Certain applications can benefit from advanced storage configurations and if a charm exists for such an application Juju can declare such requirements both at deploy time and during the lifetime of the application.
The level of sophistication is limited by the charm; a charm may support multiple storage options (e.g. persistent storage, additional cache). All this allows the user to allocate resources at a granular level. This page will refer to the PostgreSQL charm and the Ceph OSD charm.
The Ceph examples used here are based on the Ceph cluster described in the document Installing Ceph.
Outside of application deployment, Juju also has a wide array of storage management abilities. Related commands are listed below, along with a brief description of each.
- Creates and attaches a storage instance to a unit.
- Attaches an existing storage instance to a unit.
- Creates or defines a storage pool.
- Detaches a storage instance from a unit. Storage is preserved.
- Imports a filesystem into a model.
- Removes a storage instance from a model. Storage is destroyed.
- Shows details of a storage instance.
- Lists all storage instances in a model.
- Lists all storage pools in a model.
Several properties are used to dictate how storage is allocated:
- 'pool': class of storage
- 'size': size of each volume
- 'count': number of volumes
The default pool (e.g. 'ebs' for AWS, 'cinder' for OpenStack) is given by:
juju model-config storage-default-block-source
These properties are specified as constraints with the
juju deploy or
juju add-storage commands.
juju deploy <charm> [--storage <label>=<pool>,<size>,<count>]
labelis a string taken from the charm itself. It encapsulates a specific storage option/feature. Sometimes called a store.
--storagemay be specified multiple times, to support multiple labels.
If at least one constraint is specified the following default values come into effect:
- 'pool' = the default pool (see above)
- 'size' = determined from the charm's minimum storage size, or 1GiB if the charm does not specify a minimum
- 'count' = the minimum number required by the charm, or '1' if the storage is optional
In the absence of any storage constraints, the storage will be put on the root filesystem.
juju add-storage <unit> <label>[=<pool>,<size>,<count>]
juju add-unit the storage parameters used are taken from the
command corresponding to the unit.
To deploy PostgreSQL with one instance (count) of 100GiB, via the charm's 'pgdata' storage label, using the default storage pool:
juju deploy postgresql --storage pgdata=100G
Assuming an AWS model, a more explicit, but equivalent, command is:
juju deploy postgresql --storage pgdata=ebs,100G,1
juju storage-pools command to list the predefined storage pools as
well as any custom ones that may have been created with the
juju create-storage-pool command:
Here is sample output for a newly-added AWS model:
Name Provider Attrs ebs ebs ebs-ssd ebs volume-type=ssd loop loop rootfs rootfs tmpfs tmpfs
Note: The name given to a default storage pool will often be the same as the name of the storage provider upon which it is based.
Depending on the storage provider (see below), custom storage pools can be created. In the case of AWS, the 'ebs' storage provider supports several configuration attributes:
- 'volume-type': volume type (i.e. magnetic, ssd, or provisioned-iops)
- 'encrypted': enable/disable disk encryption
- 'iops': IOPS per GiB
For example, to provision a 3000 IOPS volume (100GiB x 30IOPS/GiB) by first creating a custom storage pool and then having a newly deployed PostgreSQL use it for its database storage:
juju create-storage-pool iops ebs volume-type=provisioned-iops iops=30 juju deploy postgresql --storage pgdata=iops,100G
See IOPS (Wikipedia) for background information.
Most storage can be dynamically added to, and removed from, a unit. Some types of storage, however, cannot be dynamically managed. For instance, Juju cannot disassociate MAAS disks from their respective MAAS nodes. These types of static storage can only be requested at deployment time and will be removed when the machine is removed from the model.
Certain cloud providers may also impose restrictions when attaching storage. For example, attaching an EBS volume to an EC2 instance requires that they both reside within the same availability zone. If this is not the case, Juju will return an error.
When deploying an application or unit that requires storage, using machine
--to) requires that the assigned storage be dynamic. Juju will
return an error if you try to deploy a unit to an existing machine, while also
attempting to allocate static storage.
Assuming the storage provider supports it, storage can be created and attached
to a unit using
juju add-storage. Juju will ensure the storage is allowed to
attach to the unit's machine.
Charms can specify a maximum number of storage instances. In the case of the charm 'postgresql', a maximum of one is allowed for 'pgdata'. If an attempt is made to exceed it, Juju will return an error.
Dynamic storage can be detached from units using
Charms can also define a minimum number of storage instances. The postgresql charm specifies a minimum of zero for 'pgdata' whereas another charm may specify a different number. In any case, if detaching storage from a unit would bring the total number of storage instances below the minimum, Juju will return an error.
It is not possible to add new storage to a model without also attaching it to a
unit. However, with the
juju import-filesystem command, you can add storage
to a model that has been previously released from a removed model.
To create a 32GiB EBS volume and attach it to unit 'ceph-osd/0' as its OSD storage:
juju add-storage ceph-osd/0 osd-devices=ebs,32G,1
Above, the volume was created in the same availability zone as the instance (a requirement).
To detach OSD device 'osd-devices/2' from a Ceph unit:
juju detach-storage osd-devices/2
Important: Detaching storage from a unit does not destroy the storage.
As we saw, detaching storage does not destroy the storage. In addition, when a
unit is removed from a model, and the unit has dynamic storage attached, the
storage will be detached and left intact. This allows detached storage to be
re-attached to an existing unit using
juju attach-storage, or to a new unit
--attach-storage flag of
juju deploy or
Storage is destroyed (removed from the model) by first detaching it and then
If an attempt is made to either attach or remove storage that is currently in use (i.e. it is attached to a unit) Juju will return an error.
Finally, a model cannot be destroyed while storage volumes remain without
passing a special option (
--release-storage to detach all volumes and
--destroy-storage to remove all volumes). Naturally, this applies to
the removal of a controller as well.
To attach existing storage 'osd-devices/7' to existing unit 'ceph-osd/1':
juju attach-storage ceph-osd/1 osd-devices/7
To deploy PostgreSQL with (detached) existing storage 'pgdata/0':
juju deploy postgresql --attach-storage pgdata/0
-n flags cannot be used together.
To add a new Ceph OSD unit with (detached) existing storage 'osd-devices/2':
juju add-unit ceph-osd --attach-storage osd-devices/2
To destroy already detached storage 'osd-devices/3' (remove it from the model):
juju remove-storage osd-devices/3
To upgrade the OSD journal of Ceph unit 'ceph-osd/0' from magnetic to solid state (SSD) and dispose of the unneeded original journal 'osd-journals/0':
juju add-storage ceph-osd/0 osd-journals=ebs-ssd,8G,1 juju detach-storage osd-journals/0 juju remove-storage osd-journals/0
To destroy a controller (and its models) along with all existing storage volumes:
juju destroy-controller lxd-controller --destroy-all-models --destroy-storage
To destroy a model while keeping intact all existing storage volumes:
juju destroy-model default --release-storage
Assuming the above model was LXD-based, to create a new model and import the released storage volume into it, giving it a storage name of 'pgdata':
juju add-model default juju import-filesystem lxd juju:juju-7a544c-filesystem-0 pgdata
The determination of the volume ID is dependent upon cloud type. Above, the
volume ID is given by the backing LXD pool ('juju' in this case) and the volume
name (obtained with
lxc storage volume list <lxd-pool>). The LXD storage
provider and associated LXD pools are described in detail
Storage management is currently restricted to a single model, which means it is not possible to reuse storage from one model/controller in another. Also, when a model/controller is removed, all associated storage will be destroyed. Support for releasing storage from a model, and enlisting it into another, is planned for a future release.
When upgrading a charm with the juju upgrade-charm command, the existing storage constraints specified at deployment time will be preserved.
It is also possible to change the storage constraints and define new ones by
--storage flag to
juju upgrade-charm. For example, if the
'pgdata' storage option did not exist in revision 1 of the postgresql charm,
but was introduced in revision 2, when upgrading (from 1 to 2) you could do:
juju upgrade-charm postgresql --storage pgdata=10G
If such a constraint was not provided, 'rootfs' would be used (as described in the section on deploying with storage constraints).
Warning: Specifying new constraints may be necessary when upgrading to a revision of a charm that introduces new, required, storage options.
There are several cloud-independent storage providers, which are available to all types of models:
Block-type, creates a file on the unit's root filesystem, associates a loop device with it. The loop device is provided to the charm.
Filesystem-type, creates a sub-directory on the unit's root filesystem for the unit/charm to use.
Filesystem-type, creates a temporary file storage facility that appears as a mounted file system but is stored in volatile memory.
Loop devices require extra configuration to be used within LXD. For that, please refer to Loop devices and LXD (below).
AWS-based models have access to the 'ebs' storage provider, which supports the following pool attributes:
Specifies the EBS volume type to create. You can use either the EBS volume type names, or synonyms defined by Juju (in parentheses):
- standard (magnetic)
- gp2 (ssd)
- io1 (provisioned-iops)
The default volume type is 'standard'. Since the default pool is 'ebs' the default volume for AWS will be magnetic.
The number of IOPS for provisioned-iops volume types. There are restrictions on minimum and maximum IOPS, as a ratio of the size of volumes. See Provisioned IOPS (SSD) Volumes for more information.
Boolean (true|false); indicates whether created volumes are encrypted.
Recall that pool 'ebs-ssd' is provided for all EC2 environments. This is the easiest way to get SSD-based volumes; the pool defaults the volume type to ssd/gp2. The alternate way would be to create a new pool with a 'volume-type' of 'ssd'. For example:
juju create-storage-pool myssd-pool ebs volume-type=ssd juju deploy postgresql --storage pgdata=myssd-pool,32G
For detailed information regarding EBS volume types, see the AWS EBS documentation.
OpenStack-based models have access to the 'cinder' storage provider. The Cinder provider does not currently have any specific configuration options.
MAAS has support for discovering information about machine disks, and an API for acquiring nodes with specified disk parameters. Juju's MAAS provider has an integrated 'maas' storage provider. This storage provider is static-only; it is only possible to deploy charms using 'maas' storage to a new machine in MAAS, and not to an existing machine, as described in the section on dynamic storage.
The MAAS provider currently has a single configuration attribute:
A comma-separated list of tags to match on the disks in MAAS. For example, you might tag some disks as 'fast'; you can then create a storage pool in Juju that will draw from the disks with those tags.
Azure-based models have access to the 'azure' storage provider. The Azure provider does not currently have any specific configuration options.
The Microsoft Azure provider does not currently have any storage configuration.
Google-based models have access to the 'gce' storage provider. The GCE provider does not currently have any specific configuration options.
Oracle-based models have access to the 'oracle' storage provider. The Oracle provider currently supports a single pool configuration attribute:
Volume type, a value of 'default' or 'latency'. Use 'latency' for low-latency, high IOPS requirements, and 'default' otherwise.
For convenience, the Oracle provider registers two predefined pools:
- 'oracle' (volume type is 'default')
- 'oracle-latency' (volume type is 'latency').
Note: To get an LXD version (at least 2.16) on either Ubuntu 14.04 LTS (Trusty) or Ubuntu 16.04 LTS (Xenial) that has the 'lxd' storage provider feature the LXD:stable PPA will be required.
LXD-based models have access to the 'lxd' storage provider. The LXD provider has two configuration options:
This is the LXD storage driver (e.g. zfs, btrfs, lvm, ceph).
The name to give to the corresponding storage pool in LXD.
Any other parameters will be passed to LXD (e.g. zfs.pool_name). See upstream LXD storage configuration for LXD storage parameters.
Every LXD-based model comes with a minimum of one LXD-specific Juju storage
pool called 'lxd'. If ZFS and/or BTRFS are present when the controller is
created then pools 'lxd-zfs' and/or 'lxd-btrfs' will also be available. The
following output to the
juju storage-pools command shows all three Juju
Name Provider Attrs loop loop lxd lxd lxd-btrfs lxd driver=btrfs lxd-pool=juju-btrfs lxd-zfs lxd driver=zfs lxd-pool=juju-zfs zfs.pool_name=juju-lxd rootfs rootfs tmpfs tmpfs
As can be inferred from the above output, for each Juju storage pool based on the 'lxd' storage provider there is a LXD storage pool that gets created. It is these LXD pools that will house the actual volumes.
The LXD pool corresponding to the Juju 'lxd' pool doesn't get created until the
latter is used for the first time (typically via the
juju deploy command). It
is called simply 'juju'.
lxc storage list is used to list LXD storage pools. A full
"contingent" of LXD non-custom storage pools would like like this:
+------------+-------------+--------+------------------------------------+---------+ | NAME | DESCRIPTION | DRIVER | SOURCE | USED BY | +------------+-------------+--------+------------------------------------+---------+ | default | | dir | /var/lib/lxd/storage-pools/default | 1 | +------------+-------------+--------+------------------------------------+---------+ | juju | | dir | /var/lib/lxd/storage-pools/juju | 0 | +------------+-------------+--------+------------------------------------+---------+ | juju-btrfs | | btrfs | /var/lib/lxd/disks/juju-btrfs.img | 0 | +------------+-------------+--------+------------------------------------+---------+ | juju-zfs | | zfs | /var/lib/lxd/disks/juju-zfs.img | 0 | +------------+-------------+--------+------------------------------------+---------+
To be clear, the three Juju-related pools above are for storing volumes that Juju applications can use. The fourth 'default' pool is the standard LXD storage pool where the actual containers (operating systems) live.
To deploy an application, refer to the pool as usual. Here we deploy PostgreSQL using the 'lxd' Juju storage pool, which, in turn, uses the 'juju' LXD storage pool:
juju deploy postgresql --storage pgdata=lxd,8G
See Using LXD as a cloud for how to use LXD in conjunction with Juju, including the use of ZFS as an alternative filesystem.
LXD (localhost) does not officially support attaching loopback devices for storage out of the box. However, with some configuration you can make this work.
Each container uses the 'default' LXD profile, but also uses a model-specific
profile with the name
juju-<model-name>. Editing a profile will affect all of
the containers using it, so you can add loop devices to all LXD containers by
editing the 'default' profile, or you can scope it to a model.
To add loop devices to your container, add entries to the 'default', or
model-specific, profile, with
lxc profile edit <profile>:
... devices: loop-control: major: "10" minor: "237" path: /dev/loop-control type: unix-char loop0: major: "7" minor: "0" path: /dev/loop0 type: unix-block loop1: major: "7" minor: "1" path: /dev/loop1 type: unix-block ... loop9: major: "7" minor: "9" path: /dev/loop9 type: unix-block
Doing so will expose the loop devices so the container can acquire them via the
losetup command. However, it is not sufficient to enable the container to
mount filesystems onto the loop devices. One way to achieve that is to make the
container "privileged" by adding:
config: security.privileged: "true"
For guidance on how to create a charm that uses these storage features see Writing charms that use storage.