Chapter 6. Encrypting cinder volumes

You can use barbican to manage your Block Storage (cinder) encryption keys. This configuration uses LUKS to encrypt the disks attached to your instances, including boot disks. Key management is transparent to the user; when you create a new volume using luks as the encryption type, cinder generates a symmetric key secret for the volume and stores it in barbican. When booting the instance (or attaching an encrypted volume), nova retrieves the key from barbican and stores the secret locally as a Libvirt secret on the Compute node.

Important

Nova formats encrypted volumes during their first use if they are unencrypted. The resulting block device is then presented to the Compute node.

Note

If you intend to update any configuration files, be aware that certain OpenStack services now run within containers; this applies to keystone, nova, and cinder, among others. As a result, there are administration practices to consider:

  • Do not update any configuration file you might find on the physical node’s host operating system, for example, /etc/cinder/cinder.conf. The containerized service does not reference this file.

  • Do not update the configuration file running within the container. Changes are lost once you restart the container.

    Instead, if you must change containerized services, update the configuration file in /var/lib/config-data/puppet-generated/, which is used to generate the container.

    For example:

  • keystone: /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf
  • cinder: /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf

  • nova: /var/lib/config-data/puppet-generated/nova/etc/nova/nova.conf

    Changes are applied after you restart the container.

  1. On nodes running the cinder-volume and nova-compute services, confirm that nova and cinder are both configured to use barbican for key management: 

    $ crudini --get /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf key_manager backend
castellan.key_manager.barbican_key_manager.BarbicanKeyManager

$ crudini --get /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf key_manager backend
castellan.key_manager.barbican_key_manager.BarbicanKeyManager

  2. Create a volume template that uses encryption. When you create new volumes they can be modeled off the settings you define here: 

    $ openstack volume type create --encryption-provider nova.volume.encryptors.luks.LuksEncryptor --encryption-cipher aes-xts-plain64 --encryption-key-size 256 --encryption-control-location front-end LuksEncryptor-Template-256
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field       | Value                                                                                                                                                                              |
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| description | None                                                                                                                                                                               |
| encryption  | cipher='aes-xts-plain64', control_location='front-end', encryption_id='9df604d0-8584-4ce8-b450-e13e6316c4d3', key_size='256', provider='nova.volume.encryptors.luks.LuksEncryptor' |
| id          | 78898a82-8f4c-44b2-a460-40a5da9e4d59                                                                                                                                               |
| is_public   | True                                                                                                                                                                               |
| name        | LuksEncryptor-Template-256                                                                                                                                                         |
+-------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

  3. Create a new volume and specify that it uses the LuksEncryptor-Template-256 settings:

    Note

    Ensure that the user creating the encrypted volume has the creator barbican role on the project. For more information, see the Grant user access to the creator role section. 

    $ openstack volume create --size 1 --type LuksEncryptor-Template-256 'Encrypted-Test-Volume'
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| attachments         | []                                   |
| availability_zone   | nova                                 |
| bootable            | false                                |
| consistencygroup_id | None                                 |
| created_at          | 2018-01-22T00:19:06.000000           |
| description         | None                                 |
| encrypted           | True                                 |
| id                  | a361fd0b-882a-46cc-a669-c633630b5c93 |
| migration_status    | None                                 |
| multiattach         | False                                |
| name                | Encrypted-Test-Volume                |
| properties          |                                      |
| replication_status  | None                                 |
| size                | 1                                    |
| snapshot_id         | None                                 |
| source_volid        | None                                 |
| status              | creating                             |
| type                | LuksEncryptor-Template-256           |
| updated_at          | None                                 |
| user_id             | 0e73cb3111614365a144e7f8f1a972af     |
+---------------------+--------------------------------------+

    The resulting secret is automatically uploaded to the barbican backend.

  4. Use barbican to confirm that the disk encryption key is present. In this example, the timestamp matches the LUKS volume creation time: 

    $ openstack secret list
+------------------------------------------------------------------------------------+------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
| Secret href                                                                        | Name | Created                   | Status | Content types                             | Algorithm | Bit length | Secret type | Mode | Expiration |
+------------------------------------------------------------------------------------+------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+
| https://192.168.123.169:9311/v1/secrets/24845e6d-64a5-4071-ba99-0fdd1046172e | None | 2018-01-22T02:23:15+00:00 | ACTIVE | {u'default': u'application/octet-stream'} | aes       |        256 | symmetric   | None | None       |
+------------------------------------------------------------------------------------+------+---------------------------+--------+-------------------------------------------+-----------+------------+-------------+------+------------+

  5. Attach the new volume to an existing instance. For example: 

    $ openstack server add volume testInstance Encrypted-Test-Volume

    The volume is then presented to the guest operating system and can be mounted using the built-in tools.

6.1. Migrate existing volume keys to Barbican

Previously, deployments might have used ConfKeyManager to manage disk encryption keys. This meant that a fixed key was generated and then stored in the nova and cinder configuration files. The key IDs can be migrated to barbican using the following procedure. This utility works by scanning the databases for encryption_key_id entries within scope for migration to barbican. Each entry gets a new barbican key ID and the existing ConfKeyManager secret is retained.

Note

Previously, you could reassign ownership for volumes encrypted using ConfKeyManager. This is not possible for volumes that have their keys managed by barbican.

Note

Activating barbican will not break your existing keymgr volumes.

After it is enabled, the migration process runs automatically, but it requires some configuration, described in the next section. The actual migration runs in the cinder-volume and cinder-backup process, and you can track the progress in the cinder log files.

  • cinder-volume - migrates keys stored in cinder’s Volumes and Snapshots tables.
  • cinder-backup - migrates keys in the Backups table.

6.1.1. Overview of the migration steps

  1. Deploy the barbican service.

  2. Add the creator role to the cinder service. For example: 

    #openstack role create creator
#openstack role add --user cinder creator  --project service
  3. Restart the cinder-volume and cinder-backup services.
  4. cinder-volume and cinder-backup automatically begin the migration process.
  5. Monitor the logs for the message indicating migration has finished and check that no more volumes are using the ConfKeyManager all-zeros encryption key ID.
  6. Remove the fixed_key option from cinder.conf and nova.conf. You must determine which nodes have this setting configured.
  7. Remove the creator role from the cinder service.

6.1.2. Behavioral differences

Barbican-managed encrypted volumes behave differently than volumes that use ConfKeyManager:

  • You cannot transfer ownership of encrypted volumes, because it is not currently possible to transfer ownership of the barbican secret.
  • Barbican is more restrictive about who is allowed to read and delete secrets, which can affect some cinder volume operations. For example, a user cannot attach, detach, or delete a different user’s volumes.

6.1.3. Reviewing the migration process

This section describes how you can view the status of the migration tasks. After you start the process, one of these entries appears in the logs. This indicates whether the migration started correctly, or it identifies the issue it encountered:

  • Not migrating encryption keys because the ConfKeyManager is still in use.
  • Not migrating encryption keys because the ConfKeyManager's fixed_key is not in use.
  • Not migrating encryption keys because migration to the 'XXX' key_manager backend is not supported. - This message is unlikely to appear; it is a safety check to handle the code ever encountering another Key Manager backend other than barbican. This is because the code only supports one migration scenario: From ConfKeyManager to barbican.
  • Not migrating encryption keys because there are no volumes associated with this host. - This may occur when cinder-volume is running on multiple hosts, and a particular host has no volumes associated with it. This arises because every host is responsible for handling its own volumes.
  • Starting migration of ConfKeyManager keys.

  • Migrating volume <UUID> encryption key to Barbican - During migration, all of the host’s volumes are examined, and if a volume is still using the ConfKeyManager’s key ID (identified by the fact that it’s all zeros (00000000-0000-0000-0000-000000000000)), then this message appears.

    • For cinder-backup, this message uses slightly different capitalization: Migrating Volume [...] or Migrating Backup [...]

  • After each host examines all of its volumes, the host displays a summary status message: 

    `No volumes are using the ConfKeyManager's encryption_key_id.`
`No backups are known to be using the ConfKeyManager's encryption_key_id.`

    You may also see the following entries:

    There are still %d volume(s) using the ConfKeyManager's all-zeros encryption key ID.There are still %d backup(s) using the ConfKeyManager’s all-zeros encryption key ID. Note that both of these messages can appear in the cinder-volume and cinder-backup logs. Whereas each service only handles the migration of its own entries, the service is aware of the the other’s status. As a result, cinder-volume knows if cinder-backup still has backups to migrate, and cinder-backup knows if the cinder-volume service has volumes to migrate.

    Although each host migrates only its own volumes, the summary message is based on a global assessment of whether any volume still requires migration This allows you to confirm that migration for all volumes is complete. Once you receive confirmation, remove the fixed_key setting from cinder.conf and nova.conf. See the Clean up the fixed keys section below for more information.

6.1.4. Troubleshooting the migration process

6.1.4.1. Role assignment

The barbican secret can only be created when the requestor has the creator role. This means that the cinder service itself requires the creator role, otherwise a log sequence similar to this will occur:

  1. Starting migration of ConfKeyManager keys.
  2. Migrating volume <UUID> encryption key to Barbican
  3. Error migrating encryption key: Forbidden: Secret creation attempt not allowed - please review your user/project privileges
  4. There are still %d volume(s) using the ConfKeyManager's all-zeros encryption key ID.

The key message is the third one: Secret creation attempt not allowed. To fix the problem, update the cinder account’s privileges:

  1. Run openstack role add --project service --user cinder creator
  2. Restart the cinder-volume and cinder-backup services.

As a result, the next attempt at migration should succeed.

6.1.5. Clean up the fixed keys

Important

The encryption_key_id was only recently added to the Backup table, as part of the Queens release. As a result, pre-existing backups of encrypted volumes are likely to exist. The all-zeros encryption_key_id is stored on the backup itself, but it won’t appear in the Backup database. As such, it is impossible for the migration process to know for certain whether a backup of an encrypted volume exists that still relies on the all-zeros ConfKeyMgr key ID.

After migrating your key IDs into barbican, the fixed key remains in the configuration files. This may present a security concern to some users, because the fixed_key value is not encrypted in the .conf files. To address this, you can manually remove the fixed_key values from your nova and cinder configurations. However, first complete testing and review the output of the log file before you proceed, because disks that are still dependent on this value will not be accessible.

  1. Review the existing fixed_key values. The values must match for both services. 

    crudini --get /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf keymgr fixed_key
crudini --get /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf keymgr fixed_key
  2. IMPORTANT: Make a backup of the existing fixed_key values. This allows you to restore the value if something goes wrong, or if you need to restore a backup that uses the old encryption key.

  3. Delete the fixed_key values: 

    crudini --del /var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf keymgr fixed_key
crudini --del /var/lib/config-data/puppet-generated/nova_libvirt/etc/nova/nova.conf keymgr fixed_key