Red Hat Training

A Red Hat training course is available for Red Hat OpenStack Platform

Deploy Fernet on the Overcloud

Red Hat OpenStack Platform 10

Deploy Fernet on the Red Hat OpenStack Platform director overcloud

OpenStack Documentation Team

Abstract

Deploy Fernet on the Red Hat OpenStack Platform director overcloud.

Chapter 1. Deploy Fernet on the Overcloud

This chapter describes how to configure your Overcloud to use the Fernet token provider.

  • Key Management - This example uses keystone-manage to generate the overcloud Fernet keys on the undercloud. These keys will not actually be used by the Undercloud since it is configured to use the UUID token format by default. If you do configure the undercloud to use the Fernet token format after following the procedure in this document, it will use the same keys as the overcloud (which may not be desirable).
  • Swift Artifacts - This implementation uses Heat swift artifacts, which puts a copy of the Fernet key directory on every node in your deployment (not just the Controller node). You will need to consider whether this outcome is acceptable for your deployment requirements.

1.1. Prepare the Fernet Keys

This section generates the Fernet keys on the undercloud, and uploads them into swift.

1. On the undercloud node, use keystone_manage to generate Fernet keys: 

$ . ~/stackrc
$ sudo keystone-manage fernet_setup --keystone-user keystone --keystone-group keystone

2. Create a tar file containing the Fernet keys: 

$ sudo tar -zcf keystone-fernet-keys.tar.gz -P /etc/keystone/fernet-keys
Note

the keys in the controller nodes should not be changed manually. All controller nodes should have the exact same set of Fernet keys, otherwise a token generated by one controller won’t be accepted by the others.

3. Upload the Fernet keys as swift artifacts: 

$ upload-swift-artifacts -f keystone-fernet-keys.tar.gz

1.2. Configure the Overcloud to use Fernet

This section creates a YAML file that configures keystone to use fernet as the token provider. This setting is then applied to your existing overcloud in a later step.

1. Create a file named fernet.yaml that contains the required token_provider setting: 

parameter_defaults:
  controllerExtraConfig:
    keystone::token_provider: 'fernet'

2. Deploy the overcloud, including the fernet.yaml file that was created in the previous step. For example: 

source /home/stack/stackrc
openstack overcloud deploy --templates -e /home/stack/fernet.yaml
Note

If re-deploying the overcloud in the future, you will need to ensure that you still include fernet.yaml, to prevent the token provider from being re-configured to use a different format.

The process may take some time to complete.

1.3. Review the Fernet Deployment

Review the overcloud controller configuration to confirm that the process was successful:

1. Retrieve the IP address of the controller node: 

$ openstack server list
+--------------------------------------+-------------------------+--------+---------------------+
| ID                                   | Name                    | Status | Networks            |
+--------------------------------------+-------------------------+--------+---------------------+
| 756fbd73-e47b-46e6-959c-e24d7fb71328 | overcloud-controller-0  | ACTIVE | ctlplane=192.0.2.16 |
| 62b869df-1203-4d58-8e45-fac6cd4cfbee | overcloud-novacompute-0 | ACTIVE | ctlplane=192.0.2.8  |
+--------------------------------------+-------------------------+--------+---------------------+

2. SSH to the controller: 

$ ssh heat-admin@192.0.2.16
Last login: Tue Sep  6 00:09:59 2016 from 192.0.2.1

3. Retrieve the values of the token driver and provider settings: 

$ sudo crudini --get /etc/keystone/keystone.conf token driver
sql
$ sudo crudini --get /etc/keystone/keystone.conf token provider
fernet

4. Test the Fernet provider: 

$ openstack token issue
WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| expires | 2016-09-20 05:26:17+00:00 |
| id | gAAAAABX4LppE8vaiFZ992eah2i3edpO1aDFxlKZq6a_RJzxUx56QVKORrmW0-oZK3-Xuu2wcnpYq_eek2SGLz250eLpZOzxKBR0GsoMfxJU8mEFF8NzfLNcbuS-iz7SV-N1re3XEywSDG90JcgwjQfXW-8jtCm-n3LL5IaZexAYIw059T_-cd8 |
| project_id | 26156621d0d54fc39bf3adb98e63b63d |
| user_id | 397daf32cadd490a8f3ac23a626ac06c |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

The result should include the long Fernet token. This token will still be shorter in length than the PKI token.

Note

The keys used to sign tokens are now available in the undercloud’s swift. The keys should remain in swift in case you need to deploy a new controller, however, you can delete them using the swift command, if needed: 

swift delete overcloud-artifacts keystone-fernet-keys.tar.gz

Chapter 2. Rotate the Fernet keys

It is recommended that you rotate your Fernet keys regularly, as a compromised keystone key can allow an attacker to generate their own tokens and subsequently grant themselves access to a project. During the key rotation process, the primary key is relegated to secondary key status, and a new primary key is issued, thereby reducing the value of a compromised primary key. Secondary keys can only be used to decrypt tokens that were created with previous primary keys, and cannot issue new ones.

For more information on Fernet keys, see https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/14/html-single/security_and_hardening_guide/index#fernet_tokens.

Fernet uses three types of keys, which are stored in /etc/keystone/fernet-keys. The highest-numbered file contains the primary key, which is used to generate new tokens and decrypt existing fernet tokens.

  • 0 - Contains the staged key, and will always be numbered 0. This key will be promoted to a primary key during the next rotation.
  • 1 and 2 - Contain the secondary keys.
  • 3 - Contains the primary key. This number will increment each time the keys are rotated, with the highest number always serving as the primary key.

2.1. Set the maximum number of fernet tokens

The maximum number of active keys can be determined with the following calculation: fernet-keys = token-validity(hours) / rotation-time(hours) + 2. For example, to enable 24 hours token-validity, rotation every 12 hours will resemble: 24/12 + 2 = 4.

You can configure director to manage the overcloud’s Fernet keys by adding a fernet_max_active_keys setting to the controllerExtraConfig section of your environment file:

  1. To set the maximum active keys for Fernet, add the following to your environment file: 

    parameter_defaults:
(...)
  controllerExtraConfig:
    keystone::fernet_max_active_keys: 4
Note

This approach has been replaced by a Mistral workflow in Red Hat OpenStack Platform 12 and higher.

2.2. Rotate the keys

This procedure creates a script that will rotate the Fernet tokens in your deployment, then copy them to your other controller nodes.

  1. Review the existing Fernet keys: 

    [heat-admin@overcloud-controller-0 ~]$ sudo ls /etc/keystone/fernet-keys
0  1  2
    • 0 - Contains the staged key, (which becomes the next primary key) and will always be numbered 0.
    • 1 - Contains the secondary key.

    • 2 - Contains the primary key. This number will increment each time the keys are rotated, with the highest number always serving as the primary key.

      Note
      • The maximum number of keys is determined by the fernet_max_active_keys property, using 4 by default.
      • The keys are propagated across all controllers.

  2. Create a script on the undercloud node: 

    $ sudo vi /usr/local/bin/rotate-fernet-tokens.sh

  3. Add the following contents to the script: 

    #!/bin/bash

source /home/stack/stackrc

controller_name="overcloud-controller-"

tmp_dir=/tmp/fernet_keys

controller0_ip=`openstack server show controller-0 -f value -c addresses|sed s/ctlplane=//g`
controller1_ip=`openstack server show controller-1 -f value -c addresses|sed s/ctlplane=//g`
controller2_ip=`openstack server show controller-2 -f value -c addresses|sed s/ctlplane=//g`

SSH="ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"
SCP="scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"

mkdir -p $tmp_dir
cd $tmp_dir

# rotate fernet tokens on controller-0, tar them up and copy them to the local node
$SSH heat-admin@${controller0_ip} "sudo keystone-manage fernet_rotate --keystone-user keystone --keystone-group keystone"
$SSH heat-admin@${controller0_ip} "sudo rm -f /tmp/fernet-keys.tar ; sudo find /etc/keystone/fernet-keys -maxdepth 1 -type f -execdir sudo tar -rf /tmp/fernet-keys.tar {} \; ; sudo chown heat-admin. /tmp/fernet-keys.tar && sudo tar -tf /tmp/fernet-keys.tar"
$SCP heat-admin@${controller0_ip}:/tmp/fernet-keys.tar .


# copy fernet tokens and display rotated keys on controller-1
$SCP fernet-keys.tar heat-admin@${controller1_ip}:
$SSH heat-admin@${controller1_ip} "sudo find /etc/keystone/fernet-keys -maxdepth 1 -type f | sudo xargs -I {} rm -f {} ; sudo tar -xf /home/heat-admin/fernet-keys.tar -C /etc/keystone/fernet-keys ; sudo chown keystone. /etc/keystone/fernet-keys -R ; sudo ls -al /etc/keystone/fernet-keys"


# copy fernet tokens and display rotated keys on controller-2
$SCP fernet-keys.tar heat-admin@${controller2_ip}:
$SSH heat-admin@${controller2_ip} "sudo find /etc/keystone/fernet-keys -maxdepth 1 -type f | sudo xargs -I {} rm -f {} ; sudo tar -xf /home/heat-admin/fernet-keys.tar -C /etc/keystone/fernet-keys ; sudo chown keystone. /etc/keystone/fernet-keys -R ; sudo ls -al /etc/keystone/fernet-keys"


# reload httpd
$SSH heat-admin@${controller0_ip} sudo systemctl reload httpd
$SSH heat-admin@${controller1_ip} sudo systemctl reload httpd
$SSH heat-admin@${controller2_ip} sudo systemctl reload httpd
    Note

    This script will restart the http service on your controllers. This will result in an outage to the horizon and keystone services while the service restarts.

    Note

    This script assumes you have three controller nodes. You will need to update this script if your deployment uses a different number of controllers.

  4. Make the script executable: 

    [stack@director ~]$ sudo chmod +x /usr/local/bin/rotate-fernet-tokens.sh

  5. Rotate the Fernet keys: 

    $ bash /usr/local/bin/rotate-fernet-tokens.sh

  6. On a keystone node, review the number of Fernet keys and compare with the previous result: 

    [heat-admin@overcloud-controller-0 ~]$ sudo ls /var/lib/config-data/puppet-generated/keystone/etc/keystone/fernet-keys
0  1  2  3
    • 0 - Contains the staged key, and will always be numbered 0. This key will be promoted to a primary key during the next rotation.
    • 1 and 2 - Contain the secondary keys.
    • 3 - Contains the primary key. This number will increment each time the keys are rotated, with the highest number always serving as the primary key.

  7. Consider configuring a cronjob to regularly execute the script from the undercloud. This example runs the script every 12 hours: 

    $ echo "* * * * * stack bash /usr/local/bin/rotate-fernet-tokens.sh" | sudo tee /etc/cron.d/1rotate-fernet-tokens

Legal Notice

Copyright © 2019 Red Hat, Inc.
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.