Chapter 6. Rotating service account passwords

You can periodically rotate service account passwords to improve your security posture.

6.1. Overview of overcloud password management

OpenStack services that run on the overcloud are authenticated by their Identity service (keystone) credentials. These passwords are generated during the initial deployment process and are defined as heat parameters. For example: 

            'MistralPassword',
            'BarbicanPassword',
            'AdminPassword',
            'CeilometerMeteringSecret',
            'ZaqarPassword',
            'NovaPassword',
            'MysqlRootPassword'

You can rotate the passwords used by the service accounts by using a Workflow service (mistral) workflow. However, passwords are not rotated if they are listed in DO_NOT_ROTATE, such as Key Encrypting Keys (KEK) and Fernet keys: 

DO_NOT_ROTATE_LIST = (
    'BarbicanSimpleCryptoKek',
    'SnmpdReadonlyUserPassword',
    'KeystoneCredential0',
    'KeystoneCredential1',
    'KeystoneFernetKey0',
    'KeystoneFernetKey1',
    'KeystoneFernetKeys',
)

These passwords are on the DO_NOT_ROTATE list for the following reasons:

6.2. Rotating the passwords

Use the following procedure to rotate eligible passwords. The next time you complete a stack update by running the openstack overcloud deploy command, your rotated password changes are applied. Any passwords specified in environment files take precedence over password changes that use this method. For information about outage requirements and service impact, see Outage requirements .

Important

Do not use this procedure to rotate the swift password, because this is not currently supported.

  1. As the stack user, run the password rotation workflow. This rotates all passwords, except for those on the DO_NOT_ROTATE list: 

    $ openstack workflow execution create tripleo.plan_management.v1.rotate_passwords '{"container": "overcloud"}'

    If you want to rotate only specific passwords, you can use password_list. You can also use this method to rotate passwords on the DO_NOT_ROTATE list. For example: 

    $ openstack workflow execution create tripleo.plan_management.v1.rotate_passwords '{"container": "overcloud", "password_list": ["SaharaPassword", "ManilaPassword"]}'
    The Workflow service Mistral workflow generates new passwords for the service accounts.
  2. Run a stack update to apply the new passwords.

  3. You can retrieve and view the new passwords, by creating a workflow to retrieve the passwords, and then viewing the output:

    1. Create a new workflow to retrieve the passwords. Note the ID of the workflow: 

      $ openstack workflow execution create tripleo.plan_management.v1.get_passwords '{"container": "overcloud"}'
 +--------------------+---------------------------------------------+
 | Field              | Value                                       |
 +--------------------+---------------------------------------------+
 | ID                 | edcf9103-e1a8-42f9-85c1-e505c055e0ed        |
 | Workflow ID        | 8aa2ac9b-22ee-4e7d-8240-877237ef0d0a        |
 | Workflow name      | tripleo.plan_management.v1.rotate_passwords |
 | Workflow namespace |                                             |
 | Description        |                                             |
 | Task Execution ID  | <none>                                      |
 | Root Execution ID  | <none>                                      |
 | State              | RUNNING                                     |
 | State info         | None                                        |
 | Created at         | 2020-01-22 15:47:57                         |
 | Updated at         | 2020-01-22 15:47:57                         |
 +--------------------+---------------------------------------------+

    2. Use the workflow ID to check the workflow status. You must wait until the workflow has a state of SUCCESS before you continue: 

      $ openstack workflow execution show edcf9103-e1a8-42f9-85c1-e505c055e0ed
      +--------------------+---------------------------------------------+
      | Field              | Value                                       |
      +--------------------+---------------------------------------------+
      | ID                 | edcf9103-e1a8-42f9-85c1-e505c055e0ed        |
      | Workflow ID        | 8aa2ac9b-22ee-4e7d-8240-877237ef0d0a        |
      | Workflow name      | tripleo.plan_management.v1.rotate_passwords |
      | Workflow namespace |                                             |
      | Description        |                                             |
      | Task Execution ID  | <none>                                      |
      | Root Execution ID  | <none>                                      |
      | State              | SUCCESS                                     |
      | State info         | None                                        |
      | Created at         | 2020-01-22 15:47:57                         |
      | Updated at         | 2020-01-22 15:48:39                         |
      +--------------------+---------------------------------------------+

    3. When the workflow is complete, retrieve the passwords by using the following command: 

      openstack workflow execution output show edcf9103-e1a8-42f9-85c1-e505c055e0ed
     {
          "status": "SUCCESS",
          "message": {
              "AdminPassword": "FSn0sS1aAHp8YK2fU5niM3rxu",
              "AdminToken": "dTP0Wdy7DtblG80M54r4a2yoC",
              "AodhPassword": "fB5NQdRe37BaBVEWDHVuj4etk",
              "BarbicanPassword": "rn7yk7KPafKw2PWN71MvXpnBt",
              "BarbicanSimpleCryptoKek": "lrC3sGlV7-D7-V_PI4vbDfF1Ujm5OjnAVFcnihOpbCg=",
              "CeilometerMeteringSecret": "DQ69HdlJobhnGWoBC0jM3drPF",
              "CeilometerPassword": "qI6xOpofuiXZnG95iUe8Oxv5d",
              "CephAdminKey": "AQDGVPpdAAAAABAAZMP56/VY+zCVcDT81+TOjg==",
              "CephClientKey": "AQDGVPpdAAAAABAAanYtA0ggpcoCbS1nLeDN7w==",
              "CephClusterFSID": "141a5ede-21b4-11ea-8132-52540031f76b",
              "CephDashboardAdminPassword": "AQDGVPpdAAAAABAAKhsx630YKDhQrocS4o4KzA==",
              "CephGrafanaAdminPassword": "AQDGVPpdAAAAABAAKBojG+CO72B0TdBRR0paEg==",
              "CephManilaClientKey": "AQDGVPpdAAAAABAAA1TVHrTVCC8xQ4skG4+d5A=="
          }
      }

6.3. Outage requirements

Outage requirements and service impacts can occur when you change passwords for the overcloud service accounts.

After a password has been rotated as part of the stack update, the old password becomes invalid. As a result, services are unavailable with an HTTP 401 error for the duration that it takes for the new password to be added to the service configuration settings.

In addition, you can expect to encounter brief outages when you change passwords for the supporting services, including MySQL, RabbitMQ, and High Availability.