Chapter 6. Upgrading the overcloud

This section upgrades the overcloud. This includes the following workflow:

  • Running the fast forward upgrade preparation command
  • Running the fast forward upgrade command
  • Upgrading the Controller nodes
  • Upgrading the Compute nodes
  • Upgrading the Ceph Storage nodes
  • Finalizing the fast forward upgrade.

Once you begin this workflow, you should not expect full control over the overcloud’s OpenStack services until completing all steps. This means workloads are unmanageable until all nodes have been successfully upgraded to OpenStack Platform 13. The workloads themselves will remain unaffected and continue to run. Changes or additions to any overcloud workloads need to wait until the fast forward upgrade is completed.

6.1. Performing the fast forward upgrade of the overcloud

The fast forward upgrade requires running two commands that perform the following tasks:

  • Updates the overcloud plan to OpenStack Platform 13.
  • Prepares the nodes for the fast forward upgrade.
  • Runs through upgrade steps of each subsequent version within the fast forward upgrade, including:

    • Version-specific tasks for each OpenStack Platform service.
    • Changing the repository to each sequential OpenStack Platform version within the fast forward upgrade.
    • Updates certain packages required for upgrading the database.
    • Performing database upgrades for each subsequent version.
  • Prepares the overcloud for the final upgrade to OpenStack Platform 13.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the fast forward upgrade preparation command with all relevant options and environment files appropriate to your deployment:

    $ openstack overcloud ffwd-upgrade prepare \
        --templates \
        -e /home/stack/templates/overcloud_images.yaml \
        -e /home/stack/templates/deprecated_cli_options.yaml \
        -e /home/stack/templates/custom_repositories_script.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
        -e /home/stack/templates/ceph-customization.yaml \
        -e <ENVIRONMENT FILE>

    Include the following options relevant to your environment:

    • Custom configuration environment files (-e). For example:

      • The environment file with your container image locations (overcloud_images.yaml). Note that the upgrade command might display a warning about using the --container-registry-file. You can ignore this warning as this option is deprecated in favor of using -e for the container image environment file.
      • If applicable, an environment file that maps deprecated CLI options to Heat parameters using deprecated_cli_options.yaml.
      • If applicable, an environment file with your custom repository script using custom_repositories_script.yaml.
      • If using Ceph Storage nodes, the relevant environment files.
      • Any additional environment files relevant to your environment.
    • If using a custom stack name, pass the name with the --stack option.
    • If applicable, your custom roles (roles_data) file using --roles-file.
    Important

    A prompt will ask if you are sure you want to perform the ffwd-upgrade command. Enter yes.

  3. The overcloud plan updates to the OpenStack Platform 13 version. Wait until the fast forward upgrade preparation completes.
  4. Run the fast forward upgrade command:

    $ openstack overcloud ffwd-upgrade run
    • If using a custom stack name, pass the name with the --stack option.
    Important

    A prompt will ask if you are sure you want to perform the ffwd-upgrade command. Enter yes.

  5. Wait until the fast forward upgrade completes.

At this stage:

  • Workloads are still running
  • The overcloud database has been upgraded to the OpenStack Platform 12 version
  • All overcloud services are disabled
  • Ceph Storage nodes are still at version 2

This means the overcloud is now at a state to perform the standard upgrade steps to reach OpenStack Platform 13.

6.2. Upgrading all Controller nodes

This process upgrades all the Controller nodes to OpenStack Platform 13. The process involves running the openstack overcloud upgrade run command and including the --nodes Controller option to restrict operations to the Controller nodes only.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the upgrade command:

    $ openstack overcloud upgrade run --nodes Controller --skip-tags validation
    Note

    The command uses --skip-tags validation because OpenStack Platform services are inactive on the overcloud and cannot be validated.

    • If using a custom stack name, pass the name with the --stack option.
  3. Wait until the Controller node upgrade completes.

At this stage:

  • Workloads are still running
  • The overcloud database has been upgraded to the OpenStack Platform 13 version
  • The Controller nodes have been upgraded to OpenStack Platform 13
  • All Controller services are enabled
  • The Compute nodes still require an upgrade
  • Ceph Storage nodes are still at version 2 and require an upgrade
Warning

Although Controller services are enabled, do not perform any workload operations while Compute node and Ceph Storage services are disabled. This can cause orphaned virtual machines. Wait until the entire environment is upgraded.

6.3. Upgrading test Compute nodes

This process upgrades Compute nodes selected for testing. The process involves running the openstack overcloud upgrade run command and including the --nodes option to restrict operations to the test nodes only. This procedure uses --nodes compute-0 as an example in commands.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the upgrade command:

    $ openstack overcloud upgrade run --nodes compute-0 --skip-tags validation
    Note

    The command uses --skip-tags validation because OpenStack Platform services are inactive on the overcloud and cannot be validated.

    • If using a custom stack name, pass the name with the --stack option.
  3. Wait until the test node upgrade completes.

6.4. Upgrading all Compute nodes

Important

This process upgrades all remaining Compute nodes to OpenStack Platform 13. The process involves running the openstack overcloud upgrade run command and including the --nodes Compute option to restrict operations to the Compute nodes only.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the upgrade command:

    $ openstack overcloud upgrade run --nodes Compute --skip-tags validation
    Note

    The command uses --skip-tags validation because OpenStack Platform services are inactive on the overcloud and cannot be validated.

    • If you are using a custom stack name, pass the name with the --stack option.
    • If you are using custom Compute roles, ensure that you include the role names with the --nodes option.
  3. Wait until the Compute node upgrade completes.

At this stage:

  • Workloads are still running
  • The Controller nodes and Compute nodes have been upgraded to OpenStack Platform 13
  • Ceph Storage nodes are still at version 2 and require an upgrade

6.5. Upgrading all Ceph Storage nodes

Important

This process upgrades the Ceph Storage nodes. The process involves:

  • Running the openstack overcloud upgrade run command and including the --nodes CephStorage option to restrict operations to the Ceph Storage nodes only.
  • Running the openstack overcloud ceph-upgrade run command to perform an upgrade to a containerized Red Hat Ceph Storage 3 cluster.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the upgrade command:

    $ openstack overcloud upgrade run --nodes CephStorage --skip-tags validation
    Note

    The command uses --skip-tags validation because OpenStack Platform services are inactive on the overcloud and cannot be validated.

    • If using a custom stack name, pass the name with the --stack option.
  3. Wait until the node upgrade completes.
  4. Run the Ceph Storage upgrade command. For example:

    $ openstack overcloud ceph-upgrade run \
        --templates \
        -e <ENVIRONMENT FILE> \
        -e /home/stack/templates/overcloud_images.yaml \
        -e /home/stack/templates/deprecated_cli_options.yaml \
        -e /home/stack/templates/custom_repositories_script.yaml
        -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
        -e /home/stack/templates/ceph-customization.yaml \
        --ceph-ansible-playbook '/usr/share/ceph-ansible/infrastructure-playbooks/switch-from-non-containerized-to-containerized-ceph-daemons.yml,/usr/share/ceph-ansible/infrastructure-playbooks/rolling_update.yml'

    Include the following options relevant to your environment:

    • Custom configuration environment files (-e). For example:

      • The environment file with your container image locations (overcloud_images.yaml). Note that the upgrade command might display a warning about using the --container-registry-file. You can ignore this warning as this option is deprecated in favor of using -e for the container image environment file.
      • If applicable, an environment file that maps deprecated CLI options to Heat parameters using deprecated_cli_options.yaml.
      • If applicable, an environment file with your custom repository script using custom_repositories_script.yaml.
      • The relevant environment files for your Ceph Storage nodes.
      • Any additional environment files relevant to your environment.
    • If using a custom stack name, pass the name with the --stack option.
    • If applicable, your custom roles (roles_data) file using --roles-file.
    • The following ansible playbooks:
    • /usr/share/ceph-ansible/infrastructure-playbooks/switch-from-non-containerized-to-containerized-ceph-daemons.yml
    • /usr/share/ceph-ansible/infrastructure-playbooks/rolling_update.yml
  5. Wait until the Ceph Storage node upgrade completes.

6.6. Upgrading hyperconverged nodes

If you are using only hyperconverged nodes from the ComputeHCI role, and are not using dedicated compute nodes or dedicated Ceph nodes, complete the following procedure to upgrade your nodes:

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the upgrade command:

    $ openstack overcloud upgrade run --roles ComputeHCI

    If you are using a custom stack name, pass the name to the upgrade command with the --stack option.

  3. Run the Ceph Storage upgrade command. For example:

    $ openstack overcloud ceph-upgrade run \
        --templates \
        -e /home/stack/templates/overcloud_images.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
        -e /home/stack/templates/ceph-customization.yaml \
        -e <ENVIRONMENT FILE>

    Include the following options relevant to your environment:

    • Custom configuration environment files (-e). For example:

      • The environment file with your container image locations (overcloud_images.yaml). Note that the upgrade command might display a warning about using the --container-registry-file. You can ignore this warning as this option is deprecated in favor of using -e for the container image environment file.
      • If applicable, an environment file that maps deprecated CLI options to Heat parameters using deprecated_cli_options.yaml.
      • If applicable, an environment file with your custom repository script using custom_repositories_script.yaml.
      • The relevant environment files for your Ceph Storage nodes.
    • If using a custom stack name, pass the name with the --stack option.
    • If applicable, your custom roles (roles_data) file using --roles-file.
    • The following ansible playbooks:
    • /usr/share/ceph-ansible/infrastructure-playbooks/switch-from-non-containerized-to-containerized-ceph-daemons.yml
    • /usr/share/ceph-ansible/infrastructure-playbooks/rolling_update.yml
  4. Wait until the Ceph Storage node upgrade completes.

6.7. Upgrading mixed hyperconverged nodes

If you are using dedicated compute nodes or dedicated ceph nodes in addition to hyperconverged nodes like the ComputeHCI role, complete the following procedure to upgrade your nodes:

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the upgrade command for the Compute node:

    $ openstack overcloud upgrade run --roles Compute
    If using a custom stack name, pass the name with the --stack option.
  3. Wait until the node upgrade completes.
  4. Run the upgrade command for the ComputeHCI node:

    $ openstack overcloud upgrade run --roles ComputeHCI
    If using a custom stack name, pass the name with the --stack option.
  5. Wait until the node upgrade completes.
  6. Run the upgrade command for the Ceph Storage node:

    $ openstack overcloud upgrade run --roles CephStorage
  7. Wait until the Ceph Storage node upgrade completes.
  8. Run the Ceph Storage upgrade command. For example:

    $ openstack overcloud ceph-upgrade run \
        --templates \
        -e /home/stack/templates/overcloud_images.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
        -e /home/stack/templates/ceph-customization.yaml \
        -e <ENVIRONMENT FILE>

    Include the following options relevant to your environment:

    • Custom configuration environment files (-e). For example:

      • The environment file with your container image locations (overcloud_images.yaml). Note that the upgrade command might display a warning about using the --container-registry-file. You can ignore this warning as this option is deprecated in favor of using -e for the container image environment file.
      • If applicable, an environment file that maps deprecated CLI options to Heat parameters using deprecated_cli_options.yaml.
      • If applicable, an environment file with your custom repository script using custom_repositories_script.yaml.
      • The relevant environment files for your Ceph Storage nodes.
      • Any additional environment files relevant to your environment.
    • If using a custom stack name, pass the name with the --stack option.
    • If applicable, your custom roles (roles_data) file using --roles-file.
    • The following ansible playbooks:
    • /usr/share/ceph-ansible/infrastructure-playbooks/switch-from-non-containerized-to-containerized-ceph-daemons.yml
    • /usr/share/ceph-ansible/infrastructure-playbooks/rolling_update.yml
  9. Wait until the Ceph Storage node upgrade completes.

6.8. Upgrading nodes that use custom roles

This process upgrades the custom roles that you specify to OpenStack Platform 13. The process involves running the openstack overcloud upgrade run command and including the --roles option to restrict operations to the custom roles.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the upgrade command for the custom role:

    $ openstack overcloud upgrade run --roles <CUSTOM_ROLE>

    Replace <CUSTOM_ROLE> with the name of the role that you want to upgrade. If you are using a custom stack name, pass the name with the --stack option.

  3. Wait until the node upgrade completes.

At this stage:

  • All nodes have been upgraded to OpenStack Platform 13 and workloads are still running

Although the environment is now upgraded, you must perform one last step to finalize the upgrade.

6.9. Finalizing the fast forward upgrade

The fast forward upgrade requires a final step to update the overcloud stack. This ensures the stack’s resource structure aligns with a regular deployment of OpenStack Platform 13 and allows you to perform standard openstack overcloud deploy functions in the future.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the fast forward upgrade finalization command:

    $ openstack overcloud ffwd-upgrade converge \
        --templates \
        -e /home/stack/templates/overcloud_images.yaml \
        -e /home/stack/templates/deprecated_cli_options.yaml \
        -e /home/stack/templates/custom_repositories_script.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
        -e /home/stack/templates/ceph-customization.yaml \
        -e <OTHER ENVIRONMENT FILES>

    Include the following options relevant to your environment:

    • Custom configuration environment files (-e). For example:

      • The environment file with your container image locations (overcloud_images.yaml). Note that the upgrade command might display a warning about using the --container-registry-file. You can ignore this warning as this option is deprecated in favor of using -e for the container image environment file.
      • If applicable, an environment file that maps deprecated CLI options to Heat parameters using deprecated_cli_options.yaml.
      • If applicable, an environment file with your custom repository script using custom_repositories_script.yaml.
      • If using Ceph Storage nodes, the relevant environment files.
      • Any additional environment files relevant to your environment.
    • If using a custom stack name, pass the name with the --stack option.
    • If applicable, your custom roles (roles_data) file using --roles-file.
    Important

    A prompt will ask if you are sure you want to perform the ffwd-upgrade command. Enter yes.

  3. Wait until the fast forward upgrade finalization completes.

6.10. Next Steps

The overcloud upgrade is complete. You can now perform any relevant post-upgrade overcloud configuration using the steps in Chapter 7, Executing Post Upgrade Steps. For future deployment operations, make sure to include all environment files relevant to your OpenStack Platform 13 environment, including new environment files created or converted during the upgrade.