Chapter 5. Upgrading the Overcloud

This process upgrades the overcloud.

Prerequisites

  • You have upgraded the undercloud to the latest version.
  • You have prepared your custom environment files to accommodate the changes in the upgrade.

5.1. Upgrading the Overcloud Nodes

The major-upgrade-composable-steps-docker.yaml environment file upgrades all composable services on all custom roles, except for any roles with disable_upgrade_deployment: True in the roles_data file. These nodes are updated with a separate process.

Prerequisites

  • You have upgraded the undercloud to the latest version.
  • You have prepared your custom environment files to accommodate the changes in the upgrade.

Procedure

  1. Run the openstack overcloud deploy command and include:

    • All options and custom environment files relevant to your environment, such as network isolation and storage.
    • The overcloud_images.yaml environment file generated in Section 4.2, “Preparing for Containerized Services”.
    • The major-upgrade-composable-steps-docker.yaml environment file.

      For example:

    $ openstack overcloud deploy --templates \
      -e /home.stack/templates/node_count.yaml \
      -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
      -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
      -e /home/stack/templates/network_environment.yaml \
      -e /home/stack/templates/overcloud_images.yaml  \
      -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps-docker.yaml \
      --ntp-server pool.ntp.org
  2. Wait until the overcloud updates with the new environment file’s configuration.

    Important

    The upgrade disables the OpenStack Networking (neutron) server and L3 Agent. This means you cannot create new routers during this step. You can still access instances during this period.

  3. Check if all services are active. For example, to check services on a Controllor node:

    [stack@director ~]$ ssh heat-admin@192.168.24.10
    [heat-admin@overcloud-controller-0 ~]$ sudo pcs status
    [heat-admin@overcloud-controller-0 ~]$ sudo docker ps

Related Information

  • If you encounter any issues after completing this step, please contact Red Hat and request guidance and assistance.

5.2. Upgrading the Object Storage Nodes

Standalone Object Storage nodes are not included in the main overcloud node upgrade process because you need to update each node individually to retain the services' activity. The director contains a script to execute the upgrade on an individual Object Storage nodes.

Prerequisites

  • You have previously run openstack overcloud deploy with the major-upgrade-composable-steps-docker.yaml environment file. This upgrades the main custom roles and their composable services.

Procedure

  1. Obtain a list of Object Storage nodes:

    $ openstack server list -c Name -f value --name objectstorage
  2. Perform the following steps for each Object Storage node in the list:

    1. Run the upgrade-non-controller.sh script using the node name to identify the node to upgrade:

      $ upgrade-non-controller.sh --upgrade overcloud-objectstorage-0
      Note

      If using pre-provisioned node infrastructure, see Section 4.9, “Preparing for Pre-Provisioned Nodes Upgrade” for changes with this command.

    2. Wait until the Object Storage node completes the upgrade.
    3. Reboot the Object Storage node:

      $ openstack server reboot overcloud-objectstorage-0
    4. Wait until the Object Storage node completes the reboot.

Related Information

  • If you encounter any issues after completing this step, please contact Red Hat and request guidance and assistance.

5.3. Upgrading the Compute Nodes

Compute nodes are not included in the main overcloud node upgrade process. To ensure maximum uptime of instances, you migrate each instance from a Compute node before upgrading the node. This means the Compute node upgrade process involves the following steps:

Prerequisites

  • You have previously run 'openstack overcloud deploy` with the major-upgrade-composable-steps-docker.yaml environment file. This upgrades the main custom roles and their composable services.

Procedure

Select a Compute node to upgrade:

  1. List all Compute nodes:

    $ source ~/stackrc
    $ openstack server list -c Name -f value --name compute
  2. Select a Compute node to upgrade and note its UUID and name.

Migrate instances to another Compute node:

  1. From the undercloud, select a Compute Node to reboot and disable it:

    $ source ~/overcloudrc
    (overcloud) $ openstack compute service list
    (overcloud) $ openstack compute service set [hostname] nova-compute --disable
  2. List all instances on the Compute node:

    (overcloud) $ openstack server list --host [hostname] --all-projects
  3. Use one of the following commands to migrate your instances:

    1. Migrate the instance to a specific host of your choice:

      (overcloud) $ openstack server migrate [instance-id] --live [target-host]--wait
    2. Let nova-scheduler automatically select the target host:

      (overcloud) $ nova live-migration [instance-id]
    3. Live migrate all instances at once:

      $ nova host-evacuate-live [hostname]
      Note

      The nova command might cause some deprecation warnings, which are safe to ignore.

  4. Wait until migration completes.
  5. Confirm the migration was successful:

    (overcloud) $ openstack server list --host [hostname] --all-projects
  6. Continue migrating instances until none remain on the chosen Compute Node.

Upgrade the empty Compute node:

  1. Run the upgrade-non-controller.sh script using the node name to identify the node to upgrade:

    $ upgrade-non-controller.sh --upgrade overcloud-compute-0
    Note

    If using pre-provisioned node infrastructure, see Section 4.9, “Preparing for Pre-Provisioned Nodes Upgrade” for changes with this command.

  2. Wait until the Compute node completes the upgrade.

Reboot and enable the upgraded Compute node:

  1. Log into the Compute Node and reboot it:

    [heat-admin@overcloud-compute-0 ~]$ sudo reboot
  2. Wait until the node boots.
  3. Enable the Compute Node again:

    $ source ~/overcloudrc
    (overcloud) $ openstack compute service set [hostname] nova-compute --enable
  4. Check whether the Compute node is enabled:

    (overcloud) $ openstack compute service list

Select the next node to upgrade. Migrate its instances to another Compute node before performing the upgrade. Repeat this process until you have upgraded all Compute nodes.

Related Information

  • If you encounter any issues after completing this step, please contact Red Hat and request guidance and assistance.

5.4. Finalizing the Upgrade

The director needs to run through the upgrade finalization to ensure the Overcloud stack is synchronized with the current Heat template collection. This involves an environment file (major-upgrade-converge-docker.yaml), which you include using the openstack overcloud deploy command.

Prerequisites

  • You have upgraded all nodes.

Procedure

  1. Run the openstack overcloud deploy command and include:

    • All options and custom environment files relevant to your environment, such as network isolation and storage.
    • The overcloud_images.yaml environment file generated in Section 4.2, “Preparing for Containerized Services”.
    • The major-upgrade-converge-docker.yaml environment file.

      For example:

    $ openstack overcloud deploy --templates \
      -e /home.stack/templates/node_count.yaml \
      -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
      -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
      -e /home/stack/templates/network_environment.yaml \
      -e /home/stack/templates/overcloud_images.yaml  \
      -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-converge-docker.yaml \
      --ntp-server pool.ntp.org
  2. Wait until the overcloud updates with the new environment file’s configuration.
  3. Check if all services are active. For example, to check services on a Controllor node:

    [stack@director ~]$ ssh heat-admin@192.168.24.10
    [heat-admin@overcloud-controller-0 ~]$ sudo pcs status
    [heat-admin@overcloud-controller-0 ~]$ sudo systemctl list-units 'openstack-*' 'neutron-*' 'httpd*'

Related Information

  • If you encounter any issues after completing this step, please contact Red Hat and request guidance and assistance.