Chapter 5. Upgrading the Overcloud
This process upgrades the overcloud.
Prerequisites
- You have upgraded the undercloud to the latest version.
- You have prepared your custom environment files to accommodate the changes in the upgrade.
5.1. Upgrading the Overcloud Nodes
The major-upgrade-composable-steps-docker.yaml environment file upgrades all composable services on all custom roles, except for any roles with disable_upgrade_deployment: True in the roles_data file. These nodes are updated with a separate process.
Prerequisites
- You have upgraded the undercloud to the latest version.
- You have prepared your custom environment files to accommodate the changes in the upgrade.
Procedure
Run the
openstack overcloud deploycommand and include:- All options and custom environment files relevant to your environment, such as network isolation and storage.
-
The
overcloud_images.yamlenvironment file generated in Section 4.2, “Preparing for Containerized Services”. The
major-upgrade-composable-steps-docker.yamlenvironment file.For example:
$ openstack overcloud deploy --templates \ -e /home.stack/templates/node_count.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \ -e /home/stack/templates/network_environment.yaml \ -e /home/stack/templates/overcloud_images.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps-docker.yaml \ --ntp-server pool.ntp.org
Wait until the overcloud updates with the new environment file’s configuration.
ImportantThe upgrade disables the OpenStack Networking (neutron) server and L3 Agent. This means you cannot create new routers during this step. You can still access instances during this period.
Check if all services are active. For example, to check services on a Controllor node:
[stack@director ~]$ ssh heat-admin@192.168.24.10 [heat-admin@overcloud-controller-0 ~]$ sudo pcs status [heat-admin@overcloud-controller-0 ~]$ sudo docker ps
Related Information
- If you encounter any issues after completing this step, please contact Red Hat and request guidance and assistance.
5.2. Upgrading the Object Storage Nodes
Standalone Object Storage nodes are not included in the main overcloud node upgrade process because you need to update each node individually to retain the services' activity. The director contains a script to execute the upgrade on an individual Object Storage nodes.
Prerequisites
-
You have previously run
openstack overcloud deploywith themajor-upgrade-composable-steps-docker.yamlenvironment file. This upgrades the main custom roles and their composable services.
Procedure
Obtain a list of Object Storage nodes:
$ openstack server list -c Name -f value --name objectstorage
Perform the following steps for each Object Storage node in the list:
Run the
upgrade-non-controller.shscript using the node name to identify the node to upgrade:$ upgrade-non-controller.sh --upgrade overcloud-objectstorage-0
NoteIf using pre-provisioned node infrastructure, see Section 4.9, “Preparing for Pre-Provisioned Nodes Upgrade” for changes with this command.
- Wait until the Object Storage node completes the upgrade.
Reboot the Object Storage node:
$ openstack server reboot overcloud-objectstorage-0
- Wait until the Object Storage node completes the reboot.
Related Information
- If you encounter any issues after completing this step, please contact Red Hat and request guidance and assistance.
5.3. Upgrading the Compute Nodes
Compute nodes are not included in the main overcloud node upgrade process. To ensure maximum uptime of instances, you migrate each instance from a Compute node before upgrading the node. This means the Compute node upgrade process involves the following steps:
Prerequisites
-
You have previously run 'openstack overcloud deploy` with the
major-upgrade-composable-steps-docker.yamlenvironment file. This upgrades the main custom roles and their composable services.
Procedure
Select a Compute node to upgrade:
List all Compute nodes:
$ source ~/stackrc $ openstack server list -c Name -f value --name compute
- Select a Compute node to upgrade and note its UUID and name.
Migrate instances to another Compute node:
From the undercloud, select a Compute Node to reboot and disable it:
$ source ~/overcloudrc (overcloud) $ openstack compute service list (overcloud) $ openstack compute service set [hostname] nova-compute --disable
List all instances on the Compute node:
(overcloud) $ openstack server list --host [hostname] --all-projects
Use one of the following commands to migrate your instances:
Migrate the instance to a specific host of your choice:
(overcloud) $ openstack server migrate [instance-id] --live [target-host]--wait
Let
nova-schedulerautomatically select the target host:(overcloud) $ nova live-migration [instance-id]
Live migrate all instances at once:
$ nova host-evacuate-live [hostname]
NoteThe
novacommand might cause some deprecation warnings, which are safe to ignore.
- Wait until migration completes.
Confirm the migration was successful:
(overcloud) $ openstack server list --host [hostname] --all-projects
- Continue migrating instances until none remain on the chosen Compute Node.
Upgrade the empty Compute node:
Run the
upgrade-non-controller.shscript using the node name to identify the node to upgrade:$ upgrade-non-controller.sh --upgrade overcloud-compute-0
NoteIf using pre-provisioned node infrastructure, see Section 4.9, “Preparing for Pre-Provisioned Nodes Upgrade” for changes with this command.
- Wait until the Compute node completes the upgrade.
Reboot and enable the upgraded Compute node:
Log into the Compute Node and reboot it:
[heat-admin@overcloud-compute-0 ~]$ sudo reboot
- Wait until the node boots.
Enable the Compute Node again:
$ source ~/overcloudrc (overcloud) $ openstack compute service set [hostname] nova-compute --enable
Check whether the Compute node is enabled:
(overcloud) $ openstack compute service list
Select the next node to upgrade. Migrate its instances to another Compute node before performing the upgrade. Repeat this process until you have upgraded all Compute nodes.
Related Information
- If you encounter any issues after completing this step, please contact Red Hat and request guidance and assistance.
5.4. Finalizing the Upgrade
The director needs to run through the upgrade finalization to ensure the Overcloud stack is synchronized with the current Heat template collection. This involves an environment file (major-upgrade-converge-docker.yaml), which you include using the openstack overcloud deploy command.
Prerequisites
- You have upgraded all nodes.
Procedure
Run the
openstack overcloud deploycommand and include:- All options and custom environment files relevant to your environment, such as network isolation and storage.
-
The
overcloud_images.yamlenvironment file generated in Section 4.2, “Preparing for Containerized Services”. The
major-upgrade-converge-docker.yamlenvironment file.For example:
$ openstack overcloud deploy --templates \ -e /home.stack/templates/node_count.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \ -e /home/stack/templates/network_environment.yaml \ -e /home/stack/templates/overcloud_images.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-converge-docker.yaml \ --ntp-server pool.ntp.org
- Wait until the overcloud updates with the new environment file’s configuration.
Check if all services are active. For example, to check services on a Controllor node:
[stack@director ~]$ ssh heat-admin@192.168.24.10 [heat-admin@overcloud-controller-0 ~]$ sudo pcs status [heat-admin@overcloud-controller-0 ~]$ sudo systemctl list-units 'openstack-*' 'neutron-*' 'httpd*'
Related Information
- If you encounter any issues after completing this step, please contact Red Hat and request guidance and assistance.
