Chapter 4. Updating the overcloud
After you update the undercloud, you can update the overcloud by running the overcloud and container image preparation commands, updating your nodes, and running the overcloud update converge
command. The control plane API is fully available during a minor update.
Prerequisites
- You have updated the undercloud to the latest version.
4.1. Running the overcloud update preparation
To prepare the overcloud for the update process, run the openstack overcloud update prepare
command, which performs the following tasks:
- Updates the overcloud plan to OpenStack Platform 16.1
- Prepares the nodes for the update
Prerequisites
-
If you use a Ceph subscription and have configured director to use the
overcloud-minimal
image for Ceph storage nodes, you must ensure that in theroles_data.yaml
role definition file, therhsm_enforce
parameter is set toFalse
. -
If you rendered custom NIC templates, you must regenerate the templates with the updated version of the
openstack-tripleo-heat-templates
collection to avoid incompatibility with the overcloud version. For more information about custom NIC templates, see Rendering default network interface templates for customization in the Advanced Overcloud Customization guide.
Procedure
Source the
stackrc
file:$ source ~/stackrc
Run the update preparation command:
$ openstack overcloud update prepare \ --templates \ --stack <stack_name> \ -r <roles_data_file> \ -n <network_data_file> \ -e <environment_file> \ -e <environment_file> \ …
Include the following options relevant to your environment:
-
If the name of your overcloud stack is different to the default name
overcloud
, include the--stack
option in the update preparation command and replace<stack_name>
with the name of your stack. -
If you use your own custom roles, include your custom roles (
<roles_data>) file
(-r
). -
If you use custom networks, include your composable network (
_<network_data>) file _
(-n
). -
If you deploy a high availability cluster, include the
--ntp-server
option in the update preparation command, or include theNtpServer
parameter and value in your environment file. -
Any custom configuration environment files (
-e
).
-
If the name of your overcloud stack is different to the default name
- Wait until the update preparation completes.
4.2. Running the container image preparation
The overcloud requires the latest OpenStack Platform 16.1 container images before performing the update. This involves executing the container_image_prepare
external update process. To execute this process, you must run the openstack overcloud external-update run
command against tasks tagged with the container_image_prepare
tag. These tasks perform the following actions:
- Automatically prepare all container image configuration relevant to your environment.
- Pull the relevant container images to your undercloud, unless you have previously disabled this option.
If you are not using the default stack name (overcloud
), set your stack name with the --stack <stack_name>
option replacing <stack_name>
with the name of your stack.
Procedure
Source the
stackrc
file:$ source ~/stackrc
Run the
openstack overcloud external-update run
command against tasks tagged with thecontainer_image_prepare
tag:$ openstack overcloud external-update run --stack <stack_name> --tags container_image_prepare
4.3. Optional: Updating the ovn-controller container on all overcloud servers
If you deployed your overcloud with the Modular Layer 2 Open Virtual Network mechanism driver (ML2/OVN), update the ovn-controller container to the latest RHOSP 16.1 version. The update occurs on every overcloud server that runs the ovn-controller container.
If you are not using the default stack name, which is overcloud
, set your stack name with the --stack <stack_name>
option and replace <stack_name>
with the name of your stack.
Procedure
-
Log into the undercloud as the
stack
user. Source the
stackrc
file:$ source ~/stackrc
Run the openstack overcloud external-update run command against the tasks that have the ovn tag:
$ openstack overcloud external-update run --stack <stack_name> --tags ovn
- Wait until the ovn-controller container update completes.
4.4. Updating all Controller nodes
This process updates all the Controller nodes to the latest OpenStack Platform 16.1 version. The process involves running the openstack overcloud update run
command and including the --limit Controller
option to restrict operations to the Controller nodes only. The control plane API is fully available during the minor update.
Until BZ#1872404 is resolved, for nodes based on composable roles, you must update the Database
role first, before you can update Controller
, Messaging
, Compute
, Ceph
, and other roles.
If you are not using the default stack name (overcloud
), set your stack name with the --stack <stack_name>
option replacing <stack_name>
with the name of your stack.
Procedure
Source the
stackrc
file:$ source ~/stackrc
Run the update command:
$ openstack overcloud update run --stack <stack_name> --limit Controller
- Wait until the Controller node update completes.
4.5. Updating all Compute nodes
This process updates all Compute nodes to the latest OpenStack Platform 16.1 version. The process involves running the openstack overcloud update run
command and including the --limit Compute
option to restrict operations to the Compute nodes only.
- Parallelization considerations
When you update a large number of Compute nodes, to improve performance, you can run the
openstack overcloud update run
command with the--limit Compute
option in parallel on batches of 20 nodes. For example, if you have 80 Compute nodes in your deployment, you can run the following commands to update the Compute nodes in parallel:$ openstack overcloud update run -y --limit 'Compute[0:19]' > update-compute-0-19.log 2>&1 & $ openstack overcloud update run -y --limit 'Compute[20:39]' > update-compute-20-39.log 2>&1 & $ openstack overcloud update run -y --limit 'Compute[40:59]' > update-compute-40-59.log 2>&1 & $ openstack overcloud update run -y --limit 'Compute[60:79]' > update-compute-60-79.log 2>&1 &
The
'Compute[0:19]'
,'Compute[20:39]'
,'Compute[40:59]'
, and'Compute[60:79]'
way of partitioning the nodes space is random and you don’t have control over which nodes are updated.To update specific Compute nodes, list the nodes that you want to update in a batch separated by a comma:
$ openstack overcloud update run --limit <Compute0>,<Compute1>,<Compute2>,<Compute3>
If you are not using the default stack name (overcloud
), set your stack name with the --stack <stack_name>
option replacing <stack_name>
with the name of your stack.
Procedure
Source the
stackrc
file:$ source ~/stackrc
Run the update command:
$ openstack overcloud update run --stack <stack_name> --limit Compute
- Wait until the Compute node update completes.
4.6. Updating all HCI Compute nodes
This process updates the Hyperconverged Infrastructure (HCI) Compute nodes. The process involves:
-
Running the
openstack overcloud update run
command and including the--limit ComputeHCI
option to restrict operations to the HCI nodes only. -
Running the
openstack overcloud external-update run --tags ceph
command to perform an update to a containerized Red Hat Ceph Storage 4 cluster.
If you are not using the default stack name (overcloud
), set your stack name with the --stack <stack_name>
option replacing <stack_name>
with the name of your stack.
Prerequisites
On a Ceph Monitor or Controller node that is running the
ceph-mon
service, check that the Red Hat Ceph Storage cluster status is healthy and the pg status isactive+clean
:$ sudo podman exec -it ceph-mon-controller-0 ceph -s
If the Ceph cluster is healthy, it returns a status of
HEALTH_OK
.If the Ceph cluster status is unhealthy, it returns a status of
HEALTH_WARN
orHEALTH_ERR
. For troubleshooting guidance, see the Red Hat Ceph Storage 4 Troubleshooting Guide.
Procedure
Source the
stackrc
file:$ source ~/stackrc
Run the update command:
$ openstack overcloud update run --stack <stack_name> --limit ComputeHCI
- Wait until the node update completes.
Run the Ceph Storage update command. For example:
$ openstack overcloud external-update run --stack <stack_name> --tags ceph
- Wait until the Compute HCI node update completes.
4.7. Updating all Ceph Storage nodes
This process updates the Ceph Storage nodes. The process involves:
-
Running the
openstack overcloud update run
command and including the--limit CephStorage
option to restrict operations to the Ceph Storage nodes only. -
Running the
openstack overcloud external-update run
command to runceph-ansible
as an external process and update the Red Hat Ceph Storage 3 containers.
RHOSP 16.1 is supported on RHEL 8.2. However, hosts that are mapped to the Ceph Storage role update to the latest major RHEL release. For more information, see Red Hat Ceph Storage: Supported configurations.
If you are not using the default stack name (overcloud
), set your stack name with the --stack <stack_name>
option replacing <stack_name>
with the name of your stack.
Prerequisites
On a Ceph Monitor or Controller node that is running the
ceph-mon
service, check that the Red Hat Ceph Storage cluster status is healthy and the pg status isactive+clean
:$ sudo podman exec -it ceph-mon-controller-0 ceph -s
If the Ceph cluster is healthy, it returns a status of
HEALTH_OK
.If the Ceph cluster status is unhealthy, it returns a status of
HEALTH_WARN
orHEALTH_ERR
. For troubleshooting guidance, see the Red Hat Ceph Storage 4 Troubleshooting Guide.
Procedure
Source the
stackrc
file:$ source ~/stackrc
Update group nodes.
To update all nodes in a group:
$ openstack overcloud update run --limit <GROUP_NAME>
To update a single node in a group:
$ openstack overcloud update run --limit <GROUP_NAME> [NODE_INDEX]
NoteEnsure that you update all nodes if you choose to update nodes individually.
The index of the first node in a group is zero (0). For example, to update the first node in a group named
CephStorage
:openstack overcloud update run --limit CephStorage[0]
- Wait until the node update completes.
Run the Ceph Storage container update command:
$ openstack overcloud external-update run --tags ceph
- Wait until the Ceph Storage container update completes.
4.8. Performing online database updates
Some overcloud components require an online upgrade (or migration) of their databases tables. This involves executing the online_upgrade
external update process. To execute this process, run the openstack overcloud external-update run
command against tasks tagged with the online_upgrade
tag. This performs online database updates to the following components:
- OpenStack Block Storage (cinder)
- OpenStack Compute (nova)
Procedure
Source the
stackrc
file:$ source ~/stackrc
Run the
openstack overcloud external-update run
command against tasks that use theonline_upgrade
tag:$ openstack overcloud external-update run --tags online_upgrade
4.9. Finalizing the update
The update requires a final step to update the overcloud stack. This ensures the stack’s resource structure aligns with a regular deployment of OpenStack Platform 16.1 and allows you to perform standard openstack overcloud deploy
functions in the future.
Procedure
Source the
stackrc
file:$ source ~/stackrc
-
To re-enable fencing in your overcloud, in the
fencing.yaml
environment file, set theEnableFencing
parameter totrue
. Run the update finalization command:
$ openstack overcloud update converge \ --templates \ --stack <stack_name> \ -r <roles_data_file> \ -n <network_data_file> \ -e <environment_file> \ -e <environment_file> \ ... ...
Include the following options relevant to your environment:
-
The
fencing.yaml
environment file, with theEnableFencing
parameter set totrue
. -
If the name of your overcloud stack is different to the default name
overcloud
, include the--stack
option in the update preparation command and replace<stack_name>
with the name of your stack. -
If using your own custom roles, include your custom roles (
<roles_data>
) file (-r
) -
If using custom networks, include your composable network (
<network_data>
) file (-n
) -
Any custom configuration environment files (
-e
).
-
The
- Wait until the update finalization completes.