Back up the database:

[root@director ~]# mysqldump --opt --all-databases > /root/undercloud-all-databases.sql

Archive the database backup and the configuration files:

[root@director ~]# tar --xattrs -czf undercloud-backup-`date +%F`.tar.gz /root/undercloud-all-databases.sql /etc/my.cnf.d/server.cnf /srv/node /home/stack /etc/pki/instack-certs/undercloud.pem /etc/pki/ca-trust/source/anchors/ca.crt.pem

This creates a file named undercloud-backup-[timestamp].tar.gz.

Perform the database backup:

Backup the OpenStack Telemetry database:

Backup the Redis cluster:

Backup the filesystem:

Stop the main OpenStack Platform services:

(undercloud) [stack@director ~]$ sudo systemctl stop 'openstack-*' 'neutron-*' httpd

Update the python-tripleoclient package and its dependencies to ensure you have the latest scripts for the minor version update:

(undercloud) [stack@director ~]$ sudo yum update python-tripleoclient

Run the openstack undercloud upgrade command:

(undercloud) [stack@director ~]$ openstack undercloud upgrade

Wait until this command completes its execution.

Reboot the undercloud to update the operating system’s kernel and other system packages:

(undercloud) [stack@director ~]$ sudo reboot

Check the yum log to determine if new image archives are available:

(undercloud) [stack@director ~]$ sudo grep "rhosp-director-images" /var/log/yum.log

If new archives are available, replace the current images with new images. To install the new images, first remove any existing images from the images directory on the stack user’s home (/home/stack/images):

(undercloud) [stack@director ~]$ rm -rf ~/images/*

Extract the archives:

(undercloud) [stack@director ~]$ cd ~/images
(undercloud) [stack@director ~]$ for i in /usr/share/rhosp-director-images/overcloud-full-latest-10.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-10.0.tar; do tar -xvf $i; done

Import the latest images into the director and configure nodes to use the new images

(undercloud) [stack@director ~]$ cd ~
(undercloud) [stack@director ~]$ openstack overcloud image upload --update-existing --image-path /home/stack/images/
(undercloud) [stack@director ~]$ openstack overcloud node configure $(openstack baremetal node list -c UUID -f csv --quote none | sed "1d" | paste -s -d " ")

To finalize the image update, verify the existence of the new images:

(undercloud) [stack@director ~]$ openstack image list
(undercloud) [stack@director ~]$ ls -l /httpboot

The director also retains the old images and renames them using the timestamp of when they were updated. If you no longer need these images, delete them. The director is now updated and using the latest images.

Update the current plan using the original openstack overcloud deploy command and including the --update-plan-only option. For example:

(undercloud) [stack@director ~]$ openstack overcloud deploy --update-plan-only \
  --templates  \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e /home/stack/custom-templates/network-environment.yaml \
  -e /home/stack/custom-templates/storage-environment.yaml \
  -e /home/stack/custom-templates/rhel-registration/environment-rhel-registration.yaml \
  [-e <environment_file>|...]

The --update-plan-only only updates the Overcloud plan stored in the director. Use the -e option to include environment files relevant to the Overcloud and its update path. The order of the environment files is important as the parameters and resources defined in subsequent environment files take precedence. Use the following list as an example of the environment file order:

Perform a package update on all nodes using the openstack overcloud update command. For example:

(undercloud) [stack@director ~]$ openstack overcloud update stack -i overcloud

The -i runs an interactive mode to update each node. When the update process completes a node update, the script provides a breakpoint for you to confirm. Without the -i option, the update remains paused at the first breakpoint. Therefore, it is mandatory to include the -i option.

The update process starts. During this process, the director reports an IN_PROGRESS status and periodically prompts you to clear breakpoints. For example:

not_started: [u'overcloud-controller-0', u'overcloud-controller-1', u'overcloud-controller-2']
on_breakpoint: [u'overcloud-compute-0']
Breakpoint reached, continue? Regexp or Enter=proceed, no=cancel update, C-c=quit interactive mode:

Press Enter to clear the breakpoint from last node on the on_breakpoint list. This begins the update for that node. You can also type a node name to clear a breakpoint on a specific node, or a Python-based regular expression to clear breakpoints on multiple nodes at once. However, it is not recommended to clear breakpoints on multiple controller nodes at once. Continue this process until all nodes have completed their update.

The update command reports a COMPLETE status when the update completes:

...
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
COMPLETE
update finished with status COMPLETE

If you configured fencing for the Controller nodes, the update process might disable it. When the update process completes, reenable fencing with the following command on one of the Controller nodes:

(undercloud) [stack@director ~]$ sudo pcs property set stonith-enabled=true

Reboot the node:

[heat-admin@overcloud-controller-0 ~]$ sudo reboot

Log into the node and check the services. For example:

Log into a hyperconverged Controller/Ceph Monitor node and disable the storage cluster rebalancing feature temporarily:

[stack@cntr-mon ~]$ sudo ceph osd set noout
[stack@cntr-mon ~]$ sudo ceph osd set norebalance

Reboot the node:

[stack@cntr-mon ~]$ sudo reboot

Log into the node and check the cluster status:

[stack@cntr-mon ~]$ sudo ceph -s

Check that the pgmap reports all pgs as normal (active+clean).

When complete, log into a Ceph MON or Controller node and enable cluster rebalancing again:

[stack@cntr-mon ~]$ sudo ceph osd unset noout
[stack@cntr-mon ~]$ sudo ceph osd unset norebalance

Perform a final status check to verify the cluster reports HEALTH_OK:

[stack@cntr-mon ~]$ sudo ceph status

List all the compute nodes and their UUIDs:

[stack@director ~]$ source ~/stackrc
(undercloud) [stack@director ~]$ openstack server list --name compute

Identify the UUID of the compute node you aim to reboot.

From the undercloud, select a compute node and disable it:

[stack@director ~]$ source ~/overcloudrc
(overcloud) [stack@director ~]$ openstack compute service list
(overcloud) [stack@director ~]$ openstack compute service set [hostname] nova-compute --disable

List all instances on the compute node:

(overcloud) [stack@director ~]$ openstack server list --host [hostname] --all-projects

Use one of the following commands to migrate the instances:

Confirm the migration was successful:

(overcloud) [stack@director ~]$ openstack server list --host [hostname] --all-projects

Log into the compute node and reboot it:

[heat-admin@overcloud-compute-0 ~]$ sudo reboot

Enable the compute node again:

[stack@director ~]$ source ~/overcloudrc
(overcloud) [stack@director ~]$ openstack compute service set [hostname] nova-compute --enable

Check whether the compute node is enabled:

(overcloud) [stack@director ~]$ openstack compute service list

Run yum to check the system packages:

[stack@director ~]$ sudo yum list qemu-img-rhev qemu-kvm-common-rhev qemu-kvm-rhev qemu-kvm-tools-rhev openvswitch

Run ovs-vsctl to check the version currently running:

[stack@director ~]$ sudo ovs-vsctl --version

Source the undercloud access details:

[stack@director ~]$ source ~/stackrc

Check for failed Systemd services:

[stack@director ~]$ sudo systemctl list-units --state=failed 'openstack*' 'neutron*' 'httpd' 'docker'

Check the undercloud free space:

[stack@director ~]$ df -h

Check that clocks are synchronized on the undercloud:

[stack@director ~]$ sudo ntpstat

Check the undercloud network services:

[stack@director ~]$ openstack network agent list

All agents should be Alive and their state should be UP.

Check the undercloud compute services:

[stack@director ~]$ openstack compute service list

All agents' status should be enabled and their state should be up

Check the undercloud volume services:

[stack@director ~]$ openstack volume service list

All agents' status should be enabled and their state should be up.

Source the undercloud access details:

[stack@director ~]$ source ~/stackrc

Check the status of the bare metal nodes:

[stack@director ~]$ openstack baremetal node list

All nodes should have a valid power state (on) and maintenance mode should be false.

Check for failed Systemd services:

[stack@director ~]$ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo systemctl list-units --state=failed 'openstack*' 'neutron*' 'httpd' 'docker' 'ceph*'" ; done

Check the HAProxy connection to all services. Obtain the Control Plane VIP address and authentication details for the haproxy.stats service:

[stack@director ~]$ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE sudo 'grep "listen haproxy.stats" -A 6 /etc/haproxy/haproxy.cfg'

Use these details in the following cURL request:

[stack@director ~]$ curl -s -u admin:<PASSWORD> "http://<IP ADDRESS>:1993/;csv" | egrep -vi "(frontend|backend)" | awk -F',' '{ print $1" "$2" "$18 }'

Replace <PASSWORD> and <IP ADDRESS> details with the respective details from the haproxy.stats service. The resulting list shows the OpenStack Platform services on each node and their connection status.

Check overcloud database replication health:

[stack@director ~]$ for NODE in $(openstack server list --name controller -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo clustercheck" ; done

Check RabbitMQ cluster health:

[stack@director ~]$ for NODE in $(openstack server list --name controller -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo rabbitmqctl node_health_check" ; done

Check Pacemaker resource health:

[stack@director ~]$ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo pcs status"

Look for:

Check the disk space on each overcloud node:

[stack@director ~]$ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo df -h --output=source,fstype,avail -x overlay -x tmpfs -x devtmpfs" ; done

Check health of the Red Hat Ceph Storage cluster. The following command runs the ceph tool on a Controller node to check the cluster:

[stack@director ~]$ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo ceph -s"

Check the free space on the OSD. The following command runs the ceph tool on a Controller node to check the free space:

[stack@director ~]$ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo ceph df"

Check that clocks are synchronized on overcloud nodes

[stack@director ~]$ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo ntpstat" ; done

Source the overcloud access details:

[stack@director ~]$ source ~/overcloudrc

Check the overcloud network services:

[stack@director ~]$ openstack network agent list

All agents should be Alive and their state should be UP.

Check the overcloud compute services:

[stack@director ~]$ openstack compute service list

All agents' status should be enabled and their state should be up

Check the overcloud volume services:

[stack@director ~]$ openstack volume service list

All agents' status should be enabled and their state should be up.

Disable the current OpenStack Platform repository:

[stack@director ~]$ sudo subscription-manager repos --disable=rhel-7-server-openstack-10-rpms

Enable the new OpenStack Platform repository:

[stack@director ~]$ sudo subscription-manager repos --enable=rhel-7-server-openstack-11-rpms

Stop the main OpenStack Platform services:

[stack@director ~]$ sudo systemctl stop 'openstack-*' 'neutron-*' httpd

The default Provisioning/Control Plane network has changed from 192.0.2.0/24 to 192.168.24.0/24. If you used default network values in the previous undercloud.conf file, the Provisioning/Control Plane network is set to 192.0.2.0/24. This means you need to set certain parameters in the undercloud.conf file to continue using the 192.0.2.0/24 network. These parameters are:

Set the network values in undercloud.conf to ensure continued use of the 192.0.2.0/24 CIDR during future upgrades. Ensure the network configuration set correctly before running the openstack undercloud upgrade command.

Run yum to upgrade the director’s main packages:

[stack@director ~]$ sudo yum update instack-undercloud openstack-puppet-modules openstack-tripleo-common python-tripleoclient

Run the following command to upgrade the undercloud:

[stack@director ~]$ openstack undercloud upgrade

Disable the current OpenStack Platform repository:

[stack@director ~]$ sudo subscription-manager repos --disable=rhel-7-server-openstack-11-rpms

Enable the new OpenStack Platform repository:

[stack@director ~]$ sudo subscription-manager repos --enable=rhel-7-server-openstack-12-rpms

Install the ceph-ansible package:

[stack@director ~]$ sudo yum install ceph-ansible

Run yum to upgrade the director’s main packages:

[stack@director ~]$ sudo yum update python-tripleoclient

Run the following command to upgrade the undercloud:

[stack@director ~]$ openstack undercloud upgrade

Disable the current OpenStack Platform repository:

[stack@director ~]$ sudo subscription-manager repos --disable=rhel-7-server-openstack-12-rpms

Enable the new OpenStack Platform repository:

[stack@director ~]$ sudo subscription-manager repos --enable=rhel-7-server-openstack-13-rpms

Run yum to upgrade the director’s main packages:

[stack@director ~]$ sudo yum update python-tripleoclient

Run the following command to upgrade the undercloud:

[stack@director ~]$ openstack undercloud upgrade

Reboot the undercloud to update the operating system’s kernel and other system packages:

[stack@director ~]$ sudo reboot

As the stack user, on the undercloud node, using the openstack overcloud container image prepare command to include the additional services.

Run the image prepare command:

[stack@director ~]$ openstack overcloud container image prepare \
  ...
  -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
  --set ceph_namespace=registry.access.redhat.com/rhceph \
  --set ceph_image=rhceph-3-rhel7 \
  --tag-from-label {version}-{release} \
  ...

To pull the images directly from registry.access.redhat.com in the overcloud deployment, an environment file is required to specify the image parameters. The following command automatically creates this environment file:

(undercloud) [stack@director ~]$ openstack overcloud container image prepare \
  --namespace=registry.access.redhat.com/rhosp13 \
  --prefix=openstack- \
  --tag-from-label {version}-{release} \
  --output-env-file=/home/stack/custom-templates/overcloud_images.yaml

Create a template to pull the images to the local registry:

(undercloud) [stack@director ~]$ openstack overcloud container image prepare \
  --namespace=registry.access.redhat.com/rhosp13 \
  --prefix=openstack- \
  --tag-from-label {version}-{release} \
  --output-images-file /home/stack/local_registry_images.yaml

This creates a file called local_registry_images.yaml with the container image information. Pull the images using the local_registry_images.yaml file:

(undercloud) [stack@director ~]$ sudo openstack overcloud container image upload \
  --config-file  /home/stack/local_registry_images.yaml \
  --verbose

Find the namespace of the local images. The namespace uses the following pattern:

<REGISTRY IP ADDRESS>:8787/rhosp13

Use the IP address of the undercloud, which you previously set with the local_ip parameter in the undercloud.conf file. Alternatively, you can also obtain the full namespace with the following command:

(undercloud) [stack@director ~]$ docker images | grep -v redhat.com | grep -o '^.*rhosp13' | sort -u

Create a template for using the images in our local registry on the undercloud. For example:

(undercloud) [stack@director ~]$ openstack overcloud container image prepare \
  --namespace=192.168.24.1:8787/rhosp13 \
  --prefix=openstack- \
  --tag-from-label {version}-{release} \
  --output-env-file=/home/stack/custom-templates/overcloud_images.yaml

As the stack user, source the stackrc file:

[stack@director ~]$ source ~/stackrc

List the default role templates with the openstack overcloud role list command:

[stack@director ~]$ openstack overcloud role list
BlockStorage
CephStorage
Compute
ComputeHCI
ComputeOvsDpdk
Controller
...

View a role’s YAML definition with the openstack overcloud role show command. For example:

[stack@director ~]$ openstack overcloud role show Compute

Generate a custom roles_data_custom.yaml file with the openstack overcloud roles generate command to join multiple predefined roles into a single file. For example, the following command joins the Controller, Compute, and CephStorage roles into a single file:

[stack@director ~]$ openstack overcloud roles generate -o ~/custom-templates/roles_data_custom.yaml Controller Compute ComputeHCI

The -o defines the name of the file to create.

Check that you are using the latest resources and configuration in the storage environment file. This requires the following changes:

The undercloud’s dynamic Ansible script has updated to the OpenStack Platform 12 version, which uses the RoleNetHostnameMap Heat parameter in the overcloud plan to define the inventory. However, the overcloud currently uses the OpenStack Platform 11 template versions, which do not have the RoleNetHostnameMap parameter. This means you need to create a temporary static inventory file, which you can generate with the following command:

[stack@director ~]$ openstack server list -c Networks -f value | cut -d"=" -f2 > overcloud_hosts

Create an Ansible playbook (undercloud-ca.yml) that contains the following:

- name: Add undercloud CA to overcloud nodes
  hosts: all
  user: heat-admin
  become: true
  tasks:
    - name: Copy undercloud CA
      copy:
        src: /etc/pki/ca-trust/source/anchors/cm-local-ca.pem
        dest: /etc/pki/ca-trust/source/anchors/
    - name: Update trust
      command: "update-ca-trust extract"
    - name: Get the swift endpoint
      shell: |
        source ~stack/stackrc
        openstack endpoint list -c 'Service Name' -c 'URL' -c Interface -f value | grep swift | grep public | awk -F/ '{print $3}'
      register: swift_endpoint
      delegate_to: 127.0.0.1
      become: yes
      become_user: stack
    - debug:
        var: swift_endpoint
    - name: Verify URL
      uri:
        url: https://{{ swift_endpoint.stdout }}/healthcheck
        return_content: yes
      register: verify
    - name: Report output
      debug:
        msg: "{{ ansible_hostname }} can access the undercloud's Public API"
      when: verify.content == "OK"

This playbook contains multiple tasks that perform the following on each node:

Run the playbook with the following command:

[stack@director ~]$ ansible-playbook -i overcloud_hosts undercloud-ca.yml

This uses the temporary inventory to provide Ansible with the overcloud nodes.

The resulting Ansible output should show a debug message for node. For example:

ok: [192.168.24.100] => {
    "msg": "overcloud-controller-0 can access the undercloud's Public API"
}

As the stack user, source the stackrc file:

[stack@director ~]$ source ~/stackrc

Run the fast forward upgrade preparation command:

[stack@director ~]$ openstack overcloud ffwd-upgrade prepare --templates \
  -e $ALL_ENVIRONMENTS_USED_TO_DEPLOY \
  -e /home/stack/custom-templates/overcloud_images.yaml \
  -r /home/stack/custom-templates/roles_data_custom.yaml

Include the following options relevant to the environment:

Run the fast forward upgrade command:

[stack@director ~]$ openstack overcloud ffwd-upgrade run

As the stack user, source the stackrc file:

[stack@director ~]$ source ~/stackrc

Run the upgrade command:

[stack@director ~]$ openstack overcloud upgrade run --roles Controller --skip-tags validation

Source the stackrc file:

[stack@director ~]$ source ~/stackrc

Run the upgrade command:

[stack@director ~]$ openstack overcloud upgrade run --nodes osdcompute-0 --skip-tags validation

As the stack user, source the stackrc file:

[stack@director ~]$ source ~/stackrc

Run the upgrade command:

[stack@director ~]$ openstack overcloud upgrade run --roles OsdCompute --skip-tags validation

As the stack user, source the stackrc file:

[stack@director ~]$ source ~/stackrc

Run the Ceph Storage upgrade command. For example:

[stack@director ~]$ openstack overcloud ceph-upgrade run --templates \
    -e $ALL_ENVIRONMENT_FILES_TO_DEPLOY \
    -e /home/stack/custom-templates/overcloud_images.yaml \
    -r /home/stack/custom-templates/roles_data_custom.yaml \
    --ceph-ansible-playbook '/usr/share/ceph-ansible/infrastructure-playbooks/switch-from-non-containerized-to-containerized-ceph-daemons.yml,/usr/share/ceph-ansible/infrastructure-playbooks/rolling_update.yml'

Include the following options relevant to the environment:

As the stack user, source the stackrc file:

[stack@director ~]$ source ~/stackrc

Run the fast forward upgrade finalization command:

[stack@director ~]$ openstack overcloud ffwd-upgrade converge \
    -e $ALL_ENVIRONMENT_FILES_TO_DEPLOY
    -e /home/stack/custom-templates/overcloud_images.yaml \
    -r /home/stack/custom-templates/roles_data_custom.yaml

Include the following options relevant to your environment: