Procedure

1. Source the undercloud access details:

   $ source ~/stackrc

2. Check for failed Systemd services:

   (undercloud) $ sudo systemctl list-units --state=failed 'openstack*' 'neutron*' 'httpd' 'docker'

3. Check the undercloud free space:

   (undercloud) $ df -h

   Use the "Undercloud Reqirements" as a basis to determine if you have adequate free space.

4. Check that clocks are synchronized on the undercloud:

   (undercloud) $ sudo ntpstat

5. Check the undercloud network services:

   (undercloud) $ openstack network agent list

   All agents should be Alive and their state should be UP.

6. Check the undercloud compute services:

   (undercloud) $ openstack compute service list

   All agents' status should be enabled and their state should be up

Procedure

1. Source the undercloud access details:

   $ source ~/stackrc

2. Check the status of your bare metal nodes:

   (undercloud) $ openstack baremetal node list

   All nodes should have a valid power state (on) and maintenance mode should be false.

3. Check for failed Systemd services:

   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo systemctl list-units --state=failed 'openstack*' 'neutron*' 'httpd' 'docker' 'ceph*'" ; done

4. Check the HAProxy connection to all services. Obtain the Control Plane VIP address and authentication details for the haproxy.stats service:

   (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE sudo 'grep "listen haproxy.stats" -A 6 /etc/haproxy/haproxy.cfg'

   Use these details in the following cURL request:

   (undercloud) $ curl -s -u admin:<PASSWORD> "http://<IP ADDRESS>:1993/;csv" | egrep -vi "(frontend|backend)" | awk -F',' '{ print $1" "$2" "$18 }'

   Replace <PASSWORD> and <IP ADDRESS> details with the respective details from the haproxy.stats service. The resulting list shows the OpenStack Platform services on each node and their connection status.

5. Check overcloud database replication health:

   (undercloud) $ for NODE in $(openstack server list --name controller -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo clustercheck" ; done

6. Check RabbitMQ cluster health:

   (undercloud) $ for NODE in $(openstack server list --name controller -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo rabbitmqctl node_health_check" ; done

7. Check Pacemaker resource health:

   (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo pcs status"

   Look for:

   • All cluster nodes online.
   • No resources stopped on any cluster nodes.
   • No failed pacemaker actions.

8. Check the disk space on each overcloud node:

   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo df -h --output=source,fstype,avail -x overlay -x tmpfs -x devtmpfs" ; done

9. Check overcloud Ceph Storage cluster health. The following command runs the ceph tool on a Controller node to check the cluster:

   (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo ceph -s"

10. Check Ceph Storage OSD for free space. The following command runs the ceph tool on a Controller node to check the free space:

    (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo ceph df"

11. Check that clocks are synchronized on overcloud nodes

    (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo ntpstat" ; done

12. Source the overcloud access details:

    (undercloud) $ source ~/overcloudrc

13. Check the overcloud network services:

    (overcloud) $ openstack network agent list

    All agents should be Alive and their state should be UP.

14. Check the overcloud compute services:

    (overcloud) $ openstack compute service list

    All agents' status should be enabled and their state should be up

15. Check the overcloud volume services:

    (overcloud) $ openstack volume service list

    All agents' status should be enabled and their state should be up.

Procedure

1. Log into the undercloud as the root user.

2. Back up the database:

   # mysqldump --opt --all-databases > /root/undercloud-all-databases.sql

3. Archive the database backup and the configuration files:

   # tar --xattrs -czf undercloud-backup-`date +%F`.tar.gz /root/undercloud-all-databases.sql /etc/my.cnf.d/server.cnf /srv/node /home/stack /etc/pki/instack-certs/undercloud.pem /etc/pki/ca-trust/source/anchors/ca.crt.pem

   This creates a file named undercloud-backup-[timestamp].tar.gz.

Procedure

1. Log into the director as the stack user.

2. Update the python-tripleoclient package and its dependencies to ensure you have the latest scripts for the minor version update:

   $ sudo yum update -y python-tripleoclient

3. The director uses the openstack undercloud upgrade command to update the Undercloud environment. Run the command:

   $ openstack undercloud upgrade

4. Reboot the node:

   $ sudo reboot

5. Wait until the node boots.

6. Check the status of all services:

   $ sudo systemctl list-units "openstack*" "neutron*" "openvswitch*"

   Note

   It might take approximately 10 minutes for the openstack-nova-compute to become active after a reboot.

7. Verify the existence of your overcloud and its nodes:

   $ source ~/stackrc
   $ openstack server list
   $ openstack baremetal node list
   $ openstack stack list

Procedure

1. Check the yum log to determine if new image archives are available:

   $ sudo grep "rhosp-director-images" /var/log/yum.log

2. If new archives are available, replace your current images with new images. To install the new images, first remove any existing images from the images directory on the stack user’s home (/home/stack/images):

   $ rm -rf ~/images/*

3. Extract the archives:

   $ cd ~/images
   $ for i in /usr/share/rhosp-director-images/overcloud-full-latest-11.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-11.0.tar; do tar -xvf $i; done

4. Import the latest images into the director and configure nodes to use the new images

   $ cd ~
   $ openstack overcloud image upload --update-existing --image-path /home/stack/images/
   $ openstack overcloud node configure $(openstack baremetal node list -c UUID -f csv --quote none | sed "1d" | paste -s -d " ")

5. To finalize the image update, verify the existence of the new images:

   $ openstack image list
   $ ls -l /httpboot

   The director is now updated and using the latest images. You do not need to restart any services after the update.

Procedure

1. Update the current plan using your original openstack overcloud deploy command and including the --update-plan-only option. For example:

   $ openstack overcloud deploy --update-plan-only \
     --templates  \
     -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
     -e /home/stack/templates/network-environment.yaml \
     -e /home/stack/templates/storage-environment.yaml \
     -e /home/stack/templates/rhel-registration/environment-rhel-registration.yaml \
     [-e <environment_file>|...]

   The --update-plan-only only updates the Overcloud plan stored in the director. Use the -e option to include environment files relevant to your Overcloud and its update path. The order of the environment files is important as the parameters and resources defined in subsequent environment files take precedence. Use the following list as an example of the environment file order:

   • Any network isolation files, including the initialization file (environments/network-isolation.yaml) from the heat template collection and then your custom NIC configuration file.
   • Any external load balancing environment files.
   • Any storage environment files.
   • Any environment files for Red Hat CDN or Satellite registration.
   • Any other custom environment files.

2. Perform a package update on all nodes using the openstack overcloud update command. For example:

   $ openstack overcloud update stack -i overcloud

   The -i runs an interactive mode to update each node. When the update process completes a node update, the script provides a breakpoint for you to confirm. Without the -i option, the update remains paused at the first breakpoint. Therefore, it is mandatory to include the -i option.

   Note

   Running an update on all nodes in parallel can cause problems. For example, an update of a package might involve restarting a service, which can disrupt other nodes. This is why the process updates each node using a set of breakpoints. This means nodes are updated one by one. When one node completes the package update, the update process moves to the next node.

3. The update process starts. During this process, the director reports an IN_PROGRESS status and periodically prompts you to clear breakpoints. For example:

   not_started: [u'overcloud-controller-0', u'overcloud-controller-1', u'overcloud-controller-2']
   on_breakpoint: [u'overcloud-compute-0']
   Breakpoint reached, continue? Regexp or Enter=proceed, no=cancel update, C-c=quit interactive mode:

   Press Enter to clear the breakpoint from last node on the on_breakpoint list. This begins the update for that node. You can also type a node name to clear a breakpoint on a specific node, or a Python-based regular expression to clear breakpoints on multiple nodes at once. However, it is not recommended to clear breakpoints on multiple controller nodes at once. Continue this process until all nodes have completed their update.

4. The update command reports a COMPLETE status when the update completes:

   ...
   IN_PROGRESS
   IN_PROGRESS
   IN_PROGRESS
   COMPLETE
   update finished with status COMPLETE

5. If you configured fencing for your Controller nodes, the update process might disable it. When the update process completes, reenable fencing with the following command on one of the Controller nodes:

   $ sudo pcs property set stonith-enabled=true

6. The update process does not reboot any nodes in the Overcloud automatically. Updates to the kernel or Open vSwitch require a reboot. Check the /var/log/yum.log file on each node to see if either the kernel or openvswitch packages have updated their major or minor versions. If they have, reboot each node using the "Rebooting Nodes" procedures in the Director Installation and Usage guide.

Procedure

1. Log into the director as the stack user.

2. Disable the current OpenStack Platform repository:

   $ sudo subscription-manager repos --disable=rhel-7-server-openstack-11-rpms

3. Enable the new OpenStack Platform repository:

   $ sudo subscription-manager repos --enable=rhel-7-server-openstack-12-rpms

4. Run yum to upgrade the director’s main packages:

   $ sudo yum update -y python-tripleoclient

5. Edit the /home/stack/undercloud.conf file and check that the enabled_drivers parameter does not contain the pxe_ssh driver. This driver is deprecated in favor of the Virtual Bare Metal Controller (VBMC) and removed from Red Hat OpenStack Platform. For information on switching pxe_ssh nodes to VBMC, see "Virtual Bare Metal Controller (VBMC)" in the Director Installation and Usage guide.

6. Run the following command to upgrade the undercloud:

   $ openstack undercloud upgrade

   This command upgrades the director’s packages, refreshes the director’s configuration, and populates any settings that are unset since the version change. This command does not delete any stored data, such as Overcloud stack data or data for existing nodes in your environment.

7. Reboot the node:

   $ sudo reboot

8. Wait until the node boots.

9. Check the status of all services:

   $ sudo systemctl list-units "openstack*" "neutron*" "openvswitch*"

   Note

   It might take approximately 10 minutes for the openstack-nova-compute to become active after a reboot.

10. Verify the existence of your overcloud and its nodes:

    $ source ~/stackrc
    $ openstack server list
    $ openstack baremetal node list
    $ openstack stack list

Procedure

1. Remove any existing images from the images directory on the stack user’s home (/home/stack/images):

   $ rm -rf ~/images/*

2. Extract the archives:

   $ cd ~/images
   $ for i in /usr/share/rhosp-director-images/overcloud-full-latest-12.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-12.0.tar; do tar -xvf $i; done
   $ cd ~

3. Import the latest images into the director:

   $ openstack overcloud image upload --update-existing --image-path /home/stack/images/

4. Configure your nodes to use the new images:

   $ openstack overcloud node configure $(openstack baremetal node list -c UUID -f value)

5. Verify the existence of the new images:

   $ openstack image list
   $ ls -l /httpboot

Procedure

1. Install the openstack-tripleo-heat-templates-compat package:

   $ sudo yum install openstack-tripleo-heat-templates-compat

   This installs the previous templates in the compat directory of your Heat template collection (/usr/share/openstack-tripleo-heat-templates/compat) and also creates a link to compat named after the previous version (ocata). These templates are backwards compatible with the upgraded director, which means you can use the latest version of the director to install an overcloud of the previous version.

2. Create a temporary copy of the core Heat templates:

   $ cp -a /usr/share/openstack-tripleo-heat-templates /tmp/osp12

3. Move the previous version into its own directory:

   $ mv /tmp/osp12/compat /tmp/osp11

4. Perform a diff on the contents of both directories:

   $ diff -urN /tmp/osp11 /tmp/osp12

   This shows the core template changes from one version to the next. These changes provide an idea of what should occur during the overcloud upgrade.

Procedure

1. Edit the environment file containing your registration details. For example:

   $ vi ~/templates/rhel-registration/environment-rhel-registration.yaml

2. Edit the following parameter values:

   rhel_reg_repos

   Update to include the new repositories for Red Hat OpenStack Platform 12.

   rhel_reg_activation_key

   Update the activation key to access the Red Hat OpenStack Platform 12 repositories.

   rhel_reg_sat_repo

   If using a newer version of Red Hat Satellite 6, update the repository containing Satellite 6’s management tools.

3. Save the environment file.

Procedure

1. Install the ceph-ansible package to the undercloud:

   [stack@director ~]$ sudo yum install -y ceph-ansible

2. Check that you are using the latest resources and configuration in your storage environment file. This requires the following changes:

   1. The resource_registry uses containerized services from the docker/services subdirectory of your core Heat template collection. For example:

1. Use the new CephAnsibleDisksConfig parameter to define how your disks are mapped. Previous versions of Red Hat OpenStack Platform used the ceph::profile::params::osds hieradata to define the OSD layout. Convert this hieradata to the structure of the CephAnsibleDisksConfig parameter. For example, if your hieradata contained the following:

   parameter_defaults:
     ExtraConfig:
       ceph::profile::params::osd_journal_size: 512
       ceph::profile::params::osds:
         '/dev/sdb': {}
         '/dev/sdc': {}
         '/dev/sdd': {}

   Then the CephAnsibleDisksConfig would look like this:

   parameters_default:
     CephAnsibleDisksConfig:
       devices:
       - /dev/sdb
       - /dev/sdc
       - /dev/sdd
       journal_size: 512
       osd_scenario: collocated

   For a full list of OSD disk layout options used in ceph-ansible, view the sample file in /usr/share/ceph-ansible/group_vars/osds.yml.sample.

Procedure

1. Edit the environment file containing your Ceph Storage configuration.

2. Ensure the resource_registry uses the Puppet resources. For example:

   resource_registry:
     OS::TripleO::Services::CephMon: ../puppet/services/ceph-mon.yaml
     OS::TripleO::Services::CephOSD: ../puppet/services/ceph-osd.yaml
     OS::TripleO::Services::CephClient: ../puppet/services/ceph-client.yaml

   Note

   Use the contents of the /usr/share/openstack-tripleo-heat-templates/environments/puppet-ceph.yaml as an example.

3. Upgrade your Controller-based nodes to containerized services using the instructions in Section 5.1, “Upgrading the Overcloud Nodes”.
4. Upgrade your HCI nodes using the instructions in Section 5.3, “Upgrading the Compute Nodes”

5. Edit the resource_registry in your Ceph Storage configuration to use the containerized services:

   resource_registry:
     OS::TripleO::Services::CephMon: ../docker/services/ceph-ansible/ceph-mon.yaml
     OS::TripleO::Services::CephOSD: ../docker/services/ceph-ansible/ceph-osd.yaml
     OS::TripleO::Services::CephClient: ../docker/services/ceph-ansible/ceph-client.yaml

   Note

   Use the contents of the /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml as an example.

6. Add the CephAnsiblePlaybook parameter to the parameter_defaults section of your storage environment file:

     CephAnsiblePlaybook: /usr/share/ceph-ansible/infrastructure-playbooks/switch-from-non-containerized-to-containerized-ceph-daemons.yml

7. Add the CephAnsibleDisksConfig parameter to the parameter_defaults section of your storage environment file and define the disk layout. For example:

     CephAnsibleDisksConfig:
       devices:
       - /dev/vdb
       - /dev/vdc
       - /dev/vdd
       journal_size: 512
       osd_scenario: collocated

8. Finalize the upgrade of your overcloud using the instructions in Section 5.4, “Finalizing the Upgrade”.

Procedure

1. The director’s dynamic Ansible script has updated to the OpenStack Platform 12 version, which uses the RoleNetHostnameMap Heat parameter in the overcloud plan to define the inventory. However, the overcloud currently uses the OpenStack Platform 11 template versions, which do not have the RoleNetHostnameMap parameter. This means you need to create a temporary static inventory file, which you can generate with the following command:

   $ openstack server list -c Networks -f value | cut -d"=" -f2 > overcloud_hosts

2. Create an Ansible playbook (undercloud-ca.yml) that contains the following:

   ---
   - name: Add undercloud CA to overcloud nodes
     hosts: all
     user: heat-admin
     become: true
     tasks:
       - name: Copy undercloud CA
         copy:
           src: ca.crt.pem
           dest: /etc/pki/ca-trust/source/anchors/
       - name: Update trust
         command: "update-ca-trust extract"
       - name: Get the hostname of the undercloud
         delegate_to: 127.0.0.1
         command: hostname
         register: undercloud_hostname
       - name: Verify URL
         uri:
           url: https://{{ undercloud_hostname.stdout }}:13808/healthcheck
           return_content: yes
         register: verify
       - name: Report output
         debug:
           msg: "{{ ansible_hostname }} can access the undercloud's Public API"
         when: verify.content == "OK"

   This playbook contains multiple tasks that perform the following on each node:

   • Copy the undercloud’s certificate authority file (ca.crt.pem) to the overcloud node. The name of this file and its location might vary depending on your configuration. This example uses the name and location defined during the self-signed certificate procedure (see "SSL/TLS Certificate Configuration" in the Director Installation and Usage guide).
   • Execute the command to update the certificate authority trust database on the overcloud node.
   • Checks the undercloud’s Object Storage Public API from the overcloud node and reports if successful.

3. Run the playbook with the following command:

   $ ansible-playbook -i overcloud_hosts undercloud-ca.yml

   This uses the temporary inventory to provide Ansible with your overcloud nodes.

4. The resulting Ansible output should show a debug message for node. For example:

   ok: [192.168.24.100] => {
       "msg": "overcloud-controller-0 can access the undercloud's Public API"
   }

Procedure

1. Run the following commands to save a list of node IP addresses in the OVERCLOUD_HOSTS environment variable:

   $ source ~/stackrc
   $ export OVERCLOUD_HOSTS=$(openstack server list -f value -c Networks | cut -d "=" -f 2 | tr '\n' ' ')

2. Run the following script:

   $ /usr/share/openstack-tripleo-heat-templates/deployed-server/scripts/enable-ssh-admin.sh

3. Proceed with the upgrade.

4. When upgrading Compute or Object Storage nodes, use the following:

   1. Use the -U option with the upgrade-non-controller.sh script and specify the stack user. This is because the default user for pre-provisioned nodes is stack and not heat-admin.

   2. Use the node’s IP address with the --upgrade option. This is because the node are not managed with the director’s Compute (nova) and Bare Metal (ironic) services and do not have a node name.

      For example:

      $ upgrade-non-controller.sh -U stack --upgrade 192.168.24.100

1. In the parameter_defaults section, add a network deployment parameter to run os-net-config during the upgrade process to associate the OVS 2.7 PCI address with DPDK ports:

   parameter_defaults:
     ComputeNetworkDeploymentActions: ['CREATE', 'UPDATE']

   The parameter name must match the name of the role you use to deploy DPDK. In this example, the role name is Compute so the parameter name is ComputeNetworkDeploymentActions.

   Note

   This parameter is not needed after the initial upgrade and should be removed from the environment file.

2. In the resource_registry section, override the ComputeNeutronOvsAgent service to the neutron-ovs-dpdk-agent puppet service:

   resource_registry:
     OS::TripleO::Services::ComputeNeutronOvsAgent: /usr/share/openstack-tripleo-heat-templates/puppet/services/neutron-ovs-dpdk-agent.yaml

   Red Hat OpenStack Platform 12 added a new service (OS::TripleO::Services::ComputeNeutronOvsDpdk) to support the addition of the new ComputeOvsDpdk role. The example above maps this externally for upgrades.

Procedure

1. Run the openstack overcloud deploy command and include:

   • All options and custom environment files relevant to your environment, such as network isolation and storage.
   • The overcloud_images.yaml environment file generated in Section 4.2, “Preparing for Containerized Services”.

   • The major-upgrade-composable-steps-docker.yaml environment file.

     For example:

   $ openstack overcloud deploy --templates \
     -e /home.stack/templates/node_count.yaml \
     -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
     -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
     -e /home/stack/templates/network_environment.yaml \
     -e /home/stack/templates/overcloud_images.yaml  \
     -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-composable-steps-docker.yaml \
     --ntp-server pool.ntp.org

2. Wait until the overcloud updates with the new environment file’s configuration.

   Important

   The upgrade disables the OpenStack Networking (neutron) server and L3 Agent. This means you cannot create new routers during this step. You can still access instances during this period.

3. Check if all services are active. For example, to check services on a Controllor node:

   [stack@director ~]$ ssh heat-admin@192.168.24.10
   [heat-admin@overcloud-controller-0 ~]$ sudo pcs status
   [heat-admin@overcloud-controller-0 ~]$ sudo docker ps

Procedure

1. Obtain a list of Object Storage nodes:

   $ openstack server list -c Name -f value --name objectstorage

2. Perform the following steps for each Object Storage node in the list:

   1. Run the upgrade-non-controller.sh script using the node name to identify the node to upgrade:

      $ upgrade-non-controller.sh --upgrade overcloud-objectstorage-0

      Note

      If using pre-provisioned node infrastructure, see Section 4.9, “Preparing for Pre-Provisioned Nodes Upgrade” for changes with this command.

   2. Wait until the Object Storage node completes the upgrade.

   3. Reboot the Object Storage node:

      $ openstack server reboot overcloud-objectstorage-0

   4. Wait until the Object Storage node completes the reboot.

1. Select a Compute node to upgrade
2. Migrate instances to another Compute node
3. Upgrade the empty Compute node
4. Reboot and enable the upgraded Compute node

1. List all Compute nodes:

   $ source ~/stackrc
   $ openstack server list -c Name -f value --name compute

2. Select a Compute node to upgrade and note its UUID and name.

1. From the undercloud, select a Compute Node to reboot and disable it:

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service list
   (overcloud) $ openstack compute service set [hostname] nova-compute --disable

2. List all instances on the Compute node:

   (overcloud) $ openstack server list --host [hostname] --all-projects

3. Use one of the following commands to migrate your instances:

   1. Migrate the instance to a specific host of your choice:

      (overcloud) $ openstack server migrate [instance-id] --live [target-host]--wait

   2. Let nova-scheduler automatically select the target host:

      (overcloud) $ nova live-migration [instance-id]

   3. Live migrate all instances at once:

      $ nova host-evacuate-live [hostname]

      Note

      The nova command might cause some deprecation warnings, which are safe to ignore.

4. Wait until migration completes.

5. Confirm the migration was successful:

   (overcloud) $ openstack server list --host [hostname] --all-projects

6. Continue migrating instances until none remain on the chosen Compute Node.

1. Run the upgrade-non-controller.sh script using the node name to identify the node to upgrade:

   $ upgrade-non-controller.sh --upgrade overcloud-compute-0

   Note

   If using pre-provisioned node infrastructure, see Section 4.9, “Preparing for Pre-Provisioned Nodes Upgrade” for changes with this command.

2. Wait until the Compute node completes the upgrade.

1. Log into the Compute Node and reboot it:

   [heat-admin@overcloud-compute-0 ~]$ sudo reboot

2. Wait until the node boots.

3. Enable the Compute Node again:

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service set [hostname] nova-compute --enable

4. Check whether the Compute node is enabled:

   (overcloud) $ openstack compute service list

Procedure

1. Run the openstack overcloud deploy command and include:

   • All options and custom environment files relevant to your environment, such as network isolation and storage.
   • The overcloud_images.yaml environment file generated in Section 4.2, “Preparing for Containerized Services”.

   • The major-upgrade-converge-docker.yaml environment file.

     For example:

   $ openstack overcloud deploy --templates \
     -e /home.stack/templates/node_count.yaml \
     -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
     -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
     -e /home/stack/templates/network_environment.yaml \
     -e /home/stack/templates/overcloud_images.yaml  \
     -e /usr/share/openstack-tripleo-heat-templates/environments/major-upgrade-converge-docker.yaml \
     --ntp-server pool.ntp.org

2. Wait until the overcloud updates with the new environment file’s configuration.

3. Check if all services are active. For example, to check services on a Controllor node:

   [stack@director ~]$ ssh heat-admin@192.168.24.10
   [heat-admin@overcloud-controller-0 ~]$ sudo pcs status
   [heat-admin@overcloud-controller-0 ~]$ sudo systemctl list-units 'openstack-*' 'neutron-*' 'httpd*'

Procedure

1. Create a new or edit an existing environment file. This example uses the filename undercloud-ca-map.yaml.

2. Add the CAMap parameter to the parameter_defaults section of the environment file. Use the following syntax as an example:

   parameter_defaults:
     CAMap:
       undercloud-ca: 1
         content: | 2
           -----BEGIN CERTIFICATE-----
           MIIDlTCCAn2gAwIBAgIJAOnPtx2hHEhrMA0GCSqGSIb3D
           BAYTAlVTMQswCQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZ
           UmVkIEhhdDELMAkGA1UECwwCUUUxFDASBgNVBAMMCzE5M
           ...
           ...
           -----END CERTIFICATE-----

   1

   This is the name that identifies the CA in each overcloud node’s trust database.

   2

   The content section is the actual CA certificate. Copy and paste the CA content in this section. Ensure the CA’s indentation matches the requirements for YAML syntax.

3. Save this file.
4. Include this file with subsequent execution of the openstack overcloud deploy command.

Procedure

1. Source the undercloud access details:

   $ source ~/stackrc

2. Check for failed Systemd services:

   (undercloud) $ sudo systemctl list-units --state=failed 'openstack*' 'neutron*' 'httpd' 'docker'

3. Check the undercloud free space:

   (undercloud) $ df -h

   +Use the "Undercloud Reqirements" as a basis to determine if you have adequate free space.

4. Check that clocks are synchronized on the undercloud:

   (undercloud) $ sudo ntpstat

5. Check the undercloud network services:

   (undercloud) $ openstack network agent list

   All agents should be Alive and their state should be UP.

6. Check the undercloud compute services:

   (undercloud) $ openstack compute service list

   All agents' status should be enabled and their state should be up

Procedure

1. Source the undercloud access details:

   $ source ~/stackrc

2. Check the status of your bare metal nodes:

   (undercloud) $ openstack baremetal node list

   All nodes should have a valid power state (on) and maintenance mode should be false.

3. Check for failed Systemd services:

   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo systemctl list-units --state=failed 'openstack*' 'neutron*' 'httpd' 'docker' 'ceph*'" ; done

4. Check for failed containerized services:

   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo docker ps -f 'exited=1' --all" ; done
   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo docker ps -f 'status=dead' -f 'status=restarting'" ; done

5. Check the HAProxy connection to all services. Obtain the Control Plane VIP address and authentication details for the haproxy.stats service:

   (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE sudo 'grep "listen haproxy.stats" -A 6 /var/lib/config-data/puppet-generated/haproxy/etc/haproxy/haproxy.cfg'

   Use these details in the following cURL request:

   (undercloud) $ curl -s -u admin:<PASSWORD> "http://<IP ADDRESS>:1993/;csv" | egrep -vi "(frontend|backend)" | awk -F',' '{ print $1" "$2" "$18 }'

   Replace <PASSWORD> and <IP ADDRESS> details with the respective details from the haproxy.stats service. The resulting list shows the OpenStack Platform services on each node and their connection status.

6. Check overcloud database replication health:

   (undercloud) $ for NODE in $(openstack server list --name controller -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo docker exec clustercheck clustercheck" ; done

7. Check RabbitMQ cluster health:

   (undercloud) $ for NODE in $(openstack server list --name controller -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo docker exec $(ssh heat-admin@$NODE "sudo docker ps -f 'name=.*rabbitmq.*' -q") rabbitmqctl node_health_check" ; done

8. Check Pacemaker resource health:

   (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo pcs status"

   Look for:

   • All cluster nodes online.
   • No resources stopped on any cluster nodes.
   • No failed pacemaker actions.

9. Check the disk space on each overcloud node:

   (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo df -h --output=source,fstype,avail -x overlay -x tmpfs -x devtmpfs" ; done

10. Check overcloud Ceph Storage cluster health. The following command runs the ceph tool on a Controller node to check the cluster:

    (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo ceph -s"

11. Check Ceph Storage OSD for free space. The following command runs the ceph tool on a Controller node to check the free space:

    (undercloud) $ NODE=$(openstack server list --name controller-0 -f value -c Networks | cut -d= -f2); ssh heat-admin@$NODE "sudo ceph df"

12. Check that clocks are synchronized on overcloud nodes

    (undercloud) $ for NODE in $(openstack server list -f value -c Networks | cut -d= -f2); do echo "=== $NODE ===" ; ssh heat-admin@$NODE "sudo ntpstat" ; done

13. Source the overcloud access details:

    (undercloud) $ source ~/overcloudrc

14. Check the overcloud network services:

    (overcloud) $ openstack network agent list

    All agents should be Alive and their state should be UP.

15. Check the overcloud compute services:

    (overcloud) $ openstack compute service list

    All agents' status should be enabled and their state should be up

16. Check the overcloud volume services:

    (overcloud) $ openstack volume service list

    All agents' status should be enabled and their state should be up.

Procedure

1. Log into the director as the stack user.

2. Update the python-tripleoclient package and its dependencies to ensure you have the latest scripts for the minor version update:

   $ sudo yum update -y python-tripleoclient

3. The director uses the openstack undercloud upgrade command to update the Undercloud environment. Run the command:

   $ openstack undercloud upgrade

4. Reboot the node:

   $ sudo reboot

5. Wait until the node boots.

6. Check the status of all services:

   $ sudo systemctl list-units "openstack*" "neutron*" "openvswitch*"

   Note

   It might take approximately 10 minutes for the openstack-nova-compute to become active after a reboot.

7. Verify the existence of your overcloud and its nodes:

   $ source ~/stackrc
   $ openstack server list
   $ openstack baremetal node list
   $ openstack stack list

Procedure

1. Check the yum log to determine if new image archives are available:

   $ sudo grep "rhosp-director-images" /var/log/yum.log

2. If new archives are available, replace your current images with new images. To install the new images, first remove any existing images from the images directory on the stack user’s home (/home/stack/images):

   $ rm -rf ~/images/*

3. Extract the archives:

   $ cd ~/images
   $ for i in /usr/share/rhosp-director-images/overcloud-full-latest-12.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-12.0.tar; do tar -xvf $i; done

4. Import the latest images into the director and configure nodes to use the new images:

   $ cd ~
   $ openstack overcloud image upload --update-existing --image-path /home/stack/images/
   $ openstack overcloud node configure $(openstack baremetal node list -c UUID -f csv --quote none | sed "1d" | paste -s -d " ")

5. To finalize the image update, verify the existence of the new images:

   $ openstack image list
   $ ls -l /httpboot

   The director is now updated and using the latest images. You do not need to restart any services after the update.

Procedure

1. Find the latest tag for the containerized service images:

   $ openstack overcloud container image tag discover \
     --image registry.access.redhat.com/rhosp12/openstack-base:latest \
     --tag-from-label version-release

   Make a note of the most recent tag.

2. Create an updated environment file for your container image source. Run using the openstack overcloud container image prepare command. For example, to use images from registry.access.redhat.com:

   $ openstack overcloud container image prepare \
     --namespace=registry.access.redhat.com/rhosp12 \
     --prefix=openstack- \
     --tag [TAG] \ 1
     --set ceph_namespace=registry.access.redhat.com/rhceph \
     --set ceph_image=rhceph-2-rhel7 \
     --set ceph_tag=latest \
     --env-file=/home/stack/templates/overcloud_images.yaml \
     -e /home/stack/templates/custom_environment_file.yaml 2

   1

   Replace [TAG] with the tag obtained from the previous step.

   2

   Include all additional environment files with the -e parameter. The director checks the custom resources in all included environment files and identifies the container images required for the containerized services.

   For more information about generating this environment file for different source types, see "Configuring Container Registry Details" in the Director Installation and Usage guide.

3. Run the openstack overcloud update stack command to update the container image locations in your overcloud:

   $ openstack overcloud update stack --init-minor-update \
     --container-registry-file /home/stack/templates/overcloud_images.yaml

   The --init-minor-update only performs an update of the parameters in the overcloud stack. It does not perform the actual package or container update. Wait until this command completes.

4. Perform a package and container update using the openstack overcloud update command. Using the --nodes option to upgrade node for each role. For example, the following command updates nodes in the Controller role

   $ openstack overcloud update stack --nodes Controller

   Run this command for each role group in the following order:

   • Controller
   • CephStorage
   • Compute
   • ObjectStorage
   • Any custom roles such as Database, MessageBus, Networker, and so forth.
5. The update process starts for the chosen role starts. The director uses an Ansible playbook to perform the update and displays the output of each task.
6. Update the next role group. Repeat until you have updated all nodes.
7. The update process does not reboot any nodes in the Overcloud automatically. Updates to the kernel or Open vSwitch require a reboot. Check the /var/log/yum.log file on each node to see if either the kernel or openvswitch packages have updated their major or minor versions. If they have, reboot each node using the "Rebooting Nodes" procedures in the Director Installation and Usage guide.