Red Hat Training

A Red Hat training course is available for Red Hat OpenStack Platform

Chapter 4. Updating the Overcloud

This process updates the overcloud.

Prerequisites

  • You have updated the undercloud to the latest version.

4.1. Speeding up the overcloud update

To speed up the overcloud update process, you can configure the DockerPuppetProcessCount heat parameter, archive deleted database entries, and download the required packages on the overcloud nodes before you perform an update.

For more information about speeding up the update process for large OpenStack deployments, see the Red Hat Knowledgebase article Openstack Director Node Performance Tuning for large deployments.

Procedure

  1. Log in to the undercloud as the stack user.
  2. Source the stackrc file:

    $ source ~/stackrc
  3. To increase the number of concurrent processes that container-puppet uses to generate configuration files, you must configure the DockerPuppetProcessCount parameter.

    1. Create an environment file called updates-environment.yaml in your templates directory:

      $ touch ~/templates/updates-environment.yaml
    2. Edit the file and add the following content:

      parameter_defaults:
          DockerPuppetProcessCount: 8
    3. Use the -e option to include this environment file when you run the openstack overcloud update prepare, openstack overcloud ceph-upgrade run, and openstack overcloud update converge commands.
  4. On a Controller node, archive your deleted database entries:

    1. From the overcloud, list all instances of the Controller nodes:

      $ source ~/overcloudrc
      $ openstack server list
    2. Log on to a Controller node that is running the nova_api_cron container:

      ssh heat-admin@<controller_ip>
      • Replace <controller name or IP> with the IP address of a Controller node.
    3. Archive deleted database entries:

      $ sudo docker exec -u 42436 -ti nova_api_cron bash
      $ nova-manage db archive_deleted_rows --max_rows 1000
      $ exit
  5. To download all packages that are required for an update on all the overcloud nodes, complete the following steps:

    1. Create a static inventory file of your overcloud:

      $ tripleo-ansible-inventory \
      --ansible_ssh_user heat-admin \
      --static-yaml-inventory ~/inventory.yaml
    2. Create the following Ansible playbook:

      $ cat > ~/yum-download-only.yaml <<'EOF'
      - hosts: all
        gather_facts: false
        tasks:
          - name: Pre-download all packages on all overcloud nodes
            shell:
              yum upgrade -y --downloadonly
            become: true
      EOF
    3. Run the yum-download-only.yaml Ansible playbook:

      $ ansible-playbook \
      -i ~/inventory.yaml \
      -f 20 ~/yum-download-only.yaml \
      --limit Controller,Compute,CephStorage

4.2. Consideration for custom roles

Check the following values in your roles file if your deployment includes custom roles:

  • Compare your custom roles file with the latest files in the /usr/share/openstack-tripleo-heat-templates/roles directory. Add any new parameters from the RoleParametersDefault sections for relevant roles for your environment to the equivalent roles in your custom roles file.
  • If you use Data Plane Development Kit (DPDK) and are upgrading from 13.4 or lower, ensure that the roles that contain OVS-DPDK services also contain the following mandatory parameters:

      RoleParametersDefault:
        VhostuserSocketGroup: "hugetlbfs"
        TunedProfileName: "cpu-paritioning"
        NovaLibvirtRxQueueSize: 1024
        NovaLibvirtTxQueueSize: 1024

4.3. Running the overcloud update preparation

The update requires running openstack overcloud update prepare command, which performs the following tasks:

  • Updates the overcloud plan to OpenStack Platform 13
  • Prepares the nodes for the update

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the update preparation command:

    $ openstack overcloud update prepare \
        --templates \
        -r <ROLES DATA FILE> \
        -n <NETWORK DATA FILE> \
        -e /home/stack/templates/overcloud_images.yaml \
        -e /home/stack/templates/updates-environment.yaml \
        -e <ENVIRONMENT FILE> \
        -e <ENVIRONMENT FILE> \
        --stack <STACK_NAME>
        ...

    Include the following options relevant to your environment:

    • Custom configuration environment files (-e).
    • The environment file with your new container image locations (-e). Note that the update command might display a warning about using the --container-registry-file. You can ignore this warning as this option is deprecated in favor of using -e for the container image environment file.
    • If you use your own custom roles, include your custom roles (roles_data) file (-r).
    • If you use custom networks, include your composable network (network_data) file (-n).
    • If you deploy a high availability cluster, include the --ntp-server option in the update preparation command, or include the NtpServer parameter and value in your environment file.
    • If the name of your overcloud stack is different to the default name overcloud, include the --stack option in the update preparation command and replace <STACK_NAME> with the name of your stack.
  3. Wait until the update preparation completes.

4.4. Updating the Ceph Storage cluster

This process updates the Ceph Storage cluster. The process involves running the openstack overcloud ceph-upgrade run command to perform an update to a Red Hat Ceph Storage 3 cluster.

Note

The following combinations of Ansible with ceph-ansible are supported:

  • ansible-2.6 with ceph-ansible-3.2
  • ansible-2.4 with ceph-ansible-3.1

If your environment has ansible-2.6 with ceph-ansible-3.1, update ceph-ansible to the newest version:

  # subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms
  # subscription-manager repos --enable=rhel-7-server-ansible-2.6-rpms
  # yum update ceph-ansible

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the Ceph Storage update command. For example:

    $ openstack overcloud ceph-upgrade run \
        --templates \
        -e <ENVIRONMENT FILE> \
        -e /home/stack/templates/overcloud_images.yaml \
        -e /home/stack/templates/updates-environment.yaml

    Include the following options relevant to your environment:

    • Custom configuration environment files (-e)
    • The environment file with your container image locations (-e). Note that the update command might display a warning about using the --container-registry-file. You can ignore this warning, as this option is deprecated in favor of using -e for the container image environment file.
    • If applicable, your custom roles (roles_data) file (--roles-file)
    • If applicable, your composable network (network_data) file (--networks-file)
  3. Wait until the Ceph Storage node update completes.
Note

If the Heat stack times out during the procedure, see the Red Hat Knowledgebase article During sequential update of Ceph nodes openstack overcloud ceph-upgrade run appears to time out.

4.5. Updating all Controller nodes

To update the Controller nodes to the latest Red Hat OpenStack Platform (RHOSP) 13 version, include the --nodes Controller option in the openstack overcloud update run command. The --nodes Controller option restricts the update operations to the Controller nodes only.

Warning
If you use Ceph, see the Red Hat Knowledgebase solution During minor update of OSP13/RHCS3 to latest packages Ceph services go offline and need to be manually restarted before you update the Controller nodes in order to avoid bug BZ#1910842.

Prerequisites

  • If you use the load-balancing service (octavia) and want to update from a release earlier than RHOSP 13 z13 (8 October 2020 maintenance release), to avoid bug BZ#1927169, you must run the database migrations that upgrade the load-balancing service in the correct order. You must update the bootstrap Controller node, before you can update the rest of the control plane.

    1. To identify your current maintenance release, run the following command:

      $ cat /etc/rhosp-release
    2. On the undercloud node, to identify the bootstrap Controller node, run the following command and replace <any_controller_node_IP_address> with the IP address of any of the Controller nodes in your deployment:

      $ ssh heat-admin@<any_controller_node_IP_address> sudo hiera -c /etc/puppet/hiera.yaml octavia_api_short_bootstrap_node_name
    3. On the undercloud node, run the openstack overcloud update run command to update the bootstrap Controller node:

      $ openstack overcloud update run --nodes <bootstrap_node_name>

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the update command:

    $ openstack overcloud update run --nodes Controller
  3. Wait until the Controller node update completes.

4.6. Updating all Compute nodes

This process updates all Compute nodes to the latest OpenStack Platform 13 version. The process involves running the openstack overcloud update run command and including the --nodes Compute option to restrict operations to the Compute nodes only.

Parallelization considerations

When you update a large number of Compute nodes, to improve performance, you can run the openstack overcloud update run command with the --nodes Compute option in parallel on batches of 20 nodes. For example, if you have 80 Compute nodes in your deployment, you can run the following commands to update the Compute nodes in parallel:

$ openstack overcloud update run --nodes 'Compute[0:19]' > update-compute-0-19.log 2>&1 &
$ openstack overcloud update run --nodes 'Compute[20:39]' > update-compute-20-39.log 2>&1 &
$ openstack overcloud update run --nodes 'Compute[40:59]' > update-compute-40-59.log 2>&1 &
$ openstack overcloud update run --nodes 'Compute[60:79]' > update-compute-60-79.log 2>&1 &

The 'Compute[0:19]', 'Compute[20:39]', 'Compute[40:59]', and 'Compute[60:79]' way of partitioning the nodes space is random and you don’t have control over the order in which the nodes are updated in each batch.

To update specific Compute nodes, list the nodes that you want to update in a batch separated by a comma:

$ openstack overcloud update run --nodes <Compute0>,<Compute1>,<Compute2>,<Compute3>

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the update command:

    $ openstack overcloud update run --nodes Compute
  3. Wait until the Compute node update completes.

4.7. Updating all HCI Compute nodes

This process updates the Hyperconverged Infrastructure (HCI) Compute nodes. The process involves running the openstack overcloud update run command and including the --nodes ComputeHCI option to restrict operations to the HCI nodes only.

Warning
If you are using Ceph, see the Red Hat Knowledgebase solution During minor update of OSP13/RHCS3 to latest packages Ceph services go offline and need to be manually restarted before you follow this section in order to avoid the following bug BZ#1910842.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the update command:

    $ openstack overcloud update run --nodes ComputeHCI
  3. Wait until the node update completes.

4.8. Updating all Ceph Storage nodes

This process updates the Ceph Storage nodes. The process involves running the openstack overcloud update run command and including the --nodes CephStorage option to restrict operations to the Ceph Storage nodes only.

Warning
If you are using Ceph, see the Red Hat Knowledgebase solution During minor update of OSP13/RHCS3 to latest packages Ceph services go offline and need to be manually restarted before you follow this section in order to avoid the following bug BZ#1910842.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Update group nodes.

    To update all nodes in a group:

    $ openstack overcloud update run --nodes <GROUP_NAME>

    To update a single node in a group:

    $ openstack overcloud update run --nodes <GROUP_NAME> [NODE_INDEX]
    Note

    Ensure that you update all nodes if you choose to update nodes individually.

    The index of the first node in a group is zero (0). For example, to update the first node in a group named CephStorage, the command is:

    openstack overcloud update run --nodes CephStorage[0]

  3. Wait until the node update completes.

4.9. Finalizing the update

The update requires a final step to update the overcloud stack. This ensures that the stack resource structure aligns with a regular deployment of Red Hat OpenStack Platform 13 and you can perform standard openstack overcloud deploy functions in the future.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the update finalization command:

    $ openstack overcloud update converge \
        --templates \
        --stack <STACK_NAME> \
        -e /home/stack/templates/overcloud_images.yaml \
        -e /home/stack/templates/updates-environment.yaml \
        -e <ENVIRONMENT FILE>
        ...

    Include the following options relevant to your environment:

    • Custom configuration environment files (-e)
    • The environment file with your new container image locations (-e). Note that the update command might display a warning about using the --container-registry-file. You can ignore this warning as this option is deprecated in favor of using -e for the container image environment file.
    • If applicable, your custom roles (roles_data) file (--roles-file)
    • If applicable, your composable network (network_data) file (--networks-file)
    • If the name of your overcloud stack is different than the default name overcloud, you must include the --stack option in the update preparation command and replace <STACK_NAME> with the name of your overcloud stack.
  3. Wait until the update finalization completes.