Red Hat Training

A Red Hat training course is available for Red Hat OpenStack Platform

Chapter 4. Updating the Overcloud

This process updates the overcloud.

Prerequisites

  • You have updated the undercloud to the latest version.

4.1. Speeding up the overcloud update

To speed up the overcloud update process, you can configure the DockerPuppetProcessCount heat parameter, clear your database entries, and download the required packages on the overcloud nodes before you perform an update.

For more information about speeding up the update process for large OpenStack deployments, see the Red Hat Knowledgebase article Openstack Director Node Performance Tuning for large deployments.

Procedure

  1. Log in to the undercloud as the stack user.
  2. To increase the number of concurrent processes that container-puppet uses to generate configuration files, you must configure the DockerPuppetProcessCount parameter.

    1. Create an environment file called updates-environment.yaml in your templates directory:

      $ touch ~/templates/updates-environment.yaml
    2. Edit the file and add the following content:

      parameters:
          DockerPuppetProcessCount: 8
    3. Use the -e option to include this environment file when you run the openstack overcloud update prepare, openstack overcloud ceph-upgrade run, and openstack overcloud update converge commands.
  3. Clear your database entries:

    $ sudo docker exec -u 42436 -ti nova_api_cron bash
    $ nova-manage db archive_deleted_rows --max_rows 1000 --before `date --date='today - 90 days' +\%F`  --all-cells
    $ exit
  4. To download all packages that are required for an update on all the overcloud nodes, complete the following steps:

    1. Create a static inventory file of your overcloud:

      $ tripleo-ansible-inventory \
      --ansible_ssh_user heat-admin \
      --static-yaml-inventory ~/inventory.yaml
    2. Create the following Ansible playbook:

      $ cat > ~/yum-download-only.yaml <<'EOF'
      - hosts: all
        gather_facts: false
        tasks:
          - name: Pre-download all packages on all overcloud nodes
            yum:
              download_only: yes
              state: latest
              name: '*'
            become: true
      EOF
    3. Run the yum-download-only.yaml Ansible playbook:

      $ ansible-playbook \
      -i ~/inventory.yaml \
      -f 20 ~/yum-download-only.yaml \
      --limit Controller,Compute,CephStorage

4.2. Consideration for custom roles

Check the following values in your roles file if your deployment includes custom roles:

  • Compare your custom roles file with the latest files in the /usr/share/openstack-tripleo-heat-templates/roles directory. Add any new parameters from the RoleParametersDefault sections for relevant roles for your environment to the equivalent roles in your custom roles file.
  • If you use Data Plane Development Kit (DPDK) and are upgrading from 13.4 or lower, ensure that the roles that contain OVS-DPDK services also contain the following mandatory parameters:

      RoleParametersDefault:
        VhostuserSocketGroup: "hugetlbfs"
        TunedProfileName: "cpu-paritioning"
        NovaLibvirtRxQueueSize: 1024
        NovaLibvirtTxQueueSize: 1024

4.3. Running the overcloud update preparation

The update requires running openstack overcloud update prepare command, which performs the following tasks:

  • Updates the overcloud plan to OpenStack Platform 13
  • Prepares the nodes for the update

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the update preparation command:

    $ openstack overcloud update prepare \
        --templates \
        -r <ROLES DATA FILE> \
        -n <NETWORK DATA FILE> \
        -e /home/stack/templates/overcloud_images.yaml \
        -e /home/stack/templates/updates-environment.yaml \
        -e <ENVIRONMENT FILE> \
        -e <ENVIRONMENT FILE> \
        --stack <STACK_NAME>
        ...

    Include the following options relevant to your environment:

    • Custom configuration environment files (-e)
    • The environment file with your new container image locations (-e). Note that the update command might display a warning about using the --container-registry-file. You can ignore this warning as this option is deprecated in favor of using -e for the container image environment file.
    • If using your own custom roles, include your custom roles (roles_data) file (-r)
    • If using custom networks, include your composable network (network_data) file (-n)
    • If the name of your overcloud stack is different to the default name overcloud, include the --stack option in the update preparation command and replace <STACK_NAME> with the name of your stack.
  3. Wait until the update preparation completes.

4.4. Updating all Controller nodes

To update the Controller nodes to the latest Red Hat OpenStack Platform (RHOSP) 13 version, include the --nodes Controller option in the openstack overcloud update run command. The --nodes Controller option restricts the update operations to the Controller nodes only.

Warning
If you use Ceph, see the Red Hat Knowledgebase solution During minor update of OSP13/RHCS3 to latest packages Ceph services go offline and need to be manually restarted before you update the Controller nodes in order to avoid bug BZ#1910842.

Prerequisites

  • If you use the load-balancing service (octavia) and want to update from a release earlier than RHOSP 13 z13 (8 October 2020 maintenance release), to avoid bug BZ#1927169, you must run the database migrations that upgrade the load-balancing service in the correct order. You must update the bootstrap Controller node, before you can update the rest of the control plane.

    1. To identify your current maintenance release, run the following command:

      $ cat /etc/rhosp-release
    2. On the undercloud node, to identify the bootstrap Controller node, run the following command and replace <any_controller_node_IP_address> with the IP address of any of the Controller nodes in your deployment:

      $ ssh heat-admin@<any_controller_node_IP_address> sudo hiera -c /etc/puppet/hiera.yaml octavia_api_short_bootstrap_node_name
    3. On the undercloud node, run the openstack overcloud update run command to update the bootstrap Controller node:

      $ openstack overcloud update run --nodes <bootstrap_node_name>

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the update command:

    $ openstack overcloud update run --nodes Controller
  3. Wait until the Controller node update completes.

4.5. Updating all Compute nodes

This process updates all Compute nodes to the latest OpenStack Platform 13 version. The process involves running the openstack overcloud update run command and including the --nodes Compute option to restrict operations to the Compute nodes only.

Parallelization considerations

When you update a large number of Compute nodes, to improve performance, you can run the openstack overcloud update run command with the --nodes Compute option in parallel on batches of 20 nodes. For example, if you have 80 Compute nodes in your deployment, you can run the following commands to update the Compute nodes in parallel:

$ openstack overcloud update run --nodes 'Compute[0:19]' > update-compute-0-19.log 2>&1 &
$ openstack overcloud update run --nodes 'Compute[20:39]' > update-compute-20-39.log 2>&1 &
$ openstack overcloud update run --nodes 'Compute[40:59]' > update-compute-40-59.log 2>&1 &
$ openstack overcloud update run --nodes 'Compute[60:79]' > update-compute-60-79.log 2>&1 &

The 'Compute[0:19]', 'Compute[20:39]', 'Compute[40:59]', and 'Compute[60:79]' way of partitioning the nodes space is random and you don’t have control over the order in which the nodes are updated in each batch.

To update specific Compute nodes, list the nodes that you want to update in a batch separated by a comma:

$ openstack overcloud update run --nodes <Compute0>,<Compute1>,<Compute2>,<Compute3>

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the update command:

    $ openstack overcloud update run --nodes Compute
  3. Wait until the Compute node update completes.

4.6. Updating all HCI Compute nodes

This process updates the Hyperconverged Infrastructure (HCI) Compute nodes. The process involves:

Note

Currently, the following combinations of Ansible with ceph-ansible are supported:

  • ansible-2.6 with ceph-ansible-3.2
  • ansible-2.4 with ceph-ansible-3.1

If your environment has ansible-2.6 with ceph-ansible-3.1, run the following commands to update ceph-ansible to the newest version:

  # subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms
  # subscription-manager repos --enable=rhel-7-server-ansible-2.6-rpms
  # yum update ceph-ansible

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the update command:

    $ openstack overcloud update run --nodes ComputeHCI
  3. Wait until the node update completes.
  4. Run the Ceph Storage update command. For example:

    $ openstack overcloud ceph-upgrade run \
        --templates \
        -e <ENVIRONMENT FILE> \
        -e /home/stack/templates/overcloud_images.yaml \
        -e /home/stack/templates/updates-environment.yaml

    Include the following options relevant to your environment:

    • Custom configuration environment files (-e)
    • The environment file with your container image locations (-e). Note that the update command might display a warning about using the --container-registry-file. You can ignore this warning as this option is deprecated in favor of using -e for the container image environment file.
    • If applicable, your custom roles (roles_data) file (--roles-file)
    • If applicable, your composable network (network_data) file (--networks-file)
  5. Wait until the Compute HCI node update completes.

4.7. Updating all Ceph Storage nodes

This process updates the Ceph Storage nodes. The process involves:

Note

Currently, the following combinations of Ansible with ceph-ansible are supported:

  • ansible-2.6 with ceph-ansible-3.2
  • ansible-2.4 with ceph-ansible-3.1

If your environment has ansible-2.6 with ceph-ansible-3.1, run the following commands to update ceph-ansible to the newest version:

  # subscription-manager repos --enable=rhel-7-server-rhceph-3-tools-rpms
  # subscription-manager repos --enable=rhel-7-server-ansible-2.6-rpms
  # yum update ceph-ansible

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the update command:

    $ openstack overcloud update run --nodes CephStorage
  3. Wait until the node update completes.
  4. Run the Ceph Storage update command. For example:

    $ openstack overcloud ceph-upgrade run \
        --templates \
        -e <ENVIRONMENT FILE> \
        -e /home/stack/templates/overcloud_images.yaml \
        -e /home/stack/templates/updates-environment.yaml

    Include the following options relevant to your environment:

    • Custom configuration environment files (-e)
    • The environment file with your container image locations (-e). Note that the update command might display a warning about using the --container-registry-file. You can ignore this warning as this option is deprecated in favor of using -e for the container image environment file.
    • If applicable, your custom roles (roles_data) file (--roles-file)
    • If applicable, your composable network (network_data) file (--networks-file)
  5. Wait until the Ceph Storage node update completes.

4.8. Finalizing the update

The update requires a final step to update the overcloud stack. This ensures that the stack resource structure aligns with a regular deployment of Red Hat OpenStack Platform 13 and you can perform standard openstack overcloud deploy functions in the future.

Procedure

  1. Source the stackrc file:

    $ source ~/stackrc
  2. Run the update finalization command:

    $ openstack overcloud update converge \
        --templates \
        --stack <STACK_NAME> \
        -e /home/stack/templates/overcloud_images.yaml \
        -e /home/stack/templates/updates-environment.yaml \
        -e <ENVIRONMENT FILE>
        ...

    Include the following options relevant to your environment:

    • Custom configuration environment files (-e)
    • The environment file with your new container image locations (-e). Note that the update command might display a warning about using the --container-registry-file. You can ignore this warning as this option is deprecated in favor of using -e for the container image environment file.
    • If applicable, your custom roles (roles_data) file (--roles-file)
    • If applicable, your composable network (network_data) file (--networks-file)
    • If the name of your overcloud stack is different than the default name overcloud, you must include the --stack option in the update preparation command and replace <STACK_NAME> with the name of your overcloud stack.
  3. Wait until the update finalization completes.