Red Hat Training

A Red Hat training course is available for Red Hat OpenStack Platform

Chapter 2. Director-Based Environments: Performing Updates to Minor Versions

This section explores how to update packages for your Red Hat OpenStack Platform environment within the same version. In this case, it is updates within Red Hat OpenStack Platform 10. This includes updating aspects of both the Undercloud and Overcloud.

Prerequisites

  • Before you can perform a minor update on OSP10, you must edit the storage-environment.yaml file and set the UUID in NodeDataLookup to lowercase. For more information, see the Red Hat Knowledgebase solution minor update on OSP10 fails on ceph.
Warning

With High Availaibility for Compute instances (or Instance HA, as described in High Availability for Compute Instances), upgrades or scale-up operations are not possible. Any attempts to do so will fail.

If you have Instance HA enabled, disable it before performing an upgrade or scale-up. To do so, perform a rollback as described in Rollback.

This procedure for both situations involves the following workflow:

  1. Update the Red Hat OpenStack Platform director packages
  2. Update the Overcloud images on the Red Hat OpenStack Platform director
  3. Update the Overcloud packages using the Red Hat OpenStack Platform director

2.1. Pre-Update Notes

2.1.1. General Recommendations

Before performing the update, Red Hat advises the following:

  • Perform a backup of your Undercloud node before starting any steps in the update procedure. See the Back Up and Restore the Director Undercloud guide for backup procedures.
  • Red Hat OpenStack Platform 10 ELS is only supported on Red Hat Enterprise Linux (RHEL) 7.7, therefore, it is important that you set the RHEL version to RHEL 7.7 before you perform an update.
  • Before you can perform a minor update on OSP10, you must edit the storage-environment.yaml file and set the UUID in NodeDataLookup to lowercase. For more information, see the Red Hat Knowledgebase solution minor update on OSP10 fails on ceph.
  • Run the update procedure in a test environment that includes all of the changes made before running the procedure in your production environment.
  • If necessary, please contact Red Hat and request any guidance and assistance for performing an update.

2.1.2. NFV Pre-Configuration

An Overcloud with Network Functions Virtualization (NFV) enabled requires some additional preparation to accommodate any updates to the Open vSwitch (OVS) package. To support this transition when you have OVS-DPDK configured, follow these guidelines.

Note

Red Hat OpenStack Platform 10 with OVS 2.9 operates in OVS client mode for OVS-DPDK deployments.

  1. Change the vhost user socket directory in a custom environment file, for example, network-environment.yaml: 

    parameter_defaults:
  NeutronVhostuserSocketDir: "/var/lib/vhost_sockets"

  2. Add the ovs-dpdk-permissions.yaml file to your openstack overcloud deploy command to configure the qemu group setting as hugetlbfs for OVS-DPDK: 

     -e /usr/share/openstack-tripleo-heat-templates/environments/ovs-dpdk-permissions.yaml

2.2. Updating Red Hat OpenStack Platform

2.2.1. Updating the Undercloud Packages

The director relies on standard RPM methods to update your environment. This involves ensuring your director’s host uses the latest packages through yum.

  1. Log into the director as the stack user.

  2. Stop the main OpenStack Platform services: 

    $ sudo systemctl stop 'openstack-*' 'neutron-*' httpd
    Note

    This causes a short period of downtime for the undercloud. The overcloud is still functional during the undercloud update.

  3. Set the RHEL version to RHEL 7.7: 

    $ sudo subscription-manager release --set=7.7

  4. Update the python-tripleoclient package and its dependencies to ensure you have the latest scripts for the minor version update: 

    $ sudo yum update python-tripleoclient

  5. The director uses the openstack undercloud upgrade command to update the Undercloud environment. Run the command: 

    $ openstack undercloud upgrade

Major and minor version updates to the kernel or Open vSwitch require a reboot, such as when your undercloud operating system updates from Red Hat Enterprise Linux 7.2 to 7.3, or Open vSwitch from version 2.4 to 2.5. Check the /var/log/yum.log file on the director node to see if either the kernel or openvswitch packages have updated their major or minor versions. If they have, perform a reboot of each node:

  1. Reboot the node: 

    $ sudo reboot
  2. Wait until the node boots.

  3. When the node boots, check the status of all services: 

    $ sudo systemctl list-units "openstack*" "neutron*" "openvswitch*"
    Note

    It might take approximately 10 minutes for the openstack-nova-compute to become active after a reboot.

  4. Verify the existence of your Overcloud and its nodes: 

    $ source ~/stackrc
$ openstack server list
$ openstack baremetal node list
$ openstack stack list

It is important to keep your overcloud images up to date to ensure the image configuration matches the requirements of the latest openstack-tripleo-heat-template package. To ensure successful deployments and scaling operations in the future, update your overclouds images using the instructions in Section 2.2.2, “Updating the Overcloud Images”.

2.2.2. Updating the Overcloud Images

The Undercloud update process might download new image archives from the rhosp-director-images and rhosp-director-images-ipa packages. Check the yum log to determine if new image archives are available: 

$ sudo grep "rhosp-director-images" /var/log/yum.log

If new archives are available, replace your current images with new images. To install the new images, first remove any existing images from the images directory on the stack user’s home (/home/stack/images): 

$ rm -rf ~/images/*

Extract the archives: 

$ cd ~/images
$ for i in /usr/share/rhosp-director-images/overcloud-full-latest-10.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-10.0.tar; do tar -xvf $i; done

Import the latest images into the director and configure nodes to use the new images 

$ openstack overcloud image upload --update-existing --image-path /home/stack/images/
$ openstack baremetal configure boot

To finalize the image update, verify the existence of the new images: 

$ openstack image list
$ ls -l /httpboot

The director is now updated and using the latest images. You do not need to restart any services after the update.

2.2.3. Updating the Overcloud Packages

The Overcloud relies on standard RPM methods to update the environment. This involves the following steps:

  1. Check your subscription management configuration for the rhel_reg_release parameter. If this parameter is not set, you must include it and set it to version 7.7: 

    parameter_defaults:
  ...
  rhel_reg_release: "7.7"

    Ensure that you save the changes to the overcloud subscription management environment file.

  2. Update the current plan using your original openstack overcloud deploy command and including the --update-plan-only option. For example: 

    $ openstack overcloud deploy --update-plan-only \
  --templates  \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e /home/stack/templates/network-environment.yaml \
  -e /home/stack/templates/storage-environment.yaml \
  -e /home/stack/templates/rhel-registration/environment-rhel-registration.yaml \
  [-e <environment_file>|...]

    The --update-plan-only only updates the Overcloud plan stored in the director. Use the -e to include environment files relevant to your Overcloud and its update path. The order of the environment files is important as the parameters and resources defined in subsequent environment files take precedence. Use the following list as an example of the environment file order:

    • Any network isolation files, including the initialization file (environments/network-isolation.yaml) from the heat template collection and then your custom NIC configuration file.
    • Any external load balancing environment files.
    • Any storage environment files.
    • Any environment files for Red Hat CDN or Satellite registration.
    • Any other custom environment files.

  3. Perform a package update on all nodes using the openstack overcloud update command. For example: 

    $ openstack overcloud update stack -i overcloud

    Running an update on all nodes in parallel might cause problems. For example, an update of a package might involve restarting a service, which can disrupt other nodes. This is why the process updates each node using a set of breakpoints. This means nodes are updated one by one. When one node completes the package update, the update process moves to the next node. The update process also requires the -i option, which puts the command in an interactive mode that requires confirmation at each breakpoint. Without the -i option, the update remains paused at the first breakpoint.

This starts update process. During this process, the director reports an IN_PROGRESS status and periodically prompts you to clear breakpoints. For example: 

not_started: [u'overcloud-controller-0', u'overcloud-controller-1', u'overcloud-controller-2']
on_breakpoint: [u'overcloud-compute-0']
Breakpoint reached, continue? Regexp or Enter=proceed, no=cancel update, C-c=quit interactive mode:

Press Enter to clear the breakpoint from last node on the on_breakpoint list. This begins the update for that node. You can also type a node name to clear a breakpoint on a specific node, or a Python-based regular expression to clear breakpoints on multiple nodes at once. However, it is not recommended to clear breakpoints on multiple controller nodes at once. Continue this process until all nodes have completed their update.

The update command reports a COMPLETE status when the update completes: 

...
IN_PROGRESS
IN_PROGRESS
IN_PROGRESS
COMPLETE
update finished with status COMPLETE

If you configured fencing for your Controller nodes, the update process might disable it. When the update process completes, reenable fencing with the following command on one of the Controller nodes: 

$ sudo pcs property set stonith-enabled=true

The update process does not reboot any nodes in the Overcloud automatically. Major and minor version updates to the kernel or Open vSwitch require a reboot, such as when your overcloud operating system updates from Red Hat Enterprise Linux 7.2 to 7.3, or Open vSwitch from version 2.4 to 2.5. Check the /var/log/yum.log file on each node to see if either the kernel or openvswitch packages have updated their major or minor versions. If they have, perform a reboot of each node using the "Rebooting the Overcloud' procedures in the Director Installation and Usage guide.

2.3. Post-Update Notes

2.3.1. Sshd Composable Service

The latest update of Red Hat OpenStack Platform 10 includes the OS::TripleO::Services::Sshd composable service, which is required for live migration capabilities. The director’s core template collection did not include this service in the initial release but is now included in the openstack-tripleo-heat-templates-5.2.0-12 package and later versions.

The default roles data file includes this service and the director installs the service on the overcloud on update.

If using a custom roles data file, include the OS::TripleO::Services::Sshd service on each overcloud role, then update your overcloud stack to include the new service.

For more information, see "Red Hat OpenStack Platform director (TripleO) CVE-2017-2637 bug and Red Hat OpenStack Platform ".

2.3.2. NFV Post-Configuration

If your Overcloud uses Network Functions Virtualization (NFV), follow this procedure to finish the update.

Procedure

You need to migrate your existing OVS-DPDK instances to ensure that the vhost socket mode changes from dkdpvhostuser to dkdpvhostuserclient mode in the OVS ports. We recommend that you snapshot existing instances and rebuild a new instance based on that snapshot image. See Manage Instance Snapshots for complete details on instance snapshots.

To snapshot an instance and boot a new instance from the snapshot:

  1. Find the server ID for the instance you want to take a snapshot of: 

    # openstack server list

  2. Shut down the source instance before you take the snapshot to ensure that all data is flushed to disk: 

    # openstack server stop SERVER_ID

  3. Create the snapshot image of the instance: 

    # openstack image create --id SERVER_ID SNAPSHOT_NAME

  4. Boot a new instance with this snapshot image: 

    # openstack server create --flavor DPDK_FLAVOR --nic net-id=DPDK_NET_ID --image SNAPSHOT_NAME INSTANCE_NAME

  5. Optionally, verify that the new instance status is ACTIVE: 

    # openstack server list

Repeat this procedure for all instances that you need to snapshot and relaunch.