11.3. Updating the Overcloud

This section details the steps requires to update the Overcloud. Make sure to follow each section in order and only apply the sections relevant to your environment.

11.3.1. Configuration Agent

Important

This section is only required if updating from Red Hat Enterprise Linux OpenStack Platform 7.0 or 7.1 to Red Hat Enterprise Linux OpenStack Platform 7.2 and later.
Due to a known issue (see BZ#1278181), the Overcloud requires some manual configuration of its configuration agent. This involves copying a new version of the configuration agent script from the director to each node in the Overcloud.
Log in as the stack user on the director host and source the Undercloud configuration: 
$ source ~/stackrc
Copy the configuration agent (55-heat-config) to each Overcloud node. Use the following command to do this for all hosts: 
$ for i in `nova list|awk '/Running/ {print $(NF-1)}'|awk -F"=" '{print $NF}'`; do echo $i; scp -o StrictHostKeyChecking=no /usr/share/openstack-heat-templates/software-config/elements/heat-config/os-refresh-config/configure.d/55-heat-config heat-admin@${i}: ; ssh -o StrictHostKeyChecking=no heat-admin@${i} 'sudo /bin/bash -c "cp /home/heat-admin/55-heat-config /usr/libexec/os-refresh-config/configure.d/55-heat-config"'; done
This ensures the configuration agent is up-to-date.
This Overcloud also needs to recreate some post-deployment files. The director includes a script to achieve this. Copy and execute the heat-config-rebuild-deployed script on each node. Use the following command to do this for all nodes: 
$ for i in `nova list|awk '/Running/ {print $(NF-1)}'|awk -F"=" '{print $NF}'`; do echo $i; scp  -o StrictHostKeyChecking=no /usr/share/openstack-heat-templates/software-config/elements/heat-config/bin/heat-config-rebuild-deployed heat-admin@${i}: ; ssh -o StrictHostKeyChecking=no heat-admin@${i} 'sudo /bin/bash -c "mkdir -p /usr/share/openstack-heat-templates/software-config/elements/heat-config/bin ; cp heat-config-rebuild-deployed /usr/share/openstack-heat-templates/software-config/elements/heat-config/bin/heat-config-rebuild-deployed ; chmod +x /usr/share/openstack-heat-templates/software-config/elements/heat-config/bin/heat-config-rebuild-deployed ; /usr/share/openstack-heat-templates/software-config/elements/heat-config/bin/heat-config-rebuild-deployed"' ; done

11.3.2. Modified Overcloud Templates

Important

This section is only required if using a modified template collection from Section 10.6, “Using Customized Overcloud Heat Templates”. This is because the copy is a static snapshot of the original Heat template collection from /usr/share/openstack-tripleo-heat-templates/.
To update your modified template collection, you need to:
  1. Backup your existing custom template collection: 
    $ mv ~/templates/my-overcloud/ ~/templates/my-overcloud.bak
  2. Replace the new version of the template collection from /usr/share/openstack-tripleo-heat-templates: 
    # sudo cp -rv /usr/share/openstack-tripleo-heat-templates ~/templates/my-overcloud/
  3. Check for differences between the old and new custom template collection. To see changes between the two, use the following diff command: 
    # diff -Nary ~/templates/my-overcloud.bak/ ~/templates/my-overcloud/
    This helps identify customizations from the old template collection that you can incorporate into the new template collection.
  4. Incorporate customization into the new custom template collection.

Important

Red Hat provides updates to the Heat template collection over subsequent releases. Using a modified template collection can lead to a divergence between your custom copy and the original copy in /usr/share/openstack-tripleo-heat-templates. Red Hat recommends using the methods from the following section instead of modifying the Heat template collection:
If creating a copy of the Heat template collection, you should track changes to the templates using a version control system such as git.

11.3.3. New Environment Parameters

The updated Heat template collection requires some new parameters not included in the original Red Hat Enterprise Linux OpenStack Platform 7.0 release. It is recommended to include these parameters in future updates.
Include the following additional parameters in a custom environment file (~/templates/param-updates.yaml):

Table 11.1. Additional Parameters to Include

New Parameter
Description
ControlPlaneDefaultRoute
The default route of the control plane network.
EC2MetadataIp
The IP address of the EC2 metadata server.
For example: 
parameter_defaults:
  ControlPlaneDefaultRoute: 192.168.1.1
  EC2MetadataIp: 169.254.169.254
Make sure to include this file on any subsequent updates of the Overcloud.

11.3.4. Version Specific Notes

This section contains some notes specific to your initial version of the director and the Overcloud. Use this section as a reference for any inclusions to the Overcloud stack in subsequent updates.

If you started with OpenStack Platform director 7.0 and are upgrading to OpenStack Platform director 7.2 or later:

  • Be sure to provide the same value for the ServiceNetMap parameter that was used on the initial cloud deployment (see Section 6.2.6.3, “Assigning OpenStack Services to Isolated Networks”. If a custom value was used on the initial deployment, provide the same custom value. If you are updating from 7.0 and used no custom ServiceNetMap value on the initial deployment, include the following environment file in the update command to preserve the 7.0 value:
    /usr/share/openstack-tripleo-heat-templates/environments/updates/update-from-keystone-admin-internal-api.yaml
    Make sure to include this file on any subsequent updates of the Overcloud.
    Changing the value of ServiceNetMap after Overcloud creation is not currently supported.
  • If using a single network for the Overcloud (for example, the original deployment did not include network-isolation.yaml) then include the following environment file in the update command:
    /usr/share/openstack-tripleo-heat-templates/environments/updates/update-from-publicvip-on-ctlplane.yaml
    Make sure to include this file on any subsequent updates of the Overcloud. Note that you do not need this file if using an external load balancer.

If you started with OpenStack Platform director 7.1 and are upgrading to OpenStack Platform director 7.2 or later:

  • Be sure to provide the same value for the ServiceNetMap parameter that was used on the initial cloud deployment (see Section 6.2.6.3, “Assigning OpenStack Services to Isolated Networks”. If a custom value was used on the initial deployment, provide the same custom value. If you are updating from 7.1 and used no custom value for ServiceNetMap on the initial deployment, then no additional environment file or value needs to be provided for ServiceNetMap. Changing the value of ServiceNetMap after Overcloud creation is not currently supported.
  • Include the following environment file in the update command to make sure the VIP resources remain mapped to vip.yaml:
    /usr/share/openstack-tripleo-heat-templates/environments/updates/update-from-vip.yaml
    Make sure to include this file on any subsequent updates of the Overcloud. Note that you do not need this file if using an external load balancer.
  • If updating from 7.1 and not using external load balancer, provide the control VIP for the control_virtual_ip input parameter. This is because the resource is replaced during the upgrade. To do so, find the current control_virtual_ip address with: 
    $ neutron port-show control_virtual_ip | grep ip_address
{"subnet_id": "3d7c11e0-53d9-4a54-a9d7-55865fcc1e47", "ip_address": "192.0.2.21"} |
    Add it into a custom environment file, such as ~/templates/param-updates.yaml from Section 11.3.3, “New Environment Parameters”, as follows: 
    parameters:
  ControlFixedIPs: [{'ip_address':'192.0.2.21'}]
    Make sure to include this file on any subsequent updates of the Overcloud. Note that you do not need this file if using an external load balancer.
    Delete the existing Neutron port: 
    $ neutron port-delete control_virtual_ip
    The update process replaces this VIP with a new port using the original IP address.

If upgrading from OpenStack Platform director 7.2 to OpenStack Platform director 7.3 or later:

  • Heat on the Undercloud requires an increase in the RPC response timeout to accomodate a known issue (see BZ#1305947). Edit the /etc/heat/heat.conf and set the following parameter: 
    rpc_response_timeout=600
    Then restart all Heat services: 
    $ systemctl restart openstack-heat-api.service
$ systemctl restart openstack-heat-api-cfn.service
$ systemctl restart openstack-heat-engine.service

11.3.5. Updating the Overcloud Packages

The Overcloud relies on standard RPM methods to update the environment. This involves performing an update on all nodes using the openstack overcloud update from the director.
Use the -e to include environment files relevant to your Overcloud and its upgrade path. The order of the environment files is important as the parameters and resources defined in subsequent environment files take precedence. Use the following list as an example of the environment file order:
  • The overcloud-resource-registry-puppet.yaml file from the Heat template collection. Although this file is included automatically when you run the openstack overcloud deploy command, you must include this file when you run the openstack overcloud update command.
  • Any network isolation files, including the initialization file (environments/network-isolation.yaml) from the Heat template collection and then your custom NIC configuration file. See Section 6.2.6, “Isolating all Networks into VLANs” for more information on network islocation.
  • Any external load balancing environment files.
  • Any storage environment files.
  • Any environment files for Red Hat CDN or Satellite registration.
  • Any version-specific environment files from Section 11.3.4, “Version Specific Notes”.
  • Any other custom environment files.
Running an update on all nodes in parallel might cause problems. For example, an update of a package might involve restarting a service, which can disrupt other nodes. This is why the update process updates each node using a set of breakpoints. This means nodes are updated one by one. When one node completes the package update, the update process moves on to the next node. The update process also requires the -i option, which puts the command in an interactive mode that requires confirmation at each breakpoint. Without the -i option, the update remains paused at the first breakpoint.
The following is an example update command for updating from director 7.1 to 7.2: 
$ openstack overcloud update stack overcloud -i \
  --templates  \
  -e /usr/share/openstack-tripleo-heat-templates/overcloud-resource-registry-puppet.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e /home/stack/templates/network-environment.yaml \
  -e /home/stack/templates/storage-environment.yaml \
  -e /home/stack/templates/rhel-registration/environment-rhel-registration.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/updates/update-from-vip.yaml \
  -e /home/stack/templates/param-updates.yaml
Running this command starts the update process. During this process, the director will periodically prompt you to clear breakpoints. For example: 
not_started: [u'overcloud-controller-0', u'overcloud-controller-1', u'overcloud-controller-2']
on_breakpoint: [u'overcloud-compute-0']
Breakpoint reached, continue?
Press Enter to clear the breakpoint from last node on the on_breakpoint list. This begins the update for that node. You can also type a node name to clear a breakpoint on a specific node, or a regular expression to clear breakpoints on mulitple nodes at once. However, it is not recommended to clear breakpoints on multiple controller nodes at once. Continue this process until all nodes have complete their update.

Important

The update process does not reboot any nodes in the Overcloud automatically. See Chapter 9, Rebooting the Overcloud for steps on how to reboot your environment without instance downtime.

Important

If you configured fencing for your Controller nodes, the update process might disable it. When the update process completes, reenable fencing with the following command on one of the Controller nodes: 
$ sudo pcs property set stonith-enabled=true