Chapter 3. Upgrade OpenStack All at Once

This chapter discusses the procedure to upgrade your deployment from RHEL OpenStack Platform 6 to RHEL OpenStack Platform 7 by upgrading all the services at once which involves disabling all the OpenStack services at the same time, performing the upgrade, then enabling all the services back up after the upgrade process is complete. Of all methods for upgrading to Red Hat Enterprise Linux OpenStack Platform 7, this one is the simplest.
If you do not need to upgrade the base operating system to the latest version of Red Hat Enterprise Linux 7 before upgrading the packages, no orchestration is required as all the OpenStack services are down.
In a large environment, the upgrade can result in a potentially extensive downtime while you wait for database-schema upgrades to complete. Downtime can be mitigated by proper dry-runs of your upgrade procedure on a test environment as well as scheduling a specific downtime window for the upgrade.

Note

The procedures in this chapter follow the architectural naming convention followed by all Red Hat Enterprise Linux OpenStack Platform documentation. If you are unfamiliar with this convention, refer to Architecture Guide available at: Red Hat Enterprise Linux OpenStack Platform Documentation Suite before proceeding.
Before you begin upgrading your OpenStack deployment, subscribe to the proper channels. For more information, refer Chapter 2, Prerequisites

3.1. Upgrade All OpenStack Services Simultaneously

The following procedure describes the steps you need to follow to upgrade your cloud deployment from Juno to Kilo by updating all the components at once. Some of the steps have options for both the highly available (HA) and non-highly available (non-HA) scenarios. If your cloud is highly available, then you need to use the Pacemaker commands for starting and stopping your cloud environment.
Run the following steps on all of your hosts:

Procedure 3.1. Upgrading OpenStack components on a host

  1. Install the yum repository for Red Hat Enterprise Linux OpenStack Platform 7 (Kilo).
  2. Check that the openstack-utils package is installed: 
    # yum install openstack-utils
  3. Take down all OpenStack services on all the nodes. This step depends on how your services are distributed among your nodes.
    • In a non-HA environment:
      To stop all the OpenStack services running on a node, login to the node and run: 
      # openstack-service stop
    • In an HA environment:
      1. To stop all the OpenStack services running on a node, login to the node and run: 
        # openstack-service stop
      2. Disable all Pacemaker-managed resources by setting the stop-all-resources property on the cluster. Run the following on a single member of your Pacemaker cluster: 
        # pcs property set stop-all-resources=true
        Then wait until the output of pcs status shows that all resources have stopped.
  4. Perform a complete upgrade of all packages, and then flush expired tokens in the Identity service (might decrease the time required to synchronize the database): 
    # yum upgrade
  5. Perform the necessary configuration updates on each of your services.
    1. Identity service
      In the RHEL OpenStack Platform 7 (Kilo) release, the location of the token persistence backends has changed. You need to update the driver option in the [token] section of the keystone.conf. To do this, replace any instance of keystone.token.backends with keystone.token.persistence.backends. 
      # sed -i 's/keystone.token.backends/keystone.token.persistence.backends/g' \
/etc/keystone/keystone.conf
      Package updates may include new systemd unit files, so confirm that systemd is aware of any updated files. 
      # systemctl daemon-reload
    2. OpenStack Networking service
      Once you have completed upgrading the OpenStack Networking service, you need to edit the rootwrap dhcp.filter configuration file.
      To do so, in the /usr/share/neutron/rootwrap/dhcp.filters file, replace the value of dnsmasq. For example, replace: 
      dnsmasq: EnvFilter, env, root, CONFIG_FILE=, NETWORK_ID=, dnsmasq
      with: 
      dnsmasq: CommandFilter, dnsmasq, root
  6. Upgrade the database schema for each service that uses the database. To do so, login to the node hosting the service and run: 
    # openstack-db --service SERVICENAME --update
    Use the service's project name as the SERVICENAME. For example, to upgrade the database schema of the Identity service: 
    # openstack-db --service keystone --update

    Table 3.1.  Project name of each OpenStack service that uses the database

    Service Project name
    Identity keystone
    Block Storage cinder
    Image Service glance
    Compute nova
    Networking neutron
    Orchestration heat
    Certain services require additional database maintenance as part of the Juno to Kilo upgrade that is not covered by the openstack-db command:
    1. Identity service
      Earlier versions of the installer may not have configured your system to automatically purge expired Keystone tokens. It is possible that your token table has a large number of expired entries. This can dramatically increase the time it takes to complete the database schema upgrade.
      You can alleviate this problem by running the following command before beginning the Keystone database upgrade process: 
      # keystone-manage token_flush
      This will flush expired tokens from the database. You should arrange to run this command periodically (for example, daily) using cron.
    2. Compute
      After fully upgrading to Kilo (that is, all nodes running Kilo), you should start a background migration of flavor information. Kilo conductor nodes will do this on the fly when necessary, but the rest of the idle data needs to be migrated in the the background. Run the following command as a nova user: 
      # runuser -u nova -- nova-manage db migrate_flavor_data
  7. Review the resulting configuration files. The upgraded packages will have installed .rpmnew files appropriate to the Red Hat Enterprise Linux OpenStack Platform 7 version of the service.
    New versions of OpenStack services may deprecate certain configuration options. You should also review your OpenStack logs for any deprecation warnings, because these may cause problems during a future upgrade. For more information on the new, updated and deprecated configuration options for each service , see Configuration Reference available from: Red Hat Enterprise Linux OpenStack Platform Documentation Suite.
  8. If the package upgrades you performed require a reboot (for example, if a new kernel was installed as part of the upgrade), reboot the affected hosts now while the OpenStack service is still disabled.
    • In a non-HA environment:
      To restart the OpenStack service, run the following command on each node: 
      # openstack-service start
    • In a HA environment:
      1. Allow Pacemaker to restart your resources by resetting the stop-all-resources property. On a single member of your Pacemaker cluster, run: 
        # pcs property set stop-all-resources=false
        Then wait until the output of pcs status shows that all resources have started (this may take several minutes).
      2. Restart OpenStack services on the compute nodes. On each compute node, run: 
        # openstack-service start