4.2. Upgrading OpenStack by Updating Service Individually, With Live Compute in an HA Environment

This section describes the steps you should follow to upgrade your cloud deployment by updating one service at a time with live compute in a High Availability (HA) environment.
  1. Pre-upgrade tasks:
    On all of your hosts:
    1. If you are running Puppet as configured by Staypuft, you must disable it: 
      # systemctl stop puppet
# systemctl disable puppet
      This ensures that the Staypuft-configured puppet will not revert changes made as part of the upgrade process.
    2. Install the yum repository for Red Hat Enterprise Linux OpenStack Platform 7 (Kilo).
    3. Manually upgrade all the python packages. 
      # yum upgrade python*
    4. Upgrade the openstack-selinux package, if available: 
      # yum upgrade openstack-selinux
      This is necessary to ensure that the upgraded services will run correctly on a system with SELinux enabled.
  2. Service upgrades:
    Upgrade each of your services. The following is a reasonable order in which to perform the upgrades on your controllers:
    Upgrade MariaDB:
    Perform the follow steps on each host running MariaDB. Complete the steps on one host before starting the process on another host.
    1. Stop the service from running on the local node: 
      # pcs resource ban galera-master $(crm_node -n)
    2. Wait until pcs status shows that the service is no longer running on the local node. This may take a few minutes. The local node will first transition to slave mode: 
      Master/Slave Set: galera-master [galera]
Masters: [ pcmk-mac525400aeb753 pcmk-mac525400bab8ae ]
Slaves: [ pcmk-mac5254004bd62f ]
      It will eventually transition to stopped: 
      Master/Slave Set: galera-master [galera]
Masters: [ pcmk-mac525400aeb753 pcmk-mac525400bab8ae ]
Stopped: [ pcmk-mac5254004bd62f ]
    3. Upgrade the relevant packages. 
      # yum upgrade '*mariadb*' '*galera*'
    4. Allow Pacemaker to schedule the galera resource on the local node: 
      # pcs resource clear galera-master
    5. Wait until pcs status shows that the galera resource is running on the local node as a master. The output from pcs status should include something like: 
      Master/Slave Set: galera-master [galera]
Masters: [ pcmk-mac5254004bd62f pcmk-mac525400aeb753
pcmk-mac525400bab8ae ]
    Upgrade MongoDB:
    1. Remove the mongod resource from Pacemaker's control: 
      # pcs resource unmanage mongod-clone
    2. Stop the service on all of your controllers. On each controller, run: 
      # systemctl stop mongod
    3. Upgrade the relevant packages: 
      #  yum upgrade 'mongodb*' 'python-pymongo*'
    4. Reload systemd to account for updated unit files: 
      # systemctl daemon-reload
    5. Restart the mongod service on your controllers by running, on each controller: 
      # systemctl start mongod
    6. Clean up the resource to Pacemaker control: 
      # pcs resource cleanup mongod-clone
    7. Return the resource to Pacemaker control: 
      # pcs resource manage mongod-clone
    8. Wait until the output of pcs status shows that the above resources are running.
    Upgrade Identity service (keystone):
    1. Remove Identity service from Pacemaker's control: 
      # pcs resource unmanage keystone-clone
    2. Stop the Identity service by running the following on each of your controllers: 
      # systemctl stop openstack-keystone
    3. Upgrade the relevant packages: 
      # yum upgrade 'openstack-keystone*' 'python-keystone*'
    4. Reload systemd to account for updated unit files: 
      # systemctl daemon-reload
    5. In the RHEL OpenStack Platform 7 (Kilo) release, the location of the token persistence backends has changed. You need to update the driver option in the [token] section of the keystone.conf. To do this, replace any instance of keystone.token.backends with keystone.token.persistence.backends. 
      # sed -i 's/keystone.token.backends/keystone.token.persistence.backends/g' \
/etc/keystone/keystone.conf
      Package updates may include new systemd unit files, so ensure that systemd is aware of any updated files. 
      # systemctl daemon-reload
    6. Earlier versions of the installer may not have configured your system to automatically purge expired Keystone token, it is possible that your token table has a large number of expired entries. This can dramatically increase the time it takes to complete the database schema upgrade.
      To flush expired tokens from the database and alleviate the problem, the keystone-manage command can be used before running the Identity database upgrade.
      This will flush expired tokens from the database. You can arrange to run this command periodically (e.g., daily) using cron. 
      # keystone-manage token_flush
    7. Update the Identity service database schema: 
      # openstack-db --service keystone --update
    8. Restart the service by running the following on each of your controllers: 
      # systemctl start openstack-keystone
    9. Clean up the Identity service using Pacemaker: 
      # pcs resource cleanup keystone-clone
    10. Return the resource to Pacemaker control: 
      # pcs resource manage keystone-clone
    11. Wait until the output of pcs status shows that the above resources are running.
    Upgrade Image service (glance):
    1. Stop the Image service resources in Pacemaker: 
      # pcs resource disable glance-registry-clone
# pcs resource disable glance-api-clone
    2. Wait until the output of pcs status shows that both services have stopped running.
    3. Upgrade the relevant packages: 
      # yum upgrade 'openstack-glance*' 'python-glance*'
    4. Reload systemd to account for updated unit files: 
      # systemctl daemon-reload
    5. Update the Image service database schema: 
      # openstack-db --service glance --update
    6. Clean up the Image service using Pacemaker: 
      # pcs resource cleanup glance-api-clone
# pcs resource cleanup glance-registry-clone
    7. Restart Image service resources in Pacemaker: 
      # pcs resource enable glance-api-clone
# pcs resource enable glance-registry-clone
    8. Wait until the output of pcs status shows that the above resources are running.
    Upgrade Block Storage service (cinder):
    1. Stop all Block Storage service resources in Pacemaker: 
      # pcs resource disable cinder-api-clone
# pcs resource disable cinder-scheduler-clone
# pcs resource disable cinder-volume
    2. Wait until the output of pcs status shows that the above services have stopped running.
    3. Upgrade the relevant packages: 
      # yum upgrade 'openstack-cinder*' 'python-cinder*'
    4. Reload systemd to account for updated unit files: 
      # systemctl daemon-reload
    5. Update the Block Storage service database schema: 
      # openstack-db --service cinder --update
    6. Clean up the Block Storage service using Pacemaker: 
      # pcs resource cleanup cinder-volume
# pcs resource cleanup cinder-scheduler-clone
# pcs resource cleanup cinder-api-clone
    7. Restart all Block Storage service resources in Pacemaker: 
      # pcs resource enable cinder-volume
# pcs resource enable cinder-scheduler-clone
# pcs resource enable cinder-api-clone
    8. Wait until the output of pcs status shows that the above resources are running.
    Upgrade Orchestration (heat):
    1. Stop Orchestration resources in Pacemaker: 
      # pcs resource disable heat-api-clone
# pcs resource disable heat-api-cfn-clone
# pcs resource disable heat-api-cloudwatch-clone
# pcs resource disable heat
    2. Wait until the output of pcs status shows that the above services have stopped running.
    3. Upgrade the relevant packages: 
      # yum upgrade 'openstack-heat*' 'python-heat*'
    4. Reload systemd to account for updated unit files: 
      # systemctl daemon-reload
    5. Update the Orchestration database schema: 
      # openstack-db --service heat --update
    6. Clean up the Orchestration service using Pacemaker: 
      # pcs resource cleanup heat
# pcs resource cleanup heat-api-cloudwatch-clone
# pcs resource cleanup heat-api-cfn-clone
# pcs resource cleanup heat-api-clone
    7. Restart Orchestration resources in Pacemaker: 
      # pcs resource enable heat
# pcs resource enable heat-api-cloudwatch-clone
# pcs resource enable heat-api-cfn-clone
# pcs resource enable heat-api-clone
    8. Wait until the output of pcs status shows that the above resources are running.
    Upgrade Telemetry (ceilometer):
    1. Stop all Telemetry resources in Pacemaker: 
      # pcs resource disable openstack-ceilometer-central
# pcs resource disable openstack-ceilometer-api-clone
# pcs resource disable openstack-ceilometer-alarm-evaluator-clone
# pcs resource disable openstack-ceilometer-collector-clone
# pcs resource disable openstack-ceilometer-notification-clone
# pcs resource disable openstack-ceilometer-alarm-notifier-clone
# pcs resource disable ceilometer-delay-clone
    2. Wait until the output of pcs status shows that the above services have stopped running.
    3. Upgrade the relevant packages: 
      # yum upgrade 'openstack-ceilometer*' 'python-ceilometer*'
    4. Reload systemd to account for updated unit files: 
      # systemctl daemon-reload
    5. If you are using the MySQL backend for Telemetry, update the Telemetry database schema. 
      # openstack-db --service ceilometer --update

      Note

      This step is not necessary of you are using the MongoDB backend.
    6. Clean up the Telemetry service using Pacemaker: 
      # pcs resource cleanup ceilometer-delay-clone
# pcs resource cleanup openstack-ceilometer-alarm-notifier-clone
# pcs resource cleanup openstack-ceilometer-notification-clone
# pcs resource cleanup openstack-ceilometer-collector-clone
# pcs resource cleanup openstack-ceilometer-alarm-evaluator-clone
# pcs resource cleanup openstack-ceilometer-api-clone
# pcs resource cleanup openstack-ceilometer-central
    7. Restart all Telemetry resources in Pacemaker: 
      # pcs resource enable ceilometer-delay-clone
# pcs resource enable openstack-ceilometer-alarm-notifier-clone
# pcs resource enable openstack-ceilometer-notification-clone
# pcs resource enable openstack-ceilometer-collector-clone
# pcs resource enable openstack-ceilometer-alarm-evaluator-clone
# pcs resource enable openstack-ceilometer-api-clone
# pcs resource enable openstack-ceilometer-central
    8. Wait until the output of pcs status shows that the above resources are running.
    Upgrade Compute (nova):
    1. Stop all Compute resources in Pacemaker: 
      # pcs resource disable openstack-nova-novncproxy-clone
# pcs resource disable openstack-nova-consoleauth-clone
# pcs resource disable openstack-nova-conductor-clone
# pcs resource disable openstack-nova-api-clone
# pcs resource disable openstack-nova-scheduler-clone
    2. Wait until the output of pcs status shows that the above services have stopped running.
    3. Upgrade the relevant packages: 
      # yum upgrade 'openstack-nova*' 'python-nova*'
    4. Reload systemd to account for updated unit files: 
      # systemctl daemon-reload
    5. Update the Compute database schema: 
      # openstack-db --service nova --update
      After fully upgrading to Kilo (i.e. all nodes are running Kilo), you should start a background migration of flavor information. Kilo conductor nodes will do this on the fly when necessary, but the rest of the idle data needs to be migrated in the the background. Run the following command as a nova user: 
      # runuser -u nova -- nova-manage db migrate_flavor_data
    6. If you are performing a rolling upgrade of your compute hosts you need to set explicit API version limits to ensure compatibility between your Juno and Kilo environments.
      Before starting Kilo controller or compute services, you need to set the compute option in the [upgrade_levels] section of nova.conf to juno: 
      # crudini --set /etc/nova/nova.conf upgrade_levels compute juno
      You will need to first unmanage the Compute resources by running pcs resource unmanage on one of your controllers: 
      # pcs resource unmanage openstack-nova-novncproxy-clone
# pcs resource unmanage openstack-nova-consoleauth-clone
# pcs resource unmanage openstack-nova-conductor-clone
# pcs resource unmanage openstack-nova-api-clone
# pcs resource unmanage openstack-nova-scheduler-clone
      Restart all the services on all controllers: 
      # openstack-service restart nova
      You should return control to the Pacemaker after upgrading all of your compute hosts to OpenStack Kilo. 
      # pcs resource manage openstack-nova-scheduler-clone
# pcs resource manage openstack-nova-api-clone
# pcs resource manage openstack-nova-conductor-clone
# pcs resource manage openstack-nova-consoleauth-clone
# pcs resource manage openstack-nova-novncproxy-clone
    7. Clean up all Compute resources in Pacemaker: 
      # pcs resource cleanup openstack-nova-scheduler-clone
# pcs resource cleanup openstack-nova-api-clone
# pcs resource cleanup openstack-nova-conductor-clone
# pcs resource cleanup openstack-nova-consoleauth-clone
# pcs resource cleanup openstack-nova-novncproxy-clone
    8. Restart all Compute resources in Pacemaker: 
      # pcs resource enable openstack-nova-scheduler-clone
# pcs resource enable openstack-nova-api-clone
# pcs resource enable openstack-nova-conductor-clone
# pcs resource enable openstack-nova-consoleauth-clone
# pcs resource enable openstack-nova-novncproxy-clone
    9. Wait until the output of pcs status shows that the above resources are running.
    Upgrade OpenStack Networking (neutron):
    1. Prevent Pacemaker from triggering the OpenStack Networking cleanup scripts: 
      # pcs resource unmanage neutron-ovs-cleanup-clone
# pcs resource unmanage neutron-netns-cleanup-clone
    2. Stop OpenStack Networking resources in Pacemaker: 
      # pcs resource disable neutron-server-clone
# pcs resource disable neutron-openvswitch-agent-clone
# pcs resource disable neutron-dhcp-agent-clone
# pcs resource disable neutron-l3-agent-clone
# pcs resource disable neutron-metadata-agent-clone
    3. Upgrade the relevant packages 
      # yum upgrade 'openstack-neutron*' 'python-neutron*'
    4. Install packages for the advanced Openstack Networking services enabled in the neutron.conf file, for example, openstack-neutron-vpnaas, openstack-neutron-fwaas and openstack-neutron-lbaas. 
      # yum install openstack-neutron-vpnaas
# yum install openstack-neutron-fwaas
# yum install openstack-neutron-lbaas
      Installing these packages will create the corresponding configuration files.
    5. For the VPNaaS, LBaaS service entries in the neutron.conf file, copy the service_provider entries to the corresponding neutron-*aas.conf file located in /etc/neutron and comment these entries from the neutron.conf file.
      For the FWaaS service entry, the service_provider parameters should remain in the neutron.conf file.
    6. On every node that runs the LBaaS agents, install the openstack-neutron-lbaas package. 
      # yum install openstack-neutron-lbaas
    7. Reload systemd to account for updated unit files: 
      # systemctl daemon-reload
    8. Update the OpenStack Networking database schema: 
      # openstack-db --service neutron --update
    9. Once you have completed upgrading the OpenStack Networking service, you need to edit the rootwrap dhcp.filter configuration file.
      To do so, in the /usr/share/neutron/rootwrap/dhcp.filters file, replace the value of dnsmasq. For example, replace: 
      dnsmasq: EnvFilter, env, root, CONFIG_FILE=, NETWORK_ID=, dnsmasq
      with 
      dnsmasq: CommandFilter, dnsmasq, root
    10. Clean up OpenStack Networking resources in Pacemaker: 
      # pcs resource cleanup neutron-metadata-agent-clone
# pcs resource cleanup neutron-l3-agent-clone
# pcs resource cleanup neutron-dhcp-agent-clone
# pcs resource cleanup neutron-openvswitch-agent-clone
# pcs resource cleanup neutron-server-clone
    11. Restart OpenStack Networking resources in Pacemaker: 
      # pcs resource enable neutron-metadata-agent-clone
# pcs resource enable neutron-l3-agent-clone
# pcs resource enable neutron-dhcp-agent-clone
# pcs resource enable neutron-openvswitch-agent-clone
# pcs resource enable neutron-server-clone
    12. Return the cleanup agents to Pacemaker control: 
      # pcs resource manage neutron-ovs-cleanup-clone
# pcs resource manage neutron-netns-cleanup-clone
    13. Wait until the output of pcs status shows that the above resources are running.
    Upgrade Dashboard (horizon):
    1. Stop the Dashboard resource in Pacemaker: 
      # pcs resource disable horizon-clone
    2. Wait until the output of pcs status shows that the service has stopped running.
    3. Upgrade the relevant packages: 
      # yum upgrade httpd 'openstack-dashboard*' 'python-django*'
    4. Reload systemd to account for updated unit files: 
      # systemctl daemon-reload
    5. Correct the Dashboard configuration:
      Fix Apache Configuration:
      The openstack-dashboard package installs /etc/httpd/conf.d/openstack-dashboard.conf file, but the Staypuft installer replaces this with the /etc/httpd/conf.d/15-horizon_vhost.conf file. After upgrading horizon, you will have the following configuration files:
      • 15-horizon_vhost.conf
      • openstack-dashboard.conf
      • openstack-dashboard.conf.rpmnew
      Ensure you make the following changes:
      • Remove the openstack-dashboard.conf.rpmnew file: 
        # rm openstack-dashboard.conf.rpmnew
      • Modify the 15-horizon_vhost.conf file by replacing: 
        Alias /static "/usr/share/openstack-dashboard/static"
        with 
        Alias /dashboard/static "/usr/share/openstack-dashboard/static"
      Fix Dashboard Configuration:
      The openstack-dashboard package installs the /etc/openstack-dashboard/local_settings file. After an upgrade, you will find the following configuration files:
      • /etc/openstack-dashboard/local_settings
      • /etc/openstack-dashboard/local_settings.rpmnew
      Ensure you make the following changes:
      • Backup your existing local_settings file: 
        # cp local_settings local_settings.old
      • Rename the local_settings.rpmnew file to local_settings file: 
        # mv local_settings.rpmnew local_settings
      • Replace the following configuration options with the corresponding value from your local_settings.old file:
        • ALLOWED_HOSTS
        • SECRET_KEY
        • CACHES
        • OPENSTACK_KEYSTONE_URL
      • Restart the web server on all your controllers to apply all changes: 
        # service httpd restart
    6. Clean up the Dashboard resource in Pacemaker: 
      #  pcs resource cleanup horizon-clone
    7. Restart the Dashboard resource in Pacemaker: 
      #  pcs resource enable horizon-clone
    8. Wait until the output of pcs status shows that the above resource is running.
    Upgrade Compute hosts (nova):
    On each compute host:
    1. Stop all OpenStack services on the host: 
      # openstack-service stop
    2. Upgrade all packages: 
      # yum upgrade
    3. If you are performing a rolling upgrade of your compute hosts you need to set explicit API version limits to ensure compatibility between your Juno and Kilo environments.
      Before starting Kilo controller or compute services, you need to set the compute option in the [upgrade_levels] section of nova.conf to juno: 
      # crudini --set /etc/nova/nova.conf upgrade_levels compute juno
      You need to make this change on your controllers and on your compute hosts.
    4. Start all openstack services on the host: 
      # openstack-service start
    5. After you have upgraded all of your hosts to Kilo, you will want to remove the API limits configured in the previous step. On all of your hosts: 
      # crudini --del /etc/nova/nova.conf upgrade_levels compute
    Post-upgrade tasks:
    1. After completing all of your individual service upgrades, you should perform a complete package upgrade on all of your systems: 
      # yum upgrade
      This will ensure that all packages are up-to-date. You may want to schedule a restart of your OpenStack hosts at a future date in order to ensure that all running processes are using updated versions of the underlying binaries.
    2. Review the resulting configuration files. The upgraded packages will have installed .rpmnew files appropriate to the Red Hat Enterprise Linux OpenStack Platform 7 version of the service.
      New versions of OpenStack services may deprecate certain configuration options. You should also review your OpenStack logs for any deprecation warnings, because these may cause problems during a future upgrade. For more information on the new, updated and deprecated configuration options for each service , see Configuration Reference available from: Red Hat Enterprise Linux OpenStack Platform Documentation Suite.