Procedure 3.1. Upgrading OpenStack components on a host

1. Install the yum repository for Red Hat Enterprise Linux OpenStack Platform 7 (Kilo).
2. Check that the openstack-utils package is installed:

   # yum install openstack-utils

3. Take down all OpenStack services on all the nodes. This step depends on how your services are distributed among your nodes.
   • In a non-HA environment:
     To stop all the OpenStack services running on a node, login to the node and run:

     # openstack-service stop

   • In an HA environment:
     1. To stop all the OpenStack services running on a node, login to the node and run:

        # openstack-service stop

     2. Disable all Pacemaker-managed resources by setting the stop-all-resources property on the cluster. Run the following on a single member of your Pacemaker cluster:

        # pcs property set stop-all-resources=true

        Then wait until the output of pcs status shows that all resources have stopped.
4. Perform a complete upgrade of all packages, and then flush expired tokens in the Identity service (might decrease the time required to synchronize the database):

   # yum upgrade

5. Perform the necessary configuration updates on each of your services.
   1. Identity service
      In the RHEL OpenStack Platform 7 (Kilo) release, the location of the token persistence backends has changed. You need to update the driver option in the [token] section of the keystone.conf. To do this, replace any instance of keystone.token.backends with keystone.token.persistence.backends.

      # sed -i 's/keystone.token.backends/keystone.token.persistence.backends/g' \
      /etc/keystone/keystone.conf

      Package updates may include new systemd unit files, so confirm that systemd is aware of any updated files.

      # systemctl daemon-reload

   2. OpenStack Networking service
      Once you have completed upgrading the OpenStack Networking service, you need to edit the rootwrap dhcp.filter configuration file.
      To do so, in the /usr/share/neutron/rootwrap/dhcp.filters file, replace the value of dnsmasq. For example, replace:

      dnsmasq: EnvFilter, env, root, CONFIG_FILE=, NETWORK_ID=, dnsmasq

      with:

      dnsmasq: CommandFilter, dnsmasq, root

6. Upgrade the database schema for each service that uses the database. To do so, login to the node hosting the service and run:

   # openstack-db --service SERVICENAME --update

   Use the service's project name as the SERVICENAME. For example, to upgrade the database schema of the Identity service:

   # openstack-db --service keystone --update

   Table 3.1.  Project name of each OpenStack service that uses the database

   Service	Project name
   Identity	keystone
   Block Storage	cinder
   Image Service	glance
   Compute	nova
   Networking	neutron
   Orchestration	heat

   Certain services require additional database maintenance as part of the Juno to Kilo upgrade that is not covered by the openstack-db command:
   1. Identity service
      Earlier versions of the installer may not have configured your system to automatically purge expired Keystone tokens. It is possible that your token table has a large number of expired entries. This can dramatically increase the time it takes to complete the database schema upgrade.
      You can alleviate this problem by running the following command before beginning the Keystone database upgrade process:

      # keystone-manage token_flush

      This will flush expired tokens from the database. You should arrange to run this command periodically (for example, daily) using cron.
   2. Compute
      After fully upgrading to Kilo (that is, all nodes running Kilo), you should start a background migration of flavor information. Kilo conductor nodes will do this on the fly when necessary, but the rest of the idle data needs to be migrated in the the background. Run the following command as a nova user:

      # runuser -u nova -- nova-manage db migrate_flavor_data

7. Review the resulting configuration files. The upgraded packages will have installed .rpmnew files appropriate to the Red Hat Enterprise Linux OpenStack Platform 7 version of the service.
   New versions of OpenStack services may deprecate certain configuration options. You should also review your OpenStack logs for any deprecation warnings, because these may cause problems during a future upgrade. For more information on the new, updated and deprecated configuration options for each service , see Configuration Reference available from: Red Hat Enterprise Linux OpenStack Platform Documentation Suite.
8. If the package upgrades you performed require a reboot (for example, if a new kernel was installed as part of the upgrade), reboot the affected hosts now while the OpenStack service is still disabled.
   • In a non-HA environment:
     To restart the OpenStack service, run the following command on each node:

     # openstack-service start

   • In a HA environment:
     1. Allow Pacemaker to restart your resources by resetting the stop-all-resources property. On a single member of your Pacemaker cluster, run:

        # pcs property set stop-all-resources=false

        Then wait until the output of pcs status shows that all resources have started (this may take several minutes).
     2. Restart OpenStack services on the compute nodes. On each compute node, run:

        # openstack-service start

1. Pre-upgrade tasks:
   On all of your hosts:
   1. Install the yum repository for Red Hat Enterprise Linux OpenStack Platform 7 (Kilo).
   2. Upgrade the openstack-selinux package, if available:

      # yum upgrade openstack-selinux

      This is necessary to ensure that the upgraded services will run correctly on a system with SELinux enabled.
2. Upgrade each of your services:
   The following steps provide specific instructions for each service, and the order in which they should be upgraded.
   1. Identity (keystone)
      Earlier versions of the installer may not have configured your system to automatically purge expired Keystone token, it is possible that your token table has a large number of expired entries. This can dramatically increase the time it takes to complete the database schema upgrade.
      To flush expired tokens from the database and alleviate the problem, the keystone-manage command can be used before running the Identity database upgrade.
      This will flush expired tokens from the database. You can arrange to run this command periodically (e.g., daily) using cron.
      On your Identity host, run:

      # openstack-service stop keystone
      # yum -d1 -y upgrade \*keystone\*
      # keystone-manage token_flush
      # openstack-db --service keystone --update
      # openstack-service start keystone

   2. Object Storage (swift)
      On your Object Storage hosts, run:

      # openstack-service stop swift
      # yum -d1 -y upgrade \*swift\*
      # openstack-service start swift

   3. Image Service (glance)
      On your Image Service host, run:

      # openstack-service stop glance
      # yum -d1 -y upgrade \*glance\*
      # openstack-db --service glance --update
      # openstack-service start glance

   4. Block Storage (cinder)
      On your Block Storage host, run:

      # openstack-service stop cinder
      # yum -d1 -y upgrade \*cinder\*
      # openstack-db --service cinder --update
      # openstack-service start cinder

   5. Orchestration (heat)
      On your Orchestration host, run:

      # openstack-service stop heat
      # yum -d1 -y upgrade \*heat\*
      # openstack-db --service heat --update
      # openstack-service start heat

   6. Telemetry (ceilometer)
      1. On all nodes hosting Telemetry component services, run:

         # openstack-service stop ceilometer
         # yum -d1 -y upgrade \*ceilometer\*

      2. On the controller node, where database is installed, run:

         # ceilometer-dbsync

         This command allows you to configure MySQL as a back-end for Telemetry service.
         For a list of Telemetry component services, refer to Launch the Telemetry API and Agents.
      3. After completing the package upgrade, restart the Telemetry service by running the following command on all nodes hosting Telemetry component services:

         # openstack-service start ceilometer

   7. Compute (nova)
      1. If you are performing a rolling upgrade of your compute hosts you need to set explicit API version limits to ensure compatibility between your Juno and Kilo environments.
         Before starting Kilo controller or compute services, you need to set the compute option in the [upgrade_levels] section of nova.conf to juno:

         # crudini --set /etc/nova/nova.conf upgrade_levels compute juno

         You need to make this change on your controllers and on your compute hosts.
         You should undo this operation after upgrading all of your compute hosts to OpenStack Kilo.
      2. On your Compute host, run:

         # openstack-service stop nova
         # yum -d1 -y upgrade \*nova\*
         # openstack-db --service nova --update

      3. After fully upgrading to Kilo (i.e. all nodes are running Kilo), you should start a background migration of flavor information. Kilo conductor nodes will do this on the fly when necessary, but the rest of the idle data needs to be migrated in the the background. Run the following command as a nova user:

         # runuser -u nova -- nova-manage db migrate_flavor_data

      4. After you have upgraded all of your hosts to Kilo, you will want to remove the API limits configured in the previous step. On all of your hosts:

         # crudini --del /etc/nova/nova.conf upgrade_levels compute

      5. Restart the Compute service on all the compute hosts and controllers:

         # openstack-service start nova

   8. OpenStack Networking (neutron)
      1. On your OpenStack Networking host, run:

         # openstack-service stop neutron
         # yum -d1 -y upgrade \*neutron\*
         # openstack-db --service neutron --update

      2. Once you have completed upgrading the OpenStack Networking service, you need to edit the rootwrap dhcp.filter configuration file.
         To do so, in the /usr/share/neutron/rootwrap/dhcp.filters file, replace the value of dnsmasq. For example, replace:

         dnsmasq: EnvFilter, env, root, CONFIG_FILE=, NETWORK_ID=, dnsmasq

         with:

         dnsmasq: CommandFilter, dnsmasq, root

      3. Restart the OpenStack Networking service:

         # openstack-service start neutron

   9. Dashboard (horizon)
      On your Dashboard host, run:

      # yum -y upgrade \*horizon\* \*openstack-dashboard\*
      # yum -d1 -y upgrade \*horizon\* \*python-django\*
      # systemctl restart httpd

3. Post-upgrade tasks:
   1. After completing all of your individual service upgrades, you should perform a complete package upgrade on all of your systems:

      # yum upgrade

      This will ensure that all packages are up-to-date. You may want to schedule a restart of your OpenStack hosts at a future date in order to ensure that all running processes are using updated versions of the underlying binaries.
   2. Review the resulting configuration files. The upgraded packages will have installed .rpmnew files appropriate to the Red Hat Enterprise Linux OpenStack Platform 7 version of the service.
      New versions of OpenStack services may deprecate certain configuration options. You should also review your OpenStack logs for any deprecation warnings, because these may cause problems during a future upgrade. For more information on the new, updated and deprecated configuration options for each service , see Configuration Reference available from: Red Hat Enterprise Linux OpenStack Platform Documentation Suite.

1. Pre-upgrade tasks:
   On all of your hosts:
   1. If you are running Puppet as configured by Staypuft, you must disable it:

      # systemctl stop puppet
      # systemctl disable puppet

      This ensures that the Staypuft-configured puppet will not revert changes made as part of the upgrade process.
   2. Install the yum repository for Red Hat Enterprise Linux OpenStack Platform 7 (Kilo).
   3. Manually upgrade all the python packages.

      # yum upgrade python*

   4. Upgrade the openstack-selinux package, if available:

      # yum upgrade openstack-selinux

      This is necessary to ensure that the upgraded services will run correctly on a system with SELinux enabled.
2. Service upgrades:
   Upgrade each of your services. The following is a reasonable order in which to perform the upgrades on your controllers:
   Upgrade MariaDB:
   Perform the follow steps on each host running MariaDB. Complete the steps on one host before starting the process on another host.
   1. Stop the service from running on the local node:

      # pcs resource ban galera-master $(crm_node -n)

   2. Wait until pcs status shows that the service is no longer running on the local node. This may take a few minutes. The local node will first transition to slave mode:

      Master/Slave Set: galera-master [galera]
      Masters: [ pcmk-mac525400aeb753 pcmk-mac525400bab8ae ]
      Slaves: [ pcmk-mac5254004bd62f ]

      It will eventually transition to stopped:

      Master/Slave Set: galera-master [galera]
      Masters: [ pcmk-mac525400aeb753 pcmk-mac525400bab8ae ]
      Stopped: [ pcmk-mac5254004bd62f ]

   3. Upgrade the relevant packages.

      # yum upgrade '*mariadb*' '*galera*'

   4. Allow Pacemaker to schedule the galera resource on the local node:

      # pcs resource clear galera-master

   5. Wait until pcs status shows that the galera resource is running on the local node as a master. The output from pcs status should include something like:

      Master/Slave Set: galera-master [galera]
      Masters: [ pcmk-mac5254004bd62f pcmk-mac525400aeb753
      pcmk-mac525400bab8ae ]

   Upgrade MongoDB:
   1. Remove the mongod resource from Pacemaker's control:

      # pcs resource unmanage mongod-clone

   2. Stop the service on all of your controllers. On each controller, run:

      # systemctl stop mongod

   3. Upgrade the relevant packages:

      #  yum upgrade 'mongodb*' 'python-pymongo*'

   4. Reload systemd to account for updated unit files:

      # systemctl daemon-reload

   5. Restart the mongod service on your controllers by running, on each controller:

      # systemctl start mongod

   6. Clean up the resource to Pacemaker control:

      # pcs resource cleanup mongod-clone

   7. Return the resource to Pacemaker control:

      # pcs resource manage mongod-clone

   8. Wait until the output of pcs status shows that the above resources are running.
   Upgrade Identity service (keystone):
   1. Remove Identity service from Pacemaker's control:

      # pcs resource unmanage keystone-clone

   2. Stop the Identity service by running the following on each of your controllers:

      # systemctl stop openstack-keystone

   3. Upgrade the relevant packages:

      # yum upgrade 'openstack-keystone*' 'python-keystone*'

   4. Reload systemd to account for updated unit files:

      # systemctl daemon-reload

   5. In the RHEL OpenStack Platform 7 (Kilo) release, the location of the token persistence backends has changed. You need to update the driver option in the [token] section of the keystone.conf. To do this, replace any instance of keystone.token.backends with keystone.token.persistence.backends.

      # sed -i 's/keystone.token.backends/keystone.token.persistence.backends/g' \
      /etc/keystone/keystone.conf

      Package updates may include new systemd unit files, so ensure that systemd is aware of any updated files.

      # systemctl daemon-reload

   6. Earlier versions of the installer may not have configured your system to automatically purge expired Keystone token, it is possible that your token table has a large number of expired entries. This can dramatically increase the time it takes to complete the database schema upgrade.
      To flush expired tokens from the database and alleviate the problem, the keystone-manage command can be used before running the Identity database upgrade.
      This will flush expired tokens from the database. You can arrange to run this command periodically (e.g., daily) using cron.

      # keystone-manage token_flush

   7. Update the Identity service database schema:

      # openstack-db --service keystone --update

   8. Restart the service by running the following on each of your controllers:

      # systemctl start openstack-keystone

   9. Clean up the Identity service using Pacemaker:

      # pcs resource cleanup keystone-clone

   10. Return the resource to Pacemaker control:

       # pcs resource manage keystone-clone

   11. Wait until the output of pcs status shows that the above resources are running.
   Upgrade Image service (glance):
   1. Stop the Image service resources in Pacemaker:

      # pcs resource disable glance-registry-clone
      # pcs resource disable glance-api-clone

   2. Wait until the output of pcs status shows that both services have stopped running.
   3. Upgrade the relevant packages:

      # yum upgrade 'openstack-glance*' 'python-glance*'

   4. Reload systemd to account for updated unit files:

      # systemctl daemon-reload

   5. Update the Image service database schema:

      # openstack-db --service glance --update

   6. Clean up the Image service using Pacemaker:

      # pcs resource cleanup glance-api-clone
      # pcs resource cleanup glance-registry-clone

   7. Restart Image service resources in Pacemaker:

      # pcs resource enable glance-api-clone
      # pcs resource enable glance-registry-clone

   8. Wait until the output of pcs status shows that the above resources are running.
   Upgrade Block Storage service (cinder):
   1. Stop all Block Storage service resources in Pacemaker:

      # pcs resource disable cinder-api-clone
      # pcs resource disable cinder-scheduler-clone
      # pcs resource disable cinder-volume

   2. Wait until the output of pcs status shows that the above services have stopped running.
   3. Upgrade the relevant packages:

      # yum upgrade 'openstack-cinder*' 'python-cinder*'

   4. Reload systemd to account for updated unit files:

      # systemctl daemon-reload

   5. Update the Block Storage service database schema:

      # openstack-db --service cinder --update

   6. Clean up the Block Storage service using Pacemaker:

      # pcs resource cleanup cinder-volume
      # pcs resource cleanup cinder-scheduler-clone
      # pcs resource cleanup cinder-api-clone

   7. Restart all Block Storage service resources in Pacemaker:

      # pcs resource enable cinder-volume
      # pcs resource enable cinder-scheduler-clone
      # pcs resource enable cinder-api-clone

   8. Wait until the output of pcs status shows that the above resources are running.
   Upgrade Orchestration (heat):
   1. Stop Orchestration resources in Pacemaker:

      # pcs resource disable heat-api-clone
      # pcs resource disable heat-api-cfn-clone
      # pcs resource disable heat-api-cloudwatch-clone
      # pcs resource disable heat

   2. Wait until the output of pcs status shows that the above services have stopped running.
   3. Upgrade the relevant packages:

      # yum upgrade 'openstack-heat*' 'python-heat*'

   4. Reload systemd to account for updated unit files:

      # systemctl daemon-reload

   5. Update the Orchestration database schema:

      # openstack-db --service heat --update

   6. Clean up the Orchestration service using Pacemaker:

      # pcs resource cleanup heat
      # pcs resource cleanup heat-api-cloudwatch-clone
      # pcs resource cleanup heat-api-cfn-clone
      # pcs resource cleanup heat-api-clone

   7. Restart Orchestration resources in Pacemaker:

      # pcs resource enable heat
      # pcs resource enable heat-api-cloudwatch-clone
      # pcs resource enable heat-api-cfn-clone
      # pcs resource enable heat-api-clone

   8. Wait until the output of pcs status shows that the above resources are running.
   Upgrade Telemetry (ceilometer):
   1. Stop all Telemetry resources in Pacemaker:

      # pcs resource disable openstack-ceilometer-central
      # pcs resource disable openstack-ceilometer-api-clone
      # pcs resource disable openstack-ceilometer-alarm-evaluator-clone
      # pcs resource disable openstack-ceilometer-collector-clone
      # pcs resource disable openstack-ceilometer-notification-clone
      # pcs resource disable openstack-ceilometer-alarm-notifier-clone
      # pcs resource disable ceilometer-delay-clone

   2. Wait until the output of pcs status shows that the above services have stopped running.
   3. Upgrade the relevant packages:

      # yum upgrade 'openstack-ceilometer*' 'python-ceilometer*'

   4. Reload systemd to account for updated unit files:

      # systemctl daemon-reload

   5. If you are using the MySQL backend for Telemetry, update the Telemetry database schema.

      # openstack-db --service ceilometer --update

      Note

      This step is not necessary of you are using the MongoDB backend.
   6. Clean up the Telemetry service using Pacemaker:

      # pcs resource cleanup ceilometer-delay-clone
      # pcs resource cleanup openstack-ceilometer-alarm-notifier-clone
      # pcs resource cleanup openstack-ceilometer-notification-clone
      # pcs resource cleanup openstack-ceilometer-collector-clone
      # pcs resource cleanup openstack-ceilometer-alarm-evaluator-clone
      # pcs resource cleanup openstack-ceilometer-api-clone
      # pcs resource cleanup openstack-ceilometer-central

   7. Restart all Telemetry resources in Pacemaker:

      # pcs resource enable ceilometer-delay-clone
      # pcs resource enable openstack-ceilometer-alarm-notifier-clone
      # pcs resource enable openstack-ceilometer-notification-clone
      # pcs resource enable openstack-ceilometer-collector-clone
      # pcs resource enable openstack-ceilometer-alarm-evaluator-clone
      # pcs resource enable openstack-ceilometer-api-clone
      # pcs resource enable openstack-ceilometer-central

   8. Wait until the output of pcs status shows that the above resources are running.
   Upgrade Compute (nova):
   1. Stop all Compute resources in Pacemaker:

      # pcs resource disable openstack-nova-novncproxy-clone
      # pcs resource disable openstack-nova-consoleauth-clone
      # pcs resource disable openstack-nova-conductor-clone
      # pcs resource disable openstack-nova-api-clone
      # pcs resource disable openstack-nova-scheduler-clone

   2. Wait until the output of pcs status shows that the above services have stopped running.
   3. Upgrade the relevant packages:

      # yum upgrade 'openstack-nova*' 'python-nova*'

   4. Reload systemd to account for updated unit files:

      # systemctl daemon-reload

   5. Update the Compute database schema:

      # openstack-db --service nova --update

      After fully upgrading to Kilo (i.e. all nodes are running Kilo), you should start a background migration of flavor information. Kilo conductor nodes will do this on the fly when necessary, but the rest of the idle data needs to be migrated in the the background. Run the following command as a nova user:

      # runuser -u nova -- nova-manage db migrate_flavor_data

   6. If you are performing a rolling upgrade of your compute hosts you need to set explicit API version limits to ensure compatibility between your Juno and Kilo environments.
      Before starting Kilo controller or compute services, you need to set the compute option in the [upgrade_levels] section of nova.conf to juno:

      # crudini --set /etc/nova/nova.conf upgrade_levels compute juno

      You will need to first unmanage the Compute resources by running pcs resource unmanage on one of your controllers:

      # pcs resource unmanage openstack-nova-novncproxy-clone
      # pcs resource unmanage openstack-nova-consoleauth-clone
      # pcs resource unmanage openstack-nova-conductor-clone
      # pcs resource unmanage openstack-nova-api-clone
      # pcs resource unmanage openstack-nova-scheduler-clone

      Restart all the services on all controllers:

      # openstack-service restart nova

      You should return control to the Pacemaker after upgrading all of your compute hosts to OpenStack Kilo.

      # pcs resource manage openstack-nova-scheduler-clone
      # pcs resource manage openstack-nova-api-clone
      # pcs resource manage openstack-nova-conductor-clone
      # pcs resource manage openstack-nova-consoleauth-clone
      # pcs resource manage openstack-nova-novncproxy-clone

   7. Clean up all Compute resources in Pacemaker:

      # pcs resource cleanup openstack-nova-scheduler-clone
      # pcs resource cleanup openstack-nova-api-clone
      # pcs resource cleanup openstack-nova-conductor-clone
      # pcs resource cleanup openstack-nova-consoleauth-clone
      # pcs resource cleanup openstack-nova-novncproxy-clone

   8. Restart all Compute resources in Pacemaker:

      # pcs resource enable openstack-nova-scheduler-clone
      # pcs resource enable openstack-nova-api-clone
      # pcs resource enable openstack-nova-conductor-clone
      # pcs resource enable openstack-nova-consoleauth-clone
      # pcs resource enable openstack-nova-novncproxy-clone

   9. Wait until the output of pcs status shows that the above resources are running.
   Upgrade OpenStack Networking (neutron):
   1. Prevent Pacemaker from triggering the OpenStack Networking cleanup scripts:

      # pcs resource unmanage neutron-ovs-cleanup-clone
      # pcs resource unmanage neutron-netns-cleanup-clone

   2. Stop OpenStack Networking resources in Pacemaker:

      # pcs resource disable neutron-server-clone
      # pcs resource disable neutron-openvswitch-agent-clone
      # pcs resource disable neutron-dhcp-agent-clone
      # pcs resource disable neutron-l3-agent-clone
      # pcs resource disable neutron-metadata-agent-clone

   3. Upgrade the relevant packages

      # yum upgrade 'openstack-neutron*' 'python-neutron*'

   4. Install packages for the advanced Openstack Networking services enabled in the neutron.conf file, for example, openstack-neutron-vpnaas, openstack-neutron-fwaas and openstack-neutron-lbaas.

      # yum install openstack-neutron-vpnaas
      # yum install openstack-neutron-fwaas
      # yum install openstack-neutron-lbaas

      Installing these packages will create the corresponding configuration files.
   5. For the VPNaaS, LBaaS service entries in the neutron.conf file, copy the service_provider entries to the corresponding neutron-*aas.conf file located in /etc/neutron and comment these entries from the neutron.conf file.
      For the FWaaS service entry, the service_provider parameters should remain in the neutron.conf file.
   6. On every node that runs the LBaaS agents, install the openstack-neutron-lbaas package.

      # yum install openstack-neutron-lbaas

   7. Reload systemd to account for updated unit files:

      # systemctl daemon-reload

   8. Update the OpenStack Networking database schema:

      # openstack-db --service neutron --update

   9. Once you have completed upgrading the OpenStack Networking service, you need to edit the rootwrap dhcp.filter configuration file.
      To do so, in the /usr/share/neutron/rootwrap/dhcp.filters file, replace the value of dnsmasq. For example, replace:

      dnsmasq: EnvFilter, env, root, CONFIG_FILE=, NETWORK_ID=, dnsmasq

      with

      dnsmasq: CommandFilter, dnsmasq, root

   10. Clean up OpenStack Networking resources in Pacemaker:

       # pcs resource cleanup neutron-metadata-agent-clone
       # pcs resource cleanup neutron-l3-agent-clone
       # pcs resource cleanup neutron-dhcp-agent-clone
       # pcs resource cleanup neutron-openvswitch-agent-clone
       # pcs resource cleanup neutron-server-clone

   11. Restart OpenStack Networking resources in Pacemaker:

       # pcs resource enable neutron-metadata-agent-clone
       # pcs resource enable neutron-l3-agent-clone
       # pcs resource enable neutron-dhcp-agent-clone
       # pcs resource enable neutron-openvswitch-agent-clone
       # pcs resource enable neutron-server-clone

   12. Return the cleanup agents to Pacemaker control:

       # pcs resource manage neutron-ovs-cleanup-clone
       # pcs resource manage neutron-netns-cleanup-clone

   13. Wait until the output of pcs status shows that the above resources are running.
   Upgrade Dashboard (horizon):
   1. Stop the Dashboard resource in Pacemaker:

      # pcs resource disable horizon-clone

   2. Wait until the output of pcs status shows that the service has stopped running.
   3. Upgrade the relevant packages:

      # yum upgrade httpd 'openstack-dashboard*' 'python-django*'

   4. Reload systemd to account for updated unit files:

      # systemctl daemon-reload

   5. Correct the Dashboard configuration:
      Fix Apache Configuration:
      The openstack-dashboard package installs /etc/httpd/conf.d/openstack-dashboard.conf file, but the Staypuft installer replaces this with the /etc/httpd/conf.d/15-horizon_vhost.conf file. After upgrading horizon, you will have the following configuration files:
      • 15-horizon_vhost.conf
      • openstack-dashboard.conf
      • openstack-dashboard.conf.rpmnew
      Ensure you make the following changes:
      • Remove the openstack-dashboard.conf.rpmnew file:

        # rm openstack-dashboard.conf.rpmnew

      • Modify the 15-horizon_vhost.conf file by replacing:

        Alias /static "/usr/share/openstack-dashboard/static"

        with

        Alias /dashboard/static "/usr/share/openstack-dashboard/static"

      Fix Dashboard Configuration:
      The openstack-dashboard package installs the /etc/openstack-dashboard/local_settings file. After an upgrade, you will find the following configuration files:
      • /etc/openstack-dashboard/local_settings
      • /etc/openstack-dashboard/local_settings.rpmnew
      Ensure you make the following changes:
      • Backup your existing local_settings file:

        # cp local_settings local_settings.old

      • Rename the local_settings.rpmnew file to local_settings file:

        # mv local_settings.rpmnew local_settings

      • Replace the following configuration options with the corresponding value from your local_settings.old file:
        • ALLOWED_HOSTS
        • SECRET_KEY
        • CACHES
        • OPENSTACK_KEYSTONE_URL
      • Restart the web server on all your controllers to apply all changes:

        # service httpd restart

   6. Clean up the Dashboard resource in Pacemaker:

      #  pcs resource cleanup horizon-clone

   7. Restart the Dashboard resource in Pacemaker:

      #  pcs resource enable horizon-clone

   8. Wait until the output of pcs status shows that the above resource is running.
   Upgrade Compute hosts (nova):
   On each compute host:
   1. Stop all OpenStack services on the host:

      # openstack-service stop

   2. Upgrade all packages:

      # yum upgrade

   3. If you are performing a rolling upgrade of your compute hosts you need to set explicit API version limits to ensure compatibility between your Juno and Kilo environments.
      Before starting Kilo controller or compute services, you need to set the compute option in the [upgrade_levels] section of nova.conf to juno:

      # crudini --set /etc/nova/nova.conf upgrade_levels compute juno

      You need to make this change on your controllers and on your compute hosts.
   4. Start all openstack services on the host:

      # openstack-service start

   5. After you have upgraded all of your hosts to Kilo, you will want to remove the API limits configured in the previous step. On all of your hosts:

      # crudini --del /etc/nova/nova.conf upgrade_levels compute

   Post-upgrade tasks:
   1. After completing all of your individual service upgrades, you should perform a complete package upgrade on all of your systems:

      # yum upgrade

      This will ensure that all packages are up-to-date. You may want to schedule a restart of your OpenStack hosts at a future date in order to ensure that all running processes are using updated versions of the underlying binaries.
   2. Review the resulting configuration files. The upgraded packages will have installed .rpmnew files appropriate to the Red Hat Enterprise Linux OpenStack Platform 7 version of the service.
      New versions of OpenStack services may deprecate certain configuration options. You should also review your OpenStack logs for any deprecation warnings, because these may cause problems during a future upgrade. For more information on the new, updated and deprecated configuration options for each service , see Configuration Reference available from: Red Hat Enterprise Linux OpenStack Platform Documentation Suite.