Red Hat Training

A Red Hat training course is available for Red Hat OpenStack Platform

Chapter 5. Non-Director Environments: Upgrading Individual OpenStack Services (Live Compute) in a Standard Environment

This section describes the steps you should follow to upgrade your cloud deployment by updating one service at a time with live compute in a non High Availability (HA) environment. This scenario upgrades from Red Hat OpenStack Platform 9 to Red Hat OpenStack Platform 10 in environments that do not use the director.

A live Compute upgrade minimizes interruptions to your Compute service, with only a few minutes for the smaller services, and a longer migration interval for the workloads moving to newly-upgraded Compute hosts. Existing workloads can run indefinitely, and you do not need to wait for a database migration.

Important

Due to certain package dependencies, upgrading the packages for one OpenStack service might cause Python libraries to upgrade before other OpenStack services upgrade. This might cause certain services to fail prematurely. In this situation, continue upgrading the remaining services. All services should be operational upon completion of this scenario.

Note

This method may require additional hardware resources to bring up the Compute nodes.

Note

The procedures in this chapter follow the architectural naming convention followed by all Red Hat OpenStack Platform documentation. If you are unfamiliar with this convention, refer to Architecture Guide available at Red Hat OpenStack Platform Documentation Suite before proceeding.

5.1. Pre-Upgrade Tasks

On each node, change to the Red Hat OpenStack Platform 10 repository using the subscription-manager command: 

# subscription-manager repos --disable=rhel-7-server-openstack-9-rpms
# subscription-manager repos --enable=rhel-7-server-openstack-10-rpms

Before updating, take a systemd snapshot of the OpenStack Platform services. 

# systemctl snapshot openstack-services

Disable the main OpenStack Platform services: 

sudo systemctl stop 'openstack-*'
sudo systemctl stop 'neutron-*'
sudo systemctl stop 'openvswitch'

Upgrade the openstack-selinux package: 

# yum upgrade openstack-selinux

This is necessary to ensure that the upgraded services will run correctly on a system with SELinux enabled.

5.2. Upgrading WSGI Services

Disable the Identity service and the Dashboard WSGI applets: 

# systemctl stop httpd

Update the packages for both services: 

# yum -d1 -y upgrade \*keystone\*
# yum -y upgrade \*horizon\* \*openstack-dashboard\*
# yum -d1 -y upgrade \*horizon\* \*python-django\*

It is possible that the Identity service’s token table has a large number of expired entries. This can dramatically increase the time it takes to complete the database schema upgrade. To flush expired tokens from the database and alleviate the problem, the keystone-manage command can be used before running the Identity database upgrade. 

# keystone-manage token_flush
# su -s /bin/sh -c "keystone-manage db_sync" keystone

This flushes expired tokens from the database. You can arrange to run this command periodically using cron.

Restart the httpd service. 

# systemctl start httpd

5.3. Upgrading Object Storage (swift)

On your Object Storage hosts, run: 

# systemctl stop '*swift*'
# yum -d1 -y upgrade \*swift\*
# systemctl start openstack-swift-account-auditor \
    openstack-swift-account-reaper \
    openstack-swift-account-replicator \
    openstack-swift-account \
    openstack-swift-container-auditor \
    openstack-swift-container-replicator \
    openstack-swift-container-updater \
    openstack-swift-container \
    openstack-swift-object-auditor \
    openstack-swift-object-replicator \
    openstack-swift-object-updater \
    openstack-swift-object \
    openstack-swift-proxy

5.4. Upgrading Image Service (glance)

On your Image Service host, run: 

# systemctl stop '*glance*'
# yum -d1 -y upgrade \*glance\*
# su -s /bin/sh -c "glance-manage db_sync" glance
# systemctl start openstack-glance-api \
    openstack-glance-registry

5.5. Upgrading Block Storage (cinder)

On your Block Storage host, run: 

# systemctl stop '*cinder*'
# yum -d1 -y upgrade \*cinder\*
# su -s /bin/sh -c "cinder-manage db sync" cinder
# systemctl start openstack-cinder-api \
    openstack-cinder-scheduler \
    openstack-cinder-volume

5.6. Upgrading Orchestration (heat)

On your Orchestration host, run: 

# systemctl stop '*heat*'
# yum -d1 -y upgrade \*heat\*
# su -s /bin/sh -c "heat-manage db_sync" heat
# systemctl start openstack-heat-api-cfn \
    openstack-heat-api-cloudwatch \
    openstack-heat-api \
    openstack-heat-engine

5.7. Upgrading Telemetry (ceilometer)

Note

This component has some additional upgrade procedures detailed in Chapter 7, Additional Procedures for Non-Director Environments. These additional procedures are optional for manual environments but help align with the current OpenStack Platform recommendations.

  1. On all nodes hosting Telemetry component services, run: 

    # systemctl stop '*ceilometer*'
# systemctl stop '*aodh*'
# systemctl stop '*gnocchi*'
# yum -d1 -y upgrade \*ceilometer\* \*aodh\* \*gnocchi\*

  2. On the controller node, where database is installed, run: 

    # ceilometer-dbsync
# aodh-dbsync
# gnocchi-upgrade

  3. After completing the package upgrade, restart the Telemetry service by running the following command on all nodes hosting Telemetry component services: 

    # systemctl start openstack-ceilometer-api \
    openstack-ceilometer-central \
    openstack-ceilometer-collector \
    openstack-ceilometer-notification \
    openstack-aodh-evaluator \
    openstack-aodh-listener \
    openstack-aodh-notifier \
    openstack-gnocchi-metricd \
    openstack-gnocchi-statsd

5.8. Upgrading Compute (nova)

  1. If you are performing a rolling upgrade of your compute hosts you need to set explicit API version limits to ensure compatibility in your environment.

    Before starting Compute services on Controller or Compute nodes, set the compute option in the [upgrade_levels] section of nova.conf to the previous Red Hat OpenStack Platform version (mitaka): 

    # crudini --set /etc/nova/nova.conf upgrade_levels compute mitaka

    You need to make this change on your Controller and Compute nodes.

    You should undo this operation after upgrading all of your Compute nodes.

  2. On your Compute host, run: 

    # systemctl stop '*nova*'
# yum -d1 -y upgrade \*nova\*
# su -s /bin/sh -c "nova-manage api_db sync" nova
# su -s /bin/sh -c "nova-manage db sync" nova

  3. After you have upgraded all of your hosts, you will want to remove the API limits configured in the previous step. On all of your hosts: 

    # crudini --del /etc/nova/nova.conf upgrade_levels compute

  4. Restart the Compute service on all the Controller and Compute nodes: 

    # systemctl start openstack-nova-api \
    openstack-nova-conductor \
    openstack-nova-consoleauth \
    openstack-nova-novncproxy \
    openstack-nova-scheduler

5.9. Upgrading Clustering (sahara)

  1. On all nodes hosting Clustering component services, run: 

    # systemctl stop '*sahara*'
# yum -d1 -y upgrade \*sahara\*

  2. On the controller node, where database is installed, run: 

    # su -s /bin/sh -c "sahara-db-manage upgrade heads" sahara

  3. After completing the package upgrade, restart the Clustering service by running the following command on all nodes hosting Clustering component services: 

    # systemctl start openstack-sahara-api \
    openstack-sahara-engine

5.10. Upgrading OpenStack Networking (neutron)

  1. On your OpenStack Networking host, run: 

    # systemctl stop '*neutron*'
# yum -d1 -y upgrade \*neutron\*

  2. On the same host, update the OpenStack Networking database schema: 

    # su -s /bin/sh -c "neutron-db-manage upgrade heads" neutron

  3. Restart the OpenStack Networking service: 

    # systemctl start neutron-dhcp-agent \
    neutron-l3-agent \
    neutron-metadata-agent \
    neutron-openvswitch-agent \
    neutron-server
Note

Start any additional OpenStack Networking services enabled in your environment.

5.11. Post-Upgrade Tasks

After completing all of your individual service upgrades, you should perform a complete package upgrade on all of your systems: 

# yum upgrade

This will ensure that all packages are up-to-date. You may want to schedule a restart of your OpenStack hosts at a future date in order to ensure that all running processes are using updated versions of the underlying binaries.

Review the resulting configuration files. The upgraded packages will have installed .rpmnew files appropriate to the Red Hat OpenStack Platform 10 version of the service.

New versions of OpenStack services may deprecate certain configuration options. You should also review your OpenStack logs for any deprecation warnings, because these may cause problems during a future upgrade. For more information on the new, updated and deprecated configuration options for each service, see Configuration Reference available from Red Hat OpenStack Platform Documentation Suite.