Keeping Red Hat OpenStack Platform Updated
Performing minor updates of Red Hat OpenStack Platform
Abstract
Chapter 1. Introduction
This document provides a workflow to help keep your Red Hat OpenStack Platform 13 environment updated with the latest packages and containers.
This guide provides an upgrade path through the following versions:
| Old Overcloud Version | New Overcloud Version |
|---|---|
| Red Hat OpenStack Platform 13 | Red Hat OpenStack Platform 13.z |
1.1. High level workflow
The following table provides an outline of the steps required for the upgrade process:
| Step | Description |
|---|---|
| Obtaining new container images | Create a new environment file containing the latest container images for OpenStack Platform 13 services. |
| Updating the undercloud | Update the undercloud to the latest OpenStack Platform 13.z version. |
| Updating the overcloud | Update the overcloud to the latest OpenStack Platform 13.z version. |
| Updating the Ceph Storage nodes | Upgrade all Ceph Storage 3 services. |
| Finalize the upgrade | Run the convergence command to refresh your overcloud stack. |
Chapter 2. Updating your container image source
This chapter provides information on how to update your registry source with new overcloud container images for Red Hat OpenStack Platform.
2.1. Registry Methods
Red Hat OpenStack Platform supports the following registry types:
- Remote Registry
-
The overcloud pulls container images directly from
registry.access.redhat.com. This method is the easiest for generating the initial configuration. However, each overcloud node pulls each image directly from the Red Hat Container Catalog, which can cause network congestion and slower deployment. In addition, all overcloud nodes require internet access to the Red Hat Container Catalog. - Local Registry
-
You create a local registry on the undercloud, synchronize the images from
registry.access.redhat.com, and the overcloud pulls the container images from the undercloud. This method allows you to store a registry internally, which can speed up the deployment and decrease network congestion. However, the undercloud only acts as a basic registry and provides limited life cycle management for container images. - Satellite Server
- Manage the complete application life cycle of your container images and publish them through a Red Hat Satellite 6 server. The overcloud pulls the images from the Satellite server. This method provides an enterprise grade solution to store, manage, and deploy Red Hat OpenStack Platform containers.
Select a method from the list and continue configuring your registry details.
2.2. Container image preparation command usage
This section provides an overview on how to use the openstack overcloud container image prepare command, including conceptual information on the command’s various options.
Generating a Container Image Environment File for the Overcloud
One of the main uses of the openstack overcloud container image prepare command is to create an environment file that contains a list of images the overcloud uses. You include this file with your overcloud deployment commands, such as openstack overcloud deploy. The openstack overcloud container image prepare command uses the following options for this function:
--output-env-file- Defines the resulting environment file name.
The following snippet is an example of this file’s contents:
parameter_defaults: DockerAodhApiImage: registry.access.redhat.com/rhosp13/openstack-aodh-api:latest DockerAodhConfigImage: registry.access.redhat.com/rhosp13/openstack-aodh-api:latest ...
Generating a Container Image List for Import Methods
If you aim to import the OpenStack Platform container images to a different registry source, you can generate a list of images. The syntax of list is primarily used to import container images to the container registry on the undercloud, but you can modify the format of this list to suit other import methods, such as Red Hat Satellite 6.
The openstack overcloud container image prepare command uses the following options for this function:
--output-images-file- Defines the resulting file name for the import list.
The following is an example of this file’s contents:
container_images: - imagename: registry.access.redhat.com/rhosp13/openstack-aodh-api:latest - imagename: registry.access.redhat.com/rhosp13/openstack-aodh-evaluator:latest ...
Setting the Namespace for Container Images
Both the --output-env-file and --output-images-file options require a namespace to generate the resulting image locations. The openstack overcloud container image prepare command uses the following options to set the source location of the container images to pull:
--namespace- Defines the namespace for the container images. This is usually a hostname or IP address with a directory.
--prefix- Defines the prefix to add before the image names.
As a result, the director generates the image names using the following format:
-
[NAMESPACE]/[PREFIX][IMAGE NAME]
Setting Container Image Tags
The openstack overcloud container image prepare command uses the latest tag for each container image by default. However, you can select a specific tag for an image version using one of the following options:
--tag-from-label- Use the value of the specified container image labels to discover the versioned tag for every image.
--tag-
Sets the specific tag for all images. All OpenStack Platform container images use the same tag to provide version synchronicity. When using in combination with
--tag-from-label, the versioned tag is discovered starting from this tag.
2.3. Container images for additional services
The director only prepares container images for core OpenStack Platform Services. Some additional features use services that require additional container images. You enable these services with environment files. The openstack overcloud container image prepare command uses the following option to include environment files and their respective container images:
-e- Include environment files to enable additional container images.
The following table provides a sample list of additional services that use container images and their respective environment file locations within the /usr/share/openstack-tripleo-heat-templates directory.
| Service | Environment File |
|---|---|
| Ceph Storage |
|
| Collectd |
|
| Congress |
|
| Fluentd |
|
| OpenStack Bare Metal (ironic) |
|
| OpenStack Data Processing (sahara) |
|
| OpenStack EC2-API |
|
| OpenStack Key Manager (barbican) |
|
| OpenStack Load Balancing-as-a-Service (octavia) |
|
| OpenStack Shared File System Storage (manila) |
|
| Sensu |
|
The next few sections provide examples of including additional services.
Ceph Storage
If deploying a Red Hat Ceph Storage cluster with your overcloud, you need to include the /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml environment file. This file enables the composable containerized services in your overcloud and the director needs to know these services are enabled to prepare their images.
In addition to this environment file, you also need to define the Ceph Storage container location, which is different from the OpenStack Platform services. Use the --set option to set the following parameters specific to Ceph Storage:
--set ceph_namespace-
Defines the namespace for the Ceph Storage container image. This functions similar to the
--namespaceoption. --set ceph_image-
Defines the name of the Ceph Storage container image. Usually,this is
rhceph-3-rhel7. --set ceph_tag-
Defines the tag to use for the Ceph Storage container image. This functions similar to the
--tagoption. When--tag-from-labelis specified, the versioned tag is discovered starting from this tag.
The following snippet is an example that includes Ceph Storage in your container image files:
$ openstack overcloud container image prepare \
...
-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
--set ceph_namespace=registry.access.redhat.com/rhceph \
--set ceph_image=rhceph-3-rhel7 \
--tag-from-label {version}-{release} \
...OpenStack Bare Metal (ironic)
If deploying OpenStack Bare Metal (ironic) in your overcloud, you need to include the /usr/share/openstack-tripleo-heat-templates/environments/services-docker/ironic.yaml environment file so the director can prepare the images. The following snippet is an example on how to include this environment file:
$ openstack overcloud container image prepare \ ... -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/ironic.yaml \ ...
OpenStack Data Processing (sahara)
If deploying OpenStack Data Processing (sahara) in your overcloud, you need to include the /usr/share/openstack-tripleo-heat-templates/environments/services-docker/sahara.yaml environment file so the director can prepare the images. The following snippet is an example on how to include this environment file:
$ openstack overcloud container image prepare \ ... -e /usr/share/openstack-tripleo-heat-templates/environments/services-docker/sahara.yaml \ ...
2.4. Using the Red Hat registry as a remote registry source
Red Hat hosts the overcloud container images on registry.access.redhat.com. Pulling the images from a remote registry is the simplest method because the registry is already setup and all you require is the URL and namespace of the image you aim to pull. However, during overcloud creation, the overcloud nodes all pull images from the remote repository, which can congest your external connection. If that is a problem, you can either:
- Setup a local registry
- Host the images on Red Hat Satellite 6
Procedure
To pull the images directly from
registry.access.redhat.comin your overcloud deployment, an environment file is required to specify the image parameters. The following command automatically creates this environment file:(undercloud) $ openstack overcloud container image prepare \ --namespace=registry.access.redhat.com/rhosp13 \ --prefix=openstack- \ --tag-from-label {version}-{release} \ --output-env-file=/home/stack/templates/overcloud_images.yaml-
Use the
-eoption to include any environment files for optional services. -
If using Ceph Storage, include the additional parameters to define the Ceph Storage container image location:
--set ceph_namespace,--set ceph_image,--set ceph_tag.
-
Use the
-
This creates an
overcloud_images.yamlenvironment file, which contains image locations, on the undercloud. You include this file with your deployment.
2.5. Using the undercloud as a local registry
You can configure a local registry on the undercloud to store overcloud container images. This method involves the following:
-
The director pulls each image from the
registry.access.redhat.com. - The director creates the overcloud.
- During the overcloud creation, the nodes pull the relevant images from the undercloud.
This keeps network traffic for container images within your internal network, which does not congest your external network connection and can speed the deployment process.
Procedure
Find the address of the local undercloud registry. The address will use the following pattern:
<REGISTRY IP ADDRESS>:8787
Use the IP address of your undercloud, which you previously set with the
local_ipparameter in yourundercloud.conffile. For the commands below, the address is assumed to be192.168.24.1:8787.Create a template to upload the the images to the local registry, and the environment file to refer to those images:
(undercloud) $ openstack overcloud container image prepare \ --namespace=registry.access.redhat.com/rhosp13 \ --push-destination=192.168.24.1:8787 \ --prefix=openstack- \ --tag-from-label {version}-{release} \ --output-env-file=/home/stack/templates/overcloud_images.yaml \ --output-images-file /home/stack/local_registry_images.yaml-
Use the
-eoption to include any environment files for optional services. -
If using Ceph Storage, include the additional parameters to define the Ceph Storage container image location:
--set ceph_namespace,--set ceph_image,--set ceph_tag.
-
Use the
This creates two files:
-
local_registry_images.yaml, which contains container image information from the remote source. Use this file to pull the images from the Red Hat Container Registry (registry.access.redhat.com) to the undercloud. overcloud_images.yaml, which contains the eventual image locations on the undercloud. You include this file with your deployment.Check that both files exist.
-
Pull the container images from
registry.access.redhat.comto the undercloud.(undercloud) $ sudo openstack overcloud container image upload \ --config-file /home/stack/local_registry_images.yaml \ --verbose
Pulling the required images might take some time depending on the speed of your network and your undercloud disk.
NoteThe container images consume approximately 10 GB of disk space.
The registry configuration is ready.
2.6. Using a Satellite server as a registry
Red Hat Satellite 6 offers registry synchronization capabilities. This provides a method to pull multiple images into a Satellite server and manage them as part of an application life cycle. The Satellite also acts as a registry for other container-enabled systems to use. For more details information on managing container images, see "Managing Container Images" in the Red Hat Satellite 6 Content Management Guide.
The examples in this procedure use the hammer command line tool for Red Hat Satellite 6 and an example organization called ACME. Substitute this organization for your own Satellite 6 organization.
Procedure
Create a template to pull images to the local registry:
$ source ~/stackrc (undercloud) $ openstack overcloud container image prepare \ --namespace=rhosp13 \ --prefix=openstack- \ --output-images-file /home/stack/satellite_images \
-
Use the
-eoption to include any environment files for optional services. -
If using Ceph Storage, include the additional parameters to define the Ceph Storage container image location:
--set ceph_namespace,--set ceph_image,--set ceph_tag.
NoteThis version of the
openstack overcloud container image preparecommand targets the registry on theregistry.access.redhat.comto generate an image list. It uses different values than theopenstack overcloud container image preparecommand used in a later step.-
Use the
-
This creates a file called
satellite_imageswith your container image information. You will use this file to synchronize container images to your Satellite 6 server. Remove the YAML-specific information from the
satellite_imagesfile and convert it into a flat file containing only the list of images. The followingsedcommands accomplish this:(undercloud) $ awk -F ':' '{if (NR!=1) {gsub("[[:space:]]", ""); print $2}}' ~/satellite_images > ~/satellite_images_namesThis provides a list of images that you pull into the Satellite server.
-
Copy the
satellite_images_namesfile to a system that contains the Satellite 6hammertool. Alternatively, use the instructions in the Hammer CLI Guide to install thehammertool to the undercloud. Run the following
hammercommand to create a new product (OSP13 Containers) to your Satellite organization:$ hammer product create \ --organization "ACME" \ --name "OSP13 Containers"
This custom product will contain our images.
Add the base container image to the product:
$ hammer repository create \ --organization "ACME" \ --product "OSP13 Containers" \ --content-type docker \ --url https://registry.access.redhat.com \ --docker-upstream-name rhosp13/openstack-base \ --name base
Add the overcloud container images from the
satellite_imagesfile.$ while read IMAGE; do \ IMAGENAME=$(echo $IMAGE | cut -d"/" -f2 | sed "s/openstack-//g" | sed "s/:.*//g") ; \ hammer repository create \ --organization "ACME" \ --product "OSP13 Containers" \ --content-type docker \ --url https://registry.access.redhat.com \ --docker-upstream-name $IMAGE \ --name $IMAGENAME ; done < satellite_images_names
Synchronize the container images:
$ hammer product synchronize \ --organization "ACME" \ --name "OSP13 Containers"
Wait for the Satellite server to complete synchronization.
NoteDepending on your configuration,
hammermight ask for your Satellite server username and password. You can configurehammerto automatically login using a configuration file. See the "Authentication" section in the Hammer CLI Guide.- If your Satellite 6 server uses content views, create a new content view version to incorporate the images.
Check the tags available for the
baseimage:$ hammer docker tag list --repository "base" \ --organization "ACME" \ --product "OSP13 Containers"
This displays tags for the OpenStack Platform container images.
Return to the undercloud and generate an environment file for the images on your Satellite server. The following is an example command for generating the environment file:
(undercloud) $ openstack overcloud container image prepare \ --namespace=satellite6.example.com:5000 \ --prefix=acme-osp13_containers- \ --tag-from-label {version}-{release} \ --output-env-file=/home/stack/templates/overcloud_images.yamlNoteThis version of the
openstack overcloud container image preparecommand targets the Satellite server. It uses different values than theopenstack overcloud container image preparecommand used in a previous step.When running this command, include the following data:
-
--namespace- The URL and port of the registry on the Satellite server. The default registry port on Red Hat Satellite is 5000. For example,--namespace=satellite6.example.com:5000. --prefix=- The prefix is based on a Satellite 6 convention. This differs depending on whether you use content views:-
If you use content views, the structure is
[org]-[environment]-[content view]-[product]-. For example:acme-production-myosp13-osp13_containers-. -
If you do not use content views, the structure is
[org]-[product]-. For example:acme-osp13_containers-.
-
If you use content views, the structure is
-
--tag-from-label {version}-{release}- Identifies the latest tag for each image. -
-e- Include any environment files for optional services. --set ceph_namespace,--set ceph_image,--set ceph_tag- If using Ceph Storage, include the additional parameters to define the Ceph Storage container image location. Note thatceph_imagenow includes a Satellite-specific prefix. This prefix is the same value as the--prefixoption. For example:--set ceph_image=acme-osp13_containers-rhceph-3-rhel7
This ensures the overcloud uses the Ceph container image using the Satellite naming convention.
-
-
This creates an
overcloud_images.yamlenvironment file, which contains the image locations on the Satellite server. You include this file with your deployment.
The registry configuration is ready.
2.7. Next Steps
You now have a new overcloud_images.yaml environment file that contains a list of your container image sources. Include this file with all future upgrade and deployment operations.
You can now prepare the overcloud for the update.
Chapter 3. Upgrading the Undercloud
This process upgrades the undercloud and its overcloud images to Red Hat OpenStack Platform 13.
3.1. Performing a minor update of an undercloud
The director provides commands to update the packages on the undercloud node. This allows you to perform a minor update within the current version of your OpenStack Platform environment.
Procedure
-
Log into the director as the
stackuser. Update the
python-tripleoclientpackage and its dependencies to ensure you have the latest scripts for the minor version update:$ sudo yum update -y python-tripleoclient
The director uses the
openstack undercloud upgradecommand to update the Undercloud environment. Run the command:$ openstack undercloud upgrade
- Wait until the undercloud upgrade process completes.
Reboot the undercloud to update the operating system’s kernel and other system packages:
$ sudo reboot
- Wait until the node boots.
3.2. Updating the overcloud images
You need to replace your current overcloud images with new versions. The new images ensure the director can introspect and provision your nodes using the latest version of OpenStack Platform software.
Prerequisites
- You have updated the undercloud to the latest version.
Procedure
Remove any existing images from the
imagesdirectory on thestackuser’s home (/home/stack/images):$ rm -rf ~/images/*
Extract the archives:
$ cd ~/images $ for i in /usr/share/rhosp-director-images/overcloud-full-latest-13.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-13.0.tar; do tar -xvf $i; done $ cd ~
Import the latest images into the director:
$ openstack overcloud image upload --update-existing --image-path /home/stack/images/
Configure your nodes to use the new images:
$ openstack overcloud node configure $(openstack baremetal node list -c UUID -f value)
Verify the existence of the new images:
$ openstack image list $ ls -l /httpboot
When deploying overcloud nodes, ensure the Overcloud image version corresponds to the respective Heat template version. For example, only use the OpenStack Platform 13 images with the OpenStack Platform 13 Heat templates.
3.3. Next Steps
The undercloud upgrade is complete. You can now prepare the overcloud for the upgrade.
Chapter 4. Updating the Overcloud
This process updates the overcloud.
Prerequisites
- You have updated the undercloud to the latest version.
4.1. Running the overcloud update preparation
The update requires running openstack overcloud update prepare command, which performs the following tasks:
- Updates the overcloud plan to OpenStack Platform 13
- Prepares the nodes for the update
Procedure
Source the
stackrcfile:$ source ~/stackrc
Run the update preparation command:
$ openstack overcloud update prepare \ --templates \ -e /home/stack/templates/overcloud_images.yaml \ -e <ENVIRONMENT FILE>Include the following options relevant to your environment:
-
Custom configuration environment files (
-e) -
The environment file with your new container image locations (
-e). Note that the update command might display a warning about using the--container-registry-file. You can ignore this warning as this option is deprecated in favor of using-efor the container image environment file. -
If applicable, your custom roles (
roles_data) file (--roles-file) -
If applicable, your composable network (
network_data) file (--networks-file)
-
Custom configuration environment files (
- Wait until the update preparation completes.
4.2. Updating all Controller nodes
This process updates all the Controller nodes to the latest OpenStack Platform 13 verson. The process involves running the openstack overcloud update run command and including the --roles Controller option to restrict operations to the Controller nodes only.
Procedure
Source the
stackrcfile:$ source ~/stackrc
Run the update command:
$ openstack overcloud update run --roles Controller
- Wait until the Controller node update completes.
4.3. Updating all Compute nodes
This process updates all Compute nodes to the latest OpenStack Platform 13 version. The process involves running the openstack overcloud update run command and including the --roles Compute option to restrict operations to the Compute nodes only.
Procedure
Source the
stackrcfile:$ source ~/stackrc
Run the update command:
$ openstack overcloud update run --roles Compute
- Wait until the Compute node update completes.
4.4. Updating all Ceph Storage nodes
This process updates the Ceph Storage nodes. The process involves:
-
Running the
openstack overcloud update runcommand and including the--roles CephStorageoption to restrict operations to the Ceph Storage nodes only. -
Running the
openstack overcloud ceph-upgrade runcommand to perform an update to a containerized Red Hat Ceph Storage 3 cluster.
Procedure
Source the
stackrcfile:$ source ~/stackrc
Run the update command:
$ openstack overcloud update run --roles CephStorage
- Wait until the node update completes.
Run the Ceph Storage update command. For example:
$ openstack overcloud ceph-upgrade run \ --templates \ -e <ENVIRONMENT FILE> \ -e /home/stack/templates/overcloud_images.yamlInclude the following options relevant to your environment:
-
Custom configuration environment files (
-e) -
The environment file with your container image locations (
-e). Note that the update command might display a warning about using the--container-registry-file. You can ignore this warning as this option is deprecated in favor of using-efor the container image environment file. -
If applicable, your custom roles (
roles_data) file (--roles-file) -
If applicable, your composable network (
network_data) file (--networks-file)
-
Custom configuration environment files (
- Wait until the Ceph Storage node update completes.
4.5. Finalizing the update
The update requires a final step to update the overcloud stack. This ensures the stack’s resource structure aligns with a regular deployment of OpenStack Platform 13 and allows you to perform standard openstack overcloud deploy functions in the future.
Procedure
Source the
stackrcfile:$ source ~/stackrc
Run the update finalization command:
$ openstack overcloud update converge \ --templates \ -e <ENVIRONMENT FILE>Include the following options relevant to your environment:
-
Custom configuration environment files (
-e). -
If applicable, your custom roles (
roles_data) file (--roles-file) -
If applicable, your composable network (
network_data) file (--networks-file)
-
Custom configuration environment files (
- Wait until the update finalization completes.
