Chapter 6. Upgrading director
After you complete the undercloud operating system upgrade, upgrade director. The database from the previous Red Hat OpenStack Platform 13 undercloud remains on host after the operating system upgrade. Install the new Red Hat OpenStack Platform 16.1 packages and configure the new source for Red Hat OpenStack Platform 16.1 container images before you run the
openstack undercloud upgrade command.
6.1. Locking the environment to a Red Hat Enterprise Linux release
Red Hat OpenStack Platform 16.1 is supported on Red Hat Enterprise Linux 8.2. Before you perform the update, you must lock the undercloud repositories to the Red Hat Enterprise Linux 8.2 release to avoid upgrading the operating system to a newer minor release.
Log in to the undercloud as the
Lock the undercloud to a specific verison with the
$ sudo subscription-manager release --set=8.2
6.2. Enabling repositories for the undercloud
Enable the repositories that are required for the undercloud, and update the system packages to the latest versions.
Log in to your undercloud as the
Disable all default repositories, and enable the required Red Hat Enterprise Linux repositories:
[stack@director ~]$ sudo subscription-manager repos --disable=* [stack@director ~]$ sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-eus-rpms --enable=rhel-8-for-x86_64-appstream-eus-rpms --enable=rhel-8-for-x86_64-highavailability-eus-rpms --enable=ansible-2.9-for-rhel-8-x86_64-rpms --enable=openstack-16.1-for-rhel-8-x86_64-rpms --enable=fast-datapath-for-rhel-8-x86_64-rpms
These repositories contain packages that the director installation requires.
container-toolsrepository module to version
[stack@director ~]$ sudo dnf module disable -y container-tools:rhel8 [stack@director ~]$ sudo dnf module enable -y container-tools:2.0
virtrepository module to version
[stack@director ~]$ sudo dnf module disable -y virt:rhel [stack@director ~]$ sudo dnf module enable -y virt:8.2
Synchronize the operating system to ensure that your system packages match the operating system version:
[stack@director ~]$ sudo dnf distro-sync -y [stack@director ~]$ sudo reboot
6.3. Installing director packages
Install packages relevant to Red Hat OpenStack Platform director.
Install the command line tools for director installation and configuration:
[stack@director ~]$ sudo dnf install -y python3-tripleoclient
6.4. Preparing container images
The undercloud installation requires an environment file to determine where to obtain container images and how to store them. Generate and customize this environment file that you can use to prepare your container images.
Log in to your undercloud host as the
Generate the default container image preparation file:
$ openstack tripleo container image prepare default \ --local-push-destination \ --output-env-file containers-prepare-parameter.yaml
This command includes the following additional options:
--local-push-destinationsets the registry on the undercloud as the location for container images. With this option, director pulls the necessary images from the Red Hat Container Catalog and pushes the images to the registry on the undercloud. Director uses the undercloud registry as the container image source. To pull container images directly from the Red Hat Container Catalog, omit this option.
--output-env-filespecifies an environment file that includes include the parameters for preparing your container images. In this example, the name of the file is
You can use the same
containers-prepare-parameter.yamlfile to define a container image source for both the undercloud and the overcloud.
containers-prepare-parameter.yamlto suit your requirements.
6.5. Container image preparation parameters
The default file for preparing your containers (
containers-prepare-parameter.yaml) contains the
ContainerImagePrepare heat parameter. This parameter defines a list of strategies for preparing a set of images:
parameter_defaults: ContainerImagePrepare: - (strategy one) - (strategy two) - (strategy three) ...
Each strategy accepts a set of sub-parameters that defines which images to use and what to do with the images. The following table contains information about the sub-parameters you can use with each
| || |
List of image name substrings to exclude from a strategy.
| || |
List of image name substrings to include in a strategy. At least one image name must match an existing image. All
| || |
String to append to the tag for the destination image. For example, if you pull an image with the tag
| || |
A dictionary of image labels that filter the images that you want to modify. If an image matches the labels defined, the director includes the image in the modification process.
| || |
String of ansible role names to run during upload but before pushing the image to the destination registry.
| || |
Dictionary of variables to pass to
| || |
Defines the namespace of the registry that you want to push images to during the upload process.
Do not set this parameter to
| || |
The source registry from where to pull the original container images.
| || |
A dictionary of
| || |
Defines the label pattern to tag the resulting images. Usually sets to
When you push images to the undercloud, use
push_destination: true instead of
push_destination: UNDERCLOUD_IP:PORT. The
push_destination: true method provides a level of consistency across both IPv4 and IPv6 addresses.
set parameter accepts a set of
key: value definitions:
| || |
The name of the Ceph Storage container image.
| || |
The namespace of the Ceph Storage container image.
| || |
The tag of the Ceph Storage container image.
| || |
A prefix for each OpenStack service image.
| || |
A suffix for each OpenStack service image.
| || |
The namespace for each OpenStack service image.
| || |
The driver to use to determine which OpenStack Networking (neutron) container to use. Use a null value to set to the standard
| || |
The tag that the director uses to identify the images to pull from the source registry. You usually keep this key set to the default value, which is the Red Hat OpenStack Platform version number.
The container images use multi-stream tags based on Red Hat OpenStack Platform version. This means there is no longer a
ContainerImageRegistryCredentials parameter maps a container registry to a username and password to authenticate to that registry.
If a container registry requires a username and password, you can use
ContainerImageRegistryCredentials to include credentials with the following syntax:
ContainerImagePrepare: - push_destination: true set: namespace: registry.redhat.io/... ... ContainerImageRegistryCredentials: registry.redhat.io: my_username: my_password
In the example, replace
my_password with your authentication credentials. Instead of using your individual user credentials, Red Hat recommends creating a registry service account and using those credentials to access
registry.redhat.io content. For more information, see "Red Hat Container Registry Authentication".
ContainerImageRegistryLogin parameter is used to control the registry login on the systems being deployed. This must be set to
push_destination is set to false or not used.
ContainerImagePrepare: - set: namespace: registry.redhat.io/... ... ContainerImageRegistryCredentials: registry.redhat.io: my_username: my_password ContainerImageRegistryLogin: true
If you have configured
push_destination, do not set
true. If you set this option to
true and the overcloud nodes do not have network connectivity to the registry hosts defined in
ContainerImageRegistryCredentials, the deployment might fail when trying to perform a login.
6.6. Obtaining container images from private registries
Some container image registries require authentication to access images. In this situation, use the
ContainerImageRegistryCredentials parameter in your
containers-prepare-parameter.yaml environment file.
parameter_defaults: ContainerImagePrepare: - (strategy one) - (strategy two) - (strategy three) ContainerImageRegistryCredentials: registry.example.com: username: "p@55w0rd!"
Private registries require
push_destination set to
true for their respective strategy in the
ContainerImageRegistryCredentials parameter uses a set of keys based on the private registry URL. Each private registry URL uses its own key and value pair to define the username (key) and password (value). This provides a method to specify credentials for multiple private registries.
parameter_defaults: ... ContainerImageRegistryCredentials: registry.redhat.io: myuser: 'p@55w0rd!' registry.internalsite.com: myuser2: '0th3rp@55w0rd!' '192.0.2.1:8787': myuser3: '@n0th3rp@55w0rd!'
ContainerImagePrepare parameter pulls container images from
registry.redhat.io, which requires authentication.
ContainerImageRegistryLogin parameter is used to control whether the system needs to log in to the remote registry to fetch the containers.
parameter_defaults: ... ContainerImageRegistryLogin: true
You must set
push_destination is not configured for a given strategy. If
push_destination is configured in a
ContainerImagePrepare strategy and the
ContainerImageRegistryCredentials parameter is configured, the system logs in to fetch the containers and pushes them to the remote system. If the overcloud nodes do not have network connectivity to the registry hosts defined in the
6.7. Obtaining transitional containers for upgrades
The upgrade requires containers from previous versions of Red Hat OpenStack Platform and Red Hat Ceph Storage. These containers help transition to Red Hat OpenStack Platform 16.1.
Log in to your undercloud host as the
Add the following parameters to the
parameter_defaults: ContainerImagePrepare: - push_destination: true set: ... name_prefix_stein: openstack- name_suffix_stein: '' namespace_stein: registry.redhat.io/rhosp15-rhel8 tag_stein: 15.0 ceph3_namespace: registry.redhat.io/rhceph ceph3_tag: latest ceph3_image: rhceph-3-rhel7 ...
*_steinparameters define the container images for Red Hat OpenStack Platform 15, which the upgrade process uses for database migration.
ceph3_*parameters define the current Red Hat Ceph Storage container images that the overcloud uses. The overcloud requires these containers until the transition from Red Hat Ceph Storage 3 to 4.
- If you use Red Hat Satellite for container image storage, set the namespaces to the image locations on the Red Hat Satellite server.
parameter_defaults: ContainerImagePrepare: - push_destination: true set: ... neutron_driver: openvswitch ...
The upgrade retains Open vSwitch compatibility throughout the process. After you complete the upgrade to Red Hat OpenStack Platform 16.1, you migrate the overcloud from Open vSwitch to Open Virtual Network (OVN).
6.8. Updating the undercloud.conf file
You can continue using the original
undercloud.conf file from your Red Hat OpenStack Platform 13 environment, but you must modify the file to retain compatibility with Red Hat OpenStack Platform 16.1.
Log in to your undercloud host as the
Add the following parameter to the
DEFAULTsection in the file:
container_images_file = /home/stack/containers-prepare-parameter.yaml
This parameter defines the location of the
containers-prepare-parameter.yamlenvironment file so that director pulls container images for the undercloud from the correct location.
generate_service_certificateparameter. The default for this parameter changes from
true, which enables SSL/TLS on your undercloud, during the upgrade.
local_interfaceparameter if you have migrated to a predictable NIC naming convention.
If you set the
masquerade_networkparameter in Red Hat OpenStack Platform 13, remove this parameter and set
masquerade = truefor each subnet.
- Check all other parameters in the file for any changes.
- Save the file.
6.9. Director configuration parameters
The following list contains information about parameters for configuring the
undercloud.conf file. Keep all parameters within their relevant sections to avoid errors.
The following parameters are defined in the
[DEFAULT] section of the
A list of additional (kernel) architectures that an overcloud supports. Currently the overcloud supports
When you enable support for ppc64le, you must also set
certmongernickname of the CA that signs the requested certificate. Use this option only if you have set the
generate_service_certificateparameter. If you select the
localCA, certmonger extracts the local CA certificate to
/etc/pki/ca-trust/source/anchors/cm-local-ca.pemand adds the certificate to the trust chain.
- Defines whether to wipe the hard drive between deployments and after introspection.
Cleanup temporary files. Set this to
Falseto leave the temporary files used during deployment in place after you run the deployment command. This is useful for debugging the generated files or if errors occur.
The CLI tool for container management. Leave this parameter set to
podman. Red Hat Enterprise Linux 8.2 only supports
Disables containerized service health checks. Red Hat recommends that you enable health checks and leave this option set to
Heat environment file with container image information. This file can contain the following entries:
- Parameters for all required container images
ContainerImagePrepareparameter to drive the required image preparation. Usually the file that contains this parameter is named
A list of insecure registries for
podmanto use. Use this parameter if you want to pull images from another source, such as a private container registry. In most cases,
podmanhas the certificates to pull container images from either the Red Hat Container Catalog or from your Satellite server if the undercloud is registered to Satellite.
- Additional environment files that you want to add to the undercloud installation.
The user who installs the undercloud. Leave this parameter unset to use the current default user
Sets the default driver for automatically enrolled nodes. Requires the
enable_node_discoveryparameter to be enabled and you must include the driver in the
- enable_ironic; enable_ironic_inspector; enable_mistral; enable_nova; enable_tempest; enable_validations; enable_zaqar
Defines the core services that you want to enable for director. Leave these parameters set to
Automatically enroll any unknown node that PXE-boots the introspection ramdisk. New nodes use the
fake_pxedriver as a default but you can set
discovery_default_driverto override. You can also use introspection rules to specify driver information for newly enrolled nodes.
Defines whether to install the
novajoinmetadata service in the undercloud.
- Defines whether to enable support for routed control plane networks.
- Defines whether to enable Swift encryption at-rest.
Defines whether to install OpenStack Telemetry services (gnocchi, aodh, panko) in the undercloud. Set the
trueif you want to install and configure telemetry services automatically. The default value is
false, which disables telemetry on the undercloud. This parameter is required if you use other products that consume metrics data, such as Red Hat CloudForms.
- A list of hardware types that you want to enable for the undercloud.
Defines whether to generate an SSL/TLS certificate during the undercloud installation, which is used for the
undercloud_service_certificateparameter. The undercloud installation saves the resulting certificate
/etc/pki/tls/certs/undercloud-[undercloud_public_vip].pem. The CA defined in the
certificate_generation_caparameter signs this certificate.
- URL for the heat container image to use. Leave unset.
Run host-based undercloud configuration using
heat-all. Leave as
hieradataoverride file that configures Puppet hieradata on the director, providing custom configuration to services beyond the
undercloud.confparameters. If set, the undercloud installation copies this file to the
/etc/puppet/hieradatadirectory and sets it as the first file in the hierarchy. For more information about using this feature, see Configuring hieradata on the undercloud.
Defines whether to enable extra hardware collection during the inspection process. This parameter requires the
python-hardware-detectpackages on the introspection image.
The bridge that director uses for node introspection. This is a custom bridge that the director configuration creates. The
LOCAL_INTERFACEattaches to this bridge. Leave this as the default
Runs a set of benchmarks during node introspection. Set this parameter to
trueto enable the benchmarks. This option is necessary if you intend to perform benchmark analysis when inspecting the hardware of registered nodes.
Defines the one-time password to register the undercloud node to an IPA server. This is required when
IPv6 address configuration mode for the undercloud provisioning network. The following list contains the possible values for this parameter:
- dhcpv6-stateless - Address configuration using router advertisement (RA) and optional information using DHCPv6.
- dhcpv6-stateful - Address configuration and optional information using DHCPv6.
Defines whether to use iPXE or standard PXE. The default is
true, which enables iPXE. Set this parameter to
falseto use standard PXE.
The chosen interface for the director Provisioning NIC. This is also the device that director uses for DHCP and PXE boot services. Change this value to your chosen device. To see which device is connected, use the
ip addrcommand. For example, this is the result of an
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:00:75:24:09 brd ff:ff:ff:ff:ff:ff inet 192.168.122.178/24 brd 192.168.122.255 scope global dynamic eth0 valid_lft 3462sec preferred_lft 3462sec inet6 fe80::5054:ff:fe75:2409/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noop state DOWN link/ether 42:0b:c2:a5:c1:26 brd ff:ff:ff:ff:ff:ff
In this example, the External NIC uses
eth0and the Provisioning NIC uses
eth1, which is currently not configured. In this case, set the
eth1. The configuration script attaches this interface to a custom bridge defined with the
The IP address defined for the director Provisioning NIC. This is also the IP address that director uses for DHCP and PXE boot services. Leave this value as the default
192.168.24.1/24unless you use a different subnet for the Provisioning network, for example, if this IP address conflicts with an existing IP address or subnet in your environment.
The maximum transmission unit (MTU) that you want to use for the
local_interface. Do not exceed 1500 for the undercloud.
The local subnet that you want to use for PXE boot and DHCP interfaces. The
local_ipaddress should reside in this subnet. The default is
Path to network configuration override template. If you set this parameter, the undercloud uses a JSON format template to configure the networking with
os-net-configand ignores the network parameters set in
undercloud.conf. Use this parameter when you want to configure bonding or add an option to the interface. See
/usr/share/instack-undercloud/templates/net-config.json.templatefor an example.
Networks file to override for
- Directory to output state, processed heat templates, and Ansible deployment files.
The DNS domain name that you want to use when you deploy the overcloud.Note
When you configure the overcloud, you must set the
CloudDomainparameter to a matching value. Set this parameter in an environment file when you configure your overcloud.
- The roles file that you want to use to override the default roles file for undercloud installation. It is highly recommended to leave this parameter unset so that the director installation uses the default roles file.
- The maximum number of times that the scheduler attempts to deploy an instance. This value must be greater or equal to the number of bare metal nodes that you expect to deploy at once to avoid potential race conditions when scheduling.
- The Kerberos principal for the service using the certificate. Use this parameter only if your CA requires a Kerberos principal, such as in FreeIPA.
List of routed network subnets for provisioning and introspection. The default value includes only the
ctlplane-subnetsubnet. For more information, see Subnets.
- Heat templates file to override.
The IP address or hostname defined for director Admin API endpoints over SSL/TLS. The director configuration attaches the IP address to the director software bridge as a routed IP address, which uses the
Sets the log level of undercloud services to
DEBUG. Set this value to
Enable or disable SELinux during the deployment. It is highly recommended to leave this value set to
trueunless you are debugging an issue.
- Defines the fully qualified host name for the undercloud. If set, the undercloud installation configures all system host name settings. If left unset, the undercloud uses the current host name, but you must configure all system host name settings appropriately.
The path to a log file to store the undercloud install and upgrade logs. By default, the log file is
install-undercloud.login the home directory. For example,
- A list of DNS nameservers to use for the undercloud hostname resolution.
- A list of network time protocol servers to help synchronize the undercloud date and time.
The IP address or hostname defined for director Public API endpoints over SSL/TLS. The director configuration attaches the IP address to the director software bridge as a routed IP address, which uses the
- The location and filename of the certificate for OpenStack SSL/TLS communication. Ideally, you obtain this certificate from a trusted certificate authority. Otherwise, generate your own self-signed certificate.
- Host timezone for the undercloud. If you do not specify a timezone, director uses the existing timezone configuration.
- Defines whether to update packages during the undercloud installation.
Each provisioning subnet is a named section in the
undercloud.conf file. For example, to create a subnet called
ctlplane-subnet, use the following sample in your
[ctlplane-subnet] cidr = 192.168.24.0/24 dhcp_start = 192.168.24.5 dhcp_end = 192.168.24.24 inspection_iprange = 192.168.24.100,192.168.24.120 gateway = 192.168.24.1 masquerade = true
You can specify as many provisioning networks as necessary to suit your environment.
The network that director uses to manage overcloud instances. This is the Provisioning network, which the undercloud
neutronservice manages. Leave this as the default
192.168.24.0/24unless you use a different subnet for the Provisioning network.
Defines whether to masquerade the network defined in the
cidrfor external access. This provides the Provisioning network with a degree of network address translation (NAT) so that the Provisioning network has external access through director.Note
The director configuration also enables IP forwarding automatically using the relevant
- dhcp_start; dhcp_end
- The start and end of the DHCP allocation range for overcloud nodes. Ensure that this range contains enough IP addresses to allocate your nodes.
- IP addresses to exclude in the DHCP allocation range.
DNS nameservers specific to the subnet. If no nameservers are defined for the subnet, the subnet uses nameservers defined in the
The gateway for the overcloud instances. This is the undercloud host, which forwards traffic to the External network. Leave this as the default
192.168.24.1unless you use a different IP address for director or want to use an external gateway directly.
Host routes for the Neutron-managed subnet for the overcloud instances on this network. This also configures the host routes for the
local_subneton the undercloud.
Temporary IP range for nodes on this network to use during the inspection process. This range must not overlap with the range defined by
dhcp_endbut must be in the same IP subnet.
6.10. Running the director upgrade
Complete the following steps to upgrade the director.
Run the following command to upgrade the director on the undercloud:
$ openstack undercloud upgrade
This command launches the director configuration script. The director upgrades its packages and configures its services to suit the settings in the
undercloud.conf. This script takes several minutes to complete.Note
The director configuration script prompts for confirmation before proceeding. Bypass this confirmation using the
$ openstack undercloud upgrade -y
The script also starts all OpenStack Platform service containers on the undercloud automatically. You manage each service through a
systemdresource. Check the
$ sudo systemctl list-units "tripleo_*"
systemdservice controls a container. Check the enabled containers using the following command:
$ sudo podman ps
The script adds the
stackuser to the
dockergroup to ensure that the
stackuser has access to container management commands. Refresh the
stackuser permissions with the following command:
$ exec su -l stack
The command prompts you to log in again. Enter the stack user password.
To initialize the
stackuser to use the command line tools, run the following command:
$ source ~/stackrc
The prompt now indicates OpenStack commands authenticate and execute against the undercloud;
The director upgrade is complete.