Chapter 4. Technical notes

This update includes the following enhancements:

A regression was introduced in ipmitool-1.8.18-11 that caused IPMI access to take over 2 minutes for certain BMCs that did not support the "Get Cipher Suites". As a result, introspection could fail and deployments could take much longer than previously.

With this update, ipmitool retries are handled differently, introspection passes, and deployments succeed.

Before this update, there were no retries and no timeout when downloading a final instance image with the direct deploy interface in ironic. As a result, the deployment could fail if the server that hosts the image fails to respond.

With this update, the image download process attempts 2 retries and has a connection timeout of 60 seconds. (BZ#1827721)

When you update or upgrade python3-tripleoclient, Ansible does not receive the update or upgrade and Ansible or ceph-ansible tasks fail.

When you update or upgrade, ensure that Ansible also receives the update so that playbook tasks can run successfully. (BZ#1852801)

There is an incomplete definition for TLS in the Orchestration service (heat) when you update from 16.0 to 16.1, and the update fails.

To prevent this failure, you must set the following parameter and value: InternalTLSCAFile: ''. (BZ#1840640)

Before this update, in deployments with an IPv6 internal API network, the Block Storage Service (cinder) and Compute Service (nova) were configured with a malformed glance-api endpoint URI. As a result, cinder and nova services located in a DCN or Edge deployment could not access the Image Service (glance).

With this update, the IPv6 addresses in the glance-api endpoint URI are correct and the cinder and nova services at Edge sites can access the Image Service successfully. (BZ#1815928)

Before this update, the ExtraConfigPre per_node script was not compatible with Python 3. As a result, the overcloud deployment failed at the step TASK [Run deployment NodeSpecificDeployment] with the message SyntaxError: invalid syntax.

With this update, the ExtraConfigPre per_node script is compatible with Python 3 and you can provision custom per_node hieradata. (BZ#1832920)

PowerMax configuration options have changed since Newton. This update includes the latest PowerMax configuration options and supports both iSCSI and FC drivers.

The CinderPowermaxBackend parameter also supports multiple back ends. CinderPowermaxBackendName supports a list of back ends, and you can use the new CinderPowermaxMultiConfig parameter to specify parameter values for each back end. For example syntax, see environments/cinder-dellemc-powermax-config.yaml. (BZ#1813393)

Support for Xtremio Cinder Backend

Updated the Xtremio cinder backend to support both iSCSI and FC drivers. It is also enhanceded to support multiple backends. (BZ#1852082)

PowerMax configuration options have changed since Newton. This update includes the latest PowerMax configuration options and supports both iSCSI and FC drivers.

The CinderPowermaxBackend parameter also supports multiple back ends. CinderPowermaxBackendName supports a list of back ends, and you can use the new CinderPowermaxMultiConfig parameter to specify parameter values for each back end. For example syntax, see environments/cinder-dellemc-powermax-config.yaml. (BZ#1852088)

Before this update, the data structure format that the ceph osd stat -f json command returns changed. As a result, the validation to stop the deployment unless a certain percentage of Red Hat Ceph Storage (RHCS) OSDs are running did not function correctly, and stopped the deployment regardless of how many OSDs were running.

With this update, the new version of openstack-tripleo-validations computes the percentage of running RHCS OSDs correctly and the deployment stops early if a percentage of RHCS OSDs are not running. You can use the parameter CephOsdPercentageMin to customize the percentage of RHCS OSDs that must be running. The default value is 66%. Set this parameter to 0 to disable the validation. (BZ#1845079)

With this enhancement, there are new options in the openstack tripleo container image push command that you can use to provide credentials for the source registry. The new options are --source-username and --source-password.

Before this update, you could not provide credentials when pushing a container image from a source registry that requires authentication. Instead, the only mechanism to push the container was to pull the image manually and push from the local system. (BZ#1811490)

With this update, the container_images_file parameter is now a required option in the undercloud.conf file. You must set this parameter before you install the undercloud.

With the recent move to use registry.redhat.io as the container source, you must authenticate when you fetch containers. For the undercloud, the container_images_file is the recommended option to provide the credentials when you perform the installation. Before this update, if this parameter was not set, the deployment failed with authentication errors when trying to fetch containers. (BZ#1819016)

The overcloud deployment steps included an older Ansible syntax that tagged the tripleo-bootstrap and tripleo-ssh-known-hosts roles as common_roles. This older syntax caused Ansible to run tasks tagged with the common_roles when Ansible did not use the common_roles tag. This syntax resulted in errors during the 13 to 16.1 system_upgrade process.

This update uses a newer syntax to tag the tripleo-bootstrap and tripleo-ssh-known-hosts roles as common_roles. Errors do not appear during the 13 to 16.1 system_upgrade process and you no longer include the --playbook upgrade_steps_playbook.yaml option to the system_upgrade process as a workaround. (BZ#1851914)

This update fixes a GRUB parameter naming convention that led to unpredictable behaviors on compute nodes during leapp upgrades.

Previously, the presence of the obsolete "TRIPELO" prefix on GRUB parameters caused problems.

The file /etc/default/grub has been updated with GRUB for the tripleo kernel args parameter so that leapp can upgrade it correctly. This is done by adding "upgrade_tasks" to the service "OS::TripleO::Services::BootParams", which is a new service added to all roles in the roles_data.yaml file. (BZ#1858673)

This update fixes a problem that caused baremetal nodes to become non-responsive during Leapp upgrades.

Previously, Leapp did not process transient interfaces like SR-IOV virtual functions (VF) during migration. As a result, Leapp did not find the VF interfaces during the upgrade, and nodes entered an unrecoverable state.

Now the service "OS::TripleO::Services::NeutronSriovAgent" sets the physical function (PF) to remove all VFs, and migrates workloads before the upgrade. After the successful Leapp upgrade, os-net-config runs again with the "--no-activate" flag to re-establish the VFs. (BZ#1866372)

This director enhancement automatically installs the Leapp utility on overcloud nodes to prepare for OpenStack upgrades. This enhancement includes two new Heat parameters: LeappRepoInitCommand and LeappInitCommand. In addition, if you have the following repository defaults, you do not need to pass UpgradeLeappCommandOptions values.

(BZ#1845726)

Before this update, director did not set the noout flag on Red Hat Ceph Storage OSDs before running a Leapp upgrade. As a result, additional time was required for the the OSDs to rebalance after the upgrade.

With this update, director sets the noout flag before the Leapp upgrade, which accelerates the upgrade process. Director also unsets the noout flag after the Leapp upgrade. (BZ#1853275)

Before this update, the Leapp upgrade could fail if you had any NFS shares mounted. Specifically, the nodes that run the Compute Service (nova) or the Image Service (glance) services hung if they used an NFS mount.

With this update, before the Leapp upgrade, director unmounts /var/lib/nova/instances, /var/lib/glance/images, and any Image Service staging area that you define with the GlanceNodeStagingUri parameter. (BZ#1853433)

Before this update, the Red Hat Ceph Storage dashboard listener was created in the HA Proxy configuration, even if the dashboard is disabled. As a result, upgrades of OpenStack with Ceph could fail.

With this update, the service definition has been updated to distinguish the Ceph MGR service from the dashboard service so that the dashboard service is not configured if it is not enabled and upgrades are successful. (BZ#1850991)

This update includes the following bug fix patches related to fully qualified domain names (FQDN).

This update makes it possible to run the Brocade FCZM driver in RHOSP 16.

The Brocade FCZM vendor chose not to update the driver for Python 3, and discontinued support of the driver past the Train release of OpenStack [1]. Red Hat OpenStack (RHOSP) 16 uses Python 3.6.

The upstream Cinder community assumed the maintenance of the Brocade FCZM driver on a best-effort basis, and the bugs that prevented the Brocade FCZM from running in a Python 3 environment (and hence in RHOSP 16) have been fixed.

[1] https://docs.broadcom.com/doc/12397527 (BZ#1848420)

This update fixes a problem that caused volume attachments to fail on a VxFlexOS cinder backend.

Previously, attempts to attach a volume on a VxFlexOS cinder backend failed because the cinder driver for the VxFlexOS back end did not include all of the information required to connect to the volume.

The VxFlexOS cinder driver has been updated to include all the information required in order to connect to a volume. The attachments now work correctly. (BZ#1862213)

Red Hat OpenStack Platform 16.1 includes the following PowerMax Driver updates:

Feature updates:

PowerMax Driver - Concurrent live migrations failure

Before this update, the Block Storage service (cinder) assigned the default volume type in a volume create request, ignoring alternative methods of specifying the volume type.

With this update, the Block Storage service performs as expected:

This enhancement adds a new driver for the Dell EMC PowerStore to support Block Storage service back end servers. The new driver supports the FC and iSCSI protocols, and includes these features:

The keepalived instance in the Red Hat OpenStack Platform Load-balancing service (octavia) instance (amphora) can abnormally terminate and interrupt UDP traffic. The cause of this issue is that the timeout value for the UDP health monitor is too small.

Workaround: specify a new timeout value that is greater than two seconds: $ openstack loadbalancer healthmonitor set --timeout 3 <heath_monitor_id>

For more information, search for "loadbalancer healthmonitor" in the Command Line Interface Reference. (BZ#1837316)

A known issue causes the migration of Ceph OSDs from Filestore to Bluestore to fail. In use cases where the osd_objectstore parameter was not set explicitly when you deployed OSP13 with RHCS3, the migration exits without converting any OSDs and falsely reports that the OSDs are already using Bluestore. For more information about the known issue, see https://bugzilla.redhat.com/show_bug.cgi?id=1875777

As a workaround, perform the following steps:

Undercloud timeouts (seconds):

parameter_defaults:
  TokenExpiration: 86400
  ZaqarWsTimeout: 86400

Overcloud deploy timeouts (minutes):

$ openstack overcloud deploy --timeout 1440

The timeouts are now set. (BZ#1792500)

Currently, you cannot scale down or delete compute nodes if Red Hat OpenStack Platform is deployed with TLS-e using tripleo-ipa. This is because the cleanup role, traditionally delegated to the undercloud as localhost, is now being invoked from the mistral container.

For more information, see https://access.redhat.com/solutions/5336241 (BZ#1866562)

This update fixes a bug that prevented the distributed compute nodes (DCN) compute servcie from accessing the glance service.

Previously, distributed compute nodes were configured with a glance endpoint URI that specified an IP address, even when deployed with internal transport layer security (TLS). Because TLS requires the endpoint URI to specify a fully qualified domain name (FQDN), the compute service could not access the glance service.

Now, when deployed with internal TLS, DCN services are configured with glance endpoint URI that specifies a FQDN, and the DCN compute service can access the glance service. (BZ#1873329)

This update adds support for encrypted volumes and images on distributed compute nodes (DCN).

DCN nodes can now access the Key Manager service (barbican) running in the central control plane.

For example:

$ openstack overcloud roles generate DistributedComputeHCI DistributedComputeHCIScaleOut -o ~/dcn0/roles_data.yaml

Use the appropriate path to the roles data file. (BZ#1852851)

Before this update, to successfully run a leapp upgrade during the fast forward upgrade (FFU) from RHOSP 13 to RHOSP16.1, the node where the Red Hat Enterprise Linux upgrade was occurring had to have the PermitRootLogin field defined in the ssh config file (/etc/ssh/sshd_config).

With this update, the Orchestration service (heat) no longer requires you to modify /etc/ssh/sshd_config with the PermitRootLogin field. (BZ#1855751)

Transmission of jumbo UDP frames on ML2/OVN routers depends on a kernel release that is not yet avaialbe.

After receiving a jumbo UDP frame that exceeds the maximum transmission unit of the external network, ML2/OVN routers can return ICMP "fragmentation needed" packets back to the sending VM, where the sending application can break the payload into smaller packets. To determine the packet size, this feature depends on discovery of MTU limits along the south-to-north path.

South-to-north path MTU discovery requires kernel-4.18.0-193.20.1.el8_2, which is scheduled for availability in a future release. To track availability of the kernel version, see https://bugzilla.redhat.com/show_bug.cgi?id=1860169. (BZ#1547074)

This update modifies get_device_info to use lsscsi to get [H:C:T:L] values, making it possible to support more than 255 logical unit numbers (LUNs) and host logical unit (HLU) ID values.

Previously, get_device_info used sg_scan to get these values, with a limit of 255.

You can get two device types with get_device_info:

This update fixes an incompatibility that caused VxFlex volume detachment attempts to fail.

A recent change in VxFlex cinder volume credentialing methods was not backward compatible with pre-existing volume attachments. If a VxFlex volume attachment was made before the credentialing method change, attempts to detach the volume failed.

Now the detachments do not fail. (BZ#1869346)

The entry in /etc/hosts for the undercloud duplicates anytime the Compute stack is updated on the undercloud and overcloud nodes. This occurs for split-stack deployments where the Controllers and Compute nodes are divided into multiple stacks.

Other indications of this problem are the following:

This update increases the speed of stack updates in certain cases.

Previously, stack update performance was degraded when the Ansible --limit option was not passed to ceph-ansible. During a stack update, ceph-ansible sometimes made idempotent updates on nodes even if the --limit argument was used.

Now director intercepts the Ansible --limit option and passes it to the ceph-ansible excecution. The --limit option passed to commands starting with 'openstack overcloud' deploy is passed to the ceph-ansible execution to reduce the time required for stack updates.

Before this update, the Block Storage (cinder) service would always assign newly created volumes with the default volume type, even when the volume was created from another source, such as an image, snapshot or another volume. This resulted in volumes created from another source having a different volume type from the volume type of the source.

With this update, the default volume type is assigned only after determining whether it should be assigned based on the volume type of the source. The volume type of volumes created from another source now match the volume type of the source. (BZ#1921735)

The virt-admin tool is now available for you to use to capture logs for reporting RHOSP bugs. This tool is useful for troubleshooting all libvirt and QEMU problems, as the logs provide the communications between libvirt and QEMU on the Compute nodes. You can use virt-admin to set the libvirt and QEMU debug log filters dynamically, without having to restart the nova_libvirt container.

Perform the following steps to enable libvirt and QEMU log filters on a Compute node:

Before this update, in-place upgrades from Red Hat OpenStack Platform 13 to 16.1 in a TLS everywhere environment used an incorrect rabbitmq password for the novajoin container. This caused the novajoin container on the undercloud to function incorrectly, which caused any overcloud node that ran an upgrade to fail with the following error:

2020-11-24 20:01:31.569 7 ERROR join   File "/usr/lib/python3.6/site-packages/amqp/connection.py", line 639, in _on_close
2020-11-24 20:01:31.569 7 ERROR join     (class_id, method_id), ConnectionError)
2020-11-24 20:01:31.569 7 ERROR join amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For detail see the broker logfile.

With this update, the upgrade from RHOSP 13 to 16.1 uses the correct rabbitmq password in a TLS everywhere environment so that the framework for upgrades can complete successfully. (BZ#1901157)

With this enhancement, you can manage vPMEM with two new parameters NovaPMEMMappings and NovaPMEMNamespaces.

There is currently a known issue with the mechanism that ensures the subscribed environments have the right DNF module stream set. The Advanced Virtualization repository is not always available in the subscription that the Ceph nodes use, which causes the upgrade or update of a Ceph node to fail when you try to enable virt:8.2.

Workaround:

Override the DnfStreams parameter in the upgrade or update environment file to prevent the Ceph upgrade from failing:

parameter_defaults:
  ...
  DnfStreams: [{'module':'container-tools', 'stream':'2.0'}]

For more information, see https://bugzilla.redhat.com/show_bug.cgi?id=1923887. (BZ#1866479)

Before this update, if you had NovaComputeEnableKsm enabled and you were using Red Hat Subscription Management to register the overcloud Compute nodes, the qemu-kvm-common package failed to install. This was because the configuration was sometimes applied before the Compute nodes were registered to the required repositories.

With this update, NovaComputeEnableKsm is enabled only after the Compute nodes are registered to the required repositories by using Red Hat Subscription Management, which ensures that the qemu-kvm-common package is successfully installed. (BZ#1895894)

Before this update, when deployed at an edge site the Image (glance) service was not configured to access the Key Manager (barbican) service running on the central site’s control plane. This resulted in the Image services running on edge sites being unable to access encryption keys stored in the Key Manager service.

With this update, Image services running on edge sites are now configured to access the encryption keys stored in the Key Manager service. (BZ#1899761)

This enhancement adds support for vlan transparency in the ML2/OVN mechanism driver with vlan and geneve network type drivers.

With vlan transparency, you can manage vlan tags by using instances on Networking (neutron) service networks. You can create vlan interfaces on an instance and use any vlan tag without affecting other networks. The Networking service is not aware of these vlan tags.

NOTE

Before this update, if a user configured the ContainerImagePrepare parameter to use a custom tag, such as 'tag: "latest"' or 'tag: "16.1"', instead of the standard 'tag_from_label: "{version}-{release}"', the containers did not update to the latest container images.

With this update, the container images are always fetched anytime a user runs a deployment action, including updates, and the image ID is checked against the running container to see if it needs to be rebuilt to consume the latest image. Containers are now always refreshed during deployment actions and restarted if they are updated.

(BZ#1881476)

Before this update, live migration failed when upgrading a TLS everywhere environment with local ephemeral storage and UseTLSTransportForNbd set to "False". This occurred because the default value of the UseTLSTransportForNbd configuration had changed from "False" in RHOSP 13 to "True" in RHOSP 16.x, which resulted in the correct certifications not being included in the QEMU process containers.

With this update, director checks the configuration of the previously deployed environment for global_config_settings and uses it to ensure that the UseTLSTransportForNbd state stays the same in the upgrade as on previous deployment. If global_config_settings exists in the configuration file, then director checks the configuration of the use_tls_for_nbd key. If global_config_settings does not exist, the director evaluates the hieradata key nova::compute::libvirt::qemu::nbd_tls. Keeping the UseTLSTransportForNbd state the same in the upgraded deployment as on previous deployment ensures that live migration works. (BZ#1906698)

In prior releases, the SolidFire driver created a duplicate volume whenever it retried an API request. This led to unexpected behavior due to the accumulation of unused volumes.

With this update, the Block Storage service (cinder) checks for existing volume names before it creates a volume. When Block Storage service detects a read timeout, it immediately checks for volume creation to prevent invalid API calls. This update also adds the sf_volume_create_timeout option for the SolidFire driver so that you can set an appropriate timeout value for your environment. (BZ#1939398)

This update fixes an issue that caused some API calls, such as create snapshot, to fail with an xNotPrimary error during workload re-balancing operations.

When SolidFire is under heavy load or being upgraded, the SolidFire cluster might re-balance cluster workload by automatically moving connections from primary to secondary nodes. Previously, some API calls failed with an xNotPrimary error during these workload balance operations and were not retried.

This update fixes the issue by adding the xNotPrimary exception to the SolidFire driver list of retryable exceptions. (BZ#1947474)

When an instance is created, the Compute (nova) service sanitizes the instance display name to generate a valid host name when DNS integration is enabled in the Networking (neutron) service.

Before this update, the sanitization did not replace periods ('.') in instance names, for example, 'rhel-8.4'. This could result in display names being recognized as Fully Qualified Domain Names (FQDNs) which produced invalid host names. When instance names contained periods and DNS integration was enabled in the Networking service, the Networking service rejected the invalid host name, which resulted in a failure to create the instance and a HTTP 500 server error from the Compute service.

With this update, periods are now replaced by hyphens in instance names to prevent host names being parsed as FQDNs. You can continue to use free-form strings for instance display names. (BZ#1872314)

This update modifies the registry metadata creator to handle containers with and without namespaces in their URI. On the undercloud you can now manage containers that comply with the following formats:

undercloud_host:port/namespace/container:tag undercloud_host:port/container:tag

Red Hat does not support more complex namespaces, such as undercloud_host:port/name/space/container:tag, when pushing to the undercloud. (BZ#1919445)

This update fixes a configuration problem that caused Leapp upgrades to stop and fail while executing on a CephStorage node.

Previously, CephStorage nodes were incorrectly configured to consume OpenStack highavailability, advanced-virt, and fast-datapath repos during Leapp upgrades.

Now UpgradeLeappCommand options is configurable on a per-node basis, and uses the correct default for CephStorage nodes, and Leapp upgrades succeed for CephStorage nodes. (BZ#1936419)

This update fixes a bug that caused failure of validations before openstack undercloud upgrade in some cases. Before this upgrade, a lack of permissions needed to access the requested logging directory sometimes resulted in the following failures:

Before this update, creating a volume from a snapshot of an encrypted volume could result in an unusable volume. When the destination volume is the same size as the source volume, creating an encrypted volume from a snapshot of an encrypted volume truncated the data in the new volume, which caused a size discrepancy.

With this update, the RBD back end accounts for the encryption header and does not truncate the data so that creating a volume from a snapshot of an encrypted volume does not cause the error. (BZ#1987104)

In previous releases, in Red Hat OpenStack Platform (RHOSP) deployments that use the Dell EMC XtremIO driver, attach volume operations waited for a timeout if iSCSI or FC targets were not connected to a RHOSP host. This caused attach volume operations to fail.

This release adds port filtering support for the Dell EMC XtremIO driver to allow iSCSI or FC ports that are not in use to be ignored. (BZ#1930255)

In previous releases, if Dell EMC PowerStore ports were configured for multiple purposes, such as iSCSI, replication, incorrect REST filtering caused the cinder driver to report that no accessible iSCSI targets were found.

This release fixes the Dell EMC PowerStore REST filter functionality. (BZ#1945306)

Before this update, a failure occurred when users wanted to delete the DEFAULT volume type.

With this update, you can delete the DEFAULT volume type when it is not set as the value of the default_volume_type parameter in the cinder.conf file. The default value of the default_volume_type parameter is DEFAULT so you must set it to an appropriate volume type, for example 'tripleo', so that you can delete the DEFAULT volume type. (BZ#1947415)

Before this update, the Shared File Systems service (manila) dashboard had dynamic form elements whose names could potentially cause the forms to become unresponsive. This meant that the creation of share groups, share networks, and shares within share networks did not function.

With this update, dynamic elements whose names might be problematic are encoded. The creation of share groups, share networks, and shares within share networks functions normally. (BZ#1938212)

The logic to detect the hypervisor hostname has been fixed and now returns the result consistent with libvirt driver in the Compute service (nova). With this fix, you no longer need to specify the resource_provider_hypervisors option when you use the guaranteed minimum bandwidth QoS feature.

With this update, a new option, resource_provider_default_hypervisor, has been added to the Modular Layer 2 with the Open Virtual Network mechanism driver (ML2/OVN) to replace the default hypervisor name. The option locates the root resource provider without giving a complete list of interfaces or bridges in the resource_provider_hypervisors option in case it has to be customized by the user. This new option is located in the [ovs] ini-section for the ovs-agent, and in the [sriov_nic] ini-section for the sriov-agent. (BZ#1900500)

This enhancement adds the new CinderRpcResponseTimeout and CinderApiWsgiTimeout parameters to support tuning RPC and API WSGI timeouts in the Block Storage service (cinder). Default timeout values might not be adequate for large deployments and in situations where transactions might be delayed due to system load.

It is now possible to tune the RPC and API WSGI timeouts to prevent transactions prematurely timing out. (BZ#1930806)

Currently, there is a known issue where it is not possible to simulate certain real-life scenarios when the MAC-IP addresses of a port are unknown. The RHOSP Networking service (neutron) directly specifies the MAC-IP of a port even if DHCP or security groups are not configured.

Workaround: upgrade to RHOSP 16.1.7 and install ML2/OVN v21.03. If DHCP and port security are disabled, then the addresses field of a port does not include its MAC-IP address pairs, and ML2/OVN can use the MAC learning capabilities to send traffic only to the desired port. (BZ#1898198)

Before this update, there were unhandled exceptions during connection to iSCSI portals. For example, failures in iscsiadm -m session. This occurred because the _connect_vol threads can abort unexpectedly in some failure patterns, and this abort causes a hang in subsequent steps while waiting for results from _connct_vol threads.

With this update, any exceptions during connection to iSCSI portals are handled in the _connect_vol method correctly and avoids any unexpected abort without updating thread results. (BZ#1977792)

With this update, the tripleo validator command now accepts variables and environment variables in a key-value pair format. In past releases, only JSON dictionaries allowed environment variables.

openstack tripleo validator run \
[--extra-vars key1=<val1>[,key2=val2 --extra-vars key3=<val3>] \
| --extra-vars-file EXTRA_VARS_FILE] \
[--extra-env-vars key1=<val1>[,key2=val2 --extra-env-vars key3=<val3>]]
(--validation <validation_id>[,<validation_id>,...] | --group <group>[,<group>,...])

$ openstack tripleo validator run --validation check-cpu,check-ram --extra-vars minimal_ram_gb=8 --extra-vars minimal_cpu_count=2

For the complete list of supported options, run:

$ openstack tripleo validator run --help

(BZ#1959492)

Before this update, during a tripleo validation on an OpenStack component, the following exception error occurred:

Unhandled exception during validation run.

This error occurred because a variable in the code was referenced, but never assigned.

With this update, this problem has been fixed and validations run without this error. (BZ#1959866)

Nodes rebooted during the RHOSP upgrade.

These errors were caused when the KernelArgs parameter, or the order of values in the string, changed or when a KernelArgs parameter was added.

With this update, TripleO has added upgrade tasks in kernel-boot-params-baremetal-ansible.yaml to migrate from TRIPLEO_HEAT_TEMPLATE_KERNEL_ARGS to GRUB_TRIPLEO_HEAT_TEMPLATE_KERNEL_ARGS.

This change was made to accommodate the Red Hat Enterprise Linux (RHEL) in-place upgrade tool, LEAPP, which is used to upgrade RHEL from version 7 to version 8, during the RHOSP version 13 to version 16 FFU process. LEAPP understands GRUB parameters only when the parameters start with GRUB_ in /etc/default/grub.

Despite this update, you must manually inspect each KernelArgs value to ensure that it matches the value for all hosts in the corresponding role.

The KernelArgs value may come from the PreNetworkConfig implementation from either the default tripleo-heat-templates or third-party heat templates.

If you find any mismatches, change the value of the KernelArgs parameter in the corresponding role to match the value of KernelArgs on the hosts. Perform these checks before running the openstack overcloud upgrade prepare command.

You can use the following script to check KernelArgs values:

tripleo-ansible-inventory --static-yaml-inventory inventory.yaml
KernelArgs='< KernelArgs_ FROM_THT >'
ansible -i inventory.yaml ComputeSriov -m shell -b -a "cat /proc/cmdline | grep '${KernelArgs}'"

(BZ#1980829)

Before this update, the GPFS SpectrumScale driver in the Block Storage service (cinder) did not correctly detect whether the storage back end supported copy-on-write (COW) mode. As a result, the driver disabled COW features, such as the ability to rapidly create volumes from an image. Sometimes, this caused some instances to time out when booting multiple instances simultaneously from an image.

With this update, the GPFS SpectrumScale driver properly detects COW support for storage back ends. (BZ#1960639)

Before this update, when creating a snapshot with PowerMaxOS 5978.711, REST experienced a payload response change and caused the device label to modify its format. The underlying data from the solutions enabler changed, and no longer contained a colon character (:). This resulted in an IndexError exception in the PowerMax Driver:

IndexError: list index out of range

With this update, the problem is resolved in PowerMaxOS 5978.711 and later. (BZ#1992159)

With this update, the telemetry healthchecks have been made more robust and the way the healthchecks are parsed has been simplified.

To get verbose mode when you run the healthcheck directly, run the command sudo podman -u root -e "HEALTHCHECK_DEBUG=1" <container> /openstack/healthcheck (BZ#1910939)

Before this release, the collectd container failed to start on Compute nodes because a dpdk-telemetry collectd configuration file was being automatically created despite there being no dpdk-telemetry plugin installed.

As of this release, dpdk_telemetry configuration files have been removed from the the collectd container. (BZ#1996865)

Enable the experimental rsyslog reopenOnTruncate to ensure that rsyslog immediately recognizes when a logrotation happens on a file. The setting affects every service configured to work with rsyslog.

With rsyslog reopenOnTruncate disabled, rsyslog waits for a log file to fill to its original capacity before consuming any additional logs. (BZ#1939964)

This enhancement prepares your environment for update of the metrics_qdr service to a newer AMQ Interconnect release, which requires import of the CA certificate contents from the Service Telemetry Framework (STF) deployment. Changes are not yet required by administrators when deploying or updating Red Hat OpenStack Service Platform (RHOSP) as the metrics_qdr service has not yet been updated. This functionality is in preparation of the metrics_qdr service update in a future release.

The following procedure will be required once https://bugzilla.redhat.com/show_bug.cgi?id=1949169 has shipped.

This update corrects this problem by providing a new Orchestration service (heat) parameter, MetricsQdrSSLProfiles.

To obtain a Red Hat OpenShift TLS certificate, run the following commands:

$ oc get secrets
$ oc get secret/default-interconnect-selfsigned -o jsonpath='{.data.ca\.crt}' | base64 -d

Add the MetricsQdrSSLProfiles parameter with the contents of your Red Hat OpenShift TLS certificate to a custom environment file:

MetricsQdrSSLProfiles:
    -   name: sslProfile
        caCertFileContent: |
           -----BEGIN CERTIFICATE-----
           ...
           TOpbgNlPcz0sIoNK3Be0jUcYHVMPKGMR2kk=
           -----END CERTIFICATE-----

Then, redeploy your overcloud with the openstack overcloud deploy command. (BZ#1949168)

Before this update, a do_sync_check operation could result in the incorrect deletion of non-temporary snapshots from a volume because there was no check for non-temporary snapshots deletion during the do_sync_check operation. With this update, there is a check to determine if a snapshot must be deleted. The do_sync_check operation does not perform unnecessary non-temporary snapshot deletions.

Before this update, there was a case mismatch in the conditional while checking if a storage group was a child of a parent storage group. While modifying the storage group, errors indicated that the parent storage group already contained the child storage group. With this update, the patterns used in the conditional are not case-sensitive and you can modify the storage group successfully. (BZ#2129310)

There is currently a known issue when live migrating instances that have CPUs that are incompatible with the destination host CPUs.

Workaround: Add the following configuration in the nova.conf file of each affected Compute node to skip CPU comparison on the destination host:

[workarounds]
skip_cpu_compare_on_dest = True

(BZ#2076884)

This update in RHOSP 16.1.9 fixes a bug that causes the Networking service (neutron) to fail to start after an update to RHOSP 16.1.8 and also causes OVN database instability after updates to RHOSP 16.1.8.

Instead of updating to RHOSP 16.1.8, update directly to RHOSP 16.1.9. (BZ#2125824)