Chapter 3. Release information

Before this update, director did not set the noout flag on Red Hat Ceph Storage OSDs before running a Leapp upgrade. As a result, additional time was required for the OSDs to rebalance after the upgrade.

With this update, director sets the noout flag before the Leapp upgrade, which accelerates the upgrade process. Director also unsets the noout flag after the Leapp upgrade.

PowerMax configuration options have changed since OSP10-newton. This update includes the latest PowerMax configuration options and supports both iSCSI and FC drivers.

The CinderPowermaxBackend parameter also supports multiple back ends. CinderPowermaxBackendName supports a list of back ends, and you can use the new CinderPowermaxMultiConfig parameter to specify parameter values for each back end. For example syntax, see environments/cinder-dellemc-powermax-config.yaml.

Before this update, in DCN + HCI deployments with an IPv6 internal API network, the Block Storage service (cinder) and etcd services were configured with malformed etcd URIs, and the Block Storage service and etcd services failed on startup.

With this update, the IPv6 addresses in the etcd URI are correct and the Block Storage service and etcd services start successfully.

Before this update, in deployments with an IPv6 internal API network, the Block Storage service (cinder) and Compute service (nova) were configured with a malformed glance-api endpoint URI. As a result, Block Storage service and Compute services located in a DCN or Edge deployment could not access the Image service (glance).

With this update, the IPv6 addresses in the glance-api endpoint URI are correct and the Block Storage and Compute services at Edge sites can access the Image service successfully.

Before this update, the Block Storage service (cinder) assigned the default volume type in a volume create request, ignoring alternative methods of specifying the volume type.

With this update, the Block Storage service performs correctly:

Before this update, there were no retries and no timeout when downloading a final instance image with the direct deploy interface in the Bare Metal Provisioning service (ironic). As a result, the deployment could fail if the server that hosts the image fails to respond.

With this update, the image download process attempts 2 retries and has a connection timeout of 60 seconds.

A regression was introduced in ipmitool-1.8.18-11 that caused IPMI access to take over two minutes for certain BMCs that did not support the "Get Cipher Suites". As a result, introspection could fail and deployments could take much longer than previously.

With this update, ipmitool retries are handled differently, introspection passes, and deployments succeed.

Before this update, the ExtraConfigPre per_node script was not compatible with Python 3. As a result, the overcloud deployment failed at the step TASK [Run deployment NodeSpecificDeployment] with the message SyntaxError: invalid syntax.

With this update, the ExtraConfigPre per_node script is compatible with Python 3 and you can provision custom per_node hieradata.

Before this update, the data structure format that the ceph osd stat -f json command returns changed. As a result, the validation to stop the deployment unless a certain percentage of Red Hat Ceph Storage (RHCS) OSDs are running did not function correctly, and stopped the deployment regardless of how many OSDs were running.

With this update, the new version of openstack-tripleo-validations computes the percentage of running RHCS OSDs correctly and the deployment stops early if a percentage of RHCS OSDs are not running. You can use the parameter CephOsdPercentageMin to customize the percentage of RHCS OSDs that must be running. The default value is 66%. Set this parameter to 0 to disable the validation.

Before this update, the Red Hat Ceph Storage dashboard listener was created in the HA Proxy configuration, even if the dashboard is disabled. As a result, upgrades of Red Hat OpenStack Platform with Ceph Storage could fail.

With this update, the service definition has been updated to distinguish the Ceph MGR service from the dashboard service so that the dashboard service is not configured if it is not enabled and upgrades are successful.

Before this update, the Leapp upgrade could fail if you had any NFS shares mounted. Specifically, the nodes that run the Compute service (nova) or the Image service (glance) services hung if they used an NFS mount.

With this update, before the Leapp upgrade, director unmounts /var/lib/nova/instances, /var/lib/glance/images, and any Image service staging area that you define with the GlanceNodeStagingUri parameter.

With this enhancement, there are new options in the openstack tripleo container image push command that you can use to provide credentials for the source registry. The new options are --source-username and --source-password.

Before this update, you could not provide credentials when pushing a container image from a source registry that requires authentication. Instead, the only mechanism to push the container was to pull the image manually and push from the local system.

With this enhancement, you can use policy-based routing for Red Hat OpenStack Platform nodes to configure multiple route tables and routing rules with os-net-config.

Policy-based routing uses route tables where, on a host with multiple links, you can send traffic through a particular interface depending on the source address. You can also define route rules for each interface.

With this update, the container_images_file parameter is now a required option in the undercloud.conf file. You must set this parameter before you install the undercloud.

With the recent move to use registry.redhat.io as the container source, you must authenticate when you fetch containers. For the undercloud, the container_images_file is the recommended option to provide the credentials when you perform the installation. Before this update, if this parameter was not set, the deployment failed with authentication errors when trying to fetch containers.

With this enhancement, you can manage vPMEM with two new parameters NovaPMEMMappings and NovaPMEMNamespaces.

Use NovaPMEMMappings to set the nova configuration option pmem_namespaces that reflects mappings between vPMEM and physical PMEM namespaces.

Use NovaPMEMNamespaces to create and manage physical PMEM namespaces that you use as a back end for vPMEM.

In Red Hat OpenStack Platform 16.1 there is a technology preview for routed provider networks with the ML2/OVS mechanism driver. You can use a routed provider network to enable a single provider network to represent multiple layer 2 networks (broadcast domains) or segments so that the operator can present only one network to users. This is a common network type in Edge DCN deployments and Spine-Leaf routed datacenter deployments.

Because the Compute service (nova) scheduler is not segment-aware, you must map each leaf, rack segment, or DCN edge site to a Compute service host-aggregate or availability zone. If the deployments require DHCP or the metadata service, you must also define a Compute service availability zone for each edge site or segment.

Known limitations:

For glance image conversion, the glance-direct method is not enabled by default. To enable this feature, set enabled_import_methods to [glance-direct,web-download] or [glance-direct] in the DEFAULT section of the glance-api.conf file.

The Image service (glance) must have a staging area when you use the glance-direct import method. Set the node_staging_uri option int he DEFAULT section of the glance-api.conf file to file://<absolute-directory-path>. This path must be on a shared file system that is available to all Image service API nodes.

When you upgrade from Red Hat OpenStack Platform (RHOSP) 13 DCN to RHOSP 16.1 DCN, it is not possible to migrate from the single stack RHOSP 13 deployment into a multi-stack RHOSP 16.1 deployment. The RHOSP 13 stack continues to be managed as a single stack in the Orchestration service (heat) even after you upgrade to RHOSP 16.1.

After you upgrade to RHOSP 16.1, you can deploy new DCN sites as new stacks. For more information, see the multi-stack documentation for RHOSP 16.1 DCN.

Red Hat OpenStack Platform 16.1 includes the following PowerMax Driver updates:

Feature updates:

The sg-bridge container uses the sg-bridge RPM to provide an AMQP1-to-unix socket interface for sg-core. Both components are part of the Service Telemetry Framework.

This is the initial release of the sg-bridge component.

The sg-bridge and sg-core container images provide a new data path for collectd metrics into the Service Telemetry Framework.

The sg-bridge component provides an AMQP1 to unix socket translation to the sg-core, resulting in a 500% performance increase over the legacy Smart Gateway component.

This is the initial release of the sg-bridge and sg-core container image components.

OVN serves DHCP as an openflow controller with ovn-controller directly on Compute nodes. However, SR-IOV instances are attached directly to the network through the VF/PF and so SR-IOV instances cannot receive DHCP responses.

Workaround: Change OS::TripleO::Services::NeutronDhcpAgent to OS::TripleO::Services::NeutronDhcpAgent: deployment/neutron/neutron-dhcp-container-puppet.yaml.

The keepalived instance in the Red Hat OpenStack Platform Load-balancing service (octavia) instance (amphora) can abnormally terminate and interrupt UDP traffic. The cause of this issue is that the timeout value for the UDP health monitor is too small.

Workaround: specify a new timeout value that is greater than two seconds: $ openstack loadbalancer healthmonitor set --timeout 3 <heath_monitor_id>

For more information, search for "loadbalancer healthmonitor" in the Command Line Interface Reference.

There is an incomplete definition for TLS in the Orchestration service (heat) when you update from 16.0 to 16.1, and the update fails.

To prevent this failure, you must set the following parameter and value: InternalTLSCAFile: ''.

There is a known issue when you update from 16.0 to 16.1 with Public TLS or TLS-Everywhere.

The parameter InternalTLSCAFile provides the location of the CA cert bundle for the overcloud instance. Upgrades and updates fail if this parameter is not set correctly. With new deployments, heat sets this parameter correctly, but if you upgrade a deployment that uses old heat templates, then the defaults might not be correct.

Workaround: Set the InternalTLSCAFile parameter to an empty string '' so that the undercloud uses the certificates in the default trust store.

There is a known issue when upgrading from RHOSP 13 to RHOSP 16.1. The value of HostnameFormatDefault has changed from %stackname%-compute-%index% to %stackname%-novacompute-%index%. This change in default value can result in duplicate service entries and have further impacts on operations such as live migration.

Workaround: If you upgrade from RHOSP 13 to RHOSP 16.1, you must override the HostnameFormatDefault value to configure the previous default value to ensure that the previous hostname format is retained. If you upgrade from RHOSP 15 or RHOSP 16.0, no action is required.

The output format of tripleo-ansible-inventory changed in RHOSP 16.1. As a result, the generate-inventory step fails.

Workaround: Create the inventory manually.

There is a known issue where a heat parameter InternalTLSCAFile is used during deployment when the undercloud contacts the external (public) endpoint to create initial resources and projects. If the internal and public interfaces have certificates from different Certificate Authorities (CAs), the deployment fails. Either the undercloud fails to contact the keystone public interface, or the internal interfaces receive malformed configuration.

This scenario affects deployments with TLS Everywhere, when the IPA server supplies the internal interfaces but the public interfaces have a certificate that the operator supplies. This also prevents 'brown field' deployments, where deployments with existing public certificates attempt to redeploy and configure TLS Everywhere.

There is currently no workaround for this defect.

There is a known issue in the Block Storage service (cinder) due to the following conditions:

There is a known issue with the Object Storage service (swift). If you use pre-deployed nodes, you might encounter the following error message in /var/log/containers/stdouts/swift_rsync.log:

"failed to create pid file /var/run/rsyncd.pid: File exists"

Workaround: Enter the following command on all Controller nodes that are pre-deployed:

for d in $(podman inspect swift_rsync | jq '.[].GraphDriver.Data.UpperDir') /var/lib/config-data/puppet-generated/swift; do sed -i -e '/pid file/d' $d/etc/rsyncd.conf; done

When you update or upgrade python3-tripleoclient, Ansible does not receive the update or upgrade and Ansible or ceph-ansible tasks fail.

When you update or upgrade, ensure that Ansible also receives the update so that playbook tasks can run successfully.

There is a known issue with the OVN filter packets that ovn-controller generates. Router Advertisements that receive ACL processing in OVN are dropped if there is no explicit ACL rule to allow this traffic.

Workaround: Enter the following command to create a security rule:

openstack security group rule create --ethertype IPv6 --protocol icmp --icmp-type 134 <SECURITY_GROUP>

There are some known limitations for Mellanox ConnectX-5 adapter cards in VF LAG mode in OVS OFFLOAD deployments, SRIOV Switchdev mode.

You might encounter the following known issues and limitations when you use the Mellanox ConnectX-5 adapter cards with the virtual function (VF) link aggregation group (LAG) configuration in an OVS OFFLOAD deployment, SRIOV Switchdev mode:

The Ceph Dashboard currently does not work with the TLS Everywhere framework because the dashboard_protocol parameter was incorrectly omitted from the heat template. As a result, back ends fail to appear when HAproxy is started.

As a temporary solution, create a new environment file that contains the dashboard_protocol parameter and include the environment file in your overcloud deployment with the -e option:

parameter_defaults:
  CephAnsibleExtraConfig:
    dashboard_protocol: 'https'

This solution introduces a ceph-ansible bug. For more information, see https://bugzilla.redhat.com/show_bug.cgi?id=1860815.

There is a known issue where, after an ungraceful shutdown, Ceph containers might not start automatically on system reboot.

Workaround: Remove the old container IDs manually with the podman rm command. For more information, see https://bugzilla.redhat.com/show_bug.cgi?id=1858865#c2.

There is a known issue where enabling the realtime-virtual-host tuned profile inside guest virtual machines degrades throughput and displays non-deterministic performance. ovs-dpdk PMDs are pinned incorrectly to housekeeping CPUs.

Workaround: Use the cpu-partitioning tuned profile inside guest virtual machines, write a post-deployment script to update the tuned.conf file, and reboot the node:

ps_blacklist=ksoftirqd.*;rcuc.*;rcub.*;ktimersoftd.*;.*pmd.*;.*PMD.*;^DPDK;.*qemu-kvm.*

If you change the TRIPLEO_HEAT_TEMPLATE_KERNEL_ARGS, such as the value for the hugepages parameter, during a fast-forward upgrade (FFU) from Red Hat OpenStack Platform (RHOSP) 13 to RHOSP 16.1, the upgrade fails because of duplicate entries for the kernel args.

Avoid changing the kernel args during FFU.

Workaround: In RHOSP 16.1 you can manually change the kernel args, which are usually located at /usr/share/ansible/roles/tripleo-kernel/tasks/kernelargs.yml

This director enhancement automatically installs the Leapp utility on overcloud nodes to prepare for OpenStack upgrades.https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/release_notes/index This enhancement includes two new Heat parameters: LeappRepoInitCommand and LeappInitCommand. In addition, if you have the following repository defaults, you do not need to pass UpgradeLeappCommandOptions values.

--enablerepo rhel-8-for-x86_64-baseos-eus-rpms --enablerepo rhel-8-for-x86_64-appstream-eus-rpms --enablerepo rhel-8-for-x86_64-highavailability-eus-rpm1866372s --enablerepo advanced-virt-for-rhel-8-x86_64-rpms --enablerepo ansible-2.9-for-rhel-8-x86_64-rpms --enablerepo fast-datapath-for-rhel-8-x86_64-rpms

This update fixes a bug that caused the generate-inventory step to fail during in-place migration from ML2/OVS to ML2/OVN.

Note that in the Red Hat OpenStack Platform 16.1.0 (GA release), migration from ML2/OVS to ML2/OVN was not supported. As of Red Hat OpenStack Platform 16.1.1, in-place migration is supported for non-NFV deployments, with various exceptions, limitations, and requirements as described in "Migrating from ML2/OVS to ML2/OVN." [1]

[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/networking_with_open_virtual_network/index#migrating-ml2ovs-to-ovn

Before this update, the Red Hat Ceph Storage Dashboard listener was created in the HA Proxy configuration, even if the Dashboard is disabled. As a result, upgrades of Red Hat OpenStack Platform (RHOSP) with Ceph could fail.

With this update, the service definition has been updated to distinguish the Ceph MGR service from the Dashboard service so that the Dashboard service is not configured if it is not enabled and upgrades are successful.

The overcloud deployment steps included an older Ansible syntax that tagged the tripleo-bootstrap and tripleo-ssh-known-hosts roles as common_roles. This older syntax caused Ansible to run tasks tagged with the common_roles when Ansible did not use the common_roles tag. This syntax resulted in errors during the 13 to 16.1 system_upgrade process.

This update uses a newer syntax to tag the tripleo-bootstrap and tripleo-ssh-known-hosts roles as common_roles. Errors do not appear during the 13 to 16.1 system_upgrade process and you no longer include the --playbook upgrade_steps_playbook.yaml option to the system_upgrade process as a workaround.

Before this update, director did not set the noout flag on Red Hat Ceph Storage OSDs before running a Leapp upgrade. As a result, additional time was required for the OSDs to rebalance after the upgrade.

With this update, director sets the noout flag before the Leapp upgrade, which accelerates the upgrade process. Director also unsets the noout flag after the Leapp upgrade.

Before this update, the Leapp upgrade could fail if you had any NFS shares mounted. Specifically, the nodes that run the Compute Service (nova) or the Image Service (glance) services hung if they used an NFS mount.

With this update, before the Leapp upgrade, director unmounts /var/lib/nova/instances, /var/lib/glance/images, and any Image Service staging area that you define with the GlanceNodeStagingUri parameter.

This update fixes a GRUB parameter naming convention that led to unpredictable behaviors on compute nodes during leapp upgrades.

Previously, the presence of the obsolete "TRIPELO" prefix on GRUB parameters caused problems.

The file /etc/default/grub has been updated with GRUB for the tripleo kernel args parameter so that leapp can upgrade it correctly. This is done by adding "upgrade_tasks" to the service "OS::TripleO::Services::BootParams", which is a new service added to all roles in the roles_data.yaml file.

This update fixes a problem that caused baremetal nodes to become non-responsive during Leapp upgrades.

Previously, Leapp did not process transient interfaces like SR-IOV virtual functions (VF) during migration. As a result, Leapp did not find the VF interfaces during the upgrade, and nodes entered an unrecoverable state.

Now the service "OS::TripleO::Services::NeutronSriovAgent" sets the physical function (PF) to remove all VFs, and migrates workloads before the upgrade. After the successful Leapp upgrade, os-net-config runs again with the "--no-activate" flag to re-establish the VFs.

In this release, you can use SR-IOV in an ML2/OVN deployment with native OVN DHCP. SR-IOV in an ML2/OVN deployment no longer requires the Networking service (neutron) DHCP agent.

When virtual machines boot on hypervisors that support SR-IOV NICs, the OVN controllers on the controller or network nodes can reply to the DHCP, internal DNS, and IPv6 router solicitation requests from the virtual machine.

This feature was available as a technology preview in RHOSP 16.1.0. Now it is a supported feature.

The following limitations apply to the feature in this release:

In the first maintenance release of Red Hat OpenStack Platform 16.1 there is support for routed provider networks using the ML2/OVS and SR-IOV mechanism drivers.

You can use a routed provider network to enable a single provider network to represent multiple layer 2 networks (broadcast domains) or segments so that the operator can present only one network to users. This is a common network type in edge DCN and spine-leaf routed data center deployments.

For more information, see https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html-single/networking_guide/index#deploy-routed-prov-networks_rhosp-network.

Currently, you cannot scale down or delete compute nodes if Red Hat OpenStack Platform is deployed with TLS-e using tripleo-ipa. This is because the cleanup role, traditionally delegated to the undercloud as localhost, is now being invoked from the mistral container.

For more information, see https://access.redhat.com/solutions/5336241

A Leapp issue causes failure of fast forward upgrades from Red Hat OpenStack (RHOSP) platform 13 to RHOSP 16.1.

A Leapp upgrade from RHEL 7 to RHEL 8 removes all older RHOSP packages and performs an operating system upgrade and reboot. Because Leapp installs os-net-config package at the "overcloud upgrade run" stage, os-net-config-sriov executable is not available for sriov_config serivce to configure virtual functions (VF) and switchdev mode after reboot. As a result, VFs are not configured and switchdevmode is not applied on the physical function (PF) interfaces.

As a workaround, manually create the VFs, apply switchdevmode to the VF interface, and restart the VF interface.

This update includes the following bug fix patches related to fully qualified domain names (FQDN).

Inadequate timeout values can cause an overcloud deployment to fail after four hours. To prevent these timeout failures, set the following undercloud and overcloud timeout parameters:

Before this update, the Block Storage service (cinder) assigned the default volume type in a volume create request, ignoring alternative methods of specifying the volume type.

With this update, the Block Storage service performs as expected:

This update makes it possible to run the Brocade FCZM driver in RHOSP 16.

The Brocade FCZM vendor chose not to update the driver for Python 3, and discontinued support of the driver past the Train release of OpenStack [1]. Red Hat OpenStack (RHOSP) 16 uses Python 3.6.

The upstream Cinder community assumed the maintenance of the Brocade FCZM driver on a best-effort basis, and the bugs that prevented the Brocade FCZM from running in a Python 3 environment (and hence in RHOSP 16) have been fixed.

[1] https://docs.broadcom.com/doc/12397527

This update increases the speed of stack updates in certain cases.

Previously, stack update performance was degraded when the Ansible --limit option was not passed to ceph-ansible. During a stack update, ceph-ansible sometimes made idempotent updates on nodes even if the --limit argument was used.

Now director intercepts the Ansible --limit option and passes it to the ceph-ansible excecution. The --limit option passed to commands starting with 'openstack overcloud' deploy is passed to the ceph-ansible execution to reduce the time required for stack updates.

Before this update, to successfully run a leapp upgrade during the Framework for Upgrades upgrade (FFU) from RHOSP 13 to RHOSP 16.1, the node where the Red Hat Enterprise Linux upgrade was occurring had to have the PermitRootLogin field defined in the ssh config file (/etc/ssh/sshd_config).

With this update, the Orchestration service (heat) no longer requires you to modify /etc/ssh/sshd_config with the PermitRootLogin field.

This update fixes a problem that caused volume attachments to fail on a VxFlexOS cinder backend.

Previously, attempts to attach a volume on a VxFlexOS cinder backend failed because the cinder driver for the VxFlexOS back end did not include all of the information required to connect to the volume.

The VxFlexOS cinder driver has been updated to include all the information required in order to connect to a volume. The attachments now work correctly.

This update fixes an incompatibility that caused VxFlex volume detachment attempts to fail.

A recent change ix n VxFlex cinder volume credentialing methods was not backward compatible with pre-existing volume attachments. If a VxFlex volume attachment was made before the credentialing method change, attempts to detach the volume failed.

Now the detachments do not fail.

This update modifies get_device_info to use lsscsi to get [H:C:T:L] values, making it possible to support more than 255 logical unit numbers (LUNs) and host logical unit (HLU) ID values.

Previously, get_device_info used sg_scan to get these values, with a limit of 255.

You can get two device types with get_device_info:

This update fixes a bug that prevented the distributed compute nodes (DCN) compute servcie from accessing the glance service.

Previously, distributed compute nodes were configured with a glance endpoint URI that specified an IP address, even when deployed with internal transport layer security (TLS). Because TLS requires the endpoint URI to specify a fully qualified domain name (FQDN), the compute service could not access the glance service.

Now, when deployed with internal TLS, DCN services are configured with glance endpoint URI that specifies a FQDN, and the DCN compute service can access the glance service.

This update includes the following bug fix patches related to fully qualified domain names (FQDN).

Inadequate timeout values can cause an overcloud deployment to fail after four hours. To prevent these timeout failures, set the following undercloud and overcloud timeout parameters:

Before this update, the Block Storage service (cinder) assigned the default volume type in a volume create request, ignoring alternative methods of specifying the volume type.

With this update, the Block Storage service performs as expected:

This update makes it possible to run the Brocade FCZM driver in RHOSP 16.

The Brocade FCZM vendor chose not to update the driver for Python 3, and discontinued support of the driver past the Train release of OpenStack [1]. Red Hat OpenStack (RHOSP) 16 uses Python 3.6.

The upstream Cinder community assumed the maintenance of the Brocade FCZM driver on a best-effort basis, and the bugs that prevented the Brocade FCZM from running in a Python 3 environment (and hence in RHOSP 16) have been fixed.

[1] https://docs.broadcom.com/doc/12397527

This update increases the speed of stack updates in certain cases.

Previously, stack update performance was degraded when the Ansible --limit option was not passed to ceph-ansible. During a stack update, ceph-ansible sometimes made idempotent updates on nodes even if the --limit argument was used.

Now director intercepts the Ansible --limit option and passes it to the ceph-ansible excecution. The --limit option passed to commands starting with 'openstack overcloud' deploy is passed to the ceph-ansible execution to reduce the time required for stack updates.

Before this update, to successfully run a leapp upgrade during the Framework for Upgrades upgrade (FFU) from RHOSP 13 to RHOSP 16.1, the node where the Red Hat Enterprise Linux upgrade was occurring had to have the PermitRootLogin field defined in the ssh config file (/etc/ssh/sshd_config).

With this update, the Orchestration service (heat) no longer requires you to modify /etc/ssh/sshd_config with the PermitRootLogin field.

This update fixes a problem that caused volume attachments to fail on a VxFlexOS cinder backend.

Previously, attempts to attach a volume on a VxFlexOS cinder backend failed because the cinder driver for the VxFlexOS back end did not include all of the information required to connect to the volume.

The VxFlexOS cinder driver has been updated to include all the information required in order to connect to a volume. The attachments now work correctly.

This update fixes an incompatibility that caused VxFlex volume detachment attempts to fail.

A recent change ix n VxFlex cinder volume credentialing methods was not backward compatible with pre-existing volume attachments. If a VxFlex volume attachment was made before the credentialing method change, attempts to detach the volume failed.

Now the detachments do not fail.

This update modifies get_device_info to use lsscsi to get [H:C:T:L] values, making it possible to support more than 255 logical unit numbers (LUNs) and host logical unit (HLU) ID values.

Previously, get_device_info used sg_scan to get these values, with a limit of 255.

You can get two device types with get_device_info:

This update fixes a bug that prevented the distributed compute nodes (DCN) compute servcie from accessing the glance service.

Previously, distributed compute nodes were configured with a glance endpoint URI that specified an IP address, even when deployed with internal transport layer security (TLS). Because TLS requires the endpoint URI to specify a fully qualified domain name (FQDN), the compute service could not access the glance service.

Now, when deployed with internal TLS, DCN services are configured with glance endpoint URI that specifies a FQDN, and the DCN compute service can access the glance service.

This update introduces support for encrypted images with keys managed by the Key Manager service (barbican).

For some secure workflows in which at-rest data must remain encrypted, you can upload carefully prepared encrypted images into the Image service (glance) for consumption by the Block Storage service (cinder).

This update adds support for encrypted volumes and images on distributed compute nodes (DCN).

DCN nodes can now access the Key Manager service (barbican) running in the central control plane.

For example:

$ openstack overcloud roles generate DistributedComputeHCI DistributedComputeHCIScaleOut -o ~/dcn0/roles_data.yaml

Use the appropriate path to the roles data file.

RHOSP 16.1.2 introduces a technology preview of the AMD EPYC 2 (Rome) platform with the UEFI setting NPS (Numa Per Socket)` set to 1.

Other values of NPS (2 or 4) are used in DPDK benchmarks to reach the platform peak performances, without OpenStack, on bare metal.

Red Hat continues to evaluate the operational trade-off of NPS=2 or NPS=4 with OpenStack. This configuration exposes multiple Numa nodes per socket.

Red Hat OpenStack Platform 16.1.2 introduces a technology preview of the AMD EPYC 2 (Rome) platform with the UEFI setting NPS (Numa Per Socket) set to 1.

Other values of NPS (2 or 4) are used in DPDK benchmarks to reach the platform peak performances, without OpenStack, on bare metal.

Red Hat continues to evaluate the operational trade-off of NPS=2 or NPS=4 with OpenStack. This configuration exposes multiple Numa nodes per socket.

Red Hat OpenStack Platform 16.1.2 introduces a technology preview of OVN and OVS-DPDK colocated with SR-IOV on the same hypervisor.

For related issues, see:

https://bugzilla.redhat.com/show_bug.cgi?id=1575512 and

https://bugzilla.redhat.com/show_bug.cgi?id=1575512

Red Hat OpenStack Platform 16.1.2 introduces a technology preview of OVN with OVS TC Flower-based offloads.

Note that VXLAN is not supported by OVN for regular inter-chassis communication. Thus, VXLAN with Hardware offload using OVN is not supported. See https://bugzilla.redhat.com/show_bug.cgi?id=1881704.

Red Hat OpenStack Platform 16.1 includes the following PowerMax Driver updates:

Feature updates:

In this update, the Red Hat OpenStack Platform (RHOSP) Orchestration service (heat) now enables you to deploy multiple Dell EMC XtremIO back ends with any combination of storage protocols for the Block Storage service (cinder).

A new heat parameter, CinderXtremioStorageProtocol, now enables you to choose between Fibre Channel (FC) or iSCSI storage protocols.

A new heat template enables you to deploy more than one XtremIO back end.

Previously, RHOSP director only supported one iSCSI back end for the Block Storage service. (The legacy iSCSI-only heat template will be deprecated in a future RHOSP release).

PowerMax configuration options have changed after Red Hat OpenStack Platform 10 (newton). This update includes the latest PowerMax configuration options and supports both iSCSI and FC protocols.

The CinderPowermaxBackend parameter also supports multiple back ends. CinderPowermaxBackendName supports a list of back ends, and you can use the new CinderPowermaxMultiConfig parameter to specify parameter values for each back end. For example syntax, see environments/cinder-dellemc-powermax-config.yaml.

Transmission of jumbo UDP frames on ML2/OVN routers depends on a kernel release that is not yet avaialbe.

After receiving a jumbo UDP frame that exceeds the maximum transmission unit of the external network, ML2/OVN routers can return ICMP "fragmentation needed" packets back to the sending VM, where the sending application can break the payload into smaller packets. To determine the packet size, this feature depends on discovery of MTU limits along the south-to-north path.

South-to-north path MTU discovery requires kernel-4.18.0-193.20.1.el8_2, which is scheduled for availability in a future release. To track availability of the kernel version, see https://bugzilla.redhat.com/show_bug.cgi?id=1860169.

When you enable Load-balancing service instance (amphora) log offloading, both the administrative logs and the tenant logs are written to the same file (octavia-amphora.log). This is a known issue caused by an incorrect default value for the Orchestration service (heat) parameter, OctaviaTenantLogFacility. As a workaround, perform the following steps:

Set OctaviaTenantLogFacility to zero (0) in a custom environment file and run the openstack overcloud deploy command:

parameter_defaults:
    OctaviaLogOffload: true
    OctaviaTenantLogFacility: 0
    ...

For more information, see Modifying the overcloud environment

A known issue causes the migration of Ceph OSDs from FileStore to BlueStore to fail. In use cases where the osd_objectstore parameter was not set explicitly when you deployed Red Hat OpenStack Platform 13 with Red Hat Ceph Storage 3, the migration exits without converting any OSDs and falsely reports that the OSDs are already using BlueStore. For more information about the known issue, see https://bugzilla.redhat.com/show_bug.cgi?id=1875777

As a workaround, perform the following steps:

The keepalived instance in the Red Hat OpenStack Platform Load-balancing service (octavia) instance (amphora) can abnormally terminate and interrupt UDP traffic. The cause of this issue is that the timeout value for the UDP health monitor is too small.

Workaround: specify a new timeout value that is greater than two seconds: $ openstack loadbalancer healthmonitor set --timeout 3 <heath_monitor_id>

For more information, search for "loadbalancer healthmonitor" in the Command Line Interface Reference.

Enabling the realtime-virtual-host tuned profile inside guest virtual machines degrades throughput and displays non-deterministic performance. ovs-dpdk PMDs are pinned incorrectly to housekeeping CPUs.

As a workaround, use the cpu-partitioning tuned profile inside guest virtual machines, write a post-deployment script to update the tuned.conf file, and reboot the node:

ps_blacklist=ksoftirqd.*;rcuc.*;rcub.*;ktimersoftd.*;.*pmd.*;.*PMD.*;^DPDK;.*qemu-kvm.*

Currently, you cannot scale down or delete compute nodes if Red Hat OpenStack Platform is deployed with TLS Everywhere using tripleo-ipa. This is because the cleanup role, traditionally delegated to the undercloud as localhost, is now being invoked from the Workflow service (mistral) container.

For more information, see https://access.redhat.com/solutions/5336241

This update supports the creation of volumes with tiering policy. There are four supported values:

This enhancement adds a new driver for the Dell EMC PowerStore to support Block Storage service back end servers. The new driver supports the FC and iSCSI protocols, and includes these features:

Currently, LVM filter is not set unless at least one device is listed in the LVMFilterAllowlist parameter.

Workaround: Set the LVMFilterAllowdisk parameter to contain at least one device, for example, the root disk. The LVM filter is set in /etc/lvm/lvm.conf.

There is a known issue with the Object Storage service (swift). If you use pre-deployed nodes, you might encounter the following error message in /var/log/containers/stdouts/swift_rsync.log:

"failed to create pid file /var/run/rsyncd.pid: File exists"

Workaround: Enter the following command on all pre-deployed Controller nodes:

for d in $(podman inspect swift_rsync | jq '.[].GraphDriver.Data.UpperDir') /var/lib/config-data/puppet-generated/swift; do sed -i -e '/pid file/d' $d/etc/rsyncd.conf; done

The Ceph Dashboard currently does not work with the TLS Everywhere framework because the dashboard_protocol parameter was incorrectly omitted from the heat template. As a result, back ends fail to appear when HAproxy is started.

As a temporary solution, create a new environment file that contains the dashboard_protocol parameter and include the environment file in your overcloud deployment with the -e option:

parameter_defaults:
  CephAnsibleExtraConfig:
    dashboard_protocol: 'https'

This solution introduces a ceph-ansible bug. For more information, see https://bugzilla.redhat.com/show_bug.cgi?id=1860815.

Currently, a change in OSP13 puppet module kmod has resulted in the wrong module setting for systemd-modules-load.service. This is not an issue in OSP13 but results in failure during deployment in fast forward upgrade on OSP16.1.

Workaround: Enter the following command:

rm -f /etc/modules-load.d/nf_conntrack_proto_sctp.conf

Replacement of an overcloud Controller might cause swift rings to become inconsistent across nodes. This results in decreased availability of Object Storage service.

Workaround: Log in to the previously existing Controller node using SSH, deploy the updated rings, and restart the Object Storage containers:

Before this update, if a user configured the ContainerImagePrepare parameter to use a custom tag, such as 'tag: "latest"' or 'tag: "16.1"', instead of the standard 'tag_from_label: "{version}-{release}"', the containers did not update to the latest container images.

With this update, the container images are always fetched anytime a user runs a deployment action, including updates, and the image ID is checked against the running container to see if it needs to be rebuilt to consume the latest image. Containers are now always refreshed during deployment actions and restarted if they are updated.

Before this update, when deployed at an edge site the Image (glance) service was not configured to access the Key Manager (barbican) service running on the central site’s control plane. This resulted in the Image services running on edge sites being unable to access encryption keys stored in the Key Manager service.

With this update, Image services running on edge sites are now configured to access the encryption keys stored in the Key Manager service.

Before this update, in-place upgrades from Red Hat OpenStack Platform 13 to 16.1 in a TLS everywhere environment used an incorrect rabbitmq password for the novajoin container. This caused the novajoin container on the undercloud to function incorrectly, which caused any overcloud node that ran an upgrade to fail with the following error:

2020-11-24 20:01:31.569 7 ERROR join   File "/usr/lib/python3.6/site-packages/amqp/connection.py", line 639, in _on_close
2020-11-24 20:01:31.569 7 ERROR join     (class_id, method_id), ConnectionError)
2020-11-24 20:01:31.569 7 ERROR join amqp.exceptions.AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For detail see the broker logfile.

With this update, the upgrade from RHOSP 13 to 16.1 uses the correct rabbitmq password in a TLS everywhere environment so that the framework for upgrades can complete successfully.

Before this update, live migration failed when upgrading a TLS everywhere environment with local ephemeral storage and UseTLSTransportForNbd set to "False". This occurred because the default value of the UseTLSTransportForNbd configuration had changed from "False" in RHOSP 13 to "True" in RHOSP 16.x, which resulted in the correct certifications not being included in the QEMU process containers.

With this update, director checks the configuration of the previously deployed environment for global_config_settings and uses it to ensure that the UseTLSTransportForNbd state stays the same in the upgrade as on previous deployment. If global_config_settings exists in the configuration file, then director checks the configuration of the use_tls_for_nbd key. If global_config_settings does not exist, director evaluates the hieradata key nova::compute::libvirt::qemu::nbd_tls. Keeping the UseTLSTransportForNbd state the same in the upgraded deployment as on previous deployment ensures that live migration works.

Before this update, a rebase in python-network-runner from 0.1.7 to 0.2.2 in OSP 16.1.3 caused ML2 Networking using Ansible to no longer function.

With this update, python-networking-ansible is reverted to 0.1.7, and Ansible networking returns to a functioning state.

For more information, see https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/bare_metal_provisioning/ml2-networking-ansible.

Before this update, the Block Storage (cinder) service would always assign newly created volumes with the default volume type, even when the volume was created from another source, such as an image, snapshot or another volume. This resulted in volumes created from another source having a different volume type from the volume type of the source.

With this update, the default volume type is assigned only after determining whether it should be assigned based on the volume type of the source. The volume type of volumes created from another source now match the volume type of the source.

With this enhancement, you can configure NVDIMM Compute nodes to provide persistent memory for instances. Using this feature, you can make the PMEM available to instances as virtual PMEM (vPMEM) by creating and configuring PMEM namespaces on Compute nodes that have NVDIMM hardware. Cloud users can then create instances that request vPMEM when they need the instance content to be retained after it is shut down.

Note: Due to the availability of daxio package only for x86_64 architecture, this feature is supported only on x86_64 Compute nodes.

With this enhancement, you can manage vPMEM with two new parameters NovaPMEMMappings and NovaPMEMNamespaces.

This enhancement adds support for using Open Virtual Network (OVN) with Network Functions Virtualization infrastructure (NFVi) for new deployments. This includes support for the following features:

Note: Migration from ML2/OVS to ML2/OVN is not yet supported for NFV deployments.

This enhancement adds support for vlan transparency in the ML2/OVN mechanism driver with vlan and geneve network type drivers.

With vlan transparency, you can manage vlan tags by using instances on Networking (neutron) service networks. You can create vlan interfaces on an instance and use any vlan tag without affecting other networks. The Networking service is not aware of these vlan tags.

Notes:

The virt-admin tool is now available for you to use to capture logs for reporting RHOSP bugs. This tool is useful for troubleshooting all libvirt and QEMU problems, as the logs provide the communications between libvirt and QEMU on the Compute nodes. You can use virt-admin to set the libvirt and QEMU debug log filters dynamically, without having to restart the nova_libvirt container.

Perform the following steps to enable libvirt and QEMU log filters on a Compute node:

There is currently a known issue with the mechanism that ensures the subscribed environments have the right DNF module stream set. The Advanced Virtualization repository is not always available in the subscription that the Ceph nodes use, which causes the upgrade or update of a Ceph node to fail when you try to enable virt:8.2. For more information on the known issue, see https://bugzilla.redhat.com/show_bug.cgi?id=1923887.

Workaround:

Override the DnfStreams parameter in the upgrade or update environment file to prevent the Ceph upgrade from failing:

parameter_defaults:
  ...
  DnfStreams: [{'module':'container-tools', 'stream':'2.0'}]

Systems that use UEFI boot and a UEFI bootloader in OSP13 might run into an UEFI issue that results in:

If your systems use UEFI, contact Red Hat Technical Support. For more information, see the Red Hat Knowledgebase solution FFU 13 to 16.1: Leapp fails to update the kernel on UEFI based systems and /etc/fstab does not contain the EFI partition.

There are currently known issues related to [workarounds]/disable_native_luksv1 and [workarounds]/rbd_volume_local_attach configuration options. These options are provided only as a temporary workaround for known performance regressions within libgcrypt and librdb. These workaround options will be removed once the regressions are resolved in the underlying RHEL release used by RHOSP.

There are caveats associated with using either of these workaround options. Failure to adhere to these caveats can cause issues with RDB encrypted volumes. The caveats are as follows:

If you use a Red Hat Ceph Storage subscription and have configured director to use the overcloud-minimal image for Red Hat Ceph Storage nodes, the upgrade of the operating system for Red Hat Ceph Storage nodes might fail due to a Leapp limitation.

To avoid this issue, after the system_upgrade run step, you must log in to the Red Hat Ceph Storage node to unset the RHEL minor release version, update to the latest available RHEL minor release version, and reboot the node.

If you use Red Hat Satellite Server to host RPM content for the Leapp upgrade, you must add the following 8.2 repositories to the Content View that you use:

For more information, see https://bugzilla.redhat.com/show_bug.cgi?id=1936419

There is currently a known issue in ceph-ansible that prevents an ansible playbook from finishing successfully when it is used to deploy or update a Ceph Ganesha container that is configured to use an external, unmanaged Ceph cluster.

This issue causes deployments, minor updates, or major upgrades of overclouds that use Ceph Ganesha with the Shared File Systems service (manila), and that are configured to use an external Ceph cluster, to fail.

Workaround:

If you conduct an upgrade or minor update to the overcloud, use ceph-ansible version 4.0.49.1 or later if your environment is configured to use Ceph Ganesha with the Shared File Systems service and an external Ceph cluster. Under these conditions, do not attempt the upgrade with a version of ceph-ansible earlier than 4.0.49.1 installed on the undercloud.

When an instance is created, the Compute (nova) service sanitizes the instance display name to generate a valid host name when DNS integration is enabled in the Networking (neutron) service.

Before this update, the sanitization did not replace periods ('.') in instance names, for example, 'rhel-8.4'. This could result in display names being recognized as Fully Qualified Domain Names (FQDNs) which produced invalid host names. When instance names contained periods and DNS integration was enabled in the Networking service, the Networking service rejected the invalid host name, which resulted in a failure to create the instance and a HTTP 500 server error from the Compute service.

With this update, periods are now replaced by hyphens in instance names to prevent host names being parsed as FQDNs. You can continue to use free-form strings for instance display names.

This update fixes a bug that caused failure of validations before openstack undercloud upgrade in some cases. Before this upgrade, a lack of permissions needed to access the requested logging directory sometimes resulted in the following failures:

This update fixes a configuration problem that caused Leapp upgrades to stop and fail while executing on a CephStorage node.

Previously, CephStorage nodes were incorrectly configured to consume OpenStack highavailability, advanced-virt, and fast-datapath repos during Leapp upgrades.

Now UpgradeLeappCommand options is configurable on a per-node basis, and uses the correct default for CephStorage nodes, and Leapp upgrades succeed for CephStorage nodes.

In prior releases, the SolidFire driver created a duplicate volume whenever it retried an API request. This led to unexpected behavior due to the accumulation of unused volumes.

With this update, the Block Storage service (cinder) checks for existing volume names before it creates a volume. When Block Storage service detects a read timeout, it immediately checks for volume creation to prevent invalid API calls. This update also adds the sf_volume_create_timeout option for the SolidFire driver so that you can set an appropriate timeout value for your environment.

This update fixes an issue that caused some API calls, such as create snapshot, to fail with an xNotPrimary error during workload re-balancing operations.

When SolidFire is under heavy load or being upgraded, the SolidFire cluster might re-balance cluster workload by automatically moving connections from primary to secondary nodes. Previously, some API calls failed with an xNotPrimary error during these workload balance operations and were not retried.

This update fixes the issue by adding the xNotPrimary exception to the SolidFire driver list of retryable exceptions.

In previous releases, in Red Hat OpenStack Platform (RHOSP) deployments that use the Dell EMC XtremIO driver, attach volume operations waited for a timeout if iSCSI or FC targets were not connected to a RHOSP host. This caused attach volume operations to fail.

This release adds port filtering support for the Dell EMC XtremIO driver to allow iSCSI or FC ports that are not in use to be ignored.

Before this update, the Shared File Systems service (manila) dashboard had dynamic form elements whose names could potentially cause the forms to become unresponsive. This meant that the creation of share groups, share networks, and shares within share networks did not function.

With this update, dynamic elements whose names might be problematic are encoded. Which means that creation of share groups, share networks, and shares within share networks functions normally.

In previous releases, if Dell EMC PowerStore ports were configured for multiple purposes, such as iSCSI, replication, incorrect REST filtering caused the cinder driver to report that no accessible iSCSI targets were found.

This release fixes the Dell EMC PowerStore REST filter functionality.

Before this update, a failure occurred when users wanted to delete the DEFAULT volume type.

With this update, you can delete the DEFAULT volume type when it is not set as the value of the default_volume_type parameter in the cinder.conf file. The default value of the default_volume_type parameter is DEFAULT so you must set it to an appropriate volume type, for example 'tripleo', so that you can delete the DEFAULT volume type.

Before this update, during a tripleo validation on an OpenStack component, the following exception error occurred:

Unhandled exception during validation run.

This error occurred because a variable in the code was referenced, but never assigned.

With this update, this problem has been fixed and validations run without this error.

Before this update, there were unhandled exceptions during connection to iSCSI portals. For example, failures in iscsiadm -m session. This occurred because the _connect_vol threads can abort unexpectedly in some failure patterns, and this abort causes a hang in subsequent steps while waiting for results from _connct_vol threads.

With this update, any exceptions during connection to iSCSI portals are handled in the _connect_vol method correctly and avoids any unexpected abort without updating thread results.

Before this update, changes to KernelArgs parameters caused errors in the Red Hat OpenStack Platform (RHOSP) fast forward upgrade (FFU) process for version 13 to version 16:

Before this update, creating a volume from a snapshot of an encrypted volume could result in an unusable volume. When the destination volume is the same size as the source volume, creating an encrypted volume from a snapshot of an encrypted volume truncated the data in the new volume, which caused a size discrepancy.

With this update, the RBD back end accounts for the encryption header and does not truncate the data so that creating a volume from a snapshot of an encrypted volume does not cause the error.

With this enhancement, you can use policy-based routing for Red Hat OpenStack Platform nodes to configure multiple route tables and routing rules with os-net-config.

Policy-based routing uses route tables where, on a host with multiple links, you can send traffic through a particular interface depending on the source address. You can also define route rules for each interface.

The logic to detect the hypervisor hostname has been fixed and now returns the result consistent with libvirt driver in the Compute service (nova). With this fix, you no longer need to specify the resource_provider_hypervisors option when you use the guaranteed minimum bandwidth QoS feature.

With this update, a new option, resource_provider_default_hypervisor, has been added to the Modular Layer 2 with the Open Virtual Network mechanism driver (ML2/OVN) to replace the default hypervisor name. The option locates the root resource provider without giving a complete list of interfaces or bridges in the resource_provider_hypervisors option in case it has to be customized by the user. This new option is located in the [ovs] ini-section for the ovs-agent, and in the [sriov_nic] ini-section for the sriov-agent.

This enhancement adds the new CinderRpcResponseTimeout and CinderApiWsgiTimeout parameters to support tuning RPC and API WSGI timeouts in the Block Storage service (cinder). Default timeout values might not be adequate for large deployments and in situations where transactions might be delayed due to system load.

It is now possible to tune the RPC and API WSGI timeouts to prevent transactions prematurely timing out.

With this update, the tripleo validator command now accepts variables and environment variables in a key-value pair format. In past releases, only JSON dictionaries allowed environment variables.

openstack tripleo validator run \
[--extra-vars key1=<val1>[,key2=val2 --extra-vars key3=<val3>] \
| --extra-vars-file EXTRA_VARS_FILE] \
[--extra-env-vars key1=<val1>[,key2=val2 --extra-env-vars key3=<val3>]]
(--validation <validation_id>[,<validation_id>,...] | --group <group>[,<group>,...])

$ openstack tripleo validator run --validation check-cpu,check-ram --extra-vars minimal_ram_gb=8 --extra-vars minimal_cpu_count=2

For the complete list of supported options, run:

$ openstack tripleo validator run --help

Currently, there is a known issue where it is not possible to simulate certain real-life scenarios when the MAC-IP addresses of a port are unknown. The RHOSP Networking service (neutron) directly specifies the MAC-IP of a port even if DHCP or security groups are not configured.

Workaround: upgrade to RHOSP 16.1.7 and install ML2/OVN v21.03. If DHCP and port security are disabled, then the addresses field of a port does not include its MAC-IP address pairs, and ML2/OVN can use the MAC learning capabilities to send traffic only to the desired port.

Before this update, in DCN and HCI deployments with an IPv6 internal API network, the Block Storage service (cinder) and etcd services were configured with malformed etcd URIs, and the Block Storage service and etcd services failed on startup.

With this update, the IPv6 addresses in the etcd URI are correct and the Block Storage service and etcd services start successfully.

With this update, the telemetry healthchecks have been made more robust and the way the healthchecks are parsed has been simplified.

To get verbose mode when you run the healthcheck directly, run the command sudo podman -u root -e "HEALTHCHECK_DEBUG=1" <container> /openstack/healthcheck

Before this update, the OpenStack Storage service (cinder) GPFS SpectrumScale driver would not correctly detect whether the storage backend supported copy-on-write (COW) mode. As a result, the driver would disable COW features, such as the ability to rapidly create volumes from an image. This can cause some instances to time out when booting multiple instances simultaneously from an image.

With this update, the GPFS SpectrumScale driver properly detects COW support for storage backends.

Before this update, when creating a snapshot with PowerMaxOS 5978.711, REST experienced a payload response change and caused the device label to modify its format. The underlying data from the solutions enabler changed, and no longer contained a colon character (:). This, in turn, would cause an IndexError exception in the PowerMax Driver:

IndexError: list index out of range

With this update, the problem is resolved in PowerMaxOS 5978.711 and later.

Enable the experimental rsyslog reopenOnTruncate to ensure that rsyslog immediately recognizes when a logrotation happens on a file. The setting affects every service configured to work with rsyslog.

With rsyslog reopenOnTruncate disabled, rsyslog waits for a log file to fill to its original capacity before consuming any additional logs.

This enhancement prepares your environment for update of the metrics_qdr service to a newer AMQ Interconnect release, which requires import of the CA certificate contents from the Service Telemetry Framework (STF) deployment. Changes are not yet required by administrators when deploying or updating Red Hat OpenStack Service Platform (RHOSP) as the metrics_qdr service has not yet been updated. This functionality is in preparation of the metrics_qdr service update in a future release.

The following procedure will be required once https://bugzilla.redhat.com/show_bug.cgi?id=1949169 has shipped.

This update corrects this problem by providing a new Orchestration service (heat) parameter, MetricsQdrSSLProfiles.

To obtain a Red Hat OpenShift TLS certificate, run the following commands:

$ oc get secrets
$ oc get secret/default-interconnect-selfsigned -o jsonpath='{.data.ca\.crt}' | base64 -d

Add the MetricsQdrSSLProfiles parameter with the contents of your Red Hat OpenShift TLS certificate to a custom environment file:

MetricsQdrSSLProfiles:
    -   name: sslProfile
        caCertFileContent: |
           -----BEGIN CERTIFICATE-----
           ...
           TOpbgNlPcz0sIoNK3Be0jUcYHVMPKGMR2kk=
           -----END CERTIFICATE-----

Then, redeploy your overcloud with the openstack overcloud deploy command.

With this update you can set QoS maximum bandwidth limit, egress direction rules on hardware-offloaded ports in a ML2/OVS deployment. Set the policy using the normal QoS policy/rules methods.

Note that the backend uses ip link commands to enforce the policy instead of the normal OVS QoS engine, because the OVS meter action cannot be offloaded. See meter action is not offloaded.

A known issue causes Neutron to fail to start after an update to RHOSP 16.1.8. If you then get Neutron to start, the OVN databases are unstable.

A fix is planned for RHOSP 16.1.9. Red Hat recommends that you wait to update directly to 16.1.9 if possible.

A hot fix is available for 16.1.8. If you have an urgent need to update to RHOSP 16.1.8, contact Red Hat Global Support Services to see if your environment is compatible with the hot fix.

To track this issue, see https://bugzilla.redhat.com/show_bug.cgi?id=2125824.

Before this release, the collectd container would fail to start on compute nodes because a dpdk-telemetry collectd configuration file was being automatically created despite there being no dpdk-telemetry plugin installed.

As of this release, dpdk_telemetry configuration files have been removed from the the collectd container.

This update in RHOSP 16.1.9 fixes a bug that causes the Networking service (neutron) to fail to start after an update to RHOSP 16.1.8 and also causes OVN database instability after updates to RHOSP 16.1.8.

Instead of updating to RHOSP 16.1.8, update directly to RHOSP 16.1.9.

Before this update, a do_sync_check operation could result in the incorrect deletion of non-temporary snapshots from a volume because there was no check for non-temporary snapshots deletion during the do_sync_check operation. With this update, there is a check to determine if a snapshot must be deleted. The do_sync_check operation does not perform unnecessary non-temporary snapshot deletions.

Before this update, there was a case mismatch in the conditional while checking if a storage group was a child of a parent storage group. While modifying the storage group, errors indicated that the parent storage group already contained the child storage group. With this update, the patterns used in the conditional are not case-sensitive and you can modify the storage group successfully.

Before this update, a Telemetry service (ceilometer) user had insufficient privileges to poll objects from the Object Storage service (swift). The Object Storage service client did not allow the Telemetry service user to fetch object details. With this update, the Telemetry service user is associated with the ResellerAdmin role.

Execute the following command to workaround this issue manually:

$ openstack role add --user ceilometer --project service ResellerAdmin

The associated Telemetry service user can poll Object Storage service object metrics successfully.

With this update, director supports specifying overrides for NVSv4 ID mapping when using a CephFS-NFS back end with the Shared File Systems service (manila). Ceph-NFS with the Shared File Systems service only allows client access through NFSv4.1+. With NFSv4.1, usernames and group names are sent over the wire and translated by both the server and the client. Deployers might want to customize their domain settings to better represent organization users who can access Shared File Systems service shares from multiple clients. Director supports customizing NFS ID mapping settings through these parameters:

You can use Role-based Access Control (RBAC) to share security groups between projects. However, you cannot use the --security-group argument to assign an RBAC-shared security group when you launch an instance. If you try to assign the RBAC-shared security group by using the --security-group argument in the openstack server create command, the Compute service (nova) does not find the security group and fails to create the instance. This is because the Compute service does not check for security groups that are shared through RBAC. It only checks if the security group specified with the --security-group argument exists in the project that you created the instance in.

Workaround: Create a port and assign the security group to the port. Use the --nic argument in the openstack server create command to specify the port. The Compute service does not try to create the port in the Networking service (neutron), therefore it does not check security groups.

For example:

   $ openstack port create --network net1 \
     --security-group \
     5ba835b7-22b0-4be6-bdbe-e0722d1b5f24 shared-sg-port

   $ openstack server create \
     --image cirros-0.5.1-x86_64-disk  \
     --flavor m1.tiny \
     --port shared-sg-port vm-with-shared-sg

There is currently a known issue when live migrating instances with CPUs that are incompatible with the destination host CPUs. As a workaround, you can skip the Compute service CPU comparison check on the destination host before migrating an instance, because libvirt (QEMU >= 2.9 and libvirt >= 4.4.0) correctly handles the CPU compatibility checks on the destination host during live migration.

Workaround: Before performing instance live migration, add the following configuration in the nova.conf file of each affected Compute node:

[workarounds]
skip_cpu_compare_on_dest = True