Before this update, director did not set the noout flag on Red Hat Ceph Storage OSDs before running a Leapp upgrade. As a result, additional time was required for the the OSDs to rebalance after the upgrade.

With this update, director sets the noout flag before the Leapp upgrade, which accelerates the upgrade process. Director also unsets the noout flag after the Leapp upgrade.

PowerMax configuration options have changed since Newton. This update includes the latest PowerMax configuration options and supports both iSCSI and FC drivers.

The CinderPowermaxBackend parameter also supports multiple back ends. CinderPowermaxBackendName supports a list of back ends, and you can use the new CinderPowermaxMultiConfig parameter to specify parameter values for each back end. For example syntax, see environments/cinder-dellemc-powermax-config.yaml.

Before this update, in DCN + HCI deployments with an IPv6 internal API network, the cinder and etcd services were configured with malformed etcd URIs, and the cinder and etcd services failed on starup.

With this update, the IPv6 addresses in the etcd URI are correct and the cinder and etcd services start successfully.

Before this update, in deployments with an IPv6 internal API network, the Block Storage Service (cinder) and Compute Service (nova) were configured with a malformed glance-api endpoint URI. As a result, cinder and nova services located in a DCN or Edge deployment could not access the Image Service (glance).

With this update, the IPv6 addresses in the glance-api endpoint URI are correct and the cinder and nova services at Edge sites can access the Image Service successfully.

Before this update, the Block Storage service (cinder) assigned the default volume type in a volume create request, ignoring alternative methods of specifying the volume type.

With this update, the Block Storage service performs as expected:

Before this update, there were no retries and no timeout when downloading a final instance image with the direct deploy interface in ironic. As a result, the deployment could fail if the server that hosts the image fails to respond.

With this update, the image download process attempts 2 retries and has a connection timeout of 60 seconds.

A regression was introduced in ipmitool-1.8.18-11 that caused IPMI access to take over 2 minutes for certain BMCs that did not support the "Get Cipher Suites". As a result, introspection could fail and deployments could take much longer than previously.

With this update, ipmitool retries are handled differently, introspection passes, and deployments succeed.

Before this update, the ExtraConfigPre per_node script was not compatible with Python 3. As a result, the overcloud deployment failed at the step TASK [Run deployment NodeSpecificDeployment] with the message SyntaxError: invalid syntax.

With this update, the ExtraConfigPre per_node script is compatible with Python 3 and you can provision custom per_node hieradata.

Before this update, the data structure format that the ceph osd stat -f json command returns changed. As a result, the validation to stop the deployment unless a certain percentage of Red Hat Ceph Storage (RHCS) OSDs are running did not function correctly, and stopped the deployment regardless of how many OSDs were running.

With this update, the new version of openstack-tripleo-validations computes the percentage of running RHCS OSDs correctly and the deployment stops early if a percentage of RHCS OSDs are not running. You can use the parameter CephOsdPercentageMin to customize the percentage of RHCS OSDs that must be running. The default value is 66%. Set this parameter to 0 to disable the validation.

Before this update, the Red Hat Ceph Storage dashboard listener was created in the HA Proxy configuration, even if the dashboard is disabled. As a result, upgrades of OpenStack with Ceph could fail.

With this update, the service definition has been updated to distinguish the Ceph MGR service from the dashboard service so that the dashboard service is not configured if it is not enabled and upgrades are successful.

Before this update, the Leapp upgrade could fail if you had any NFS shares mounted. Specifically, the nodes that run the Compute Service (nova) or the Image Service (glance) services hung if they used an NFS mount.

With this update, before the Leapp upgrade, director unmounts /var/lib/nova/instances, /var/lib/glance/images, and any Image Service staging area that you define with the GlanceNodeStagingUri parameter.

With this enhancement, there are new options in the openstack tripleo container image push command that you can use to provide credentials for the source registry. The new options are --source-username and --source-password.

Before this update, you could not provide credentials when pushing a container image from a source registry that requires authentication. Instead, the only mechanism to push the container was to pull the image manually and push from the local system.

With this enhancement, you can use policy-based routing for Red Hat OpenStack Platform nodes to configure multiple route tables and routing rules with os-net-config.

Policy-based routing uses route tables where, on a host with multiple links, you can send traffic through a particular interface depending on the source address. You can also define route rules for each interface.

With this update, the container_images_file parameter is now a required option in the undercloud.conf file. You must set this parameter before you install the undercloud.

With the recent move to use registry.redhat.io as the container source, you must authenticate when you fetch containers. For the undercloud, the container_images_file is the recommended option to provide the credentials when you perform the installation. Before this update, if this parameter was not set, the deployment failed with authentication errors when trying to fetch containers.

With this enhancement, you can manage vPMEM with two new parameters NovaPMEMMappings and NovaPMEMNamespaces.

Use NovaPMEMMappings to set the nova configuration option pmem_namespaces that reflects mappings between vPMEM and physical PMEM namespaces.

Use NovaPMEMNamespaces to create and manage physical PMEM namespaces that you use as a back end for vPMEM.

In Red Hat OpenStack Platform 16.1 there is a technology preview for routed provider networks with the ML2/OVS mechanism driver. You can use a routed provider network to enable a single provider network to represent multiple layer 2 networks (broadcast domains) or segments so that the operator can present only one network to users. This is a common network type in Edge DCN deployments and Spine-Leaf routed datacenter deployments.

Because Nova scheduler is not segment-aware, you must map each leaf, rack segment, or DCN edge site to a Nova host-aggregate or availability zone. If the deployments require DHCP or the metadata service, you must also define a Nova availability zone for each edge site or segment.

Known Limitations:

For glance image conversion, the glance-direct method is not enabled by default. To enable this feature, set enabled_import_methods to [glance-direct,web-download] or [glance-direct] in the DEFAULT section of the glance-api.conf file.

The Image Service (glance) must have a staging area when you use the glance-direct import method. Set the node_staging_uri option int he DEFAULT section of the glance-api.conf file to file://<absolute-directory-path>. This path must be on a shared file system that is available to all Image Service API nodes.

When you upgrade from Red Hat OpenStack Platform (RHOSP) 13 DCN to RHOSP 16.1 DCN, it is not possible to migrate from the single stack RHOSP 13 deployment into a multi-stack RHOSP 16.1 deployment. The RHOSP 13 stack continues to be managed as a single stack in the Orchestration service (heat) even after you upgrade to RHOSP 16.1.

After you upgrade to RHOSP 16.1, you can deploy new DCN sites as new stacks. For more information, see the multi-stack documentation for RHOSP 16.1 DCN.

Red Hat OpenStack Platform 16.1 includes the following PowerMax Driver updates:

Feature updates:

The sg-bridge container uses the sg-bridge RPM to provide an AMQP1-to-unix socket interface for sg-core. Both components are part of the Service Telemetry Framework.

This is the initial release of the sg-bridge component.

The sg-bridge and sg-core container images provide a new data path for collectd metrics into the Service Telemetry Framework.

The sg-bridge component provides an AMQP1 to unix socket translation to the sg-core, resulting in a 500% performance increase over the legacy Smart Gateway component.

This is the initial release of the sg-bridge and sg-core container image components.

OVN serves DHCP as an openflow controller with ovn-controller directly on Compute nodes. However, SR-IOV instances are attached directly to the network through the VF/PF and so SR-IOV instances cannot receive DHCP responses.

Workaround: Change OS::TripleO::Services::NeutronDhcpAgent to OS::TripleO::Services::NeutronDhcpAgent: deployment/neutron/neutron-dhcp-container-puppet.yaml.

Because of a core OVN bug, virtual machines with floating IP (FIP) addresses cannot route to other networks in an ML2/OVN deployment with distributed virtual routing (DVR) enabled. Core OVN sets a bad next hop when routing SNAT IPv4 traffic from a VM with a floating ip with DVR enabled. Instead of the gateway IP, OVN sets the destination IP. As a result, the router sends an ARP request for an unknown IP instead of routing the request to the gateway.

Workaround: Before you deploy a new overcloud with ML2/OVN, disable DVR by setting NeutronEnableDVR: false in an environment file. If you have ML2/OVN in an existing deployment, complete the following steps:

1) Set enable_distributed_floating_ips to 'False' in the neutron.conf file:

(undercloud) [stack@undercloud-0 ~]$ ansible -i /usr/bin/tripleo-ansible-inventory -m shell -b -a "crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/ml2_conf.ini ovn enable_distributed_floating_ip False" Controller

2) Restart neutron server containers:

(undercloud) [stack@undercloud-0 ~]$ ansible -i /usr/bin/tripleo-ansible-inventory -m shell -b -a "podman restart neutron_api" Controller

3) Centralize all of the FIP traffic through gateway nodes. Run the following command on any overcloud node:

$ export NB=$(sudo ovs-vsctl get open . external_ids:ovn-remote | sed -e 's/\"//g' | sed -e 's/6642/6641/g') $ alias ovn-nbctl='sudo podman exec ovn_controller ovn-nbctl --db=$NB' $ for fip in $(ovn-nbctl --bare --columns _uuid find nat type=dnat_and_snat); do ovn-nbctl clear NAT $fip external_mac; done

When the fix is available in RHOSP 16.1.1, you can re-enable distributed FIP traffic:

1) Set enable_distributed_floating_ips back to 'True' in the neutron.conf file:

(undercloud) [stack@undercloud-0 ~]$ ansible -i /usr/bin/tripleo-ansible-inventory -m shell -b -a "crudini --set /var/lib/config-data/puppet-generated/neutron/etc/neutron/plugins/ml2/ml2_conf.ini ovn enable_distributed_floating_ip True" Controller

2) Restart neutron server containers:

(undercloud) [stack@undercloud-0 ~]$ ansible -i /usr/bin/tripleo-ansible-inventory -m shell -b -a "podman restart neutron_api" Controller

3) Trigger the update in all of the FIPs. Run the following command on any overcloud node:

$ export NB=$(sudo ovs-vsctl get open . external_ids:ovn-remote | sed -e 's/\"//g' | sed -e 's/6642/6641/g') $ alias ovn-nbctl='sudo podman exec ovn_controller ovn-nbctl --db=$NB' $ for i in $(ovn-nbctl --bare --columns logical_port find nat type=dnat_and_snat); do ovn-nbctl set logical_switch_port $i up=false; done

The keepalived instance in the Red Hat OpenStack Platform Load-balancing service (octavia) instance (amphora) can abnormally terminate and interrupt UDP traffic. The cause of this issue is that the timeout value for the UDP health monitor is too small.

Workaround: specify a new timeout value that is greater than two seconds: $ openstack loadbalancer healthmonitor set --timeout 3 <heath_monitor_id>

For more information, search for "loadbalancer healthmonitor" in the Command Line Interface Reference.

There is an incomplete definition for TLS in the Orchestration service (heat) when you update from 16.0 to 16.1, and the update fails.

To prevent this failure, you must set the following parameter and value: InternalTLSCAFile: ''.

There is a known issue when you update from 16.0 to 16.1 with Public TLS or TLS-Everywhere.

The parameter InternalTLSCAFile provides the location of the CA cert bundle for the overcloud instance. Upgrades and updates fail if this parameter is not set correctly. With new deployments, heat sets this parameter correctly, but if you upgrade a deployment that uses old heat templates, then the defaults might not be correct.

Workaround: Set the InternalTLSCAFile parameter to an empty string '' so that the undercloud uses the certificates in the default trust store.

There is a known issue when upgrading from RHOSP 13 to RHOSP 16.1. The value of HostnameFormatDefault has changed from %stackname%-compute-%index% to %stackname%-novacompute-%index%. This change in default value can result in duplicate service entries and have further impacts on operations such as live migration.

Workaround: If you upgrade from RHOSP 13 to RHOSP 16.1, you must override the HostnameFormatDefault value to configure the previous default value to ensure that the previous hostname format is retained. If you upgrade from RHOSP 15 or RHOSP 16.0, no action is required.

The output format of tripleo-ansible-inventory changed in RHOSP 16.1. As a result, the generate-inventory step fails.

Workaround: Create the inventory manually.

There is a known issue where a heat parameter InternalTLSCAFile is used during deployment when the undercloud contacts the external (public) endpoint to create initial resources and projects. If the internal and public interfaces have certificates from different Certificate Authorities (CAs), the deployment fails. Either the undercloud fails to contact the keystone public interface, or the internal interfaces receive malformed configuration.

This scenario affects deployments with TLS Everywhere, when the IPA server supplies the internal interfaces but the public interfaces have a certificate that the operator supplies. This also prevents 'brown field' deployments, where deployments with existing public certificates attempt to redeploy and configure TLS Everywhere.

There is currently no workaround for this defect.

There is a known issue in the Block Storage Service (cinder) due to the following conditions:

There is a known issue with the Object Storage service (swift). If you use pre-deployed nodes, you might encounter the following error message in /var/log/containers/stdouts/swift_rsync.log:

"failed to create pid file /var/run/rsyncd.pid: File exists"

Workaround: Enter the following command on all Controller nodes that are pre-deployed:

for d in $(podman inspect swift_rsync | jq '.[].GraphDriver.Data.UpperDir') /var/lib/config-data/puppet-generated/swift; do sed -i -e '/pid file/d' $d/etc/rsyncd.conf; done

When you update or upgrade python3-tripleoclient, Ansible does not receive the update or upgrade and Ansible or ceph-ansible tasks fail.

When you update or upgrade, ensure that Ansible also receives the update so that playbook tasks can run successfully.

There is a known issue with the OVN filter packets that ovn-controller generates. Router Advertisements that receive ACL processing in OVN are dropped if there is no explicit ACL rule to allow this traffic.

Workaround: Enter the following command to create a security rule:

openstack security group rule create --ethertype IPv6 --protocol icmp --icmp-type 134 <SECURITY_GROUP>

There are some known limitations for Mellanox ConnectX-5 adapter cards in VF LAG mode in OVS OFFLOAD deployments, SRIOV Switchdev mode.

You might encounter the following known issues and limitations when you use the Mellanox ConnectX-5 adapter cards with the virtual function (VF) link aggregation group (LAG) configuration in an OVS OFFLOAD deployment, SRIOV Switchdev mode:

The Ceph Dashboard currently does not work with the TLS Everywhere framework because the dashboard_protocol parameter was incorrectly omitted from the heat template. As a result, back ends fail to appear when HAproxy is started.

As a temporary solution, create a new environment file that contains the the dashboard_protocol parameter and include the environment file in your overcloud deployment with the -e option:

parameter_defaults:
  CephAnsibleExtraConfig:
    dashboard_protocol: 'https'

This solution introduces ceph-ansible bug. For more information, see https://bugzilla.redhat.com/show_bug.cgi?id=1860815.

There is a known issue where, after an ungraceful shutdown, Ceph containers might not start automatically on system reboot.

Workaround: Remove the old container IDs manually with the podman rm command. For more information, see https://bugzilla.redhat.com/show_bug.cgi?id=1858865#c2.

There is a known issue where enabling the realtime-virtual-host tuned profile inside guest virtual machines degrades throughput and displays non-deterministic performance. ovs-dpdk PMDs are pinned incorrectly to housekeeping CPUs.

Workaround: Use the cpu-partitioning tuned profile inside guest virtual machines, write a post-deployment script to update the tuned.conf file, and reboot the node:

ps_blacklist=ksoftirqd.*;rcuc.*;rcub.*;ktimersoftd.*;.*pmd.*;.*PMD.*;^DPDK;.*qemu-kvm.*