Red Hat Training

A Red Hat training course is available for Red Hat OpenStack Platform

Chapter 3. Release Information

These release notes highlight technology preview items, recommended practices, known issues, and deprecated functionality to be taken into consideration when deploying this release of Red Hat OpenStack Platform.

Notes for updates released during the support lifecycle of this Red Hat OpenStack Platform release will appear in the advisory text associated with each update.

3.1. Red Hat OpenStack Platform 13 GA

These release notes highlight technology preview items, recommended practices, known issues, and deprecated functionality to be taken into consideration when deploying this release of Red Hat OpenStack Platform.

3.1.1. Enhancements

This release of Red Hat OpenStack Platform features the following enhancements:

BZ#1419556

The Object Store service (swift) can now integrate with Barbican to transparently encrypt and decrypt your stored (at-rest) objects. At-rest encryption is distinct from in-transit encryption and refers to the objects being encrypted while being stored on disk.

Swift objects are stored as clear text on disk. These disks can pose a security risk if not properly disposed of when they reach end-of-life. Encrypting the objects mitigates that risk.

Swift performs these encryption tasks transparently, with the objects being automatically encrypted when uploaded to swift, then automatically decrypted when served to a user. This encryption and decryption is done using the same (symmetric) key, which is stored in Barbican.

BZ#1540239

This enhancement adds support for sending metrics data to a Gnocchi DB instance.

The following new parameters for collectd composable service were added. If CollectdGnocchiAuthMode is set to 'simple', then CollectdGnocchiProtocol, CollectdGnocchiServer, CollectdGnocchiPort and CollectdGnocchiUser are taken into account for configuration.

If CollectdGnocchiAuthMode is set to 'keystone', then CollectdGnocchiKeystone* parameters are taken into account for configuration.

Following is a detailed description of added parameters:

CollectdGnocchiAuthMode

type: string

description: Type of authentication Gnocchi server is using. Supported values are’simple' and 'keystone'.

default: 'simple'

CollectdGnocchiProtocol

type: string

description: API protocol Gnocchi server is using.

default: 'http'

CollectdGnocchiServer

type: string

description: The name or address of a gnocchi endpoint to which we should send metrics.

default: nil

CollectdGnocchiPort

type: number

description: The port to which we will connect on the Gnocchi server.

default: 8041

CollectdGnocchiUser

type: string

description: Username for authenticating to the remote Gnocchi server using simple authentication.

default: nil

CollectdGnocchiKeystoneAuthUrl

type: string

description: Keystone endpoint URL to authenticate to.

default: nil

CollectdGnocchiKeystoneUserName

type: string

description: Username for authenticating to Keystone.

default: nil

CollectdGnocchiKeystoneUserId

type: string

description: User ID for authenticating to Keystone.

default: nil

CollectdGnocchiKeystonePassword

type: string

description: Password for authenticating to Keystone

default: nil

CollectdGnocchiKeystoneProjectId

type: string

description: Project ID for authenticating to Keystone.

default: nil

CollectdGnocchiKeystoneProjectName

type: string

description: Project name for authenticating to Keystone.

default: nil

CollectdGnocchiKeystoneUserDomainId

type: string

description: User domain ID for authenticating to Keystone.

default: nil

CollectdGnocchiKeystoneUserDomainName

type: string

description: User domain name for authenticating to Keystone.

default: nil

CollectdGnocchiKeystoneProjectDomainId

type: string

description: Project domain ID for authenticating to Keystone.

default: nil

CollectdGnocchiKeystoneProjectDomainName

type: string

description: Project domain name for authenticating to Keystone.

default: nil

CollectdGnocchiKeystoneRegionName

type: string

description: Region name for authenticating to Keystone.

default: nil

CollectdGnocchiKeystoneInterface

type: string

description: Type of Keystone endpoint to authenticate to.

default: nil

CollectdGnocchiKeystoneEndpoint

type: string

description: Explicitly state Gnocchi server URL if you want to override Keystone value

default: nil

CollectdGnocchiResourceType

type: string

description: Default resource type created by the collectd-gnocchi plugin in Gnocchi to store hosts.

default: 'collectd'

CollectdGnocchiBatchSize

type: number

description: Minimum number of values Gnocchi should batch.

default: 10

BZ#1592823

Logs from Ansible playbooks now include timestamps that provide information about the timing of actions during deployment, updates, and upgrades.

3.1.2. Technology Preview

The items listed in this section are provided as Technology Previews. For further information on the scope of Technology Preview status, and the associated support implications, refer to https://access.redhat.com/support/offerings/techpreview/.

BZ#1446311

This release adds support for PCI device NUMA affinity policies, which are configured as part of the “[pci]alias” configuration options. Three policies are supported:

“required” (must have) “legacy” (default; must have, if available) “preferred” (nice to have)

In all cases, strict NUMA affinity is provided, if possible. These policies allow you to configure how strict your NUMA affinity should be per PCI alias to maximize resource utilization. The key difference between the policies is how much NUMA affinity you’re willing to forsake before failing to schedule.

When the “preferred” policy is configured for a PCI device, nova uses CPUs on a different NUMA node from the NUMA node of the PCI device, if it is available. This results in increased resource utilization, but performance is reduced for these instances.

BZ#1488095

From RHOS-12 onwards, the OpenStack services are becoming containerized. In this release, we containerize OpenStack Tempest as well. The containerized OpenStack Tempest is available as a Technology Preview.

3.1.3. Release Notes

This section outlines important details about the release, including recommended practices and notable changes to Red Hat OpenStack Platform. You must take this information into account to ensure the best possible outcomes for your deployment.

BZ#1468020

The Shared File System service (manila) now provides IPv6 access rule support with NetApp ONTAP cDOT driver, which lets you use manila with IPv6 environments.

As a result, the Shared File System service now supports exporting shares backed by NetApp ONTAP back ends over IPv6 networks. Access to the exported shares is controlled by IPv6 client addresses.

BZ#1469208

The Shared File System service (manila) supports mounting shared file systems backed by a Ceph File System (CephFS) via the NFSv4 protocol. NFS-Ganesha servers operating on Controller nodes are used to export CephFS to tenants with high availability (HA). Tenants are isolated from one another and may only access CephFS through the provided NFS gateway interface. This new feature is fully integrated into director, enabling CephFS back end deployment and configuration for the Shared File System service.

BZ#1496584

When neutron services are containerized, trying to run commands in a network namespace might fail with the following error:

# ip netns exec qrouter...
RTNETLINK answers: Invalid argument

In order to run a command inside a network namespace, you must do it from the neutron container that created the namespace. For example, the l3-agent creates network namespace for routers, so the command would need to change to:

# docker exec neutron_l3_agent ip netns exec qrouter...

Similarly with network namespaces beginning with 'qdhcp' you would need to exec from the 'neutron_dhcp' container.

BZ#1503521

This version introduces support for internal DNS resolution in networking-ovn. Although there are two know limitations, one is .BZ#1581332 which prevents proper resolution of internal fqdn requests via internal dns.

Please note that the extension is not configured by default by tripleo on the GA release. See .BZ#1577592 for a workaround.

BZ#1533206

The openstack-gnocchi packages have been renamed to gnocchi. The openstack- prefix was removed because of an upstream project scoping change. Gnocchi has been moved out of the OpenStack umbrella and is maintained as a stand-alone project.

BZ#1556933

Since version 2.1, python-cryptography checks that the CNS Names used in certificates are compliant with IDN standards. If the found names do not follow this specification, cryptography will fail to validate the certificate and different errors may be found when using OpenStack command line interface or in OpenStack service logs.

BZ#1563412

The reserved host memory for OpenStack Compute (nova) has increased from 2048 MB to 4096 MB. This can affect capacity estimations for your environment. If necessary, you can reconfigure the reserved memory using the 'NovaReservedHostMemory' parameter in a environment file. For example:

parameter_defaults: NovaReservedHostMemory: 2048

BZ#1564176

The python-mistralclient is not part of any supported overcloud use-cases so it is being dropped from the -tools channels for the OSP 13 release.

BZ#1567735

OSP13 using OVN as the networking backend won’t include IPv6 support in the first release. There is a problem with the responses to the Neighbor Solicitation requests coming from guests VMs which causes a loss of the default routes.

BZ#1575752

In previous versions, the *NetName parameters (e.g. InternalApiNetName) changed the names of the default networks. This is no longer supported.

To change the names of the default networks, use a custom composable network file (network_data.yaml) and include it with your 'openstack overcloud deploy' command using the '-n' option. In this file you should set the "name_lower" field to the custom net name for the network you want to change. For more information, see "Using Composable Networks" in the Advanced Overcloud Customization guide.

In addition, you need to add a local parameter for the ServiceNetMap table to network_environment.yaml and override all the default values for the old network name to the new custom name. The default values can be found in /usr/share/openstack-tripleo-heat-templates/network/service_net_map.j2.yaml. This requirement to modify ServiceNetMap will not be necessary in future OSP-13 releases.

BZ#1577537

Fixes OSP 13 Beta issue where some container images were not available.

BZ#1578312

When the OVSDB server fails over to a different controller node, a reconnection from neutron-server/metadata-agent does not take place because they are not detecting this condition.

As a result, booting VMs may not work as metadata-agent will not provision new metadata namespaces and the clustering is not behaving as expected.

A possible workaround is to restart the ovn_metadata_agent container in all the compute nodes after a new controller has been promoted as master for OVN databases. Also increase the ovsdb_probe_interval on the plugin.ini to a value of 600000 milliseconds.

BZ#1589849

When the OVN metadata agent is stopped in a Compute node, all the VMs on that node will not have access to the metadata service. The impact is that if a new VM is spawned or an existing VM is rebooted, the VM will fail to access metadata until the OVN metadata agent is brought up back again.

BZ#1592528

In rare circumstances, after rebooting controller nodes several times, RabbitMQ may be running in an inconsistent state that will block API operations on the overcloud.

The symptoms for this issue are: - Entries in any of the OpenStack service logs of the form: DuplicateMessageError: Found duplicate message(629ff0024219488499b0fac0cacaa3a5). Skipping it. - "openstack network agent list" returns that some agents are DOWN

To restore normal operation, run the following command on any of the controller nodes (you only need to do this on one controller): pcs resource restart rabbitmq-bundle

3.1.4. Known Issues

These known issues exist in Red Hat OpenStack Platform at this time:

BZ#1321179

OpenStack command-line clients that use python-requests can not currently validate certificates that have an IP address in the SAN field.

BZ#1461132

When using Red Hat Ceph Storage as a Block Storage backend for both Cinder volume and Cinder backup, any attempts to perform an incremental backup will result in a full backup instead, without any warning. This is a known issue.

BZ#1508449

OVN serves DHCP as an openflow controller with ovn-controller directly on compute nodes. But SR-IOV instances are directly attached to the network through the VF/PF. As such, SR-IOV instances will not be able to get DHCP responses from anywhere.

To workaround this issue, change OS::TripleO::Services::NeutronDhcpAgent to:

OS::TripleO::Services::NeutronDhcpAgent: docker/services/neutron-dhcp.yaml

BZ#1515815

When the router gateway is cleared, the Layer 3 flows related to learned IP addresses is not removed. The learned IP addresses include the PNF and external gateway IP addresses. This leads stale flows, but not any functional issue. The external gateway and IP address does not change frequently. The stale flows will be removed when the external network is deleted.

BZ#1518126

Redis is unable to correctly replicate data across nodes in a HA deployment with TLS enabled. Redis follower nodes will not contain any data from the leader node. It is recommended to disable TLS for Redis deployments.

BZ#1519783

Neutron may issue an error claiming that the Quota has been exceed for Neutron Router creation. This is a known issue where multiple router resources are created with a single create request in Neutron DB due to a bug with networking-odl. The workaround for this issue is to delete the duplicated routers using the OpenStack Neutron CLI and create a router again, resulting with a single instance.

BZ#1557794

A regression was identified in the procedure for backing up and restoring the director undercloud. As a result, the procedure requires modification and verification before it can be published.

The book 'Back Up and Restore the Director Undercloud' is therefore not available with the general availability of Red Hat OpenStack Platform 13. The procedure will be updated as a priority after the general availability release, and published as soon as it is verified.

BZ#1559055

OpenDaylight logging might be missing earlier logs. This is a known issue with journald logging of OpenDaylight (using the “docker logs opendaylght_api” command). The current workaround is to switch OpenDaylight logging to the “file” mechanism which will log inside of the container to /opt/opendaylight/data/logs/karaf.log. To do this, configure the following heat parameter: OpenDaylightLogMechanism: ‘file’.

BZ#1568012

Connecting to an external IP fails when associating a floating IP to an instance then disassociating the floating IP. This situation happens in a tenant VLAN network when: * a VM spawned on a non-NAPT switch is associated with a floating IP and * the floating IP is removed. This results in a missing flow (sporadically) in the FIB table of NAPT switch.

Due to the missing FIB table entry, the VM loses connectivity to the public network.

Associating the floating IP to the VM restores connectivity to the public network. As long as the floating IP is associated with the VM, it will be able to connect to the internet. However, you will lose a public IP/floating IP from the external network.

BZ#1568311

Layer 3 connectivity between nova instances across multiple subnets may fail when an instance without a floating IP tries to reach another instance that has a floating IP on another router. This occurs when nova instances are spread across multiple compute nodes. There is no suitable workaround for this issue.

BZ#1568976

During deployment, one or more OpenDaylight instances may fail to start correctly due to a feature loading bug. This may lead to a deployment or functional failure.

When a deployment passes, only two of the three OpenDaylight instances must be functional for the deployment to succeed. It is possible that the third OpenDaylight instance started incorrectly. Check the health status of each container with the docker ps command. If it is unhealthy, restart the container with docker restart opendaylight_api.

When a deployment fails, the only option is to restart the deployment. For TLS-based deployments, all OpenDaylight instances must boot correctly or deployment will fail.

BZ#1571864

Temporary removal of Heat stack resources during fast-forward upgrade preparation triggers RHEL unregistration. As a result, RHEL unregistration is stalled because Heat software deployment signalling does not work properly.

To avoid the problem, while the overcloud is still on OSP 10 and ready to perform the last overcloud minor version update: 1. Edit the template file /usr/share/openstack-tripleo-heat-templates/extraconfig/pre_deploy/rhel-registration/rhel-registration.yaml 2. Delete RHELUnregistration and RHELUnregistrationDeployment resources from the template. 3. Proceed with the minor update and fast-forward upgrade procedure.

BZ#1573597

A poorly performing Swift cluster used as a Gnocchi back end can generate 503 errors in the collectd log and "ConnectionError: ('Connection aborted.', CannotSendRequest())" errors in in gnocchi-metricd.conf. To mitigate the problem, increase the value of the CollectdDefaultPollingInterval parameter or improve the Swift cluster performance.

BZ#1574708

When an OpenDaylight instance is removed from a cluster and reconnected, the instance may not successfully join the cluster. The node will eventually re-join the cluster.

The following actions should be taken in such a situation: * Restart the faulty node. * Monitor the REST endpoint to verify the cluster member is healthy: http://$ODL_IP:8081/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore * The response should contain a field “SyncStatus”, and a value of “true” will indicate a healthy cluster member.

BZ#1574725

When multiple VMs in the same subnet of a VLAN provider network are scheduled to two different Compute nodes, ARP between the VMs fails sporadically.

Since ARP packets between those VMs fails, there is essentially no networking between the two VMs.

BZ#1575023

The manila-share service fails to initialize because changes to ceph-ansible’s complex ceph-keys processing generate incorrect content in the /etc/ceph/ceph.client.manila.keyring file.

To allow the manila-share service to initialize:

1) Make a copy of /usr/share/openstack/tripleo-heat-templates to use for the overcloud deploy.

2) Edit the …​/tripleo-heat-templates/docker/services/ceph-ansible/ceph-base.yaml file to change all triple backslashes in line 295 to single backslashes.

Before:

mon_cap: 'allow r, allow command \\\"auth del\\\", allow command \\\"auth caps\\\", allow command \\\"auth get\\\", allow command \\\"auth get-or-create\\\"'

After:

mon_cap: 'allow r, allow command \"auth del\", allow command \"auth caps\", allow command \"auth get\", allow command \"auth get-or-create\"'

3) Deploy the overcloud substituting the path to the copy of tripleo-heat-templates wherever /usr/share/openstack-tripleo-heat templates occurred in your original overcloud-deploy command.

The ceph key /etc/ceph/ceph.client.manila.keyring file will have proper contents and the manila-share service will initialize properly.

BZ#1575118

Ceph Release 12.2.1 lowers the maximum number of PGs allowed for each OSD. The lower limit may cause the monitor to prematurely issue a HEALTH_WARN message.

The monitor warning threshold has been reduced from 300 to 200 PGs per OSD. 200 is still twice the generally recommended target of 100 PGs per OSD. This limit can be adjusted via the mon_max_pg_per_osd option on the monitors. The older mon_pg_warn_max_per_osd option has been removed.

The amount of PGs consumed by a pool can not be decreased. If the upgrade causes a pre-existing deployment to reach the maximum limit, you can raise the limit to its pre-upgrade value during the ceph-upgrade step. In an environment file, add a parameter setting like this:

parameter_defaults:
  CephConfigOverrides:
    mon_max_pg_per_osd: 300

The setting is applied into ceph.conf and the cluster stays in HEALTH_OK state.

BZ#1575150

There is a known issue where the OpenDaylight cluster may stop responding for up to 30 minutes when an OpenDaylight cluster member is stopped (due to failure or otherwise). The workaround is wait until the cluster becomes active again.

BZ#1575496

When using a physical host interface for external network with Director, if the interface is not attached to an OVS bridge, the interface will not pass traffic in an OpenDaylight setup. Traffic will not pass and you should avoid this type of configuration.

Always use an OVS bridge in the NIC templates for an overcloud external network. This bridge is named "br-ex" by default in Director (although you may use any name). You should attach the physical host interface used for the external network to this OVS bridge.

When you use an interface attached to an OVS bridge, the deployment will function correctly and the external network traffic to tenants will work correctly.

BZ#1577975

OpenDaylight may experience periods of very high CPU usage. This issue should not affect the functionality of OpenDaylight, although it could potentially impact other system services.

BZ#1579025

OVN pacemaker Resource Agent (RA) script sometimes does not handle the promotion action properly when pacemaker tries to promote a slave node. This is seen when the ovsdb-servers report the status as master to the RA script when the master ip is moved to the node. The issue is fixed upstream.

When the issue occurs, the neutron server will not be able to connect the OVN North and South DB servers and all Create/Update/Delete APIs to the neutron server will fail.

Restarting the ovn-dbs-bundle resource will resolve the issue. Run the below command in one of the controller node:

"pcs resource restart ovn-dbs-bundle"

BZ#1579417

SNAT support requires configuring VXLAN tunnels regardless of the encapsulation used in the tenant networks. It is also necessary to configure the MTU correctly when using VLAN tenant networks, since the VXLAN Tunnel header is added to the payload and this could cause the packet to exceed the default MTU (1500 Bytes).

The VXLAN tunnels have to be properly configured in order for the SNAT traffic to flow through them. When using VLAN tenant networks, use one of the following methods to configure MTU so that SNAT traffic can flow through the VXLAN tunnels:: * Configure VLAN tenant based networks to use an MTU of 1450 on a per network configuration. * Set NeutronGlobalPhysnetMtu heat parameter to 1450. Note: the implication of this means all flat/VLAN provider networks will have a 1450 MTU, which may not be desirable (especially for external provider networks). * Configure tenant network underlay with MTU of 1550 (or higher). This includes setting the MTU in the NIC templates for tenant network NIC.

BZ#1581337

HAProxy, used for network load balancing, must be version 1.6 or higher to correctly support the PING type health monitor.

The version of HAProxy included with Red Hat OpenStack Platform 13 is an older version than 1.6 that uses TCP connect instead when you configure the PING type health monitor.

BZ#1583541

SRIOV based Compute instances have no connectivity to OVS Compute instances if they are on different networks. The workaround is to use an external router that is connected to both VLAN provider networks.

BZ#1584518

RHOSP does not configure the availability of DifferentHostFilter / SameHostFilter by default in nova, and these settings are necessary to properly complete some tests. As such, several security group tests might randomly fail.

You should skip those tests, or alternatively add those filters to your nova configuration.

BZ#1584762

If Telemetry is manually enabled on the undercloud, hardware.* metrics does not work due to a misconfiguration of the firewall on each of the nodes.

As a workaround, you need to manually set the snmpd subnet with the control plane network by adding an extra template for the undercloud deployment as follows"

parameter_defaults: SnmpdIpSubnet: 192.168.24.0/24

BZ#1588186

A race condition causes Open vSwitch to not connect to the Opendaylight openflowplugin. A fix is currently being implemented for a 13.z release of this product.

BZ#1590114

If Telemetry is manually enabled on the undercloud, hardware.* metrics does not work due to a misconfiguration of the firewall on each of the nodes.

As a workaround, you need to manually set the snmpd subnet with the control plane network by adding an extra template for the undercloud deployment as follows"

parameter_defaults:
  SnmpdIpSubnet: 192.168.24.0/24

BZ#1590560

The ceph-ansible utility does not always remove the ceph-create-keys container from the same node where it was created.

Because of this, the deployment may fail with the message "Error response from daemon: No such container: ceph-create-keys." This may affect any ceph-ansible run, including fresh deployments, that have: * multiple compute notes or * a custom role behaving as ceph client which is also hosting a service consuming ceph.

BZ#1590938

If you deploy more than three OSDs on RHCS3 and set the PG number for your pools as determined by pgcalc (https://access.redhat.com/labs/cephpgc), deployment will fail because ceph-ansible creates pools before all OSDs are active.

To avoid the problem, set the default PG number to 32 and when the deployment is finished, manually raise the PG number as described in the Storage Strategies Guide, https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/storage_strategies_guide/placement_groups_pgs#set_the_number_of_pgs.

BZ#1590939

Because ceph-ansible OpenStack pool tasks have an incorrect container name, it is not yet possible to colocate Ceph MONs and OSDs. Standard HCI (Computes + OSDs) is not affected.

BZ#1593290

After restarting the nova-compute service when a guest with SR-IOV-based network interface(s) attached is running and removing the guest, it is no longer possible to attach SR-IOV VFs on that node to any guest. This is because available devices are enumerated on service startup but as the device is attached to a guest it is not included in the list of host devices.

You must restart the 'nova-compute' service after removing the guest. After removing the guest and restarting the service, the list of available SR-IOV devices will be correct.

BZ#1593715

Insecure registry list is being updated later than some container images are pulled during a major upgrade. As such, container images from newly introduced insecure registry fails to download during openstack overcloud upgrade run command.

You can use one of the following workarounds:

Option A: Update the /etc/sysconfig/docker file manually on nodes which have containers managed by Pacemaker, and add any newly introduced insecure registries.

Option B: run openstack overcloud deploy command right before upgrading, and provide the desired new insecure registry list using an environment file with the DockerInsecureRegistryAddress parameter.

All container images should download successfully during upgrade.

BZ#1593757

Enabling Octavia on an existing overcloud deployment reports as a success, but the Octavia API endpoints are not reachable because the firewall rules on the Controller nodes are misconfigured.

Workaround On all controller nodes, add firewall rules and make sure they are inserted before the DROP rule.

IPv4:
  # iptables -A INPUT -p tcp -m multiport --dports 9876 -m state --state NEW -m comment --comment "100 octavia_api_haproxy ipv4" -j ACCEPT
  # iptables -A INPUT -p tcp -m multiport --dports 13876 -m state --state NEW -m comment --comment "100 octavia_api_haproxy_ssl ipv4" -j ACCEPT
  # iptables -A INPUT -p tcp -m multiport --dports 9876,13876 -m state --state NEW -m comment --comment "120 octavia_api ipv4" -j ACCEPT
IPv6:
  # ip6tables -A INPUT -p tcp -m multiport --dports 9876 -m state --state NEW -m comment --comment "100 octavia_api_haproxy ipv6" -j ACCEPT
  # ip6tables -A INPUT -p tcp -m multiport --dports 13876 -m state --state NEW -m comment --comment "100 octavia_api_haproxy_ssl ipv6" -j ACCEPT
  # ip6tables -A INPUT -p tcp -m multiport --dports 9876,13876 -m state --state NEW -m comment --comment "120 octavia_api ipv6" -j ACCEPT

Restart HAProxy:

  # docker restart haproxy-bundle-docker-0

BZ#1595363

During the fast forward upgrade process, users upgrade the undercloud from version 10 to version 11. In some situations, the nova-api.log might report the following error:

Unexpected API Error. Table 'nova_cell0.instances' doesn’t exist

You can resolve this error by running the following command:

$ sudo nova-manage api_db sync

This issue is non-critical and should not impede the fast forward upgrade process in a major way.

BZ#1790653

Because of the manner in which OpenStack Networking binds ports, the live migration of network instances in DVR environments might cause existing connections using a floating IP address to become disconnected. Currently, there is no workaround in RHOSP 13. However, this issue has been fixed in RHOSP 14 and later releases.

3.2. Red Hat OpenStack Platform 13 Maintenance Release - July 19, 2018

These release notes highlight technology preview items, recommended practices, known issues, and deprecated functionality to be taken into consideration when deploying this release of Red Hat OpenStack Platform.

3.2.1. Enhancements

This release of Red Hat OpenStack Platform features the following enhancements:

BZ#1592823

Logs from Ansible playbooks now include timestamps that provide information about the timing of actions during deployment, updates, and upgrades.

3.2.2. Release Notes

This section outlines important details about the release, including recommended practices and notable changes to Red Hat OpenStack Platform. You must take this information into account to ensure the best possible outcomes for your deployment.

BZ#1578312

When the OVSDB server fails over to a different controller node, a reconnection from neutron-server/metadata-agent does not take place because they are not detecting this condition.

As a result, booting VMs may not work as metadata-agent will not provision new metadata namespaces and the clustering is not behaving as expected.

A possible workaround is to restart the ovn_metadata_agent container in all the compute nodes after a new controller has been promoted as master for OVN databases. Also increase the ovsdb_probe_interval on the plugin.ini to a value of 600000 milliseconds.

3.2.3. Known Issues

These known issues exist in Red Hat OpenStack Platform at this time:

BZ#1515815

When the router gateway is cleared, the Layer 3 flows related to learned IP addresses is not removed. The learned IP addresses include the PNF and external gateway IP addresses. This leads stale flows, but not any functional issue. The external gateway and IP address does not change frequently. The stale flows will be removed when the external network is deleted.

BZ#1519783

Neutron may issue an error claiming that the Quota has been exceed for Neutron Router creation. This is a known issue where multiple router resources are created with a single create request in Neutron DB due to a bug with networking-odl. The workaround for this issue is to delete the duplicated routers using the OpenStack Neutron CLI and create a router again, resulting with a single instance.

BZ#1559055

OpenDaylight logging might be missing earlier logs. This is a known issue with journald logging of OpenDaylight (using the “docker logs opendaylght_api” command). The current workaround is to switch OpenDaylight logging to the “file” mechanism which will log inside of the container to /opt/opendaylight/data/logs/karaf.log. To do this, configure the following heat parameter: OpenDaylightLogMechanism: ‘file’.

BZ#1568311

Layer 3 connectivity between nova instances across multiple subnets may fail when an instance without a floating IP tries to reach another instance that has a floating IP on another router. This occurs when nova instances are spread across multiple compute nodes. There is no suitable workaround for this issue.

BZ#1568976

During deployment, one or more OpenDaylight instances may fail to start correctly due to a feature loading bug. This may lead to a deployment or functional failure.

When a deployment passes, only two of the three OpenDaylight instances must be functional for the deployment to succeed. It is possible that the third OpenDaylight instance started incorrectly. Check the health status of each container with the docker ps command. If it is unhealthy, restart the container with docker restart opendaylight_api.

When a deployment fails, the only option is to restart the deployment. For TLS-based deployments, all OpenDaylight instances must boot correctly or deployment will fail.

BZ#1583541

SRIOV based Compute instances have no connectivity to OVS Compute instances if they are on different networks. The workaround is to use an external router that is connected to both VLAN provider networks.

BZ#1588186

A race condition causes Open vSwitch to not connect to the Opendaylight openflowplugin. A fix is currently being implemented for a 13.z release of this product.

BZ#1593757

Enabling Octavia on an existing overcloud deployment reports as a success, but the Octavia API endpoints are not reachable because the firewall rules on the Controller nodes are misconfigured.

Workaround:

On all controller nodes, add firewall rules and make sure they are inserted before the DROP rule:

IPv4:
  # iptables -A INPUT -p tcp -m multiport --dports 9876 -m state --state NEW -m comment --comment "100 octavia_api_haproxy ipv4" -j ACCEPT
  # iptables -A INPUT -p tcp -m multiport --dports 13876 -m state --state NEW -m comment --comment "100 octavia_api_haproxy_ssl ipv4" -j ACCEPT
  # iptables -A INPUT -p tcp -m multiport --dports 9876,13876 -m state --state NEW -m comment --comment "120 octavia_api ipv4" -j ACCEPT
IPv6:
  # ip6tables -A INPUT -p tcp -m multiport --dports 9876 -m state --state NEW -m comment --comment "100 octavia_api_haproxy ipv6" -j ACCEPT
  # ip6tables -A INPUT -p tcp -m multiport --dports 13876 -m state --state NEW -m comment --comment "100 octavia_api_haproxy_ssl ipv6" -j ACCEPT
  # ip6tables -A INPUT -p tcp -m multiport --dports 9876,13876 -m state --state NEW -m comment --comment "120 octavia_api ipv6" -j ACCEPT

Restart HAProxy:

  # docker restart haproxy-bundle-docker-0

3.3. Red Hat OpenStack Platform 13 Maintenance Release - August 29, 2018

These release notes highlight technology preview items, recommended practices, known issues, and deprecated functionality to be taken into consideration when deploying this release of Red Hat OpenStack Platform.

3.3.1. Enhancements

This release of Red Hat OpenStack Platform features the following enhancements:

BZ#1561961

This feature adds support for PCI device NUMA affinity policies. These are configured as part of the [pci]alias configuration options. There are three policies supported: - required - legacy - preferred In all cases, strict NUMA affinity is provided if possible. The key difference between the policies is how much NUMA affinity you can forsake before failing to schedule. These policies allow you to configure how strict your NUMA affinity is on a per-device basis or, more specifically, per device alias. This is useful to ensure maximum resource utilization. When the 'preferred' policy is configured for a PCI device, nova now utilizes CPUs on a different NUMA node from the NUMA node of the PCI device if this is all that is available. This results in increased resource utilization with the downside of reduced performance for these instances.

BZ#1564918

Previously, Ironic considered just one IPMI error as retryable. That might have caused unjustified Ironic failure. With this enhancement, Ironic treats more types of IPMI error messages as retryable by the IPMI-backed hardware interfaces, such as power and management hardware interfaces. Specifically, "Node busy", "Timeout", "Out of space", and "BMC initialization in progress" IPMI errors cause Ironic to retry the IPMI command. The result is improved reliability of IPMI based communication with BMC.

BZ#1571741

Nova’s libvirt driver now allows the specification of granular CPU feature flags when configuring CPU models.

One benefit of this change is the alleviation of a performance degradation experienced on guests running with certain Intel-based virtual CPU models after application of the "Meltdown" CVE fixes. This guest performance impact is reduced by exposing the CPU feature flag 'PCID' ("Process-Context ID") to the guest CPU, assuming that the PCID flag is available in the physical hardware itself.

For more details, refer to the documentation of [libvirt]/cpu_model_extra_flags in nova.conf for usage details.

BZ#1574349

It is possible to create the stonith resources for the cluster automatically before the overcloud deployment. Before the start of the deployment, run the following command: openstack overcloud generate fencing --ipmi-lanplus --output /home/stack/fencing.yaml /home/stack/instackenv.json

Then pass '-e /home/stack/fencing.yaml' to the list of arguments to the deploy command. This creates the necessary stonith resources for the cluster automatically.

BZ#1578633

rhosp-director-images are now multi-arch. OSP 13 now has overcloud full and ironic python agent images for ppc64le. The resulting rhosp-director-images were adjusted to accommodate this change. As a result, rhosp-director-images and rhosp-director-images-ipa are now meta-packages, with rhosp-director-images-<arch> and rhosp-director-images-ipa-<arch> rpms added for multi-arch support.

BZ#1578636

rhosp-director-images are now multi-arch. OSP 13 now has overcloud full and ironic python agent images for ppc64le. The resulting rhosp-director-images were adjusted to accommodate this change. As a result, rhosp-director-images and rhosp-director-images-ipa are now meta-packages, with rhosp-director-images-<arch> and rhosp-director-images-ipa-<arch> rpms added for multi-arch support.

BZ#1579691

Nova’s libvirt driver now allows the specification of granular CPU feature flags when configuring CPU models. One benefit of this is the alleviation of a performance degradation experienced on guests running with certain Intel-based virtual CPU models after application of the "Meltdown" CVE fixes. This guest performance impact is reduced by exposing the CPU feature flag 'PCID' ("Process-Context ID") to the guest CPU, assuming that the PCID flag is available in the physical hardware itself. This change removes the restriction of having only 'PCID' as the only CPU feature flag and allows for the addition and removal of multiple CPU flags, making way for other use cases. For more information, refer to the documentation of [libvirt]/cpu_model_extra_flags in nova.conf.

BZ#1601472

The procedures for upgrading from RHOSP 10 to RHOSP 13 with NFV deployed have been retested and updated for DPDK and SR-IOV environments.

BZ#1606224

With this update, Ceph storage is supported by KVM virtualization on all CPU architectures supported by Red Hat.

BZ#1609352

This enhancement sees the addition of GA containers for nova and utilities, and Technology Preview containers for Cinder, Glance, Keystone, Neutron, and Swift on IBM Power LE.

BZ#1619311

rhosp-director-images are now multi-arch. OSP 13 now has overcloud full and ironic python agent images for ppc64le. The resulting rhosp-director-images were adjusted to accommodate this change. As a result, rhosp-director-images and rhosp-director-images-ipa are now meta-packages, with rhosp-director-images-<arch> and rhosp-director-images-ipa-<arch> rpms added for multi-arch support.

3.3.2. Release Notes

This section outlines important details about the release, including recommended practices and notable changes to Red Hat OpenStack Platform. You must take this information into account to ensure the best possible outcomes for your deployment.

BZ#1523864

This update adds support for use of Manila IPv6 export locations and access rules with Dell-EMC Unity and VNX back ends.

BZ#1549770

Containers are now the default deployment method. There is still a way to deploy the baremetal services in environments/baremetal-services.yaml, but this is expected to eventually disappear.

Environment files with resource registries referencing environments/services-docker must be altered to the environments/services paths. If you need to retain any of the deployed baremetal services, update references to environments/services-baremetal instead of the originally placed environments/services.

BZ#1565028

README has been added to /var/log/opendaylight, stating the correct OpenDaylight log path.

BZ#1570039

The compress option for the containerized logrotate service to compress rotated logs by default has been added. The delaycompress option ensures the first rotation of a log file remains uncompressed.

BZ#1575752

In previous versions, the *NetName parameters (e.g. InternalApiNetName) changed the names of the default networks. This is no longer supported. To change the names of the default networks, use a custom composable network file (network_data.yaml) and include it with your 'openstack overcloud deploy' command using the '-n' option. In this file, set the "name_lower" field to the custom net name for the network you want to change. For more information, see "Using Composable Networks" in the Advanced Overcloud Customization guide. In addition, you need to add a local parameter for the ServiceNetMap table to network_environment.yaml and override all the default values for the old network name to the new custom name. You can find the default values in /usr/share/openstack-tripleo-heat-templates/network/service_net_map.j2.yaml. This requirement to modify ServiceNetMap will not be necessary in future OSP-13 releases.

BZ#1592528

In rare circumstances, after rebooting controller nodes several times, RabbitMQ may be running in an inconsistent state that will block API operations on the overcloud.

The symptoms for this issue are: - Entries in any of the OpenStack service logs of the form: DuplicateMessageError: Found duplicate message(629ff0024219488499b0fac0cacaa3a5). Skipping it. - "openstack network agent list" returns that some agents are DOWN

To restore normal operation, run the following command on any of the controller nodes (you only need to do this on one controller): pcs resource restart rabbitmq-bundle

3.3.3. Known Issues

These known issues exist in Red Hat OpenStack Platform at this time:

BZ#1557794

A regression was identified in the procedure for backing up and restoring the director undercloud. As a result, the procedure requires modification and verification before it can be published.

The book 'Back Up and Restore the Director Undercloud' is therefore not available with the general availability of Red Hat OpenStack Platform 13. The procedure will be updated as a priority after the general availability release, and published as soon as it is verified.

BZ#1579025

OVN pacemaker Resource Agent (RA) script sometimes does not handle the promotion action properly when pacemaker tries to promote a slave node. This is seen when the ovsdb-servers report the status as master to the RA script when the master ip is moved to the node. The issue is fixed upstream.

When the issue occurs, the neutron server will not be able to connect the OVN North and South DB servers and all Create/Update/Delete APIs to the neutron server will fail.

Restarting the ovn-dbs-bundle resource will resolve the issue. Run the below command in one of the controller node:

"pcs resource restart ovn-dbs-bundle"

BZ#1584762

If Telemetry is manually enabled on the undercloud, hardware.* metrics does not work due to a misconfiguration of the firewall on each of the nodes. As a workaround, you need to manually set the snmpd subnet with the control plane network by adding an extra template for the undercloud deployment as follows: parameter_defaults: SnmpdIpSubnet: 192.168.24.0/24

3.4. Red Hat OpenStack Platform 13 Maintenance Release - November 13, 2018

These release notes highlight technology preview items, recommended practices, known issues, and deprecated functionality to be taken into consideration when deploying this release of Red Hat OpenStack Platform.

3.4.1. Enhancements

This release of Red Hat OpenStack Platform features the following enhancements:

BZ#1466117

To set MTU as a part of OSPD, this release adds neutron::plugins::ml2::physical_network_mtus as a NeutronML2PhysicalNetworkMtus in the heat template to enable MTU in the ml2 plugin. Neutron::plugins::ml2::physical_network_mtus is set based on values from the TripleO heat template.

BZ#1545151

Director uploads the latest amphora image to glance when OpenStack is updated and/or upgraded. The latest amphora image ensures amphora instances run with the latest general bug and security fixes, not only for Octavia agent fixes, but also for operating system fixes.

With this release, newly created and recreated amphora instances are made with the latest amphora image. Previous amphora images will remain stored in glance and be renamed to include the timestamp in the suffix.

BZ#1561961

This feature adds support for PCI device NUMA affinity policies. These are configured as part of the [pci]alias configuration options. There are three policies supported: - required - legacy - preferred In all cases, strict NUMA affinity is provided if possible. The key difference between the policies is how much NUMA affinity you can forsake before failing to schedule. These policies allow you to configure how strict your NUMA affinity is on a per-device basis or, more specifically, per device alias. This is useful to ensure maximum resource utilization. When the 'preferred' policy is configured for a PCI device, nova now utilizes CPUs on a different NUMA node from the NUMA node of the PCI device if this is all that is available. This results in increased resource utilization with the downside of reduced performance for these instances.

BZ#1571286

You can use the weigher CPUWeigher to spread (default) or pack workloads on hosts based on their vCPU usage. You can set the nova.conf [filter_scheduler] cpu_weight_multiplier configuration option to -1.0 or 1.0. If you set this option to 1.0, instances are spread across hosts. If you set this option to -1.0, instances are packed on a host. If you set the value to 0, the weigher is disabled.

BZ#1619485

There is a case where multipathd show status doesn’t return an error code as it should, so we are now checking stdout as a workaround for this issue to properly detect that multipathd is in an error state.

BZ#1628786

To prevent Ceph setup failures in a multi-architecture environment, this update ensures that Ceph setup of client users, keys, and pools is not run on a non-x86_64 system.

Ceph setup is supported only on x86_64 systems. Prior to this update, if the first inventory item in the clients group was not an x86_64 system, a Ceph setup attempt on that system would fail and cause the ceph-install to abort.

BZ#1629873

iSCSI device detection checked for the presence of devices based on the re-scan time. Devices becoming available between scans went undetected. With this release, searching and rescanning are independent operations working at different cadences with checks happening every second.

BZ#1631009

In prior versions, undercloud hieradata overrides could be used to tune some service configurations using the <service>::config options similar to the overcloud. However, this functionality was not available for all deployed OpenStack services. With this version, any configuration values not currently available can be updated via the <service>::config hieradata.

BZ#1640833

Support was added for volume retype and migration operations to the Block Storage service’s HPE Nimble Storage driver.

3.4.2. Release Notes

This section outlines important details about the release, including recommended practices and notable changes to Red Hat OpenStack Platform. You must take this information into account to ensure the best possible outcomes for your deployment.

BZ#1496584

When neutron services are containerized, trying to run commands in a network namespace might fail with the following error:

# ip netns exec qrouter...
RTNETLINK answers: Invalid argument

To run a command inside a network namespace, you must do it from the neutron container that created the namespace. For example, the l3-agent creates network namespace for routers, so the command should be:

# docker exec neutron_l3_agent ip netns exec qrouter...

Similarly, with network namespaces beginning with 'qdhcp' you would need to exec from the 'neutron_dhcp' container.

BZ#1567735

OSP13 using OVN as the networking backend won’t include IPv6 support in the first release. There is a problem with the responses to the Neighbor Solicitation requests coming from guests VMs that causes a loss of the default routes.

BZ#1589849

When the OVN metadata agent is stopped in a Compute node, all the VMs on that node will not have access to the metadata service. The impact is that if a new VM is spawned or an existing VM is rebooted, the VM will fail to access metadata until the OVN metadata agent is brought up back again.

3.4.3. Known Issues

These known issues exist in Red Hat OpenStack Platform at this time:

BZ#1621062

octavia-amphora file naming change can result in broken symlinks or symlinks with improper naming. As a workaround, run /usr/sbin/rhosp-director-image-update. This resolves the issue.

3.5. Red Hat OpenStack Platform 13 Maintenance Release - January 16, 2019

These release notes highlight technology preview items, recommended practices, known issues, and deprecated functionality to be taken into consideration when deploying this release of Red Hat OpenStack Platform.

3.5.1. Enhancements

This release of Red Hat OpenStack Platform features the following enhancements:

BZ#1476282

You can now deploy a minimal version of the overcloud with the overcloud-minimal qcow2 image. This minimal installation does not require OpenStack entitlements for updates.

BZ#1600115

Previously, the first packet of a new connection using an OVN logical router was used to discover the MAC address of the destination. This resulted in the loss of the first packet on the new connection.

This enhancement adds the capability to correctly queue the first packet of a new connection, which prevents the loss of that packet.

BZ#1607362

This feature adds support for deploying OpenDaylight (ODL) on IPv6 addresses.

BZ#1635892

Octavia previously assigned the Octavia project-id to the security group associated with the VIP and VRRP Amphora ports. This prevented the user from restricting access to the load-balancer. This fix adds the option to change SG ownership to belong to the user project (for certain whitelisted projects), which enables the user to refine access policies for the load-balancers.

BZ#1636395

This feature allows you to create instances with trusted SR-IOV virtual functions (VFs), such as changing the MAC address of the VF and enable promiscuous mode directly from the guest instance. These functions help you configure failover VFs for instances directly from the instance.

To configure trusted mode for VFs, you first set the trusted value of the [pci] passthrough_whitelist JSON configuration option in nova.conf. You then create the port with the trusted=true attribute in the binding profile. Make sure that the vnic-type attribute has the hw_veb or direct values.

BZ#1644020

This feature adds a new parameter NovaLibvirtVolumeUseMultipath (boolean), which sets the multipath configuration parameter libvirt/volume_use_multipath in the nova.conf file for Compute nodes. This parameter can be set for each Compute role. Default value is False.

BZ#1646907

This feature adds the capability to boot whole security-hardened images in UEFI mode.

BZ#1648261

This enhancement adds the parameter NeutronOVSTunnelCsum, which allows you to configure neutron::agents::ml2::ovs::tunnel_csum in the heat template. This parameter sets or removes the tunnel header checksum on the GRE/VXLAN tunnel that carries outgoing IP packets in the OVS agent.

BZ#1656495

This feature adds the parameter NovaSchedulerWorkers, which allows you to configure multiple nova-schedule workers for each scheduler node. Default value is 1.

3.5.2. Known Issues

These known issues exist in Red Hat OpenStack Platform at this time:

BZ#1574708

When an OpenDaylight instance is removed from a cluster and reconnected, the instance may not successfully join the cluster. The node will eventually re-join the cluster.

The following actions should be taken in such a situation.

  1. Restart the faulty node.
  2. Monitor the REST endpoint to verify the cluster member is healthy: http://$ODL_IP:8081/jolokia/read/org.opendaylight.controller:Category=ShardManager,name=shard-manager-config,type=DistributedConfigDatastore

The response should contain a field “SyncStatus”, and a value of “true” will indicate a healthy cluster member.

BZ#1579025

OVN pacemaker Resource Agent (RA) script sometimes does not handle the promotion action properly when pacemaker tries to promote a slave node. This is seen when the ovsdb-servers report the status as master to the RA script when the master ip is moved to the node. The issue is fixed upstream.

When the issue occurs, the neutron server will not be able to connect the OVN North and South DB servers and all Create/Update/Delete APIs to the neutron server will fail.

Restarting the ovn-dbs-bundle resource will resolve the issue. Run the below command in one of the controller node:

pcs resource restart ovn-dbs-bundle

3.6. Red Hat OpenStack Platform 13 Maintenance Release - March 13, 2019

These release notes highlight technology preview items, recommended practices, known issues, and deprecated functionality to be taken into consideration when deploying this release of Red Hat OpenStack Platform.

3.6.1. Enhancements

This release of Red Hat OpenStack Platform features the following enhancements:

BZ#1636496

With this update, you can use the following parameters to set the default Octavia timeouts for backend member and frontend client:

  • OctaviaTimeoutClientData: Frontend client inactivity timeout
  • OctaviaTimeoutMemberConnect: Backend member connection timeout
  • OctaviaTimeoutMemberData: Backend member inactivity timeout
  • OctaviaTimeoutTcpInspect: Time to wait for TCP packets for content inspection

The value for all of these parameters is in milliseconds.

BZ#1636895

With this update, you can now create tunnel endpoints on IPv6 addresses.

BZ#1650576

Previously, OpenDaylight packaging used the default OpenDaylight log_pattern values and included the PaxOsgi appender. These default values are not always appropriate for every deployment and it is appropriate to configure custom values.

With this update, puppet-opendaylight has two additional configuration variables:

  • log_pattern: Use this variable to configure which log pattern you want to use with the OpenDaylight logger log4j2.
  • enable_paxosgi_appender: Use this boolean flag to enable or disable the PaxOsgi appender.

puppet-opendaylight also modifies the OpenDaylight defaults. Deployments that use puppet-opendaylight have new defaults:

  • log_pattern: %d{ISO8601} | %-5p | %-16t | %-60c{6} | %m%n
  • enable_paxosgi_appender: false

New variable configuration options

log_pattern

String that controls the log pattern used for logging.

Default: %d{ISO8601} | %-5p | %-16t | %-60c{6} | %m%n

Valid options: A string that is a valid log4j2 pattern.

enable_paxosgi_logger

Boolean that controls whether the PaxOsgi appender is enabled for logging.

If you enable the enable_paxosgi_logger variable, you must also modify the log pattern to utilize the additional capabilities. Modify the log_pattern variable and include a pattern that contains the PaxOsgi tokens. For example, set the log_pattern variable to a string that includes the following values:

'%X{bundle.id} - %X{bundle.name} -
+
%X{bundle.version}'

If you do not edit the log_pattern variable, the PaxOsgi appender is still enabled and continues to run but logging does not utilize the additional functionality.

For example, set the enable_paxosgi_logger variable to true and set the log_pattern variable to the following value:

'%d{ISO8601} | %-5p | %-16t | %-32c{1} | %X{bundle.id} - %X{bundle.name} - %X{bundle.version} | %m%n'

Default: false

Valid options: The boolean values true and false.

3.6.2. Release Notes

This section outlines important details about the release, including recommended practices and notable changes to Red Hat OpenStack Platform. You must take this information into account to ensure the best possible outcomes for your deployment.

BZ#1597666

With this update, OpenDaylight minor update is now included in the Red Hat OpenStack Platform minor update workflow.

BZ#1611960

With this update, Compute nodes in a Red Hat OpenStack Platform environment that uses OpenDaylight as a back end can be scaled successfully.

3.6.3. Known Issues

These known issues exist in Red Hat OpenStack Platform at this time:

BZ#1574725

When multiple VMs in the same subnet of a VLAN provider network are scheduled to two different Compute nodes, ARP between the VMs fails sporadically.

Since ARP packets between those VMs fails, there is essentially no networking between the two VMs.

BZ#1575150

There is a known issue where the OpenDaylight cluster may stop responding for up to 30 minutes when an OpenDaylight cluster member is stopped (due to failure or otherwise). The workaround is wait until the cluster becomes active again.

BZ#1577975

OpenDaylight may experience periods of very high CPU usage. This issue should not affect the functionality of OpenDaylight, although it could potentially impact other system services.

3.6.4. Removed Functionality

The following functionality has been removed from this release of Red Hat OpenStack Platform.

BZ#1431431

Block Storage - Highly Available Active-Active Volume Service is no longer available as a technology preview, and is not supported for this release.

BZ#1683464

Block Storage - RBD Cinder Volume Replication is no longer available for technology preview and is not supported.

3.7. Red Hat OpenStack Platform 13 Maintenance Release - April 30, 2019

These release notes highlight technology preview items, recommended practices, known issues, and deprecated functionality to be taken into consideration when deploying this release of Red Hat OpenStack Platform.

3.7.1. Enhancements

This release of Red Hat OpenStack Platform features the following enhancements:

BZ#1654079
Previously, the undercloud upgrade failed if the overcloud was in a failed state. As a result, you could not deploy fixes by modifying the undercloud.

With this update, you can use the new undercloud upgrade option --force. This option causes the undercloud upgrade process to ignore the overcloud state.

BZ#1692872

With this update, you can use the following new tags to control the configuration management Ansible playbooks and tasks:

  • container_config
  • container_config_tasks
  • container_config_scripts
  • container_startup_configs
  • host_config
  • step1
  • step2
  • step3
  • step4
  • step5

3.7.2. Known Issues

These known issues exist in Red Hat OpenStack Platform at this time:

BZ#1664701

A recent change made memory allocation for instances with NUMA topologies pagesize aware. With this change, memory for instances with NUMA topologies can no longer be oversubscribed.

As a result, memory oversubscription is currently disabled for all instances with a NUMA topology, whereas previously only instances with hugepages were not allowed to use oversubscription. This affects instances with an explicit NUMA topology and those with an implicit topology. An instance can have an implicit NUMA topology due to the use of hugepages or CPU pinning.

If possible, avoid the use of explicit NUMA topologies. If CPU pinning is required, resulting in an implicit NUMA topology, there is no workaround.

3.8. Red Hat OpenStack Platform 13 Maintenance Release - July 10, 2019

These release notes highlight technology preview items, recommended practices, known issues, and deprecated functionality to be taken into consideration when deploying this release of Red Hat OpenStack Platform.

3.8.1. Enhancements

This release of Red Hat OpenStack Platform features the following enhancements:

BZ#1589505
With this update you can now configure NUMA affinity to ensure instances are placed on the same NUMA node as the NIC providing external connectivity for the vSwitch. This feature is available for layer 2 networks of type 'flat' or 'vlan' and layer 3 networks of type 'vxlan', 'gre', or 'geneve'.
BZ#1688461

There are two new CLI arguments you can use with config-download:

  • Monitor the deployment in a separate CLI session or with the API with openstack overcloud status.
  • Log and save Ansible errors for future analysis with openstack overcloud failures.
BZ#1698467
This enhancement adds new features and usability enhancements to the Octavia Horizon dashboard.
BZ#1698683
This update adds a new director parameter, CinderNfsSnapshotSupport, which you can use to control Block Storage service (cinder) snapshots on NFS back ends.
BZ#1701427

Previously, if you enabled TLS throughout your environment, the communication between internal services, such as the haproxy and the manila API, was not secured.

With this update, the manila API supports TLS endpoints on the internal API network.

BZ#1714227
This update adds support for a second ceph Storage Tier deployment capability through director.

3.8.2. Technology Preview

The items listed in this section are provided as Technology Previews. For further information on the scope of Technology Preview status, and the associated support implications, refer to https://access.redhat.com/support/offerings/techpreview/.

BZ#1624971

With this Technology Preview update, you can attach a single volume to multiple hosts with the multi-attach feature. The volumes are connected without a storage protocol and data consistency must be provided by an appropriate software application such as a clustered filesystem.

Contact Red Hat to determine if your target storage driver supports the multi-attach feature in this release. Cinder volume encryption is not supported on multi-attached volumes and other feature restrictions may apply.

3.8.3. Release Notes

This section outlines important details about the release, including recommended practices and notable changes to Red Hat OpenStack Platform. You must take this information into account to ensure the best possible outcomes for your deployment.

BZ#1651191

This update adds support for the Ansible Networking ML2 plugin, which provides the following functionality:

  • Isolated Tenant network capability for ironic baremetal guests.
  • Automated switch configuration for baremetal nodes.
  • Use of the same ML2 driver for multiple switch platforms.
BZ#1696332
This update provides OpenStack Messaging Service (zaqar) support for IPv6 endpoints, to facilitate an Undercloud deployed with IPv6.
BZ#1701425

Previously, the manila API was not deployed with Apache httpd server.

With this update, the Apache logs are located in /var/log/containers/httpd/manila-api on the nodes with manila API container.

3.8.4. Known Issues

These known issues exist in Red Hat OpenStack Platform at this time:

BZ#1581337

HAProxy, used for network load balancing, must be version 1.6 or higher to correctly support the PING type health monitor.

The version of HAProxy included with Red Hat OpenStack Platform 13 is an older version than 1.6 that uses TCP connect instead when you configure the PING type health monitor.

3.9. Red Hat OpenStack Platform 13 Maintenance Release - September 4, 2019

These release notes highlight technology preview items, recommended practices, known issues, and deprecated functionality to be taken into consideration when deploying this release of Red Hat OpenStack Platform.

3.9.1. Enhancements

This release of Red Hat OpenStack Platform features the following enhancements:

BZ#1614361

With this update, you can now use the Red Hat OpenStack Platform 13 host-config-and-reboot environment during fast-forward upgrade:

  1. Remove the NodeUserData mapping from the first-boot script
  2. Add the host-config-and-reboot.yaml environment file to the deploy command
  3. Add KernelArgs and TunedProfile configured to the OvS-DPDK role using role-specific parameters
  4. Ensure that the KernelArgs and TunedProfile correspond to the OpenStack Platform 10 values. Any changes result in the node rebooting during fast-forward upgrade and the upgrade fails.

Ansible cannot handle reboots performed by the heat stack configuration. Any incorrect configuration that results in reboot causes the fast-forward upgrade process to fail.

Note

You can still perform the fast-forward upgrade with the existing first-boot scripts, even with the new patches present.

BZ#1631107

With this update, Red Hat OpenStack Platform contains a new parameter DnsSearchDomains. You can use this parameter for IDM and FreeIPA environments that have different DNS subdomains. Set this parameter in the parameter_defaults section of an environment file to add a list of DNS search domains to resolv.conf.

BZ#1679267

Previously, it was not possible to upgrade to TLS Everywhere in an existing deployment.

With this update, you can secure the in-flight connections between internal OpenStack services without reinstallation.

BZ#1688385

With this enhancement, you can pass arbitrary mysql config options to the mysql cluster on the overcloud with the tripleo::profile::base::database::mysql::mysql_server_options hiera hash key.

BZ#1706992

With this update, you can now configure automatic restart of instances on a Compute node if the node reboots without migrating the instances. You can configure Nova and the libvirt-guests agent to shut down the instances gracefully and restart them when the Compute node reboots. The following parameters are new in Red Hat OpenStack Platform 13: NovaResumeGuestsStateOnHostBoot (True/False) NovaResumeGuestsShutdownTimeout (default 300s)

BZ#1713761

With this update, you can set the domain for an ifcfg configuration with a key for interfaces called domain. Use this to improve DNS search, which is needed to support IDM/FreeIPA environments with different DNS subdomains.

BZ#1732220

With this update, rabbitmq-management interface is now enabled on localhost by default on the overcloud so that it is simpler to monitor and query the state of rabbitmq via its management API.

3.9.2. Technology Preview

The items listed in this section are provided as Technology Previews. For further information on the scope of Technology Preview status, and the associated support implications, refer to https://access.redhat.com/support/offerings/techpreview/.

BZ#1700882

This update expands the Block Storage service multi attach feature to the Ceph RBD driver.

3.9.3. Release Notes

This section outlines important details about the release, including recommended practices and notable changes to Red Hat OpenStack Platform. You must take this information into account to ensure the best possible outcomes for your deployment.

BZ#1592528

In rare circumstances, when you reboot a controller node several times, RabbitMQ can run in an inconsistent state that blocks API operations on the overcloud.

The symptoms for this issue are: - Entries in any of the OpenStack service logs of the form: DuplicateMessageError: Found duplicate message(629ff0024219488499b0fac0cacaa3a5). Skipping it. - "openstack network agent list" returns that some agents are DOWN

To restore normal operation, run the following command on any one of the controller nodes: pcs resource restart rabbitmq-bundle

BZ#1656978

Previously, the Neutron SR-IOV agent set two possible states for virtual functions (VFs), enable or disable, which forced the VF link state regardless of the physical function (PF) link state.

With this update, the Neutron SR-IOV agent sets VFs to auto or disable. The auto state replicates the PF up or down automatically. As a result, if the PF is in the down state, the VF does not transmit or receive, even with other VFs in the same embedded switch (NIC).

Note

This behavior is not standard and depends on the NIC vendor implementation. Check the driver manual for the actual behavior of a VF in the auto state when the PF is down.

BZ#1708705

This update enables multi-attach capability for hpe3par driver.

3.9.4. Known Issues

These known issues exist in Red Hat OpenStack Platform at this time:

BZ#1745130

There is currently a known issue for TLS Everywhere in-place upgrades, where overcloud nodes are consequently unable to enroll in IdM. As a workaround, remove /etc/ipa/ca.crt/ from all overcloud nodes before running the overcloud deploy. For more information, see https://bugzilla.redhat.com/show_bug.cgi?id=1732564.

3.9.5. Deprecated Functionality

The items in this section are either no longer supported, or will no longer be supported in a future release.

BZ#1541829

File injection from the Compute REST API. This will continue to be supported for now if using API microversion < 2.56. However, nova will eventually remove this functionality. The changes are as follows:

  • Deprecate the personality parameter from the POST /servers create server API and the POST /servers/{server_id}/action rebuild server API. Specifying the personality parameter in the request body to either of these APIs will result in a 400 Bad Request error response.
  • Add support to pass user_data to the rebuild server API as a result of this change.
  • Stop returning maxPersonality and maxPersonalitySize response values from the GET /limits API.
  • Stop accepting and returning injected_files, injected_file_path_bytes, injected_file_content_bytes from the os-quota-sets and os-quota-class-sets APIs.

3.10. Red Hat OpenStack Platform 13 Maintenance Release - November 6, 2019

These release notes highlight technology preview items, recommended practices, known issues, and deprecated functionality to be taken into consideration when deploying this release of Red Hat OpenStack Platform.

3.10.1. Enhancements

This release of Red Hat OpenStack Platform features the following enhancements:

BZ#1561961

This release includes support for PCI device NUMA affinity policies. You can configure these policies as part of the [pci]alias configuration options. The following policies are supported:

  • required
  • legacy
  • preferred

In all cases, strict NUMA affinity is provided if possible. The key difference between the policies is the amount of NUMA affinity you can sacrifice before failing to schedule.

You can use these policies to configure how strict your NUMA affinity is for each device, or more specifically, for each device alias. This is useful to ensure maximum resource utilization.

When you configure the 'preferred' policy for a PCI device, nova now utilizes CPUs on a different NUMA node from the NUMA node of the PCI device, if only a different node is available. This results in increased resource utilization with the downside of reduced performance for these instances.

BZ#1656292

With this update, you can now use NVIDIA vGPU GRID drivers with Red Hat OpenStack Platform 13. Red Hat fully supports installations that include these drivers, excluding support for third-party applications developed around the NVIDIA GPU and dedicated driver.

BZ#1684421

With this release, you can configure NovaEnableRbdBackend on a per-role basis so that you can deploy a subset of compute hosts with RBD ephemeral disks. The remaining hosts continue using the default local ephemeral disk.

NOTE For best performance, the images that you deploy to RBD ephemeral compute hosts must be in RAW format, and the images that you deploy to local ephemeral compute hosts must be in QCOW2 format.

BZ#1726733

Before this update, the default amphora timeouts were too high for production environments.

With this update, the default amphora timeouts are more suitable for production environments. You can also use new director parameters to override the defaults.

BZ#1731210

This enhancement upgrades facter to version 3, which improves performance when you deploy and run updates on systems with a large number of network interfaces. This version of facter supports fact caching and generates the fact list significantly faster.

NOTE You must run facter version 3 in the same containers that you deploy on the host system when you use the version of openstack-tripleo-heat-templates that implements facter version 3.

BZ#1732220

With this update, rabbitmq-management interface is now enabled on localhost by default on the overcloud so that it is simpler to monitor and query the state of rabbitmq with its management API.

BZ#1733260

With this update, openstack-tripleo-common is enhanced so that you can run the openstack overcloud generate fencing command to create proper fencing configuration for ironic nodes that use the staging-ovirt power driver, similar to virtual machines that you deploy on RHV.

puppet-tripleo has also been enhanced and now properly configures pacemaker fence-rhevm stonith agents for virtual machines on RHV.

BZ#1746112

With this update, the overcloud role name can now start with a number, for example, 10Controller or 99Compute.

BZ#1762167

This enhancement adds the following plugins to the collectd container: - connectivity - mysql - ping - procevent - snmp-agent - sysevent

3.11. Red Hat OpenStack Platform 13 Maintenance Release - December 19, 2019

These release notes highlight technology preview items, recommended practices, known issues, and deprecated functionality to be taken into consideration when deploying this release of Red Hat OpenStack Platform.

3.11.1. Enhancements

This release of Red Hat OpenStack Platform features the following enhancements:

BZ#1766735

This enhancement allows OpenStack Image Service (glance) volumes to be mounted in a containerized control plane, as required by ScaleIO.

3.11.2. Deprecated Functionality

The items in this section are either no longer supported, or will no longer be supported in a future release.

BZ#1719815

The OpenStack Telemetry Event Storage (panko) service is now deprecated. Support for panko is limited to usage from Red Hat Cloudforms only. Red Hat does not recommend using panko outside of the Red Hat Cloudforms use-case. You can use the following options instead of using panko:

  • Poll the OpenStack Telemetry Metrics (gnocchi) service instead of polling panko. This gives you access to the resource history.
  • Use the OpenStack Telemetry Alarming (aodh) service to trigger alarms when an event arises. You can use OpenStack Messaging Service (zaqar) to store alarms in a queue if an application cannot be reached directly by the OpenStack Telemetry Alarming (aodh) service.

3.12. Red Hat OpenStack Platform 13 Maintenance Release - March 10, 2020

These release notes highlight technology preview items, recommended practices, known issues, and deprecated functionality to be taken into consideration when deploying this release of Red Hat OpenStack Platform.

3.12.1. Enhancements

This release of Red Hat OpenStack Platform features the following enhancements:

BZ#1726733

This enhancement provides default Octavia timeouts that are suitable for production environments, and the following new heat parameters to override the defaults:

  • OctaviaConnectionMaxRetries
  • OctaviaBuildActiveRetries
  • OctaviaPortDetachTimeout
BZ#1757886
You can now configure PCI NUMA affinity on an instance-level basis. This is required to configure NUMA affinity for instances with SR-IOV-based network interfaces. Previously, NUMA affinity was only configurable at a host-level basis for PCI passthrough devices.
BZ#1760567
This update adds a feature that verifies that ceph-ansible is installed from the ceph-tools repository.
BZ#1760871

This update adds the following parameters to fine-tune Octavia keepalived VRRP settings:

octavia::controller::vrrp_advert_int

Amphora role and priority advertisement internal in seconds. Defaults to $::os_service_default

octavia::controller::vrrp_check_interval

VRRP health check script run interval in seconds. Defaults to $::os_service_default

octavia::controller::vrrp_fail_count

Number of successive failures before transition to a fail rate. Defaults to $::os_service_default.

octavia::controller::vrrp_success_count

Number of consecutive successes before transition to a success rate. Defaults to $::os_service_default.

octavia::controller::vrrp_garp_refresh_interval

Time in seconds between gratuitous ARP announcements from the MASTER. Defaults to $::os_service_default.

octavia::controller::vrrp_garp_refresh_count

Number of gratuitous ARP announcements to make on each refresh interval. Defaults to $::os_service_default.

To customize your environment, see https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html/advanced_overcloud_customization/.

BZ#1766735
This enhancement allows OpenStack Image Service (glance) volumes to be mounted in a containerized control plane, as required by ScaleIO.
BZ#1777993
With this update, support for the extension of attached volumes is now enabled for the Libvirt OpenStack Nova virt driver.
BZ#1782229

This enhancement adds two new parameters, GlanceImageImportPlugins and GlanceImageConversionOutputFormat, to enable Image service (glance) plugins during the image import process.

For example, if you enable the image_conversion plugin, the following command imports a qcow2 image, stores it in raw format, and converts it automatically after import:

glance image-create-via-import --disk-format qcow2 --container-format bare --name cirros --import-method web-download --uri http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img

This means that you can store images always in raw format when you use RBD as an Image service driver.

3.13. Red Hat OpenStack Platform 13 Maintenance Release - June 24, 2020

These release notes highlight technology preview items, recommended practices, known issues, and deprecated functionality to be taken into consideration when deploying this release of Red Hat OpenStack Platform.

3.13.1. Bug Fix

These bugs were fixed in this release of Red Hat OpenStack Platform:

BZ#1650046

Before this update, SELinux limitations prevented RabbitMQ logs from rotating correctly on the undercloud when logged in as root with su.

The rabbitmq-server package is updated and the openstack-selinux has a new policy rule to enable correct log rotation.

BZ#1783210
This update puts the correct ipc:host setting in the pacemaker template version of the cinder-backup container.
BZ#1805840

Before this update, deploying Ceph RadosGW did not create the swiftoperator role, and users with the swiftoperator role were not granted administration permissions for RadosGW Swift endpoints.

With this update, deploying Swift or Ceph RadosGW creates the swiftoperator role automatically, and users with the swiftoperator role can now administer RadosGW objects as well as Swift objects.

BZ#1811122

Before this update, the xtremio driver reported incorrect available space capacity, which prevented virtual machine instances that rely on the storage backend from provisioning with insufficient space.

After this update, the xtremio driver now reports the correct free space capacity, and virtual machine instances can provision correctly.

BZ#1813640
This update assigns higher priority to port create and update messages received by the neutron DHCP agent during processing to decrease 'Failed to allocate network' errors when booting instances.
BZ#1813642

This update fixes a bug that caused failure of upgrades from the OpenStack Platform maintenance release of 10 July 2019 (RHOSP 13.0.7). Specifically, it fixes a cell management error in OpenStack Compute (nova).

Now you can upgrade to newer OpenStack versions from the OpenStack Platform maintenance release of 10 July 2019 (RHOSP 13.0.7).

BZ#1829284
This update correctly associates a cinder volume with its corresponding VxFlex OS volume, so that when you delete a volume, the corresponding backend volume is also deleted. Previously, deletion of a cinder volume did not trigger deletion of the corresponding VxFlex OS volume.
BZ#1829765

Before this update, the cinder RBD driver did not perform trim or discard operations, which prevented users from trimming unused spaces from cinder RBD volumes.

With this update, the cinder RBD driver now supports trim and discard operations.

BZ#1835870
This update fixes a bug that caused Cinder offline volume migration for HPE 3par storage to fail.

3.13.2. Enhancements

This release of Red Hat OpenStack Platform features the following enhancements:

BZ#1670592
This update adds support for hyper-coverged infrastructure (HCI) deployments with OVS-DPDK. An HCI architecture features co-location of overcloud nodes with Compute and Ceph Storage services, configured for better resource usage.
BZ#1759254
This enhancement adds the OctaviaConnectionLogging parameter so that you can disable connection flow logging. The default setting is true, which means that connection flows are logged.
BZ#1823416
This enhancement updates the cinder Datera driver to version 2.2.
BZ#1841194
This enhancement corrects a condition that caused network traffic flooding with the openvswitch firewall driver. Previously, the lack of an forwarding database (FDB) entry for the destination MAC address of relevant packets caused flooding across all ports in a given VLAN on the integration bridge. Now the traffic is directed only to the correct port.

3.13.3. Release Notes

This section outlines important details about the release, including recommended practices and notable changes to Red Hat OpenStack Platform. You must take this information into account to ensure the best possible outcomes for your deployment.

BZ#1804412
This update lets you disable connection logging inside the amphora.

3.14. Red Hat OpenStack Platform 13 Maintenance Release - October 28, 2020

These release notes highlight technology preview items, recommended practices, known issues, and deprecated functionality to be taken into consideration when deploying this release of Red Hat OpenStack Platform.

3.14.1. Bug Fix

These bugs were fixed in this release of Red Hat OpenStack Platform:

BZ#1723482

Before this update, the Compute (nova) service not releasing resources, such as network ports, until a Compute node is restored was causing the Load-balancing service (octavia) failover to fail when it was unable to detach a network port from an instance on a Compute node that is down.

With this update, the failover flow in the Load-balancing service has been updated to work around this Compute service issue. The Load-balancing service will now abandon ports that the Compute service will not release, leaving them in a "pending delete" state for the Compute service or Networking service to clean up once the Compute node is restored. This resolves the issue, allowing failover to succeed even if the Compute node is still failed.

BZ#1806975

Before this update, when several restores were being performed concurrently, the backup service was failing because the system was running out of memory.

With this update, we have increased the rate at which Python frees memory during backup restore operations by reducing the reference count to the data sooner, to allow Python to garbage collect the data as soon as it is decompressed rather than waiting until the restoration is complete. This resolves the issue, allowing the backup service to handle multiple restorations concurrently.

BZ#1841157
Before this update, FC live migration was failing. With this update, the correct device information is now sent to os-brick for FC for the corresponding host. Also, the device is now removed from the correct masking view when the live migration process has failed on the Compute node.
BZ#1841866
Before this update, the 3PAR driver did not examine the _name_id field for a possible volume ID, which caused volumes to be unusable after a live migration. With this update, the driver is now aware of the _name_id field as an alternative location for the volume ID, and live migrated volumes now work as expected.
BZ#1843196

Before this update, the internal temporary snapshot, created during async migration when creating a volume from a snapshot, was not being deleted from the VNX storage.

For example, if we create a new volume, V2, from snapshot S1, which we created from volume V1, an internal temporary snapshot, S2, is created from copying S1. V1 now has two snapshots, S1 and S2. Although we delete V1, V2 and S1 from OpenStack Block Storage (cinder), S2 is not deleted. This causes both V1 and S2 to remain on the VNX storage.

With this update, the temporary snapshot, S2, is deleted, and V1 can be successfully deleted.

BZ#1854950

Before this update, instances were unable to access their volumes after upgrading from RHOSP 10 to RHOSP 13, because the NFS share being used as a backend for OpenStack Block Storage (cinder) was not unmounted before migrating the OpenStack Block Storage services from the host to the containers. Therefore, when the containerized service started up and changed the ownership of all files in OpenStack Block Storage service directory, it also changed the ownership of files on the NFS share.

With this update, OpenStack Block Storage NFS shares are unmounted prior to upgrading the services to run in containers. This resolves the issue, and instances can now access their volumes after upgrading to RHOSP 13.

BZ#1861084

Before this update, if the OpenStack Shared File Systems (manila) service was configured with VServer-scoped ONTAP credentials it caused share provisioning to fail. This was due to a recent change to the NetApp ONTAP driver causing the share manager service to become stuck in a restart loop while trying to determine the storage system capabilities.

With this update, the NetApp ONTAP driver now checks for Vserver-scoped NetApp users and adds a fall-back path for determining storage system capabilities, which resolves the issue. The OpenStack Shared File Systems share manager service can now successfully determine the storage system capabilities, and share provisioning succeeds.

BZ#1862105

Before this update, initial connection errors with the agent were disrupting the retry logic, which sometimes resulted in the agent failing to communicate with the Ironic services, and logging a misleading TypeError to the agent console.

With this update, the exception handling has been fixed to explicitly handle known possible connection and lookup failure cases, and the logging has been updated to provide clarity on what is happening with the agent. Connections are now retried as designed by the agent and logging should no longer report just a TypeError in the event of an unexpected failure.

BZ#1867817
Before this update, using the ceilometer-metrics-qdr.yaml environment file resulted in a standalone Redis configuration rather than clustered redis instances as required by OpenStack Telemetry (ceilometer). This update now uses the correct services file in the resource registry to resolve the issue.
BZ#1847305

During startup, the ironic-conductor service could lose the reservation lock on a bare metal node for work accepted early in the ironic-conductor restart process. Losing the lock caused a race condition for work submitted to an OpenStack Bare Metal Provisioning (ironic) deployment during a restart of the ironic-conductor service, which resulted in requests failing with a "NodeNotLocked" error.

With this update, the database clean-up checks are performed before work can be accepted by an ironic-conductor process, which resolves the issue.

3.14.2. Enhancements

This release of Red Hat OpenStack Platform features the following enhancements:

BZ#1802038
This enhancement adds support for an external Red Hat Ceph Storage 4.1 cluster.
BZ#1845802

This enhancement adds a new configuration option to the Networking (neutron) service, http_retries, which you can use to configure the number of times API calls to the Compute (nova) service or OpenStack Bare Metal Provisioning (ironic) service should be retried if the first attempt fails. By default, the API calls are retried 3 times in case of failure.

With this enhancement, the Networking service can retry API requests to prevent errors in booting instances, for example, to ensure that the Compute service receives notification that a port is ready for use.

BZ#1815202

When using the config-download Tech Preview functionality, the generated Ansible playbooks do not include a default ansible.cfg that is tailored to the config-download playbooks. The default Ansible settings are not not ideal for large scale deployments.

This enhancement allows you to use the following command to generate an ansible.cfg that can be used with the config-download playbooks:

$ openstack tripleo config generate ansible

3.14.3. Known Issues

These known issues exist in Red Hat OpenStack Platform at this time:

BZ#1891014

There is currently a known issue with TLS Everywhere environments when live migrating instances during a minor update.

With the introduction of support for full QEMU-native TLS encryption when live migrating (BZ1754791), instance live migration is failing when performing a minor update on a RHOSP deployment that has running instances. This is because the certificates for the TLS NBD block migration, that do not already exist in the libvirtd container, are created during the update. The certificates are merged into the container directory tree during creation of the libvirt container, instead of being directly bind mounted from the host. Therefore, the QEMU processes of the instances that need migrated during the update do not get the new certificate automatically and the NBD setup process fails with the following error:

libvirtError: internal error: unable to execute QEMU command 'object-add': Unable to access credentials /etc/pki/qemu/ca-cert.pem: No such file or directory

Live migration works for instances created after the update.

Workaround:

You can use one of the following options to workaround this issue:

  • Stop and start the instances that fail to live migrate after the update is complete, so that new QEMU processes get created by libvirt container that has the certificate details.
  • Add the following configuration to the overcloud to disable TLS transport encryption for NBD, and deploy the overcloud:

      parameter_defaults:
        UseTLSTransportForNbd: False
BZ#1726270

There is a known issue that causes the glance db purge command to fail with an IntegrityError and a message similar to “Cannot delete or update a parent row: a foreign key constraint fails” when you try to purge records from the images table when related records have not yet been purged from other tables.

Workaround:

Manually delete the records from other tables before you purge the records from the images table.

3.15. Red Hat OpenStack Platform 13 Maintenance Release - December 16, 2020

These release notes highlight technology preview items, recommended practices, known issues, and deprecated functionality to be taken into consideration when deploying this release of Red Hat OpenStack Platform.

3.15.1. Bug Fix

These bugs were fixed in this release of Red Hat OpenStack Platform:

BZ#1879531

Before this update, when a Red Hat OpenStack Platform (RHOSP) user configured their containers to use the latest tag, RHOSP did not fetch nor rebuild these containers to use the updated container images.

With this update the issue is resolved. Now, anytime a user runs a deployment action (including an update), RHOSP always fetches the container images and checks the image ID for each running container to determine if it should be rebuilt to consume the latest image. RHOSP restarts any containers that it updates.

Important

This update is a change from previous versions for how RHOSP manages container updates. In past versions, RHOSP would only check if the image existed. Now, RHOSP always refreshes containers during deployment actions, and restarts any containers that it updates. For this reason, you should not reuse tags, like latest and always use the --tag-from-labels option, unless you are controlling container tags with a Red Hat Satellite deployment.

3.16. Red Hat OpenStack Platform 13 Maintenance Release - March 17, 2021

These release notes highlight technology preview items, recommended practices, known issues, and deprecated functionality to be taken into consideration when deploying this release of Red Hat OpenStack Platform.

3.16.1. Bug Fix

These bugs were fixed in this release of Red Hat OpenStack Platform:

BZ#1908366

This update fixes an incompatibility that caused VxFlex volume detachment attempts to fail.

A recent change in VxFlex cinder volume credentialing methods was not backward compatible with pre-existing volume attachments. If a VxFlex volume attachment was made before the credentialing method change, attempts to detach the volume failed.

Now the detachments do not fail.

3.16.2. Known Issues

These known issues exist in Red Hat OpenStack Platform at this time:

BZ#1841371

When transferring a volume with snapshots to another user, the volume is transferred but the snapshots remain owned by the previous user.

As a workaround, manage snapshots manually in Red Hat Openstack Platform 13. See "Managing instance snapshots" [1] in the Red Hat OpenStack Instances and Images Guide.

[1] https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/instances_and_images_guide/index#section-instance-snapshots

3.17. Red Hat OpenStack Platform 13 Maintenance Release - June 16, 2021

These release notes highlight technology preview items, recommended practices, known issues, and deprecated functionality to be taken into consideration when deploying this release of Red Hat OpenStack Platform.

3.17.1. Bug Fix

These bugs were fixed in this release of Red Hat OpenStack Platform:

BZ#1888417

Before this update, API calls to the NetApp SolidFire back end for the Block Storage service (cinder) could fail with a xNotPrimary error. This type of error occurred when an operation was made to a volume at the same time that SolidFire automatically moved connections to rebalance the cluster workload.

With this update, a SolidFire driver patch adds the xNotPrimary exception to the list of exceptions that can be retried.

BZ#1888469

Before this update, users experienced timeouts in certain environments, mostly when volumes were too big. Often these multi-terabyte volumes experienced poor network performance or upgrade issues that involved the SolidFire cluster.

With this update, two timeout settings have been added to the SolidFire driver to allow users to set the appropriate timeouts for their environment.

BZ#1914590

Before this update, when a Block Storage service (cinder) API response was lost, the NetApp SolidFire back end created an unused duplicate volume.

With this update, a patch to the SolidFire driver first checks if the volume name already exists before trying to create it. The patch also checks for volume creation immediately after it detects a read timeout, and prevents invalid API calls.

BZ#1934440

Before this update, the Service Telemetry Framework (STF) client could not connect to the STF server, because the latest version of Red Hat AMQ Interconnect does not allow TLS connections without a CA certificate.

This update corrects this problem by providing a new Orchestration service (heat) parameter, MetricsQdrSSLProfiles.

To obtain a Red Hat OpenShift TLS certificate, enter these commands:

$ oc get secrets
$ oc get secret/default-interconnect-selfsigned -o jsonpath='{.data.ca\.crt}' | base64 -d

Add the MetricsQdrSSLProfiles parameter with the contents of your Red Hat OpenShift TLS certificate to a custom environment file:

MetricsQdrSSLProfiles:
    -   name: sslProfile
        caCertFileContent: |
           -----BEGIN CERTIFICATE-----
           ...
           TOpbgNlPcz0sIoNK3Be0jUcYHVMPKGMR2kk=
           -----END CERTIFICATE-----

Then, redeploy your overcloud with the openstack overcloud deploy command.

BZ#1940153

Before this update, when using the Block Storage service (cinder) to create a large number of instances (bootable volumes) from snapshots on HP3Par Storage back end servers, timeouts occurred. An HP variable (convert_to_base) was set to true which caused HP3Par to create a thick volume of the original volume. This was an unnecessary and unwanted action.

With this update, a newer HP driver (4.0.11) has been backported to RHOSP 13 that includes a new spec:

hpe3par:convert_to_base=True | False
  • True (default) - The volume is created independently from the snapshot (HOS8 behavior).
  • False - The volume is created as a child of snapshot (HOS5 behavior).

    Usage

    You can set this new spec for HPE3Par volumes by using the cinder type-key command:

    cinder type-key <volume-type-name-or-ID> set hpe3par:convert_to_base=False | True

    Example

    $ cinder type-key myVolType set hpe3par:convert_to_base=False
    $ cinder create --name v1 --volume-type myVolType 10
    $ cinder snapshot-create --name s1 v1
    $ cinder snapshot-list
    $ cinder create --snapshot-id <snap_id> --volume-type myVolType --name v2 10

    Notes

    If the size of v2 is greater than size of v1, then the volume cannot be grown. In this case, to avoid any error, v2 is converted to a base volume (convert_to_base=True).

BZ#1943181

Before this update, when the Compute service (nova) made a terminate-connection call to the Block Storage service (cinder), single and multipath devices were not being flushed and there was a risk of data loss because these devices were in a leftover state.

The cause of this problem was that the os-brick disconnect_volume code assumed that the use_multipath parameter had the same value as the connector that was used in the original connect_volume call.

With this update, the Block Storage service changes how it performs disconnects. The os-brick code now properly flushes and detaches volumes when the multipath configuration in the Compute service changes for volumes attached to instances.

3.17.2. Enhancements

This release of Red Hat OpenStack Platform features the following enhancements:

BZ#1875508

This enhancement enables you to override the Orchestration service (heat) parameter, ServiceNetMap, for a role when you deploy the overcloud.

On spine-and-leaf (edge) deployments that use TLS-everywhere, hiera interpolation has been problematic when used to map networks on roles. Overriding the ServiceNetMap per role fixes the issues seen in some TLS-everywhere deployments, provides an easier interface, and replaces the need for the more complex hiera interpolation.

3.17.3. Release Notes

This section outlines important details about the release, including recommended practices and notable changes to Red Hat OpenStack Platform. You must take this information into account to ensure the best possible outcomes for your deployment.

BZ#1924727
The Block Storage backup service sometimes needs access to files on the host that would otherwise not be available in the container that runs the service. This enhancement adds the CinderBackupOptVolumes parameter, which you can use to specify additional container volume mounts for the Block Storage backup service.