Random connectivity issues upon network redeployment with Red Hat OpenStack Director in a VXLAN underlay network with ARP suppression

Issue

Random connectivity issues upon network redeployment with Red Hat OpenStack Director. An update of the network configuration was triggered by applying:

NetworkDeploymentActions: ['CREATE','UPDATE']

This caused restarts of the network environment. After this, random connectivity issues occur. Nodes cannot ping each other seemingly randomly until the ARP caches are cleared with arp -d <IP address>. This also affects services. Once the ARP caches are populated correctly, pcs resource cleanup needs to be run on the controllers and on the computes openstack-service restart.

The datacenter's underlay network uses VXLAN with ARP suppression (in the datacenter, not the overcloud's VXLAN network).