OVS refuse to reload rules at restart

Issue

• After controller reboot, neutron-openvswitch-agent refuses to load all rules and this causes loss of connectivity to some instances.

• /var/log/neutron/openvswitch-agent.log contains errors similar to this one:

2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-1e9c4d37-3d5b-47aa-b975-8b1a6fc3b533 - - - - -] Error while processing VIF ports
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent Traceback (most recent call last):
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2054, in rpc_loop
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     port_info, ovs_restarted)
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 154, in wrapper
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return f(*args, **kwargs)
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1658, in process_network_ports
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     failed_devices['added'] |= self._bind_devices(need_binding_devices)
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 897, in _bind_devices
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.conf.host)
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 181, in update_device_list
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     agent_id=agent_id, host=host)
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 185, in call
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     time.sleep(wait)
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     self.force_reraise()
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     six.reraise(self.type_, self.value, self.tb)
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/neutron/common/rpc.py", line 162, in call
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     return self._original_context.call(ctxt, method, **kwargs)
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=self.retry)
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     timeout=timeout, retry=retry)
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 584, in send
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     retry=retry)
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 573, in _send
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     result = self._waiter.wait(msg_id, timeout)
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 456, in wait
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     message = self.waiters.get(msg_id, timeout=timeout)
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 347, in get
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent     'to message ID %s' % msg_id)
2019-10-15 09:11:31.923 1007783 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent MessagingTimeout: Timed out waiting for a reply to message ID f1f909f439014e82bdcb6d1ab6824eea