New VMs are unreachable when connect to a newly created network

Issue

• We have recently noticed that any new tenant networks created by neutron do not serve dhcp information to the VMs.

• Existing tenant networks behave as expected.

• New tenant networks also appear to be created successfully but we do not see dhcp replies from the containers. In fact, we can not ping anything from inside the containers - so it looks like there is some problem with the container connectivity.

• The strange thing is neutron removes all the provision blocks and sets the VM to ACTIVE.

• The following logs are seen in /var/log/containers/neutron/neutron-l3-agent.log:

2020-01-19 03:20:00.998 869619 ERROR neutron.agent.l3.dvr_local_router [-] DVR: Failed updating arp entry: NetlinkError: (105, 'No buffer space available')
2020-01-19 03:20:00.998 869619 ERROR neutron.agent.l3.dvr_local_router Traceback (most recent call last):
2020-01-19 03:20:00.998 869619 ERROR neutron.agent.l3.dvr_local_router   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 261, in _update_arp_entry
2020-01-19 03:20:00.998 869619 ERROR neutron.agent.l3.dvr_local_router     device.neigh.add(ip, mac)
2020-01-19 03:20:00.998 869619 ERROR neutron.agent.l3.dvr_local_router   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 846, in add
2020-01-19 03:20:00.998 869619 ERROR neutron.agent.l3.dvr_local_router     **kwargs)
2020-01-19 03:20:00.998 869619 ERROR neutron.agent.l3.dvr_local_router   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 988, in add_neigh_entry
2020-01-19 03:20:00.998 869619 ERROR neutron.agent.l3.dvr_local_router     **kwargs)
2020-01-19 03:20:00.998 869619 ERROR neutron.agent.l3.dvr_local_router   File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 207, in _wrap
2020-01-19 03:20:00.998 869619 ERROR neutron.agent.l3.dvr_local_router     return self.channel.remote_call(name, args, kwargs)
2020-01-19 03:20:00.998 869619 ERROR neutron.agent.l3.dvr_local_router   File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202, in remote_call
2020-01-19 03:20:00.998 869619 ERROR neutron.agent.l3.dvr_local_router     raise exc_type(*result[2])
2020-01-19 03:20:00.998 869619 ERROR neutron.agent.l3.dvr_local_router NetlinkError: (105, 'No buffer space available')