Misconfigured tenant network causes neutron-dhcp-agent to continuously restart

Issue

• While doing some stress test on the Neutron configuration (creating 200 tenants, each with 2 private networks and 2 virtual routers), we noticed a high load (100% CPU usage) on the neutron-server process, even after all networks were created. After some research, we found the following on the dhcp_agent.log file:

2014-04-21 03:14:57.503 10044 ERROR neutron.agent.dhcp_agent [-] Unable to enable dhcp for a60ebdeb-0239-4fff-a8d0-b34a1084b0c3.
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent Traceback (most recent call last):
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/dhcp_agent.py", line 128, in call_driver
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent     getattr(driver, action)(**action_kwargs)
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/dhcp.py", line 167, in enable
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent     reuse_existing=True)
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/dhcp.py", line 730, in setup
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent     self._set_default_route(network)
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/dhcp.py", line 618, in _set_default_route
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent     device.route.add_gateway(subnet.gateway_ip)
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/ip_lib.py", line 368, in add_gateway
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent     self._as_root(*args)
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/ip_lib.py", line 217, in _as_root
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent     kwargs.get('use_root_namespace', False))
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/ip_lib.py", line 70, in _as_root
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent     namespace)
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/ip_lib.py", line 81, in _execute
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent     root_helper=root_helper)
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent   File "/usr/lib/python2.6/site-packages/neutron/agent/linux/utils.py", line 75, in execute
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent     raise RuntimeError(m)
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent RuntimeError: 
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-a60ebdeb-0239-4fff-a8d0-b34a1084b0c3', 'ip', 'route', 'replace', 'default', 'via', '192.168.1.1', 'dev', 'tapd198f0a7-cd']
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent Exit code: 2
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent Stdout: ''
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent Stderr: 'RTNETLINK answers: No such process\n'
2014-04-21 03:14:57.503 10044 TRACE neutron.agent.dhcp_agent 
2014-04-21 03:15:33.273 10044 INFO neutron.agent.dhcp_agent [-] Synchronizing state

• This error was being displayed every few minutes, and we kept seeing dnsmasq processes being stopped and restarted. The network was misconfigured, as it had a CIDR of x.x.x.x/24 and a gateway IP of x.x.x.x . Fixing this reduced the neutron-server load to normal levels.