Dnsmasq scale limitation for subnets in OpenStack Platform
Issue
Out of the box, the total number of subnets that can be created on the cloud is limited to less than 128, because by default the value of fs.inotify.max_user_instance
kernel parameter is set to 128.
Each subnet created in OpenStack Platform spawns a dnsmasq process on the controllers as follows:
nobody 290502 0.0 0.0 19756 1000 ? S Oct26 0:00 dnsmasq --no-hosts --no-resolv --strict-order --except-interface=lo --pid-file=/var/lib/neutron/dhcp/75a1b2ce-9478-4b8e-9626-731ce2ffc08b/pid --dhcp-hostsfile=/var/lib/neutron/dhcp/75a1b2ce-9478-4b8e-9626-731ce2ffc08b/host --addn-hosts=/var/lib/neutron/dhcp/75a1b2ce-9478-4b8e-9626-731ce2ffc08b/addn_hosts --dhcp-optsfile=/var/lib/neutron/dhcp/75a1b2ce-9478-4b8e-9626-731ce2ffc08b/opts --dhcp-leasefile=/var/lib/neutron/dhcp/75a1b2ce-9478-4b8e-9626-731ce2ffc08b/leases --dhcp-match=set:ipxe,175 --bind-interfaces --interface=tap53824559-bc --dhcp-range=set:tag0,10.10.0.0,static,86400s --dhcp-option-force=option:mtu,1450 --dhcp-lease-max=65536 --conf-file= --domain=openstacklocal
Each dnsmasq process uses one inotify socket to watch the /etc/resolv.conf
file. A few other inotify sockets are used by the system, thereby limiting the total number of subnets that can be successfully created to less than 128. When the user tries to create more subnets an error is seen in the neutron dhcp-agent logs:
2017-09-13 21:45:38.124 91663 ERROR neutron.agent.dhcp.agent [req-288477aa-318a-40c3-954e-dd6fc98c6c1b - - - - -] Unable to enable dhcp for bb6cdb16-72c0-4cc4-a316-69ebcd7633b2.: ProcessExecutionError: Exit code: 5; Stdin: ; Stdout: ; Stderr:
dnsmasq: failed to create inotify: Too many open files
2017-09-13 21:45:38.124 91663 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2017-09-13 21:45:38.124 91663 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 142, in call_driver
2017-09-13 21:45:38.124 91663 ERROR neutron.agent.dhcp.agent getattr(driver, action)(**action_kwargs)
2017-09-13 21:45:38.124 91663 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 218, in enable
2017-09-13 21:45:38.124 91663 ERROR neutron.agent.dhcp.agent self.spawn_process()
2017-09-13 21:45:38.124 91663 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 439, in spawn_process
2017-09-13 21:45:38.124 91663 ERROR neutron.agent.dhcp.agent self._spawn_or_reload_process(reload_with_HUP=False)
2017-09-13 21:45:38.124 91663 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 453, in _spawn_or_reload_process
2017-09-13 21:45:38.124 91663 ERROR neutron.agent.dhcp.agent pm.enable(reload_cfg=reload_with_HUP)
2017-09-13 21:45:38.124 91663 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/external_process.py", line 96, in enable
2017-09-13 21:45:38.124 91663 ERROR neutron.agent.dhcp.agent run_as_root=self.run_as_root)
2017-09-13 21:45:38.124 91663 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 903, in execute
2017-09-13 21:45:38.124 91663 ERROR neutron.agent.dhcp.agent log_fail_as_error=log_fail_as_error, **kwargs)
2017-09-13 21:45:38.124 91663 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 151, in execute
2017-09-13 21:45:38.124 91663 ERROR neutron.agent.dhcp.agent raise ProcessExecutionError(msg, returncode=returncode)
2017-09-13 21:45:38.124 91663 ERROR neutron.agent.dhcp.agent ProcessExecutionError: Exit code: 5; Stdin: ; Stdout: ; Stderr:
2017-09-13 21:45:38.124 91663 ERROR neutron.agent.dhcp.agent dnsmasq: failed to create inotify: Too many open files
2017-09-13 21:45:38.124 91663 ERROR neutron.agent.dhcp.agent
2017-09-13 21:45:38.124 91663 ERROR neutron.agent.dhcp.agent
2017-09-13 21:45:38.260 91663 ERROR neutron.agent.linux.utils [req-d0ade748-22ea-4a45-ba10-277d45f20981 - - - - -] Exit code: 5; Stdin: ; Stdout: ; Stderr:
dnsmasq: failed to create inotify: Too many open files
Environment
- Red Hat OpenStack Platform 10.
- Red Hat OpenStack Platform 11.
- Red Hat OpenStack Platform 12.
- Red Hat Enterprise Linux 7.3.
- Red Hat Enterprise Linux 7.4.
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.