Openstack instances on error due to too many open files

Solution In Progress - Updated -

Environment

  • Red Hat OpenStack Platform 16.2 (RHOSP)

Issue

Several Nova libvirt instances from same compute node went to ERROR state while trying to attach a tap interface.

Resolution

  1. Increase default values in ContainerNovaLibvirtUlimit in your templates (eg: Double the current 'nofile=131072', 'nproc=126960'):
parameter_defaults:
  ContainerNovaLibvirtUlimit:
    default: ['nofile=262144', 'nproc=253920']

  1. Run deployment.

  2. Validate instance edition.

Root Cause

Due to a huge environment the limit of open files was reached. Number of guests bigger than 400 in one compute node.

Diagnostic Steps

  1. Check the error message in openstack server show <UUID similar to below:
(overcloud) [stack@director ~]$ openstack server show -f json 55f3aace-8b62-40dd-be02-e4ca103fa800
...
  "OS-EXT-SRV-ATTR:host": "compute05.domain.com",
  "OS-EXT-SRV-ATTR:hostname": "node3",
  "OS-EXT-SRV-ATTR:hypervisor_hostname": "compute05.domain.com",
  "OS-EXT-SRV-ATTR:instance_name": "instance-020211e8",
...
  "fault": {
    "code": 500,
    "created": "2025-06-16T07:00:51Z",
    "message": "libvirtError",
    "details": "Traceback (most recent call last):\n  File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 207, in decorated_function\n    return function(self, context, *args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 3718, in reboot_instance\n    do_reboot_instance(context, instance, block_device_info, reboot_type)\n  File \"/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py\", line 328, in inner\n    return f(*args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 3717, in do_reboot_instance\n    reboot_type)\n  File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 3808, in _reboot_instance\n    self._set_instance_obj_error_state(context, instance)\n  File \"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py\", line 220, in __exit__\n    self.force_reraise()\n  File \"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py\", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n  File \"/usr/lib/python3.6/site-packages/six.py\", line 693, in reraise\n    raise value\n  File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 3782, in _reboot_instance\n    bad_volumes_callback=bad_volumes_callback)\n  File \"/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py\", line 3365, in reboot\n    block_device_info)\n  File \"/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py\", line 3482, in _hard_reboot\n    vifs_already_plugged=True)\n  File \"/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py\", line 6683, in _create_domain_and_network\n    cleanup_instance_disks=cleanup_instance_disks)\n  File \"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py\", line 220, in __exit__\n    self.force_reraise()\n  File \"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py\", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n  File \"/usr/lib/python3.6/site-packages/six.py\", line 693, in reraise\n    raise value\n  File \"/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py\", line 6649, in _create_domain_and_network\n    post_xml_callback=post_xml_callback)\n  File \"/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py\", line 6578, in _create_domain\n    guest.launch(pause=pause)\n  File \"/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py\", line 149, in launch\n    self._encoded_xml, errors='ignore')\n  File \"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py\", line 220, in __exit__\n    self.force_reraise()\n  File \"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py\", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n  File \"/usr/lib/python3.6/site-packages/six.py\", line 693, in reraise\n    raise value\n  File \"/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py\", line 144, in launch\n    return self._domain.createWithFlags(flags)\n  File \"/usr/lib/python3.6/site-packages/eventlet/tpool.py\", line 190, in doit\n    result = proxy_call(self._autowrap, f, *args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/eventlet/tpool.py\", line 148, in proxy_call\n    rv = execute(f, *args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/eventlet/tpool.py\", line 129, in execute\n    six.reraise(c, e, tb)\n  File \"/usr/lib/python3.6/site-packages/six.py\", line 693, in reraise\n    raise value\n  File \"/usr/lib/python3.6/site-packages/eventlet/tpool.py\", line 83, in tworker\n    rv = meth(*args, **kwargs)\n  File \"/usr/lib64/python3.6/site-packages/libvirt.py\", line 1385, in createWithFlags\n    raise libvirtError('virDomainCreateWithFlags() failed')\nlibvirt.libvirtError: internal error: Unable to add port tap815bec00-93 to OVS bridge br-int\n"
  },
...

(overcloud) [stack@director ~]$ openstack server show -f json 47d4504d-174c-40cf-ac50-724c466dfb00
...
  "OS-EXT-SRV-ATTR:host": "compute05.domain.com",
  "OS-EXT-SRV-ATTR:hostname": "node2",
  "OS-EXT-SRV-ATTR:hypervisor_hostname": "compute05.domain.com",
  "OS-EXT-SRV-ATTR:instance_name": "instance-02021100",
...
  "fault": {
    "code": 500,
    "created": "2025-06-16T07:00:52Z",
    "message": "libvirtError",
...
raise libvirtError('virDomainCreateWithFlags() failed')\nlibvirt.libvirtError: internal error: Unable to add port tapabd141df-00 to OVS bridge br-int\n"
...

(overcloud) [stack@director ~]$ openstack server show -f json 38504ee3-1536-4ca4-8246-0aa8a5eb3a00
...
  "OS-EXT-SRV-ATTR:host": "compute05.domain.com",
  "OS-EXT-SRV-ATTR:hostname": "node1",
  "OS-EXT-SRV-ATTR:hypervisor_hostname": "compute05.domain.com",
  "OS-EXT-SRV-ATTR:instance_name": "instance-02021100",
...
  "fault": {
    "code": 500,
    "created": "2025-06-16T07:00:43Z",
    "message": "libvirtError",
    "details": "Traceback (most recent call last):\n  File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 207, in decorated_function\n    return function(self, context, *args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 3718, in reboot_instance\n    do_reboot_instance(context, instance, block_device_info, reboot_type)\n  File \"/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py\", line 328, in inner\n    return f(*args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 3717, in do_reboot_instance\n    reboot_type)\n  File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 3808, in _reboot_instance\n    self._set_instance_obj_error_state(context, instance)\n  File \"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py\", line 220, in __exit__\n    self.force_reraise()\n  File \"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py\", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n  File \"/usr/lib/python3.6/site-packages/six.py\", line 693, in reraise\n    raise value\n  File \"/usr/lib/python3.6/site-packages/nova/compute/manager.py\", line 3782, in _reboot_instance\n    bad_volumes_callback=bad_volumes_callback)\n  File \"/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py\", line 3365, in reboot\n    block_device_info)\n  File \"/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py\", line 3482, in _hard_reboot\n    vifs_already_plugged=True)\n  File \"/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py\", line 6683, in _create_domain_and_network\n    cleanup_instance_disks=cleanup_instance_disks)\n  File \"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py\", line 220, in __exit__\n    self.force_reraise()\n  File \"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py\", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n  File \"/usr/lib/python3.6/site-packages/six.py\", line 693, in reraise\n    raise value\n  File \"/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py\", line 6649, in _create_domain_and_network\n    post_xml_callback=post_xml_callback)\n  File \"/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py\", line 6578, in _create_domain\n    guest.launch(pause=pause)\n  File \"/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py\", line 149, in launch\n    self._encoded_xml, errors='ignore')\n  File \"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py\", line 220, in __exit__\n    self.force_reraise()\n  File \"/usr/lib/python3.6/site-packages/oslo_utils/excutils.py\", line 196, in force_reraise\n    six.reraise(self.type_, self.value, self.tb)\n  File \"/usr/lib/python3.6/site-packages/six.py\", line 693, in reraise\n    raise value\n  File \"/usr/lib/python3.6/site-packages/nova/virt/libvirt/guest.py\", line 144, in launch\n    return self._domain.createWithFlags(flags)\n  File \"/usr/lib/python3.6/site-packages/eventlet/tpool.py\", line 190, in doit\n    result = proxy_call(self._autowrap, f, *args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/eventlet/tpool.py\", line 148, in proxy_call\n    rv = execute(f, *args, **kwargs)\n  File \"/usr/lib/python3.6/site-packages/eventlet/tpool.py\", line 129, in execute\n    six.reraise(c, e, tb)\n  File \"/usr/lib/python3.6/site-packages/six.py\", line 693, in reraise\n    raise value\n  File \"/usr/lib/python3.6/site-packages/eventlet/tpool.py\", line 83, in tworker\n    rv = meth(*args, **kwargs)\n  File \"/usr/lib64/python3.6/site-packages/libvirt.py\", line 1385, in createWithFlags\n    raise libvirtError('virDomainCreateWithFlags() failed')\nlibvirt.libvirtError: Unable to open /dev/net/tun, is tun module loaded?: Too many open files\n"
  },
...
  1. The log file shows many occurrences of Too many open files error message around that time in compute node.
grep "Too many open files" /var/log/containers/libvirtd/libvirtd.log
...
2025-06-16 06:53:59.767+0000: 1490928: error : virPipeImpl:1901 : Unable to create pipes: Too many open files
2025-06-16 06:53:59.767+0000: 1490928: error : virNetDevOpenvswitchAddPort:185 : internal error: Unable to add port tap479b0f00-00 to OVS bridge br-int
2025-06-16 06:54:31.999+0000: 846322: error : getDevNull:359 : cannot open /dev/null: Too many open files
2025-06-16 06:54:36.178+0000: 1490928: error : getDevNull:359 : cannot open /dev/null: Too many open files
...
grep "Too many open files" /var/log/containers/nova/nova-compute.log
...
2025-06-16 06:54:34.280 7 ERROR nova.virt.libvirt.driver [req-f83bc57c-a84a-4c93-89ab-0db8dce0c89b a12ab43a55134d45aac1e3df09ab5deb 893193f681194ca1966becc9ab6cfad1 - default default] [instance: 85273add-029a-4d45-a76e-5fec90fcafd1] Failed to start libvirt guest: libvirt.libvirtError: cannot open /dev/null: Too many open files
...
  1. If instances are manually stopped and started, it will be running.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments