"conf_read_file: error code 24" tracebacks occur when destroying too many VMs on the same compute node at the same time

Solution In Progress - Updated -

Issue

  • When destroying too many VMs on the same compute node at the same time, delete requests start failing with "conf_read_file: error code 24".

  • The CEPH storage backend is used for the VM block devices.

  • The following error messages might be seen in /var/log/nova/nova-compute.log:

019-05-07 12:09:49.203 4113 ERROR nova.compute.manager [req-b9e51e9f-ed92-4449-8d4d-8ddbe5367488 41b2e24ba2554adfbc56afe0b5fbda5b be0920719a534c38865b5da52c00092e - - -] [instance: 745edb73-2c77-4b76-bcd9-20253b1cb5b9] Failed to deallocate network for instance. Error: Unable to establish connection to http://10.10.10.10:9696/v2.0/ports.json?device_id=745edb73-2c77-4b76-bcd9-20253b1cb5b9: HTTPConnectionPool(host='10.10.10.10', port=9696): Max retries exceeded with url: /v2.0/ports.json?device_id=745edb73-2c77-4b76-bcd9-20253b1cb5b9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 24] Too many open files',))
2019-05-07 12:09:49.356 4113 ERROR nova.compute.manager [instance: 745edb73-2c77-4b76-bcd9-20253b1cb5b9] ConnectFailure: Unable to establish connection to http://10.10.10.10:9696/v2.0/ports.json?device_id=745edb73-2c77-4b76-bcd9-20253b1cb5b9: HTTPConnectionPool(host='10.10.10.10', port=9696): Max retries exceeded with url: /v2.0/ports.json?device_id=745edb73-2c77-4b76-bcd9-20253b1cb5b9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 24] Too many open files',))
2019-05-07 12:09:49.744 4113 ERROR oslo_messaging.rpc.server ConnectFailure: Unable to establish connection to http://10.10.10.10:9696/v2.0/ports.json?device_id=745edb73-2c77-4b76-bcd9-20253b1cb5b9: HTTPConnectionPool(host='10.10.10.10', port=9696): Max retries exceeded with url: /v2.0/ports.json?device_id=745edb73-2c77-4b76-bcd9-20253b1cb5b9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 24] Too many open files',))
2019-05-07 12:26:50.910 4113 ERROR oslo_service.periodic_task IOError: [Errno 24] Too many open files: '/var/lib/nova/instances/locks/nova-storage-registry-lock'
2019-05-07 13:08:22.910 4113 ERROR oslo_service.periodic_task IOError: [Errno 24] Too many open files: '/var/lib/nova/instances/locks/nova-storage-registry-lock'
2019-05-07 13:48:58.909 4113 ERROR oslo_service.periodic_task IOError: [Errno 24] Too many open files: '/var/lib/nova/instances/locks/nova-storage-registry-lock'
2019-05-07 13:52:22.543 4113 ERROR nova.compute.manager [req-22f65da8-62fd-4273-9dce-23eefde2a42c 97af81cdf77d416e93d7a7a33a872e75 11aa576ca2ce4d33907e8be97cdaae3a - - -] [instance: 23137a57-9389-4a6d-81d2-636b23540b16] Cannot reboot instance: [Errno 24] Too many open files: '/etc/machine-id'
2019-05-07 13:52:22.772 4113 ERROR oslo_messaging.rpc.server IOError: [Errno 24] Too many open files: '/etc/machine-id'
2019-05-07 13:52:26.435 4113 ERROR nova.compute.manager [req-4e96a09c-2974-48b8-8b71-fd8f2085aad6 97af81cdf77d416e93d7a7a33a872e75 11aa576ca2ce4d33907e8be97cdaae3a - - -] [instance: 23137a57-9389-4a6d-81d2-636b23540b16] Cannot reboot instance: [Errno 24] Too many open files: '/etc/machine-id'
2019-05-07 13:52:26.672 4113 ERROR oslo_messaging.rpc.server IOError: [Errno 24] Too many open files: '/etc/machine-id'

Environment

  • Red Hat OpenStack Platform 10.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content