"conf_read_file: error code 24" tracebacks occur when destroying too many VMs on the same compute node at the same time

Solution In Progress - Updated -

Issue

  • When destroying too many VMs on the same compute node at the same time, delete requests start failing with "conf_read_file: error code 24".

  • The CEPH storage backend is used for the VM block devices.

  • The following error messages might be seen in /var/log/nova/nova-compute.log:

019-05-07 12:09:49.203 4113 ERROR nova.compute.manager [req-b9e51e9f-ed92-4449-8d4d-8ddbe5367488 41b2e24ba2554adfbc56afe0b5fbda5b be0920719a534c38865b5da52c00092e - - -] [instance: 745edb73-2c77-4b76-bcd9-20253b1cb5b9] Failed to deallocate network for instance. Error: Unable to establish connection to http://10.10.10.10:9696/v2.0/ports.json?device_id=745edb73-2c77-4b76-bcd9-20253b1cb5b9: HTTPConnectionPool(host='10.10.10.10', port=9696): Max retries exceeded with url: /v2.0/ports.json?device_id=745edb73-2c77-4b76-bcd9-20253b1cb5b9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 24] Too many open files',))
2019-05-07 12:09:49.356 4113 ERROR nova.compute.manager [instance: 745edb73-2c77-4b76-bcd9-20253b1cb5b9] ConnectFailure: Unable to establish connection to http://10.10.10.10:9696/v2.0/ports.json?device_id=745edb73-2c77-4b76-bcd9-20253b1cb5b9: HTTPConnectionPool(host='10.10.10.10', port=9696): Max retries exceeded with url: /v2.0/ports.json?device_id=745edb73-2c77-4b76-bcd9-20253b1cb5b9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 24] Too many open files',))
2019-05-07 12:09:49.744 4113 ERROR oslo_messaging.rpc.server ConnectFailure: Unable to establish connection to http://10.10.10.10:9696/v2.0/ports.json?device_id=745edb73-2c77-4b76-bcd9-20253b1cb5b9: HTTPConnectionPool(host='10.10.10.10', port=9696): Max retries exceeded with url: /v2.0/ports.json?device_id=745edb73-2c77-4b76-bcd9-20253b1cb5b9 (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 24] Too many open files',))
2019-05-07 12:26:50.910 4113 ERROR oslo_service.periodic_task IOError: [Errno 24] Too many open files: '/var/lib/nova/instances/locks/nova-storage-registry-lock'
2019-05-07 13:08:22.910 4113 ERROR oslo_service.periodic_task IOError: [Errno 24] Too many open files: '/var/lib/nova/instances/locks/nova-storage-registry-lock'
2019-05-07 13:48:58.909 4113 ERROR oslo_service.periodic_task IOError: [Errno 24] Too many open files: '/var/lib/nova/instances/locks/nova-storage-registry-lock'
2019-05-07 13:52:22.543 4113 ERROR nova.compute.manager [req-22f65da8-62fd-4273-9dce-23eefde2a42c 97af81cdf77d416e93d7a7a33a872e75 11aa576ca2ce4d33907e8be97cdaae3a - - -] [instance: 23137a57-9389-4a6d-81d2-636b23540b16] Cannot reboot instance: [Errno 24] Too many open files: '/etc/machine-id'
2019-05-07 13:52:22.772 4113 ERROR oslo_messaging.rpc.server IOError: [Errno 24] Too many open files: '/etc/machine-id'
2019-05-07 13:52:26.435 4113 ERROR nova.compute.manager [req-4e96a09c-2974-48b8-8b71-fd8f2085aad6 97af81cdf77d416e93d7a7a33a872e75 11aa576ca2ce4d33907e8be97cdaae3a - - -] [instance: 23137a57-9389-4a6d-81d2-636b23540b16] Cannot reboot instance: [Errno 24] Too many open files: '/etc/machine-id'
2019-05-07 13:52:26.672 4113 ERROR oslo_messaging.rpc.server IOError: [Errno 24] Too many open files: '/etc/machine-id'

Environment

  • Red Hat OpenStack Platform 10.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In