Unable to create new VM after upgrade Openstack 16.2.5 -> 17.1.1
Issue
- We recently upgraded our openstack from 16.2.5 to 17.1.1 and all VMs creation are now failing:
(overcloud) [Fri Apr 05 11:01:01] [stack@undercloud:~]$ openstack server show test-upgrade-rhel8 --fit
+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hostname | test-upgrade-rhel8 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | instance-00000012 |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-w7801x10 |
| OS-EXT-SRV-ATTR:root_device_name | None |
| OS-EXT-SRV-ATTR:user_data | None |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | error |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| config_drive | |
| created | 2024-04-05T07:59:19Z |
| description | None |
| fault | {'code': 500, 'created': '2024-04-05T07:59:20Z', 'message': 'No valid host was found. There are not enough hosts available.', 'details': |
| | 'Traceback (most recent call last):\n File "/usr/lib/python3.9/site-packages/nova/conductor/manager.py", line 1552, in |
| | schedule_and_build_instances\n host_lists = self._schedule_instances(context, request_specs[0],\n File "/usr/lib/python3.9/site- |
| | packages/nova/conductor/manager.py", line 913, in _schedule_instances\n host_lists = self.query_client.select_destinations(\n File |
| | "/usr/lib/python3.9/site-packages/nova/scheduler/client/query.py", line 41, in select_destinations\n return |
| | self.scheduler_rpcapi.select_destinations(context, spec_obj,\n File "/usr/lib/python3.9/site-packages/nova/scheduler/rpcapi.py", line 160, in |
| | select_destinations\n return cctxt.call(ctxt, \'select_destinations\', **msg_args)\n File "/usr/lib/python3.9/site- |
| | packages/oslo_messaging/rpc/client.py", line 175, in call\n self.transport._send(self.target, msg_ctxt, msg,\n File |
| | "/usr/lib/python3.9/site-packages/oslo_messaging/transport.py", line 123, in _send\n return self._driver.send(target, ctxt, message,\n File |
| | "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in send\n return self._send(target, ctxt, message, |
| | wait_for_reply, timeout,\n File "/usr/lib/python3.9/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 673, in _send\n raise |
| | result\nnova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available.\nTraceback (most recent call |
| | last):\n\n File "/usr/lib/python3.9/site-packages/oslo_messaging/rpc/server.py", line 241, in inner\n return func(*args, **kwargs)\n\n |
| | File "/usr/lib/python3.9/site-packages/nova/scheduler/manager.py", line 203, in select_destinations\n selections = |
| | self.driver.select_destinations(ctxt, spec_obj,\n\n File "/usr/lib/python3.9/site-packages/nova/scheduler/filter_scheduler.py", line 93, in |
| | select_destinations\n host_selections = self._schedule(context, spec_obj, instance_uuids,\n\n File "/usr/lib/python3.9/site- |
| | packages/nova/scheduler/filter_scheduler.py", line 263, in _schedule\n self._ensure_sufficient_hosts(context, claimed_hosts, |
| | num_instances,\n\n File "/usr/lib/python3.9/site-packages/nova/scheduler/filter_scheduler.py", line 301, in _ensure_sufficient_hosts\n |
| | raise exception.NoValidHost(reason=reason)\n\nnova.exception.NoValidHost: No valid host was found. There are not enough hosts available.\n\n'} |
| flavor | disk='40', ephemeral='0', extra_specs.HA='HA_STD', original_name='g-4-4-std', ram='4096', swap='0', vcpus='4' |
| hostId | |
| host_status | |
| id | 32c8298e-c121-4f27-b91f-38cedc3a55e0 |
| image | RHEL_8_7 (9ff897c8-fb5e-4d31-80b3-ae74f06e8dd1) |
| key_name | ansible |
| locked | False |
| locked_reason | None |
| name | test-upgrade-rhel8 |
| project_id | 6cf763dffb0d4063b73460fc51b7d2ce |
| properties | |
| server_groups | [] |
| status | ERROR |
| tags | |
| trusted_image_certificates | None |
| updated | 2024-04-05T07:59:20Z |
| user_id | 2dc0f86a1ff3477fa6a33d2a5b378624 |
| volumes_attached | |
+-------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+
- We found the following scheduling failure in
/var/log/containers/nova/nova-scheduler.log:
2024-04-05 09:59:20.274 31 INFO nova.filters [req-5fe72817-f9b0-446c-8f0b-9ddb8b83db3c 2dc0f86a1ff3477fa6a33d2a5b378624 6cf763dffb0d4063b73460fc51b7d2ce - default default] Filter MetricsFilter returned 0 hosts
2024-04-05 09:59:20.276 31 INFO nova.filters [req-5fe72817-f9b0-446c-8f0b-9ddb8b83db3c 2dc0f86a1ff3477fa6a33d2a5b378624 6cf763dffb0d4063b73460fc51b7d2ce - default default] Filtering removed all hosts for the request with instance ID '32c8298e-c121-4f27-b91f-38cedc3a55e0'. Filter results: ['AggregateInstanceExtraSpecsFilter: (start: 2, end: 2)', 'AvailabilityZoneFilter: (start: 2, end: 2)', 'ComputeFilter: (start: 2, end: 2)', 'ComputeCapabilitiesFilter: (start: 2, end: 2)', 'ImagePropertiesFilter: (start: 2, end: 2)', 'ServerGroupAntiAffinityFilter: (start: 2, end: 2)', 'ServerGroupAffinityFilter: (start: 2, end: 2)', 'MetricsFilter: (start: 2, end: 0)']
Environment
- Red Hat OpenStack 17.1 (RHOSP)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.
