Overcloud deployment failed for "No valid host was found...code:500"

Latest response

Hi,
I tried to deploy my OSP9 overcloud in nested virtual environment. That means the control, compute and ceph-storage nodes all are VMs. The undercloud node is also a VM. So totally there are 10 VMs running in my VMhost. I successfully installed the director nodes in one of VM node (called OSP-D as below). Then when I try to deploy overcloud from OSP-D, I always get error Message: No valid host was found. There are not enough hosts available., Code: 500. The introspection was successful. And from "nova hypervisor-stats", the resource seems quite OK before deployment. What could be the problem?

Enclosed please find the nova-api.log and nova-conductor.log. Thanks!

[root@redhatvmhost scripts]# virsh list --all

Id Name State

41 OSP-D running
220 overcloud-node1 running
221 overcloud-node2 running
222 overcloud-node3 running
223 overcloud-node4 running
224 overcloud-node5 running
225 overcloud-node6 running
226 overcloud-node7 running
227 overcloud-node8 running
228 overcloud-node9 running

[stack@osp-d ~]$ ironic node-list
+--------------------------------------+-----------------+--------------------------------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+-----------------+--------------------------------------+-------------+--------------------+-------------+
| 4bbcd3c1-3c6c-4826-8e72-52a9b71258d6 | overcloud-node1 | fc677759-d2ff-4419-974c-f7e6f27a8176 | power on | available | False |
| 8989f7c1-5012-43c3-a653-1e5f1ff7fe18 | overcloud-node2 | 7d8a6365-f6ac-4fb1-85b6-e08f65b84dd5 | power on | available | False |
| 14aef095-12ca-41de-93c4-668428f923fd | overcloud-node3 | 89c74b76-ffec-4f51-a411-d215a6dd5111 | power on | available | False |
| 0078f452-9f5f-4c79-8885-10e1169c6060 | overcloud-node4 | 75d86d28-f237-42c7-8bed-82adc6cf8f78 | power on | available | False |
| 4772d9b4-099d-48f0-9f8f-8dd39d21e8a3 | overcloud-node5 | 75637932-ba6a-4e6f-9ff0-571f2a27807a | power on | available | False |
| 853c7540-cf6e-4bd9-b065-9aba340af5cf | overcloud-node6 | 3f796335-5c41-44b4-a8f8-db5a9fde0cdf | power on | available | False |
| 62182238-ae17-4bd2-95f9-40a5e9237045 | overcloud-node7 | 65c4f34e-8a03-4731-9d80-ed6a0d91ea8a | power on | available | False |
| 96744a84-5710-4a6f-a53e-ac628a459c55 | overcloud-node8 | 0f75c8aa-a885-48f3-b7f3-324e77acac4a | power on | available | False |
| 12029189-7f79-401e-9634-93d8b5a5950b | overcloud-node9 | c818667e-5dcb-4a50-a362-81f4a23d6efe | power on | available | False |
+--------------------------------------+-----------------+--------------------------------------+-------------+--------------------+-------------+
[stack@osp-d ~]$ openstack stack list
+--------------------------------------+------------+---------------+---------------------+--------------+
| ID | Stack Name | Stack Status | Creation Time | Updated Time |
+--------------------------------------+------------+---------------+---------------------+--------------+
| 8e95fa28-8e97-47e8-97c6-3ba46315912a | overcloud | CREATE_FAILED | 2016-11-25T10:25:30 | None |
+--------------------------------------+------------+---------------+---------------------+--------------+

Deployment logs

2016-11-25 10:27:35 [NovaCompute]: CREATE_FAILED ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:36 [Controller]: CREATE_FAILED ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:36 [2]: CREATE_FAILED ResourceInError: resources[2].resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:36 [NovaCompute]: CREATE_FAILED ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:36 [NovaCompute]: CREATE_FAILED ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:36 [overcloud-Compute-eamlrfwlozpy-2-gxh6ct45rrnn]: CREATE_FAILED Resource CREATE failed: ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:37 [overcloud-Controller-sm4p5ipb4nqm-1-6wsyzem6omd2]: CREATE_FAILED Resource CREATE failed: ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:37 [1]: CREATE_FAILED ResourceInError: resources[1].resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:37 [overcloud-Compute-eamlrfwlozpy-1-yuewpgvn4qzo]: CREATE_FAILED Resource CREATE failed: ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:37 [overcloud-Compute-eamlrfwlozpy-0-omrvilffysoh]: CREATE_FAILED Resource CREATE failed: ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:38 [1]: CREATE_FAILED ResourceInError: resources[1].resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:38 [0]: CREATE_FAILED ResourceInError: resources[0].resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:39 [Controller]: CREATE_IN_PROGRESS state changed
2016-11-25 10:27:39 [Controller]: CREATE_IN_PROGRESS state changed
2016-11-25 10:27:39 [overcloud-Compute-eamlrfwlozpy]: CREATE_FAILED Resource CREATE failed: ResourceInError: resources[2].resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:40 [Compute]: CREATE_FAILED ResourceInError: resources.Compute.resources[2].resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:41 [Controller]: CREATE_FAILED ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:41 [Controller]: CREATE_FAILED ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:42 [overcloud-Controller-sm4p5ipb4nqm-0-lgjkqxoadfeu]: CREATE_FAILED Resource CREATE failed: ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:42 [overcloud-Controller-sm4p5ipb4nqm-2-vlrjf4xfkrht]: CREATE_FAILED Resource CREATE failed: ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:43 [0]: CREATE_FAILED ResourceInError: resources[0].resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:44 [2]: CREATE_FAILED ResourceInError: resources[2].resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:45 [Controller]: CREATE_FAILED ResourceInError: resources.Controller.resources[1].resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
2016-11-25 10:27:45 [overcloud-Controller-sm4p5ipb4nqm]: CREATE_FAILED Resource CREATE failed: ResourceInError: resources[1].resources.Controller: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500"
Stack overcloud CREATE_FAILED
Deployment failed: Heat Stack create failed.

Nova-conductor.log

2016-11-25 11:27:30.104 28499 WARNING nova.scheduler.utils [req-dd8efd32-4319-4a80-a3ad-31fac99c35b6 8449634a091b4d2f98151f93b15ce30c 961d8f774bdd4a8bad023c95dd37cc1d - - -] [instance: 6f7a50e7-d7a5-414c-abc6-211fd78ae363] Setting instance to ERROR state.
2016-11-25 11:27:30.148 28499 DEBUG nova.objects.instance [req-dd8efd32-4319-4a80-a3ad-31fac99c35b6 8449634a091b4d2f98151f93b15ce30c 961d8f774bdd4a8bad023c95dd37cc1d - - -] Lazy-loading 'metadata' on Instance uuid 6f7a50e7-d7a5-414c-abc6-211fd78ae363 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:895
2016-11-25 11:27:30.180 28499 DEBUG nova.objects.instance [req-dd8efd32-4319-4a80-a3ad-31fac99c35b6 8449634a091b4d2f98151f93b15ce30c 961d8f774bdd4a8bad023c95dd37cc1d - - -] Lazy-loading 'info_cache' on Instance uuid 6f7a50e7-d7a5-414c-abc6-211fd78ae363 obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:895
2016-11-25 11:27:30.217 28499 DEBUG nova.network.neutronv2.api [req-dd8efd32-4319-4a80-a3ad-31fac99c35b6 8449634a091b4d2f98151f93b15ce30c 961d8f774bdd4a8bad023c95dd37cc1d - - -] [instance: 6f7a50e7-d7a5-414c-abc6-211fd78ae363] deallocate_for_instance() deallocate_for_instance /usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py:862
2016-11-25 11:27:30.218 28499 DEBUG keystoneauth.session [req-dd8efd32-4319-4a80-a3ad-31fac99c35b6 8449634a091b4d2f98151f93b15ce30c 961d8f774bdd4a8bad023c95dd37cc1d - - -] REQ: curl -g -i -X GET http://192.0.2.1:9696/v2.0/ports.json?device_id=6f7a50e7-d7a5-414c-abc6-211fd78ae363 -H "User-Agent: python-neutronclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}5e0b250b3cdf1389c161d41255eca84ebb746c83" _http_log_request /usr/lib/python2.7/site-packages/keystoneauth1/session.py:248
2016-11-25 11:27:30.238 28499 DEBUG keystoneauth.session [req-dd8efd32-4319-4a80-a3ad-31fac99c35b6 8449634a091b4d2f98151f93b15ce30c 961d8f774bdd4a8bad023c95dd37cc1d - - -] RESP: [200] Content-Type: application/json; charset=UTF-8 Content-Length: 13 X-Openstack-Request-Id: req-0a78270f-da77-43c4-9a46-82dcd53764af Date: Fri, 25 Nov 2016 10:27:30 GMT Connection: keep-alive
RESP BODY: {"ports": []}
_http_log_response /usr/lib/python2.7/site-packages/keystoneauth1/session.py:277
2016-11-25 11:27:30.239 28499 DEBUG nova.network.neutronv2.api [req-dd8efd32-4319-4a80-a3ad-31fac99c35b6 8449634a091b4d2f98151f93b15ce30c 961d8f774bdd4a8bad023c95dd37cc1d - - -] [instance: 6f7a50e7-d7a5-414c-abc6-211fd78ae363] Instance cache missing network info. _get_preexisting_port_ids /usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py:1720
2016-11-25 11:27:30.239 28499 DEBUG nova.network.base_api [req-dd8efd32-4319-4a80-a3ad-31fac99c35b6 8449634a091b4d2f98151f93b15ce30c 961d8f774bdd4a8bad023c95dd37cc1d - - -] [instance: 6f7a50e7-d7a5-414c-abc6-211fd78ae363] Updating instance_info_cache with network_info: [] update_instance_cache_with_nw_info /usr/lib/python2.7/site-packages/nova/network/base_api.py:43
2016-11-25 11:27:34.312 28503 WARNING nova.scheduler.utils [req-c9135a9a-c01f-41e1-8247-8f07925551e3 8449634a091b4d2f98151f93b15ce30c 961d8f774bdd4a8bad023c95dd37cc1d - - -] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):

File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 150, in inner
return func(*args, **kwargs)

File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 104, in select_destinations
dests = self.driver.select_destinations(ctxt, spec_obj)

File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 74, in select_destinations
raise exception.NoValidHost(reason=reason)

NoValidHost: No valid host was found. There are not enough hosts available.

Attachments

GY

Responses