Stack overcloud CREATE_FAILED - "Message: No valid host was found. , Code: 500"
I was trying to deploy RHOSP 13 on RHEV 4.3 platform. But every-time it is getting failed during openstack overcloud deploy step. I found lot of articles about this error in google and also Red Hat solution but nothing helped. I have enough resource on RHEV Hypervisor node (40 Cores CPU, 128GB RAM and 1TB Disk) Any further help highly appreciated.
openstack overcloud deploy error:
Stack overcloud CREATE_FAILED
overcloud.Controller.1.Controller:
resource_type: OS::TripleO::ControllerServer
physical_resource_id: e36a191c-041c-4d06-99ef-03fdac37e137
status: CREATE_FAILED
status_reason: |
ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. , Code: 500"
overcloud.Controller.0.Controller:
resource_type: OS::TripleO::ControllerServer
physical_resource_id: e40fff3c-1643-4150-9557-496ca764a8e4
status: CREATE_FAILED
status_reason: |
ResourceInError: resources.Controller: Went to status ERROR due to "Message: No valid host was found. , Code: 500"
overcloud.Compute.1.NovaCompute:
resource_type: OS::TripleO::ComputeServer
physical_resource_id: 5e7a230a-358c-457c-a733-45c0a9bd62d6
status: CREATE_FAILED
status_reason: |
ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. , Code: 500"
overcloud.Compute.0.NovaCompute:
resource_type: OS::TripleO::ComputeServer
physical_resource_id: fea4c120-ac3b-40fa-806d-a74f9cab2cf4
status: CREATE_FAILED
status_reason: |
ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: No valid host was found. , Code: 500"
overcloud.CephStorage.1.CephStorage:
resource_type: OS::TripleO::CephStorageServer
physical_resource_id: db62a251-621f-4a11-86f3-713f0d78280d
status: CREATE_FAILED
status_reason: |
ResourceInError: resources.CephStorage: Went to status ERROR due to "Message: No valid host was found. , Code: 500"
overcloud.CephStorage.0.CephStorage:
resource_type: OS::TripleO::CephStorageServer
physical_resource_id: 9997e37e-eb54-4361-be9c-7d04f095f651
status: CREATE_FAILED
status_reason: |
ResourceInError: resources.CephStorage: Went to status ERROR due to "Message: No valid host was found. , Code: 500"
Heat Stack create failed.
Heat Stack create failed.
(undercloud) stack@undercloud.example.com:/home/stack>openstack flavor list
+--------------------------------------+---------------+-------+------+-----------+-------+-----------+
| ID | Name | RAM | Disk | Ephemeral | VCPUs | Is Public |
+--------------------------------------+---------------+-------+------+-----------+-------+-----------+
| 05bb09dd-65d7-43e1-a732-8e52a164ab4c | block-storage | 4096 | 40 | 0 | 1 | True |
| 121f9b30-1536-4beb-82fc-4f10b3daa673 | ceph-storage | 16384 | 40 | 0 | 1 | True |
| 52b7a88a-0cd2-4986-875b-edbc1bf1aa8c | baremetal | 4096 | 40 | 0 | 1 | True |
| bda3a743-2105-4a6d-a698-4c6dc6b45bd6 | swift-storage | 4096 | 40 | 0 | 1 | True |
| c455fb5a-4a50-49b5-89db-18c1e4517f77 | compute | 4096 | 40 | 0 | 1 | True |
| dd16a84d-b22e-4a82-a890-6cdedd194b47 | control | 4096 | 40 | 0 | 1 | True |
+--------------------------------------+---------------+-------+------+-----------+-------+-----------+
(undercloud) stack@undercloud.example.com:/home/stack>
(undercloud) stack@undercloud.example.com:/home/stack>openstack baremetal node list
+--------------------------------------+-------------+---------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+-------------+---------------+-------------+--------------------+-------------+
| 2fc8cc9a-c5d6-4b52-b0a4-b1108e2817a2 | controller1 | None | power off | available | False |
| 6767d1e1-84ec-40ef-95e2-92a671b8add6 | controller2 | None | power off | available | False |
| aef937a9-8cda-415c-8363-90376d921b2c | compute1 | None | power off | available | False |
| a11b0bb3-177e-4ac8-9afc-d159e45f953a | compute2 | None | power off | available | False |
| 2a8bf848-8546-4b8d-ba70-24f1ffce8de4 | ceph1 | None | power off | available | False |
| 365995b7-ce77-4125-837c-4d5fc55c3468 | ceph2 | None | power off | available | False |
+--------------------------------------+-------------+---------------+-------------+--------------------+-------------+
(undercloud) stack@undercloud.example.com:/home/stack>
(undercloud) stack@undercloud.example.com:/home/stack>openstack overcloud profiles list
+--------------------------------------+-------------+-----------------+-----------------+-------------------+
| Node UUID | Node Name | Provision State | Current Profile | Possible Profiles |
+--------------------------------------+-------------+-----------------+-----------------+-------------------+
| 2fc8cc9a-c5d6-4b52-b0a4-b1108e2817a2 | controller1 | available | control | |
| 6767d1e1-84ec-40ef-95e2-92a671b8add6 | controller2 | available | control | |
| aef937a9-8cda-415c-8363-90376d921b2c | compute1 | available | compute | |
| a11b0bb3-177e-4ac8-9afc-d159e45f953a | compute2 | available | compute | |
| 2a8bf848-8546-4b8d-ba70-24f1ffce8de4 | ceph1 | available | ceph-storage | |
| 365995b7-ce77-4125-837c-4d5fc55c3468 | ceph2 | available | ceph-storage | |
+--------------------------------------+-------------+-----------------+-----------------+-------------------+
(undercloud) stack@undercloud.example.com:/home/stack>
nova-conductor.log output:
2019-12-24 21:19:22.213 4289 DEBUG oslo_db.sqlalchemy.engines [req-242fa720-0306-4985-949c-9dd476a18539 e91ab96ba445474095c6a56f0e8b5e63 2c1ef3d24f5c49258602f4439a6a6c53 - default default] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:290
2019-12-24 21:19:22.596 4290 DEBUG oslo_db.sqlalchemy.engines [req-d0e42106-aca0-42f6-a204-96e5f9f1ed7b e91ab96ba445474095c6a56f0e8b5e63 2c1ef3d24f5c49258602f4439a6a6c53 - default default] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:290
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager [req-242fa720-0306-4985-949c-9dd476a18539 e91ab96ba445474095c6a56f0e8b5e63 2c1ef3d24f5c49258602f4439a6a6c53 - default default] Failed to schedule instances: NoValidHost_Remote: No valid host was found.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner
return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 139, in select_destinations
raise exception.NoValidHost(reason="")
NoValidHost: No valid host was found.
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager Traceback (most recent call last):
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 1165, in schedule_and_build_instances
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager instance_uuids, return_alternates=True)
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 760, in _schedule_instances
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager return_alternates=return_alternates)
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/utils.py", line 793, in wrapped
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager return func(*args, **kwargs)
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 53, in select_destinations
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager instance_uuids, return_objects, return_alternates)
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager return getattr(self.instance, __name)(*args, **kwargs)
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py", line 42, in select_destinations
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager instance_uuids, return_objects, return_alternates)
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 158, in select_destinations
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager return cctxt.call(ctxt, 'select_destinations', **msg_args)
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 174, in call
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager retry=self.retry)
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 131, in _send
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager timeout=timeout, retry=retry)
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 625, in send
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager retry=retry)
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 616, in _send
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager raise result
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager NoValidHost_Remote: No valid host was found.
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager Traceback (most recent call last):
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager return func(*args, **kwargs)
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 139, in select_destinations
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager raise exception.NoValidHost(reason="")
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager NoValidHost: No valid host was found.
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager
2019-12-24 21:19:22.918 4289 ERROR nova.conductor.manager
2019-12-24 21:19:22.938 4289 DEBUG oslo_concurrency.lockutils [req-242fa720-0306-4985-949c-9dd476a18539 e91ab96ba445474095c6a56f0e8b5e63 2c1ef3d24f5c49258602f4439a6a6c53 - default default] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273
2019-12-24 21:19:22.940 4289 DEBUG oslo_concurrency.lockutils [req-242fa720-0306-4985-949c-9dd476a18539 e91ab96ba445474095c6a56f0e8b5e63 2c1ef3d24f5c49258602f4439a6a6c53 - default default] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.002s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
2019-12-24 21:19:22.957 4289 DEBUG oslo_db.sqlalchemy.engines [req-242fa720-0306-4985-949c-9dd476a18539 e91ab96ba445474095c6a56f0e8b5e63 2c1ef3d24f5c49258602f4439a6a6c53 - default default] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py:290
2019-12-24 21:19:23.212 4289 DEBUG nova.conductor.manager [req-242fa720-0306-4985-949c-9dd476a18539 e91ab96ba445474095c6a56f0e8b5e63 2c1ef3d24f5c49258602f4439a6a6c53 - default default] [instance: 6258eabf-d7b9-451a-b5a6-989eb5981aa6] block_device_mapping [BlockDeviceMapping(attachment_id=<?>,boot_index=0,connection_info=None,created_at=<?>,delete_on_termination=True,deleted=<?>,deleted_at=<?>,destination_type='local',device_name=None,device_type='disk',disk_bus=None,guest_format=None,id=<?>,image_id='17d31491-de85-4bc5-b9db-739eb8baeca4',instance=<?>,instance_uuid=<?>,no_device=False,snapshot_id=None,source_type='image',tag=None,updated_at=<?>,uuid=<?>,volume_id=None,volume_size=None)] _create_block_device_mapping /usr/lib/python2.7/site-packages/nova/conductor/manager.py:1063
2019-12-24 21:19:23.215 4289 DEBUG oslo_concurrency.lockutils [req-242fa720-0306-4985-949c-9dd476a18539 e91ab96ba445474095c6a56f0e8b5e63 2c1ef3d24f5c49258602f4439a6a6c53 - default default] Lock "00000000-0000-0000-0000-000000000000" acquired by "nova.context.get_or_set_cached_cell_and_set_connections" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:273
2019-12-24 21:19:23.215 4289 DEBUG oslo_concurrency.lockutils [req-242fa720-0306-4985-949c-9dd476a18539 e91ab96ba445474095c6a56f0e8b5e63 2c1ef3d24f5c49258602f4439a6a6c53 - default default] Lock "00000000-0000-0000-0000-000000000000" released by "nova.context.get_or_set_cached_cell_and_set_connections" :: held 0.001s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:285
2019-12-24 21:19:23.248 4289 WARNING nova.scheduler.utils [req-242fa720-0306-4985-949c-9dd476a18539 e91ab96ba445474095c6a56f0e8b5e63 2c1ef3d24f5c49258602f4439a6a6c53 - default default] Failed to compute_task_build_instances: No valid host was found.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 229, in inner
return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 139, in select_destinations
raise exception.NoValidHost(reason="")
instackenv.json: (MAC,IP Address changed)
{
"nodes":[
{
"mac":[
"bb:bb:bb:bb:bb:bb"
],
"name":"controller1",
"cpu":"1",
"memory":"4096",
"disk":"40",
"arch":"x86_64",
"pm_type":"staging-ovirt",
"pm_user":"admin@internal",
"pm_password":"password123",
"pm_addr":"10.10.0.10",
"pm_vm_name":"controller1"
},
{
"mac":[
"bb:bb:bb:bb:bb:bb"
],
"name":"controller2",
"cpu":"1",
"memory":"4096",
"disk":"40",
"arch":"x86_64",
"pm_type":"staging-ovirt",
"pm_user":"admin@internal",
"pm_password":"password123",
"pm_addr":"10.10.0.10",
"pm_vm_name":"controller2"
},
{
"mac":[
"bb:bb:bb:bb:bb:bb"
],
"name":"compute1",
"cpu":"1",
"memory":"4096",
"disk":"40",
"arch":"x86_64",
"pm_type":"staging-ovirt",
"pm_user":"admin@internal",
"pm_password":"password123",
"pm_addr":"10.10.0.10",
"pm_vm_name":"compute1"
},
{
"mac":[
"bb:bb:bb:bb:bb:bb"
],
"name":"compute2",
"cpu":"1",
"memory":"4096",
"disk":"40",
"arch":"x86_64",
"pm_type":"staging-ovirt",
"pm_user":"admin@internal",
"pm_password":"password123",
"pm_addr":"10.10.0.10",
"pm_vm_name":"compute2"
},
{
"mac":[
"bb:bb:bb:bb:bb:bb"
],
"name":"ceph1",
"cpu":"1",
"memory":"16384",
"disk":"40",
"arch":"x86_64",
"pm_type":"staging-ovirt",
"pm_user":"admin@internal",
"pm_password":"password123",
"pm_addr":"10.10.0.10",
"pm_vm_name":"ceph1"
},
{
"mac":[
"bb:bb:bb:bb:bb:bb"
],
"name":"ceph2",
"cpu":"1",
"memory":"16384",
"disk":"40",
"arch":"x86_64",
"pm_type":"staging-ovirt",
"pm_user":"admin@internal",
"pm_password":"password123",
"pm_addr":"10.10.0.10",
"pm_vm_name":"ceph2"
}
]
}
