overcloud deploy fails with "no vailid host" because undercloud /var is full
Issue
[stack@calo-rtp12-undercloud ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------------------------+--------+------------+-------------+----------+
| 6e61851d-aa3f-4b81-84c3-6df4c494e908 | overcloud-compute-0 | ERROR | - | NOSTATE | |
| 4a065728-df9e-4a8a-a0b0-64ab6fb92fae | overcloud-compute-1 | ERROR | - | NOSTATE | |
| 8089d33c-3ea1-49fd-be99-97be8ebff64c | overcloud-compute-1 | ERROR | - | NOSTATE | |
| 61959ad6-bd54-4225-b7ce-3a5d19e12997 | overcloud-controller-1 | ERROR | - | NOSTATE | |
+--------------------------------------+------------------------+--------+------------+-------------+----------+
[stack@calo-rtp12-undercloud ~]$ ironic node-list
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| d758aa8e-899a-433f-978a-04345a3cd3a6 | None | None | power off | available | False |
| 6615bb71-6483-4d7d-8945-f70ed319d1c0 | None | None | power off | available | False |
| e283d061-e51f-4768-bc49-2e64692cbc9e | None | None | power off | available | False |
| e0f782a8-35cb-44f3-bd4f-1844ce9d9294 | None | None | power off | available | False |
| 77f68770-d18f-4926-b0dd-cc86056bbff8 | None | None | power off | available | False |
| d22f2063-a086-48d7-a1c4-09d3cc087169 | None | None | power off | available | False |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
- nova conductor.log
2017-04-18 14:59:22.091 2301 ERROR nova.scheduler.utils [req-139108e9-7ffb-485f-8e85-65fca219a690 ead78c7620504e1ba7ec31db080c658d 4f3179843af1434facb012a5b1403b59 - - -] [instance: 4a065728-df9e-4a8a-a0b0-64ab6fb92fae] Error from last host: calo-rtp12-undercloud (node e0f782a8-35cb-44f3-bd4f-1844ce9d9294): [u'Traceback (most recent call last):\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1779, in _do_build_and_run_instance\n filter_properties)\n', u' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1976, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance 4a065728-df9e-4a8a-a0b0-64ab6fb92fae was re-scheduled: Failed to provision instance 4a065728-df9e-4a8a-a0b0-64ab6fb92fae: Failed to deploy. Error: Disk volume where '/var/lib/ironic/master_images/tmpBrkiNU' is located doesn't have enough disk space. Required 5252 MiB, only 4444 MiB available space present.\n"]
2017-04-18 14:59:22.141 2301 WARNING nova.scheduler.utils [req-139108e9-7ffb-485f-8e85-65fca219a690 ead78c7620504e1ba7ec31db080c658d 4f3179843af1434facb012a5b1403b59 - - -] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 199, in inner
return func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 104, in select_destinations
dests = self.driver.select_destinations(ctxt, spec_obj)
File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 74, in select_destinations
raise exception.NoValidHost(reason=reason)
NoValidHost: No valid host was found. There are not enough hosts available.
2017-04-18 14:59:22.141 2301 WARNING nova.scheduler.utils [req-139108e9-7ffb-485f-8e85-65fca219a690 ead78c7620504e1ba7ec31db080c658d 4f3179843af1434facb012a5b1403b59 - - -] [instance: 4a065728-df9e-4a8a-a0b0-64ab6fb92fae] Setting instance to ERROR state.
Environment
- Red Hat OpenStack Platform 10.0
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.