overcloud deploy fails with "no vailid host" because undercloud /var is full

Solution In Progress - Updated -

Issue

[stack@calo-rtp12-undercloud ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+----------+
| ID                                   | Name                   | Status | Task State | Power State | Networks |
+--------------------------------------+------------------------+--------+------------+-------------+----------+
| 6e61851d-aa3f-4b81-84c3-6df4c494e908 | overcloud-compute-0    | ERROR  | -          | NOSTATE     |          |
| 4a065728-df9e-4a8a-a0b0-64ab6fb92fae | overcloud-compute-1    | ERROR  | -          | NOSTATE     |          |
| 8089d33c-3ea1-49fd-be99-97be8ebff64c | overcloud-compute-1    | ERROR  | -          | NOSTATE     |          |
| 61959ad6-bd54-4225-b7ce-3a5d19e12997 | overcloud-controller-1 | ERROR  | -          | NOSTATE     |          |
+--------------------------------------+------------------------+--------+------------+-------------+----------+

[stack@calo-rtp12-undercloud ~]$ ironic node-list
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| UUID                                 | Name | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
| d758aa8e-899a-433f-978a-04345a3cd3a6 | None | None          | power off   | available          | False       |
| 6615bb71-6483-4d7d-8945-f70ed319d1c0 | None | None          | power off   | available          | False       |
| e283d061-e51f-4768-bc49-2e64692cbc9e | None | None          | power off   | available          | False       |
| e0f782a8-35cb-44f3-bd4f-1844ce9d9294 | None | None          | power off   | available          | False       |
| 77f68770-d18f-4926-b0dd-cc86056bbff8 | None | None          | power off   | available          | False       |
| d22f2063-a086-48d7-a1c4-09d3cc087169 | None | None          | power off   | available          | False       |
+--------------------------------------+------+---------------+-------------+--------------------+-------------+
  • nova conductor.log
2017-04-18 14:59:22.091 2301 ERROR nova.scheduler.utils [req-139108e9-7ffb-485f-8e85-65fca219a690 ead78c7620504e1ba7ec31db080c658d 4f3179843af1434facb012a5b1403b59 - - -] [instance: 4a065728-df9e-4a8a-a0b0-64ab6fb92fae] Error from last host: calo-rtp12-undercloud (node e0f782a8-35cb-44f3-bd4f-1844ce9d9294): [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1779, in _do_build_and_run_instance\n    filter_properties)\n', u'  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1976, in _build_and_run_instance\n    instance_uuid=instance.uuid, reason=six.text_type(e))\n', u"RescheduledException: Build of instance 4a065728-df9e-4a8a-a0b0-64ab6fb92fae was re-scheduled: Failed to provision instance 4a065728-df9e-4a8a-a0b0-64ab6fb92fae: Failed to deploy. Error: Disk volume where '/var/lib/ironic/master_images/tmpBrkiNU' is located doesn't have enough disk space. Required 5252 MiB, only 4444 MiB available space present.\n"]

2017-04-18 14:59:22.141 2301 WARNING nova.scheduler.utils [req-139108e9-7ffb-485f-8e85-65fca219a690 ead78c7620504e1ba7ec31db080c658d 4f3179843af1434facb012a5b1403b59 - - -] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 199, in inner
    return func(*args, **kwargs)

  File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 104, in select_destinations
    dests = self.driver.select_destinations(ctxt, spec_obj)

  File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 74, in select_destinations
    raise exception.NoValidHost(reason=reason)

NoValidHost: No valid host was found. There are not enough hosts available.

2017-04-18 14:59:22.141 2301 WARNING nova.scheduler.utils [req-139108e9-7ffb-485f-8e85-65fca219a690 ead78c7620504e1ba7ec31db080c658d 4f3179843af1434facb012a5b1403b59 - - -] [instance: 4a065728-df9e-4a8a-a0b0-64ab6fb92fae] Setting instance to ERROR state.

Environment

  • Red Hat OpenStack Platform 10.0

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content