Undercloud keeps shutting down overcloud instances after their recovering from failure.

Solution Verified - Updated -

Issue

What problem/issue/behavior are you having trouble with?

We did a power shutdown of all our cloud members and after that only controller0 stayed up (after powering on all controllers and compute nodes, all but controller0 was fenced withing like five minutes).
I would have expected all nodes to stay up after power on and the overcloud to be ready to start my virtual machine instances.

In order to get also the other controller nodes alive

$ ironic node-set-power-state 5a69baed-4149-4f2c-bf2b-354a6b79aac6 on

And now we have mismatch on the statuses of the system

$ ironic node-list
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
| UUID                                 | Name | Instance UUID                        | Power State | Provision State | Maintenance |
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+
| d504d776-9981-4bd2-a926-795d91aaf04e | None | 6729d339-99fa-48d0-aedc-01a834a47969 | power on    | active          | False       |
| 5a69baed-4149-4f2c-bf2b-354a6b79aac6 | None | 7387c9d3-ee3d-4a42-bf48-d6d58ffb2ce6 | power on    | active          | False       |
| 8737165b-1770-4b40-9df7-c956d59df194 | None | 813b7285-3289-45bd-b7e3-62b3f3776586 | power on    | active          | False       |
| aae71c7d-ba6d-4446-8f90-aaf9fe1b528c | None | cdb5b493-aadc-4b91-886e-dc36ea6dede9 | power off   | active          | False       |
| df8cde67-6255-4c8d-baee-faa825290080 | None | f0f2723b-335a-4536-ae88-4d173730977e | power off   | active          | False       |
| ecdebb34-c2cb-4252-8647-cc640635bb4a | None | 6c447710-e5a4-4d1d-b310-4086e26efa71 | power off   | active          | False       |
| a5116003-9300-4f4d-95f9-beb07720ecfd | None | None                                 | power off   | available       | False       |
+--------------------------------------+------+--------------------------------------+-------------+-----------------+-------------+

$ nova list
+--------------------------------------+------------------------+---------+------------+-------------+---------------------+
| ID                                   | Name                   | Status  | Task State | Power State | Networks            |
+--------------------------------------+------------------------+---------+------------+-------------+---------------------+
| 6c447710-e5a4-4d1d-b310-4086e26efa71 | overcloud-compute-0    | SHUTOFF | -          | Shutdown    | ctlplane=192.0.2.12 |
| cdb5b493-aadc-4b91-886e-dc36ea6dede9 | overcloud-compute-1    | SHUTOFF | -          | Shutdown    | ctlplane=192.0.2.11 |
| f0f2723b-335a-4536-ae88-4d173730977e | overcloud-compute-2    | SHUTOFF | -          | Shutdown    | ctlplane=192.0.2.14 |
| 813b7285-3289-45bd-b7e3-62b3f3776586 | overcloud-controller-0 | ACTIVE  | -          | Running     | ctlplane=192.0.2.16 |
| 6729d339-99fa-48d0-aedc-01a834a47969 | overcloud-controller-1 | SHUTOFF | -          | Shutdown    | ctlplane=192.0.2.13 |
| 7387c9d3-ee3d-4a42-bf48-d6d58ffb2ce6 | overcloud-controller-2 | SHUTOFF | -          | Shutdown    | ctlplane=192.0.2.15 |
+--------------------------------------+------------------------+---------+------------+-------------+---------------------+

Overcloud stays down after simulated power outage

Environment

  • Red Hat OpenStack Platform 7.0
  • Red Hat OpenStack Director 7

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content