Compute nodes are stuck with clean wait status during overcloud node introspection

Solution Verified - Updated -

Environment

  • Red Hat OpenStack Platform 11

Issue

  • Compute nodes stuck with "clean wait" during overcloud node introspection. After sometime they moved to clean failed state.

Resolution

  • Disable the cleaning, restart ironic and re introspect the nodes.

    Edit the /etc/ironic/ironic.conf to disable cleaning:
    [conductor]
    automated_clean = false
    
  • Restart ironic:

    # systemctl restart openstack-ironic-api.service
    # systemctl restart openstack-ironic-conductor.service
    # systemctl restart openstack-ironic-inspector-dnsmasq.service
    # systemctl restart openstack-ironic-inspector.service
    

Root Cause

  • Cleaning was enabled manually, By default it is disable.Cleaning takes lots of time ( Default value for timeout is 1800sec), Due to timeout introspection was failing.

  • From the ironic.conf file

    # Enables or disables automated cleaning. Automated cleaning
     # is a configurable set of steps, such as erasing disk drives,
     # that are performed on the node to ensure it is in a baseline
     # state and ready to be deployed to. This is done after
     # instance deletion as well as during the transition from a
     # "manageable" to "available" state. When enabled, the
     # particular steps performed to clean a node depend on which
     # driver that node is managed by; see the individual driver's
     # documentation for details. NOTE: The introduction of the
     # cleaning operation causes instance deletion to take
     # significantly longer. In an environment where all tenants
     # are trusted (eg, because there is only one tenant), this
     # option could be safely disabled. (boolean value)
    
     # Timeout (seconds) to wait for a callback from the ramdisk
     # doing the cleaning. If the timeout is reached the node will
     # be put in the "clean failed" provision state. Set to 0 to
     # disable timeout. (integer value)
     #clean_callback_timeout = 1800
    

Diagnostic Steps

  • While doing bulk introspection it failed with below error:

    Introspection for UUID 9700540f-6964-4c7c-86d0-2f84899f1bb5 finished successfully.
    Introspection completed.
    Started Mistral Workflow tripleo.baremetal.v1.provide_manageable_nodes. Execution ID: 4d8626bd-b4a5-4812-a56e-6d5839c46212
    Waiting for messages on queue '8056d8c6-2177-4b23-a6d2-702dfa0f897c' with no timeout.
    socket is already closed.
    
  • We have tried to do introspection for one node and it also failed with below error:

    # openstack overcloud node introspect 6b669a27-e2ec-4164-b775-d14dd6ef5d41 --provide
    Started Mistral Workflow tripleo.baremetal.v1.introspect. Execution ID: 62b37499-d6fa-40dd-954c-36d59d95afcd
    Waiting for introspection to finish...
    Waiting for messages on queue '7b9d075c-8a9a-4316-a443-355a9e316f54' with no timeout.
    Successfully introspected all nodes.
    Introspection completed.
    Started Mistral Workflow tripleo.baremetal.v1.provide. Execution ID: 2ae43f27-bb71-4c43-a2c7-86f2d58a4a10
    Waiting for messages on queue '7b9d075c-8a9a-4316-a443-355a9e316f54' with no timeout.
    Failed to set nodes to available state:  IronicAction.node.set_power_state failed: <class 'ironicclient.common.apiclient.exceptions.BadRequest'>: The requested action "power off" can not be performed on node "6b669a27-e2ec-4164-b775-d14dd6ef5d41" while it is in state "clean wait".
    [stack@ ~]$
    
  • When we are checking the introspection status , it is showing True in Finished column.

    $ openstack baremetal introspection bulk status
    +--------------------------------------+----------+-------+
    | Node UUID                            | Finished | Error |
    +--------------------------------------+----------+-------+
    | d62e7834-35f3-4c0e-8d58-b67324d30838 | True     | None  |
    | 9cd445b4-52d2-4d0a-b0ed-502ff8cbe0b3 | True     | None  |
    | d7bb62e9-12e0-4e40-88bc-09f9d1e12522 | True     | None  |
    | 6b669a27-e2ec-4164-b775-d14dd6ef5d41 | True     | None  |
    | 08f084fe-ba43-48a6-b041-759952c9000e | True     | None  |
    | 136c4835-b313-4644-9f29-6a0caaa6354f | True     | None  |
    | 156973c6-9cef-4cc7-89ce-65dfe2266bbe | True     | None  |
    | 9700540f-6964-4c7c-86d0-2f84899f1bb5 | True     | None  |
    | 76dd228a-d58a-4c54-a67a-2bb8c0de1fc1 | True     | None  |
    | da952e34-0514-4241-a2e7-fd7476de65fa | True     | None  |
    | 28611a1b-2dcf-46eb-a2b6-26baac85aef7 | True     | None  |
    | 9b45c9fa-51cb-4fa9-bffb-1cd06fc92547 | True     | None  |
    +--------------------------------------+----------+-------+
    
  • tried to clean up one compute node with fresh rhel 7 installation and did introspection but no luck. It is failing while changing the state to poweroff after introspection.

    stack@l ~]$ openstack overcloud node introspect 08f084fe-ba43-48a6-b041-759952c9000e --provide
    Started Mistral Workflow tripleo.baremetal.v1.introspect. Execution ID: 87094e34-5568-4fc1-9d79-b6166da7d541
    Waiting for introspection to finish...
    Waiting for messages on queue 'da4040fd-7921-46a4-9e20-eb5caf72dc51' with no timeout.
    Successfully introspected all nodes.
    Introspection completed.
    Started Mistral Workflow tripleo.baremetal.v1.provide. Execution ID: bf76e148-977f-4998-9ee8-03af3ba45ad5
    Waiting for messages on queue 'da4040fd-7921-46a4-9e20-eb5caf72dc51' with no timeout.
    Failed to set nodes to available state:  IronicAction.node.set_power_state failed: <class 'ironicclient.common.apiclient.exceptions.BadRequest'>: The requested action "power off" can not be performed on node "08f084fe-ba43-48a6-b041-759952c9000e" while it is in state "clean wait"
    

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.