Unable to set ironic node maintenance state to off, getting HTTP 409 response

Issue

• Unable to set ironic node maintenance state to off, getting HTTP 409 response

• nova list works :

[stack@overcloud-ospd ~]$ source stackrc
[stack@overcloud-ospd ~]$ nova list
+--------------------------------------+------------------------+--------+------------+-------------+------------------------+
| ID                                   | Name                   | Status | Task State | Power State | Networks               |
+--------------------------------------+------------------------+--------+------------+-------------+------------------------+
| d4dd3690-154b-4d56-bb43-e2a87dd26a2a | overcloud-compute-0     | ACTIVE | -          | Running     | ctlplane=10.10.10.118 |
| a6efd945-135b-4fb0-b42c-9d4e3f83a17d | overcloud-compute-1     | ACTIVE | -          | Running     | ctlplane=10.10.10.134 |
| 5ab4b28a-316e-450e-aeae-1aaafe374e83 | overcloud-compute-10    | ACTIVE | -          | Running     | ctlplane=10.10.10.128 |
| 67264d13-43ea-4d25-9dc5-875e3157e723 | overcloud-compute-11    | ACTIVE | -          | Running     | ctlplane=10.10.10.125 |
| a7d4da2a-24eb-40a1-96fa-f960dca304f2 | overcloud-compute-12    | ACTIVE | -          | Running     | ctlplane=10.10.10.124 |
| 6247b855-41d0-4d8c-a059-c490a8ae2ee8 | overcloud-compute-13    | ACTIVE | -          | Running     | ctlplane=10.10.10.131 |
| 76eeec4a-410c-4d62-8843-9126a1ce48bb | overcloud-compute-14    | ACTIVE | -          | Running     | ctlplane=10.10.10.114 |
| 8c47b190-6aa6-461a-ad16-b5acc2951422 | overcloud-compute-15    | ACTIVE | -          | Running     | ctlplane=10.10.10.136 |
| 6835af4b-e641-4fcc-b04a-c7836dd20d72 | overcloud-compute-16    | ACTIVE | -          | Running     | ctlplane=10.10.10.133 |
| 18d54a5d-d957-4f5c-8d9a-1da890608e34 | overcloud-compute-17    | ACTIVE | -          | Running     | ctlplane=10.10.10.112 |
| baead58f-98a2-4822-a07a-316bdcdca338 | overcloud-compute-2     | ACTIVE | -          | Running     | ctlplane=10.10.10.102 |
| f2b70aa5-b023-4f4c-9744-38c5e59ba11e | overcloud-compute-3     | ACTIVE | -          | Running     | ctlplane=10.10.10.120 |
| 05add4e7-fb67-457f-8a81-f1638bb0e26f | overcloud-compute-4     | ACTIVE | -          | Running     | ctlplane=10.10.10.108 |
| 26625249-4677-4c78-8cf0-15701afa8417 | overcloud-compute-5     | ACTIVE | -          | Running     | ctlplane=10.10.10.109 |
| ce455955-a2a1-442c-a267-dc51cb5ccb74 | overcloud-compute-6     | ACTIVE | -          | Running     | ctlplane=10.10.10.104 |
| 023f3cef-e80f-4696-abb7-5f13033ff4c6 | overcloud-compute-7     | ACTIVE | -          | Running     | ctlplane=10.10.10.127 |
| 499e77d3-0e5a-4fa6-8347-98594106f92c | overcloud-compute-8     | ACTIVE | -          | Running     | ctlplane=10.10.10.122 |
| 25695b75-9236-4ebb-bd5b-24a7970f4829 | overcloud-compute-9     | ACTIVE | -          | Running     | ctlplane=10.10.10.106 |
| fadfbee4-ff98-47bb-ad2e-73f6924f963a | overcloud-controller-0  | ACTIVE | -          | Running     | ctlplane=10.10.10.119 |
| 7b24d369-06ab-499d-a4d8-3cbd2eaad294 | overcloud-controller-1  | ACTIVE | -          | Running     | ctlplane=10.10.10.117 |
| ec5d86ed-4eff-461d-89d8-c34e3906b460 | overcloud-controller-2  | ACTIVE | -          | Running     | ctlplane=10.10.10.113 |
| f92d9549-4891-4a1d-8d8a-489df82d1beb | overcloud-osd-compute-0 | ACTIVE | -          | Running     | ctlplane=10.10.10.110 |
| db175aba-d2dc-4835-88d4-5080fef6c6fc | overcloud-osd-compute-1 | ACTIVE | -          | Running     | ctlplane=10.10.10.103 |
| b76902c2-2123-4a5f-9548-c9cdb6b1776f | overcloud-osd-compute-2 | ACTIVE | -          | Running     | ctlplane=10.10.10.111 |
| 68f2c909-07df-4b26-a408-d6a3e12e5225 | overcloud-osd-compute-3 | ACTIVE | -          | Running     | ctlplane=10.10.10.105 |
+--------------------------------------+------------------------+--------+------------+-------------+------------------------+

• ironic node-list works:

[stack@overcloud-ospd ~]$ ironic node-list
+--------------------------------------+-------------------+--------------------------------------+-------------+--------------------+-------------+
| UUID                                 | Name              | Instance UUID                        | Power State | Provisioning State | Maintenance |
+--------------------------------------+-------------------+--------------------------------------+-------------+--------------------+-------------+
| 4dc997d5-80e6-4308-942e-40f3e3849d28 | overcloud-OSC-A    | fadfbee4-ff98-47bb-ad2e-73f6924f963a | power on    | active             | False       |
| 5fb27da1-66ff-47c5-bdd1-0608cc31eb99 | overcloud-OSC-B    | 7b24d369-06ab-499d-a4d8-3cbd2eaad294 | power on    | active             | False       |
| 9f053e87-a099-4abc-877a-234ec44365fd | overcloud-OSC-C    | ec5d86ed-4eff-461d-89d8-c34e3906b460 | power on    | active             | False       |
| cd4fc634-8718-4ea8-a282-f90be1e37477 | overcloud-OSDC1-01 | f92d9549-4891-4a1d-8d8a-489df82d1beb | power on    | active             | False       |
| c86ea1b6-4ccb-4aa3-937c-070be7054bde | overcloud-OSDC1-02 | db175aba-d2dc-4835-88d4-5080fef6c6fc | power on    | active             | False       |
| 65c376f7-a451-4713-989a-b69c985fa98f | overcloud-OSDC2-01 | b76902c2-2123-4a5f-9548-c9cdb6b1776f | power on    | active             | False       |
| f5361314-2beb-4046-a80b-5fd3147229fd | overcloud-OSDC2-02 | 68f2c909-07df-4b26-a408-d6a3e12e5225 | power on    | active             | False       |
| 6a2609ec-4e13-4834-9203-fc6c8e9f5e01 | overcloud-CMP1-01  | d4dd3690-154b-4d56-bb43-e2a87dd26a2a | power on    | active             | False       |
| b0b083e6-6f9f-4428-b4dc-c72dfd4e383c | overcloud-CMP1-02  | a6efd945-135b-4fb0-b42c-9d4e3f83a17d | power on    | active             | False       |
| 54a90ae0-fe02-4d9a-a35e-88f0876b2d5f | overcloud-CMP1-03  | baead58f-98a2-4822-a07a-316bdcdca338 | None        | active             | True        |
| 1af9f6e8-b92d-4c2e-a8d4-ebc4fee77ae7 | overcloud-CMP1-04  | f2b70aa5-b023-4f4c-9744-38c5e59ba11e | power on    | active             | False       |
| f72f414c-a2c3-4ab6-b1db-e0b8c4b10987 | overcloud-CMP1-05  | 05add4e7-fb67-457f-8a81-f1638bb0e26f | power on    | active             | False       |
| b4d34dd1-1e0d-4fd0-849a-c5fe399f5fe9 | overcloud-CMP1-06  | 26625249-4677-4c78-8cf0-15701afa8417 | power on    | active             | False       |
| 3666a19d-b69d-4e24-a0e6-1d729831b49a | overcloud-CMP1-07  | ce455955-a2a1-442c-a267-dc51cb5ccb74 | power on    | active             | False       |
| ff19b964-cf69-49ce-be95-4004bf9fcf76 | overcloud-CMP1-08  | 023f3cef-e80f-4696-abb7-5f13033ff4c6 | power on    | active             | False       |
| 01020f7f-a59b-43ec-9985-498cdb5fdd88 | overcloud-CMP1-09  | 499e77d3-0e5a-4fa6-8347-98594106f92c | power on    | active             | False       |
| bc78afd9-fb81-4e13-8cc4-c3ab61dd2e5c | overcloud-CMP2-01  | 25695b75-9236-4ebb-bd5b-24a7970f4829 | power on    | active             | False       |
| 9f349266-7330-4b24-9f6a-19312f23313e | overcloud-CMP2-02  | 5ab4b28a-316e-450e-aeae-1aaafe374e83 | power on    | active             | False       |
| 000b0fdc-89b3-4383-b88d-936fc4ee425f | overcloud-CMP2-03  | 67264d13-43ea-4d25-9dc5-875e3157e723 | power on    | active             | False       |
| 3f90868b-f84d-42fd-b9d6-7526b4bbb6e6 | overcloud-CMP2-04  | a7d4da2a-24eb-40a1-96fa-f960dca304f2 | power on    | active             | False       |
| fe513cf1-b207-47c8-8fe3-191effa060b3 | overcloud-CMP2-05  | 6247b855-41d0-4d8c-a059-c490a8ae2ee8 | power on    | active             | False       |
| c63c7cce-9e27-4c6b-b9cf-92d3f48b6ded | overcloud-CMP2-06  | 76eeec4a-410c-4d62-8843-9126a1ce48bb | power on    | active             | False       |
| e7ac7e15-ef32-43ef-9c25-50e15bf651ee | overcloud-CMP2-07  | 8c47b190-6aa6-461a-ad16-b5acc2951422 | power on    | active             | False       |
| 39971c5b-544d-4c18-aca5-ab17aacf3001 | overcloud-CMP2-08  | 6835af4b-e641-4fcc-b04a-c7836dd20d72 | power on    | active             | False       |
| 203f1b6b-2e2d-419e-8174-92d35ceaa1c5 | overcloud-CMP2-09  | 18d54a5d-d957-4f5c-8d9a-1da890608e34 | power on    | active             | False       |
+--------------------------------------+-------------------+--------------------------------------+-------------+--------------------+-------------+

• The following data was imported in ironic:

[stack@overcloud-ospd ~]$ less scripts/instackenv.json | grep compute-2 -A10
            "capabilities": "node:osd-compute-2,boot_option:local",
            "cpu": "24",
            "memory": "256000",
            "disk": "3000",
            "arch": "x86_64",
            "pm_type": "pxe_ipmitool",
            "pm_user": "admin",
            "pm_password": "admin",
            "pm_addr": "10.10.12.6"
        },
        {
--
            "capabilities": "node:compute-2,boot_option:local",
            "cpu": "24",
            "memory": "256000",
            "disk": "3000",
            "arch": "x86_64",
            "pm_type": "pxe_ipmitool",
            "pm_user": "admin",
            "pm_password": "admin",
            "pm_addr": "10.10.11.10"
        },

• Using the above data works:

[stack@overcloud-ospd ~]$ ipmitool -I lanplus -U admin -P password -H 10.10.11.10 chassis power status
Chassis Power is on

• ironic node-show UUID on the node returns an error message similar to this one:

[stack@overcloud-ospd ~]$ ironic node-show 54a90ae0-fe02-4d9a-a35e-88f0876b2d5f
+------------------------+--------------------------------------------------------------------------+
| Property               | Value                                                                    |
+------------------------+--------------------------------------------------------------------------+
| chassis_uuid           |                                                                          |
| clean_step             | {}                                                                       |
| console_enabled        | False                                                                    |
| created_at             | 2019-06-22T18:24:57+00:00                                                |
| driver                 | pxe_ipmitool                                                             |
| driver_info            | {u'deploy_kernel': u'a5b30536-c6b1-4d73-958c-58f98c09a4a1',              |
|                        | u'ipmi_address': u'10.10.11.10', u'deploy_ramdisk': u'f804ba75-17fb-    |
|                        | 4bac-9ce4-5a27ca9b1cab', u'ipmi_password': u'******', u'ipmi_username':  |
|                        | u'admin'}                                                                |
| driver_internal_info   | {u'agent_url': u'http://10.10.10.102:9999', u'root_uuid_or_disk_id':    |
|                        | u'1d4b88c1-ed31-4893-85b6-b001119ae483', u'is_whole_disk_image': False,  |
|                        | u'agent_last_heartbeat': 1561229836}                                     |
| extra                  | {u'hardware_swift_object': u'extra_hardware-54a90ae0-fe02-4d9a-a35e-     |
|                        | 88f0876b2d5f'}                                                           |
| inspection_finished_at | None                                                                     |
| inspection_started_at  | None                                                                     |
| instance_info          | {u'root_gb': u'40', u'display_name': u'overcloud-compute-2',              |
|                        | u'image_source': u'0af558ab-240e-436d-a505-9433c37b0419',                |
|                        | u'capabilities': u'{"boot_option": "local"}', u'memory_mb': u'4096',     |
|                        | u'vcpus': u'1', u'local_gb': u'743', u'configdrive': u'******',          |
|                        | u'swap_mb': u'0'}                                                        |
| instance_uuid          | baead58f-98a2-4822-a07a-316bdcdca338                                     |
| last_error             | During sync_power_state, max retries exceeded for node 54a90ae0-fe02     |
|                        | -4d9a-a35e-88f0876b2d5f, node state None does not match expected state   |
|                        | 'power on'. Updating DB state to 'None' Switching node to maintenance    |
|                        | mode. Error: IPMI call failed: power status.                             |
| maintenance            | True                                                                     |
| maintenance_reason     | During sync_power_state, max retries exceeded for node 54a90ae0-fe02     |
|                        | -4d9a-a35e-88f0876b2d5f, node state None does not match expected state   |
|                        | 'power on'. Updating DB state to 'None' Switching node to maintenance    |
|                        | mode. Error: IPMI call failed: power status.                             |
| name                   | overcloud-CMP1-03                                                         |
| network_interface      |                                                                          |
| power_state            | None                                                                     |
| properties             | {u'memory_mb': u'393216', u'cpu_arch': u'x86_64', u'local_gb': u'743',   |
|                        | u'cpus': u'40', u'capabilities': u'node:compute-2,cpu_vt:true,cpu_hugepa |
|                        | ges:true,boot_option:local,cpu_txt:true,cpu_aes:true,cpu_hugepages_1g:tr |
|                        | ue'}                                                                     |
| provision_state        | active                                                                   |
| provision_updated_at   | 2019-06-22T19:06:19+00:00                                                |
| raid_config            |                                                                          |
| reservation            | overcloud-ospd.localdomain                                               |
| resource_class         |                                                                          |
| target_power_state     | None                                                                     |
| target_provision_state | None                                                                     |
| target_raid_config     |                                                                          |
| updated_at             | 2019-12-13T06:31:21+00:00                                                |
| uuid                   | 54a90ae0-fe02-4d9a-a35e-88f0876b2d5f                                     |
+------------------------+--------------------------------------------------------------------------+

• We are unable to remove the maintenance mode from the node:

[stack@overcloud-ospd ~]$ ironic node-set-maintenance 54a90ae0-fe02-4d9a-a35e-88f0876b2d5f off
Node 54a90ae0-fe02-4d9a-a35e-88f0876b2d5f is locked by host overcloud-ospd.localdomain, please retry after the current operation is completed. (HTTP 409)