Compute Node Introspection fails to set boot device to PXE

Issue

• We are able to import baremetal nodes for management:

(undercloud) [stack@undercloud-rhosp15 ~]$ openstack baremetal node list
+--------------------------------------+------------+---------------+-------------+--------------------+-------------+
| UUID                                 | Name       | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+------------+---------------+-------------+--------------------+-------------+
| 3730689b-3b64-479f-b611-8a72d00923ce | DL380Gen10 | None          | power off   | manageable         | False       |
+--------------------------------------+------------+---------------+-------------+--------------------+-------------+

• However, introspection of the node fails with the error Failed to set boot device to PXE. :

(undercloud) [stack@undercloud-rhosp15 ~]$ openstack overcloud node introspect --all-manageable --provide
Waiting for introspection to finish...
Waiting for messages on queue 'tripleo' with no timeout.
Introspection of node 3730689b-3b64-479f-b611-8a72d00923ce failed.
Retrying 1 nodes that failed introspection. Attempt 1 of 3
Introspection of node 3730689b-3b64-479f-b611-8a72d00923ce completed. Status:FAILED. Errors:Failed to set boot device to PXE: Setting pxe as boot device failed, error: [iLO 10.10.10.1 90] The Redfish controller failed to set one time boot device NETWORK. Error: HTTP PATCH https://10.10.10.190/redfish/v1/Systems/1 returned code 400. iLO.0.10.ExtendedInfo: See @Messa ge.ExtendedInfo for more information. (HTTP 500)
Retrying 1 nodes that failed introspection. Attempt 2 of 3
Introspection of node 3730689b-3b64-479f-b611-8a72d00923ce completed. Status:FAILED. Errors:Failed to set boot device to PXE: Setting pxe as boot device failed, error: [iLO 10.10.10.1 90] The Redfish controller failed to set one time boot device NETWORK. Error: HTTP PATCH https://10.10.10.190/redfish/v1/Systems/1 returned code 400. iLO.0.10.ExtendedInfo: See @Messa ge.ExtendedInfo for more information. (HTTP 500)
Retrying 1 nodes that failed introspection. Attempt 3 of 3
Introspection of node 3730689b-3b64-479f-b611-8a72d00923ce completed. Status:FAILED. Errors:Failed to set boot device to PXE: Setting pxe as boot device failed, error: [iLO 10.10.10.1 90] The Redfish controller failed to set one time boot device NETWORK. Error: HTTP PATCH https://10.10.10.190/redfish/v1/Systems/1 returned code 400. iLO.0.10.ExtendedInfo: See @Messa ge.ExtendedInfo for more information. (HTTP 500)
Retry limit reached with 1 nodes still failing introspection
{'result': 'Failure caused by error in tasks: send_message\n\n  send_message [task_ex_id=7c7202ef-ec16-4274-bcb9-c0657e300d7b] -> Workflow failed due to message status. Status:FAILED Me ssage:Retry limit reached with 1 nodes still failing introspection\n    [wf_ex_id=47b465d7-f7c4-4baa-8511-7c25222cd0b9, idx=0]: Workflow failed due to message status. Status:FAILED Mess age:Retry limit reached with 1 nodes still failing introspection\n', 'introspection_attempt': 3, 'introspected_nodes': {'3730689b-3b64-479f-b611-8a72d00923ce': {'error': 'Failed to set  boot device to PXE: Setting pxe as boot device failed, error: [iLO 10.10.10.190] The Redfish controller failed to set one time boot device NETWORK. Error: HTTP PATCH https://15.113.21 9.190/redfish/v1/Systems/1 returned code 400. iLO.0.10.ExtendedInfo: See @Message.ExtendedInfo for more information. (HTTP 500)', 'finished': True, 'finished_at': '2020-09-10T23:02:40',  'links': [{'href': 'http://10.10.10.51:13050/v1/introspection/3730689b-3b64-479f-b611-8a72d00923ce', 'rel': 'self'}], 'started_at': '2020-09-10T23:02:36', 'state': 'error', 'uuid': ' 3730689b-3b64-479f-b611-8a72d00923ce'}}, 'failed_introspection': ['3730689b-3b64-479f-b611-8a72d00923ce'], 'status': 'FAILED', 'message': 'Retry limit reached with 1 nodes still failing  introspection', 'node_uuids': ['3730689b-3b64-479f-b611-8a72d00923ce']}
Exception introspecting nodes: {'result': 'Failure caused by error in tasks: send_message\n\n  send_message [task_ex_id=7c7202ef-ec16-4274-bcb9-c0657e300d7b] -> Workflow failed due to m essage status. Status:FAILED Message:Retry limit reached with 1 nodes still failing introspection\n    [wf_ex_id=47b465d7-f7c4-4baa-8511-7c25222cd0b9, idx=0]: Workflow failed due to mes sage status. Status:FAILED Message:Retry limit reached with 1 nodes still failing introspection\n', 'introspection_attempt': 3, 'introspected_nodes': {'3730689b-3b64-479f-b611-8a72d0092 3ce': {'error': 'Failed to set boot device to PXE: Setting pxe as boot device failed, error: [iLO 10.10.10.190] The Redfish controller failed to set one time boot device NETWORK. Erro r: HTTP PATCH https://10.10.10.190/redfish/v1/Systems/1 returned code 400. iLO.0.10.ExtendedInfo: See @Message.ExtendedInfo for more information. (HTTP 500)', 'finished': True, 'finis hed_at': '2020-09-10T23:02:40', 'links': [{'href': 'http://10.10.10.51:13050/v1/introspection/3730689b-3b64-479f-b611-8a72d00923ce', 'rel': 'self'}], 'started_at': '2020-09-10T23:02:3 6', 'state': 'error', 'uuid': '3730689b-3b64-479f-b611-8a72d00923ce'}}, 'failed_introspection': ['3730689b-3b64-479f-b611-8a72d00923ce'], 'status': 'FAILED', 'message': 'Retry limit rea ched with 1 nodes still failing introspection', 'node_uuids': ['3730689b-3b64-479f-b611-8a72d00923ce']}
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.10.10.51', 43766)>
sys:1: ResourceWarning: unclosed <ssl.SSLSocket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('10.10.10.51', 34140)>

• We noticed that the tripleo_ironic_inspector_dnsmasq.service is in a failed state but we are unsure if it is related to this error:

 tripleo_ironic_inspector_dnsmasq.service             loaded failed   failed  ironic_inspector_dnsmasq container