Baremetal nodes import fails for HPe ILO or redfish in Red Hat OpenStack Platform 16.0

Issue

Baremetal nodes import fails for HPe ILO or redfish in Red Hat OpenStack Platform 16.0

When importing the node, the import fails with:

(undercloud) $ openstack overcloud node import ../undercloud/instackenv.json
Waiting for messages on queue 'tripleo' with no timeout.

[{'result': 'Node <UUID> did not reach state "manageable", the state is "enroll", error: Failed to get power state for node <UUID>. Error: [iLO 192.168.123.22] The Redfish controller at "https://192.168.123.22" has thrown error. Error Unable to connect to https://192.168.123.22/redfish/v1/. Error: HTTPSConnectionPool(host=\'192.168.123.22\', port=443): Max retries exceeded with url: /redfish/v1/ (Caused by ProxyError(\'Cannot connect to proxy.\', OSError(\'Tunnel connection failed: 403 Forbidden\',)))'}]
{'result': 'Failure caused by error in tasks: send_message\n\n  send_message [task_ex_id=<task UUID>] -> Workflow failed due to message status. Status:FAILED Message:({\'result\': \'Node <UUID> did not reach state "manageable", the state is "enroll", error: Failed to get power state for node <UUID>. Error: [iLO 192.168.123.22] The Redfish controller at "https://192.168.123.22" has thrown error. Error Unable to connect to https://192.168.123.22/redfish/v1/. Error: HTTPSConnectionPool(host=\\\'192.168.123.22\\\', port=443): Max retries exceeded with url: /redfish/v1/ (Caused by ProxyError(\\\'Cannot connect to proxy.\\\', OSError(\\\'Tunnel connection failed: 403 Forbidden\\\',)))\'},)\n    [wf_ex_id=<wf_ex_id UUID>, idx=0]: Workflow failed due to message status. Status:FAILED Message:({\'result\': \'Node <UUID> did not reach state "manageable", the state is "enroll", error: Failed to get power state for node <UUID>. Error: [iLO 192.168.123.22] The Redfish controller at "https://192.168.123.22" has thrown error. Error Unable to connect to https://192.168.123.22/redfish/v1/. Error: HTTPSConnectionPool(host=\\\'192.168.123.22\\\', port=443): Max retries exceeded with url: /redfish/v1/ (Caused by ProxyError(\\\'Cannot connect to proxy.\\\', OSError(\\\'Tunnel connection failed: 403 Forbidden\\\',)))\'},)\n', 'status': 'FAILED', 'message': [{'result': 'Node <UUID> did not reach state "manageable", the state is "enroll", error: Failed to get power state for node <UUID>. Error: [iLO 192.168.123.22] The Redfish controller at "https://192.168.123.22" has thrown error. Error Unable to connect to https://192.168.123.22/redfish/v1/. Error: HTTPSConnectionPool(host=\'192.168.123.22\', port=443): Max retries exceeded with url: /redfish/v1/ (Caused by ProxyError(\'Cannot connect to proxy.\', OSError(\'Tunnel connection failed: 403 Forbidden\',)))'}]}
Exception registering nodes: {'result': 'Failure caused by error in tasks: send_message\n\n  send_message [task_ex_id=<task UUID>] -> Workflow failed due to message status. Status:FAILED Message:({\'result\': \'Node <UUID> did not reach state "manageable", the state is "enroll", error: Failed to get power state for node <UUID>. Error: [iLO 192.168.123.22] The Redfish controller at "https://192.168.123.22" has thrown error. Error Unable to connect to https://192.168.123.22/redfish/v1/. Error: HTTPSConnectionPool(host=\\\'192.168.123.22\\\', port=443): Max retries exceeded with url: /redfish/v1/ (Caused by ProxyError(\\\'Cannot connect to proxy.\\\', OSError(\\\'Tunnel connection failed: 403 Forbidden\\\',)))\'},)\n    [wf_ex_id=<wf_ex_id UUID>, idx=0]: Workflow failed due to message status. Status:FAILED Message:({\'result\': \'Node <UUID> did not reach state "manageable", the state is "enroll", error: Failed to get power state for node <UUID>. Error: [iLO 192.168.123.22] The Redfish controller at "https://192.168.123.22" has thrown error. Error Unable to connect to https://192.168.123.22/redfish/v1/. Error: HTTPSConnectionPool(host=\\\'192.168.123.22\\\', port=443): Max retries exceeded with url: /redfish/v1/ (Caused by ProxyError(\\\'Cannot connect to proxy.\\\', OSError(\\\'Tunnel connection failed: 403 Forbidden\\\',)))\'},)\n', 'status': 'FAILED', 'message': [{'result': 'Node <UUID> did not reach state "manageable", the state is "enroll", error: Failed to get power state for node <UUID>. Error: [iLO 192.168.123.22] The Redfish controller at "https://192.168.123.22" has thrown error. Error Unable to connect to https://192.168.123.22/redfish/v1/. Error: HTTPSConnectionPool(host=\'192.168.123.22\', port=443): Max retries exceeded with url: /redfish/v1/ (Caused by ProxyError(\'Cannot connect to proxy.\', OSError(\'Tunnel connection failed: 403 Forbidden\',)))'}]}

One can further see this issue by following req- in /var/log/containers/ironic/ironic-conductor.log.

One can see that the ILO driver makes an HTTPS call to verify the power state of the node and this fails as the connection is attempted via the proxy:

2020-04-19 20:49:53.969 7 DEBUG ironic.conductor.task_manager [req-<req UUID> 2ca0122e9c9248728f34b76439b04ead 7a668372af6740aca63088bcbe3a5bda - default default] Attempting to get exclusive lock on node <node UUID> (for provision action manage) __init__ /usr/lib/python3.6/site-packages/ironic/conductor/task_manager.py:222
2020-04-19 20:49:53.991 7 DEBUG ironic.conductor.task_manager [req-<req UUID> 2ca0122e9c9248728f34b76439b04ead 7a668372af6740aca63088bcbe3a5bda - default default] Node <node UUID> successfully reserved for provision action manage (took 0.02 seconds) reserve_node /usr/lib/python3.6/site-packages/ironic/conductor/task_manager.py:277
(...)
2020-04-19 20:49:54.034 7 DEBUG ironic.common.states [req-<req UUID> 2ca0122e9c9248728f34b76439b04ead 7a668372af6740aca63088bcbe3a5bda - default default] Exiting old state 'enroll' in response to event 'manage
' on_exit /usr/lib/python3.6/site-packages/ironic/common/states.py:294
2020-04-19 20:49:54.035 7 DEBUG ironic.common.states [req-<req UUID> 2ca0122e9c9248728f34b76439b04ead 7a668372af6740aca63088bcbe3a5bda - default default] Entering new state 'verifying' in response to event 'ma
nage' on_enter /usr/lib/python3.6/site-packages/ironic/common/states.py:300
(...)
2020-04-19 20:49:54.065 7 INFO ironic.conductor.task_manager [req-<req UUID> 2ca0122e9c9248728f34b76439b04ead 7a668372af6740aca63088bcbe3a5bda - default default] Node <node UUID> moved to provision state "verifying" from state "enroll"; target provision state is "manageable"
(...)
2020-04-19 20:49:54.070 7 DEBUG ironic.conductor.manager [req-<req UUID> 2ca0122e9c9248728f34b76439b04ead 7a668372af6740aca63088bcbe3a5bda - default default] Starting power credentials verification for node <node UUID> _do_node_verify /usr/lib/python3.6/site-packages/ironic/conductor/manager.py:1498
2020-04-19 20:49:54.071 7 DEBUG proliantutils.ilo.ribcl [req-<req UUID> 2ca0122e9c9248728f34b76439b04ead 7a668372af6740aca63088bcbe3a5bda - default default] [iLO 192.168.123.22] POST https://192.168.123.22:443/ribcl with request data: {'headers': {'Content-length': '145'}, 'data': '<RIBCL VERSION="2.0"><LOGIN PASSWORD="*****" USER_LOGIN="*****"><SERVER_INFO MODE="read"><GET_PRODUCT_NAME ></GET_PRODUCT_NAME></SERVER_INFO></LOGIN></RIBCL>\r\n', 'verify': False} _request_ilo /usr/lib/python3.6/site-packages/proliantutils/ilo/ribcl.py:147
(...)
2020-04-19 20:49:54.097 7 DEBUG proliantutils.ilo.ribcl [req-<req UUID> 2ca0122e9c9248728f34b76439b04ead 7a668372af6740aca63088bcbe3a5bda - default default] [iLO 192.168.123.22] Unable to connect to iLO. HTTPSConnectionPool(host='192.168.123.22', port=443): Max retries exceeded with url: /ribcl (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden',))) _request_ilo /usr/lib/python3.6/site-packages/proliantutils/ilo/ribcl.py:151
2020-04-19 20:49:54.098 7 DEBUG proliantutils.redfish.redfish [req-<req UUID> 2ca0122e9c9248728f34b76439b04ead 7a668372af6740aca63088bcbe3a5bda - default default] Redfish address: https://192.168.123.22 __init__ /usr/lib/python3.6/site-packages/proliantutils/redfish/redfish.py:147
2020-04-19 20:49:54.099 7 DEBUG sushy.connector [req-<req UUID> 2ca0122e9c9248728f34b76439b04ead 7a668372af6740aca63088bcbe3a5bda - default default] HTTP request: GET https://192.168.123.22/redfish/v1/; headers: {'OData-Version': '4.0'}; body: None; session arguments: {}; _op /usr/lib/python3.6/site-packages/sushy/connector.py:89
(...)
2020-04-19 20:49:56.114 7 DEBUG sushy.connector [req-<req UUID> 2ca0122e9c9248728f34b76439b04ead 7a668372af6740aca63088bcbe3a5bda - default default] HTTP request: GET https://192.168.123.22/redfish/v1/; headers: {'OData-Version': '4.0'}; body: None; session arguments: {}; _op /usr/lib/python3.6/site-packages/sushy/connector.py:89
2020-04-19 20:49:58.126 7 DEBUG sushy.connector [req-<req UUID> 2ca0122e9c9248728f34b76439b04ead 7a668372af6740aca63088bcbe3a5bda - default default] HTTP request: GET https://192.168.123.22/redfish/v1/; headers: {'OData-Version': '4.0'}; body: None; session arguments: {}; _op /usr/lib/python3.6/site-packages/sushy/connector.py:89
2020-04-19 20:49:58.136 7 DEBUG proliantutils.redfish.redfish [req-<req UUID> 2ca0122e9c9248728f34b76439b04ead 7a668372af6740aca63088bcbe3a5bda - default default] [iLO 192.168.123.22] The Redfish controller at "https://192.168.123.22" has thrown error. Error Unable to connect to https://192.168.123.22/redfish/v1/. Error: HTTPSConnectionPool(host='192.168.123.22', port=443): Max retries exceeded with url: /redfish/v1/ (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden',))) __init__ /usr/lib/python3.6/site-packages/proliantutils/redfish/redfish.py:164
2020-04-19 20:49:58.137 7 ERROR ironic.conductor.manager [req-<req UUID> 2ca0122e9c9248728f34b76439b04ead 7a668372af6740aca63088bcbe3a5bda - default default] Failed to get power state for node <node UUID>. Error: [iLO 192.168.123.22] The Redfish controller at "https://192.168.123.22" has thrown error. Error Unable to connect to https://192.168.123.22/redfish/v1/. Error: HTTPSConnectionPool(host='192.168.123.22', port=443): Max retries exceeded with url: /redfish/v1/ (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden',)))
(...)