overcloud deployment or node provision fails with 403 Forbidden error

Issue

• In Red Hat OpenStack Platform 16.2 environment:

  • openstack overcloud deploy fails with the following error.

    overcloud.Compute.0.NovaCompute:
      resource_type: OS::TripleO::ComputeServer
      physical_resource_id: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa
      status: CREATE_FAILED
      status_reason: |
        ResourceInError: resources.NovaCompute: Went to status ERROR due to "Message: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa., Code: 500"
    

  • /var/log/containers/ironic/ironic-conductor.log shows 403 error when attempting to access 9999 port of a overcloud node.

    ERROR ironic.drivers.modules.agent_base_vendor [-] Asynchronous exception: Failed checking if deploy is done. Exception: Expecting value: line 1 column 1 (char 0) for node aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa: simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
    ERROR ironic.drivers.modules.agent_base_vendor Traceback (most recent call last):
    ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/agent_base_vendor.py", line 387, in heartbeat
    ERROR ironic.drivers.modules.agent_base_vendor     if not self.deploy_has_started(task):
    ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python3.6/site-packages/ironic_lib/metrics.py", line 60, in wrapped
    ERROR ironic.drivers.modules.agent_base_vendor     result = f(*args, **kwargs)
    ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/agent.py", line 173, in deploy_has_started
    ERROR ironic.drivers.modules.agent_base_vendor     commands = self._client.get_commands_status(task.node)
    ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python3.6/site-packages/ironic_lib/metrics.py", line 60, in wrapped
    ERROR ironic.drivers.modules.agent_base_vendor     result = f(*args, **kwargs)
    ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python3.6/site-packages/retrying.py", line 68, in wrapped_f
    ERROR ironic.drivers.modules.agent_base_vendor     return Retrying(*dargs, **dkw).call(f, *args, **kw)
    ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
    ERROR ironic.drivers.modules.agent_base_vendor     return attempt.get(self._wrap_exception)
    ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
    ERROR ironic.drivers.modules.agent_base_vendor     six.reraise(self.value[0], self.value[1], self.value[2])
    ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
    ERROR ironic.drivers.modules.agent_base_vendor     raise value
    ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
    ERROR ironic.drivers.modules.agent_base_vendor     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
    ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/agent_client.py", line 188, in get_commands_status
    ERROR ironic.drivers.modules.agent_base_vendor     result = resp.json()['commands']
    ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib/python3.6/site-packages/requests/models.py", line 897, in json
    ERROR ironic.drivers.modules.agent_base_vendor     return complexjson.loads(self.text, **kwargs)
    ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib64/python3.6/site-packages/simplejson/__init__.py", line 518, in loads
    ERROR ironic.drivers.modules.agent_base_vendor     return _default_decoder.decode(s)
    ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib64/python3.6/site-packages/simplejson/decoder.py", line 370, in decode
    ERROR ironic.drivers.modules.agent_base_vendor     obj, end = self.raw_decode(s)
    ERROR ironic.drivers.modules.agent_base_vendor   File "/usr/lib64/python3.6/site-packages/simplejson/decoder.py", line 400, in raw_decode
    ERROR ironic.drivers.modules.agent_base_vendor     return self.scan_once(s, idx=_w(s, idx).end())
    ERROR ironic.drivers.modules.agent_base_vendor simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
    ERROR ironic.drivers.modules.agent_base_vendor
    DEBUG ironic.drivers.modules.agent_client [-] Executing agent command log.collect_system_logs for node aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa _command /usr/lib/python3.6/site-packages/ironic/drivers/modules/agent_client.py:93
    ERROR ironic.drivers.modules.agent_client [-] Unable to decode response as JSON.
    Request URL: http://192.168.24.5:9999/v1/commands
    Request body: "{"name": "log.collect_system_logs", "params": {}}"
    Response status code: 403
    Response: "<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
    <html><head>
    <meta type="copyright" content="Copyright (C) 1996-2016 The Squid Software Foundation and contributors">
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
    <title>ERROR: The requested URL could not be retrieved</title>
    <style type="text/css"><!--
    

• In Red Hat OpenStack Platform 17.0 environment:

  • openstack overcloud node provision fails with the following error.

    2023-01-01 01:17:32.000000 | aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa |       TASK | Provision instances
    2023-01-01 01:14:33.000000 | aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa |      FATAL | Provision instances | localhost | error=......
    

  • /var/log/containers/ironic/ironic-conductor.log shows 403 error when attempting to access 9999 port of a overcloud node.

    ERROR ironic.drivers.modules.agent_client [-] Failed to connect to the agent running on node aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa  to collect commands status. Error: HTTPSConnectionPool(host='192.168.24.5', port=9999): Max retries exceeded with url: /v1/commands/ (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden'))): requests.exceptions.ProxyError: HTTPSConnectionPool(host='192.168.24.5', port=9999): Max retries exceeded with url: /v1/commands/ (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden')))