ironic-conductor is restarting in loop

Solution In Progress - Updated -

Issue

  • While performing scale out activity, getting below error while importing new nodes:
$ openstack overcloud node import  ~/instack_env/allnodes.json
Error contacting Ironic server: Resource temporarily unavailable, please retry. (HTTP 503). Attempt 6 of 6
Exception occured while running the command
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 32, in run
    super(Command, self).run(parsed_args)
  File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python3.9/site-packages/cliff/command.py", line 186, in run
    return_code = self.take_action(parsed_args) or 0
  File "/usr/lib/python3.9/site-packages/tripleoclient/v2/overcloud_node.py", line 108, in take_action
    nodes = baremetal.register_or_update(
  File "/usr/lib/python3.9/site-packages/tripleoclient/workflows/baremetal.py", line 90, in register_or_update
    registered_nodes = node_utils.register_all_nodes(
  File "/usr/lib/python3.9/site-packages/tripleo_common/utils/nodes.py", line 601, in register_all_nodes
    node = _update_or_register_ironic_node(node, node_map, client=client)
  File "/usr/lib/python3.9/site-packages/tripleo_common/utils/nodes.py", line 564, in _update_or_register_ironic_node
    ironic_node = client.node.update(node_uuid, node_patch)
  File "/usr/lib/python3.9/site-packages/ironicclient/v1/node.py", line 416, in update
    return self._update(resource_id=node_id, patch=patch,
  File "/usr/lib/python3.9/site-packages/ironicclient/common/base.py", line 271, in _update
    resp, body = self.api.json_request(method, url, **kwargs)
  File "/usr/lib/python3.9/site-packages/ironicclient/common/http.py", line 405, in json_request
    resp = self._http_request(url, method, **kwargs)
  File "/usr/lib/python3.9/site-packages/ironicclient/common/http.py", line 287, in wrapper
    return func(self, url, method, **kwargs)
  File "/usr/lib/python3.9/site-packages/ironicclient/common/http.py", line 386, in _http_request
    raise exc.from_response(resp, error_json.get('error_message'),
ironicclient.common.apiclient.exceptions.ServiceUnavailable: Resource temporarily unavailable, please retry. (HTTP 503)
Resource temporarily unavailable, please retry. (HTTP 503)
  • One of the ironic pod is in starting state as you can see below:
3554dfa89963  undercloudnew.ctlplane.localdomain:8787/ttsl-rh_9_osp_17_1-redhat_9_rhosp-osp_containers-ironic-conductor:17.1           kolla_start           46 minutes ago  Up 1 second (starting)               ironic_conductor
e50a29a56088  undercloudnew.ctlplane.localdomain:8787/ttsl-rh_9_osp_17_1-redhat_9_rhosp-osp_containers-ironic-pxe:17.1                 kolla_start           46 minutes ago  Up 46 minutes (healthy)              ironic_pxe_tftp
88e9f5881316  undercloudnew.ctlplane.localdomain:8787/ttsl-rh_9_osp_17_1-redhat_9_rhosp-osp_containers-ironic-pxe:17.1                 kolla_start           46 minutes ago  Up 46 minutes (healthy)              ironic_pxe_http
b6a5585bd33a  undercloudnew.ctlplane.localdomain:8787/ttsl-rh_9_osp_17_1-redhat_9_rhosp-osp_containers-neutron-dhcp-agent:17.1         /usr/sbin/dnsmasq...  45 minutes ago  Up 45 minutes                        neutron-dnsmasq-qdhcp-d928afb6-382b-4853-9749-f0ed45b4b518

Environment

  • Red Hat OpenStack Platform 17.1 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content