ironic-conductor is restarting in loop
Issue
- While performing scale out activity, getting below error while importing new nodes:
$ openstack overcloud node import ~/instack_env/allnodes.json
Error contacting Ironic server: Resource temporarily unavailable, please retry. (HTTP 503). Attempt 6 of 6
Exception occured while running the command
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/tripleoclient/command.py", line 32, in run
super(Command, self).run(parsed_args)
File "/usr/lib/python3.9/site-packages/osc_lib/command/command.py", line 39, in run
return super(Command, self).run(parsed_args)
File "/usr/lib/python3.9/site-packages/cliff/command.py", line 186, in run
return_code = self.take_action(parsed_args) or 0
File "/usr/lib/python3.9/site-packages/tripleoclient/v2/overcloud_node.py", line 108, in take_action
nodes = baremetal.register_or_update(
File "/usr/lib/python3.9/site-packages/tripleoclient/workflows/baremetal.py", line 90, in register_or_update
registered_nodes = node_utils.register_all_nodes(
File "/usr/lib/python3.9/site-packages/tripleo_common/utils/nodes.py", line 601, in register_all_nodes
node = _update_or_register_ironic_node(node, node_map, client=client)
File "/usr/lib/python3.9/site-packages/tripleo_common/utils/nodes.py", line 564, in _update_or_register_ironic_node
ironic_node = client.node.update(node_uuid, node_patch)
File "/usr/lib/python3.9/site-packages/ironicclient/v1/node.py", line 416, in update
return self._update(resource_id=node_id, patch=patch,
File "/usr/lib/python3.9/site-packages/ironicclient/common/base.py", line 271, in _update
resp, body = self.api.json_request(method, url, **kwargs)
File "/usr/lib/python3.9/site-packages/ironicclient/common/http.py", line 405, in json_request
resp = self._http_request(url, method, **kwargs)
File "/usr/lib/python3.9/site-packages/ironicclient/common/http.py", line 287, in wrapper
return func(self, url, method, **kwargs)
File "/usr/lib/python3.9/site-packages/ironicclient/common/http.py", line 386, in _http_request
raise exc.from_response(resp, error_json.get('error_message'),
ironicclient.common.apiclient.exceptions.ServiceUnavailable: Resource temporarily unavailable, please retry. (HTTP 503)
Resource temporarily unavailable, please retry. (HTTP 503)
- One of the ironic pod is in starting state as you can see below:
3554dfa89963 undercloudnew.ctlplane.localdomain:8787/ttsl-rh_9_osp_17_1-redhat_9_rhosp-osp_containers-ironic-conductor:17.1 kolla_start 46 minutes ago Up 1 second (starting) ironic_conductor
e50a29a56088 undercloudnew.ctlplane.localdomain:8787/ttsl-rh_9_osp_17_1-redhat_9_rhosp-osp_containers-ironic-pxe:17.1 kolla_start 46 minutes ago Up 46 minutes (healthy) ironic_pxe_tftp
88e9f5881316 undercloudnew.ctlplane.localdomain:8787/ttsl-rh_9_osp_17_1-redhat_9_rhosp-osp_containers-ironic-pxe:17.1 kolla_start 46 minutes ago Up 46 minutes (healthy) ironic_pxe_http
b6a5585bd33a undercloudnew.ctlplane.localdomain:8787/ttsl-rh_9_osp_17_1-redhat_9_rhosp-osp_containers-neutron-dhcp-agent:17.1 /usr/sbin/dnsmasq... 45 minutes ago Up 45 minutes neutron-dnsmasq-qdhcp-d928afb6-382b-4853-9749-f0ed45b4b518
Environment
- Red Hat OpenStack Platform 17.1 (RHOSP)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.