Hardware introspection failed in OpenStack 7

Solution Unverified - Updated -

Issue

Hardware introspection fails when adding nodes to a running overcloud and dracut falls into emergency mode.

Ironic-discoverd logs contains the following:

Nov 26 07:49:12 <hostname> ironic-discoverd[831]: ERROR:ironic_discoverd.utils:Unexpected exception during processing
Nov 26 07:49:12 <hostname> ironic-discoverd[831]: INFO:werkzeug:171.21.212.148 - - [26/Nov/2015 07:49:12] "POST /v1/continue HTTP/1.1" 400 -
Nov 26 11:26:26 <hostname> ironic-discoverd[831]: INFO:ironic_discoverd.utils:Node 057031e4-2d9f-48c9-8e68-53a6b1dfe395 is in mainten...s check
Nov 26 11:26:26 <hostname> ironic-discoverd[831]: INFO:werkzeug:172.21.41.4 - - [26/Nov/2015 11:26:26] "POST /v1/introspection/057031..." 202 -
Nov 26 11:26:26 <hostname> ironic-discoverd[831]: INFO:ironic_discoverd.introspect:Whitelisting MAC's [u'a0:36:9f:75:a8:22'] for node...irewall
Nov 26 11:30:45 <hostname> ironic-discoverd[831]: INFO:ironic_discoverd.plugins.standard:Discovered data: CPUs: 48 x86_64, memory 393...277 GiB
Nov 26 11:30:45 <hostname> ironic-discoverd[831]: INFO:ironic_discoverd.plugins.standard:PXE boot interface was a0:36:9f:75:a8:22
Nov 26 11:30:45 <hostname> ironic-discoverd[831]: WARNING:ironic_discoverd.plugins.standard:The following interfaces were invalid or ...u'ip': 
Nov 26 11:30:46 <hostname> ironic-discoverd[831]: INFO:ironic_discoverd.plugins.standard:Eligible interfaces are {u'p2p2': {u'ip': u'...8:22'}}
Nov 26 11:30:46 <hostname> ironic-discoverd[831]: INFO:ironic_discoverd.utils:Node 057031e4-2d9f-48c9-8e68-53a6b1dfe395 is in mainten...s check
Nov 26 11:32:48 <hostname> ironic-discoverd[831]: ERROR:ironicclient.common.http:Error contacting Ironic server: A port with MAC addr...1 of 61
Nov 26 11:32:48 <hostname> ironic-discoverd[831]: WARNING:ironic_discoverd.process:MAC a0:36:9f:75:a8:22 appeared in introspection da...kipping
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: ERROR:swiftclient:('Connection aborted.', error(111, 'ECONNREFUSED'))
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: Traceback (most recent call last):
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1261, in _retry
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: rv = func(self.url, self.token, *args, **kwargs)
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 731, in put_container
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: conn.request(method, path, '', headers)
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 212, in request
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: files=files, **self.requests_args)
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 195, in _request
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: return self.request_session.request(*arg, **kwarg)
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: resp = self.send(prep, **send_kwargs)
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: r = adapter.send(request, **kwargs)
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: raise ConnectionError(err, request=request)
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: ConnectionError: ('Connection aborted.', error(111, 'ECONNREFUSED'))
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: ERROR:ironic_discoverd.process:Unexpected exception during processing
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: Traceback (most recent call last):
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/ironic_discoverd/process.py", line 95, in process
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: return _process_node(ironic, node, node_info, cached_node)
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/ironic_discoverd/process.py", line 142, in _...ss_node
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: node_patches, port_patches = _run_post_hooks(node, ports, node_info)
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/ironic_discoverd/process.py", line 114, in _...t_hooks
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: node_info)
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/ironic_discoverd/plugins/edeploy.py", line 5..._update
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: json.dumps(node_info['data']))
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/ironic_discoverd/plugins/edeploy.py", line 6...ardware
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: swift_api.create_object(name, data)
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/ironic_discoverd/common/swift.py", line 110,..._object
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: self.connection.put_container(container)
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1337, in put_container
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: response_dict=response_dict)
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1261, in _retry
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: rv = func(self.url, self.token, *args, **kwargs)
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 731, in put_container
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: conn.request(method, path, '', headers)
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 212, in request
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: files=files, **self.requests_args)
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 195, in _request
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: return self.request_session.request(*arg, **kwarg)
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: resp = self.send(prep, **send_kwargs)
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: r = adapter.send(request, **kwargs)
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: raise ConnectionError(err, request=request)
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: ConnectionError: ('Connection aborted.', error(111, 'ECONNREFUSED'))
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: ERROR:ironic_discoverd.utils:Unexpected exception during processing
Nov 26 11:32:51 <hostname> ironic-discoverd[831]: INFO:werkzeug:171.21.212.149 - - [26/Nov/2015 11:32:51] "POST /v1/continue HTTP/1.1" 400 -

I am going to look after the ECONNREFUSED.

Regards

Environment

  • Red Hat OpenStack 7

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content