An instance is stuck in a building state due to qpid connection errors.

Solution Verified - Updated -

Issue

  • An instance is stuck in a building state due to qpid connection errors.
  • Following seen in /var/log/nova/scheduler.log
2014-06-10 12:12:14.709 30225 TRACE root 
2014-06-10 12:13:15.955 30225 ERROR root [-] Unexpected exception occurred 61 time(s)... retrying.
2014-06-10 12:13:15.955 30225 TRACE root Traceback (most recent call last):
2014-06-10 12:13:15.955 30225 TRACE root   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 78, in inner_func
2014-06-10 12:13:15.955 30225 TRACE root     return infunc(*args, **kwargs)
2014-06-10 12:13:15.955 30225 TRACE root   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 698, in _consumer_thread
2014-06-10 12:13:15.955 30225 TRACE root     self.consume()
2014-06-10 12:13:15.955 30225 TRACE root   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 689, in consume
2014-06-10 12:13:15.955 30225 TRACE root     it.next()
2014-06-10 12:13:15.955 30225 TRACE root   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 606, in iterconsume
2014-06-10 12:13:15.955 30225 TRACE root     yield self.ensure(_error_callback, _consume)
2014-06-10 12:13:15.955 30225 TRACE root   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 540, in ensure
2014-06-10 12:13:15.955 30225 TRACE root     return method(*args, **kwargs)
2014-06-10 12:13:15.955 30225 TRACE root   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 597, in _consume
2014-06-10 12:13:15.955 30225 TRACE root     nxt_receiver = self.session.next_receiver(timeout=timeout)
2014-06-10 12:13:15.955 30225 TRACE root   File "<string>", line 6, in next_receiver
2014-06-10 12:13:15.955 30225 TRACE root   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 660, in next_receiver
2014-06-10 12:13:15.955 30225 TRACE root     if self._ecwait(lambda: self.incoming, timeout):
2014-06-10 12:13:15.955 30225 TRACE root   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
2014-06-10 12:13:15.955 30225 TRACE root     result = self._ewait(lambda: self.closed or predicate(), timeout)
2014-06-10 12:13:15.955 30225 TRACE root   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 566, in _ewait
2014-06-10 12:13:15.955 30225 TRACE root     result = self.connection._ewait(lambda: self.error or predicate(), timeout)
2014-06-10 12:13:15.955 30225 TRACE root   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 209, in _ewait
2014-06-10 12:13:15.955 30225 TRACE root     self.check_error()
2014-06-10 12:13:15.955 30225 TRACE root   File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 202, in check_error
2014-06-10 12:13:15.955 30225 TRACE root     raise self.error
2014-06-10 12:13:15.955 30225 TRACE root InternalError: Traceback (most recent call last):
2014-06-10 12:13:15.955 30225 TRACE root   File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 651, in write
2014-06-10 12:13:15.955 30225 TRACE root     self._op_dec.write(*self._seg_dec.read())
2014-06-10 12:13:15.955 30225 TRACE root   File "/usr/lib/python2.6/site-packages/qpid/framing.py", line 269, in write
2014-06-10 12:13:15.955 30225 TRACE root     if self.op.headers is None:
2014-06-10 12:13:15.955 30225 TRACE root AttributeError: 'NoneType' object has no attribute 'headers'

Environment

  • Red Hat Enterprise Linux Openstack Platform 4.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content