An instance is stuck in a building state due to qpid connection errors.
Issue
- An instance is stuck in a building state due to qpid connection errors.
- Following seen in
/var/log/nova/scheduler.log
2014-06-10 12:12:14.709 30225 TRACE root
2014-06-10 12:13:15.955 30225 ERROR root [-] Unexpected exception occurred 61 time(s)... retrying.
2014-06-10 12:13:15.955 30225 TRACE root Traceback (most recent call last):
2014-06-10 12:13:15.955 30225 TRACE root File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 78, in inner_func
2014-06-10 12:13:15.955 30225 TRACE root return infunc(*args, **kwargs)
2014-06-10 12:13:15.955 30225 TRACE root File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 698, in _consumer_thread
2014-06-10 12:13:15.955 30225 TRACE root self.consume()
2014-06-10 12:13:15.955 30225 TRACE root File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 689, in consume
2014-06-10 12:13:15.955 30225 TRACE root it.next()
2014-06-10 12:13:15.955 30225 TRACE root File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 606, in iterconsume
2014-06-10 12:13:15.955 30225 TRACE root yield self.ensure(_error_callback, _consume)
2014-06-10 12:13:15.955 30225 TRACE root File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 540, in ensure
2014-06-10 12:13:15.955 30225 TRACE root return method(*args, **kwargs)
2014-06-10 12:13:15.955 30225 TRACE root File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 597, in _consume
2014-06-10 12:13:15.955 30225 TRACE root nxt_receiver = self.session.next_receiver(timeout=timeout)
2014-06-10 12:13:15.955 30225 TRACE root File "<string>", line 6, in next_receiver
2014-06-10 12:13:15.955 30225 TRACE root File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 660, in next_receiver
2014-06-10 12:13:15.955 30225 TRACE root if self._ecwait(lambda: self.incoming, timeout):
2014-06-10 12:13:15.955 30225 TRACE root File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 50, in _ecwait
2014-06-10 12:13:15.955 30225 TRACE root result = self._ewait(lambda: self.closed or predicate(), timeout)
2014-06-10 12:13:15.955 30225 TRACE root File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 566, in _ewait
2014-06-10 12:13:15.955 30225 TRACE root result = self.connection._ewait(lambda: self.error or predicate(), timeout)
2014-06-10 12:13:15.955 30225 TRACE root File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 209, in _ewait
2014-06-10 12:13:15.955 30225 TRACE root self.check_error()
2014-06-10 12:13:15.955 30225 TRACE root File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 202, in check_error
2014-06-10 12:13:15.955 30225 TRACE root raise self.error
2014-06-10 12:13:15.955 30225 TRACE root InternalError: Traceback (most recent call last):
2014-06-10 12:13:15.955 30225 TRACE root File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 651, in write
2014-06-10 12:13:15.955 30225 TRACE root self._op_dec.write(*self._seg_dec.read())
2014-06-10 12:13:15.955 30225 TRACE root File "/usr/lib/python2.6/site-packages/qpid/framing.py", line 269, in write
2014-06-10 12:13:15.955 30225 TRACE root if self.op.headers is None:
2014-06-10 12:13:15.955 30225 TRACE root AttributeError: 'NoneType' object has no attribute 'headers'
Environment
- Red Hat Enterprise Linux Openstack Platform 4.
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.