Ceph: After OSD node was added to Ceph cluster Open Stack Nova instances shut down
Issue
-
An OSD node was added to an existing Ceph Storage cluster to expand space. After the OSD node was successfully added all Nova instances in Openstack region to shut down.
-
From Nova Compute log:
015-09-03 11:22:17.903 18735 ERROR nova.openstack.common.periodic_task [-] Error during ComputeManager.update_available_resource: error calling connect
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/nova/openstack/common/periodic_task.py", line 198, in run_periodic_tasks
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task task(self, context)
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6027, in update_available_resource
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task rt.update_available_resource(context)
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 370, in update_available_resource
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task resources = self.driver.get_available_resource(self.nodename)
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5095, in get_available_resource
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task stats = self.get_host_stats(refresh=True)
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6178, in get_host_stats
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task return self.host_state.get_host_stats(refresh=refresh)
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6752, in get_host_stats
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task self.update_status()
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6775, in update_status
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task disk_info_dict = self.driver._get_local_gb_info()
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4688, in _get_local_gb_info
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task info = LibvirtDriver._get_rbd_driver().get_pool_info()
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/rbd_utils.py", line 286, in get_pool_info
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task with RADOSClient(self) as client:
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/rbd_utils.py", line 86, in __init__
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task self.cluster, self.ioctx = driver._connect_to_rados(pool)
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/rbd_utils.py", line 110, in _connect_to_rados
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task client.connect()
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/site-packages/rados.py", line 419, in connect
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task raise make_ex(ret, "error calling connect")
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task TimedOut: error calling connect
cant connect
- Display from 'rbd list' command on Ceph cluster:
terminate called after throwing an instance of 'ceph::buffer::malformed_input'
what(): buffer::malformed_input: unsupported bucket algorithm: 5
Environment
- Red Had Ceph Storage 1.3
- Red Hat Open Stack 6.0
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
