Ceph: After OSD node was added to Ceph cluster Open Stack Nova instances shut down

Issue

• An OSD node was added to an existing Ceph Storage cluster to expand space. After the OSD node was successfully added all Nova instances in Openstack region to shut down.

• From Nova Compute log:

015-09-03 11:22:17.903 18735 ERROR nova.openstack.common.periodic_task [-] Error during ComputeManager.update_available_resource: error calling connect
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task  File "/usr/lib/python2.7/site-packages/nova/openstack/common/periodic_task.py", line 198, in run_periodic_tasks
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task    task(self, context)
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6027, in update_available_resource
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task    rt.update_available_resource(context)
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task  File "/usr/lib/python2.7/site-packages/nova/compute/resource_tracker.py", line 370, in update_available_resource
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task    resources = self.driver.get_available_resource(self.nodename)
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5095, in get_available_resource
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task    stats = self.get_host_stats(refresh=True)
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6178, in get_host_stats
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task    return self.host_state.get_host_stats(refresh=refresh)
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6752, in get_host_stats
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task    self.update_status()
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6775, in update_status
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task    disk_info_dict = self.driver._get_local_gb_info()
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4688, in _get_local_gb_info
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task    info = LibvirtDriver._get_rbd_driver().get_pool_info()
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/rbd_utils.py", line 286, in get_pool_info
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task    with RADOSClient(self) as client:
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/rbd_utils.py", line 86, in __init__
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task    self.cluster, self.ioctx = driver._connect_to_rados(pool)
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/rbd_utils.py", line 110, in _connect_to_rados
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task    client.connect()
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task  File "/usr/lib/python2.7/site-packages/rados.py", line 419, in connect
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task    raise make_ex(ret, "error calling connect")
2015-09-03 11:22:17.903 18735 TRACE nova.openstack.common.periodic_task TimedOut: error calling connect
cant connect

• Display from 'rbd list' command on Ceph cluster:

terminate called after throwing an instance of 'ceph::buffer::malformed_input'
what():  buffer::malformed_input: unsupported bucket algorithm: 5