Red Hat Ceph Storage: aborted: Failure prepping block device

Solution Unverified - Updated -

Environment

  • Red Hat Ceph Storage
  • Red Hat Openstack 6
  • Red Hat Openstack 7

Issue

We are trying to build an instance using a 100GB rhel6.6 raw image and get the following error.

Message
Build of instance 516f17c4-580d-4d9b-8588-1fc63bd6962b aborted: Failure prepping block device.
Code
500
Details
File "/usr/lib/python2.7/site-packages/nova/compute/", line 2075, in do_build_and_run_instance filter_properties) File "/usr/lib/python2.7/site-packages/nova/compute/", line 2174, in _build_and_run_instance 'create.error', fault=e) File "/usr/lib/python2.7/site-packages/nova/openstack/common/", line 82, in exit six.reraise(self.type, self.value, self.tb) File "/usr/lib/python2.7/site-packages/nova/compute/", line 2147, in _build_and_run_instance block_device_mapping) as resources: File "/usr/lib64/python2.7/", line 17, in enter return self.gen.next() File "/usr/lib/python2.7/site-packages/nova/compute/", line 2285, in _build_resources reason=msg)

Resolution

  • Modify /etc/glance/glance-api.conf to include the following.
enable_v2_api=True
show_image_direct_url=True
  • Restart the glance-api service and nova-compute.
service openstack-glance-api restart
service nova-compute restart

Red Hat Openstack 7 Director
For an automated way to set the show_image_direct_url parameter when a ceph backend is being used with OSP-d, bugzilla Enable glance-api show_image_direct_url for COW has been filed to track the backport of upstream fix https://review.openstack.org/#/c/226324/ .

As a workaround the following modification needs to be done to the OSP-d template to achieve the same. Add the glance::api::show_image_direct_url line to the controller.yaml hieradata file:

$ diff -u /usr/share/openstack-tripleo-heat-templates/puppet/hieradata/controller.yaml my_templates/puppet/hieradata/controller.yaml 

--- /usr/share/openstack-tripleo-heat-templates/puppet/hieradata/controller.yaml        2015-10-01 10:30:10.000000000 -0400
+++ my_templates/puppet/hieradata/controller.yaml       2015-11-23 14:57:02.466280952 -0500
@@ -62,6 +62,7 @@

 # glance
 glance::api::pipeline: 'keystone'
+glance::api::show_image_direct_url: true
 glance::registry::pipeline: 'keystone'
 glance::backend::swift::swift_store_create_container_on_put: true
 glance_file_pcmk_directory: '/var/lib/glance/images'

Root Cause

  • The operation times out because the full image is being pushed from Glance to Cinder instead of using copy on write snapshots.

Diagnostic Steps

  • Find an instance of the error in the nova-compute.log
2015-10-06 12:07:44.155 15544 ERROR nova.compute.manager [-] [instance: 139da521-38a9-4cd8-a55e-b8c67518d53b] Build of instance 139da521-38a9-4cd8-a55e-b8c67518d53b aborted: Failure prepping block device.
  • If we dump out the entire transaction by grepping for the instance ID we can see there is an earlier timeout error.
2015-10-06 12:07:44.132 15544 TRACE nova.compute.manager [instance: 139da521-38a9-4cd8-a55e-b8c67518d53b] VolumeNotCreated: Volume 9a8a9c83-a757-43cd-bba5-3d94a3336b49 did not finish being created even after we waited 188 seconds or 61 attempts.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments