Live Migration not working failing when using ceph as a backend

Issue

Trying to validate support for live migration of instances with OSP5 and RHEL7. Nova configuration is using images_type rbd and both source and dest are able to create rbd backed volumes properly.

nova.conf has:

live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE

/etc/libvirt/libvirtd.conf contains:

listen_tls=0, auth_tcp="none"

/etc/sysconfig/libvirtd contains:

 LIBVIRTD_ARGS="--listen"

Currently Instances get stuck in migrating state and never error or complete. Retrieved below stack trace from nova-compute.log after enabling debug.

2014-10-09 04:45:59.342 6282 ERROR nova.compute.manager [req-dcef86a2-81b4-4870-becd-ade01f200489 3671e07e02c440198b4acdd0e12c747e 7edb1fe949614b30ab04dc256a5667fe] [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa] Pre live migration failed at svl6-csl-b-nova1-003
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa] Traceback (most recent call last):
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4543, in live_migration
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]     block_migration, disk, dest, migrate_data)
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]   File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 595, in pre_live_migration
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]     disk=disk, migrate_data=migrate_data)
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 150, in call
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]     wait_for_reply=True, timeout=timeout)
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]   File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]     timeout=timeout)
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]     return self._send(target, ctxt, message, wait_for_reply, timeout)
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]     raise result
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa] DestinationDiskExists_Remote: The supplied disk path (/var/lib/nova/instances/ccfc08c6-2068-408d-9f6f-4959b2f87afa) already exists, it is expected not to exist.
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa] Traceback (most recent call last):
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa] 
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]     incoming.message))
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa] 
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]     return self._do_dispatch(endpoint, method, ctxt, args)
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa] 
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]     result = getattr(endpoint, method)(ctxt, **new_args)
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa] 
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in decorated_function
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]     return function(self, context, *args, **kwargs)
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa] 
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]     payload)
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa] 
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]     six.reraise(self.type_, self.value, self.tb)
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa] 
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]     return f(self, context, *args, **kw)
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa] 
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 303, in decorated_function
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]     e, sys.exc_info())
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa] 
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]     six.reraise(self.type_, self.value, self.tb)
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa] 
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 290, in decorated_function
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]     return function(self, context, *args, **kwargs)
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa] 
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4499, in pre_live_migration
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]     migrate_data)
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa] 
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4616, in pre_live_migration
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa]     raise exception.DestinationDiskExists(path=instance_dir)
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa] 
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa] DestinationDiskExists: The supplied disk path (/var/lib/nova/instances/ccfc08c6-2068-408d-9f6f-4959b2f87afa) already exists, it is expected not to exist.
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa] 
2014-10-09 04:45:59.342 6282 TRACE nova.compute.manager [instance: ccfc08c6-2068-408d-9f6f-4959b2f87afa] 
2014-10-09 04:45:59.418 6282 DEBUG nova.openstack.common.lockutils [req-dcef86a2-81b4-4870-becd-ade01f200489 3671e07e02c440198b4acdd0e12c747e 7edb1fe949614b30ab04dc256a5667fe] Got semaphore "compute_resources" lock /usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py:168
2014-10-09 04:45:59.418 6282 DEBUG nova.openstack.common.lockutils [req-dcef86a2-81b4-4870-becd-ade01f200489 3671e07e02c440198b4acdd0e12c747e 7edb1fe949614b30ab04dc256a5667fe] Got semaphore / lock "update_usage" inner /usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py:248
2014-10-09 04:45:59.485 6282 DEBUG nova.openstack.common.lockutils [req-dcef86a2-81b4-4870-becd-ade01f200489 3671e07e02c440198b4acdd0e12c747e 7edb1fe949614b30ab04dc256a5667fe] Semaphore / lock released "update_usage" inner /usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py:252
2014-10-09 04:45:59.577 6282 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: The supplied disk path (/var/lib/nova/instances/ccfc08c6-2068-408d-9f6f-4959b2f87afa) already exists, it is expected not to exist.
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
    incoming.message))

  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)

  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
    payload)

  File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
    six.reraise(self.type_, self.value, self.tb)

  File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
    return f(self, context, *args, **kw)

  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 303, in decorated_function
    e, sys.exc_info())

  File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
    six.reraise(self.type_, self.value, self.tb)

  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 290, in decorated_function
    return function(self, context, *args, **kwargs)

  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4499, in pre_live_migration
    migrate_data)

  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4616, in pre_live_migration
    raise exception.DestinationDiskExists(path=instance_dir)

DestinationDiskExists: The supplied disk path (/var/lib/nova/instances/ccfc08c6-2068-408d-9f6f-4959b2f87afa) already exists, it is expected not to exist.