Evacuation fails with `InvalidSharedStorage: Invalid state of instance files on shared storage` when using Instance High Availability (High Availability for Compute Instances) with Ceph in Red Hat OpenStack Platform 13

Issue

• Evacuation fails with InvalidSharedStorage: Invalid state of instance files on shared storage when using Instance High Availability (High Availability for Compute Instances) with Ceph in Red Hat OpenStack Platform 13.

• Instance evacuation fails and destination compute host reports:

2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [req-e1ec84ef-bece-4f4f-8d4e-f2f40ee293f9 2922683c94c042a29b23528738264aa6 27e66c047b6b445880dea615d65a9b25 - default default] [instance:
 88fd302f-ffea-4d39-b64b-ccf9a42a133e] Setting instance vm_state to ERROR: InvalidSharedStorage: Invalid state of instance files on shared storage
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e] Traceback (most recent call last):
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7442, in _error_
out_instance_on_exception
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e]     yield
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2888, in rebuild_instance
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e]     migration, request_spec)
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2950, in _do_rebuild_instance_with_claim
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e]     self._do_rebuild_instance(*args, **kwargs)
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2991, in _do_rebuild_instance
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e]     _("Invalid state of instance files on shared"
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e] InvalidSharedStorage: Invalid state of instance files on shared storage
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e] 
2018-11-22 04:34:05.658 1 INFO nova.compute.manager [req-e1ec84ef-bece-4f4f-8d4e-f2f40ee293f9 2922683c94c042a29b23528738264aa6 27e66c047b6b445880dea615d65a9b25 - default default] [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e] Successfully reverted task state from rebuilding on failure for instance.
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e] Traceback (most recent call last):
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 7442, in _error_out_instance_on_exception
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e]     yield
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2888, in rebuild_instance
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e]     migration, request_spec)
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2950, in _do_rebuild_instance_with_claim
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e]     self._do_rebuild_instance(*args, **kwargs)
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2991, in _do_rebuild_instance
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e]     _("Invalid state of instance files on shared"
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e] InvalidSharedStorage: Invalid state of instance files on shared storage
2018-11-22 04:34:05.461 1 ERROR nova.compute.manager [instance: 88fd302f-ffea-4d39-b64b-ccf9a42a133e]

• Boot from volume instance with Ceph:

cat var/lib/config-data/puppet-generated/cinder/etc/cinder/cinder.conf
(...)
enabled_backends=tripleo_ceph
(...)
[tripleo_ceph]
backend_host=hostgroup
volume_backend_name=tripleo_ceph
volume_driver=cinder.volume.drivers.rbd.RBDDriver
rbd_ceph_conf=/etc/ceph/ceph.conf
rbd_user=openstack
rbd_pool=volumes