Ceph - Failure to start migrated instance after OpenStack compute crash

Resolution

• In a Ceph monitor node check the capabilities of the OpenStack client.

  [root@ceph-mon]# ceph auth list
  

• If "osd blacklist" is missing then modify the client keyring for your OpenStack client to enable osd blacklist tomon 'allow r, allow command "osd blacklist".

  1. Export the CephX authentication keyring for your OSP environment, and create a backup:

     [root@ceph-mon]# ceph auth export client.openstack -o client.openstack.export
     [root@ceph-mon]# cp client.openstack.export client.openstack.export.backup
     

  2. Edit the client.${name}.export to modify the 'caps mon' line like the following:

     ⓘ You must use double quotes to enclose the value, and use escaped double quotes to enclose osd blacklist.

     ⓘ If after importing the authentication list you still have access denied problems, try this same procedure again but replace blacklist with blocklist. Newer versions might not work with blacklist anymore.

         caps mon = "allow r, allow command \"osd blacklist\""
     

  3. Check ceph auth list to ensure caps are correct.

     The below three items show the results of configuring the key. The first one shows a correct configured key, the other two show the results for common key configuration errors.

     ⓘ If you get either of the bad results, import your backup before you begin troubleshooting.

     1. ✅ This output shows a correct configuration:

        [root@ceph-mon]# ceph auth list | grep client.openstack -A3
        client.openstack
                key: AQAeo/5dAAAAABAAiZTSeas0vRYYeTcYmBeRtA==
                caps mon: allow r, allow command "osd blacklist"
                caps osd: allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rwx pool=images
        

     2. ❌ The phrase was enclosed in single quotes:

        [root@ceph-mon]# ceph auth list | grep client.openstack -A3
        client.openstack
                key: AQAeo/5dAAAAABAAiZTSeas0vRYYeTcYmBeRtA==
                caps mon: 'allow r, allow command "osd blacklist"'
                caps osd: allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rwx pool=images
        

     3. ❌ The internal quotes where not escaped:

        [root@ceph-mon]# ceph auth list | grep client.openstack -A3
        client.openstack
                key: AQAeo/5dAAAAABAAiZTSeas0vRYYeTcYmBeRtA==
                caps osd: allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rwx pool=images
        

  4. Save the file, then import it into Ceph:

     [root@ceph-mon]# ceph auth import -i client.openstack.export
     

• As an alternative, you can also change the active keyring with the following command, but it is recommended to take an export of the keyring first.
  The same note as above applies, if commands still fail with access denied, try to replace blacklist with blocklist.

  [root@ceph-mon]# ceph auth caps client.<ID> mon 'allow r, allow command "osd blacklist"' osd '<existing osd caps>'
  

  Example:

  [root@ceph-mon]# ceph auth caps client.cinder mon 'allow r, allow command "osd blacklist"' osd 'allow class-read object_prefix rbd_children, allow rwx pool=cinder, allow rx pool=glance'
  

  ⓘ There is no need to update actual keyring files on Overcloud nodes as actual key remains the same.