Post Fast Forward Upgrade ceph error: auth entities have invalid capabilities

Solution Verified - Updated -

Environment

  • Red Hat Openstack Platform (RHOSP) version is 13.
  • Fast Forward Upgrade (FFU) install RHOSP 16.1 .

Issue

  • After the Fast Forward Upgrade (FFU) from Red Hat Openstack Platform (RHOSP) version 13 Z 14 to RHOSP 16.1 Z 3, the ceph health status is found to HEALTH_ERR.

Resolution

  • An upstream patch to fix this issue is present. Please follow the following link: Link to the FIX.

  • Until the patch installation is possible to manually remove the osd cap from client.openstack. To manually do that, please follow the following steps:

    - ceph auth export client.openstack > client.openstack
    - edit client.openstack and remove the osd caps line
    - reimport the key without the osd section
    

Root Cause

  • The issue is the osd where the cap was missing:

    client.openstack
            key: <KEY>
            caps: [mgr] allow *
            caps: [mon] profile rbd
            caps: [osd] 
    
  • In this case ceph is used only using rgw as a swift backend, so it is expected that no capabilities are added to osd.

Diagnostic Steps

  • Verify the ceph status using the following commands:

    ceph -s
    ceph health detail
    ceph auth list| grep -A10 client.openstack
    
  • Here is an example of the results of the above commands:

    [root@ctr0 ~]# ceph -s
      cluster:
        id:     <ID>
        health: HEALTH_ERR
                1 auth entities have invalid capabilities
    
      services:
        mon: 3 daemons, quorum ctr0,ctr1,ctr2 (age 15h)
        mgr: ctr0(active, since 16h), standbys: ctr1, ctr2
        osd: 9 osds: 9 up (since 6d), 9 in (since 6d)
        rgw: 3 daemons active (ctr0.rgw0, ctr1.rgw0, ctr2.rgw0)
    
      task status:
    
      data:
        pools:   11 pools, 96 pgs
        objects: 1.48k objects, 4.9 GiB
        usage:   2.6 TiB used, 9.8 TiB / 12 TiB avail
        pgs:     96 active+clean
    
    [root@ctr0 ~]# ceph health detail
    HEALTH_ERR 1 auth entities have invalid capabilities
    AUTH_BAD_CAPS 1 auth entities have invalid capabilities
        client.openstack osd capability parse failed, stopped at '' of ''
    
    [root@ctr0 ~]# ceph auth list | grep -A10 client.openstack
    installed auth entries:
    
    client.openstack
            key: <KEY>
            caps: [mgr] allow *
            caps: [mon] profile rbd
            caps: [osd] 
    client.radosgw
            key: <KEY>
            caps: [mgr] allow *
            caps: [mon] allow rw
            caps: [osd] allow rwx
    client.rgw.ctr0
    

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments