Post Fast Forward Upgrade ceph error: auth entities have invalid capabilities

Solution Verified - Updated -

Environment

  • Red Hat Openstack Platform (RHOSP) version is 13.
  • Fast Forward Upgrade (FFU) install RHOSP 16.1 .

Issue

  • After the Fast Forward Upgrade (FFU) from Red Hat Openstack Platform (RHOSP) version 13 Z 14 to RHOSP 16.1 Z 3, the ceph health status is found to HEALTH_ERR.

Resolution

  • An upstream patch to fix this issue is present. Please follow the following link: Link to the FIX.

  • Until the patch installation is possible to manually remove the osd cap from client.openstack. To manually do that, please follow the following steps:

    - ceph auth export client.openstack > client.openstack
- edit client.openstack and remove the osd caps line
- reimport the key without the osd section

Root Cause

  • The issue is the osd where the cap was missing:

    client.openstack
        key: <KEY>
        caps: [mgr] allow *
        caps: [mon] profile rbd
        caps: [osd]

  • In this case ceph is used only using rgw as a swift backend, so it is expected that no capabilities are added to osd.

Diagnostic Steps

  • Verify the ceph status using the following commands:

    ceph -s
ceph health detail
ceph auth list| grep -A10 client.openstack

  • Here is an example of the results of the above commands:

    [root@ctr0 ~]# ceph -s
  cluster:
    id:     <ID>
    health: HEALTH_ERR
            1 auth entities have invalid capabilities

  services:
    mon: 3 daemons, quorum ctr0,ctr1,ctr2 (age 15h)
    mgr: ctr0(active, since 16h), standbys: ctr1, ctr2
    osd: 9 osds: 9 up (since 6d), 9 in (since 6d)
    rgw: 3 daemons active (ctr0.rgw0, ctr1.rgw0, ctr2.rgw0)

  task status:

  data:
    pools:   11 pools, 96 pgs
    objects: 1.48k objects, 4.9 GiB
    usage:   2.6 TiB used, 9.8 TiB / 12 TiB avail
    pgs:     96 active+clean

[root@ctr0 ~]# ceph health detail
HEALTH_ERR 1 auth entities have invalid capabilities
AUTH_BAD_CAPS 1 auth entities have invalid capabilities
    client.openstack osd capability parse failed, stopped at '' of ''

[root@ctr0 ~]# ceph auth list | grep -A10 client.openstack
installed auth entries:

client.openstack
        key: <KEY>
        caps: [mgr] allow *
        caps: [mon] profile rbd
        caps: [osd] 
client.radosgw
        key: <KEY>
        caps: [mgr] allow *
        caps: [mon] allow rw
        caps: [osd] allow rwx
client.rgw.ctr0

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments