Post Fast Forward Upgrade ceph error: auth entities have invalid capabilities
Environment
- Red Hat Openstack Platform (RHOSP) version is 13.
- Fast Forward Upgrade (FFU) install RHOSP 16.1 .
Issue
- After the Fast Forward Upgrade (FFU) from Red Hat Openstack Platform (RHOSP) version 13 Z 14 to RHOSP 16.1 Z 3, the ceph health status is found to HEALTH_ERR.
Resolution
-
An upstream patch to fix this issue is present. Please follow the following link: Link to the FIX.
-
Until the patch installation is possible to manually remove the osd cap from
client.openstack
. To manually do that, please follow the following steps:- ceph auth export client.openstack > client.openstack - edit client.openstack and remove the osd caps line - reimport the key without the osd section
Root Cause
-
The issue is the osd where the cap was missing:
client.openstack key: <KEY> caps: [mgr] allow * caps: [mon] profile rbd caps: [osd]
-
In this case ceph is used only using rgw as a swift backend, so it is expected that no capabilities are added to osd.
Diagnostic Steps
-
Verify the ceph status using the following commands:
ceph -s ceph health detail ceph auth list| grep -A10 client.openstack
-
Here is an example of the results of the above commands:
[root@ctr0 ~]# ceph -s cluster: id: <ID> health: HEALTH_ERR 1 auth entities have invalid capabilities services: mon: 3 daemons, quorum ctr0,ctr1,ctr2 (age 15h) mgr: ctr0(active, since 16h), standbys: ctr1, ctr2 osd: 9 osds: 9 up (since 6d), 9 in (since 6d) rgw: 3 daemons active (ctr0.rgw0, ctr1.rgw0, ctr2.rgw0) task status: data: pools: 11 pools, 96 pgs objects: 1.48k objects, 4.9 GiB usage: 2.6 TiB used, 9.8 TiB / 12 TiB avail pgs: 96 active+clean [root@ctr0 ~]# ceph health detail HEALTH_ERR 1 auth entities have invalid capabilities AUTH_BAD_CAPS 1 auth entities have invalid capabilities client.openstack osd capability parse failed, stopped at '' of '' [root@ctr0 ~]# ceph auth list | grep -A10 client.openstack installed auth entries: client.openstack key: <KEY> caps: [mgr] allow * caps: [mon] profile rbd caps: [osd] client.radosgw key: <KEY> caps: [mgr] allow * caps: [mon] allow rw caps: [osd] allow rwx client.rgw.ctr0
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments