Ceph: Status shows "Reduced data availability: xx pgs inactive, xx pgs peering"
Issue
Ceph status returns "[WRN] PG_AVAILABILITY: Reduced data availability: xx pgs inactive, xx pgs peering"
Example:
# ceph -s
cluster:
id: 5b3c2fd{Cluster ID Obfuscated}16bfb00
health: HEALTH_WARN
1 MDSs report slow metadata IOs
1 MDSs report slow requests
1 MDSs behind on trimming
Reduced data availability: 6 pgs inactive, 6 pgs peering <--- Here
services:
mon: 3 daemons, quorum e,f,h (age 22h)
mgr: a(active, since 25h)
mds: 1/1 daemons up, 1 hot standby
osd: 6 osds: 6 up (since 3h), 6 in (since 3h); 60 remapped pgs
data:
volumes: 1/1 healthy
pools: 4 pools, 193 pgs
objects: 378.73k objects, 98 GiB
usage: 315 GiB used, 2.6 TiB / 2.9 TiB avail
pgs: 3.109% pgs not active
243503/1136184 objects misplaced (21.432%)
127 active+clean
59 active+remapped+backfill_wait
6 peering <--- Here
1 active+remapped+backfilling
io:
client: 102 B/s wr, 0 op/s rd, 0 op/s wr
recovery: 1.7 MiB/s, 8 objects/s
From Ceph Health Detail
# ceph health detail:
HEALTH_WARN 1 MDSs report slow metadata IOs; 1 MDSs report slow requests; 1 MDSs behind on trimming; Reduced data availability: 6 pgs inactive, 6 pgs peering
[WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
mds.ocs-storagecluster-cephfilesystem-b(mds.0): 100+ slow metadata IOs are blocked > 30 secs, oldest blocked for 13158 secs
[WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests
mds.ocs-storagecluster-cephfilesystem-b(mds.0): 84356 slow requests are blocked > 30 secs
[WRN] MDS_TRIM: 1 MDSs behind on trimming
mds.ocs-storagecluster-cephfilesystem-b(mds.0): Behind on trimming (261/128) max_segments: 128, num_segments: 261
[WRN] PG_AVAILABILITY: Reduced data availability: 6 pgs inactive, 6 pgs peering \
pg 1.25 is stuck peering for 86m, current state peering, last acting [2,3,1] \
pg 1.43 is stuck peering for 74m, current state peering, last acting [2,1,3] \
pg 1.44 is stuck peering for 86m, current state peering, last acting [2,3,5] Here
pg 1.63 is stuck peering for 97m, current state peering, last acting [2,5,3] /
pg 3.4 is stuck peering for 74m, current state peering, last acting [2,5,3] /
pg 3.1f is stuck peering for 86m, current state peering, last acting [2,5,0] /
Environment
Red Hat Ceph Storage (RHCS) 4
Red Hat Ceph Storage (RHCS) 5
Red Hat OpenShift Cluster Platform (OCP) 4
Red Hat OpenShift Container Storage (OCS) 4
Red Hat OpenShift Data Foundation (ODF) 4
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.