Ceph: OSD_TOO_MANY_REPAIRS: Too many repaired reads on 1 OSDs along with "PG_DEGRADED" and "SLOW_OPS".
Issue
The output of ceph health detail
shows "OSD_TOO_MANY_REPAIRS
": Too many repaired reads on 1 OSDs along with "PG_DEGRADED
" and "SLOW_OPS
".
Example:
[root@edon-00 ~]# ceph health detail
HEALTH_WARN Too many repaired reads on 1 OSDs; Degraded data redundancy: 1/104988123404 objects degraded (0.000%), 1 pg degraded; 1 slow ops, oldest one blocked for 25234 sec, osd.33 has slow ops
[WRN] OSD_TOO_MANY_REPAIRS: Too many repaired reads on 1 OSDs
osd.33 had 246285 reads repaired
[WRN] PG_DEGRADED: Degraded data redundancy: 1/104988123404 objects degraded (0.000%), 1 pg degraded
pg 8.53a is active+recovering+degraded+repair, acting [33,50,14]
[WRN] SLOW_OPS: 1 slow ops, oldest one blocked for 25234 sec, osd.33 has slow ops
Also, this will be seen in the osd log files:
[root@edon-02 ~]# vim ceph-osd.33.log
2023-12-08T12:45:40.749+0000 log_channel(cluster) log [ERR] : 8.53a full-object read crc 0x6b9bf47a != expected 0xffffffff on 8:5ca91bd9:::300017b6c72.00000000:head
2023-12-08T12:45:40.749+0000 log_channel(cluster) log [ERR] : 8.53a missing primary copy of 8:5ca91bd9:::300017b6c72.00000000:head, will try copies on 14,50
2023-12-08T12:45:40.769+0000 osd.33 592819 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.213685203.0:1223450696 8.53a 8:5ca91bd9:::300017b6c72.00000000:head [read 0~16384 [1508@0] out=16384b] snapc 0=[] RETRY=82 ondisk+retry+read+known_if_redirected e592608)
Environment
Red Hat Ceph Storage (RHCS) 5.x
Red Hat Ceph Storage (RHCS) 6.x
Red Hat Ceph Storage (RHCS) 7.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.