Ceph / ODF: HEALTH_WARN pgs not scrubbed / deep-scrubbed in time while some PGs remain in scrubbing / deep-scrubbing "indefinitely".

Issue

HEALTH_WARN pgs not scrubbed / deep-scrubbed in time while some PGs remain in scrubbing / deep-scrubbing "indefinitely".

It is possible for the PG currently being scrubbed / deep-scrubbed to hang (become a zombie or be in a defunct state). If this issue persists, it will prevent other PGs from being scrubbed until the "HEALTH_WARN" is triggered

  health: HEALTH_WARN
            468 pgs not deep-scrubbed in time   <-- [2]

  services:
    mon: 3 daemons, quorum a,b,c (age 29h)
    mgr: a(active, since 29h), standbys: b
    mds: 8/8 daemons up, 1 standby
    osd: 326 osds: 326 up (since 5w), 326 in (since 5w)
    rgw: 30 daemons active (15 hosts, 2 zones)

  data:
    volumes: 2/2 healthy
    pools:   20 pools, 19441 pgs
    objects: 2.51G objects, 415 TiB
    usage:   899 TiB used, 1.5 PiB / 2.4 PiB avail
    pgs:     19423 active+clean
             18    active+clean+scrubbing+deep      <-- [3]

  io:
    client:   1.2 GiB/s rd, 699 MiB/s wr, 107.69k op/s rd, 11.06k op/s wr

$ ceph health detail
[....]
[WRN] PG_NOT_SCRUBBED: 468 pgs not scrubbed in time
    pg 31.1c19 not deep-scrubbed since 2023-10-09T03:21:24.661671+0000
    pg 31.1ffd not deep-scrubbed since 2023-10-09T05:15:54.268777+0000
    pg 31.1ff6 not deep-scrubbed since 2023-10-09T02:38:08.356159+0000
    pg 31.1fbd not deep-scrubbed since 2023-10-09T01:33:09.841035+0000
[....]

• This solution is only valid if there are PGs deep-scrubbing tasks active [3] and also PGs not deep-scrubbed in time [2].
• Before applying this solution, ensure the system is not affected by the mClock Scheduler, KCS 7092973

• For "PGs not deep-scrubbed in time" with no active deep-scrubbing activity, see KCS Article 7049262.

  See the following two solutions for related issues.

Ceph: Deep scrubs taking too long
Ceph pgs not deep-scrubbed in time