ODF: Ceph PGs inconsistent - "bluestore(/var/lib/ceph/osd/ceph-num) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x6000, got 0x30ab1d33, expected 0xc5c08f40"

Issue

Note: This issue is platform-specific and affects the IBM POWER-based ppc64 or ppc64le architecture ONLY. Architectures such as x86 ARE NOT affected by this issue. If you're reviewing this solution and ARE NOT on IBM POWER-based ppc64 or ppc64le architecture, open a Red Hat Support case to have the OpenShift Data Foundation or Ceph team review the issue.

Ceph PGs inconsistent - "bluestore(/var/lib/ceph/osd/ceph-num) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x6000, got 0x30ab1d33, expected 0xc5c08f40"

For the PowerPC Hardware platform (ppc64le) utilizing certain OCP/ODF releases between 4.12 and 4.15, inconsistent PGs can be observed.

Example of inconsistent PGs:

-bash 5.1 $ ceph status
...
  data:
    volumes: 1/1 healthy
    pools:   12 pools, 185 pgs
    objects: 161.79k objects, 497 GiB
    usage:   1.4 TiB used, 1.6 TiB / 3 TiB avail
    pgs:     3/485367 objects degraded (0.001%)
             1/161789 objects unfound (0.001%)
             175 active+clean
             9   active+clean+inconsistent
             1   active+recovery_unfound+undersized+degraded+remapped

OSD logs reporting crc32 checksum errors:

bluestore(/var/lib/ceph/osd/ceph-16) _verify_csum bad crc32c/0x1000 checksum at blob offset 0xe000, got 0xee0556b6, expected 0xf2e78d2a, device location [0x2e421000~1000], logical extent 0x32e000~1000, object #10:b5391834:::200.0000003c:head#
bluestore(/var/lib/ceph/osd/ceph-11) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x2000, got 0x513ec87a, expected 0xc0838dd6, device location [0x20b60000~1000], logical extent 0xf2000~1000, object #10:f376ae95:::200.00000016:head#
bluestore(/var/lib/ceph/osd/ceph-12) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x4000, got 0x33d267cf, expected 0x22d56e3b, device location [0x31848000~1000], logical extent 0x254000~1000, object #1:0c52f2e6:::rbd_data.3e5322cd8965.000000000000002f:head#
bluestore(/var/lib/ceph/osd/ceph-14) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x4000, got 0xda15e262, expected 0x539e3cf2, device location [0x1ce9b000~1000], logical extent 0x244000~1000, object #10:0a8213bc:::200.00000020:head#