Ceph : Unsafe Inconsistent PG, not safe to run pg repair , How to fix them ?

Solution Verified - Updated -

Environment

  • Red Hat Ceph Storage 1.3.3
  • Red Hat Ceph Storage 2.x
  • Red Hat Ceph Storage 3.x
  • Red Hat Ceph Storage 4.x
  • Red Hat Ceph Storage 5.x
  • Red Hat Ceph Storage 6.x

Issue

  • Unsafe inconsistent PG
  • We have another inconsistent PG. It is of the same type as the last one which we filed a case for, with one twist: the primary appears to be the shard with the problem. Here are the errors from a deep scrub:
2015-05-20 19:33:45.740993 7fc237ddc700  0 log [ERR] : 5.e010 shard 46: soid 45a4e010/default.118038857.24732__shadow_.kelxKI7tQ64YCK2RNrKZp7NYh_VurPu_1/head//5 digest 733226906 != known digest 608489496
2015-05-20 19:33:45.740998 7fc237ddc700  0 log [ERR] : 5.e010 shard 135: soid 45a4e010/default.118038857.24732__shadow_.kelxKI7tQ64YCK2RNrKZp7NYh_VurPu_1/head//5 digest 733226906 != known digest 608489496
2015-05-20 19:36:29.672930 7fc237ddc700  0 log [ERR] : 5.e010 deep-scrub 0 missing, 1 inconsistent objects
2015-05-20 19:36:29.672940 7fc237ddc700  0 log [ERR] : 5.e010 deep-scrub 2 errors

  • The non-primary replicas agree, while the tools report that there are 2 errors because the primary disagrees with them.

  • I assume that the proper thing to do here follows the procedure for a digest mismatch, but I am filing this ticket out of an abundance of caution. Please just let me know if I should manually repair the object as before, or if this is a special case because the primary is the replica with the mismatch.

  • PG 5.a096 (osd.1279 on cephnode1) came up inconsistent in an automated scrub. I ran a deep scrub and the error reported is:

2015-05-13 16:40:54.849618 7ff8e1020700  0 log [ERR] : 5.a096 shard 327: soid 6a3aa096/default.86750274.2235__multipart_1. test/8. AUGUST 2014/test1 + test2 VID RAR/3. test4 Day/CUTAWAY/CARD 1.part303.rar.5RJ1X66LDt6rI76J2iB7irFZ_73q3d1.4/head//5 candidate had a read error, digest 1144609603 != known digest 404770159
2015-05-13 16:42:41.842931 7ff8e1020700  0 log [ERR] : 5.a096 deep-scrub 0 missing, 1 inconsistent objects
2015-05-13 16:42:41.842937 7ff8e1020700  0 log [ERR] : 5.a096 deep-scrub 1 errors
  • According to docs, this is not safe to use a 'ceph pg repair' on.

Resolution

  • What you will need to do in this case it to remove the object from the osd manually. To do that you can find the object by checking the pg directory for osd.[number].

  • You should be able to find the object with something similar to this, the object names are slightly encoded so it is sometimes easier to just looks for a part of the filename that doesn't have any spaces special characters.

/var/lib/ceph/osd/<osd-number>/current/<pg_number>_head# find ./ | grep [object_name]

  • From here you should be able to find the object. Before deleting the object, shut the osd down, back the object up, then remove the object.

  • After starting the osd back up, upon rescrubbing, the placement group should have fixed itself, or it may now need a manual repair.

  • If the primary is incorrect, the procedure will be the same as the above but will just need to be applied to the primary osd instead.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments