Ceph - OSD reboots continusously

Solution Unverified - Updated -

Issue

one osd alwayse repeat reboot,when I kill the process and mark it out.

another osd repeat the same as the previous osd.

some simple logs are:

    -5> 2015-08-23 08:41:58.413487 7fbc1bab5700  2 osd.123 pg_epoch: 43766 pg[4.241( v 43673'1447948 (28547'1444939,43673'1447948] local-les=43765 n=709 ec=3 les/c 43765/43765 43764/43764/437
64) [123,64,109] r=0 lpr=43764 crt=43673'1447948 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub   osd.123 has 22 items
    -4> 2015-08-23 08:41:58.413538 7fbc1bab5700  2 osd.123 pg_epoch: 43766 pg[4.241( v 43673'1447948 (28547'1444939,43673'1447948] local-les=43765 n=709 ec=3 les/c 43765/43765 43764/43764/437
64) [123,64,109] r=0 lpr=43764 crt=43673'1447948 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub replica 64 has 22 items
    -3> 2015-08-23 08:41:58.413564 7fbc1bab5700  2 osd.123 pg_epoch: 43766 pg[4.241( v 43673'1447948 (28547'1444939,43673'1447948] local-les=43765 n=709 ec=3 les/c 43765/43765 43764/43764/437
64) [123,64,109] r=0 lpr=43764 crt=43673'1447948 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub replica 109 has 22 items
    -2> 2015-08-23 08:41:58.414046 7fbc1bab5700  2 osd.123 pg_epoch: 43766 pg[4.241( v 43673'1447948 (28547'1444939,43673'1447948] local-les=43765 n=709 ec=3 les/c 43765/43765 43764/43764/437
64) [123,64,109] r=0 lpr=43764 crt=43673'1447948 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] 
    -1> 2015-08-23 08:41:58.414190 7fbc1bab5700  0 log [ERR] : deep-scrub 4.241 ac7b5241/rbd_data.5193e2ae8944a.0000000000009480/49//4 on disk size (0) does not match object info size (315392
0) ajusted for ondisk to (3153920)
     0> 2015-08-23 08:41:58.417590 7fbc1bab5700 -1 osd/osd_types.cc: In function 'uint64_t SnapSet::get_clone_bytes(snapid_t) const' thread 7fbc1bab5700 time 2015-08-23 08:41:58.414217
osd/osd_types.cc: 3537: FAILED assert(clone_size.count(clone))

 ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)
 1: (SnapSet::get_clone_bytes(snapid_t) const+0x15f) [0x715b8f]
 2: (ReplicatedPG::_scrub(ScrubMap&)+0x108f) [0x7db95f]
 3: (PG::scrub_compare_maps()+0x144c) [0x76a15c]
 4: (PG::chunky_scrub(ThreadPool::TPHandle&)+0x1c3) [0x76abb3]
 5: (PG::scrub(ThreadPool::TPHandle&)+0x395) [0x76e725]
 6: (OSD::ScrubWQ::_process(PG*, ThreadPool::TPHandle&)+0x13) [0x66cff3]
 7: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0xa6eab6]
 8: (ThreadPool::WorkThread::entry()+0x10) [0xa70ad0]
 9: (()+0x7e9a) [0x7fbc3bce8e9a]
10: (clone()+0x6d) [0x7fbc3abe231d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Environment

  • Inktank Ceph Enterprise 1.2

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In