Ceph - OSD reboots continusously

Solution Unverified - Updated -

Issue

one osd alwayse repeat reboot,when I kill the process and mark it out.

another osd repeat the same as the previous osd.

some simple logs are:

    -5> 2015-08-23 08:41:58.413487 7fbc1bab5700  2 osd.123 pg_epoch: 43766 pg[4.241( v 43673'1447948 (28547'1444939,43673'1447948] local-les=43765 n=709 ec=3 les/c 43765/43765 43764/43764/437
64) [123,64,109] r=0 lpr=43764 crt=43673'1447948 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub   osd.123 has 22 items
    -4> 2015-08-23 08:41:58.413538 7fbc1bab5700  2 osd.123 pg_epoch: 43766 pg[4.241( v 43673'1447948 (28547'1444939,43673'1447948] local-les=43765 n=709 ec=3 les/c 43765/43765 43764/43764/437
64) [123,64,109] r=0 lpr=43764 crt=43673'1447948 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub replica 64 has 22 items
    -3> 2015-08-23 08:41:58.413564 7fbc1bab5700  2 osd.123 pg_epoch: 43766 pg[4.241( v 43673'1447948 (28547'1444939,43673'1447948] local-les=43765 n=709 ec=3 les/c 43765/43765 43764/43764/437
64) [123,64,109] r=0 lpr=43764 crt=43673'1447948 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub replica 109 has 22 items
    -2> 2015-08-23 08:41:58.414046 7fbc1bab5700  2 osd.123 pg_epoch: 43766 pg[4.241( v 43673'1447948 (28547'1444939,43673'1447948] local-les=43765 n=709 ec=3 les/c 43765/43765 43764/43764/437
64) [123,64,109] r=0 lpr=43764 crt=43673'1447948 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] 
    -1> 2015-08-23 08:41:58.414190 7fbc1bab5700  0 log [ERR] : deep-scrub 4.241 ac7b5241/rbd_data.5193e2ae8944a.0000000000009480/49//4 on disk size (0) does not match object info size (315392
0) ajusted for ondisk to (3153920)
     0> 2015-08-23 08:41:58.417590 7fbc1bab5700 -1 osd/osd_types.cc: In function 'uint64_t SnapSet::get_clone_bytes(snapid_t) const' thread 7fbc1bab5700 time 2015-08-23 08:41:58.414217
osd/osd_types.cc: 3537: FAILED assert(clone_size.count(clone))

 ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)
 1: (SnapSet::get_clone_bytes(snapid_t) const+0x15f) [0x715b8f]
 2: (ReplicatedPG::_scrub(ScrubMap&)+0x108f) [0x7db95f]
 3: (PG::scrub_compare_maps()+0x144c) [0x76a15c]
 4: (PG::chunky_scrub(ThreadPool::TPHandle&)+0x1c3) [0x76abb3]
 5: (PG::scrub(ThreadPool::TPHandle&)+0x395) [0x76e725]
 6: (OSD::ScrubWQ::_process(PG*, ThreadPool::TPHandle&)+0x13) [0x66cff3]
 7: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0xa6eab6]
 8: (ThreadPool::WorkThread::entry()+0x10) [0xa70ad0]
 9: (()+0x7e9a) [0x7fbc3bce8e9a]
10: (clone()+0x6d) [0x7fbc3abe231d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Environment

  • Inktank Ceph Enterprise 1.2

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content