Ceph - OSD reboots continusously
Issue
one osd alwayse repeat reboot,when I kill the process and mark it out.
another osd repeat the same as the previous osd.
some simple logs are:
-5> 2015-08-23 08:41:58.413487 7fbc1bab5700 2 osd.123 pg_epoch: 43766 pg[4.241( v 43673'1447948 (28547'1444939,43673'1447948] local-les=43765 n=709 ec=3 les/c 43765/43765 43764/43764/437
64) [123,64,109] r=0 lpr=43764 crt=43673'1447948 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub osd.123 has 22 items
-4> 2015-08-23 08:41:58.413538 7fbc1bab5700 2 osd.123 pg_epoch: 43766 pg[4.241( v 43673'1447948 (28547'1444939,43673'1447948] local-les=43765 n=709 ec=3 les/c 43765/43765 43764/43764/437
64) [123,64,109] r=0 lpr=43764 crt=43673'1447948 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub replica 64 has 22 items
-3> 2015-08-23 08:41:58.413564 7fbc1bab5700 2 osd.123 pg_epoch: 43766 pg[4.241( v 43673'1447948 (28547'1444939,43673'1447948] local-les=43765 n=709 ec=3 les/c 43765/43765 43764/43764/437
64) [123,64,109] r=0 lpr=43764 crt=43673'1447948 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub replica 109 has 22 items
-2> 2015-08-23 08:41:58.414046 7fbc1bab5700 2 osd.123 pg_epoch: 43766 pg[4.241( v 43673'1447948 (28547'1444939,43673'1447948] local-les=43765 n=709 ec=3 les/c 43765/43765 43764/43764/437
64) [123,64,109] r=0 lpr=43764 crt=43673'1447948 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep]
-1> 2015-08-23 08:41:58.414190 7fbc1bab5700 0 log [ERR] : deep-scrub 4.241 ac7b5241/rbd_data.5193e2ae8944a.0000000000009480/49//4 on disk size (0) does not match object info size (315392
0) ajusted for ondisk to (3153920)
0> 2015-08-23 08:41:58.417590 7fbc1bab5700 -1 osd/osd_types.cc: In function 'uint64_t SnapSet::get_clone_bytes(snapid_t) const' thread 7fbc1bab5700 time 2015-08-23 08:41:58.414217
osd/osd_types.cc: 3537: FAILED assert(clone_size.count(clone))
ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)
1: (SnapSet::get_clone_bytes(snapid_t) const+0x15f) [0x715b8f]
2: (ReplicatedPG::_scrub(ScrubMap&)+0x108f) [0x7db95f]
3: (PG::scrub_compare_maps()+0x144c) [0x76a15c]
4: (PG::chunky_scrub(ThreadPool::TPHandle&)+0x1c3) [0x76abb3]
5: (PG::scrub(ThreadPool::TPHandle&)+0x395) [0x76e725]
6: (OSD::ScrubWQ::_process(PG*, ThreadPool::TPHandle&)+0x13) [0x66cff3]
7: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0xa6eab6]
8: (ThreadPool::WorkThread::entry()+0x10) [0xa70ad0]
9: (()+0x7e9a) [0x7fbc3bce8e9a]
10: (clone()+0x6d) [0x7fbc3abe231d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Environment
- Inktank Ceph Enterprise 1.2
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.