Ceph - OSD reboots continusously
Issue
one osd alwayse repeat reboot,when I kill the process and mark it out.
another osd repeat the same as the previous osd.
some simple logs are:
-5> 2015-08-23 08:41:58.413487 7fbc1bab5700 2 osd.123 pg_epoch: 43766 pg[4.241( v 43673'1447948 (28547'1444939,43673'1447948] local-les=43765 n=709 ec=3 les/c 43765/43765 43764/43764/437
64) [123,64,109] r=0 lpr=43764 crt=43673'1447948 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub osd.123 has 22 items
-4> 2015-08-23 08:41:58.413538 7fbc1bab5700 2 osd.123 pg_epoch: 43766 pg[4.241( v 43673'1447948 (28547'1444939,43673'1447948] local-les=43765 n=709 ec=3 les/c 43765/43765 43764/43764/437
64) [123,64,109] r=0 lpr=43764 crt=43673'1447948 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub replica 64 has 22 items
-3> 2015-08-23 08:41:58.413564 7fbc1bab5700 2 osd.123 pg_epoch: 43766 pg[4.241( v 43673'1447948 (28547'1444939,43673'1447948] local-les=43765 n=709 ec=3 les/c 43765/43765 43764/43764/437
64) [123,64,109] r=0 lpr=43764 crt=43673'1447948 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep] scrub replica 109 has 22 items
-2> 2015-08-23 08:41:58.414046 7fbc1bab5700 2 osd.123 pg_epoch: 43766 pg[4.241( v 43673'1447948 (28547'1444939,43673'1447948] local-les=43765 n=709 ec=3 les/c 43765/43765 43764/43764/437
64) [123,64,109] r=0 lpr=43764 crt=43673'1447948 lcod 0'0 mlcod 0'0 active+clean+scrubbing+deep]
-1> 2015-08-23 08:41:58.414190 7fbc1bab5700 0 log [ERR] : deep-scrub 4.241 ac7b5241/rbd_data.5193e2ae8944a.0000000000009480/49//4 on disk size (0) does not match object info size (315392
0) ajusted for ondisk to (3153920)
0> 2015-08-23 08:41:58.417590 7fbc1bab5700 -1 osd/osd_types.cc: In function 'uint64_t SnapSet::get_clone_bytes(snapid_t) const' thread 7fbc1bab5700 time 2015-08-23 08:41:58.414217
osd/osd_types.cc: 3537: FAILED assert(clone_size.count(clone))
ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3)
1: (SnapSet::get_clone_bytes(snapid_t) const+0x15f) [0x715b8f]
2: (ReplicatedPG::_scrub(ScrubMap&)+0x108f) [0x7db95f]
3: (PG::scrub_compare_maps()+0x144c) [0x76a15c]
4: (PG::chunky_scrub(ThreadPool::TPHandle&)+0x1c3) [0x76abb3]
5: (PG::scrub(ThreadPool::TPHandle&)+0x395) [0x76e725]
6: (OSD::ScrubWQ::_process(PG*, ThreadPool::TPHandle&)+0x13) [0x66cff3]
7: (ThreadPool::worker(ThreadPool::WorkThread*)+0x4e6) [0xa6eab6]
8: (ThreadPool::WorkThread::entry()+0x10) [0xa70ad0]
9: (()+0x7e9a) [0x7fbc3bce8e9a]
10: (clone()+0x6d) [0x7fbc3abe231d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Environment
- Inktank Ceph Enterprise 1.2
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
