ceph::HeartbeatMap::_check() is getting assert with "OSD hit suicide timeout" followed by "FileStore: sync_entry timed out after 600 seconds" log, Why ?

Issue

• ceph::HeartbeatMap::_check() is getting assert with "OSD hit suicide timeout" followed by "FileStore: sync_entry timed out after 600 seconds" log, Why ?

Apr 26 17:38:23 ceph-osd-045 bash: common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, const char*, time_t)' thread 7f239f6d4700 time 2016-04-26 17:38:23.168776
Apr 26 17:38:23 ceph-osd-045 bash: common/HeartbeatMap.cc: 79: FAILED assert(0 == "hit suicide timeout")
Apr 26 17:38:23 ceph-osd-045 bash: ceph version 0.94.5-9.el7cp (deef183a81111fa5e128ec88c90a32c9587c615d)
Apr 26 17:38:23 ceph-osd-045 bash: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xb075d5]
Apr 26 17:38:23 ceph-osd-045 bash: 2: (ceph::HeartbeatMap::_check(ceph::heartbeat_handle_d*, char const*, long)+0x2d9) [0xa3c879]
Apr 26 17:38:23 ceph-osd-045 bash: 3: (ceph::HeartbeatMap::is_healthy()+0xde) [0xa3d16e]
Apr 26 17:38:23 ceph-osd-045 bash: 4: (ceph::HeartbeatMap::check_touch_file()+0x2c) [0xa3d88c]
Apr 26 17:38:23 ceph-osd-045 bash: 5: (CephContextServiceThread::entry()+0x15b) [0xb179eb]
Apr 26 17:38:23 ceph-osd-045 bash: 6: (()+0x7df5) [0x7f23a2ed9df5]
Apr 26 17:38:23 ceph-osd-045 bash: 7: (clone()+0x6d) [0x7f23a19bc1ad]
Apr 26 17:38:23 ceph-osd-045 bash: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

• hit suicide timeout assert followed by below given FileStore asserts :

Apr 26 17:40:47 ceph-osd-045 bash: 2016-04-26 17:40:47.464587 7f1eea2bd700 -1 FileStore: sync_entry timed out after 600 seconds.

Apr 26 17:40:47 ceph-osd-045 bash: ceph version 0.94.5-9.el7cp (deef183a81111fa5e128ec88c90a32c9587c615d)
Apr 26 17:40:47 ceph-osd-045 bash: 1: (Context::complete(int)+0x9) [0x6833c9]
Apr 26 17:40:47 ceph-osd-045 bash: 2: (SafeTimer::timer_thread()+0x104) [0xaefec4]
Apr 26 17:40:47 ceph-osd-045 bash: 3: (SafeTimerThread::entry()+0xd) [0xaf0e7d]
Apr 26 17:40:47 ceph-osd-045 bash: 4: (()+0x7df5) [0x7f1ef8bc5df5]
Apr 26 17:40:47 ceph-osd-045 bash: 5: (clone()+0x6d) [0x7f1ef76a81ad]

Apr 26 17:40:47 ceph-osd-045 bash: os/FileStore.cc: In function 'virtual void SyncEntryTimeout::finish(int)' thread 7f1eea2bd700 time 2016-04-26 17:40:47.464650
Apr 26 17:40:47 ceph-osd-045 bash: os/FileStore.cc: 3454: FAILED assert(0)

Apr 26 17:40:47 ceph-osd-045 bash: ceph version 0.94.5-9.el7cp (deef183a81111fa5e128ec88c90a32c9587c615d)
Apr 26 17:40:47 ceph-osd-045 bash: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x85) [0xb075d5]
Apr 26 17:40:47 ceph-osd-045 bash: 2: (SyncEntryTimeout::finish(int)+0xbd) [0x8e71ed]
Apr 26 17:40:47 ceph-osd-045 bash: 3: (Context::complete(int)+0x9) [0x6833c9]
Apr 26 17:40:47 ceph-osd-045 bash: 4: (SafeTimer::timer_thread()+0x104) [0xaefec4]
Apr 26 17:40:47 ceph-osd-045 bash: 5: (SafeTimerThread::entry()+0xd) [0xaf0e7d]
Apr 26 17:40:47 ceph-osd-045 bash: 6: (()+0x7df5) [0x7f1ef8bc5df5]
Apr 26 17:40:47 ceph-osd-045 bash: 7: (clone()+0x6d) [0x7f1ef76a81ad]
Apr 26 17:40:47 ceph-osd-045 bash: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.