Slow request from pool metrics on Ceph with Gnocchi

Issue

• On Red Hat OpenStack Platform with backed Red Hat Ceph Storage and Gnocchi for collecting metrics, slow requests are experienced against PGs from pool metrics. These slow request are affecting other pools which share the OSDs with pool metrics. OSDs may failing with:

  osd/ReplicatedPG.cc: 387: FAILED assert(needs_recovery)
  

  • or with:

    7fd479246700  1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fd4517d0700' had suicide timed out after 150
    7fd479246700 -1 common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(const ceph::heartbeat_handle_d*, const char*, time_t)' thread 7fd479246700 time 2017-05-30 23:10:57.905609
    common/HeartbeatMap.cc: 86: FAILED assert(0 == "hit suicide timeout")