Slow request from pool metrics on Ceph with Gnocchi
Issue
-
On Red Hat OpenStack Platform with backed Red Hat Ceph Storage and Gnocchi for collecting metrics, slow requests are experienced against
PGs
from pool metrics. These slow request are affecting other pools which share theOSDs
with pool metrics.OSDs
may failing with:osd/ReplicatedPG.cc: 387: FAILED assert(needs_recovery)
-
or with:
7fd479246700 1 heartbeat_map is_healthy 'OSD::osd_op_tp thread 0x7fd4517d0700' had suicide timed out after 150 7fd479246700 -1 common/HeartbeatMap.cc: In function 'bool ceph::HeartbeatMap::_check(const ceph::heartbeat_handle_d*, const char*, time_t)' thread 7fd479246700 time 2017-05-30 23:10:57.905609 common/HeartbeatMap.cc: 86: FAILED assert(0 == "hit suicide timeout")
-
Environment
- Red Hat Ceph Storage (RHCS)
- 1.3.x
- 2.x
- Red Hat OpenStack Platform (RHOSP)
- 10
- 11
- Gnocchi
- 3.0.13 and below for RHOSP 10
- 3.1.10 and below for RHOSP 11
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.