Ceph librbd: thread_pool worker queue crash in void *item = wq->_void_dequeue()

Solution In Progress - Updated -

Issue

  • Ceph librbd: thread_pool worker queue crash in void *item = wq->_void_dequeue()
  • nova-compute randomly segfaults when librbd in use
Core was generated by `/usr/bin/python2 /usr/bin/nova-compute --config-file /etc/nova/nova.conf --conf'.
Program terminated with signal 11, Segmentation fault.
#0  ThreadPool::worker (this=0x726e5a0, wt=0x2d9a4f0) at common/WorkQueue.cc:120
120     void *item = wq->_void_dequeue();

(gdb) bt
#0  ThreadPool::worker (this=0x726e5a0, wt=0x2d9a4f0) at common/WorkQueue.cc:120
#1  0x00007f1e59b614d0 in ThreadPool::WorkThread::entry (this=) at common/WorkQueue.h:318
#2  0x00007f1ef0abedc5 in start_thread (arg=0x7f1e599fa700) at pthread_create.c:308
#3  0x00007f1ef00e31cd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
  • Some times with below given back trace:
#0  0x00007f0ca7b7620b in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/pt-raise.c:37
#1  0x00007f0cad50192d in reraise_fatal (signum=6) at global/signal_handler.cc:59
#2  handle_fatal_signal (signum=6) at global/signal_handler.cc:109
#3  <signal handler called>
#4  0x00007f0ca6634cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#5  0x00007f0ca66380d8 in __GI_abort () at abort.c:89
#6  0x00007f0ca6f3f535 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7  0x00007f0ca6f3d6d6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8  0x00007f0ca6f3d703 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#9  0x00007f0ca6f3e1bf in __cxa_pure_virtual () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#10 0x00007f0caa7acfe4 in ThreadPool::WorkQueueVal<std::pair<Context*, int>, std::pair<Context*, int> >::_void_dequeue (
    this=0x7f0cb09d6360) at ./common/WorkQueue.h:197
#11 0x00007f0caa8aae84 in ThreadPool::worker (this=0x7f0cb09cfdf0, wt=0x7f0cb09c1970) at common/WorkQueue.cc:120
#12 0x00007f0caa8ac220 in ThreadPool::WorkThread::entry (this=<optimized out>) at common/WorkQueue.h:362
#13 0x00007f0ca7b6e182 in start_thread (arg=0x7f0c9a7fc700) at pthread_create.c:312
#14 0x00007f0ca66f847d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

Environment

  • Red Hat Ceph Storage 1.3.2
ceph-common-0.94.5-14.el7cp.x86_64
librbd1-0.94.5-14.el7cp.x86_64
python-rbd-0.94.5-14.el7cp.x86_64
librados2-0.94.5-14.el7cp.x86_64
python-rados-0.94.5-14.el7cp.x86_64

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content