Ceph - Kernel libceph module has a deadlock

Solution Verified - Updated -

Issue

  • Ceph - Ceph - Kernel libceph module has a deadlock
  • Ceph - Kernel RBD client has many D status processes in libceph kernel module
  • libceph: stop allocating a new cipher on every crypto request
Dec 21 09:42:05 node01 kernel: INFO: task kworker/6:5:4025750 blocked for more than 120 seconds.
Dec 21 09:42:05 node01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 21 09:42:05 node01 kernel: kworker/6:5     D ffff8808634f07c8     0 4025750      2 0x00000080
Dec 21 09:42:05 node01 kernel: Workqueue: ceph-msgr con_work [libceph]
Dec 21 09:42:05 node01 kernel: ffff88005fb6fb90 0000000000000046 ffff8800426a9700 ffff88005fb6ffd8
Dec 21 09:42:05 node01 kernel: ffff88005fb6ffd8 ffff88005fb6ffd8 ffff8800426a9700 ffff8808634f07c0
Dec 21 09:42:05 node01 kernel: ffff8808634f07c4 ffff8800426a9700 00000000ffffffff ffff8808634f07c8
Dec 21 09:42:05 node01 kernel: Call Trace:
Dec 21 09:42:05 node01 kernel: [<ffffffff8163cc19>] schedule_preempt_disabled+0x29/0x70
Dec 21 09:42:05 node01 kernel: [<ffffffff8163a915>] __mutex_lock_slowpath+0xc5/0x1c0
Dec 21 09:42:05 node01 kernel: [<ffffffff81639d7f>] mutex_lock+0x1f/0x2f
Dec 21 09:42:05 node01 kernel: [<ffffffffa060bca7>] alloc_msg+0xb7/0x2b0 [libceph]
Dec 21 09:42:05 node01 kernel: [<ffffffffa0606334>] read_partial_message+0x2a4/0x940 [libceph]
Dec 21 09:42:05 node01 kernel: [<ffffffff81511a1a>] ? kernel_recvmsg+0x3a/0x50
Dec 21 09:42:05 node01 kernel: [<ffffffff81511a1a>] ? kernel_recvmsg+0x3a/0x50
Dec 21 09:42:05 node01 kernel: [<ffffffffa0606d28>] try_read+0x358/0x10c0 [libceph]
Dec 21 09:42:05 node01 kernel: [<ffffffff810c22de>] ? dequeue_task_fair+0x42e/0x640
Dec 21 09:42:05 node01 kernel: [<ffffffff810bb7c5>] ? sched_clock_cpu+0x85/0xc0
Dec 21 09:42:05 node01 kernel: [<ffffffffa0607b49>] con_work+0xb9/0x640 [libceph]
Dec 21 09:42:05 node01 kernel: [<ffffffff8109d69b>] process_one_work+0x17b/0x470
Dec 21 09:42:05 node01 kernel: [<ffffffff8109e46b>] worker_thread+0x11b/0x400
Dec 21 09:42:05 node01 kernel: [<ffffffff8109e350>] ? rescuer_thread+0x400/0x400
Dec 21 09:42:05 node01 kernel: [<ffffffff810a5b8f>] kthread+0xcf/0xe0
Dec 21 09:42:05 node01 kernel: [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
Dec 21 09:42:05 node01 kernel: [<ffffffff81646a98>] ret_from_fork+0x58/0x90
Dec 21 09:42:05 node01 kernel: [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140

Environment

  • Red Hat Enterprise Linux 7.2 - 3.10.0-327.36.3.el7.x86_64
  • Red Hat Ceph Storage 1.3.3

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.