Ceph - Kernel libceph module has a deadlock

Issue

• Ceph - Ceph - Kernel libceph module has a deadlock
• Ceph - Kernel RBD client has many D status processes in libceph kernel module
• libceph: stop allocating a new cipher on every crypto request

Dec 21 09:42:05 node01 kernel: INFO: task kworker/6:5:4025750 blocked for more than 120 seconds.
Dec 21 09:42:05 node01 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 21 09:42:05 node01 kernel: kworker/6:5     D ffff8808634f07c8     0 4025750      2 0x00000080
Dec 21 09:42:05 node01 kernel: Workqueue: ceph-msgr con_work [libceph]
Dec 21 09:42:05 node01 kernel: ffff88005fb6fb90 0000000000000046 ffff8800426a9700 ffff88005fb6ffd8
Dec 21 09:42:05 node01 kernel: ffff88005fb6ffd8 ffff88005fb6ffd8 ffff8800426a9700 ffff8808634f07c0
Dec 21 09:42:05 node01 kernel: ffff8808634f07c4 ffff8800426a9700 00000000ffffffff ffff8808634f07c8
Dec 21 09:42:05 node01 kernel: Call Trace:
Dec 21 09:42:05 node01 kernel: [<ffffffff8163cc19>] schedule_preempt_disabled+0x29/0x70
Dec 21 09:42:05 node01 kernel: [<ffffffff8163a915>] __mutex_lock_slowpath+0xc5/0x1c0
Dec 21 09:42:05 node01 kernel: [<ffffffff81639d7f>] mutex_lock+0x1f/0x2f
Dec 21 09:42:05 node01 kernel: [<ffffffffa060bca7>] alloc_msg+0xb7/0x2b0 [libceph]
Dec 21 09:42:05 node01 kernel: [<ffffffffa0606334>] read_partial_message+0x2a4/0x940 [libceph]
Dec 21 09:42:05 node01 kernel: [<ffffffff81511a1a>] ? kernel_recvmsg+0x3a/0x50
Dec 21 09:42:05 node01 kernel: [<ffffffff81511a1a>] ? kernel_recvmsg+0x3a/0x50
Dec 21 09:42:05 node01 kernel: [<ffffffffa0606d28>] try_read+0x358/0x10c0 [libceph]
Dec 21 09:42:05 node01 kernel: [<ffffffff810c22de>] ? dequeue_task_fair+0x42e/0x640
Dec 21 09:42:05 node01 kernel: [<ffffffff810bb7c5>] ? sched_clock_cpu+0x85/0xc0
Dec 21 09:42:05 node01 kernel: [<ffffffffa0607b49>] con_work+0xb9/0x640 [libceph]
Dec 21 09:42:05 node01 kernel: [<ffffffff8109d69b>] process_one_work+0x17b/0x470
Dec 21 09:42:05 node01 kernel: [<ffffffff8109e46b>] worker_thread+0x11b/0x400
Dec 21 09:42:05 node01 kernel: [<ffffffff8109e350>] ? rescuer_thread+0x400/0x400
Dec 21 09:42:05 node01 kernel: [<ffffffff810a5b8f>] kthread+0xcf/0xe0
Dec 21 09:42:05 node01 kernel: [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
Dec 21 09:42:05 node01 kernel: [<ffffffff81646a98>] ret_from_fork+0x58/0x90
Dec 21 09:42:05 node01 kernel: [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140