The kernel crashes due to a blocked task detection. The blocked task is waiting on an rw_semaphore that is held by a task looping over the loop in blk_mq_get_tag()

Issue

• The kernel crashes due to a blocked task detection. The blocked task is waiting on an rw_semaphore that is held by a task looping over the loop in blk_mq_get_tag().

[256921.913067] INFO: task stathist sample:2533 blocked for more than 120 seconds.
[256921.914292] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[256921.915587] stathist sample D ffff88ea4d11acc0     0  2533   2322 0x00000000
[256921.915591] Call Trace:
[256921.915599]  [<ffffffff8a44c0b8>] ? do_sys_open+0x198/0x220
[256921.915602]  [<ffffffff8a985da9>] schedule+0x29/0x70
[256921.915605]  [<ffffffff8a9874e5>] rwsem_down_write_failed+0x215/0x3c0
[256921.915608]  [<ffffffff8a5966f7>] call_rwsem_down_write_failed+0x17/0x30
[256921.915610]  [<ffffffff8a9850fd>] down_write+0x2d/0x3d
[256921.915614]  [<ffffffff8a3de5d0>] vm_mmap_pgoff+0xa0/0x120
[256921.915618]  [<ffffffff8a3fa9b6>] SyS_mmap_pgoff+0x116/0x290
[256921.915622]  [<ffffffff8a231f12>] SyS_mmap+0x22/0x30
[256921.915626]  [<ffffffff8a992ed2>] system_call_fastpath+0x25/0x2a
[256921.915628] sending NMI to all CPUs:
[256921.916690] NMI backtrace for cpu 0 skipped: idling at pc 0xffffffff8a987e8b
[256921.916691] NMI backtrace for cpu 1 skipped: idling at pc 0xffffffff8a987e8b
[256921.916692] NMI backtrace for cpu 2 skipped: idling at pc 0xffffffff8a987e8b
[256921.916693] NMI backtrace for cpu 3 skipped: idling at pc 0xffffffff8a987e8b
[256921.916694] NMI backtrace for cpu 4 skipped: idling at pc 0xffffffff8a987e8b
[256921.916695] NMI backtrace for cpu 5 skipped: idling at pc 0xffffffff8a987e8b
[256921.916696] NMI backtrace for cpu 6
[256921.916698] CPU: 6 PID: 50 Comm: khungtaskd Kdump: loaded Tainted: P           OE  ------------   3.10.0-1127.8.2.el7.x86_64 #1
[256921.916699] Hardware name: Amazon EC2 r5.2xlarge/, BIOS 1.0 10/16/2017
[256921.916700] task: ffff88ded0b19070 ti: ffff88ded0bd8000 task.ti: ffff88ded0bd8000
[256921.916701] RIP: 0010:[<ffffffff8a26d5fa>]  [<ffffffff8a26d5fa>] native_write_msr_safe+0xa/0x10
[256921.916702] RSP: 0018:ffff88ded0bdbdb8  EFLAGS: 00000046
[256921.916703] RAX: 0000000000000400 RBX: 0000000000000006 RCX: 0000000000000830
[256921.916704] RDX: 0000000000000005 RSI: 0000000000000400 RDI: 0000000000000830
[256921.916705] RBP: ffff88ded0bdbdb8 R08: ffffffff8af5b020 R09: 0000000000000000
[256921.916706] R10: 0000000000000270 R11: ffff88ded0bdbb26 R12: ffffffff8af5b020
[256921.916707] R13: 0000000000000006 R14: 000000000000e02e R15: 0000000000000002
[256921.916708] FS:  0000000000000000(0000) GS:ffff88ea4d180000(0000) knlGS:0000000000000000
[256921.916709] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[256921.916710] CR2: 00007f588f46d5b0 CR3: 0000000d40082000 CR4: 00000000007606e0
[256921.916711] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[256921.916712] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[256921.916712] PKRU: 00000000
[256921.916713] Call Trace:
[256921.916714]  [<ffffffff8a2639e2>] __x2apic_send_IPI_mask+0xb2/0xe0
[256921.916715]  [<ffffffff8a263a83>] x2apic_send_IPI_mask+0x13/0x20
[256921.916716]  [<ffffffff8a25ee03>] arch_trigger_all_cpu_backtrace+0x2c3/0x2d0
[256921.916717]  [<ffffffff8a34ed40>] watchdog+0x260/0x2c0
[256921.916718]  [<ffffffff8a34eae0>] ? reset_hung_task_detector+0x20/0x20
[256921.916719]  [<ffffffff8a2c6691>] kthread+0xd1/0xe0
[256921.916720]  [<ffffffff8a2c65c0>] ? insert_kthread_work+0x40/0x40
[256921.916721]  [<ffffffff8a992d37>] ret_from_fork_nospec_begin+0x21/0x21
[256921.916722]  [<ffffffff8a2c65c0>] ? insert_kthread_work+0x40/0x40
[256921.916723] Code: 00 55 89 f9 48 89 e5 0f 32 31 c9 89 c0 48 c1 e2 20 89 0e 48 09 c2 48 89 d0 5d c3 66 0f 1f 44 00 00 55 89 f0 89 f9 48 89 e5 0f 30 <31> c0 5d c3 66 90 55 89 f9 48 89 e5 0f 33 89 c0 48 c1 e2 20 48 
[256921.916725] NMI backtrace for cpu 7 skipped: idling at pc 0xffffffff8a987e8b
[256921.916726] Kernel panic - not syncing: hung_task: blocked tasks
[256921.917708] CPU: 6 PID: 50 Comm: khungtaskd Kdump: loaded Tainted: P           OE  ------------   3.10.0-1127.8.2.el7.x86_64 #1
[256921.919565] Hardware name: Amazon EC2 r5.2xlarge/, BIOS 1.0 10/16/2017
[256921.920645] Call Trace:
[256921.921077]  [<ffffffff8a97ffa5>] dump_stack+0x19/0x1b
[256921.921943]  [<ffffffff8a979541>] panic+0xe8/0x21f
[256921.922739]  [<ffffffff8a34ed4e>] watchdog+0x26e/0x2c0
[256921.923585]  [<ffffffff8a34eae0>] ? reset_hung_task_detector+0x20/0x20
[256921.924666]  [<ffffffff8a2c6691>] kthread+0xd1/0xe0
[256921.925472]  [<ffffffff8a2c65c0>] ? insert_kthread_work+0x40/0x40
[256921.926485]  [<ffffffff8a992d37>] ret_from_fork_nospec_begin+0x21/0x21
[256921.927565]  [<ffffffff8a2c65c0>] ? insert_kthread_work+0x40/0x40

PID: 2535   TASK: ffff88e9f7141070  CPU: 4   COMMAND: "VM Periodic Tas"
 #0 [ffff88e9f7006fe8] __schedule at ffffffff8a9858fa
 #1 [ffff88e9f7007078] schedule at ffffffff8a985da9
 #2 [ffff88e9f7007088] schedule_timeout at ffffffff8a9838b1
 #3 [ffff88e9f7007138] io_schedule_timeout at ffffffff8a98549d
 #4 [ffff88e9f7007168] io_schedule at ffffffff8a985538
 #5 [ffff88e9f7007178] blk_mq_get_tag at ffffffff8a5632e1
 #6 [ffff88e9f70071e8] __blk_mq_alloc_request at ffffffff8a55e69b
 #7 [ffff88e9f7007218] blk_mq_sched_get_request at ffffffff8a5656fb
 #8 [ffff88e9f7007258] blk_mq_make_request at ffffffff8a5605d9
 #9 [ffff88e9f70072e8] generic_make_request at ffffffff8a5543d7
#10 [ffff88e9f7007340] submit_bio at ffffffff8a554680
#11 [ffff88e9f7007398] __swap_writepage at ffffffff8a408f29
#12 [ffff88e9f7007488] swap_writepage at ffffffff8a408fa9
#13 [ffff88e9f70074a8] shrink_page_list at ffffffff8a3d2f12
#14 [ffff88e9f70075e0] shrink_inactive_list at ffffffff8a3d38d6
#15 [ffff88e9f70076a8] shrink_lruvec at ffffffff8a3d43b5
#16 [ffff88e9f70077a8] shrink_zone at ffffffff8a3d47e6
#17 [ffff88e9f7007800] do_try_to_free_pages at ffffffff8a3d4cd0
#18 [ffff88e9f7007878] try_to_free_pages at ffffffff8a3d51fc
#19 [ffff88e9f7007910] __alloc_pages_slowpath at ffffffff8a97b25c
#20 [ffff88e9f7007a00] __alloc_pages_nodemask at ffffffff8a3c91a6
#21 [ffff88e9f7007aa8] alloc_pages_current at ffffffff8a418e78
#22 [ffff88e9f7007af0] __page_cache_alloc at ffffffff8a3be3d7
#23 [ffff88e9f7007b28] __do_page_cache_readahead at ffffffff8a3cb5ad
#24 [ffff88e9f7007be8] ra_submit at ffffffff8a3cbce1
#25 [ffff88e9f7007bf8] filemap_fault at ffffffff8a3c0e25
#26 [ffff88e9f7007c68] __xfs_filemap_fault at ffffffffc0438a4e [xfs]
#27 [ffff88e9f7007cc8] xfs_filemap_fault at ffffffffc0438c4c [xfs]
#28 [ffff88e9f7007cd8] __do_fault at ffffffff8a3edf4a
#29 [ffff88e9f7007d58] do_shared_fault at ffffffff8a3ee6ac
#30 [ffff88e9f7007db8] handle_mm_fault at ffffffff8a3f5793
#31 [ffff88e9f7007e80] __do_page_fault at ffffffff8a98d653
 ...