VM running on QEMU/KVM hangs waiting for pending I/O operations
Issue
- A VM running on QEMU/KVM hangs waiting for I/O operations. A message like this may appear in the kernel log:
[18120.095501] INFO: task kworker/7:0:15034 blocked for more than 120 seconds.
[18120.096821] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[18120.098284] kworker/7:0 D ffff88043feaad10 0 15034 2 0x00000080
[18120.099654] Workqueue: xfs-cil/vda1 xlog_cil_push_work [xfs]
[18120.100896] ffff880313ebb7b0 0000000000000046 ffff880420bf1fa0 ffff880313ebbfd8
[18120.102292] ffff880313ebbfd8 ffff880313ebbfd8 ffff880420bf1fa0 ffff88043fdd6cc0
[18120.103685] 0000000000000000 7fffffffffffffff ffff88043fddec80 ffff88043feaad10
[18120.105087] Call Trace:
[18120.106122] [<ffffffff816a94c9>] schedule+0x29/0x70
[18120.107311] [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[18120.108534] [<ffffffff81062efe>] ? kvm_clock_get_cycles+0x1e/0x20
[18120.109837] [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[18120.111106] [<ffffffff816a8be8>] io_schedule+0x18/0x20
[18120.112289] [<ffffffff81305fe5>] bt_get+0x135/0x1c0
[18120.113479] [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
[18120.114762] [<ffffffff81306525>] blk_mq_get_tag+0x45/0xe0
[18120.115975] [<ffffffff8130158b>] __blk_mq_alloc_request+0x1b/0x200
[18120.117188] [<ffffffff813030ec>] blk_mq_map_request+0x18c/0x1e0
[18120.118448] [<ffffffff81303370>] blk_sq_make_request+0x80/0x3d0
[18120.119610] [<ffffffff812f706f>] ? generic_make_request_checks+0x24f/0x380
[18120.120829] [<ffffffff812f8b45>] generic_make_request+0x105/0x310
[18120.122053] [<ffffffff8123b439>] ? bvec_alloc+0x59/0x120
[18120.123233] [<ffffffff812f8dc0>] submit_bio+0x70/0x150
[18120.124339] [<ffffffff8123b713>] ? bio_alloc_bioset+0x213/0x310
[18120.125552] [<ffffffffc0206313>] _xfs_buf_ioapply+0x2f3/0x460 [xfs]
[18120.126748] [<ffffffffc0227d5b>] ? xlog_bdstrat+0x2b/0x60 [xfs]
[18120.127984] [<ffffffffc0207b70>] xfs_buf_submit+0x70/0x1f0 [xfs]
[18120.129235] [<ffffffffc0227d5b>] xlog_bdstrat+0x2b/0x60 [xfs]
[18120.130456] [<ffffffffc0229c7e>] xlog_sync+0x2fe/0x420 [xfs]
[18120.131643] [<ffffffffc0229e1b>] xlog_state_release_iclog+0x7b/0xd0 [xfs]
[18120.132911] [<ffffffffc022a8fa>] xlog_write+0x5da/0x720 [xfs]
[18120.134081] [<ffffffffc022c1d8>] xlog_cil_push+0x2a8/0x430 [xfs]
[18120.135349] [<ffffffffc022c375>] xlog_cil_push_work+0x15/0x20 [xfs]
[18120.136586] [<ffffffff810a881a>] process_one_work+0x17a/0x440
[18120.137792] [<ffffffff810a9638>] worker_thread+0x278/0x3c0
[18120.138988] [<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[18120.140228] [<ffffffff810b098f>] kthread+0xcf/0xe0
[18120.141307] [<ffffffff8108ddeb>] ? do_exit+0x6bb/0xa40
[18120.142421] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[18120.143559] [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[18120.144652] [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40>
Environment
- Host based on Red Hat Enterprise Linux 7.4
- qemu-kvm-rhev < 2.9.0-16.el7_4.12
- qemu-kvm-rhev < 2.10.0-7.el7
- qemu-kvm-rhev from 2.6.x and previous series are not affected by this bug
- qemu-kvm from 1.5.x series are not affected either
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.