VM running on QEMU/KVM hangs waiting for pending I/O operations

Solution Verified - Updated -

Issue

  • A VM running on QEMU/KVM hangs waiting for I/O operations. A message like this may appear in the kernel log:
[18120.095501] INFO: task kworker/7:0:15034 blocked for more than 120 seconds.
[18120.096821] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[18120.098284] kworker/7:0     D ffff88043feaad10     0 15034      2 0x00000080
[18120.099654] Workqueue: xfs-cil/vda1 xlog_cil_push_work [xfs]
[18120.100896]  ffff880313ebb7b0 0000000000000046 ffff880420bf1fa0 ffff880313ebbfd8
[18120.102292]  ffff880313ebbfd8 ffff880313ebbfd8 ffff880420bf1fa0 ffff88043fdd6cc0
[18120.103685]  0000000000000000 7fffffffffffffff ffff88043fddec80 ffff88043feaad10
[18120.105087] Call Trace:
[18120.106122]  [<ffffffff816a94c9>] schedule+0x29/0x70
[18120.107311]  [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[18120.108534]  [<ffffffff81062efe>] ? kvm_clock_get_cycles+0x1e/0x20
[18120.109837]  [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[18120.111106]  [<ffffffff816a8be8>] io_schedule+0x18/0x20
[18120.112289]  [<ffffffff81305fe5>] bt_get+0x135/0x1c0
[18120.113479]  [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
[18120.114762]  [<ffffffff81306525>] blk_mq_get_tag+0x45/0xe0
[18120.115975]  [<ffffffff8130158b>] __blk_mq_alloc_request+0x1b/0x200
[18120.117188]  [<ffffffff813030ec>] blk_mq_map_request+0x18c/0x1e0
[18120.118448]  [<ffffffff81303370>] blk_sq_make_request+0x80/0x3d0
[18120.119610]  [<ffffffff812f706f>] ? generic_make_request_checks+0x24f/0x380
[18120.120829]  [<ffffffff812f8b45>] generic_make_request+0x105/0x310
[18120.122053]  [<ffffffff8123b439>] ? bvec_alloc+0x59/0x120
[18120.123233]  [<ffffffff812f8dc0>] submit_bio+0x70/0x150
[18120.124339]  [<ffffffff8123b713>] ? bio_alloc_bioset+0x213/0x310
[18120.125552]  [<ffffffffc0206313>] _xfs_buf_ioapply+0x2f3/0x460 [xfs]
[18120.126748]  [<ffffffffc0227d5b>] ? xlog_bdstrat+0x2b/0x60 [xfs]
[18120.127984]  [<ffffffffc0207b70>] xfs_buf_submit+0x70/0x1f0 [xfs]
[18120.129235]  [<ffffffffc0227d5b>] xlog_bdstrat+0x2b/0x60 [xfs]
[18120.130456]  [<ffffffffc0229c7e>] xlog_sync+0x2fe/0x420 [xfs]
[18120.131643]  [<ffffffffc0229e1b>] xlog_state_release_iclog+0x7b/0xd0 [xfs]
[18120.132911]  [<ffffffffc022a8fa>] xlog_write+0x5da/0x720 [xfs]
[18120.134081]  [<ffffffffc022c1d8>] xlog_cil_push+0x2a8/0x430 [xfs]
[18120.135349]  [<ffffffffc022c375>] xlog_cil_push_work+0x15/0x20 [xfs]
[18120.136586]  [<ffffffff810a881a>] process_one_work+0x17a/0x440
[18120.137792]  [<ffffffff810a9638>] worker_thread+0x278/0x3c0
[18120.138988]  [<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[18120.140228]  [<ffffffff810b098f>] kthread+0xcf/0xe0
[18120.141307]  [<ffffffff8108ddeb>] ? do_exit+0x6bb/0xa40
[18120.142421]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[18120.143559]  [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[18120.144652]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40>

Environment

  • Host based on Red Hat Enterprise Linux 7.4
  • qemu-kvm-rhev < 2.9.0-16.el7_4.12
  • qemu-kvm-rhev < 2.10.0-7.el7
  • qemu-kvm-rhev from 2.6.x and previous series are not affected by this bug
  • qemu-kvm from 1.5.x series are not affected either

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In