VM running on QEMU/KVM hangs waiting for pending I/O operations

Solution Verified - Updated -

Issue

  • A VM running on QEMU/KVM hangs waiting for I/O operations. A message like this may appear in the kernel log:
[18120.095501] INFO: task kworker/7:0:15034 blocked for more than 120 seconds.
[18120.096821] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[18120.098284] kworker/7:0     D ffff88043feaad10     0 15034      2 0x00000080
[18120.099654] Workqueue: xfs-cil/vda1 xlog_cil_push_work [xfs]
[18120.100896]  ffff880313ebb7b0 0000000000000046 ffff880420bf1fa0 ffff880313ebbfd8
[18120.102292]  ffff880313ebbfd8 ffff880313ebbfd8 ffff880420bf1fa0 ffff88043fdd6cc0
[18120.103685]  0000000000000000 7fffffffffffffff ffff88043fddec80 ffff88043feaad10
[18120.105087] Call Trace:
[18120.106122]  [<ffffffff816a94c9>] schedule+0x29/0x70
[18120.107311]  [<ffffffff816a6fd9>] schedule_timeout+0x239/0x2c0
[18120.108534]  [<ffffffff81062efe>] ? kvm_clock_get_cycles+0x1e/0x20
[18120.109837]  [<ffffffff816a8b4d>] io_schedule_timeout+0xad/0x130
[18120.111106]  [<ffffffff816a8be8>] io_schedule+0x18/0x20
[18120.112289]  [<ffffffff81305fe5>] bt_get+0x135/0x1c0
[18120.113479]  [<ffffffff810b1910>] ? wake_up_atomic_t+0x30/0x30
[18120.114762]  [<ffffffff81306525>] blk_mq_get_tag+0x45/0xe0
[18120.115975]  [<ffffffff8130158b>] __blk_mq_alloc_request+0x1b/0x200
[18120.117188]  [<ffffffff813030ec>] blk_mq_map_request+0x18c/0x1e0
[18120.118448]  [<ffffffff81303370>] blk_sq_make_request+0x80/0x3d0
[18120.119610]  [<ffffffff812f706f>] ? generic_make_request_checks+0x24f/0x380
[18120.120829]  [<ffffffff812f8b45>] generic_make_request+0x105/0x310
[18120.122053]  [<ffffffff8123b439>] ? bvec_alloc+0x59/0x120
[18120.123233]  [<ffffffff812f8dc0>] submit_bio+0x70/0x150
[18120.124339]  [<ffffffff8123b713>] ? bio_alloc_bioset+0x213/0x310
[18120.125552]  [<ffffffffc0206313>] _xfs_buf_ioapply+0x2f3/0x460 [xfs]
[18120.126748]  [<ffffffffc0227d5b>] ? xlog_bdstrat+0x2b/0x60 [xfs]
[18120.127984]  [<ffffffffc0207b70>] xfs_buf_submit+0x70/0x1f0 [xfs]
[18120.129235]  [<ffffffffc0227d5b>] xlog_bdstrat+0x2b/0x60 [xfs]
[18120.130456]  [<ffffffffc0229c7e>] xlog_sync+0x2fe/0x420 [xfs]
[18120.131643]  [<ffffffffc0229e1b>] xlog_state_release_iclog+0x7b/0xd0 [xfs]
[18120.132911]  [<ffffffffc022a8fa>] xlog_write+0x5da/0x720 [xfs]
[18120.134081]  [<ffffffffc022c1d8>] xlog_cil_push+0x2a8/0x430 [xfs]
[18120.135349]  [<ffffffffc022c375>] xlog_cil_push_work+0x15/0x20 [xfs]
[18120.136586]  [<ffffffff810a881a>] process_one_work+0x17a/0x440
[18120.137792]  [<ffffffff810a9638>] worker_thread+0x278/0x3c0
[18120.138988]  [<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[18120.140228]  [<ffffffff810b098f>] kthread+0xcf/0xe0
[18120.141307]  [<ffffffff8108ddeb>] ? do_exit+0x6bb/0xa40
[18120.142421]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[18120.143559]  [<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[18120.144652]  [<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40>

Environment

  • Host based on Red Hat Enterprise Linux 7.4
  • qemu-kvm-rhev < 2.9.0-16.el7_4.12
  • qemu-kvm-rhev < 2.10.0-7.el7
  • qemu-kvm-rhev from 2.6.x and previous series are not affected by this bug
  • qemu-kvm from 1.5.x series are not affected either

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content