RHEL7: I/O stalled from critical I/O task being throttled by throttle_vm_writeout

Solution Verified - Updated -

Issue

  • Hang occurred with following log:
[966792.129150] INFO: task kswapd1:308 blocked for more than 120 seconds.
[966792.129208] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[966792.129270] kswapd1         D ffff88613fb66eb0     0   308      2 0x00000080
[966792.129278] Call Trace:
[966792.129288]  [<ffffffff97312f49>] schedule+0x29/0x70
[966792.129313]  [<ffffffffc084778e>] raid1_write_request+0x7ae/0xa10 [raid1]
[966792.129321]  [<ffffffff96df835b>] ? __kmalloc+0x1eb/0x230
[966792.129329]  [<ffffffffc0846fd4>] ? r1bio_pool_alloc+0x24/0x30 [raid1]
[966792.129341]  [<ffffffff96d96afe>] ? mempool_alloc+0x6e/0x170
[966792.129346]  [<ffffffff96cbbe20>] ? wake_up_atomic_t+0x30/0x30
[966792.129355]  [<ffffffffc0847a82>] raid1_make_request+0x92/0x500 [raid1]
[966792.129361]  [<ffffffff96d969a5>] ? mempool_alloc_slab+0x15/0x20
[966792.129372]  [<ffffffff97147ba0>] md_handle_request+0xd0/0x150
[966792.129396]  [<ffffffffc049eba8>] raid_map+0x28/0x40 [dm_raid]
[966792.129419]  [<ffffffffc03f1126>] __map_bio+0x96/0x190 [dm_mod]
[966792.129432]  [<ffffffffc03ef680>] ? queue_io+0x80/0x80 [dm_mod]
[966792.129447]  [<ffffffffc03f1427>] __clone_and_map_data_bio+0x177/0x280 [dm_mod]
[966792.129461]  [<ffffffffc03f1801>] __split_and_process_bio+0x2d1/0x520 [dm_mod]
[966792.129475]  [<ffffffffc03f1d6d>] dm_make_request+0x11d/0x1a0 [dm_mod]
[966792.129500]  [<ffffffff96f1a6bb>] generic_make_request+0x10b/0x320
[966792.129504]  [<ffffffff96e57395>] ? bio_alloc_bioset+0x115/0x310
[966792.129506]  [<ffffffff96f1a940>] submit_bio+0x70/0x150
[966792.129511]  [<ffffffff96dd9860>] ? init_emergency_isa_pool+0x60/0x60
[966792.129513]  [<ffffffff96dd9e09>] __swap_writepage+0x239/0x280
[966792.129517]  [<ffffffff96dd229e>] ? page_remove_rmap+0xce/0x160
[966792.129520]  [<ffffffff96dd25bf>] ? try_to_unmap_one+0x28f/0x7c0
[966792.129522]  [<ffffffff96ddab32>] ? _swap_info_get+0x12/0x60
[966792.129525]  [<ffffffff96ddc8b0>] ? page_swapcount+0x50/0xa0
[966792.129527]  [<ffffffff96dd9e89>] swap_writepage+0x39/0x80
[966792.129531]  [<ffffffff96da814f>] shrink_page_list+0x4df/0xaf0
[966792.129534]  [<ffffffff96da8d8d>] shrink_inactive_list+0x1cd/0x5d0
[966792.129536]  [<ffffffff96da9885>] shrink_lruvec+0x385/0x730
[966792.129539]  [<ffffffff96da9ca6>] shrink_zone+0x76/0x1a0
[966792.129541]  [<ffffffff96daaf9c>] balance_pgdat+0x48c/0x5e0
[966792.129544]  [<ffffffff96dab263>] kswapd+0x173/0x440
[966792.129545]  [<ffffffff96cbbe20>] ? wake_up_atomic_t+0x30/0x30
[966792.129547]  [<ffffffff96dab0f0>] ? balance_pgdat+0x5e0/0x5e0
[966792.129551]  [<ffffffff96cbae31>] kthread+0xd1/0xe0
[966792.129553]  [<ffffffff96cbad60>] ? insert_kthread_work+0x40/0x40
[966792.129556]  [<ffffffff9731f637>] ret_from_fork_nospec_begin+0x21/0x21
[966792.129558]  [<ffffffff96cbad60>] ? insert_kthread_work+0x40/0x40

Environment

  • Red Hat Enterprise Linux 7

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content