RHEL7: I/O stalled from critical I/O task being throttled by throttle_vm_writeout
Issue
- Hang occurred with following log:
[966792.129150] INFO: task kswapd1:308 blocked for more than 120 seconds.
[966792.129208] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[966792.129270] kswapd1 D ffff88613fb66eb0 0 308 2 0x00000080
[966792.129278] Call Trace:
[966792.129288] [<ffffffff97312f49>] schedule+0x29/0x70
[966792.129313] [<ffffffffc084778e>] raid1_write_request+0x7ae/0xa10 [raid1]
[966792.129321] [<ffffffff96df835b>] ? __kmalloc+0x1eb/0x230
[966792.129329] [<ffffffffc0846fd4>] ? r1bio_pool_alloc+0x24/0x30 [raid1]
[966792.129341] [<ffffffff96d96afe>] ? mempool_alloc+0x6e/0x170
[966792.129346] [<ffffffff96cbbe20>] ? wake_up_atomic_t+0x30/0x30
[966792.129355] [<ffffffffc0847a82>] raid1_make_request+0x92/0x500 [raid1]
[966792.129361] [<ffffffff96d969a5>] ? mempool_alloc_slab+0x15/0x20
[966792.129372] [<ffffffff97147ba0>] md_handle_request+0xd0/0x150
[966792.129396] [<ffffffffc049eba8>] raid_map+0x28/0x40 [dm_raid]
[966792.129419] [<ffffffffc03f1126>] __map_bio+0x96/0x190 [dm_mod]
[966792.129432] [<ffffffffc03ef680>] ? queue_io+0x80/0x80 [dm_mod]
[966792.129447] [<ffffffffc03f1427>] __clone_and_map_data_bio+0x177/0x280 [dm_mod]
[966792.129461] [<ffffffffc03f1801>] __split_and_process_bio+0x2d1/0x520 [dm_mod]
[966792.129475] [<ffffffffc03f1d6d>] dm_make_request+0x11d/0x1a0 [dm_mod]
[966792.129500] [<ffffffff96f1a6bb>] generic_make_request+0x10b/0x320
[966792.129504] [<ffffffff96e57395>] ? bio_alloc_bioset+0x115/0x310
[966792.129506] [<ffffffff96f1a940>] submit_bio+0x70/0x150
[966792.129511] [<ffffffff96dd9860>] ? init_emergency_isa_pool+0x60/0x60
[966792.129513] [<ffffffff96dd9e09>] __swap_writepage+0x239/0x280
[966792.129517] [<ffffffff96dd229e>] ? page_remove_rmap+0xce/0x160
[966792.129520] [<ffffffff96dd25bf>] ? try_to_unmap_one+0x28f/0x7c0
[966792.129522] [<ffffffff96ddab32>] ? _swap_info_get+0x12/0x60
[966792.129525] [<ffffffff96ddc8b0>] ? page_swapcount+0x50/0xa0
[966792.129527] [<ffffffff96dd9e89>] swap_writepage+0x39/0x80
[966792.129531] [<ffffffff96da814f>] shrink_page_list+0x4df/0xaf0
[966792.129534] [<ffffffff96da8d8d>] shrink_inactive_list+0x1cd/0x5d0
[966792.129536] [<ffffffff96da9885>] shrink_lruvec+0x385/0x730
[966792.129539] [<ffffffff96da9ca6>] shrink_zone+0x76/0x1a0
[966792.129541] [<ffffffff96daaf9c>] balance_pgdat+0x48c/0x5e0
[966792.129544] [<ffffffff96dab263>] kswapd+0x173/0x440
[966792.129545] [<ffffffff96cbbe20>] ? wake_up_atomic_t+0x30/0x30
[966792.129547] [<ffffffff96dab0f0>] ? balance_pgdat+0x5e0/0x5e0
[966792.129551] [<ffffffff96cbae31>] kthread+0xd1/0xe0
[966792.129553] [<ffffffff96cbad60>] ? insert_kthread_work+0x40/0x40
[966792.129556] [<ffffffff9731f637>] ret_from_fork_nospec_begin+0x21/0x21
[966792.129558] [<ffffffff96cbad60>] ? insert_kthread_work+0x40/0x40
Environment
- Red Hat Enterprise Linux 7
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.