rhel7: system hangs, dump backtrace shows device mapper threads
Issue
We frequently experience hangs:
The following task is waiting for I/O completion of dm-236
for about 1,600 sec.
PID: 43412 TASK: ffff8807a60dc500 CPU: 0 COMMAND: "jbd2/dm-236-8"
#0 [ffff88000f29fa70] __schedule at ffffffff8163a32d
#1 [ffff88000f29fad8] schedule at ffffffff8163a9c9
#2 [ffff88000f29fae8] schedule_timeout at ffffffff816386b9
#3 [ffff88000f29fb98] io_schedule_timeout at ffffffff81639ffe
#4 [ffff88000f29fbc8] io_schedule at ffffffff8163a098
#5 [ffff88000f29fbd8] sleep_on_buffer at ffffffff812112de
#6 [ffff88000f29fbe8] __wait_on_bit at ffffffff81638840
#7 [ffff88000f29fc28] out_of_line_wait_on_bit at ffffffff816388f7
#8 [ffff88000f29fc98] __wait_on_buffer at ffffffff8121289a
#9 [ffff88000f29fca8] jbd2_journal_commit_transaction at ffffffffa01b562b [jbd2]
#10 [ffff88000f29fe48] kjournald2 at ffffffffa01b9d79 [jbd2]
#11 [ffff88000f29fec8] kthread at ffffffff810a5acf
#12 [ffff88000f29ff50] ret_from_fork at ffffffff81645998
dm-236 is a dm thin device of dm thin pool device, dm-1.
The I/O of dm-1 is handled by PID: 129724 "kworker/u256:5".
PID: 129724 locks dm_bufio_client->lock via dm_bufio_lock()
and issues the I/O to /dev/loop1.
PID: 129724 TASK: ffff88045fb68000 CPU: 0 COMMAND: "kworker/u256:5"
#0 [ffff88000b5979f0] __schedule at ffffffff8163a32d
#1 [ffff88000b597a58] schedule at ffffffff8163a9c9
#2 [ffff88000b597a68] schedule_timeout at ffffffff816386b9
#3 [ffff88000b597b18] io_schedule_timeout at ffffffff81639ffe
#4 [ffff88000b597b48] io_schedule at ffffffff8163a098
#5 [ffff88000b597b58] do_io_schedule at ffffffffa04d30ee [dm_bufio]
#6 [ffff88000b597b68] __wait_on_bit at ffffffff81638840
#7 [ffff88000b597ba8] out_of_line_wait_on_bit at ffffffff816388f7
#8 [ffff88000b597c20] dm_bufio_write_dirty_buffers at ffffffffa04d50f1 [dm_bufio]
#9 [ffff88000b597c80] dm_bm_flush at ffffffffa04e631c [dm_persistent_data]
#10 [ffff88000b597c90] dm_tm_pre_commit at ffffffffa04e9d8f [dm_persistent_data]
#11 [ffff88000b597ca8] __commit_transaction at ffffffffa05046a7 [dm_thin_pool]
#12 [ffff88000b597d20] dm_pool_commit_metadata at ffffffffa0505e77 [dm_thin_pool]
#13 [ffff88000b597d48] commit at ffffffffa04fed26 [dm_thin_pool]
#14 [ffff88000b597d78] do_worker at ffffffffa05031b2 [dm_thin_pool]
#15 [ffff88000b597e20] process_one_work at ffffffff8109d5db
#16 [ffff88000b597e68] worker_thread at ffffffff8109e3ab
#17 [ffff88000b597ec8] kthread at ffffffff810a5acf
#18 [ffff88000b597f50] ret_from_fork at ffffffff81645998
Environment
- Red Hat Enterprise Linux (RHEL) 7
- hang
- device mapper
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
