System I/O hang with 'seos' module loaded
Issue
-
System I/O hang with seos module loaded
-
I/O hung and processes blocked. Oldest 5 processes:
crash> ps -m | grep UN | tail -n5
[ 0 02:21:00.721] [UN] PID: 19181 TASK: ffff881539e76040 CPU: 9 COMMAND: "oracle"
[ 0 02:21:00.765] [UN] PID: 29194 TASK: ffff8805882bd540 CPU: 16 COMMAND: "ora_lgwr_sisid0"
[ 0 02:21:00.764] [UN] PID: 19175 TASK: ffff885010ea6ae0 CPU: 1 COMMAND: "oracle"
[ 0 02:21:00.863] [UN] PID: 12922 TASK: ffff881284ea8080 CPU: 5 COMMAND: "ora_j000_sisid0"
[ 0 02:21:00.870] [UN] PID: 28918 TASK: ffff8801c253aae0 CPU: 11 COMMAND: "ora_dbw2_sisid0"
- Processes hung trying to get I/O flushed to disk:
crash> bt ffff8801c253aae0
PID: 28918 TASK: ffff8801c253aae0 CPU: 11 COMMAND: "ora_dbw2_sisid0"
#0 [ffff8802941bba18] schedule at ffffffff81529970
#1 [ffff8802941bbaf0] schedule_timeout at ffffffff8152a8c5
#2 [ffff8802941bbba0] wait_for_common at ffffffff8152a543
#3 [ffff8802941bbc30] wait_for_completion at ffffffff8152a65d
#4 [ffff8802941bbc40] __blkdev_issue_flush at ffffffff8127531d
#5 [ffff8802941bbca0] blkdev_issue_flush at ffffffff81275366
#6 [ffff8802941bbcb0] ext4_sync_file at ffffffffa02a70a5 [ext4]
#7 [ffff8802941bbd00] vfs_fsync_range at ffffffff811c07f1
#8 [ffff8802941bbd50] generic_write_sync at ffffffff811c089b
#9 [ffff8802941bbd60] generic_file_aio_write at ffffffff811269e6
#10 [ffff8802941bbdb0] ext4_file_write at ffffffffa02a6e08 [ext4]
#11 [ffff8802941bbdf0] aio_rw_vect_retry at ffffffff811daf44
#12 [ffff8802941bbe20] aio_run_iocb at ffffffff811dc904
#13 [ffff8802941bbe50] do_io_submit at ffffffff811ddd31
#14 [ffff8802941bbf70] sys_io_submit at ffffffff811de3d0
#15 [ffff8802941bbf80] system_call_fastpath at ffffffff8100b072
RIP: 0000003fc8200667 RSP: 00007ffffd8e2678 RFLAGS: 00000206
RAX: 00000000000000d1 RBX: ffffffff8100b072 RCX: 0000800000010000
RDX: 00007ffffd8db2e0 RSI: 0000000000000001 RDI: 00007f415a719000
RBP: 00007ffffd8dd310 R8: 00007ffffd8ddbb8 R9: 00000002ca1b82d0
R10: 0000000000000001 R11: 0000000000000246 R12: ffffffff811de3d0
R13: ffff8802941bbf78 R14: 0000000000000001 R15: 00007ffffd8ddbb8
ORIG_RAX: 00000000000000d1 CS: 0033 SS: 002b
Environment
- Red Hat Enterprise Linux 6
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.