I/O to the devices hung after online storage addition
Issue
- Added a few devices for use by ORACLE ASM. The process that is used for online storage addition has been thoroughly vetted in many different systems. But on this particular server, ORACLE ASM was unable to query the devices anymore. A
ddwas run to test the access and it hung with the following messages in dmesg
Oct 18 21:59:24 hostname kernel: INFO: task dd:29504 blocked for more than 120 seconds.
Oct 18 21:59:24 hostname kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 18 21:59:24 hostname kernel: dd D ffffffff80157f06 0 29504 24217 (NOTLB)
Oct 18 21:59:24 hostname kernel: ffff8103be84bcb8 0000000000000082 0000000000000400 ffffffff8001c836
Oct 18 21:59:24 hostname kernel: ffff810033280940 0000000000000001 ffff81010f2217b0 ffff81061fc240c0
Oct 18 21:59:24 hostname kernel: 00088e7ded78c3ce 000000000001fd3f ffff81010f221998 0000000bfffffff4
Oct 18 21:59:24 hostname kernel: Call Trace:
Oct 18 21:59:24 hostname kernel: [<ffffffff8001c836>] generic_make_request+0x211/0x228
Oct 18 21:59:24 hostname kernel: [<ffffffff8006ed98>] do_gettimeofday+0x40/0x90
Oct 18 21:59:24 hostname kernel: [<ffffffff800637de>] io_schedule+0x3f/0x67
Oct 18 21:59:24 hostname kernel: [<ffffffff800fa644>] __blockdev_direct_IO+0x8d5/0xa88
Oct 18 21:59:24 hostname kernel: [<ffffffff800ea2a3>] blkdev_direct_IO+0x32/0x37
Oct 18 21:59:24 hostname kernel: [<ffffffff800ea1db>] blkdev_get_blocks+0x0/0x96
Oct 18 21:59:24 hostname kernel: [<ffffffff8000c665>] __generic_file_aio_read+0xb8/0x198
Oct 18 21:59:24 hostname kernel: [<ffffffff8013212b>] inode_has_perm+0x56/0x63
Oct 18 21:59:24 hostname kernel: [<ffffffff800ca9b9>] generic_file_read+0xac/0xc5
Oct 18 21:59:24 hostname kernel: [<ffffffff800a3ccf>] autoremove_wake_function+0x0/0x2e
Oct 18 21:59:24 hostname kernel: [<ffffffff80134d14>] selinux_file_permission+0x9f/0xb5
Oct 18 21:59:24 hostname kernel: [<ffffffff8000b72f>] vfs_read+0xcb/0x171
Oct 18 21:59:24 hostname kernel: [<ffffffff80011da1>] sys_read+0x45/0x6e
Oct 18 21:59:24 hostname kernel: [<ffffffff8005d29e>] tracesys+0xd5/0xdf
Environment
- Red Hat Enterprise Linux 5.9
- Kernel: 2.6.18-348.6.1.el5
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
