RHEL5.6+: Panic in raw_ctl_ioctl called by Oracle emagent due to memory corruption of raw_devices table
Issue
- RHEL5 servers kernel crashes, 10/300 servers crashed while running an informational gathering script provided by EMC.
- Upgrading from Red Hat Enterprise Linux 5.5 to Red Hat Enterprise Linux 5.6 causes the system to panic due to corruption of raw_devices table
-
Crash dump shows the following backtrace
Pid: 11265, comm: raw Not tainted 2.6.18-238.5.1.el5 #1 RIP: 0010:[<ffffffff801b8d5d>] [<ffffffff801b8d5d>] raw_ctl_ioctl+0x193/0x1fb RSP: 0018:ffff810ef5ed9e68 EFLAGS: 00010286 RAX: ffff81057b39d080 RBX: 000000000001e490 RCX: 0000000000000000 RDX: ffff000000000000 RSI: 00007fff12a67638 RDI: ffffffff8033f770 Process raw (pid: 11265, threadinfo ffff810ef5ed8000, task ffff8101606290c0) Stack: 00007fff00001e49 0000000000000000 0000000000000000 0000000000030002 0000000000000000 ffff810510c9d1c0 00007fff12a67620 000000000000ac01 0000000000000003 ffffffff8004241d ffff810510c9d1c0 00007fff12a67620 Call Trace: [] do_ioctl+0x55/0x6b [] vfs_ioctl+0x457/0x4b9 [] remove_vma+0x5d/0x64 [] sys_ioctl+0x59/0x78 [] system_call+0x7e/0x83 Code: 8b 02 c1 e8 14 89 c0 48 89 44 24 08 8b 02 25 ff ff 0f 00 48 RIP [] raw_ctl_ioctl+0x193/0x1fb
Environment
- Red Hat Enterprise Linux 5
- Kernels at least RHEL5.6 and above
- Seen on 5.6 2.6.18-238.5.1.el5, 2.6.18-348.1.1.el5, 2.6.18-348.4.1.el5, 2.6.18-407.el5
- iSCSI loaded, including rdma_* modules
- CCISS driver used for local storage
- Oracle
- HP firmware 8.6.3
- Problem did not occur on Red Hat Enterprise Linux 5.5 (2.6.18-194.el5)
- However, due to the small nature of the memory corruption, this could just be chance (kernel memory still corrupted, but in less fatal area)
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.
