McAfee's LinuxShield caused kernel panic / hung tasks
Issue
-
Ran the 'yum update' utility , and now every night servers experience a kernel panic
KERNEL: /usr/lib/debug/lib/modules/2.6.18-274.7.1.el5/vmlinux DUMPFILE: /var/crash/2011-11-30-04:08/vmcore CPUS: 4 DATE: Wed Nov 30 04:06:46 2011 UPTIME: 23:48:10 LOAD AVERAGE: 1.41, 0.91, 0.38 TASKS: 181 NODENAME: xyz RELEASE: 2.6.18-274.7.1.el5 VERSION: #1 SMP Mon Oct 17 11:57:40 EDT 2011 MACHINE: i686 (2133 Mhz) MEMORY: 1.5 GB PANIC: "Oops: 0000 [#1]" (check log for details) PID: 281 COMMAND: "pdflush" TASK: c9d79550 [THREAD_INFO: c9d7b000] CPU: 1 STATE: TASK_RUNNING (PANIC) BUG: unable to handle kernel NULL pointer dereference at virtual address 0000001c printing eip: f8dfc0ec *pde = 5bd72067 Oops: 0000 [#1] SMP last sysfs file: /devices/pci0000:00/0000:00:00.0/irq Modules linked in: linuxshield(U) lshook(U) nfsd exportfs auth_rpcgss ipv6 xfrm_nalgo crypto_api autofs4 nfs nfs_acl lockd sunrpc ip_conntrack_netbios_ns iptable_nat ip_nat iptable_mangle ipt_REJECT xt_state ip_conntrack nfnetlink xt_tcpudp iptable_filter ip_tables x_tables vsock(U) vmmemctl(U) acpiphp loop dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi ac lp parport_pc ide_cd i2c_piix4 parport floppy tpm_tis tpm cdrom i2c_core tpm_bios serio_raw vmci(U) sg pcspkr pvscsi(U) vmxnet3(U) vmxnet(U) dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata mptspi mptscsih mptbase scsi_transport_spi sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd CPU: 1 EIP: 0060:[<f8dfc0ec>] Tainted: G ---- VLI EFLAGS: 00010246 (2.6.18-274.7.1.el5 #1) EIP is at fileOpenHook+0x64/0x265 [lshook] eax: 00000000 ebx: f6777180 ecx: c9d79550 edx: c9d7be94 esi: d57a6e70 edi: c9d7bed4 ebp: 00000000 esp: c9d7be88 ds: 007b es: 007b ss: 0068 Process pdflush (pid: 281, ti=c9d7b000 task=c9d79550 task.ti=c9d7b000) Stack: f8b5b8df f7306e60 00000000 00000000 f6777180 d57a6e70 c9d7bed4 00000000 f8b8a494 f8b9e91c f6777180 f7306e60 d57a6e70 d57a6e70 00000001 c9d7bed4 d57a6d2c f8b8adbd 00000000 c9d7bed4 c9d7bed4 00000000 d57a6e70 d57a6f18 Call Trace: [<f8b5b8df>] rpc_init_task_wq+0x35/0x11d [sunrpc] [<f8b8a494>] nfs_commit_list+0x108/0x200 [nfs] [<f8b8adbd>] nfs_commit_inode+0x5b/0x78 [nfs] [<f8b83655>] nfs_write_inode+0x41/0x5c [nfs] [<c04969de>] __writeback_single_inode+0x199/0x2a5 [<c0496dce>] sync_sb_inodes+0x17e/0x221 [<c049701d>] writeback_inodes+0x6a/0xb0 [<c045f986>] wb_kupdate+0xcb/0x130 [<c045fda7>] pdflush+0x0/0x1a1 [<c045feb2>] pdflush+0x10b/0x1a1 [<c045f8bb>] wb_kupdate+0x0/0x130 [<c0436e4e>] kthread+0xc0/0xee [<c0436d8e>] kthread+0x0/0xee [<c0405c87>] kernel_thread_helper+0x7/0x10 ======================= Code: 89 d0 ff 73 20 25 ff ff 0f 00 c1 ea 14 50 52 68 0c fa df f8 68 7e d2 df f8 e8 d9 9e 62 c7 83 c4 1c 8b 83 9c 00 00 00 8d 54 24 0c <8b> 40 1c e8 44 e1 ff ff 8b 44 24 0c 83 c0 14 e8 30 71 82 c7 8b EIP: [<f8dfc0ec>] fileOpenHook+0x64/0x265 [lshook] SS:ESP 0068:c9d7be88 -
Server is having hung tasks constantly, with the following stack trace in the logs:
Aug 6 07:07:03 localhost kernel: INFO: task mysqld:29636 blocked for more than 120 seconds.
Aug 6 07:07:03 localhost kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 6 07:07:03 localhost kernel: mysqld D 000419EB 2572 29636 3478 29638 3715 (NOTLB)
Aug 6 07:07:03 localhost kernel: e7726eac 00000082 ca2e64fc 000419eb f3da43c0 f8dc266e 00000004 00000009
Aug 6 07:07:03 localhost kernel: f14a35f0 ca347aaf 000419eb 000615b3 00000002 f14a36fc c2e20488 f2d1be40
Aug 6 07:07:03 localhost kernel: f8eceae8 00000000 00000286 c042f301 e0e9e501 c2f62000 00000286 ffffffff
Aug 6 07:07:03 localhost kernel: Call Trace:
Aug 6 07:07:03 localhost kernel: [<f8dc266e>] fileOpenHook+0x1ed/0x265 [lshook]
Aug 6 07:07:03 localhost kernel: [<c042f301>] lock_timer_base+0x15/0x2f
Aug 6 07:07:03 localhost kernel: [<f8ec5d8b>] doScan+0x840/0xc2a [linuxshield]
Aug 6 07:07:03 localhost kernel: [<c04383ff>] autoremove_wake_function+0x0/0x2d
Aug 6 07:07:03 localhost kernel: [<f8ec6e45>] SC_doScan+0xea/0xf4 [linuxshield]
Aug 6 07:07:03 localhost kernel: [<f8ec441e>] releaseHook+0xed/0x153 [linuxshield]
Aug 6 07:07:03 localhost kernel: [<f8dc1f4a>] invokeCallbacks+0x1fd/0x354 [lshook]
Aug 6 07:07:03 localhost kernel: [<f8dc2438>] fileReleaseHook+0xa0/0xe9 [lshook]
Aug 6 07:07:03 localhost kernel: [<c047a2c0>] __fput+0xb9/0x184
Aug 6 07:07:03 localhost kernel: [<c0477c37>] filp_close+0x4e/0x54
Aug 6 07:07:03 localhost kernel: [<c0478e90>] sys_close+0x71/0xa0
Aug 6 07:07:03 localhost kernel: [<c0404f9b>] syscall_call+0x7/0xb
Aug 6 07:07:03 localhost kernel: =======================
Aug 6 07:09:03 localhost kernel: INFO: task mysqld:29636 blocked for more than 120 seconds.
Aug 6 07:09:03 localhost kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug 6 07:09:03 localhost kernel: mysqld D 000419EB 2572 29636 3478 29638 3715 (NOTLB)
Aug 6 07:09:03 localhost kernel: e7726eac 00000082 ca2e64fc 000419eb f3da43c0 f8dc266e 00000004 00000009
Aug 6 07:09:03 localhost kernel: f14a35f0 ca347aaf 000419eb 000615b3 00000002 f14a36fc c2e20488 f2d1be40
Aug 6 07:09:03 localhost kernel: f8eceae8 00000000 00000286 c042f301 e0e9e501 c2f62000 00000286 ffffffff
Aug 6 07:09:03 localhost kernel: Call Trace:
Aug 6 07:09:03 localhost kernel: [<f8dc266e>] fileOpenHook+0x1ed/0x265 [lshook]
Aug 6 07:09:03 localhost kernel: [<c042f301>] lock_timer_base+0x15/0x2f
Aug 6 07:09:03 localhost kernel: [<f8ec5d8b>] doScan+0x840/0xc2a [linuxshield]
Aug 6 07:09:03 localhost kernel: [<c04383ff>] autoremove_wake_function+0x0/0x2d
Aug 6 07:09:03 localhost kernel: [<f8ec6e45>] SC_doScan+0xea/0xf4 [linuxshield]
Aug 6 07:09:03 localhost kernel: [<f8ec441e>] releaseHook+0xed/0x153 [linuxshield]
Aug 6 07:09:03 localhost kernel: [<f8dc1f4a>] invokeCallbacks+0x1fd/0x354 [lshook]
Aug 6 07:09:03 localhost kernel: [<f8dc2438>] fileReleaseHook+0xa0/0xe9 [lshook]
Aug 6 07:09:03 localhost kernel: [<c047a2c0>] __fput+0xb9/0x184
Aug 6 07:09:03 localhost kernel: [<c0477c37>] filp_close+0x4e/0x54
Aug 6 07:09:03 localhost kernel: [<c0478e90>] sys_close+0x71/0xa0
Aug 6 07:09:03 localhost kernel: [<c0404f9b>] syscall_call+0x7/0xb
Aug 6 07:09:03 localhost kernel: =======================
Environment
- Red Hat Enterprise Linux (all versions)
- McAfee
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.
