McAfee's LinuxShield caused kernel panic / hung tasks

Solution Unverified - Updated -

Issue

  • Ran the 'yum update' utility , and now every night servers experience a kernel panic

    KERNEL: /usr/lib/debug/lib/modules/2.6.18-274.7.1.el5/vmlinux
    DUMPFILE: /var/crash/2011-11-30-04:08/vmcore
        CPUS: 4
        DATE: Wed Nov 30 04:06:46 2011
      UPTIME: 23:48:10
LOAD AVERAGE: 1.41, 0.91, 0.38
       TASKS: 181
    NODENAME: xyz
     RELEASE: 2.6.18-274.7.1.el5
     VERSION: #1 SMP Mon Oct 17 11:57:40 EDT 2011
     MACHINE: i686  (2133 Mhz)
      MEMORY: 1.5 GB
       PANIC: "Oops: 0000 [#1]" (check log for details)
         PID: 281
     COMMAND: "pdflush"
        TASK: c9d79550  [THREAD_INFO: c9d7b000]
         CPU: 1
       STATE: TASK_RUNNING (PANIC)

BUG: unable to handle kernel NULL pointer dereference at virtual address 0000001c
 printing eip:
f8dfc0ec
*pde = 5bd72067
Oops: 0000 [#1]
SMP
last sysfs file: /devices/pci0000:00/0000:00:00.0/irq
Modules linked in: linuxshield(U) lshook(U) nfsd exportfs auth_rpcgss ipv6 xfrm_nalgo crypto_api autofs4 nfs nfs_acl lockd sunrpc 
ip_conntrack_netbios_ns iptable_nat ip_nat iptable_mangle ipt_REJECT xt_state ip_conntrack nfnetlink xt_tcpudp iptable_filter ip_tables 
x_tables vsock(U) vmmemctl(U) acpiphp loop dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery
asus_acpi ac lp parport_pc ide_cd i2c_piix4 parport floppy tpm_tis tpm cdrom i2c_core tpm_bios serio_raw vmci(U) sg pcspkr pvscsi(U)
vmxnet3(U) vmxnet(U) dm_raid45 dm_message dm_region_hash dm_mem_cache dm_snapshot dm_zero dm_mirror dm_log dm_mod ata_piix libata mptspi 
mptscsih mptbase scsi_transport_spi sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd

CPU:    1
EIP:    0060:[<f8dfc0ec>]    Tainted: G     ---- VLI
EFLAGS: 00010246   (2.6.18-274.7.1.el5 #1)
EIP is at fileOpenHook+0x64/0x265 [lshook]
eax: 00000000   ebx: f6777180   ecx: c9d79550   edx: c9d7be94
esi: d57a6e70   edi: c9d7bed4   ebp: 00000000   esp: c9d7be88
ds: 007b   es: 007b   ss: 0068
Process pdflush (pid: 281, ti=c9d7b000 task=c9d79550 task.ti=c9d7b000)
Stack: f8b5b8df f7306e60 00000000 00000000 f6777180 d57a6e70 c9d7bed4 00000000
       f8b8a494 f8b9e91c f6777180 f7306e60 d57a6e70 d57a6e70 00000001 c9d7bed4
       d57a6d2c f8b8adbd 00000000 c9d7bed4 c9d7bed4 00000000 d57a6e70 d57a6f18
Call Trace:
 [<f8b5b8df>] rpc_init_task_wq+0x35/0x11d [sunrpc]
 [<f8b8a494>] nfs_commit_list+0x108/0x200 [nfs]
 [<f8b8adbd>] nfs_commit_inode+0x5b/0x78 [nfs]
 [<f8b83655>] nfs_write_inode+0x41/0x5c [nfs]
 [<c04969de>] __writeback_single_inode+0x199/0x2a5
 [<c0496dce>] sync_sb_inodes+0x17e/0x221
 [<c049701d>] writeback_inodes+0x6a/0xb0
 [<c045f986>] wb_kupdate+0xcb/0x130
 [<c045fda7>] pdflush+0x0/0x1a1
 [<c045feb2>] pdflush+0x10b/0x1a1
 [<c045f8bb>] wb_kupdate+0x0/0x130
 [<c0436e4e>] kthread+0xc0/0xee
 [<c0436d8e>] kthread+0x0/0xee
 [<c0405c87>] kernel_thread_helper+0x7/0x10

=======================
Code: 89 d0 ff 73 20 25 ff ff 0f 00 c1 ea 14 50 52 68 0c fa df f8 68 7e d2 df f8 e8 d9 9e 62 c7 83 c4 1c 8b 83 9c 00 00 00 8d 54 24 0c <8b> 40 1c e8 44 e1 ff ff 8b 44 24 0c 83 c0 14 e8 30 71 82 c7 8b
EIP: [<f8dfc0ec>] fileOpenHook+0x64/0x265 [lshook] SS:ESP 0068:c9d7be88

  • Server is having hung tasks constantly, with the following stack trace in the logs:

Aug  6 07:07:03 localhost kernel: INFO: task mysqld:29636 blocked for more than 120 seconds.
Aug  6 07:07:03 localhost kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug  6 07:07:03 localhost kernel: mysqld        D 000419EB  2572 29636   3478         29638  3715 (NOTLB)
Aug  6 07:07:03 localhost kernel:        e7726eac 00000082 ca2e64fc 000419eb f3da43c0 f8dc266e 00000004 00000009 
Aug  6 07:07:03 localhost kernel:        f14a35f0 ca347aaf 000419eb 000615b3 00000002 f14a36fc c2e20488 f2d1be40 
Aug  6 07:07:03 localhost kernel:        f8eceae8 00000000 00000286 c042f301 e0e9e501 c2f62000 00000286 ffffffff 
Aug  6 07:07:03 localhost kernel: Call Trace:
Aug  6 07:07:03 localhost kernel:  [<f8dc266e>] fileOpenHook+0x1ed/0x265 [lshook]
Aug  6 07:07:03 localhost kernel:  [<c042f301>] lock_timer_base+0x15/0x2f
Aug  6 07:07:03 localhost kernel:  [<f8ec5d8b>] doScan+0x840/0xc2a [linuxshield]
Aug  6 07:07:03 localhost kernel:  [<c04383ff>] autoremove_wake_function+0x0/0x2d
Aug  6 07:07:03 localhost kernel:  [<f8ec6e45>] SC_doScan+0xea/0xf4 [linuxshield]
Aug  6 07:07:03 localhost kernel:  [<f8ec441e>] releaseHook+0xed/0x153 [linuxshield]
Aug  6 07:07:03 localhost kernel:  [<f8dc1f4a>] invokeCallbacks+0x1fd/0x354 [lshook]
Aug  6 07:07:03 localhost kernel:  [<f8dc2438>] fileReleaseHook+0xa0/0xe9 [lshook]
Aug  6 07:07:03 localhost kernel:  [<c047a2c0>] __fput+0xb9/0x184
Aug  6 07:07:03 localhost kernel:  [<c0477c37>] filp_close+0x4e/0x54
Aug  6 07:07:03 localhost kernel:  [<c0478e90>] sys_close+0x71/0xa0
Aug  6 07:07:03 localhost kernel:  [<c0404f9b>] syscall_call+0x7/0xb
Aug  6 07:07:03 localhost kernel:  =======================
Aug  6 07:09:03 localhost kernel: INFO: task mysqld:29636 blocked for more than 120 seconds.
Aug  6 07:09:03 localhost kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Aug  6 07:09:03 localhost kernel: mysqld        D 000419EB  2572 29636   3478         29638  3715 (NOTLB)
Aug  6 07:09:03 localhost kernel:        e7726eac 00000082 ca2e64fc 000419eb f3da43c0 f8dc266e 00000004 00000009 
Aug  6 07:09:03 localhost kernel:        f14a35f0 ca347aaf 000419eb 000615b3 00000002 f14a36fc c2e20488 f2d1be40 
Aug  6 07:09:03 localhost kernel:        f8eceae8 00000000 00000286 c042f301 e0e9e501 c2f62000 00000286 ffffffff 
Aug  6 07:09:03 localhost kernel: Call Trace:
Aug  6 07:09:03 localhost kernel:  [<f8dc266e>] fileOpenHook+0x1ed/0x265 [lshook]
Aug  6 07:09:03 localhost kernel:  [<c042f301>] lock_timer_base+0x15/0x2f
Aug  6 07:09:03 localhost kernel:  [<f8ec5d8b>] doScan+0x840/0xc2a [linuxshield]
Aug  6 07:09:03 localhost kernel:  [<c04383ff>] autoremove_wake_function+0x0/0x2d
Aug  6 07:09:03 localhost kernel:  [<f8ec6e45>] SC_doScan+0xea/0xf4 [linuxshield]
Aug  6 07:09:03 localhost kernel:  [<f8ec441e>] releaseHook+0xed/0x153 [linuxshield]
Aug  6 07:09:03 localhost kernel:  [<f8dc1f4a>] invokeCallbacks+0x1fd/0x354 [lshook]
Aug  6 07:09:03 localhost kernel:  [<f8dc2438>] fileReleaseHook+0xa0/0xe9 [lshook]
Aug  6 07:09:03 localhost kernel:  [<c047a2c0>] __fput+0xb9/0x184
Aug  6 07:09:03 localhost kernel:  [<c0477c37>] filp_close+0x4e/0x54
Aug  6 07:09:03 localhost kernel:  [<c0478e90>] sys_close+0x71/0xa0
Aug  6 07:09:03 localhost kernel:  [<c0404f9b>] syscall_call+0x7/0xb
Aug  6 07:09:03 localhost kernel:  =======================

Environment

  • Red Hat Enterprise Linux (all versions)
  • McAfee

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.