Server reboots automatically

Issue

• A critical mail server automatically reboots which creates an issue in the Environment. The system reboots frequently without warning. The timing of the reboots is also sporadic - as often as 3 times per day or as infrequent as twice per week.

• If kdump is configured check for a resultant vmcore from one of the reboots. In this specific case 22 coredumps were found under /var/crash spread over the course of one month. Each indicated the same BUG condition and RIP:

<1>BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
<1>IP: [<ffffffffa001b73b>] custom_file_open+0x10b/0x150 [lshook]

• The call trace contains no other evidence - no other kernel functions are shown in the stack. However it was noticed that in each coredump the system had hit the fs.file-max limit prior to the crash as indicated by the following in vmcore-dmesg:

<6> VFS: file-max limit 32768 reached

• The RIP - custom_file_open [lshook] is third party closed source code so we can not be sure of it's purpose however based on the name it could be related to opening a file. Hitting the file-max limit will prevent any additional files from being opened and influence said path.

Full call trace:

<7>linuxshield module is older than RHEL 6.2 ... applying fixups
<6>VFS: file-max limit 32768 reached
<6>VFS: file-max limit 32768 reached
<1>BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
<1>IP: [<ffffffffa001b73b>] custom_file_open+0x10b/0x150 [lshook]
<4>PGD 3f415e067 PUD 410999067 PMD 0 
<4>Oops: 0000 [#1] SMP 
<4>last sysfs file: /sys/devices/system/cpu/online
<4>CPU 1 
<4>Modules linked in: linuxshield(U) lshook(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipv6 ppdev parport_pc parport microcode vmware_balloon sg i2c_piix4 i2c_core shpchp ext3 jbd mbcache sd_mod crc_t10dif sr_mod cdrom vmxnet3 mptspi mptscsih mptbase scsi_transport_spi pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
<4>
<4>Pid: 31475, comm: deliver Not tainted 2.6.32-504.8.1.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
<4>RIP: 0010:[<ffffffffa001b73b>]  [<ffffffffa001b73b>] custom_file_open+0x10b/0x150 [lshook]
<4>RSP: 0018:ffff8804108b9f38  EFLAGS: 00010246
<4>RAX: ffff8803f8a60c40 RBX: 000000000000000b RCX: 000000000000000d
<4>RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff8803f8a60c40
<4>RBP: ffff8804108b9f78 R08: ffff8803f8a60bc0 R09: 0000000000000000
<4>R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffffffffe9
<4>R13: 000000000195f238 R14: 00000000000001b6 R15: ffff8803f8a60c40
<4>FS:  00007fa7688357c0(0000) GS:ffff880028240000(0000) knlGS:0000000000000000
<4>CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>CR2: 0000000000000018 CR3: 00000005ab3f5000 CR4: 00000000000007e0
<4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>Process deliver (pid: 31475, threadinfo ffff8804108b8000, task ffff8805e9b33500)
<4>Stack:
<4> ffff8804108b9f78 ffff8803f8a60bc0 0000000000000001 00000000562e4412
<4><d> 000000000199e3e0 00007fffd11bffe0 0000000000000000 0000000000000000
<4><d> 0000000000000001 ffffffff8100b072 0000000000000246 0000000000000008
<4>Call Trace:
<4> [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
<4>Code: e6 ff 15 a1 31 00 00 4c 89 ff 49 89 c4 e8 4e 17 51 e1 4c 8b 45 c8 49 8b 40 08 48 8b 40 08 4a 8b 14 e0 4c 89 f8 c7 00 00 00 00 00 <48> 8b 42 18 31 ff 48 8b 70 10 31 c0 e8 14 e2 ff ff 44 89 e7 41 
<1>RIP  [<ffffffffa001b73b>] custom_file_open+0x10b/0x150 [lshook]