RHEL6.4: kernel BUG at mm/slab.c:3069 involving size-32 kmem cache with twnotify and secfs2 3rd party modules

Issue

• Software installed on the system, once started crashes the server with a kernel panic. Looking into what component of this software is crashing the server. This happens whenever they start the processes, and is a repeatable crash.
• RHEL6 panic in cache_alloc_refill, kernel BUG at mm/slab.c:3069
• kernel assertion BUG_ON(slabp->inuse >= cachep->num) hit involving size-32 kmem cache, and the inuse value is equal to the num value.

kernel BUG at mm/slab.c:3069!
invalid opcode: 0000 [#1] SMP 
last sysfs file: /sys/devices/virtual/dmi/id/product_serial
CPU 6 
Modules linked in: twnotify(U) nfsd exportfs nfs lockd fscache auth_rpcgss nfs_acl sunrpc secvm2(P)(U) secfs2(P)(U) vsock(U) ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables ext3 jbd ppdev parport_pc parport e1000 microcode vmware_balloon sg vmci(U) i2c_piix4 i2c_core shpchp ext4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom mptspi mptscsih mptbase scsi_transport_spi pata_acpi ata_generic ata_piix dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 19534, comm: java Tainted: P           ---------------    2.6.32-358.el6.x86_64 #1 VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform
RIP: 0010:[<ffffffff81167274>]  [<ffffffff81167274>] cache_alloc_refill+0x1e4/0x240
RSP: 0018:ffff88019c50d9b8  EFLAGS: 00010046
RAX: 0000000000000035 RBX: ffff88023fc00040 RCX: 0000000000000070
RDX: ffff88023999a000 RSI: ffff88023fc22440 RDI: ffff88019c711000
RBP: ffff88019c50da18 R08: ffff88023999a000 R09: ffff8801a6641448
R10: 0000000100700afc R11: 0000000000000000 R12: ffff88023cf64000
R13: ffff88023fc22440 R14: 0000000000000035 R15: ffff88019c711000
FS:  00007ff394cda700(0000) GS:ffff88002f6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007ff3c8141000 CR3: 00000002395ab000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process java (pid: 19534, threadinfo ffff88019c50c000, task ffff88019c4b3500)
Stack:
 ffff88019c50d9c8 000000008111c477 ffff88023fc22480 000412d08111c5c5
<d> ffff88023fc22460 ffff88023fc22450 000000109c50da18 000000000000001c
<d> 0000000000000010 00000000000000d0 ffff88023fc00040 ffffffff8113a0a9
Call Trace:
 [<ffffffff8113a0a9>] ? kmemdup+0x29/0x50
 [<ffffffff81168159>] __kmalloc+0x1a9/0x220
 [<ffffffff8113a0a9>] kmemdup+0x29/0x50
 [<ffffffffa046c541>] nfs_readdir_page_filler+0x3f1/0x580 [nfs]
 [<ffffffffa046c8d8>] nfs_readdir_xdr_to_array+0x208/0x2b0 [nfs]
 [<ffffffffa046c9a6>] nfs_readdir_filler+0x26/0xa0 [nfs]
 [<ffffffff8111a25e>] ? add_to_page_cache_lru+0x3e/0x50
 [<ffffffff8111bc4b>] do_read_cache_page+0x7b/0x180
 [<ffffffffa046c980>] ? nfs_readdir_filler+0x0/0xa0 [nfs]
 [<ffffffff81196100>] ? filldir+0x0/0xe0
 [<ffffffff81196100>] ? filldir+0x0/0xe0
 [<ffffffff8111bd99>] read_cache_page_async+0x19/0x20
 [<ffffffff8111bdae>] read_cache_page+0xe/0x20
 [<ffffffffa046cb92>] nfs_readdir+0x172/0x5f0 [nfs]
 [<ffffffffa0485a40>] ? nfs3_decode_dirent+0x0/0x3d0 [nfs]
 [<ffffffff81196100>] ? filldir+0x0/0xe0
 [<ffffffff81196380>] vfs_readdir+0xc0/0xe0
 [<ffffffff81196100>] ? filldir+0x0/0xe0
 [<ffffffffa025d4b1>] op_readdir+0xd1/0x160 [secfs2]
 [<ffffffff81196100>] ? filldir+0x0/0xe0
 [<ffffffff81196100>] ? filldir+0x0/0xe0
 [<ffffffff81196380>] vfs_readdir+0xc0/0xe0
 [<ffffffff81196509>] sys_getdents+0x89/0xf0
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Code: 89 ff e8 c0 1b 12 00 eb 99 66 0f 1f 44 00 00 41 c7 45 60 01 00 00 00 4d 8b 7d 20 4c 39 7d c0 0f 85 f2 fe ff ff eb 84 0f 0b eb fe <0f> 0b 66 2e 0f 1f 84 00 00 00 00 00 eb f4 8b 55 ac 8b 75 bc 31 
RIP  [<ffffffff81167274>] cache_alloc_refill+0x1e4/0x240
 RSP <ffff88019c50d9b8>