Veritas Cluster Server (VCS) node running RHEL 6 crashes in llt_free_mblk()

Issue

• General protection fault in llt_free_mblk() crashing the system with console messages

[11323092.205077] general protection fault: 0000 [#1] SMP
[11323092.240957] last sysfs file: /sys/devices/system/cpu/online
[11323092.278900] CPU 0
[11323092.282119] Modules linked in: tcp_diag inet_diag nls_utf8 ext2 raid1 raid0 linear cdrom msdos nfsd exportfs bridge ipt_REJECT xt_multiport nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_LOG xt_limit iptable_filter ip_tables amf(P)(U) vxfen(P)(U) gab(P)(U) llt(P)(U) rdma_cm ib_cm iw_cm ib_sa ib_mad seos(P)(U) nfs lockd fscache auth_rpcgss nfs_acl sunrpc acpi_cpufreq freq_table mperf 8021q garp stp llc bonding ipmi_devintf vfat fat iTCO_wdt iTCO_vendor_support microcode joydev lpc_ich mfd_core hpilo hpwdt ioatdma dca ipmi_si ipmi_msghandler sg ext4 jbd2 mbcache dm_round_robin sd_mod crc_t10dif qla2xxx scsi_transport_fc scsi_tgt crc32c_intel be2iscsi bnx2i(U) cnic(U) uio cxgb4i iw_cxgb4 cxgb4 cxgb3i libcxgbi iw_cxgb3 ib_core ib_addr ipv6 cxgb3 libiscsi_tcp qla4xxx iscsi_boot_sysfs bnx2x(U) libiscsi scsi_transport_iscsi libcrc32c mdio dm_multipath dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
[11323092.822973]
[11323092.887891] Pid: 17523, comm: had Tainted: P           -- ------------    2.6.32-573.12.1.el6.x86_64 #1 HP Superdome2 16s x86
[11323093.039272] RIP: 0010:[<ffffffffa0a5b230>]  [<ffffffffa0a5b230>] llt_free_mblk+0x10/0x30 [llt]
[11323093.124418] RSP: 0018:ffff88a2c1b97db8  EFLAGS: 00010296
[11323093.206279] RAX: ffffffffa0a9ec20 RBX: 0000000000000000 RCX: 0000000000000007
[11323093.292684] RDX: 000000000000a398 RSI: 0000000001200000 RDI: 7346000000084600
[11323093.379265] RBP: ffff88a2c1b97dc8 R08: ffffffffa0ae3830 R09: 00000000ffc42ab8
[11323093.467431] R10: 0000000000000001 R11: 0000000000000000 R12: ffff8833efc49470
[11323093.557118] R13: ffff8833efc49440 R14: 0000000000000102 R15: 0000000000000000
[11323093.648016] FS:  0000000000000000(0000) GS:ffff880285e00000(0063) knlGS:00000000f77388e0
[11323093.742381] CS:  0010 DS: 002b ES: 002b CR0: 0000000080050033
[11323093.835192] CR2: 000000000cebb490 CR3: 000000a28f2cc000 CR4: 00000000001407f0
[11323093.931956] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[11323094.029432] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[11323094.126552] Process had (pid: 17523, threadinfo ffff88a2c1b94000, task ffff88c065596ab0)
[11323094.227467] Stack:
[11323094.320814]  ffff88a2c1b97dd8 0000000000000000 ffff88a2c1b97dd8 ffffffffa0ae2102
[11323094.332347] <d> ffff88a2c1b97e98 ffffffffa0ac6b61 ffff88a2c1b97df8 ffff880468f19e48
[11323094.436690] <d> ffff88a2c1b97ed8 ffffffff811e4575 ffff88a2c1b97e38 ffffffff00000000
[11323094.638127] Call Trace:
[11323094.739096]  [<ffffffffa0ae2102>] gab_ifreemsg+0x22/0x30 [gab]
[11323094.846856]  [<ffffffffa0ac6b61>] gabread+0x811/0xd80 [gab]
[11323094.955075]  [<ffffffff811e4575>] ? fcntl_setlk+0x75/0x320
[11323095.064209]  [<ffffffffa0ae387f>] gab_linux_read+0x4f/0x70 [gab]
[11323095.174643]  [<ffffffff81192315>] vfs_read+0xb5/0x1a0
[11323095.284781]  [<ffffffff811930c6>] ? fget_light_pos+0x16/0x50
[11323095.396032]  [<ffffffff81192661>] sys_read+0x51/0xb0
[11323095.507185]  [<ffffffff81055e50>] sysenter_dispatch+0x7/0x2e
[11323095.620155] Code: a8 a0 31 c0 e8 eb d0 ad e0 31 c0 eb c4 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 48 83 ec 08 0f 1f 44 00 00 66 90 <48> 8b 5f 30 48 c7 47 30 00 00 00 00 e8 8f 30 a0 e0 48 85 db 48
[11323095.869168] RIP  [<ffffffffa0a5b230>] llt_free_mblk+0x10/0x30 [llt]
[11323095.991522]  RSP <ffff88a2c1b97db8>

• The kernel panic stack trace:

crash> bt
PID: 17523  TASK: ffff88c065596ab0  CPU: 0   COMMAND: "had"
 #0 [ffff88a2c1b97b40] machine_kexec at ffffffff8103d1ab
 #1 [ffff88a2c1b97ba0] crash_kexec at ffffffff810cc4a2
 #2 [ffff88a2c1b97c70] oops_end at ffffffff8153cff0
 #3 [ffff88a2c1b97ca0] die at ffffffff81010f5b
 #4 [ffff88a2c1b97cd0] do_general_protection at ffffffff8153cae2
 #5 [ffff88a2c1b97d00] general_protection at ffffffff8153c2b5
    [exception RIP: llt_free_mblk+0x10]
    RIP: ffffffffa0a5b230  RSP: ffff88a2c1b97db8  RFLAGS: 00010296
    RAX: ffffffffa0a9ec20  RBX: 0000000000000000  RCX: 0000000000000007
    RDX: 000000000000a398  RSI: 0000000001200000  RDI: 7346000000084600
    RBP: ffff88a2c1b97dc8   R8: ffffffffa0ae3830   R9: 00000000ffc42ab8
    R10: 0000000000000001  R11: 0000000000000000  R12: ffff8833efc49470
    R13: ffff8833efc49440  R14: 0000000000000102  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #6 [ffff88a2c1b97dd0] gab_ifreemsg at ffffffffa0ae2102 [gab]
 #7 [ffff88a2c1b97de0] gabread at ffffffffa0ac6b61 [gab]
 #8 [ffff88a2c1b97ea0] gab_linux_read at ffffffffa0ae387f [gab]
 #9 [ffff88a2c1b97ef0] vfs_read at ffffffff81192315
#10 [ffff88a2c1b97f30] sys_read at ffffffff81192661
#11 [ffff88a2c1b97f80] sysenter_dispatch at ffffffff81055e50