Issue

The system with Veritas VxVM/VxFS modules installed crashes frequently in [vxfs] code or at __jbd2_journal_remove_checkpoint+0xc9 or at do_get_write_access+0x5b with the call traces similar to these.

The first crash type:

crash> log
BUG: unable to handle kernel NULL pointer dereference at 0000000000000622
IP: [<ffffffffa01369b9>] __jbd2_journal_remove_checkpoint+0xc9/0x190 [jbd2]
PGD 70e00e067 PUD 6b7348067 PMD 0 
Oops: 0000 [#1] SMP 
last sysfs file: /sys/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.3/speed
CPU 0 
Modules linked in: mptctl mptbase vxodm(P)(U) vxgms(P)(U) amf(P)(U) vxglm(P)(U) vxfen(P)(U) gab(P)(U) llt(P)(U) sunrpc dmpjbod(P)(U)
dmpap(P)(U) dmpaa(P)(U) vxspec(P)(U) vxio(P)(U) vxdmp(P)(U) bonding 8021q garp stp llc ipv6 vxportal(P)(U) fdd(P)(U) vxfs(P)(U) exportfs
ext3 jbd hpilo hpwdt power_meter sg be2net serio_raw iTCO_wdt iTCO_vendor_support ioatdma dca shpchp ext4 mbcache jbd2 sd_mod crc_t10dif
lpfc scsi_transport_fc scsi_tgt hpsa(U) video output dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Pid: 2415, comm: jbd2/dm-4-8 Tainted: P           ---------------    2.6.32-358.0.1.el6.x86_64 #1 HP ProLiant BL460c Gen8
RIP: 0010:[<ffffffffa01369b9>]  [<ffffffffa01369b9>] __jbd2_journal_remove_checkpoint+0xc9/0x190 [jbd2]
RSP: 0018:ffff88082f9d1ce0  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8810314833c0 RCX: 0000000000000000
RDX: ffff880fc9842d68 RSI: 0000000000000004 RDI: ffff880fc9842d68
RBP: ffff88082f9d1d20 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000021 R11: ffff8808349eff88 R12: 0000000000000292
R13: ffff881031483448 R14: ffff880fc9842d68 R15: ffff880830ac7540
FS:  0000000000000000(0000) GS:ffff880045600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000622 CR3: 00000006f15bd000 CR4: 00000000000407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process jbd2/dm-4-8 (pid: 2415, threadinfo ffff88082f9d0000, task ffff8808333ecae0)
Stack:
 ffff88082f9d1d00 ffff880fb5b61588 ffff88082f9d1d20 ffff88082fbefb9c
<d> ffff881030f889c0 ffff88082fbef800 ffff880fc9842d68 ffff880830ac7540
<d> ffff88082f9d1e60 ffffffffa0135236 ffff88082f9d1d90 ffff88083121f740
Call Trace:
 [<ffffffffa0135236>] jbd2_journal_commit_transaction+0xc06/0x14b0 [jbd2]
 [<ffffffff81081a5b>] ? try_to_del_timer_sync+0x7b/0xe0
 [<ffffffffa013af38>] kjournald2+0xb8/0x220 [jbd2]
 [<ffffffff81096ca0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa013ae80>] ? kjournald2+0x0/0x220 [jbd2]
 [<ffffffff81096936>] kthread+0x96/0xa0
 [<ffffffff8100c0ca>] child_rip+0xa/0x20
 [<ffffffff810968a0>] ? kthread+0x0/0xa0
 [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
RIP  [<ffffffffa01369b9>] __jbd2_journal_remove_checkpoint+0xc9/0x190 [jbd2]
 RSP <ffff88082f9d1ce0>
CR2: 0000000000000622

The second crash type:

crash> log
BUG: unable to handle kernel NULL pointer dereference at 0000000000000292
IP: [<ffffffffa0133e1b>] do_get_write_access+0x5b/0x520 [jbd2]
PGD 10304d6067 PUD 10328de067 PMD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.3/speed
CPU 0
Modules linked in: mptctl mptbase vxodm(P)(U) vxgms(P)(U) amf(P)(U) vxglm(P)(U) vxfen(P)(U) gab(P)(U) llt(P)(U) sunrpc dmpjbod(P)(U)
dmpap(P)(U) dmpaa(P)(U) vxspec(P)(U) vxio(P)(U) vxdmp(P)(U) bonding 8021q garp stp llc ipv6 vxportal(P)(U) fdd(P)(U) vxfs(P)(U) exportfs
ext3 jbd power_meter hpilo hpwdt sg be2net serio_raw iTCO_wdt iTCO_vendor_support ioatdma dca shpchp ext4 mbcache jbd2 sd_mod crc_t10dif
lpfc scsi_transport_fc scsi_tgt hpsa(U) video output dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Pid: 6591, comm: cmastdeqd Tainted: P           ---------------    2.6.32-358.0.1.el6.x86_64 #1 HP ProLiant BL460c Gen8
RIP: 0010:[<ffffffffa0133e1b>]  [<ffffffffa0133e1b>] do_get_write_access+0x5b/0x520 [jbd2]
RSP: 0018:ffff881033125bd8  EFLAGS: 00010246
RAX: 0000000000000292 RBX: ffff8807c6721748 RCX: ffff8810345c6f58
RDX: 0000000000000000 RSI: ffff8807c6721748 RDI: ffff8807b2701528
RBP: ffff881033125ca8 R08: 0000000000000003 R09: ffffea001c4a1500
R10: ffff881032f0dc00 R11: 0000000000003213 R12: ffff8807c6721748
R13: ffff88082ff425c0 R14: ffff8807b2701528 R15: ffff881032f0dc00
FS:  00007f0c7d8ad700(0000) GS:ffff880045800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000292 CR3: 0000001031de2000 CR4: 00000000000407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process cmastdeqd (pid: 6591, threadinfo ffff881033124000, task ffff8810315fd540)
Stack:
 ffff881033125c48 0000000000001000 ffff881033125d68 ffff8807ff6a54c0
<d> ffff881033125d68 ffff881031048b40 ffff881033125c78 ffffffff811b572c
<d> ffff88079a79a021 0000000000000004 0000000000000292 ffff881033125d38
Call Trace:
 [<ffffffff811b572c>] ? __getblk+0x2c/0x2a0
 [<ffffffffa0134431>] jbd2_journal_get_write_access+0x31/0x50 [jbd2]
 [<ffffffffa017f708>] __ext4_journal_get_write_access+0x38/0x80 [ext4]
 [<ffffffffa015ab33>] ext4_reserve_inode_write+0x73/0xa0 [ext4]
 [<ffffffffa015abac>] ext4_mark_inode_dirty+0x4c/0x1d0 [ext4]
 [<ffffffffa0133455>] ? jbd2_journal_start+0xb5/0x100 [jbd2]
 [<ffffffffa015aea0>] ext4_dirty_inode+0x40/0x60 [ext4]
 [<ffffffff811ac21b>] __mark_inode_dirty+0x3b/0x160
 [<ffffffff8119e39c>] inode_setattr+0x4c/0x60
 [<ffffffffa015eb5c>] ext4_setattr+0x10c/0x330 [ext4]
 [<ffffffff8119e768>] notify_change+0x168/0x340
 [<ffffffff8117e808>] sys_fchmod+0xf8/0x130
 [<ffffffff810dc645>] ? __audit_syscall_exit+0x265/0x290
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
RIP  [<ffffffffa0133e1b>] do_get_write_access+0x5b/0x520 [jbd2]
 RSP <ffff881033125bd8>
CR2: 0000000000000292

The third crash type:

crash> log
BUG: unable to handle kernel NULL pointer dereference at 000000000000029a
IP: [<ffffffffa02dfab4>] vx_msgblkalloc+0x34/0x1c0 [vxfs]

crash> bt
PID: 2569   TASK: ffff880831ec6ae0  CPU: 1   COMMAND: "vx_worklist_thr"
 #0 [ffff880831ecb6f0] machine_kexec at ffffffff81035b7b
 #1 [ffff880831ecb750] crash_kexec at ffffffff810c0dd2
 #2 [ffff880831ecb820] oops_end at ffffffff815112b0
 #3 [ffff880831ecb850] no_context at ffffffff81046bfb
 #4 [ffff880831ecb8a0] __bad_area_nosemaphore at ffffffff81046e85
 #5 [ffff880831ecb8f0] bad_area_nosemaphore at ffffffff81046f53
 #6 [ffff880831ecb900] __do_page_fault at ffffffff810476b1
 #7 [ffff880831ecba20] do_page_fault at ffffffff815131fe
 #8 [ffff880831ecba50] page_fault at ffffffff815105b5
    [exception RIP: vx_msgblkalloc+52]
    RIP: ffffffffa02dfab4  RSP: ffff880831ecbb00  RFLAGS: 00010006
    RAX: 0000000000000246  RBX: ffff88082f0dc480  RCX: ffffffffa04ddfc0
    RDX: 0000000000000292  RSI: 0000000000080019  RDI: ffffffffa04ddfd4
    RBP: ffff880831ecbb10   R8: 0000000000000005   R9: ffff880831ecbc38
    R10: 0000000000000300  R11: 0000000000000000  R12: 0000000000000001
    R13: 0000000000080019  R14: 000000000c709c40  R15: ffff880831ecbc20
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #9 [ffff880831ecbb18] vx_msg_broadcast at ffffffffa02e2764 [vxfs]
#10 [ffff880831ecbb88] vx_bcast_fsesumsync at ffffffffa0302e5e [vxfs]
#11 [ffff880831ecbc78] vx_cfs_efree at ffffffffa030327d [vxfs]
#12 [ffff880831ecbca8] vx_edele_mode_check at ffffffffa030340c [vxfs]
#13 [ffff880831ecbcd8] vx_edele_release_fs at ffffffffa0305668 [vxfs]
#14 [ffff880831ecbd18] vx_do_fsext at ffffffffa03d1658 [vxfs]
#15 [ffff880831ecbd38] vx_workitem_process at ffffffffa0384b28 [vxfs]
#16 [ffff880831ecbd58] vx_worklist_process at ffffffffa0384c91 [vxfs]
#17 [ffff880831ecbdb8] vx_walk_fslist_threaded at ffffffffa03d13ed [vxfs]
#18 [ffff880831ecbe18] vx_walk_fslist at ffffffffa03d1621 [vxfs]
#19 [ffff880831ecbe68] vx_edele_release at ffffffffa03005d0 [vxfs]
#20 [ffff880831ecbe78] vx_workitem_process at ffffffffa0384b28 [vxfs]
#21 [ffff880831ecbe98] vx_worklist_process at ffffffffa0384db3 [vxfs]
#22 [ffff880831ecbef8] vx_worklist_thread at ffffffffa038fd2d [vxfs]
#23 [ffff880831ecbf18] vx_kthread_init at ffffffffa03f31ab [vxfs]
#24 [ffff880831ecbf48] kernel_thread at ffffffff8100c0ca