System Crashed at kernel BUG at fs/buffer.c:3154! in Red Hat Enterprise Linux

Issue

• System Crashed at kernel BUG at fs/buffer.c:3154! with following call trace :

<2>EXT4-fs error (device dm-129): htree_dirblock_to_tree: bad entry in directory #12: rec_len % 4 != 0 - block=48offset=0(0), inode=32774, rec_len=32774, name_len=1
<4>------------[ cut here ]------------
<2>kernel BUG at fs/buffer.c:3154!
<4>invalid opcode: 0000 [#1] SMP
<4>last sysfs file: /sys/devices/system/cpu/online
<4>CPU 6
<4>Modules linked in: joydev iptable_filter ip_tables bridge mptctl mptbase oracleasm(U) 8021q garp stp llc bonding deadman(U) dm_round_robin scsi_dh_alua dm_multipath iTCO_wdt iTCO_vendor_support microcode serio_raw ses enclosure ipmi_devintf power_meter acpi_ipmi ipmi_si ipmi_msghandler sb_edac edac_core lpc_ich mfd_core hpilo hpwdt ioatdma sg ext4 jbd2 mbcache sd_mod crc_t10dif qla2xxx scsi_transport_fc scsi_tgt crc32c_intel be2iscsi bnx2i cnic uio cxgb4i iw_cxgb4 cxgb4 cxgb3i igb libcxgbi dca iw_cxgb3 i2c_algo_bit i2c_core ib_core ptp ib_addr pps_core ipv6 cxgb3 mdio libiscsi_tcp qla4xxx iscsi_boot_sysfs libiscsi scsi_transport_iscsi hpsa wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
<4>
<4>Pid: 5100, comm: ls Not tainted 2.6.32-573.22.1.el6.x86_64 #1 HP ProLiant DL580 Gen9/ProLiant DL580 Gen9
<4>RIP: 0010:[<ffffffff811c81e2>]  [<ffffffff811c81e2>] submit_bh+0x152/0x1f0
<4>RSP: 0018:ffff882561197b48  EFLAGS: 00010246
<4>RAX: 0000000000000005 RBX: ffff88257e7eb678 RCX: ffff8831291c0000
<4>RDX: 0000000000000000 RSI: ffff88257e7eb678 RDI: 00000000000004c1
<4>RBP: ffff882561197b68 R08: 0000000000000001 R09: 0000000000000000
<4>R10: 0000000000000000 R11: 0000000000000198 R12: 00000000000004c1
<4>R13: ffffffff81c17bc0 R14: ffff88257e7eb678 R15: ffff8814cc01d400
<4>FS:  00007f991befa7a0(0000) GS:ffff883129080000(0000) knlGS:0000000000000000
<4>CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
<4>CR2: 00000000015f2368 CR3: 0000005edaba3000 CR4: 00000000001407e0
<4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
<4>Process ls (pid: 5100, threadinfo ffff882561194000, task ffff88255f6bcab0)
<4>Stack:
<4> ffff88257e7eb678 00000000000004c1 ffffffff81c17bc0 ffff88257e7eb678
<4><d> ffff882561197b88 ffffffff811ca203 0000000000001072 ffff88203dcd4c00
<4><d> ffff882561197b98 ffffffff811ca2b3 ffff882561197be8 ffffffffa0384c68
<4>Call Trace:
<4> [<ffffffff811ca203>] __sync_dirty_buffer+0x53/0xf0
<4> [<ffffffff811ca2b3>] sync_dirty_buffer+0x13/0x20
<4> [<ffffffffa0384c68>] ext4_commit_super+0x158/0x1e0 [ext4]
<4> [<ffffffffa0385222>] ext4_handle_error+0x72/0xd0 [ext4]
<4> [<ffffffffa0385612>] __ext4_error+0x82/0x90 [ext4]
<4> [<ffffffff8113e9b0>] ? __lru_cache_add+0x40/0x90
<4> [<ffffffffa036825d>] ext4_check_dir_entry+0xbd/0x100 [ext4]
<4> [<ffffffffa0376b40>] htree_dirblock_to_tree+0xc0/0x190 [ext4]
<4> [<ffffffffa0377558>] ext4_htree_fill_tree+0xa8/0x260 [ext4]
<4> [<ffffffff81153559>] ? handle_mm_fault+0x299/0x3d0
<4> [<ffffffffa03683c7>] ext4_readdir+0x127/0x700 [ext4]
<4> [<ffffffff8104f204>] ? __do_page_fault+0x1f4/0x500
<4> [<ffffffff811a8d20>] ? filldir+0x0/0xe0
<4> [<ffffffff810e62fd>] ? audit_filter_rules+0x2d/0xdd0
<4> [<ffffffff8123f29b>] ? selinux_file_permission+0xfb/0x150
<4> [<ffffffff811a8d20>] ? filldir+0x0/0xe0
<4> [<ffffffff811a8fa0>] vfs_readdir+0xc0/0xe0
<4> [<ffffffff811a9129>] sys_getdents+0x89/0xf0
<4> [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
<4>Code: 3c 01 45 19 e4 41 f7 d4 e8 cc 46 00 00 41 83 e4 a1 5b 44 89 e0 41 5c 41 5d 41 5e c9 c3 0f 0b eb fe 0f 0b eb fe 0f 0b 66 90 eb fc <0f> 0b eb fe 0f 0b 0f 1f 84 00 00 00 00 00 eb f6 48 8b 53 48 c1
<1>RIP  [<ffffffff811c81e2>] submit_bh+0x152/0x1f0
<4> RSP <ffff882561197b48>

• Sometimes before the crash, storage errors have been observed.

sd 1:0:0:233: reservation conflict
qla2xxx 0000:44:00.0: vpd r/w failed.  This is likely a firmware bug on this device.  Contact the card vendor for a firmware update.
scsi 3:0:0:116: alua: Detached

sd 1:0:0:0: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatically remap LUN assignments.

 rport-1:0-0: blocked FC remote port time out: removing target and saving binding
sd 1:0:0:68: rejecting I/O to offline device
sd 1:0:0:13: rejecting I/O to offline device
sd 1:0:0:13: [sdo] killing request
sd 1:0:0:13: rejecting I/O to offline device
sd 1:0:0:0: [sdb]  Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
device-mapper: multipath: Failing path 8:224.
sd 1:0:0:0: [sdb] CDB: Write(10): 2a 00 03 64 d2 d8 00 00 08 00
end_request: I/O error, dev sdb, sector 56939224
sd 1:0:0:13: [sdo]  Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
sd 1:0:0:13: [sdo] CDB: Write(10): 2a 00 00 05 07 40 00 00 08 00
end_request: I/O error, dev sdo, sector 329536
device-mapper: multipath: Failing path 8:16.
device-mapper: multipath: Failing path 66:32.
device-mapper: multipath: Failing path 69:48.