RHEL 6.4 BUG: soft lockups on xfsdatad

Issue

• Seeing "BUG: soft lockup - CPU#12 stuck for 67s! [xfsdatad/12:4514]" on RHEL 6.4 server.

Jan  5 20:56:59 <hostname> kernel: BUG: soft lockup - CPU#12 stuck for 67s! [xfsdatad/12:4514]
Jan  5 20:56:59 <hostname> kernel: CPU 12 
Jan  5 20:56:59 <hostname> kernel: Modules linked in: tcp_diag inet_diag fuse nfs fscache nfsd lockd nfs_acl auth_rpcgss autofs4
 sunrpc target_core_iblock target_core_file target_core_pscsi target_core_mod configfs bonding 8021q garp stp llc iptable_nat nf
_nat nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrac
k ip6table_filter ip6_tables rdma_ucm(U) ib_ucm(U) rdma_cm(U) iw_cm(U) ib_addr(U) ib_ipoib(U) ib_cm(U) ipv6 ib_uverbs(U) ib_umad
(U) mlx5_ib(U) mlx5_core(U) mlx4_en(U) mlx4_ib(U) ib_sa(U) ib_mad(U) ib_core(U) mlx4_core(U) compat(U) xfs exportfs knem(U) uinp
ut ses enclosure hpilo hpwdt ixgbe ptp pps_core mdio power_meter microcode serio_raw sg tg3 iTCO_wdt iTCO_vendor_support ioatdma
 dca shpchp ext4 jbd2 mbcache dm_round_robin sd_mod crc_t10dif qla2xxx(U) scsi_transport_fc scsi_tgt hpsa(U) dm_multipath dm_mir
ror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Jan  5 20:56:59 <hostname> kernel: 
Jan  5 20:56:59 <hostname> kernel: Pid: 4514, comm: xfsdatad/12 Not tainted 2.6.32-358.28.1.el6.x86_64 #1 HP ProLiant DL560 Gen8
Jan  5 20:56:59 <hostname> kernel: RIP: 0010:[<ffffffff81510ab7>]  [<ffffffff81510ab7>] _spin_unlock_irqrestore+0x17/0x20
Jan  5 20:56:59 <hostname> kernel: RSP: 0018:ffff88103329fd00  EFLAGS: 00000282
Jan  5 20:56:59 <hostname> kernel: RAX: ffff880028048e30 RBX: ffff88103329fd00 RCX: ffff880028048e30
Jan  5 20:56:59 <hostname> kernel: RDX: ffff880028048e28 RSI: 0000000000000282 RDI: 0000000000000282
Jan  5 20:56:59 <hostname> kernel: RBP: ffffffff8100bb8e R08: 0000000000000001 R09: 0000000000000001
Jan  5 20:56:59 <hostname> kernel: R10: 0000000000000000 R11: 0000000000000400 R12: ffff88103329fc80
Jan  5 20:56:59 <hostname> kernel: R13: 0000000000000000 R14: ffff88103329fd50 R15: ffff880028048e30
Jan  5 20:56:59 <hostname> kernel: FS:  0000000000000000(0000) GS:ffff88085c480000(0000) knlGS:0000000000000000
Jan  5 20:56:59 <hostname> kernel: CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
Jan  5 20:56:59 <hostname> kernel: CR2: 00007f7169e110a0 CR3: 000000102a540000 CR4: 00000000000407e0
Jan  5 20:56:59 <hostname> kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Jan  5 20:56:59 <hostname> kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Jan  5 20:56:59 <hostname> kernel: Process xfsdatad/12 (pid: 4514, threadinfo ffff88103329e000, task ffff88103147aaa0)
Jan  5 20:56:59 <hostname> kernel: Stack:
Jan  5 20:56:59 <hostname> kernel: ffff88103329fd40 ffffffff81055ad3 ffff88103329fd60 ffffea000e5a7c50
Jan  5 20:56:59 <hostname> kernel: <d> ffffea000e5a7c50 ffff88074de80f00 0000000000000202 ffffe8efffa8b248
Jan  5 20:56:59 <hostname> kernel: <d> ffff88103329fd60 ffffffff81096d51 ffffea000e5a7c50 ffffea000000000d
Jan  5 20:56:59 <hostname> kernel: Call Trace:
Jan  5 20:56:59 <hostname> kernel: [<ffffffff81055ad3>] ? __wake_up+0x53/0x70
Jan  5 20:56:59 <hostname> kernel: [<ffffffff81096d51>] ? __wake_up_bit+0x31/0x40
Jan  5 20:56:59 <hostname> kernel: [<ffffffff81119f05>] ? end_page_writeback+0x45/0x60
Jan  5 20:56:59 <hostname> kernel: [<ffffffff811b66f8>] ? end_buffer_async_write+0xd8/0x190
Jan  5 20:56:59 <hostname> kernel: [<ffffffffa02b071c>] ? xfs_destroy_ioend+0x3c/0xa0 [xfs]
Jan  5 20:56:59 <hostname> kernel: [<ffffffffa02b0970>] ? xfs_end_io+0x0/0xf0 [xfs]
Jan  5 20:56:59 <hostname> kernel: [<ffffffffa02b09bc>] ? xfs_end_io+0x4c/0xf0 [xfs]
Jan  5 20:56:59 <hostname> kernel: [<ffffffff81090be0>] ? worker_thread+0x170/0x2a0
Jan  5 20:56:59 <hostname> kernel: [<ffffffff81096da0>] ? autoremove_wake_function+0x0/0x40
Jan  5 20:56:59 <hostname> kernel: [<ffffffff81090a70>] ? worker_thread+0x0/0x2a0
Jan  5 20:56:59 <hostname> kernel: [<ffffffff81096a36>] ? kthread+0x96/0xa0
Jan  5 20:56:59 <hostname> kernel: [<ffffffff8100c0ca>] ? child_rip+0xa/0x20
Jan  5 20:56:59 <hostname> kernel: [<ffffffff810969a0>] ? kthread+0x0/0xa0
Jan  5 20:56:59 <hostname> kernel: [<ffffffff8100c0c0>] ? child_rip+0x0/0x20
Jan  5 20:56:59 <hostname> kernel: Code: 00 00 00 01 74 05 e8 a9 2d d7 ff c9 c3 0f 1f 80 00 00 00 00 55 48 89 e5 0f 1f 44 00 00 48 89 fa 66 ff 02 66 66 90 48 89 f7 57 9d <0f> 1f 44 00 00 c9 c3 66 90 55 48 89 e5 0f 1f 44 00 00 f0 ff 07