System panics with "BUG: scheduling while atomic" at 'task_rq_lock+0x4d/0xa0' with XFS file system(s) mounted in RHEL 6

Issue

• Alternating nodes in a 2 node cluster are being semi-regularly fenced due to a hang or panic
• System with XFS file system mounted panics:

<3>BUG: scheduling while atomic: oracle/19822/0xffffffff
<1>BUG: unable to handle kernel paging request at 000000038a60e5a8
<1>IP: [<ffffffff81058fdd>] task_rq_lock+0x4d/0xa0
<4>PGD 0 
<4>Oops: 0000 [#1] SMP 
<4>last sysfs file: /sys/devices/system/node/node1/meminfo
<4>CPU 0 
<4>Modules linked in: xfs exportfs dlm configfs autofs4 nfs lockd fscache auth_rpcgss nfs_acl sunrpc bonding 8021q garp stp llc ipv6 ipt_REJECT xt_comment nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables emcpvlumd(P)(U) emcpxcrypt(P)(U) emcpdm(P)(U) emcpgpx(P)(U) emcpmpx(P)(U) emcp(P)(U) iTCO_wdt iTCO_vendor_support dcdbas microcode sb_edac edac_core lpc_ich mfd_core shpchp power_meter ses enclosure sg tg3 ptp pps_core ext4 jbd2 mbcache sr_mod cdrom sd_mod crc_t10dif ahci qla2xxx scsi_transport_fc scsi_tgt megaraid_sas wmi dm_mirror dm_region_hash dm_log dm_mod [last unloaded: speedstep_lib]
<4>
<4>Pid: 47546, comm: xfslogd/0 Tainted: P           ---------------    2.6.32-431.1.2.el6.x86_64 #1 Dell Inc. PowerEdge R820/0YWR73
<4>RIP: 0010:[<ffffffff81058fdd>]  [<ffffffff81058fdd>] task_rq_lock+0x4d/0xa0
<4>RSP: 0018:ffff88202754d9e0  EFLAGS: 00010082
<4>RAX: 00000000811422c9 RBX: 0000000000016840 RCX: 00000000000008c2
<4>RDX: 0000000000000082 RSI: ffff88202754da38 RDI: ffff8815d6db1540
<4>RBP: ffff88202754da00 R08: ffff8815d5012c88 R09: 00000000ffffffff
<4>R10: 0000000000000000 R11: 0000000000000000 R12: ffff8815d6db1540
<4>R13: ffff88202754da38 R14: 0000000000016840 R15: 000000000000000f
<4>FS:  0000000000000000(0000) GS:ffff880065800000(0000) knlGS:0000000000000000
<4>CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
<4>CR2: 000000038a60e5a8 CR3: 00000015d9f05000 CR4: 00000000000407f0
<4>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
<4>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
<4>Process xfslogd/0 (pid: 47546, threadinfo ffff88202754c000, task ffff88201f698080)
<4>Stack:
<4> ffff8815d6db1540 ffff8815d5ec6540 0000000000000000 0000000000000000
<4><d> ffff88202754da70 ffffffff81065a4c ffff88202754da30 000000008100bdae
<4><d> ffff88202754dab0 ffffea004c754b68 ffff88202754daa0 0000000000000082
<4>Call Trace:
<4> [<ffffffff81065a4c>] try_to_wake_up+0x3c/0x3e0
<4> [<ffffffff81065e45>] wake_up_process+0x15/0x20
<4> [<ffffffff8152965a>] __up+0x2a/0x40
<4> [<ffffffff810a12f2>] up+0x42/0x50
<4> [<ffffffffa06504a2>] xfs_buf_unlock+0x32/0x90 [xfs]
<4> [<ffffffffa061997f>] xfs_buf_item_unpin+0xcf/0x1a0 [xfs]
<4> [<ffffffffa064618c>] xfs_trans_committed_bulk+0x29c/0x2b0 [xfs]
<4> [<ffffffff81068ff5>] ? enqueue_entity+0x125/0x450
<4> [<ffffffff81060b13>] ? perf_event_task_sched_out+0x33/0x70
<4> [<ffffffff81068a53>] ? dequeue_entity+0x113/0x2e0
<4> [<ffffffffa063a26d>] xlog_cil_committed+0x3d/0x100 [xfs]
<4> [<ffffffffa063679d>] xlog_state_do_callback+0x15d/0x2b0 [xfs]
<4> [<ffffffffa061905e>] ? xfs_buf_iodone_callbacks+0x5e/0x200 [xfs]
<4> [<ffffffffa063696e>] xlog_state_done_syncing+0x7e/0xb0 [xfs]
<4> [<ffffffffa06370e9>] xlog_iodone+0x59/0xb0 [xfs]
<4> [<ffffffffa0651e30>] ? xfs_buf_iodone_work+0x0/0x50 [xfs]
<4> [<ffffffffa0651e56>] xfs_buf_iodone_work+0x26/0x50 [xfs]
<4> [<ffffffff81094d30>] worker_thread+0x170/0x2a0
<4> [<ffffffff8109b2b0>] ? autoremove_wake_function+0x0/0x40
<4> [<ffffffff81094bc0>] ? worker_thread+0x0/0x2a0
<4> [<ffffffff8109af06>] kthread+0x96/0xa0
<4> [<ffffffff8100c20a>] child_rip+0xa/0x20
<4> [<ffffffff8109ae70>] ? kthread+0x0/0xa0
<4> [<ffffffff8100c200>] ? child_rip+0x0/0x20
<4>Code: c3 40 68 01 00 49 89 fc 49 89 f5 9c 58 0f 1f 44 00 00 48 89 c2 fa 66 0f 1f 44 00 00 49 89 55 00 49 8b 44 24 08 49 89 de 8b 40 18 <4c> 03 34 c5 60 cf bf 81 4c 89 f7 e8 13 13 4d 00 49 8b 44 24 08 
<1>RIP  [<ffffffff81058fdd>] task_rq_lock+0x4d/0xa0
<4> RSP <ffff88202754d9e0>
<4>CR2: 000000038a60e5a8