Soft lockup messages seen with RIP __inet_lookup_established in TCP receive path

Issue

• Routinely see soft lockup errors on our NetBackup server.
• RHEL6.4 rpciod or nfsiod soft lockup, RIP of __inet_lookup_established suggests stuck in tcp/ip stack
• Soft lockup messages seen with RIP __inet_lookup_established in receive TCP path.

Example panic trace:

BUG: soft lockup - CPU#1 stuck for 67s! [process:PID]

RIP: 0010:[<ffffffff81487e39>]  [<ffffffff81487e39>] __inet_lookup_established+0xd9/0x2a0

Call Trace:
 <IRQ> 
 [<ffffffff814a161e>] ? tcp_v4_rcv+0x1ce/0x8d0
 [<ffffffff8147f610>] ? ip_local_deliver_finish+0x0/0x2d0
 [<ffffffff8147f6ed>] ? ip_local_deliver_finish+0xdd/0x2d0
 [<ffffffff8147f978>] ? ip_local_deliver+0x98/0xa0
 [<ffffffff8147ee3d>] ? ip_rcv_finish+0x12d/0x440
 [<ffffffff8147f3c5>] ? ip_rcv+0x275/0x350
 [<ffffffff814e6e50>] ? vlan_hwaccel_do_receive+0x70/0x130
 [<ffffffff8144858b>] ? __netif_receive_skb+0x4ab/0x750
 [<ffffffff8144a968>] ? netif_receive_skb+0x58/0x60
 [<ffffffff8144aa70>] ? napi_skb_finish+0x50/0x70
 [<ffffffff814e75a4>] ? vlan_gro_receive+0x84/0xa0
 [<ffffffffa0150abb>] ? e1000_receive_skb+0x8b/0x90 [e1000]
 [<ffffffffa01555b2>] ? e1000_clean_rx_irq+0x2b2/0x5e0 [e1000]
 [<ffffffffa0152cd1>] ? e1000_clean+0x1f1/0xa50 [e1000]
 [<ffffffff81063516>] ? rebalance_domains+0x1a6/0x5a0
 [<ffffffff8102df6a>] ? smp_reschedule_interrupt+0x2a/0x30
 [<ffffffff8144d133>] ? net_rx_action+0x103/0x2f0
 [<ffffffff81076fb1>] ? __do_softirq+0xc1/0x1e0
 [<ffffffff8100c1cc>] ? call_softirq+0x1c/0x30
 [<ffffffff8100de05>] ? do_softirq+0x65/0xa0
 [<ffffffff81076d95>] ? irq_exit+0x85/0x90
 [<ffffffff815171c5>] ? do_IRQ+0x75/0xf0
 [<ffffffff8100b9d3>] ? ret_from_intr+0x0/0x11

Code: 66 90 48 8b 3a 40 f6 c7 01 48 89 f8 74 19 e9 ca 00 00 00 0f 1f 80 00 00 00 00 48 8b 3f 40 f6 c7 01 0f 85 b3 00 00 00 44 39 77 14 <48> 89 f9 75 ea 48 3b 5f 38 75 e4 4c 39 a7 58 02 00 00 75 db 44 
...
Kernel panic - not syncing: softlockup: hung tasks

Or alternately:

    [exception RIP: __inet_lookup_established+176]
 #7 [ffff880028203ae8] handle_edge_irq at ffffffff810e3e68
 #8 [ffff880028203b58] ip_local_deliver_finish at ffffffff8147fa6d
 #9 [ffff880028203b88] ip_local_deliver at ffffffff8147fcf8
#10 [ffff880028203bb8] ip_rcv_finish at ffffffff8147f1bd
#11 [ffff880028203bf8] ip_rcv at ffffffff8147f745
...