Multiple soft lockup messages observed for netstat command when system has huge amount of ESTABLISHED-state TCP connections

Issue

• Multiple soft lockup messages observed for netstat command when system has huge amount of ESTABLISHED-state TCP connections.
• The system has a huge amount of ESTABLISHED-state TCP connections:

 $ wc -l sos_commands/networking/netstat_-neopa
 145749 sos_commands/networking/netstat_-neopa

 $ egrep -c ESTAB sos_commands/networking/netstat_-neopa
 142924

• Backtrace looks like following snippet.

 RIP: 0010:[<ffffffff8146f31d>]  [<ffffffff8146f31d>] skb_release_data+0x3d/0x110
 RSP: 0018:ffff880121a03948  EFLAGS: 00000257
 RAX: ffff8834601f1140 RBX: ffff880121a03968 RCX: 00000000fffeffff
 RDX: 0000000000010001 RSI: 0000000000000005 RDI: ffff883630985580
 RBP: ffffffff81557483 R08: 0000000000000001 R09: 000000001c3d85d0
 R10: 0000000000000000 R11: ffff8820517e0020 R12: ffff880121a038c0
 R13: ffff883630985580 R14: ffff88207ffc9350 R15: ffffffff81558c15
 FS:  00007f8e168ca7a0(0000) GS:ffff880121a00000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 0000003b14803020 CR3: 000000171b970000 CR4: 00000000001607f0
 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
 Process netstat (pid: 55206, threadinfo ffff884e7bcac000, task ffff88604ee6c040)
 Stack:
 0000000000000004 ffff883630985580 0000000000000004 ffff883630985580
 <d> ffff880121a03988 ffffffff8146eebe 0000000000000006 ffff8817541d00c0
 <d> ffff880121a03a58 ffffffff814d024c ffff880121a03a08 0000000000000286
 Call Trace:
 <IRQ>
 [<ffffffff8146f0ba>] ? skb_release_head_state+0x6a/0x110
 [<ffffffff8146eeb6>] ? __kfree_skb+0x16/0xa0
 [<ffffffff8146ef7b>] ? consume_skb+0x3b/0x80
 [<ffffffff8147ccfd>] ? dev_kfree_skb_any+0x3d/0x50
 [<ffffffffa021b64c>] ? ixgbe_poll+0xec/0x790 [ixgbe]
 [<ffffffff814800e8>] ? netif_receive_skb+0x58/0x60
 [<ffffffff81481d53>] ? net_rx_action+0x103/0x300
 [<ffffffff810859a5>] ? __do_softirq+0xe5/0x230
 [<ffffffff81557bfc>] ? call_softirq+0x1c/0x30
 <EOI>
 [<ffffffff8100e545>] ? do_softirq+0x65/0xa0
 [<ffffffff810866c8>] ? local_bh_enable_ip+0x98/0xa0
 [<ffffffff8154ee4b>] ? _spin_unlock_bh+0x1b/0x20
 [<ffffffff814d908b>] ? established_get_next+0xbb/0x160
 [<ffffffff814d93e0>] ? tcp_get_idx+0x90/0xa0
 [<ffffffff814d9429>] ? tcp_seq_start+0x39/0x40
 [<ffffffff811c07b6>] ? seq_read+0x96/0x400
 [<ffffffff8120669e>] ? proc_reg_read+0x7e/0xc0
 [<ffffffff8119aeb5>] ? vfs_read+0xb5/0x1a0
 [<ffffffff8119bc76>] ? fget_light_pos+0x16/0x50
 [<ffffffff8119b201>] ? sys_read+0x51/0xb0
 [<ffffffff810eecee>] ? __audit_syscall_exit+0x25e/0x290
 [<ffffffff815566d6>] ? system_call_fastpath+0x16/0x1b
 Code: 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 10 48 89 1c 24 4c 89 64 24 08 0f 1f 44 00 00 48 8b 4f 10 f0 0f ba b1 70 06 00 00 00 <19> c0 85 c0 74 0f f0 0f ba a9 70 06 00 00 01 19 c0 85 c0 74 18