Multiple soft lockup messages observed for netstat command when system has huge amount of ESTABLISHED-state TCP connections
Issue
- Multiple soft lockup messages observed for netstat command when system has huge amount of
ESTABLISHED-state
TCP connections. - The system has a huge amount of
ESTABLISHED-state
TCP connections:
$ wc -l sos_commands/networking/netstat_-neopa
145749 sos_commands/networking/netstat_-neopa
$ egrep -c ESTAB sos_commands/networking/netstat_-neopa
142924
- Backtrace looks like following snippet.
RIP: 0010:[<ffffffff8146f31d>] [<ffffffff8146f31d>] skb_release_data+0x3d/0x110
RSP: 0018:ffff880121a03948 EFLAGS: 00000257
RAX: ffff8834601f1140 RBX: ffff880121a03968 RCX: 00000000fffeffff
RDX: 0000000000010001 RSI: 0000000000000005 RDI: ffff883630985580
RBP: ffffffff81557483 R08: 0000000000000001 R09: 000000001c3d85d0
R10: 0000000000000000 R11: ffff8820517e0020 R12: ffff880121a038c0
R13: ffff883630985580 R14: ffff88207ffc9350 R15: ffffffff81558c15
FS: 00007f8e168ca7a0(0000) GS:ffff880121a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000003b14803020 CR3: 000000171b970000 CR4: 00000000001607f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Process netstat (pid: 55206, threadinfo ffff884e7bcac000, task ffff88604ee6c040)
Stack:
0000000000000004 ffff883630985580 0000000000000004 ffff883630985580
<d> ffff880121a03988 ffffffff8146eebe 0000000000000006 ffff8817541d00c0
<d> ffff880121a03a58 ffffffff814d024c ffff880121a03a08 0000000000000286
Call Trace:
<IRQ>
[<ffffffff8146f0ba>] ? skb_release_head_state+0x6a/0x110
[<ffffffff8146eeb6>] ? __kfree_skb+0x16/0xa0
[<ffffffff8146ef7b>] ? consume_skb+0x3b/0x80
[<ffffffff8147ccfd>] ? dev_kfree_skb_any+0x3d/0x50
[<ffffffffa021b64c>] ? ixgbe_poll+0xec/0x790 [ixgbe]
[<ffffffff814800e8>] ? netif_receive_skb+0x58/0x60
[<ffffffff81481d53>] ? net_rx_action+0x103/0x300
[<ffffffff810859a5>] ? __do_softirq+0xe5/0x230
[<ffffffff81557bfc>] ? call_softirq+0x1c/0x30
<EOI>
[<ffffffff8100e545>] ? do_softirq+0x65/0xa0
[<ffffffff810866c8>] ? local_bh_enable_ip+0x98/0xa0
[<ffffffff8154ee4b>] ? _spin_unlock_bh+0x1b/0x20
[<ffffffff814d908b>] ? established_get_next+0xbb/0x160
[<ffffffff814d93e0>] ? tcp_get_idx+0x90/0xa0
[<ffffffff814d9429>] ? tcp_seq_start+0x39/0x40
[<ffffffff811c07b6>] ? seq_read+0x96/0x400
[<ffffffff8120669e>] ? proc_reg_read+0x7e/0xc0
[<ffffffff8119aeb5>] ? vfs_read+0xb5/0x1a0
[<ffffffff8119bc76>] ? fget_light_pos+0x16/0x50
[<ffffffff8119b201>] ? sys_read+0x51/0xb0
[<ffffffff810eecee>] ? __audit_syscall_exit+0x25e/0x290
[<ffffffff815566d6>] ? system_call_fastpath+0x16/0x1b
Code: 0f 1f 84 00 00 00 00 00 55 48 89 e5 48 83 ec 10 48 89 1c 24 4c 89 64 24 08 0f 1f 44 00 00 48 8b 4f 10 f0 0f ba b1 70 06 00 00 00 <19> c0 85 c0 74 0f f0 0f ba a9 70 06 00 00 01 19 c0 85 c0 74 18
Environment
- Red Hat Enterprise Linux 6.9
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.