Repeated occurrences of list_add corruption or double addition detected, leading to a system crash

Solution Unverified - Updated -

Issue

  • Repeated occurrences of list_add corruption or double addition detected, leading to a system crash:
------------[ cut here ]------------
WARNING: CPU: 80 PID: 0 at lib/list_debug.c:29 __list_add+0x65/0xc0
list_add corruption. next->prev should be prev (ffff9aadbf853c38), but was ffff9aadbf854048. (next=ffff9a458168c210).
Modules linked in: ...
CPU: 80 PID: 0 Comm: swapper/80 Kdump: loaded Tainted: P        W  OE  ------------   3.10.0-1160.42.2.el7.x86_64 #1
Hardware name: HP Superdome2 16s x86, BIOS Bundle: 008.008.034 SFW: 045.018.000 10/01/2019
Call Trace:
 <IRQ>  [<ffffffff94183539>] dump_stack+0x19/0x1b
 [<ffffffff93a9b278>] __warn+0xd8/0x100
 [<ffffffff93a9b2ff>] warn_slowpath_fmt+0x5f/0x80
 [<ffffffff93da66c5>] __list_add+0x65/0xc0
 [<ffffffff93aad6d7>] __internal_add_timer+0xc7/0x130
 [<ffffffff93aada45>] internal_add_timer+0x45/0xe0
 [<ffffffff93aaed80>] mod_timer+0x100/0x230
 [<ffffffff9403ae68>] sk_reset_timer+0x18/0x30
 [<ffffffff940bb5d2>] tcp_rearm_rto+0x82/0x110
 [<ffffffff940bba8c>] tcp_ack+0x42c/0x12f0
 [<ffffffff9403e6f7>] ? kfree_skbmem+0x37/0x90
 [<ffffffff9403f78a>] ? __kfree_skb+0x1a/0x20
 [<ffffffff940b88f8>] ? tcp_data_queue+0x248/0xdc0
 [<ffffffff940bcf26>] tcp_rcv_established+0x1d6/0x7a0
 [<ffffffff940c800a>] tcp_v4_do_rcv+0x10a/0x350
 [<ffffffff940c985c>] tcp_v4_rcv+0x7dc/0x9e0
 [<ffffffff940a1a9d>] ip_local_deliver_finish+0xbd/0x200
 [<ffffffff940a1d90>] ip_local_deliver+0x60/0xe0
 [<ffffffff940c9032>] ? tcp_v4_early_demux+0x112/0x160
 [<ffffffff940a1700>] ip_rcv_finish+0x90/0x370
 [<ffffffff940a20d0>] ip_rcv+0x2c0/0x420
 [<ffffffffc0543e98>] ? bond_handle_frame+0x78/0x290 [bonding]
 [<ffffffff94056829>] __netif_receive_skb_core+0x729/0xa10
 [<ffffffff94056b28>] __netif_receive_skb+0x18/0x60
 [<ffffffff94056bb0>] netif_receive_skb_internal+0x40/0xc0
 [<ffffffff940579eb>] napi_gro_frags+0x18b/0x1f0
 [<ffffffffc028e8e4>] be_poll+0x7b4/0xb40 [be2net]
 [<ffffffff93ad2c60>] ? task_rq_unlock+0x20/0x20
 [<ffffffff940571cf>] net_rx_action+0x26f/0x390
 [<ffffffff93aa4bf5>] __do_softirq+0xf5/0x280
 [<ffffffff941994ec>] call_softirq+0x1c/0x30
 [<ffffffff93a2f715>] do_softirq+0x65/0xa0
 [<ffffffff93aa4f75>] irq_exit+0x105/0x110
 [<ffffffff9419a8d6>] do_IRQ+0x56/0xf0
 [<ffffffff9418c36a>] common_interrupt+0x16a/0x16a
 <EOI>  [<ffffffff93fc8da7>] ? cpuidle_enter_state+0x57/0xd0
 [<ffffffff93fc8efe>] cpuidle_idle_call+0xde/0x230
 [<ffffffff93a37c8e>] arch_cpu_idle+0xe/0xc0
 [<ffffffff93b0181a>] cpu_startup_entry+0x14a/0x1e0
 [<ffffffff93a5a827>] start_secondary+0x1f7/0x270
 [<ffffffff93a000d5>] start_cpu+0x5/0x14
---[ end trace 83a5f002137f2175 ]---
------------[ cut here ]------------
WARNING: CPU: 73 PID: 144033 at lib/list_debug.c:29 __list_add+0x65/0xc0
list_add corruption. next->prev should be prev (ffff9aadbf854048), but was dead000000000200. (next=ffff9a458168c210).
Modules linked in: ...
CPU: 73 PID: 144033 Comm: oracle_144033_m Kdump: loaded Tainted: P        W  OE  ------------   3.10.0-1160.42.2.el7.x86_64 #1
Hardware name: HP Superdome2 16s x86, BIOS Bundle: 008.008.034 SFW: 045.018.000 10/01/2019
Call Trace:
 <IRQ>  [<ffffffff94183539>] dump_stack+0x19/0x1b
 [<ffffffff93a9b278>] __warn+0xd8/0x100
 [<ffffffff93a9b2ff>] warn_slowpath_fmt+0x5f/0x80
 [<ffffffff93da66c5>] __list_add+0x65/0xc0
 [<ffffffff93aad6d7>] __internal_add_timer+0xc7/0x130
 [<ffffffff93aada45>] internal_add_timer+0x45/0xe0
 [<ffffffff93aaed80>] mod_timer+0x100/0x230
 [<ffffffff9403ae68>] sk_reset_timer+0x18/0x30
 [<ffffffff940bffe5>] tcp_schedule_loss_probe+0x145/0x1e0
 [<ffffffff940c0f43>] tcp_write_xmit+0x353/0xd40
 [<ffffffff940c1bbe>] __tcp_push_pending_frames+0x2e/0xc0
 [<ffffffff940bcf95>] tcp_rcv_established+0x245/0x7a0
 [<ffffffff940c800a>] tcp_v4_do_rcv+0x10a/0x350
 [<ffffffff940c985c>] tcp_v4_rcv+0x7dc/0x9e0
 [<ffffffff940a1a9d>] ip_local_deliver_finish+0xbd/0x200
 [<ffffffff940a1d90>] ip_local_deliver+0x60/0xe0
 [<ffffffff940c9032>] ? tcp_v4_early_demux+0x112/0x160
 [<ffffffff940a1700>] ip_rcv_finish+0x90/0x370
 [<ffffffff940a20d0>] ip_rcv+0x2c0/0x420
 [<ffffffffc0543e98>] ? bond_handle_frame+0x78/0x290 [bonding]
 [<ffffffff94056829>] __netif_receive_skb_core+0x729/0xa10
 [<ffffffff94056b28>] __netif_receive_skb+0x18/0x60
 [<ffffffff94056bb0>] netif_receive_skb_internal+0x40/0xc0
 [<ffffffff940579eb>] napi_gro_frags+0x18b/0x1f0
 [<ffffffffc028e8e4>] be_poll+0x7b4/0xb40 [be2net]
 [<ffffffff940571cf>] net_rx_action+0x26f/0x390
 [<ffffffff93aa4bf5>] __do_softirq+0xf5/0x280
 [<ffffffff941994ec>] call_softirq+0x1c/0x30
 [<ffffffff93a2f715>] do_softirq+0x65/0xa0
 [<ffffffff93aa4f75>] irq_exit+0x105/0x110
 [<ffffffff9419a8d6>] do_IRQ+0x56/0xf0
 [<ffffffff9418c36a>] common_interrupt+0x16a/0x16a
 <EOI> 
---[ end trace 83a5f002137f2176 ]---
------------[ cut here ]------------
WARNING: CPU: 73 PID: 144033 at lib/list_debug.c:36 __list_add+0x8a/0xc0
list_add double add: new=ffff9a458168c210, prev=ffff9aadbf854048, next=ffff9a458168c210.
Modules linked in: ...
CPU: 73 PID: 144033 Comm: oracle_144033_m Kdump: loaded Tainted: P        W  OE  ------------   3.10.0-1160.42.2.el7.x86_64 #1
Hardware name: HP Superdome2 16s x86, BIOS Bundle: 008.008.034 SFW: 045.018.000 10/01/2019
Call Trace:
 <IRQ>  [<ffffffff94183539>] dump_stack+0x19/0x1b
 [<ffffffff93a9b278>] __warn+0xd8/0x100
 [<ffffffff93a9b2ff>] warn_slowpath_fmt+0x5f/0x80
 [<ffffffff93da66ea>] __list_add+0x8a/0xc0
 [<ffffffff93aad6d7>] __internal_add_timer+0xc7/0x130
 [<ffffffff93aada45>] internal_add_timer+0x45/0xe0
 [<ffffffff93aaed80>] mod_timer+0x100/0x230
 [<ffffffff9403ae68>] sk_reset_timer+0x18/0x30
 [<ffffffff940bffe5>] tcp_schedule_loss_probe+0x145/0x1e0
 [<ffffffff940c0f43>] tcp_write_xmit+0x353/0xd40
 [<ffffffff940c1bbe>] __tcp_push_pending_frames+0x2e/0xc0
 [<ffffffff940bcf95>] tcp_rcv_established+0x245/0x7a0
 [<ffffffff940c800a>] tcp_v4_do_rcv+0x10a/0x350
 [<ffffffff940c985c>] tcp_v4_rcv+0x7dc/0x9e0
 [<ffffffff940a1a9d>] ip_local_deliver_finish+0xbd/0x200
 [<ffffffff940a1d90>] ip_local_deliver+0x60/0xe0
 [<ffffffff940c9032>] ? tcp_v4_early_demux+0x112/0x160
 [<ffffffff940a1700>] ip_rcv_finish+0x90/0x370
 [<ffffffff940a20d0>] ip_rcv+0x2c0/0x420
 [<ffffffffc0543e98>] ? bond_handle_frame+0x78/0x290 [bonding]
 [<ffffffff94056829>] __netif_receive_skb_core+0x729/0xa10
 [<ffffffff94056b28>] __netif_receive_skb+0x18/0x60
 [<ffffffff94056bb0>] netif_receive_skb_internal+0x40/0xc0
 [<ffffffff940579eb>] napi_gro_frags+0x18b/0x1f0
 [<ffffffffc028e8e4>] be_poll+0x7b4/0xb40 [be2net]
 [<ffffffff940571cf>] net_rx_action+0x26f/0x390
 [<ffffffff93aa4bf5>] __do_softirq+0xf5/0x280
 [<ffffffff941994ec>] call_softirq+0x1c/0x30
 [<ffffffff93a2f715>] do_softirq+0x65/0xa0
 [<ffffffff93aa4f75>] irq_exit+0x105/0x110
 [<ffffffff9419a8d6>] do_IRQ+0x56/0xf0
 [<ffffffff9418c36a>] common_interrupt+0x16a/0x16a
 <EOI> 
---[ end trace 83a5f002137f2177 ]---
kernel tried to execute NX-protected page - exploit attempt? (uid: 1020)
BUG: unable to handle kernel paging request at ffff9aadbf853d58
IP: [<ffff9aadbf853d58>] 0xffff9aadbf853d58
PGD 4bfffff7067 PUD 33cfeadb063 PMD 8000033cff8000e3 
Oops: 0011 [#1] SMP 
Modules linked in: ...
CPU: 73 PID: 126472 Comm: ora_lms8_sid Kdump: loaded Tainted: P        W  OE  ------------   3.10.0-1160.42.2.el7.x86_64 #1
Hardware name: HP Superdome2 16s x86, BIOS Bundle: 008.008.034 SFW: 045.018.000 10/01/2019
task: ffff9a4559430000 ti: ffff9a44f9ad8000 task.ti: ffff9a44f9ad8000
RIP: 0010:[<ffff9aadbf853d58>]  [<ffff9aadbf853d58>] 0xffff9aadbf853d58
RSP: 0018:ffff9aadbf843ed0  EFLAGS: 00010246
RAX: ffff9a44f9adbfd8 RBX: ffff9aadbf853940 RCX: ffff9aadbf853d58
RDX: ffff9aadbf853d58 RSI: ffff9aadbf853d58 RDI: ffff9aadbf853d58
RBP: ffff9aadbf843f00 R08: ffff9aadbf853d78 R09: 0000000000000000
R10: 0000000000000003 R11: ffff9a452406b800 R12: ffff9aadbf853d38
R13: 0000000000000101 R14: ffff9aadbf853d58 R15: ffff9aadbf853d58
FS:  00007f46c6c15700(0000) GS:ffff9aadbf840000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffff9aadbf853d58 CR3: 0000004f16348000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <IRQ> 
 [<ffffffff93aabdb8>] ? call_timer_fn+0x38/0x110
 [<ffffffff93aae3cd>] run_timer_softirq+0x25d/0x340
 [<ffffffff93aa4bf5>] __do_softirq+0xf5/0x280
 [<ffffffff941994ec>] call_softirq+0x1c/0x30
 <EOI> 
 [<ffffffff93a2f715>] do_softirq+0x65/0xa0
 [<ffffffff93aa404b>] __local_bh_enable_ip+0x9b/0xb0
 [<ffffffff93aa4077>] local_bh_enable+0x17/0x20
 [<ffffffff940a7874>] ip_finish_output+0x284/0x8d0
 [<ffffffff940a81eb>] ip_output+0x7b/0xf0
 [<ffffffff940a8dee>] ? ip_make_skb+0xfe/0x120
 [<ffffffff940a5c17>] ip_local_out_sk+0x37/0x40
 [<ffffffff940a8c56>] ip_send_skb+0x16/0x50
 [<ffffffff940d10bc>] udp_send_skb+0xac/0x2b0
 [<ffffffff940a5950>] ? ip_copy_metadata+0x170/0x170
 [<ffffffff940d2621>] udp_sendmsg+0x5e1/0xa70
 [<ffffffff940d3b2d>] ? udp_recvmsg+0x14d/0x480
 [<ffffffff940dfe09>] inet_sendmsg+0x69/0xb0
 [<ffffffff940dffb0>] ? inet_recvmsg+0x80/0xb0
 [<ffffffff940363a6>] sock_sendmsg+0xb6/0xf0
 [<ffffffff940364f5>] ? sock_recvmsg+0xc5/0x100
 [<ffffffff93a7dfe9>] ? switch_mm_irqs_off+0x109/0x290
 [<ffffffff93a7dff5>] ? switch_mm_irqs_off+0x115/0x290
 [<ffffffff93a7dfe9>] ? switch_mm_irqs_off+0x109/0x290
 [<ffffffff93a7dff5>] ? switch_mm_irqs_off+0x115/0x290
 [<ffffffff93a7dfe9>] ? switch_mm_irqs_off+0x109/0x290
 [<ffffffff93a7dff5>] ? switch_mm_irqs_off+0x115/0x290
 [<ffffffff93a7dfe9>] ? switch_mm_irqs_off+0x109/0x290
 [<ffffffff94037269>] ___sys_sendmsg+0x3e9/0x400
 [<ffffffff93b3b1ea>] ? audit_comparator+0x2a/0xa0
 [<ffffffff93b3d527>] ? audit_filter_rules.isra.10+0x787/0xf90
 [<ffffffff93c9d663>] ? ep_poll+0x123/0x360
 [<ffffffff94038921>] __sys_sendmsg+0x51/0x90
 [<ffffffff94038972>] SyS_sendmsg+0x12/0x20
 [<ffffffff94195f92>] system_call_fastpath+0x25/0x2a
Code: 9a ff ff 28 3d 85 bf ad 9a ff ff 00 00 00 00 00 00 00 00 00 02 00 00 00 00 ad de 48 3d 85 bf ad 9a ff ff 48 3d 85 bf ad 9a ff ff <58> 3d 85 bf ad 9a ff ff 58 3d 85 bf ad 9a ff ff 68 3d 85 bf ad 
RIP  [<ffff9aadbf853d58>] 0xffff9aadbf853d58
 RSP <ffff9aadbf843ed0>
CR2: ffff9aadbf853d58

Environment

  • Red Hat Enterprise Linux 7.9.z

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content