System crashes after a burst of "IPv6 duplicate address ... used by ... detected!" messages

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 8.6 and newer
  • IPv6 is enabled
  • primarily seen on cloned virtual machines

Issue

  • The system experiences a kernel panic after a burst of IPv6 duplicate address ... used by ... detected! messages.
  • What is CVE-2024-35969.
[594453.712295] IPv6: ens192: IPv6 duplicate address ... used by ... detected!
[594453.735815] ------------[ cut here ]------------
[594453.735824] kernel BUG at mm/slub.c:373!
[594453.735944] invalid opcode: 0000 [#1] SMP NOPTI
[594453.735985] CPU: 4 PID: 3069741 Comm: kworker/4:1 Kdump: loaded Not tainted 4.18.0-372.16.1.el8_6.x86_64 #1
[

Resolution

Red Hat Enterprise Linux 8.10.z

  • The issue has been resolved in the RHEL kernel version 4.18.0-553.8.1.el8_10 via Errata RHSA-2024:4211.
  • The issue was tracked at private JIRA RHEL-29783.

Red Hat Enterprise Linux 8.8.z

  • The issue has been resolved in the RHEL kernel version 4.18.0-477.74.1.el8_8 via Errata RHSA-2024:6993.
  • The issue was tracked at private JIRA RHEL-39011.

Red Hat Enterprise Linux 8.6.z AUS

  • The issue has been resolved in the RHEL kernel version 4.18.0-372.111.1.el8_6 via Errata RHSA-2024:4447.
  • The issue was tracked at private JIRA RHEL-39009.

Workaround

  • Preventing duplicate IPv6 addresses from occurring is expected to avoid triggering this issue. If the system is a cloned VM, the same stable-privacy criteria should not be in effect. eui64 may be used instead.

  • Alternatively, disabling IPv6 for the whole system can be performed, or IPv6 can be disabled for one interface only:

$ sudo nmcli connection modify eth0 ipv6.method "disabled"
$ sudo nmcli dev reapply eth0

Root Cause

  • The issue has been caused by a race condition between ipv6_get_ifaddr() and ipv6_del_addr() functions as described in the 7633c4da919a ("ipv6: fix race condition between ipv6_get_ifaddr and ipv6_del_addr") upstream commit.

  • Duplicate IPv6 addresses primarily occur on cloned VMs that have the same stable-privacy criteria that IPv6 uses to generate a unique link-local address. This triggers DAD to detect those and take action. Due to a bug in this mechanism, a memory corruption happens.

Diagnostic Steps

The following burst of messages is observed before the system crashes:

IPv6: eth0: IPv6 duplicate address fe80::dead:beef:dead:beef used by 00:xx:xx:xx:xx:xx detected!
IPv6: eth0: IPv6 duplicate address fe80::0000:aaaa:bbbb:0000 used by 00:yy:yy:yy:yy:yy detected!
IPv6: eth0: IPv6 duplicate address fe80::1111:2222:3333:4444 used by 00:zz:zz:zz:zz:zz detected!
IPv6: eth0: IPv6 duplicate address fe80::0101:0101:0101:0101 used by 00:aa:aa:aa:aa:aa detected!

The following message might be seen in the kernel log:

kernel BUG at mm/slub.c:373!

The following stack traces of the panic tasks may apply:

crash> bt
PID: 1285     TASK: ffff9a12062e0000  CPU: 6    COMMAND: "NetworkManager"
…
 #6 [ffffb9264472bb60] invalid_op at ffffffff85a00d64
    [exception RIP: set_freepointer]
…
 #7 [ffffb9264472bc10] kfree at ffffffff8530a05e
 #8 [ffffb9264472bc58] consume_skb at ffffffff857c4e77
 #9 [ffffb9264472bc70] skb_free_datagram at ffffffff857cbc31
#10 [ffffb9264472bc80] netlink_recvmsg at ffffffff85851469
#11 [ffffb9264472bd00] ____sys_recvmsg at ffffffff857b60b1
#12 [ffffb9264472bdc8] ___sys_recvmsg at ffffffff857b9f7b
#13 [ffffb9264472beb0] __sys_recvmsg at ffffffff857ba044
#14 [ffffb9264472bf38] do_syscall_64 at ffffffff8500430b
…

or:

crash> bt
PID: 518127   TASK: ffff8c628c7a4000  CPU: 6    COMMAND: "kworker/6:1"
…
 #6 [ffffb6ef84e0bd20] invalid_op at ffffffff9f600d64
    [exception RIP: kmem_cache_free_bulk+1941]
…
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffffb6ef84e0be30] kfree_rcu_work at ffffffff9ed6e6ca
 #8 [ffffb6ef84e0be98] process_one_work at ffffffff9ed0b547
 #9 [ffffb6ef84e0bed8] worker_thread at ffffffff9ed0bc00
#10 [ffffb6ef84e0bf10] kthread at ffffffff9ed12a2a
…

or:

crash> bt                                                              
PID: 0        TASK: ffff9b0581f10000  CPU: 10   COMMAND: "swapper/10"
…
 #7 [ffffacda41ba0910] page_fault at ffffffffb880114e                        
    [exception RIP: ipv6_get_ifaddr+0x54]                         
    RIP: ffffffffb8758fb4  RSP: ffffacda41ba09c0  RFLAGS: 00010286
    RAX: ffff9b05882f8200  RBX: ffff9b05a5efbc00  RCX: 0000000000000001
    RDX: ffff9b05874af000  RSI: ffff9b0582147a7e  RDI: ffffffffb9b30840
    RBP: ffff9b05874af000   R8: 0000000000000000   R9: 0000000000000003
    R10: 0000000000000000  R11: ff00000000000000  R12: ffff9b0582147a4e
    R13: ffff9b0582147a76  R14: ffff9b0582147a90  R15: ffff9b0582147a7e
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #8 [ffffacda41ba09c0] ndisc_recv_na at ffffffffb876f353
 #9 [ffffacda41ba0ab8] ndisc_rcv at ffffffffb8770888           
#10 [ffffacda41ba0ac8] icmpv6_rcv at ffffffffb8779648
#11 [ffffacda41ba0b08] ip6_protocol_deliver_rcu at ffffffffb87504bf                                            
#12 [ffffacda41ba0b50] ip6_input_finish at ffffffffb8750591              
#13 [ffffacda41ba0b58] ip6_input at ffffffffb8750644
#14 [ffffacda41ba0ba8] ip6_mc_input at ffffffffb8750738
#15 [ffffacda41ba0be0] ipv6_rcv at ffffffffb874fe11
#16 [ffffacda41ba0c48] __netif_receive_skb_core at ffffffffb8620290
#17 [ffffacda41ba0ce8] netif_receive_skb_internal at ffffffffb86209ad
#18 [ffffacda41ba0d10] napi_gro_receive at ffffffffb862145c
#19 [ffffacda41ba0d30] receive_buf at ffffffffc041acb4 [virtio_net]
#20 [ffffacda41ba0e00] virtnet_poll at ffffffffc041b834 [virtio_net]
#21 [ffffacda41ba0ea8] __napi_poll at ffffffffb8621d8d
#22 [ffffacda41ba0ed8] net_rx_action at ffffffffb8622252
#23 [ffffacda41ba0f58] __softirqentry_text_start at ffffffffb8a000dc
…

The exact stack trace may vary due to an unpredictable nature of memory corruption and user-after-free/double free bug.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments