kernel panic in __xfrm_state_delete ().

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 8.6
  • Red Hat Enterprise Linux 9.

Issue

  • kernel panic in __xfrm_state_delete () in RHEL8.6
[175887.177687] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
[175887.181611] PGD 22ca6f067 P4D 0 
[175887.183254] Oops: 0002 [#1] SMP NOPTI
[175887.185092] CPU: 1 PID: 3511882 Comm: pluto Kdump: loaded Tainted: P            E    --------- -  - 4.18.0-372.26.1.el8_6.x86_64 #1
[175887.191499] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008  12/07/2018
[175887.196258] RIP: 0010:__xfrm_state_delete+0xd2/0x1b0
[175887.198780] Code: 48 89 50 08 48 b8 00 02 00 00 00 00 ad de 8b 93 b4 00 00 00 48 89 43 20 85 d2 74 2b 48 8b 83 58 03 00 00 48 8b 93 60 03 00 00 <48> 89 02 48 85 c0 74 04 48 89 50 08 48 b8 00 02 00 00 00 00 a
d de
[175887.207763] RSP: 0018:ff691fda48ac3a78 EFLAGS: 00010206
[175887.210463] RAX: 0000000000000000 RBX: ff27f9309cbb8700 RCX: dead000000000200
[175887.214266] RDX: 0000000000000000 RSI: ff27f9309cbb87a0 RDI: ff27f9309cbb87a0
[175887.217737] RBP: ffffffff94b31100 R08: 0000000000000001 R09: 0000000000000002
[175887.221124] R10: 0000000000000002 R11: 0000000000000032 R12: ffffffff94b2f8c0
[175887.224600] R13: ffffffff93d00710 R14: ff691fda48ac3c60 R15: 0000000000000000
[175887.228067] FS:  00007fa6082e8740(0000) GS:ff27f9379fa40000(0000) knlGS:0000000000000000
[175887.231992] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[175887.234586] CR2: 0000000000000000 CR3: 000000017c9e4004 CR4: 0000000000371ee0
[175887.237564] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[175887.240626] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[175887.243851] Call Trace:
[175887.244928]  xfrm_state_delete+0x1e/0x30
[175887.246597]  xfrm_del_sa+0xb8/0x100
[175887.248161]  xfrm_user_rcv_msg+0x133/0x1e0
[175887.249902]  ? hist_field_none+0x10/0x10
[175887.251556]  ? event_hist_trigger+0x347/0x410
[175887.253423]  ? bpf_lsm_socket_shutdown+0x10/0x10
[175887.255384]  ? copy_to_user_state_extra+0x3f0/0x3f0
[175887.257459]  netlink_rcv_skb+0x4c/0x120
[175887.259120]  xfrm_netlink_rcv+0x30/0x40
[175887.260735]  netlink_unicast+0x196/0x230
[175887.262393]  netlink_sendmsg+0x204/0x3d0
[175887.264040]  sock_sendmsg+0x4c/0x50
[175887.265521]  sock_write_iter+0x97/0x100
[175887.267157]  new_sync_write+0x112/0x160
[175887.268796]  vfs_write+0xa5/0x1a0
[175887.270462]  ksys_write+0x4f/0xb0

Resolution

  • The issue is worked by following private Red Hat bugs

    • RHEL8 - Fix is released in kernel-4.18.0-477.10.1.el8_8.x86_64 with RHSA-2023:2951
    • RHEL9 - Fix is released in kernel-5.14.0-284.11.1.el9_2.x86_64 with RHSA-2023:2458

  • Kindly open support ticket to get more information on the bug.

Root Cause

  • The issue was introduced in RHEL8.6 by upstream commit fe9f1d8779cb ("xfrm: add state hashtable keyed by seq").
  • This kernel panic is fixed upstream via following commit.
commit b97df039a68b2f3e848e238df5d5d06343ea497b
Author: Thomas Jarosch 
Date:   Wed Nov 2 11:18:48 2022 +0100

    xfrm: Fix oops in __xfrm_state_delete()

    Kernel 5.14 added a new "byseq" index to speed
    up xfrm_state lookups by sequence number in commit
    fe9f1d8779cb ("xfrm: add state hashtable keyed by seq")

    While the patch was thorough, the function pfkey_send_new_mapping()
    in net/af_key.c also modifies x->km.seq and never added
    the current xfrm_state to the "byseq" index.

    This leads to the following kernel Ooops:
        BUG: kernel NULL pointer dereference, address: 0000000000000000
        ..
        RIP: 0010:__xfrm_state_delete+0xc9/0x1c0
        ..
        Call Trace:
        <TASK>
        xfrm_state_delete+0x1e/0x40
        xfrm_del_sa+0xb0/0x110 [xfrm_user]
        xfrm_user_rcv_msg+0x12d/0x270 [xfrm_user]
        ? remove_entity_load_avg+0x8a/0xa0
        ? copy_to_user_state_extra+0x580/0x580 [xfrm_user]
        netlink_rcv_skb+0x51/0x100
        xfrm_netlink_rcv+0x30/0x50 [xfrm_user]
        netlink_unicast+0x1a6/0x270
        netlink_sendmsg+0x22a/0x480
        __sys_sendto+0x1a6/0x1c0
        ? __audit_syscall_entry+0xd8/0x130
        ? __audit_syscall_exit+0x249/0x2b0
        __x64_sys_sendto+0x23/0x30
        do_syscall_64+0x3a/0x90
        entry_SYSCALL_64_after_hwframe+0x61/0xcb

Diagnostic Steps

  • Following is snippet from vmcore of this issue.
crash> bt
PID: 3511882  TASK: ff27f9303a9b8000  CPU: 1    COMMAND: "pluto"
 #0 [ff691fda48ac3730] machine_kexec at ffffffff92e6564e
 #1 [ff691fda48ac3788] __crash_kexec at ffffffff92fa576d
 #2 [ff691fda48ac3850] panic at ffffffff92eed98b
 #3 [ff691fda48ac38d0] oops_end.cold.10 at ffffffff92e2660f
 #4 [ff691fda48ac38f0] no_context at ffffffff92e763bf
 #5 [ff691fda48ac3948] __bad_area_nosemaphore at ffffffff92e7671c
 #6 [ff691fda48ac3990] do_page_fault at ffffffff92e76fb7
 #7 [ff691fda48ac39c0] page_fault at ffffffff9380111e
    [exception RIP: __xfrm_state_delete+210]
    RIP: ffffffff936edbd2  RSP: ff691fda48ac3a78  RFLAGS: 00010206
    RAX: 0000000000000000  RBX: ff27f9309cbb8700  RCX: dead000000000200
    RDX: 0000000000000000  RSI: ff27f9309cbb87a0  RDI: ff27f9309cbb87a0
    RBP: ffffffff94b31100   R8: 0000000000000001   R9: 0000000000000002
    R10: 0000000000000002  R11: 0000000000000032  R12: ffffffff94b2f8c0
    R13: ffffffff93d00710  R14: ff691fda48ac3c60  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #8 [ff691fda48ac3a90] xfrm_state_delete at ffffffff936edcce
 #9 [ff691fda48ac3aa8] xfrm_del_sa at ffffffff936fb298
#10 [ff691fda48ac3af0] xfrm_user_rcv_msg at ffffffff936f8943
#11 [ff691fda48ac3c58] netlink_rcv_skb at ffffffff9365592c
#12 [ff691fda48ac3ca8] xfrm_netlink_rcv at ffffffff936f7370
#13 [ff691fda48ac3cc0] netlink_unicast at ffffffff936550f6
#14 [ff691fda48ac3d00] netlink_sendmsg at ffffffff93655394
#15 [ff691fda48ac3d70] sock_sendmsg at ffffffff935b853c
#16 [ff691fda48ac3d88] sock_write_iter at ffffffff935b85d7
#17 [ff691fda48ac3df8] new_sync_write at ffffffff9313d4f2
#18 [ff691fda48ac3e80] vfs_write at ffffffff93140be5
#19 [ff691fda48ac3eb0] ksys_write at ffffffff93140e5f
#20 [ff691fda48ac3ee8] unload_network_ops_symbols at ffffffffc0a48f78 [falcon_lsm_pinned_14306]
#21 [ff691fda48ac3f38] do_syscall_64 at ffffffff92e0430b
#22 [ff691fda48ac3f50] entry_SYSCALL_64_after_hwframe at ffffffff938000ad
    RIP: 00007fa6069b2a07  RSP: 00007ffed0b0d5f0  RFLAGS: 00000293
    RAX: ffffffffffffffda  RBX: 000000000000000d  RCX: 00007fa6069b2a07
    RDX: 0000000000000028  RSI: 00007ffed0b0dac0  RDI: 000000000000000d
    RBP: 00007ffed0b0dac0   R8: 0000000000000000   R9: 00007ffed0b0dac0
    R10: 00007ffed0b0dd77  R11: 0000000000000293  R12: 0000000000000028
    R13: 00007ffed0b0dfd0  R14: 00007ffed0b0dfa0  R15: 0000559033bc4398
    ORIG_RAX: 0000000000000001  CS: 0033  SS: 002b

crash> sys|grep PANIC
       PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000000"

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments