RHEL7: kernel panic at br_handle_frame_finish+0x3e

Diagnostic Steps

• Vmcore analysis:

     KERNEL: /cores/retrace/repos/kernel/x86_64/usr/lib/debug/lib/modules/3.10.0-1127.el7.x86_64/vmlinux
    DUMPFILE: /cores/retrace/tasks/624011979/crash/vmcore  [PARTIAL DUMP]
        CPUS: 12
        DATE: Mon Apr  5 05:33:27 KST 2021
      UPTIME: 199 days, 14:10:48
LOAD AVERAGE: 0.13, 0.18, 0.20
       TASKS: 1547
     RELEASE: 3.10.0-1127.el7.x86_64
     VERSION: #1 SMP Tue Feb 18 16:39:12 EST 2020
     MACHINE: x86_64  (2593 Mhz)
      MEMORY: 96 GB
       PANIC: "general protection fault: 0000 [#1] SMP "
         PID: 26219
     COMMAND: "weaver"
        TASK: ffff91230b22c1c0  [THREAD_INFO: ffff91230b35c000]
         CPU: 6
       STATE: TASK_RUNNING (PANIC)

        DMI_BIOS_VENDOR: VMware, Inc.
       DMI_BIOS_VERSION: VMW71.00V.0.B64.1809080740
          DMI_BIOS_DATE: 09/08/2018

[17244377.759437] weave: port 14(vethwepl0971c9f) entered disabled state
[17244378.577697] general protection fault: 0000 [#1] SMP 
[17244378.577725] Modules linked in: ip_set_list_set ip6table_nat ip6_tables ipt_REJECT nf_reject_ipv4 xt_physdev xt_NFLOG nfnetlink_log veth dummy vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_defrag_ipv6 ip_set_hash_ip xt_set ip_set xt_nat xt_statistic ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_mangle xt_comment xt_mark xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache iptable_filter nls_utf8 isofs loop vmw_vsock_vmci_transport vsock sunrpc vfat fat iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
[17244378.577989]  ppdev ablk_helper cryptd vmw_balloon joydev pcspkr sg nfit vmw_vmci i2c_piix4 libnvdimm parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
[17244378.578138] CPU: 6 PID: 26219 Comm: weaver Kdump: loaded Tainted: G               ------------ T 3.10.0-1127.el7.x86_64 #1
[17244378.578167] Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.0.B64.1809080740 09/08/2018
[17244378.579172] task: ffff91230b22c1c0 ti: ffff91230b35c000 task.ti: ffff91230b35c000
[17244378.579191] RIP: 0010:[<ffffffffc0644fde>]  [<ffffffffc0644fde>] br_handle_frame_finish+0x3e/0x580 [bridge]
[17244378.579224] RSP: 0018:ffff91237fc03818  EFLAGS: 00010202
[17244378.579239] RAX: ffff911dcdfcb000 RBX: ffff91232ed5aef8 RCX: 0000000000000000
[17244378.579257] RDX: 00089336e5f11642 RSI: ffff91232ed5aef8 RDI: 0000000000000000
[17244378.579275] RBP: ffff91237fc03880 R08: 0000000000000000 R09: 0000000000000034
[17244378.579293] R10: ffffffff83315bc0 R11: ffffffff82c7ff90 R12: ffff91232ed5aef8
[17244378.579315] R13: ffff912346a92000 R14: 0000000000000042 R15: 2f65336162363561
[17244378.579334] FS:  00007f4acbfff700(0000) GS:ffff91237fc00000(0000) knlGS:0000000000000000
[17244378.579354] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[17244378.579369] CR2: 00007fcd0abb9ae0 CR3: 0000000bcf496000 CR4: 00000000007607e0
[17244378.579433] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[17244378.579451] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[17244378.579469] PKRU: 55555554
[17244378.579476] Call Trace:
[17244378.581126]  <IRQ> 
[17244378.581140]  [<ffffffff82ca3ea4>] ? ip_finish_output+0x284/0x8d0
[17244378.582399]  [<ffffffffc06100ac>] br_nf_dev_xmit+0x5c/0x70 [br_netfilter]
[17244378.583033]  [<ffffffffc0640ad2>] br_dev_xmit+0x52/0x2d0 [bridge]
[17244378.583656]  [<ffffffff82c53386>] dev_hard_start_xmit+0x246/0x3b0
[17244378.584268]  [<ffffffff82c56389>] __dev_queue_xmit+0x519/0x650
[17244378.584885]  [<ffffffff82c564d0>] dev_queue_xmit+0x10/0x20
[17244378.585513]  [<ffffffff82c6312d>] neigh_resolve_output+0x11d/0x220
[17244378.586138]  [<ffffffff82c65818>] neigh_update+0x488/0x650
[17244378.586781]  [<ffffffff82cd41c0>] arp_process+0x310/0x8e0
[17244378.587417]  [<ffffffff82c53fa0>] ? netif_receive_skb_internal+0x40/0xc0
[17244378.588065]  [<ffffffffc0644e29>] ? br_netif_receive_skb+0x49/0x60 [bridge]
[17244378.588721]  [<ffffffffc0644f07>] ? br_pass_frame_up+0xc7/0x160 [bridge]
[17244378.589383]  [<ffffffff82cd48dd>] arp_rcv+0x12d/0x1b0
[17244378.590042]  [<ffffffff82c53c19>] __netif_receive_skb_core+0x729/0xa10
[17244378.590712]  [<ffffffff82659000>] ? cpuid_read+0x90/0x120
[17244378.591398]  [<ffffffff82c53f18>] __netif_receive_skb+0x18/0x60
[17244378.592078]  [<ffffffff82c53fa0>] netif_receive_skb_internal+0x40/0xc0
[17244378.592741]  [<ffffffff82c5403c>] netif_receive_skb+0x1c/0x70
[17244378.593399]  [<ffffffffc0644e08>] br_netif_receive_skb+0x28/0x60 [bridge]
[17244378.594032]  [<ffffffffc0644f07>] br_pass_frame_up+0xc7/0x160 [bridge]
[17244378.594651]  [<ffffffffc064245a>] ? br_fdb_update+0xca/0x220 [bridge]
[17244378.595252]  [<ffffffffc06451c1>] br_handle_frame_finish+0x221/0x580 [bridge]
[17244378.595847]  [<ffffffffc06456f9>] br_handle_frame+0x1d9/0x330 [bridge]
[17244378.596432]  [<ffffffffc0644fa0>] ? br_pass_frame_up+0x160/0x160 [bridge]
[17244378.596992]  [<ffffffff82c536ea>] __netif_receive_skb_core+0x1fa/0xa10
[17244378.597537]  [<ffffffff82c53f18>] __netif_receive_skb+0x18/0x60
[17244378.598062]  [<ffffffff82c54eee>] process_backlog+0xae/0x180
[17244378.598571]  [<ffffffff82c545bf>] net_rx_action+0x26f/0x390
[17244378.599069]  [<ffffffff826a5695>] __do_softirq+0xf5/0x280
[17244378.599557]  [<ffffffff82d9642c>] call_softirq+0x1c/0x30
[17244378.600018]  <EOI> 
[17244378.600030]  [<ffffffff8262f715>] do_softirq+0x65/0xa0
[17244378.600920]  [<ffffffff826a4aeb>] __local_bh_enable_ip+0x9b/0xb0
[17244378.601355]  [<ffffffff826a4b17>] local_bh_enable+0x17/0x20
[17244378.601783]  [<ffffffffc0753ce2>] ovs_packet_cmd_execute+0x2b2/0x2e0 [openvswitch]
[17244378.602213]  [<ffffffff82c930e8>] genl_family_rcv_msg+0x208/0x430
[17244378.602652]  [<ffffffff82c9336b>] genl_rcv_msg+0x5b/0xc0
[17244378.603078]  [<ffffffff82c93310>] ? genl_family_rcv_msg+0x430/0x430
[17244378.603503]  [<ffffffff82c9135b>] netlink_rcv_skb+0xab/0xc0
[17244378.603924]  [<ffffffff82c91898>] genl_rcv+0x28/0x40
[17244378.604336]  [<ffffffff82c90ce0>] netlink_unicast+0x170/0x210
[17244378.604746]  [<ffffffff8299bb92>] ? memcpy_fromiovec+0x62/0xb0
[17244378.605149]  [<ffffffff82c91088>] netlink_sendmsg+0x308/0x420
[17244378.605561]  [<ffffffff826e4de6>] ? update_curr+0x86/0x1e0
[17244378.605969]  [<ffffffff82c333a6>] sock_sendmsg+0xb6/0xf0
[17244378.606378]  [<ffffffff826d7c52>] ? check_preempt_curr+0x92/0xa0
[17244378.606784]  [<ffffffff826d7c79>] ? ttwu_do_wakeup+0x19/0xe0
[17244378.607202]  [<ffffffff826d7daf>] ? ttwu_do_activate+0x6f/0x80
[17244378.607609]  [<ffffffff82c33ad1>] SYSC_sendto+0x121/0x1c0
[17244378.608011]  [<ffffffff82c355ee>] SyS_sendto+0xe/0x10
[17244378.608409]  [<ffffffff82d92ed2>] system_call_fastpath+0x25/0x2a
[17244378.608808] Code: 53 48 89 f3 48 83 ec 40 66 89 4d ca 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 48 8b 46 20 4c 8b b8 b8 02 00 00 4d 85 ff 74 07 <41> 80 7f 31 00 75 33 48 89 df e8 33 77 5f c2 31 c0 48 8b 4d d0 
[17244378.609720] RIP  [<ffffffffc0644fde>] br_handle_frame_finish+0x3e/0x580 [bridge]
[17244378.610164]  RSP <ffff91237fc03818>

crash> bt
PID: 26219  TASK: ffff91230b22c1c0  CPU: 6   COMMAND: "weaver"
 #0 [ffff91237fc03588] machine_kexec at ffffffff82666044
 #1 [ffff91237fc035e8] __crash_kexec at ffffffff82722ee2
 #2 [ffff91237fc036b8] crash_kexec at ffffffff82722fd0
 #3 [ffff91237fc036d0] oops_end at ffffffff82d8a798
 #4 [ffff91237fc036f8] die at ffffffff82630a7b
 #5 [ffff91237fc03728] do_general_protection at ffffffff82d8a092
 #6 [ffff91237fc03760] general_protection at ffffffff82d89718
    [exception RIP: br_handle_frame_finish+62]
    RIP: ffffffffc0644fde  RSP: ffff91237fc03818  RFLAGS: 00010202
    RAX: ffff911dcdfcb000  RBX: ffff91232ed5aef8  RCX: 0000000000000000
    RDX: 00089336e5f11642  RSI: ffff91232ed5aef8  RDI: 0000000000000000
    RBP: ffff91237fc03880   R8: 0000000000000000   R9: 0000000000000034
    R10: ffffffff83315bc0  R11: ffffffff82c7ff90  R12: ffff91232ed5aef8
    R13: ffff912346a92000  R14: 0000000000000042  R15: 2f65336162363561
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
 #7 [ffff91237fc03888] br_nf_dev_xmit at ffffffffc06100ac [br_netfilter]
 #8 [ffff91237fc038a8] br_dev_xmit at ffffffffc0640ad2 [bridge]
 #9 [ffff91237fc038e8] dev_hard_start_xmit at ffffffff82c53386
#10 [ffff91237fc03960] __dev_queue_xmit at ffffffff82c56389
#11 [ffff91237fc039d0] dev_queue_xmit at ffffffff82c564d0
#12 [ffff91237fc039e0] neigh_resolve_output at ffffffff82c6312d
#13 [ffff91237fc03a20] neigh_update at ffffffff82c65818
#14 [ffff91237fc03a88] arp_process at ffffffff82cd41c0
#15 [ffff91237fc03b50] arp_rcv at ffffffff82cd48dd
#16 [ffff91237fc03bc0] __netif_receive_skb_core at ffffffff82c53c19
#17 [ffff91237fc03c38] __netif_receive_skb at ffffffff82c53f18
#18 [ffff91237fc03c58] netif_receive_skb_internal at ffffffff82c53fa0
#19 [ffff91237fc03c88] netif_receive_skb at ffffffff82c5403c
#20 [ffff91237fc03ca8] br_netif_receive_skb at ffffffffc0644e08 [bridge]
#21 [ffff91237fc03cc0] br_pass_frame_up at ffffffffc0644f07 [bridge]
#22 [ffff91237fc03d40] br_handle_frame_finish at ffffffffc06451c1 [bridge]
#23 [ffff91237fc03db8] br_handle_frame at ffffffffc06456f9 [bridge]
#24 [ffff91237fc03e30] __netif_receive_skb_core at ffffffff82c536ea
#25 [ffff91237fc03ea0] __netif_receive_skb at ffffffff82c53f18
#26 [ffff91237fc03ec0] process_backlog at ffffffff82c54eee
#27 [ffff91237fc03f00] net_rx_action at ffffffff82c545bf
#28 [ffff91237fc03f80] __do_softirq at ffffffff826a5695
#29 [ffff91237fc03ff0] call_softirq at ffffffff82d9642c
--- <IRQ stack> ---
#30 [ffff91230b35fa18] do_softirq at ffffffff8262f715
#31 [ffff91230b35fa38] __local_bh_enable_ip at ffffffff826a4aeb
#32 [ffff91230b35fa50] local_bh_enable at ffffffff826a4b17
#33 [ffff91230b35fa60] ovs_packet_cmd_execute at ffffffffc0753ce2 [openvswitch]
#34 [ffff91230b35fac0] genl_family_rcv_msg at ffffffff82c930e8
#35 [ffff91230b35fb88] genl_rcv_msg at ffffffff82c9336b
#36 [ffff91230b35fbb8] netlink_rcv_skb at ffffffff82c9135b
#37 [ffff91230b35fbe0] genl_rcv at ffffffff82c91898
#38 [ffff91230b35fbf8] netlink_unicast at ffffffff82c90ce0
#39 [ffff91230b35fc40] netlink_sendmsg at ffffffff82c91088
#40 [ffff91230b35fcc8] sock_sendmsg at ffffffff82c333a6
#41 [ffff91230b35fe28] SYSC_sendto at ffffffff82c33ad1
#42 [ffff91230b35ff40] sys_sendto at ffffffff82c355ee
#43 [ffff91230b35ff50] system_call_fastpath at ffffffff82d92ed2

crash> dis -rl ffffffffc0644fde | tail
0xffffffffc0644fc8 <br_handle_frame_finish+40>: mov    %rax,-0x30(%rbp)
0xffffffffc0644fcc <br_handle_frame_finish+44>: xor    %eax,%eax
/usr/src/debug/kernel-3.10.0-1127.el7/linux-3.10.0-1127.el7.x86_64/net/bridge/br_input.c: 132
0xffffffffc0644fce <br_handle_frame_finish+46>: mov    0x20(%rsi),%rax
/usr/src/debug/kernel-3.10.0-1127.el7/linux-3.10.0-1127.el7.x86_64/net/bridge/br_private.h: 279
0xffffffffc0644fd2 <br_handle_frame_finish+50>: mov    0x2b8(%rax),%r15
/usr/src/debug/kernel-3.10.0-1127.el7/linux-3.10.0-1127.el7.x86_64/net/bridge/br_input.c: 141
0xffffffffc0644fd9 <br_handle_frame_finish+57>: test   %r15,%r15
0xffffffffc0644fdc <br_handle_frame_finish+60>: je     0xffffffffc0644fe5 <br_handle_frame_finish+69>
0xffffffffc0644fde <br_handle_frame_finish+62>: cmpb   $0x0,0x31(%r15)

R15: 2f65336162363561: Invalid memory address

129 /* note: already called with rcu_read_lock */
130 int br_handle_frame_finish(struct sock *sk, struct sk_buff *skb)
131 {
132         struct net_bridge_port *p = br_port_get_rcu(skb->dev);
133         enum br_pkt_type pkt_type = BR_PKT_UNICAST;
134         struct net_bridge_fdb_entry *dst = NULL;
135         struct net_bridge_mdb_entry *mdst;
136         bool local_rcv, mcast_hit = false;
137         const unsigned char *dest;
138         struct net_bridge *br;
139         u16 vid = 0;
140 
141         if (!p || p->state == BR_STATE_DISABLED) << panic context


crash> net_bridge_port -ox|grep 2b8
  [0x2b8] struct hlist_node rlist;

crash> sk_buff.dev -ox
struct sk_buff {
  [0x20] struct net_device *dev;
}

Checking RAX which is pointing out net_device, but it's already freed.

crash> sk_buff.dev ffff91232ed5aef8
  dev = 0xffff911dcdfcb000

crash> kmem ffff911dcdfcb000
CACHE             OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE  NAME
ffff910cbfc03300     4096       1009      1256    157    32k  kmalloc-4096
  SLAB              MEMORY            NODE  TOTAL  ALLOCATED  FREE
  ffffe7670a37f200  ffff911dcdfc8000     1      8          2     6
  FREE / [ALLOCATED]
   ffff911dcdfcb000  (cpu 8 cache)

      PAGE         PHYSICAL      MAPPING       INDEX CNT FLAGS
ffffe7670a37f2c0 128dfcb000                0        0  0 6fffff00008000 tail

crash> rd 0xffff911dcdfcb2b8 30
ffff911dcdfcb2b8:  2f65336162363561 61762f3a66666964   a56ba3e/diff:/va
ffff911dcdfcb2c8:  6f642f62696c2f72 65766f2f72656b63   r/lib/docker/ove
ffff911dcdfcb2d8:  62622f3279616c72 3239613235613732   rlay2/bb27a52a92
ffff911dcdfcb2e8:  6638653063303339 3634353263613730   930c0e8f07ac2546
ffff911dcdfcb2f8:  3536373564356466 3366366565623564   fd5d5765d5bee6f3
ffff911dcdfcb308:  3635353863393036 3239306635363865   609c8556e865f092
ffff911dcdfcb318:  642f643162353038 7261762f3a666669   805b1d/diff:/var
ffff911dcdfcb328:  636f642f62696c2f 7265766f2f72656b   /lib/docker/over
ffff911dcdfcb338:  6434352f3279616c 6634366664366530   lay2/54d0e6df64f
ffff911dcdfcb348:  6233623333343863 3837343564656163   c8433b3bcaed5478
ffff911dcdfcb358:  3236636539386263 6266383131363237   cb89ec62726118fb
ffff911dcdfcb368:  6262613033386262 6539663333626433   bb830abb3db33f9e
ffff911dcdfcb378:  69642f3832303837 2f7261762f3a6666   78028/diff:/var/
ffff911dcdfcb388:  6b636f642f62696c 6c7265766f2f7265   lib/docker/overl
ffff911dcdfcb398:  343437392f327961 3939633332333636   ay2/974466323c99

crash> mod -t
NAME     TAINTS
overlay  T

$ git lol kernel-3.10.0-1127.el7..kernel-3.10.0-1160.24.1.el7 net|wc -l
104

$ git lol --grep free kernel-3.10.0-1127.el7..kernel-3.10.0-1160.24.1.el7 net    
383cad0cbd81 [net] fix struct pid memory leak
16f7ce20b6c7 [net] ipv6: use in6_dev_put in dad timer handler instead of __in6_dev_put
43b3b2ae62ba [net] openvswitch: free vport unless register_netdevice() succeeds
b252f6eae21c [net] openvswitch: do not free vport if register_netdevice() is failed
dea336b3e0dd [net] netfilter: nf_queue: do not release refcouts until nf_reinject is done
22e95fb360d4 [net] netfilter: nf_queue: make nf_queue_entry_release_refs static
448d648a77d9 [net] net-sysfs: call dev_hold if kobject_init_and_add success
ec9750a1a81e [net] Fix one possible memleak in ip_setup_cork
cc00475a9d13 [net] fix null de-reference of device refcount
62718d3c5fda [net] netem: fix error path for corrupted GSO frames
afd47541cd01 [net] xfrm: policy: Fix doulbe free in xfrm_policy_timer
490a0ca56ee1 [net] rtnetlink: validate IFLA_MTU attribute in rtnl_create_link()
cc5a8a57fc05 [net] sctp: fix refcount bug in sctp_wfree
d29a337ec044 [net] sctp: move the format error check out of __sctp_sf_do_9_1_abort
065bb218e7bf [net] sctp: free cmd->obj.chunk for the unprocessed SCTP_CMD_REPLY
48180b2daddb [net] sit: fix memory leak in sit_init_net()
91e925b7568f [net] netfilter: nf_tables: use-after-free in dynamic operations
404c0231438b [nvdimm] Revert "driver boilerplate changes to properly manage device_rh"
8d81cbc35cfc [net] netfilter: ctnetlink: netns exit must wait for callbacks

commit 43b3b2ae62baffda9f7d92e244a768dd0f1bf6cf
Author: Timothy Redaelli <tredaelli@redhat.com>
Date:   Mon Oct 12 14:58:05 2020 -0400

    [net] openvswitch: free vport unless register_netdevice() succeeds


commit b252f6eae21cfe111b6145701b3cbe6893520ca2
Author: Timothy Redaelli <tredaelli@redhat.com>
Date:   Mon Oct 12 14:58:04 2020 -0400

    [net] openvswitch: do not free vport if register_netdevice() is failed

commit dea336b3e0dd5d95620fd2fa258cc95365b62d36
Author: Florian Westphal <fwestpha@redhat.com>
Date:   Thu Oct 8 19:43:29 2020 -0400

    [net] netfilter: nf_queue: do not release refcouts until nf_reinject is done

commit 448d648a77d9487367dbe50e8a6f0750071e5f4a
Author: Hangbin Liu <haliu@redhat.com>
Date:   Wed Jun 24 01:10:11 2020 -0400

    [net] net-sysfs: call dev_hold if kobject_init_and_add success

commit cc00475a9d13571b9f93f5622f2a7f5ec76a2af1
Author: Guillaume Nault <gnault@redhat.com>
Date:   Fri May 22 22:03:12 2020 -0400

    [net] fix null de-reference of device refcount