NULL pointer dereference happens in vmxnet3_rq_rx_complete()

Solution Verified - Updated -

Issue

  • The kernel crashed due to NULL pointer dereference that happened in vmxnet3_rq_rx_complete():
[1728352.515347] BUG: unable to handle kernel NULL pointer dereference at 0000000000000034
[1728352.515374] PGD 0 P4D 0 
[1728352.515382] Oops: 0000 [#1] SMP PTI
[1728352.515393] CPU: 7 PID: 0 Comm: swapper/7 Kdump: loaded Tainted: P           OE    --------- -  - 4.18.0-425.10.1.el8_7.x86_64 #1
[1728352.515418] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[1728352.515440] RIP: 0010:memcpy_orig+0x54/0x130
[1728352.515455] Code: 89 07 4c 89 4f 08 4c 89 57 10 4c 89 5f 18 48 8d 7f 20 73 d4 83 c2 20 eb 44 48 01 d6 48 01 d7 48 83 ea 20 0f 1f 00 48 83 ea 20 <4c> 8b 46 f8 4c 8b 4e f0 4c 8b 56 e8 4c 8b 5e e0 48 8d 76 e0 4c 89
[1728352.515494] RSP: 0018:ffffa61341ad4df8 EFLAGS: 00010287
[1728352.515506] RAX: ffff9194ee761840 RBX: ffff919694660000 RCX: 0000000000000200
[1728352.515522] RDX: fffffffffffffffc RSI: 000000000000003c RDI: ffff9194ee76187c
[1728352.515542] RBP: ffff9193d0ff2ac0 R08: ffffa61341ad4d00 R09: 0000000000000000
[1728352.515558] R10: ffff9193d5160000 R11: 0000000000000000 R12: 0000000000000000
[1728352.515574] R13: ffff9193ef350000 R14: ffff9193d0ff0ac0 R15: ffff9193d0ff2ac0
[1728352.515590] FS:  0000000000000000(0000) GS:ffff9196efdc0000(0000) knlGS:0000000000000000
[1728352.515608] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1728352.515622] CR2: 0000000000000034 CR3: 00000001d5410003 CR4: 00000000003706e0
[1728352.515655] Call Trace:
[1728352.515665]  <IRQ>
[1728352.515672]  vmxnet3_rq_rx_complete+0x419/0xef0 [vmxnet3]
[1728352.515690]  vmxnet3_poll_rx_only+0x31/0xa0 [vmxnet3]
[1728352.515704]  __napi_poll+0x2d/0x130
[1728352.515716]  net_rx_action+0x252/0x320
[1728352.515727]  __do_softirq+0xd7/0x2c8
[1728352.515738]  irq_exit_rcu+0xd3/0xe0
[1728352.516207]  irq_exit+0xa/0x10
[1728352.516613]  do_IRQ+0x7f/0xd0
[1728352.517015]  common_interrupt+0xf/0xf
[1728352.517411]  </IRQ>
[1728352.517819] RIP: 0010:native_safe_halt+0xe/0x20
[1728352.518204] Code: 00 f0 80 48 02 20 48 8b 00 a8 08 75 c0 e9 79 ff ff ff 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 0f 00 2d d6 96 41 00 fb f4 <e9> ed 09 21 00 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00
[1728352.519017] RSP: 0018:ffffa61341963e30 EFLAGS: 00000246 ORIG_RAX: ffffffffffffffd9
[1728352.519437] RAX: 0000000080004000 RBX: 0000000000000001 RCX: 0000000000000020
[1728352.519854] RDX: 0000000000000001 RSI: ffffffffa8ec41c0 RDI: ffff9193c2580864
[1728352.520268] RBP: ffff9193c2580864 R08: 0000000000000001 R09: ffff9193c2580800
[1728352.520695] R10: 00000000000003bd R11: ffff9196efde9c04 R12: 0000000000000001
[1728352.521118] R13: ffffffffa8ec41c0 R14: 0000000000000001 R15: 0000000000000001
[1728352.521543]  acpi_idle_do_entry+0x4a/0x60
[1728352.521967]  acpi_idle_enter+0x5a/0xd0
[1728352.522392]  cpuidle_enter_state+0x86/0x3d0
[1728352.522818]  cpuidle_enter+0x2c/0x40
[1728352.523248]  do_idle+0x268/0x2d0
[1728352.523673]  cpu_startup_entry+0x6f/0x80
[1728352.524098]  start_secondary+0x18c/0x1d0
[1728352.524516]  secondary_startup_64_no_verify+0xc2/0xcb
[1728352.524939] Modules linked in: [...]
[1728352.528309] Red Hat flags: eBPF/event
[1728352.528798] CR2: 0000000000000034
  • Just before the occurrence of the NULL dereference, page allocation failure happened in vmxnet3_rq_create():
[1728352.477993] ethtool: page allocation failure: order:9, mode:0x6000c0(GFP_KERNEL), nodemask=(null),cpuset=/,mems_allowed=0
[1728352.478005] CPU: 0 PID: 1885963 Comm: ethtool Kdump: loaded Tainted: P           OE    --------- -  - 4.18.0-425.10.1.el8_7.x86_64 #1
[1728352.478007] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
[1728352.478009] Call Trace:
[1728352.478028]  dump_stack+0x41/0x60
[1728352.478035]  warn_alloc.cold.120+0x7b/0x11b
[1728352.478038]  ? _cond_resched+0x15/0x30
[1728352.478042]  ? __alloc_pages_direct_compact+0x15f/0x170
[1728352.478043]  __alloc_pages_slowpath+0xcd3/0xd10
[1728352.478047]  __alloc_pages_nodemask+0x2e2/0x320
[1728352.478049]  __dma_direct_alloc_pages.constprop.25+0x8a/0x120
[1728352.478053]  dma_direct_alloc+0x5a/0x2a0
[1728352.478056]  vmxnet3_rq_create.part.57+0x17c/0x1f0 [vmxnet3]
[1728352.478060]  vmxnet3_create_queues+0x267/0x360 [vmxnet3]
[1728352.478063]  vmxnet3_set_ringparam+0x243/0x320 [vmxnet3]
[1728352.478065]  ethnl_set_rings+0x1f7/0x2f0
[1728352.478070]  genl_family_rcv_msg_doit.isra.17+0x113/0x150
[1728352.478073]  genl_family_rcv_msg+0xb7/0x170
[1728352.478075]  ? rings_prepare_data+0x80/0x80
[1728352.478077]  genl_rcv_msg+0x47/0xa0
[1728352.478079]  ? genl_family_rcv_msg+0x170/0x170
[1728352.478080]  netlink_rcv_skb+0x4c/0x130
[1728352.478084]  genl_rcv+0x24/0x40
[1728352.478085]  netlink_unicast+0x19a/0x230
[1728352.478088]  netlink_sendmsg+0x204/0x3d0
[1728352.478090]  sock_sendmsg+0x50/0x60
[1728352.478093]  __sys_sendto+0xee/0x160
[1728352.478096]  ? syscall_trace_enter+0x1ff/0x2d0
[1728352.478099]  __x64_sys_sendto+0x24/0x30
[1728352.478101]  do_syscall_64+0x5b/0x1b0
[1728352.478103]  entry_SYSCALL_64_after_hwframe+0x61/0xc6
[1728352.478107] RIP: 0033:0x7fd16ab1f9db
[1728352.478114] Code: 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 f3 0f 1e fa 48 8d 05 45 4d 29 00 41 89 ca 8b 00 85 c0 75 14 b8 2c 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 75 c3 0f 1f 40 00 41 57 4d 89 c7 41 56 41 89
[1728352.478115] RSP: 002b:00007ffd98eae868 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
[1728352.478117] RAX: ffffffffffffffda RBX: 000055b23ff232a0 RCX: 00007fd16ab1f9db
[1728352.478118] RDX: 000000000000003c RSI: 000055b23ff233b0 RDI: 0000000000000003
[1728352.478121] RBP: 00007ffd98eae900 R08: 00007fd16adb8480 R09: 000000000000000c
[1728352.478122] R10: 0000000000000000 R11: 0000000000000246 R12: 000055b23ff23350
[1728352.478123] R13: 000055b23ff23340 R14: 0000000000000000 R15: 00007ffd98eaea88
...
[1728352.478188] vmxnet3 0000:0b:00.0 ens192: rx data ring will be disabled
[1728352.487552] vmxnet3 0000:0b:00.0 ens192: intr type 3, mode 0, 9 vectors allocated
[1728352.488691] vmxnet3 0000:0b:00.0 ens192: NIC Link is Up 10000 Mbps

Environment

  • Red Hat Enterprise Linux 8
  • RHEL guest running on VMware ESXi
  • vmxnet3 NIC driver

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content