The kernel crashes due to double fault on a RHEL guest running on VMware ESXi

Solution Verified - Updated -

Issue

  • The kernel crashed due to double fault on a RHEL VM running on VMware ESXi hypervisor:
WARNING: stack going in the wrong direction? at double_fault+0x1e/0x30
WARNING: stack recursion on stack type 5
BUG: unable to handle kernel paging request at ffff92cfcc958850
PGD 3212a02067 P4D 3212a02067 PUD 10cf17063 PMD a3755e063 PTE 800000010c958163
Oops: 0000 [#1] SMP NOPTI
CPU: 15 PID: 0 Comm: swapper/15 Kdump: loaded Tainted: P           OE     -------- -  - 4.18.0-553.75.1.el8_10.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
RIP: 0010:__do_page_fault+0x25/0x440
Code: dd 0f 1f 40 00 0f 1f 44 00 00 41 57 41 56 41 55 41 54 49 89 d4 55 48 89 fd 53 48 89 f3 65 4c 8b 2c 25 40 dc 01 00 48 83 ec 18 <49> 8b 85 50 08 00 00 0f 0d 48 70 48 89 d7 e8 78 ff ff ff 84 c0 0f
RSP: 0018:fffffe39ebf2b9a8 EFLAGS: 00010092
RAX: ffff92cfcc958850 RBX: 0000000000000000 RCX: ffffffffad000c17
RDX: ffff92cfcc958850 RSI: 0000000000000000 RDI: fffffe39ebf2ba28
RBP: fffffe39ebf2ba28 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff92cfcc958850
R13: ffff92cfcc958000 R14: 0000000000000000 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff930523dc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffff92cfcc958850 CR3: 0000003211010005 CR4: 0000000000770ee0
PKRU: 55555554
Call Trace:
 <#DF>
 ? __die_body+0x1a/0x60
 ? no_context+0x1ba/0x3f0
 ? __bad_area_nosemaphore+0x157/0x180
 ? do_page_fault+0x37/0x12d
 ? page_fault+0x1e/0x30
 ? native_iret+0x7/0x7
 ? __do_page_fault+0x25/0x440
 do_page_fault+0x37/0x12d
 page_fault+0x1e/0x30
RIP: 0010:__do_page_fault+0x25/0x440
Code: dd 0f 1f 40 00 0f 1f 44 00 00 41 57 41 56 41 55 41 54 49 89 d4 55 48 89 fd 53 48 89 f3 65 4c 8b 2c 25 40 dc 01 00 48 83 ec 18 <49> 8b 85 50 08 00 00 0f 0d 48 70 48 89 d7 e8 78 ff ff ff 84 c0 0f
RSP: 0018:fffffe39ebf2bad8 EFLAGS: 00010096
RAX: ffff92cfcc958850 RBX: 0000000000000000 RCX: ffffffffad000c17
RDX: ffff92cfcc958850 RSI: 0000000000000000 RDI: fffffe39ebf2bb58
RBP: fffffe39ebf2bb58 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff92cfcc958850
R13: ffff92cfcc958000 R14: 0000000000000000 R15: 0000000000000000
 ? native_iret+0x7/0x7
 do_page_fault+0x37/0x12d
 page_fault+0x1e/0x30
RIP: 0010:__do_page_fault+0x25/0x440
Code: dd 0f 1f 40 00 0f 1f 44 00 00 41 57 41 56 41 55 41 54 49 89 d4 55 48 89 fd 53 48 89 f3 65 4c 8b 2c 25 40 dc 01 00 48 83 ec 18 <49> 8b 85 50 08 00 00 0f 0d 48 70 48 89 d7 e8 78 ff ff ff 84 c0 0f
RSP: 0018:fffffe39ebf2bc08 EFLAGS: 00010092
RAX: ffff92cfcc958850 RBX: 0000000000000000 RCX: ffffffffad000c17
RDX: ffff92cfcc958850 RSI: 0000000000000000 RDI: fffffe39ebf2bc88
RBP: fffffe39ebf2bc88 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff92cfcc958850
R13: ffff92cfcc958000 R14: 0000000000000000 R15: 0000000000000000
 ? native_iret+0x7/0x7
 do_page_fault+0x37/0x12d
 page_fault+0x1e/0x30
RIP: 0010:__do_page_fault+0x25/0x440
Code: dd 0f 1f 40 00 0f 1f 44 00 00 41 57 41 56 41 55 41 54 49 89 d4 55 48 89 fd 53 48 89 f3 65 4c 8b 2c 25 40 dc 01 00 48 83 ec 18 <49> 8b 85 50 08 00 00 0f 0d 48 70 48 89 d7 e8 78 ff ff ff 84 c0 0f
RSP: 0018:fffffe39ebf2bd38 EFLAGS: 00010092
RAX: ffff92cfcc9592e8 RBX: 0000000000000000 RCX: ffffffffad000c17
RDX: ffff92cfcc9592e8 RSI: 0000000000000000 RDI: fffffe39ebf2bdb8
RBP: fffffe39ebf2bdb8 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff92cfcc9592e8
R13: ffff92cfcc958000 R14: 0000000000000000 R15: 0000000000000000
 ? native_iret+0x7/0x7
 do_page_fault+0x37/0x12d
 page_fault+0x1e/0x30
RIP: 0010:uprobe_get_trap_addr+0xe/0x40
Code: 00 48 8b 87 80 00 00 00 48 83 e8 01 e9 26 85 a6 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 65 48 8b 04 25 40 dc 01 00 <48> 8b 80 e8 12 00 00 48 85 c0 75 0c 48 8b 87 80 00 00 00 e9 f5 84
RSP: 0018:fffffe39ebf2be68 EFLAGS: 00010046
RAX: ffff92cfcc958000 RBX: fffffe39ebf2be88 RCX: ffffffffad000c17
RDX: 0000000000000000 RSI: 0000000000000000 RDI: fffffe39ebf2be88
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
 ? native_iret+0x7/0x7
 do_invalid_op+0x12/0x40
 invalid_op+0x14/0x20
RIP: 0010:do_double_fault+0x179/0x180
Code: 48 8b 52 20 48 c7 00 00 00 00 00 48 89 50 28 48 c7 87 80 00 00 00 b0 11 00 ad 48 89 87 98 00 00 00 5b 5d 41 5c e9 62 ad cd 00 <0f> 0b 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 f1 48 89 f5 48 89 fa
RSP: 0018:fffffe39ebf2bf38 EFLAGS: 00010046
RAX: 0000000000f00000 RBX: fffffe39ebf2bf58 RCX: 0000000000000000
RDX: fffffe39ebf2a000 RSI: 0000000000000000 RDI: fffffe39ebf2bf58
RBP: ffff92cfcc958000 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
 ? do_double_fault+0x3f/0x180
 double_fault+0x1e/0x30
 ? native_iret+0x7/0x7
 ? page_fault+0x8/0x30
 ? srso_alias_safe_ret+0x7/0x7
 </#DF>
Modules linked in: [...]
Red Hat flags: eBPF/event eBPF/rawtrace
CR2: ffff92cfcc958850

Environment

  • (include, but not be limited to,) Red Hat Enterprise Linux 8
  • (include, but not be limited to,) AMD EPYC 7500 Processors
  • (include, but not be limited to,) VMware ESXi Hypervisors
    • Similar issues can happen due to some hypervisor malfunction even with different kernel versions, processor models.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content