The kernel crashes due to double fault on a RHEL guest running on VMware ESXi
Issue
- The kernel crashed due to double fault on a RHEL VM running on VMware ESXi hypervisor:
WARNING: stack going in the wrong direction? at double_fault+0x1e/0x30
WARNING: stack recursion on stack type 5
BUG: unable to handle kernel paging request at ffff92cfcc958850
PGD 3212a02067 P4D 3212a02067 PUD 10cf17063 PMD a3755e063 PTE 800000010c958163
Oops: 0000 [#1] SMP NOPTI
CPU: 15 PID: 0 Comm: swapper/15 Kdump: loaded Tainted: P OE -------- - - 4.18.0-553.75.1.el8_10.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
RIP: 0010:__do_page_fault+0x25/0x440
Code: dd 0f 1f 40 00 0f 1f 44 00 00 41 57 41 56 41 55 41 54 49 89 d4 55 48 89 fd 53 48 89 f3 65 4c 8b 2c 25 40 dc 01 00 48 83 ec 18 <49> 8b 85 50 08 00 00 0f 0d 48 70 48 89 d7 e8 78 ff ff ff 84 c0 0f
RSP: 0018:fffffe39ebf2b9a8 EFLAGS: 00010092
RAX: ffff92cfcc958850 RBX: 0000000000000000 RCX: ffffffffad000c17
RDX: ffff92cfcc958850 RSI: 0000000000000000 RDI: fffffe39ebf2ba28
RBP: fffffe39ebf2ba28 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff92cfcc958850
R13: ffff92cfcc958000 R14: 0000000000000000 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff930523dc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffff92cfcc958850 CR3: 0000003211010005 CR4: 0000000000770ee0
PKRU: 55555554
Call Trace:
<#DF>
? __die_body+0x1a/0x60
? no_context+0x1ba/0x3f0
? __bad_area_nosemaphore+0x157/0x180
? do_page_fault+0x37/0x12d
? page_fault+0x1e/0x30
? native_iret+0x7/0x7
? __do_page_fault+0x25/0x440
do_page_fault+0x37/0x12d
page_fault+0x1e/0x30
RIP: 0010:__do_page_fault+0x25/0x440
Code: dd 0f 1f 40 00 0f 1f 44 00 00 41 57 41 56 41 55 41 54 49 89 d4 55 48 89 fd 53 48 89 f3 65 4c 8b 2c 25 40 dc 01 00 48 83 ec 18 <49> 8b 85 50 08 00 00 0f 0d 48 70 48 89 d7 e8 78 ff ff ff 84 c0 0f
RSP: 0018:fffffe39ebf2bad8 EFLAGS: 00010096
RAX: ffff92cfcc958850 RBX: 0000000000000000 RCX: ffffffffad000c17
RDX: ffff92cfcc958850 RSI: 0000000000000000 RDI: fffffe39ebf2bb58
RBP: fffffe39ebf2bb58 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff92cfcc958850
R13: ffff92cfcc958000 R14: 0000000000000000 R15: 0000000000000000
? native_iret+0x7/0x7
do_page_fault+0x37/0x12d
page_fault+0x1e/0x30
RIP: 0010:__do_page_fault+0x25/0x440
Code: dd 0f 1f 40 00 0f 1f 44 00 00 41 57 41 56 41 55 41 54 49 89 d4 55 48 89 fd 53 48 89 f3 65 4c 8b 2c 25 40 dc 01 00 48 83 ec 18 <49> 8b 85 50 08 00 00 0f 0d 48 70 48 89 d7 e8 78 ff ff ff 84 c0 0f
RSP: 0018:fffffe39ebf2bc08 EFLAGS: 00010092
RAX: ffff92cfcc958850 RBX: 0000000000000000 RCX: ffffffffad000c17
RDX: ffff92cfcc958850 RSI: 0000000000000000 RDI: fffffe39ebf2bc88
RBP: fffffe39ebf2bc88 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff92cfcc958850
R13: ffff92cfcc958000 R14: 0000000000000000 R15: 0000000000000000
? native_iret+0x7/0x7
do_page_fault+0x37/0x12d
page_fault+0x1e/0x30
RIP: 0010:__do_page_fault+0x25/0x440
Code: dd 0f 1f 40 00 0f 1f 44 00 00 41 57 41 56 41 55 41 54 49 89 d4 55 48 89 fd 53 48 89 f3 65 4c 8b 2c 25 40 dc 01 00 48 83 ec 18 <49> 8b 85 50 08 00 00 0f 0d 48 70 48 89 d7 e8 78 ff ff ff 84 c0 0f
RSP: 0018:fffffe39ebf2bd38 EFLAGS: 00010092
RAX: ffff92cfcc9592e8 RBX: 0000000000000000 RCX: ffffffffad000c17
RDX: ffff92cfcc9592e8 RSI: 0000000000000000 RDI: fffffe39ebf2bdb8
RBP: fffffe39ebf2bdb8 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff92cfcc9592e8
R13: ffff92cfcc958000 R14: 0000000000000000 R15: 0000000000000000
? native_iret+0x7/0x7
do_page_fault+0x37/0x12d
page_fault+0x1e/0x30
RIP: 0010:uprobe_get_trap_addr+0xe/0x40
Code: 00 48 8b 87 80 00 00 00 48 83 e8 01 e9 26 85 a6 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 65 48 8b 04 25 40 dc 01 00 <48> 8b 80 e8 12 00 00 48 85 c0 75 0c 48 8b 87 80 00 00 00 e9 f5 84
RSP: 0018:fffffe39ebf2be68 EFLAGS: 00010046
RAX: ffff92cfcc958000 RBX: fffffe39ebf2be88 RCX: ffffffffad000c17
RDX: 0000000000000000 RSI: 0000000000000000 RDI: fffffe39ebf2be88
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
? native_iret+0x7/0x7
do_invalid_op+0x12/0x40
invalid_op+0x14/0x20
RIP: 0010:do_double_fault+0x179/0x180
Code: 48 8b 52 20 48 c7 00 00 00 00 00 48 89 50 28 48 c7 87 80 00 00 00 b0 11 00 ad 48 89 87 98 00 00 00 5b 5d 41 5c e9 62 ad cd 00 <0f> 0b 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 f1 48 89 f5 48 89 fa
RSP: 0018:fffffe39ebf2bf38 EFLAGS: 00010046
RAX: 0000000000f00000 RBX: fffffe39ebf2bf58 RCX: 0000000000000000
RDX: fffffe39ebf2a000 RSI: 0000000000000000 RDI: fffffe39ebf2bf58
RBP: ffff92cfcc958000 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
? do_double_fault+0x3f/0x180
double_fault+0x1e/0x30
? native_iret+0x7/0x7
? page_fault+0x8/0x30
? srso_alias_safe_ret+0x7/0x7
</#DF>
Modules linked in: [...]
Red Hat flags: eBPF/event eBPF/rawtrace
CR2: ffff92cfcc958850
Environment
- (include, but not be limited to,) Red Hat Enterprise Linux 8
- (include, but not be limited to,) AMD EPYC 7500 Processors
- (include, but not be limited to,) VMware ESXi Hypervisors
- Similar issues can happen due to some hypervisor malfunction even with different kernel versions, processor models.
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.