Kernel panic with `Missing ENDBR`

Root Cause

Summary

A third-party module or ebpf program attempted an indirect function call to a location that does not contain the endbr instruction. This triggered the Indirect Branch Tracking (IBT) security feature of the CPU and kernel resulting in a kernel panic.

Details

Background

Function calls within programs can be direct or indirect. A direct function call executes the function associated with the name while an indirect function call will instead use a provided address to execute at where this address is typically referred to as a function pointer. Indirect function calls are useful in many ways, for example, when the function called to do some action can vary based on some additional information. System Calls (syscalls), for example, all call the same function directly with a syscall number. The number is used to identify which syscall to execute and which function pointer to use to execute. For example, the function pointer for sys_open will be different for sys_read.

Executing function pointers is found throughout most any software, operating system, and firmware. Malicious code could derive a calculated function pointer to execute arbitrary code. To greatly reduce this problem surface area, Intel introduced Indirect Branch Tracking (IBT) as part of its Control Flow Enforcement (CET). The general premise is to guarantee the address you start executing from a function pointer is safe to reach at that specific point.

Implementation

Intel introduced the endbr assembly instruction (or more precisely endbr32 and endbr64). The instruction just executes as a no-op, which more or less means an instruction to execute nothing for a cpu cycle. Indirect branches are expected to land on this endbr instruction. If so, execution continues as expected. If not, then the CPU throws a control-protection (CP) exception.

The Linux kernel handles the control-protection exception by first omitting an error Missing ENDBR: <function location>;

arch/x86/kernel/cet.c
 84 static void do_kernel_cp_fault(struct pt_regs *regs, unsigned long error_code)
 85 {    
...
 96     pr_err("Missing ENDBR: %pS\n", (void *)instruction_pointer(regs));
 97     if (!ibt_fatal) {
 98         printk(KERN_DEFAULT CUT_HERE);
 99         __warn(__FILE__, __LINE__, (void *)regs->ip, TAINT_WARN, regs, NULL);
100         return;
101     }
102     BUG();
103 }    

For example,

Missing ENDBR: kallsyms_lookup_name+0x4/0xd0

The above indicates the caller attempted to start executing at kallsyms_lookup_name+0x4 using a function pointer and indirect function calling. However, that instruction is not endbr64;

crash> dis kallsyms_lookup_name | head -3
0xffffffffaaffe380 <kallsyms_lookup_name>:  nopw   (%rax)
0xffffffffaaffe384 <kallsyms_lookup_name+0x4>:  nopl   0x0(%rax,%rax,1)     <---
0xffffffffaaffe389 <kallsyms_lookup_name+0x9>:  push   %rbx

That function pointer points to a nopl instruction instead. As such, the cpu threw the control-protection exception resulting in a kernel panic.

In contrast;

crash> dis -r ffffffffc1701015 | tail -5
0xffffffffc170100e <_MODULE_INIT_TEXT_START_traps+0xe>: nop
0xffffffffc170100f <_MODULE_INIT_TEXT_START_traps+0xf>: nop
0xffffffffc1701010 <_MODULE_INIT_TEXT_START_traps+0x10>:    endbr64          <---
0xffffffffc1701014 <_MODULE_INIT_TEXT_START_traps+0x14>:    nopl   0x0(%rax,%rax,1)
0xffffffffc1701019 <_MODULE_INIT_TEXT_START_traps+0x19>:    push   %rbp

An indirect function call landing at 0xffffffffc1701010 would succeed because such a call would land on endbr64. The assembly instruction indicates there is an expectation to call to that location elsewhere and doing so is safe.