Booting System With 'mem' or 'memmap' Randomly Results in an Undersized Device

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 9
  • Systems booted with UEFI

Issue

  • The mem= and memmap= kernel command line parameters do not work as expected
  • Booting the system with a memmap=nn[KMG]!ss[KMG] reservation results in an undersized /dev/pmem0 device sometimes

Resolution

Red Hat Enterprise Linux 8

Upgrade to kernel-4.18.0-553.72.1.el8_10 or above.

Red Hat Enterprise Linux 9

Upgrade to kernel-5.14.0-603.el9 or above.

Considerations

Note This patch disables physical Kernel Address Space Layout Randomization (KASLR) when mem=, memmap=, efi_fake_mem=, or hugepages= is applied to the kernel command line. When any of those parameters are applied to the kernel command line, the kernel will emit the following log line;

cmdline memmap override detected on the kernel command line - disabling physical KASLR

This log line is safe to ignore and only meant to inform the kernel is disabling physical KASLR; virtual KASLR will still be enabled unless KASLR itself is disabled. Removing the relevant parameter on boot will re-enable physical KASLR.

In the event both hugepages and physical KASLR are needed, remove the hugepage kernel parameter, restart the system, and allocate the hugepages after boot by either writing the desired amount of hugepages to /proc/sys/vm/nr_hugepages or setting the vm.nr_hugepages sysctl. For more information on this, please refer to the following knowledgebase article;

Root Cause

Kernel Address Space Layout Randomization (KASLR) will randomize where the kernel is placed on boot for security reason. The memmap parameter creates a hole in memory for custom drivers to take advantage of but the kernel to not use in typical operations. The issue is KASLR may inadvertently place the kernel in the memmap hole, causing the hole size to be smaller than it is designated as, and to be partitioned.

Diagnostic Steps

The issue can be observed across multiple reboots by checking /proc/iomem for the addresses of the kernel and the memmap hole, and lsblk can be checked for the size of the hole;

Steps to Reproduce

  1. Set the memmap kernel parameter, starts at the 4 GiB point and is 3 GiB in size

    # grubby --args='memmap=3G!4G' --update-kernel=ALL && reboot

  2. Confirm the parameter is set after a reboot

    # cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-4.18.0-553.46.1.el8_10.x86_64 root=/dev/mapper/rhel-root ro crashkernel=auto resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet console=tty0 console=ttyS0,115200 memmap=3G!4G

Verifying the Symptoms

The following command can be ran on a separate system to check for the symptoms across reboot. Make sure to replace root@r8 with the appropriate user and hostname of the system being tested;

```
$ while true; do ssh root@r8 'grep -e Kernel -e legacy /proc/iomem; echo; lsblk | grep pmem; echo; reboot'; sleep 10; done
[...]
Connection to r8 closed by remote host.
2de00000-2ec00f28 : Kernel code
2ec00f29-2fdd18bf : Kernel data
30488000-311fffff : Kernel bss
100000000-1bfffffff : Persistent Memory (legacy)

pmem0         259:0    0    3G  0 disk 

Connection to r8 closed by remote host.
100000000-1b7bfffff : Persistent Memory (legacy)
1b7c00000-1b8a00f28 : Kernel code
1b8a00f29-1b9bd18bf : Kernel data
1ba288000-1baffffff : Kernel bss
1bb02c000-1bfffffff : Persistent Memory (legacy)

pmem0         259:0    0  2.9G  0 disk 

Connection to r8 closed by remote host.
34600000-35400f28 : Kernel code
35400f29-365d18bf : Kernel data
36c88000-379fffff : Kernel bss
100000000-1bfffffff : Persistent Memory (legacy)

pmem0         259:0    0    3G  0 disk 

Connection to r8 closed by remote host.
[...]
```

In the second set of output above, the memmap hole (Persistent Memory) is partitioned while it is contiguous in the first and last sets. The variation in location and the physical addresses listed next to the entries show KASLR is still enabled and confirms the issue is occurring.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments