Server is not booting up. Stuck after "Welcome to Red Hat Enterprise Linux Server"

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux

Issue

  • After a scheduled reboot the system hangs immediately following "Welcome to Red Hat Enterprise Linux Server"
  • No updates were applied and no known changes were made prior to the reboot
  • Removed "rhgb quiet" from the command line but we still do not see any errors
  • Older installed kernel and single user mode hang in the same location

  • After reducing RAM, system will not boot and loops with Out of memory and Killed process messages before init completes.

Resolution

  • Remove any erroneous hugepage allocation vm.nr_hugepages from /etc/sysctl.conf
  • On Red Hat Enterprise Linux 7, backup and rebuild the initramfs to have the new setting apply

Root Cause

  • Hugepage allocation was set to a larger value than total RAM 
# grep vm.nr_hugepages /etc/sysctl.conf
vm.nr_hugepages = 79252380

# grep MemTotal /proc/meminfo
MemTotal:       137995282 kB

$ echo '79252380*2' | bc
158504760
  • The HugePage allocation exceeds total installed RAM.

Diagnostic Steps

  • Interrupt grub during boot and add "init=/bin/bash" to the kernel command line. Boot with these changes and drop into a bash shell prior to applying any sysctls.

  • Remount the root filesystem read-write

bash-4.1# mount -o remount rw /
  • Edit /etc/sysctl.conf to comment out an incorrect hugepage setting
bash-4.1# vi /etc/sysctl.conf
..........
#vm.nr_hugepages = 79252380
  • Remount read only then reset the machine via power switch
bash-4.1# mount -o remount ro /

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.