System fails to boot with the error " Failed to mount Configuration File System".

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 7

Issue

  • System falls into emergency mode with the below errors:
[FAILED]  Failed to mount Configuration File System.
See `systemctl status sys-kernel-config.mount` for details.

Resolution

  • Recreate the missing symbolic link:
# rm -rf /bin
# ln -s usr/bin /bin
  • Verify if the filesystem package is intact after above changes:
# rpm -V filesystem

The above command should not return any output.

  • Rebuild the initramfs from the working kernel or from the rescue iso:
# dracut -f -v  /boot/initramfs-<release_version_of_kernel>.el7.x86_64.img <release_version_of_kernel>

Follow the link for more information on How to rebuild the initial ramdisk image in Red Hat Enterprise Linux

Root Cause

  • On an ideal system below configuration is seen:
# ls -ld /bin
lrwxrwxrwx. 1 root root 7 Oct 29 08:33 /bin -> usr/bin
  • On the affected system , the newly generated initramfs had the symlink missing for /bin. It was not pointing to /usr/bin. As important binaries like mount failed to execute during early startup, this resulted the system to fall into emergency mode.

Diagnostic Steps

  • Check if /bin is a symlink to /usr/bin in the initramfs:
# lsinitrd initramfs-<affected-kernel>.img | grep -w "bin"
Arguments: -v --install '/usr/bin/mount' -f
drwxr-xr-x   2 root     root        0 Nov  3 03:10 bin <------- no symlink
  • Check if /bin is a symlink to /usr/bin:
# ls -ld /bin
drwxrwxr-x. 2 user  user 32768 Nov  2 09:35 /bin <------- wrong permission/ownership and missing symlink
  • Run the below command to identify the missing or modified binaries/files and check if filesystem package is one of them:
# rpm -Va &> /tmp/rpm.out 
# rpm -V filesystem
.M.......    /bin <--------------- M stands for Mode differs (includes permissions and file type)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments