system is failing to boot in systemd in Red Hat Enterprise Linux 7

Solution Unverified - Updated -

Environment

  • Red Hat Enterprise Linux 7
  • Systemd
  • grub

Issue

  • RHEL7 system is failing to boot in systemd with following point
[[32m  OK  [0m] Started Tell Plymouth To Write Out Runtime Data.
[[32m  OK  [0m] Started Update UTMP about System Boot/Shutdown.
[[32m  OK  [0m] Reached target System Initialization.
[[32m  OK  [0m] Reached target Timers.
[[32m  OK  [0m] Listening on D-Bus System Message Bus Socket.
[[32m  OK  [0m] Reached target Sockets.
[[32m  OK  [0m] Reached target Basic System.
[[32m  OK  [0m] Reached target Multi-User System.
         Starting Dump dmesg to /var/log/dmesg...
[[32m  OK  [0m] Started Dump dmesg to /var/log/dmesg.
  • How to debug Red Hat Enterprise Linux 7 boot issue with systemd

Resolution

Run the following commands in the back ground shell with Ctrl-Alt-F9, which can enabled by systemctl enable debug-shell.service

  # rm /etc/systemd/system/default.target
  # systemctl set-default multi-user.target
  # systemctl get-default                      <==== This should return 'multi-user.target'

This recreates /etc/systemd/system/default.target as a symbolic link to multi-user.target.

Root Cause

/etc/systemd/system/default.target was a normal file (copied from multi-user.target). It should be a symbolic link to /lib/systemd/system/multi-user.target.
Otherwise, systemd cannot resolve properly the dependency for default.target, and many services are not activated.

Diagnostic Steps

  • Reproduce the boot problem with a boot parameter "systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M" added to the kernel line
  • As grub is in use not grub2 from RHEL7, edit /etc/grub.conf to add them
  • Enable debug-shell.service
# systemctl enable debug-shell.service

Reproduce the problem, and run the following command in the background shell with Ctrl-Alt-F9:

  # systemctl list-jobs > /tmp/systemctl-listjobs.log
  # ps -ef > /tmp/psef.log
  # lsof > /tmp/lsof.log
  # systemctl get-default > /tmp/systemctl-getdefault.log
  # systemctl list-units > /tmp/systemctl-listunits.log
  # systemctl list-units --all > /tmp/systemctl-listunitsall.log
  # systemctl get-default > /tmp/systemctl-getdefault.log
  # journalctl > /tmp/journal.log

  # tar zcvf /tmp/systemd-dbg.tar.gz /tmp/systemctl-listjobs.log /tmp/systemctl-getdefault.log /tmp/systemctl-listunits.log /tmp/systemctl-listunitsall.log /tmp/journal.log /tmp/psef.log /tmp/lsof.log /etc/systemd/system /lib/systemd/system
  • systemctl get-default returned default.target (not multi-user.target)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments