system is failing to boot in systemd in Red Hat Enterprise Linux 7

Solution Unverified - Updated -

Environment

  • Red Hat Enterprise Linux 7
  • Systemd
  • grub

Issue

  • RHEL7 system is failing to boot in systemd with following point
[  OK  ] Started Tell Plymouth To Write Out Runtime Data.
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Reached target System Initialization.
[  OK  ] Reached target Timers.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
[  OK  ] Reached target Multi-User System.
         Starting Dump dmesg to /var/log/dmesg...
[  OK  ] Started Dump dmesg to /var/log/dmesg.
  • How to debug Red Hat Enterprise Linux 7 boot issue with systemd

Resolution

Run the following commands in the back ground shell with Ctrl-Alt-F9, which can enabled by systemctl enable debug-shell.service

  # rm /etc/systemd/system/default.target
  # systemctl set-default multi-user.target
  # systemctl get-default                      <==== This should return 'multi-user.target'

This recreates /etc/systemd/system/default.target as a symbolic link to multi-user.target.

Root Cause

/etc/systemd/system/default.target was a normal file (copied from multi-user.target). It should be a symbolic link to /lib/systemd/system/multi-user.target.
Otherwise, systemd cannot resolve properly the dependency for default.target, and many services are not activated.

Diagnostic Steps

  • Reproduce the boot problem with a boot parameter "systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M" added to the kernel line
  • As grub is in use not grub2 from RHEL7, edit /etc/grub.conf to add them
  • Enable debug-shell.service
# systemctl enable debug-shell.service

Reproduce the problem, and run the following command in the background shell with Ctrl-Alt-F9:

  # systemctl list-jobs > /tmp/systemctl-listjobs.log
  # ps -ef > /tmp/psef.log
  # lsof > /tmp/lsof.log
  # systemctl get-default > /tmp/systemctl-getdefault.log
  # systemctl list-units > /tmp/systemctl-listunits.log
  # systemctl list-units --all > /tmp/systemctl-listunitsall.log
  # systemctl get-default > /tmp/systemctl-getdefault.log
  # journalctl > /tmp/journal.log

  # tar zcvf /tmp/systemd-dbg.tar.gz /tmp/systemctl-listjobs.log /tmp/systemctl-getdefault.log /tmp/systemctl-listunits.log /tmp/systemctl-listunitsall.log /tmp/journal.log /tmp/psef.log /tmp/lsof.log /etc/systemd/system /lib/systemd/system
  • systemctl get-default returned default.target (not multi-user.target)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.