System boot process hangs after "Starting udev Kernel Device Manager" being outputted.

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 8

Issue

  • System hangs during reboot with the following repeated errors. However, the system can boot eventually after 12~13 hours.

    [FAILED] Failed to start udev Kernel Device Manager.
See `systemctl status systemd-udevd.service' for dtails.
[  OK  ] Stopped udev Kernel Device Manager.
         Starting udev Kernel Device Manager...

  • systemd-udevd.service failed to start during boot process.

Resolution

Change the order of services used for determining "group" in /etc/nsswitch.conf.

  • Before

    group:      sss nis files systemd

  • After

    group:      files sss nis systemd

Root Cause

During boot, udevd tried to match groups according to some customized rules. Since sss and nis are set firstly as the services which are used to determine group in nsswitch.conf, it tried to access 127.0.0.1:111. Thus, udevd can't start until rpcbind starts up.

Diagnostic Steps

$ grep GROUP * | cut -d: -f1 | sort -u
50-xxx.rules
60-xxx.rules

$ cat /etc/nsswitch.conf | grep -v "^#" | grep -v "^$" | grep ^group
group:      sss nis files systemd

Add strace in systemd-udevd.service unit file.

ExecStart=/usr/bin/strace -fttTvyy -D -s 128 -- /usr/lib/systemd/systemd-udevd

The following messages are outputted in messages.

systemd[1]: systemd-udevd.service: start operation timed out. Terminating. <<<----------- udev Kernel Device Manager timed out
...
strace[3429]: 10:59:07.333524 connect(10<TCP:[70288]>, {sa_family=AF_INET, sin_port=htons(111), sin_addr=inet_addr("127.0.0.1")}, 16) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) <89.931070>
...
systemd[3435]: rpcbind.service: Executing: /usr/bin/rpcbind -w -f <<<--------------------- RPC Bind starting
systemd[3436]: var-lib-nfs-rpc_pipefs.mount: Executing: /usr/bin/mount sunrpc /var/lib/nfs/rpc_pipefs -t rpc_pipefs
systemd[3437]: Applying namespace mount on /run/systemd/unit-root/dev
systemd[3437]: Applying namespace mount on /run/systemd/unit-root/proc
...
systemd-update-utmp[3465]: Bus n/a: changing state RUNNING → CLOSED
strace[3429]: 11:00:37.264890 connect(10<TCP:[127.0.0.1:637->127.0.0.1:111]>, {sa_family=AF_INET, sin_port=htons(111), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 ETIMEDOUT (接続がタイムアウトしました) <40.993262>

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments