RHEL7.6: Booting a large system ends up entering emergency target due to storage devices timing out

Issue

• When booting a system having numerous storage devices and many CPUs, the boot ends up in emergency target instead of default target and many storage devices are failing in time out

  # journalctl -b | grep "timed out"
  [...]
  systemd[1]: Job dev-mapper-rootvg\x2dlv_var.device/start timed out.
  systemd[1]: Job dev-mapper-rootvg\x2dlv_home.device/start timed out.
  [...]
  

  Note: output may vary depending on the volume group abd logical volumes names used.

• Storage devices are discovered quickly upon boot, but device nodes are still not created after 1m30 (the default systemd start timeout)

  # journalctl -b
  
  ... Kernel booting message (kernel release may vary) ...
  Mar 01 14:57:07 XXX kernel: Linux version 3.10.0-957.5.1.el7.x86_64 (mockbuild@x86-019.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Wed Dec 19 10:46:58 EST 2018
  
  ... Storage devices messages (device names and properties may vary) ...
  Mar 01 14:57:09 XXX kernel: sd 0:1:0:0: [sda] Write Protect is off
  [...]
  Mar 01 14:57:25 XXX kernel: sd 14:0:15:76: [sdxd] Write Protect is off
  
  ... Devices timing out (device names may vary) ...
  Mar 01 14:58:59 XXX systemd[1]: Job dev-mapper-rootvg\x2dlv_var.device/start timed out.
  

  In the example above, we can see that storage devices were all seen by the kernel after 18 seconds, but the LVM device mapper built on these storage devices were not yet available after more than 1min30 since booting.