RHEL 8.6 installation fails while booting from kickstart file. The host drops into emergency dracut shell

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 8.10
  • IBM Power 9
  • LPAR

Issue

  • We have a working kickstart file which deploys RHEL 8.6 on an LPAR on a Power 9 hosts. I have updated the kickstart to point to 8.10, and updated the kernel and initrd to match. Once booted, the host will fail, dropping into an emergency dracut shell.

Resolution

Change the 'inst.stage2' command line parameter From:

inst.stage2=/tftpboot/<hostname>

To:

inst.stage2=http://<server_name_which_holds_anacondas_stage2_image>/<hostname>

Root Cause

  • The error on /dev/root indicates that the code running in the initramfs was not able to find the root file system, which is at this step the Stage 2 image.
  • The boot command line from the 'rdsosreport.txt' is:
BOOT_IMAGE=/tftpboot/<hostname>/vmlinuz ro inst.ks=http://<dummy_IP_adress_http>/cblr/svc/op/ks/system/<hostname> inst.text inst.geoloc=0 ip=<dummy_host_name_and_IP_addresses>eth0:none rd.debug net.ifnames=0 gateway=10.27.121.129 KSnetwork=no dns=10.240.47.3 nameserver=10.240.41.3 nameserver=10.240.47.3 inst.stage2=/tftpboot/<hostname> ipv6.disable=1
  • Note the 'inst.stage2' parameter, but the parameter value is not a URI, just a path. Indeed, as per the documentation, the 'inst.stage2' scheme is defaulting to 'hd:LABEL=RHEL-x-0-0-BaseOS-x86_64' pattern, which searches on the DVD (which is not applicable in this case since system is getting installed through a pure network), or is a HTTP or FTP URI.
  • So in this case it should be something like this:
inst.stage2=http://xxxx/<hostname>

where the URI would point to the base where a 'images/install.img' file is stored.

what is Anaconda's stage 2 ?
Anaconda installer is divided in two stages:
- Stage 1: Very small boot setup i.e. grub + initrd + kernel. Under this stage system just boots and starts the installation. If stage 1 is missing then system will stop booting at very early stage.
- Stage 2: Full installer i.e. all the necessary packages, GUI/TUI, storage, network tools etc. This full installation image (stage2) could be on DVD, ISO, NFS, network URL etc. If stage 2 is missing then Anaconda does not have its environment which is necessary to assemble the root device, initialize the GUI/TUI etc.

Diagnostic Steps

All the pointers listed below are saying the system is not able to find/assemble the root device at the time of installation.

  • Pointers:

1) It is dropping right after the 'dracut-initque' hook. Dracut-initqueue's job is to prepare the root, swap device etc.
2) Since system has reached till the 'dracut-initqueue' that means udev stage has been completed without any issue.
3) This (below given) particular script of 'dracut-initqueue' hook which is looking for '/dev/root' is always returning 1 (failure):

[  191.941870] <dummy_host_name_and_IP_addresses> dracut-initqueue[845]: /lib/dracut-lib.sh@447(check_finished): '[' -e '/lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2froot.sh' ']'
[  191.942179] <dummy_host_name_and_IP_addresses> dracut-initqueue[7970]: /lib/dracut-lib.sh@447(check_finished): . '/lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2froot.sh'
[  191.942179] <dummy_host_name_and_IP_addresses> dracut-initqueue[7970]: //lib/dracut/hooks/initqueue/finished/devexists-\x2fdev\x2froot.sh@1(source): '[' -e /dev/root ']'
[  191.942554] <dummy_host_name_and_IP_addresses> dracut-initqueue[845]: /lib/dracut-lib.sh@447(check_finished): return 1


$ grep -i '/lib/dracut-lib.sh@447(check_finished): return 1' without_debug_trace_rsosreport.txt  | wc -l
861

This is what is leading to the emergency shell since it is not able to assemble the root device for the installation.

  • The scsci (sd*) disks and mpath devices have been correctly recognized by the kernel - the below listed output is from 'rdsosreport.txt':
+ ls -l /dev/disk/by-id /dev/disk/by-label /dev/disk/by-path /dev/disk/by-uuid
/dev/disk/by-id:
total 0
lrwxrwxrwx 1 root root 10 Apr 17 20:24 dm-name-mpatha -> ../../dm-0
lrwxrwxrwx 1 root root 10 Apr 17 20:24 dm-uuid-mpath-3600507681081829ba800000000000100 -> ../../dm-0
lrwxrwxrwx 1 root root 10 Apr 17 20:24 scsi-3600507681081829ba800000000000100 -> ../../dm-0
lrwxrwxrwx 1 root root 10 Apr 17 20:24 wwn-0x600507681081829ba800000000000100 -> ../../dm-0

/dev/disk/by-label:
total 0
lrwxrwxrwx 1 root root 9 Apr 17 20:24 config-2 -> ../../sr0

/dev/disk/by-path:
total 0
lrwxrwxrwx 1 root root 9 Apr 17 20:24 fc-0x500507681015532f-lun-0 -> ../../sdf
lrwxrwxrwx 1 root root 9 Apr 17 20:24 fc-0x5005076810155375-lun-0 -> ../../sde
lrwxrwxrwx 1 root root 9 Apr 17 20:24 fc-0x500507681017532f-lun-0 -> ../../sdc
lrwxrwxrwx 1 root root 9 Apr 17 20:24 fc-0x5005076810175375-lun-0 -> ../../sdd
lrwxrwxrwx 1 root root 9 Apr 17 20:24 fc-0x500507681025532f-lun-0 -> ../../sdg
lrwxrwxrwx 1 root root 9 Apr 17 20:24 fc-0x5005076810255375-lun-0 -> ../../sdh
lrwxrwxrwx 1 root root 9 Apr 17 20:24 fc-0x500507681027532f-lun-0 -> ../../sdb
lrwxrwxrwx 1 root root 9 Apr 17 20:24 fc-0x5005076810275375-lun-0 -> ../../sda

/dev/disk/by-uuid:
total 0
lrwxrwxrwx 1 root root 9 Apr 17 20:24 <dummy_date>-00 -> ../../sr0
+ for _i in /etc/conf.d/*.conf
+ '[' -f /etc/conf.d/systemd.conf ']'
+ echo /etc/conf.d/systemd.conf
/etc/conf.d/systemd.conf
+ cat /etc/conf.d/systemd.conf
+ sed -e 's/\(ftp:\/\/.*\):.*@/\1:*******@/g;s/\(cifs:\/\/.*\):.*@/\1:*******@/g;s/cifspass=[^ ]*/cifspass=*******/g;s/iscsi:.*@/iscsi:******@/g;s/rd.iscsi.password=[^ ]*/rd.iscsi.password=******/g;s/rd.iscsi.in.password=[^ ]*/rd.iscsi.in.password=******/g'
systemdutildir="/usr/lib/systemd"
systemdsystemunitdir="/usr/lib/systemd/system"
systemdsystemconfdir="/etc/systemd/system"
+ command -v lvm
+ lvm pvdisplay
  WARNING: locking_type (4) is deprecated, using --sysinit --readonly.
+ lvm vgdisplay
  WARNING: locking_type (4) is deprecated, using --sysinit --readonly.
+ lvm lvdisplay
  WARNING: locking_type (4) is deprecated, using --sysinit --readonly.
+ command -v dmsetup
+ dmsetup ls --tree
mpatha (253:0)
 |- (8:96)
 |- (8:80)
 |- (8:32)
 |- (8:16)
 |- (8:112)
 |- (8:64)
 |- (8:48)
 `- (8:0)
+ cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] [linear] 
unused devices: <none>
  • So it is clear that the issue lies in system is not able to assemble the '/dev/root' device. In booting procedure kernel refers to root filesystem as '/dev/root' and down the line through initramfs the '/dev/root' gets resolved to the actual root device such as scsi (sd), nvme, UUID, mpath etc.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments