How to enter rescue mode on IBM Z Virtualization

Solution Unverified - Updated -

Environment

  • Red Hat Enterprise Linux 4
  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 8
  • IBM Z Virtualization

Issue

  • How do I boot into rescue mode in my Red Hat Enterprise Linux system on z/VM?

Resolution

Disclaimer Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content. For additional assistance, please engage your support representative for IBM products.

Red Hat Enterirpse Linux 7 and 8

To enter a rescue environment, start an interactive Linux installation. For example, complete the following steps to enter a rescue environment on the LINUX2 virtual machine:

  1. Log on to the target Linux system by using the 3270 terminal console: for example, LINUX2. For the question DO YOU WANT TO IPL LINUX FROM B747 AS STORAGE STORAGE WWPN 50050763 09141145 AT LOOKUP NUMBER 40014022 00000000? Y/N that is asked by the REXX script that is loaded by the PROFILE EXEC, answer n:

    LOGON LINUX2
    00: NIC 0600 is created; devices 0600-0602 defined
    00: NIC 0620 is created; devices 0620-0622 defined
    00: NIC 0640 is created; devices 0640-0642 defined
    00: z/VM Version 7 Release 1.0, Service Level 2001 (64-bit),
    00: built on IBM Virtualization Technology
    00: There is no logmsg data
    00: FILES: 0003 RDR, NO PRT, NO PUN
    00: LOGON AT 15:29:02 EDT WEDNESDAY 10/21/20
    00: Command complete
    00: Command complete
    z/VM V7.1.0 2020-03-13 12:46
    00: RPIMGR031E RESOURCE LNXADMIN SPECIFIED BY SPOOL COMMAND NOT FOUND
    LINUX2 AT RDBKZVM3 VIA * 2020-10-21 15:29:02 EDT WEDNESDAY
    DMSACP723I T (592) R/O
    DMSACR723I D (LNX:LNXADMIN.) R/O
    DMSACR723I B (LNX:LNXADMIN.REDHAT82) R/O
    DMSACR723I C (LNX:LNXADMIN.SWAPGEN) R/O
    SWPSWA0012I DIAG swap disk defined at virtual address 0300 with
    SWPSWA0012I 64988 4K pages of swap space.
    SWPSWA0012I DIAG swap disk defined at virtual address 0301 with
    SWPSWA0012I 129980 4K pages of swap space.
    DO YOU WANT TO IPL LINUX FROM B747 AS
    STORAGE WWPN 50050763 09141145 AT LOOKUP NUMBER 40014022 00000000? Y/N
    n
    
  2. Create a copy of the LINUX1 PRM file on the same local disk LNX:LINUX2. Rename it to RESCUE PRM:

    ===> copyfile linux1 prm a rescue prm a
    
  3. Create a copy of the REDHAT EXEC file from LNX:LNXADMIN. to the local disk LNX:LINUX1. Rename it to RESCUE EXEC:

    ===> copyfile redhat exec a rescue exec a
    
  4. Edit the RESCUE EXEC file and replace the GENERIC PRM with the RESCUE PRM parameter:

    /* */
    'CL RDR'
    'PURGE RDR ALL'
    'SPOOL PUNCH * RDR'
    'PUNCH KERNEL IMG * (NOH'
    'PUNCH RESCUE PRM * (NOH'
    'PUNCH INITRD IMG * (NOH'
    'CH RDR ALL KEEP NOHOLD'
    'I 00C'
    
  5. Edit the GENERIC PRM file and replace any kickstart or Virtual Network Computing (VNC) lines with the rescue command-line option:

    ===> x rescue prm a
    ro ramdisk_size=47000 cio_ignore=all,!condev
    inst.repo=http://lnxadmin:lnx4rdbk@X.X.X.X/rhel82
    rd.znet=qeth,0.0.0640,0.0.0641,0.0.0642,layer2=1,portno=0,portname=Z
    ip=X.X.X.X::X.X.X.X:24:linux2.hostname.com:enc640:none
    nameserver=X.X.X.X nameserver=X.X.X.X
    rd.zfcp=0.0.b747,0x5005076309141145,0x4001402200000000
    inst.cmdline
    rescue
    
  6. Run the RESCUE EXEC to start the rescue environment:

    ===> rescue exec a
    00: 0000003 FILES CHANGED
    [ 0.101778] Linux version 4.18.0-193.el8.s390x
    (mockbuild@s390-018.build.eng.bos.redhat.com) (gcc version 8.3.1 20191121 (Red
    Hat
     8.3.1-5) (GCC)) #1 SMP Fri Mar 27 14:43:09 UTC 2020
    [ 0.101781] setup: Linux is running as a z/VM guest operating system in
    64-bit mode
    [ 0.101841] setup: The maximum memory size is 8192MB
    [ 0.101843] cma: Reserved 4 MiB at 0x00000001ffc00000
    [ 0.101867] numa: NUMA mode: plain
    [ 0.102019] cpu: 2 configured CPUs, 0 standby CPUs
    [ 0.102155] Write protected kernel read-only data: 9456k
    [ 0.102204] Zone ranges:
    [ 0.102205] DMA [mem 0x0000000000000000-0x000000007fffffff]
    [ 0.102207] Normal [mem 0x0000000080000000-0x00000001ffffffff]
    [ 0.102208] Movable zone start for each node
    [ 0.102209] Early memory node ranges
    [ 0.102210] node 0: [mem 0x0000000000000000-0x00000001ffffffff]
    [ 0.102212] Initmem setup node 0 [mem 0x0000000000000000-0x00000001ffffffff]
    [ 0.159615] percpu: Embedded 34 pages/cpu s99584 r8192 d31488 u139264
    [ 0.159644] Built 1 zonelists, mobility grouping on. Total pages: 2064384
    [ 0.159644] Policy zone: Normal
    [ 0.159646] Kernel command line: ro ramdisk_size=47000
    cio_ignore=all,!condev inst.repo=http:/
    /lnxadmin:lnx4rdbk@X.X.X.X/rhel82
    rd.znet=qeth,0.0.0640,0.0.0641,0.0.0642,layer2=1,portno=0,portname=Z
    ip=X.X.X.X::X.X.X.X:24:linux2.pbm.ihost.com:enc640:none
    nameserver=X.X.X.X nameserver=X.X.X.X
    rd.zfcp=0.0.b747,0x5005076309141145,0x4001402200000000
    inst.cmdline
     rescue
    ...
    Reloading system manager configuration
    Starting default target
    [ 31.266141] anaconda[2036]: Starting installer, one moment...
    [ 31.266235] anaconda[2036]: 19:56:43 Please ssh install@linux2.pbm.ihost.com
    (129.40.23.144) to begin the install.
    
  7. The rescue process directs you to a Secure Shell (SSH) to connect to the IP address of your Linux server to begin the rescue environment. Use an SSH client to connect to the IP address and log in as user install.

  8. In the next instructions, the rescue environment prompts you to continue to mount the partitions that it finds on /mnt/sysimage, mount the found partitions as Read-Only, or skip this step:

    Starting installer, one moment...
    anaconda 29.19.2.17-1.el8 for Red Hat Enterprise Linux 8.2 started.
     * installation log files are stored in /tmp during the installation
     * shell is available in second TMUX pane (ctrl+b, then press 2)
     * if the graphical installation interface fails to start, try again with the
     inst.text bootoption to start text installation
     * when reporting a bug add logs from /tmp as separate text/plain attachments
    ===============================================================================
    =
    ===============================================================================
    =
    Rescue
    The rescue environment will now attempt to find your Linux installation and
    mount it under the directory : /mnt/sysimage. You can then make any changes
    required to your system. Choose '1' to proceed with this step.
    You can choose to mount your file systems read-only instead of read-write by
    choosing '2'.
    If for some reason this process does not work choose '3' to skip directly to a
    shell.
    1) Continue
    2) Read-only mount
    3) Skip to shell
    4) Quit (Reboot)
    Please make a selection from the above:
    
    • Note: If you use PuTTY, under Settings → Window → Translation, change Character Set Translation on Received Data to UTF-8 for a better user experience.
  9. Choose 1 to continue. The following instructions appear:

    ======================================================================
    ======================================================================
    Rescue Shell
    Your system has been mounted under /mnt/sysimage.
    If you would like to make the root of your system the root of the active
    system,
    run the command:
     chroot /mnt/sysimage
    When finished, please exit from the shell and your system will reboot.
    Please press ENTER to get a shell:
     <Tab>/<Alt-Tab> between elements | <Space> selects | <F12> next screen
    [anaconda] 1:main* 2:shell 3:log 4:storage-log 5:program-log
    
  10. Press Enter. Your system was mounted under /mnt/sysimage. The rescue environment presents a command-line prompt:

    sh-4.4#
    When finished please exit from the shell and your system will reboot.
    
    • Note: If you want to use system tools to modify your system, use the chroot command line:

      sh-4.4# chroot /mnt/sysimage/
      bash-4.4# ls
      bin dev home lib64 mnt proc run serial-getty@hvc0.service sys usr
      boot etc lib media opt root sbin srv tmp var
      
  • Note To exit the rescue mode, enter exit. Return to the 3270 console to enter the following command; otherwise, the rescue environment is loaded again:

    ===> #CP IPL CMS
    
  • Note If the rescue image cannot find your partition, you can attempt to mount it manually by using the mount command; for example, if the Linux system did not mount logical volumes (LVs) automatically. To mount failed Logical Volume Manager (LVM) systems, enter the following commands:

    bash-4.4# vgchange -ay
     1 logical volume(s) in volume group "vg_system" now active
    bash-4.4# vgs
     VG #PV #LV #SN Attr VSize VFree
     vg_system 1 1 0 wz--n- <59.00g 0
    bash-4.4# ls /dev/mapper/
    control live-base live-rw mpatha mpatha1 mpatha2 vg_system-root
    sh-4.4# mount /dev/mapper/vg_system-root /mnt/sysimage
    sh-4.4# ls /mnt/sysimage/
    bin dev home lib64 mnt proc run serial-getty@hvc0.service sys usr
    boot etc lib media opt root sbin srv tmp var
    
    • Now, you can modify the files on the vg_system-root logical volume.

Red Hat Enterprise Linux 6

To enter a rescue environment, initiate an interactive Linux installation. Perform the following steps to enter a rescue environment on the RH64GOLD virtual machine:

  1. Log on to LNXMAINT. Copy the RHEL64 EXEC file to a new file named RESCUE EXEC, and copy the user’s PARM-RH6 file to a new file (RH64GOLD RESCUE in this example):

    ==> copy rhel64 exec d rescue = =
    ==> copy rh64gold parm-rh6 d = rescue =
    
  2. Edit RESCUE EXEC to point to the new RESCUE file:

    ==> x rescue exec d
    ...
    Address 'COMMAND'
     'CP SPOOL PUN *'
     'CP CLOSE RDR'
     'CP PURGE RDR ALL'
     'PUNCH RHEL64 KERNEL * (NOHEADER'
     'PUNCH' 'USERID'() 'RESCUE * (NOHEADER'
     'PUNCH RHEL64 INITRD * (NOHEADER'
     'CP CHANGE RDR ALL KEEP'
     'CP IPL 00C CLEAR'
     Exit
    
  3. Edit the RH64GOLD RESCUE file, replacing any kickstart or VNC lines with the rescue command-line option:

    ==> x rh64gold rescue d
    root=/dev/ram0 ro ip=off ramdisk_size=40000
    CMSDASD=191 CMSCONFFILE=RH64GOLD.CONF-RH6
    rescue
    
  4. Log off from LNXMAINT.

  5. Log on to RH64GOLD and answer no to IPL from 100 question.
  6. Define storage to 1 GB.

    ==> def stor 1g
    00: STORAGE = 1G
    00: Storage cleared - system reset.
    Enter
    
  7. IPL CMS:

    ==> ipl cms
    
  8. Run the RESCUE EXEC:

    ==> rescue
    00: NO FILES PURGED
    00: RDR FILE 0001 SENT FROM RH62GOLD PUN WAS 0001 RECS 113K CPY 001 A NOHOLD NO
    KEEP
    00: RDR FILE 0005 SENT FROM RH62GOLD PUN WAS 0005 RECS 0003 CPY 001 A NOHOLD NO
    KEEP
    00: RDR FILE 0009 SENT FROM RH62GOLD PUN WAS 0009 RECS 224K CPY 001 A NOHOLD NO
    KEEP
    00: 0000003 FILES CHANGED
    00: 0000003 FILES CHANGED
    ...
    Kernel command line: root=/dev/ram0 ro ip=off ramdisk_size=40000
     CMSDASD=191 CMSCONFFILE=RH64GOLD.CONF-RH6
     rescue
    ...
    Starting sshd to allow login over the network.
    Connect now to 9.60.18.144 and log in as user 'install' to start the installatio
    n.
    E.g. using: ssh -x install@9.60.18.144
    For VNC or text mode, disable X11 forwarding (recommended) with 'ssh -x'.
    For X11, enable X11 forwarding with 'ssh -X'.
    You may log in as the root user to start an interactive shell.
    
  9. The installation process directs you to Telnet or Secure Shell (SSH) to the IP address of your Linux server to begin the first stage of the installation. Use SSH to connect to the IP address and log in as install.

  10. You should get an SSH session and see a Choose a Language panel. Choose your language.
  11. The rescue environment will prompt you for the location of the rescue image, which is located in the installation tree on the Linux administration system. Choose NFS directory, then enter the IP address of the Linux administration system and the path /var/nfs/rhel64.

    +------------------------------¦ NFS Setup +------------------------------+
    ¦                                                                         ¦
    ¦ Please enter the server and NFSv3 path to your Red Hat Enterprise Linux ¦
    ¦ installation image and optionally additional NFS mount options.         ¦
    ¦                                                                         ¦
    ¦ NFS server name: X.X.X.X_____________                                   ¦
    ¦ Red Hat Enterprise Linux directory: /var/nfs/rhel64_________            ¦
    
  12. The Rescue window appears. Choose Continue. The rescue image will search for your Linux installation.

  13. Hopefully, it will prompt you to mount the partitions that it finds:

    +--------------¦ Rescue +---------------+
    ¦                                       ¦
    ¦ Your system has been mounted under    ¦
    ¦ /mnt/sysimage.                        ¦
    ¦                                       ¦
    ¦ Press <return> to get a shell. If you ¦
    ¦ would like to make your system the    ¦
    ¦ root environment, run the command:    ¦
    ¦                                       ¦
    ¦        chroot /mnt/sysimage           ¦
    ¦                                       ¦
    ¦ The system will reboot automatically  ¦
    ¦ when you exit from the shell.         ¦
    
  • Note Type exit to leave the shell and exit rescue mode.
  • Note If the rescue image cannot find your partition, you can try to mount it yourself with the mount command. For example:

    # mount /dev/dasda1 /mnt/runtime/
    # ls /mnt/runtime/
    bin home media root sys
    boot lib mnt sbin tmp
    dev lib64 opt selinux usr
    etc lost+found proc srv var
    

Red Hat Enterprise Linux 5

To enter a rescue environment, initiate an interactive Linux installation. Perform the following steps to enter a rescue environment on the LINUX03 user ID;

  1. Logon to LNXMAINT. Copy the RHEL5 EXEC file to a new file named RESCUE EXEC, and copy the user’s PARM-RH5 file to a new file (LINUX03 RESCUE, in this example):

    ==> copy rhel52 exec d rescue = =
    ==> copy linux03 parm-rh5 d = rescue =
    
  2. Next, edit RESCUE EXEC to point to the new LINUX03 RESCUE file:

    ==> xedit RESCUE EXEC
    ...
    00005 'PUN RHEL52 KERNEL * (NOH'
    00006 'PUN' userid() ' RESCUE * (NOH'
    00007 'PUN RHEL52 INITRD * (NOH'
    ...
    ==> file
    
  3. Edit the LINUX03 RESCUE file, replacing any kickstart or VNC lines with the rescue command line option:

    ==> xedit LINUX03 RESCUE
    ramdisk_size=40000 root=/dev/ram0 ro ip=off
    CMSDASD=191 CMSCONFFILE= LINUX03.CONF-RH5
    rescue
    
  4. Now logoff from LNXMAINT and logon to LINUX03. Respond no to the IPL from DASD question, and run RESCUE EXEC;

    ==> rescue
    Linux version 2.6.18-92.el5 (brewbuilder@spark.z900.redhat.com) (gcc version 4.1
    .2 20071124 (Red Hat 4.1.2-41)) #1 SMP Tue Apr 29 13:16:58 EDT 2008
    We are running under VM (64 bit mode)
    Detected 4 CPU's
    Boot cpu address 0
    Built 1 zonelists. Total pages: 524288
    Kernel command line: ramdisk_size=40000 root=/dev/ram0 ro ip=off
     CMSDASD=191 CMSCONFFILE=Z12.conf
     rescue
    ...
    
  5. The install process directs you to telnet or SSH to the IP address of your Linux server to begin the first stage of the installation. Use SSH to connect and log in as root. After choosing the language, the rescue environment will prompt you for the location of the rescue image, which is located in the install tree on the controller.

  6. Choose NFS image, then enter the IP address or host name of the controller and the path /nfs/rhel5.2
  7. The rescue image will search for your Linux installation and will prompt you to mount the partitions it finds. When prompted, select Continue to mount the partitions. After this, you will be presented with a shell prompt. Because you chose to mount your root partition, it will be automatically mounted to the /mnt/sysimage directory.

    • You can use the chroot command to run an interactive bash shell from the /mnt/sysimage directory. To do this, run the following command:

      sh-3.2# chroot /mnt/sysimage /bin/bash
      
    • The /bin/bash argument instructs chroot to run the bash shell after changing the root directory to /mnt/sysimage. From the chroot shell, the system appears almost exactly as it would if it were running (the only difference is that, because udev is not running, there may be no device nodes under /dev). If you need to rewrite the master boot record (MBR), run zipl from the chroot shell. To exit the shell, type: exit.

    • If the rescue image cannot find your partition, or if you chose Skip when prompted to search for the existing file systems, you can mount the file systems manually.
    • To manually mount the filesystems, first, bring the LVM logical volumes online with the following two commands:

      sh-3.2# lvm vgscan
      Reading all physical volumes. This may take a while...
        Found volume group "system_vg" using metadata type lvm2
      sh-3.2# lvm vgchange -a y
        5 logical volume(s) in volume group "system_vg" now active
      
    • When the LVM is online, create a mount point, and then mount the file systems.

      sh-3.2# mkdir /mnt/sysimage
      sh-3.2# mount /dev/system_vg/root_lv /mnt/sysimage/
      sh-3.2# mount /dev/system_vg/tmp_lv /mnt/sysimage/tmp
      sh-3.2# mount /dev/system_vg/opt_lv /mnt/sysimage/opt
      sh-3.2# mount /dev/system_vg/var_lv /mnt/sysimage/var
      sh-3.2# mount /dev/system_vg/usr_lv /mnt/sysimage/usr
      
    • Lastly, mount the /boot file system located on dasda1:

      sh-3.2# mount /dev/dasda1 /mnt/sysimage/boot
      
  8. When finished, type: exit to leave the chroot shell, then type: exit again to leave rescue mode.

Red Hat Enterprise Linux 4

To enter a rescue environment, initiate an interactive Linux installation.

  1. Run the rh4u3x exec. The install process directs you to telnet or SSH to the IP address of your Linux server to begin the first stage of the installation.
  2. Use SSH to connect to the IP address and log in as root.
  3. Before answering any questions in this window, use SSH in a new window to connect to
    the same IP address. This second SSH session brings you to a shell prompt:

    Welcome to the Red Hat Linux install environment 1.1 for zSeries
    loader has already been run. Starting shell.
    -/bin/sh-3.00#
    
  4. Return to the first SSH session, and proceed through stage one of the interactive installation until the Virtual Network Computing (VNC) server is up and displays something similar to the following;

    Welcome to the Red Hat Linux install environment 1.1 for zSeries
    Running anaconda, the Red Hat Enterprise Linux system installer - please wait...
    Starting VNC...
    The VNC server is now running.
    Please connect to 10.1.40.99:1 to begin the install...
    Starting graphical installation...
    XKB extension not present on :1
    
  5. Do not use VNC to connect to the IP address. Instead, return to the shell prompt of the second SSH session. At this point, you can access your root file system by its device node (/dev/dasda1 for example). To scan the root file system for errors, use the e2fsck command similar to the following;

    -/bin/sh-3.00# e2fsck /dev/dasda1
    e2fsck 1.35 (28-Feb-2004)
    /: recovering journal
    /: clean, 24005/142560 files, 171477/284928 blocks
    
  6. To mount the root file system, first create the directory /mnt/sysimage, then use the mount command:

    -/bin/sh-3.00# mkdir /mnt/sysimage
    -/bin/sh-3.00# mount /dev/dasda1 /mnt/sysimage/
    

You can now access your root file system through this mount point. If additional DASD need to be onlined, Please refer to IBM Redbooks and/or your respective IBM support representative for additional assistance.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments