How to enter rescue mode on IBM Z Virtualization

Resolution

Disclaimer Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content. For additional assistance, please engage your support representative for IBM products.

Red Hat Enterirpse Linux 7 and 8

To enter a rescue environment, start an interactive Linux installation. For example, complete the following steps to enter a rescue environment on the LINUX2 virtual machine:

1. Log on to the target Linux system by using the 3270 terminal console: for example, LINUX2. For the question DO YOU WANT TO IPL LINUX FROM B747 AS STORAGE STORAGE WWPN 50050763 09141145 AT LOOKUP NUMBER 40014022 00000000? Y/N that is asked by the REXX script that is loaded by the PROFILE EXEC, answer n:

   LOGON LINUX2
   00: NIC 0600 is created; devices 0600-0602 defined
   00: NIC 0620 is created; devices 0620-0622 defined
   00: NIC 0640 is created; devices 0640-0642 defined
   00: z/VM Version 7 Release 1.0, Service Level 2001 (64-bit),
   00: built on IBM Virtualization Technology
   00: There is no logmsg data
   00: FILES: 0003 RDR, NO PRT, NO PUN
   00: LOGON AT 15:29:02 EDT WEDNESDAY 10/21/20
   00: Command complete
   00: Command complete
   z/VM V7.1.0 2020-03-13 12:46
   00: RPIMGR031E RESOURCE LNXADMIN SPECIFIED BY SPOOL COMMAND NOT FOUND
   LINUX2 AT RDBKZVM3 VIA * 2020-10-21 15:29:02 EDT WEDNESDAY
   DMSACP723I T (592) R/O
   DMSACR723I D (LNX:LNXADMIN.) R/O
   DMSACR723I B (LNX:LNXADMIN.REDHAT82) R/O
   DMSACR723I C (LNX:LNXADMIN.SWAPGEN) R/O
   SWPSWA0012I DIAG swap disk defined at virtual address 0300 with
   SWPSWA0012I 64988 4K pages of swap space.
   SWPSWA0012I DIAG swap disk defined at virtual address 0301 with
   SWPSWA0012I 129980 4K pages of swap space.
   DO YOU WANT TO IPL LINUX FROM B747 AS
   STORAGE WWPN 50050763 09141145 AT LOOKUP NUMBER 40014022 00000000? Y/N
   n
   

2. Create a copy of the LINUX1 PRM file on the same local disk LNX:LINUX2. Rename it to RESCUE PRM:

   ===> copyfile linux1 prm a rescue prm a
   

3. Create a copy of the REDHAT EXEC file from LNX:LNXADMIN. to the local disk LNX:LINUX1. Rename it to RESCUE EXEC:

   ===> copyfile redhat exec a rescue exec a
   

4. Edit the RESCUE EXEC file and replace the GENERIC PRM with the RESCUE PRM parameter:

   /* */
   'CL RDR'
   'PURGE RDR ALL'
   'SPOOL PUNCH * RDR'
   'PUNCH KERNEL IMG * (NOH'
   'PUNCH RESCUE PRM * (NOH'
   'PUNCH INITRD IMG * (NOH'
   'CH RDR ALL KEEP NOHOLD'
   'I 00C'
   

5. Edit the GENERIC PRM file and replace any kickstart or Virtual Network Computing (VNC) lines with the rescue command-line option:

   ===> x rescue prm a
   ro ramdisk_size=47000 cio_ignore=all,!condev
   inst.repo=http://lnxadmin:lnx4rdbk@X.X.X.X/rhel82
   rd.znet=qeth,0.0.0640,0.0.0641,0.0.0642,layer2=1,portno=0,portname=Z
   ip=X.X.X.X::X.X.X.X:24:linux2.hostname.com:enc640:none
   nameserver=X.X.X.X nameserver=X.X.X.X
   rd.zfcp=0.0.b747,0x5005076309141145,0x4001402200000000
   inst.cmdline
   rescue
   

6. Run the RESCUE EXEC to start the rescue environment:

   ===> rescue exec a
   00: 0000003 FILES CHANGED
   [ 0.101778] Linux version 4.18.0-193.el8.s390x
   (mockbuild@s390-018.build.eng.bos.redhat.com) (gcc version 8.3.1 20191121 (Red
   Hat
    8.3.1-5) (GCC)) #1 SMP Fri Mar 27 14:43:09 UTC 2020
   [ 0.101781] setup: Linux is running as a z/VM guest operating system in
   64-bit mode
   [ 0.101841] setup: The maximum memory size is 8192MB
   [ 0.101843] cma: Reserved 4 MiB at 0x00000001ffc00000
   [ 0.101867] numa: NUMA mode: plain
   [ 0.102019] cpu: 2 configured CPUs, 0 standby CPUs
   [ 0.102155] Write protected kernel read-only data: 9456k
   [ 0.102204] Zone ranges:
   [ 0.102205] DMA [mem 0x0000000000000000-0x000000007fffffff]
   [ 0.102207] Normal [mem 0x0000000080000000-0x00000001ffffffff]
   [ 0.102208] Movable zone start for each node
   [ 0.102209] Early memory node ranges
   [ 0.102210] node 0: [mem 0x0000000000000000-0x00000001ffffffff]
   [ 0.102212] Initmem setup node 0 [mem 0x0000000000000000-0x00000001ffffffff]
   [ 0.159615] percpu: Embedded 34 pages/cpu s99584 r8192 d31488 u139264
   [ 0.159644] Built 1 zonelists, mobility grouping on. Total pages: 2064384
   [ 0.159644] Policy zone: Normal
   [ 0.159646] Kernel command line: ro ramdisk_size=47000
   cio_ignore=all,!condev inst.repo=http:/
   /lnxadmin:lnx4rdbk@X.X.X.X/rhel82
   rd.znet=qeth,0.0.0640,0.0.0641,0.0.0642,layer2=1,portno=0,portname=Z
   ip=X.X.X.X::X.X.X.X:24:linux2.pbm.ihost.com:enc640:none
   nameserver=X.X.X.X nameserver=X.X.X.X
   rd.zfcp=0.0.b747,0x5005076309141145,0x4001402200000000
   inst.cmdline
    rescue
   ...
   Reloading system manager configuration
   Starting default target
   [ 31.266141] anaconda[2036]: Starting installer, one moment...
   [ 31.266235] anaconda[2036]: 19:56:43 Please ssh install@linux2.pbm.ihost.com
   (129.40.23.144) to begin the install.
   

7. The rescue process directs you to a Secure Shell (SSH) to connect to the IP address of your Linux server to begin the rescue environment. Use an SSH client to connect to the IP address and log in as user install.

8. In the next instructions, the rescue environment prompts you to continue to mount the partitions that it finds on /mnt/sysimage, mount the found partitions as Read-Only, or skip this step:

   Starting installer, one moment...
   anaconda 29.19.2.17-1.el8 for Red Hat Enterprise Linux 8.2 started.
    * installation log files are stored in /tmp during the installation
    * shell is available in second TMUX pane (ctrl+b, then press 2)
    * if the graphical installation interface fails to start, try again with the
    inst.text bootoption to start text installation
    * when reporting a bug add logs from /tmp as separate text/plain attachments
   ===============================================================================
   =
   ===============================================================================
   =
   Rescue
   The rescue environment will now attempt to find your Linux installation and
   mount it under the directory : /mnt/sysimage. You can then make any changes
   required to your system. Choose '1' to proceed with this step.
   You can choose to mount your file systems read-only instead of read-write by
   choosing '2'.
   If for some reason this process does not work choose '3' to skip directly to a
   shell.
   1) Continue
   2) Read-only mount
   3) Skip to shell
   4) Quit (Reboot)
   Please make a selection from the above:
   

   • Note: If you use PuTTY, under Settings → Window → Translation, change Character Set Translation on Received Data to UTF-8 for a better user experience.

9. Choose 1 to continue. The following instructions appear:

   ======================================================================
   ======================================================================
   Rescue Shell
   Your system has been mounted under /mnt/sysimage.
   If you would like to make the root of your system the root of the active
   system,
   run the command:
    chroot /mnt/sysimage
   When finished, please exit from the shell and your system will reboot.
   Please press ENTER to get a shell:
    <Tab>/<Alt-Tab> between elements | <Space> selects | <F12> next screen
   [anaconda] 1:main* 2:shell 3:log 4:storage-log 5:program-log
   

10. Press Enter. Your system was mounted under /mnt/sysimage. The rescue environment presents a command-line prompt:

    sh-4.4#
    When finished please exit from the shell and your system will reboot.
    

    • Note: If you want to use system tools to modify your system, use the chroot command line:

      sh-4.4# chroot /mnt/sysimage/
      bash-4.4# ls
      bin dev home lib64 mnt proc run serial-getty@hvc0.service sys usr
      boot etc lib media opt root sbin srv tmp var
      

• Note To exit the rescue mode, enter exit. Return to the 3270 console to enter the following command; otherwise, the rescue environment is loaded again:

  ===> #CP IPL CMS
  

• Note If the rescue image cannot find your partition, you can attempt to mount it manually by using the mount command; for example, if the Linux system did not mount logical volumes (LVs) automatically. To mount failed Logical Volume Manager (LVM) systems, enter the following commands:

  bash-4.4# vgchange -ay
   1 logical volume(s) in volume group "vg_system" now active
  bash-4.4# vgs
   VG #PV #LV #SN Attr VSize VFree
   vg_system 1 1 0 wz--n- <59.00g 0
  bash-4.4# ls /dev/mapper/
  control live-base live-rw mpatha mpatha1 mpatha2 vg_system-root
  sh-4.4# mount /dev/mapper/vg_system-root /mnt/sysimage
  sh-4.4# ls /mnt/sysimage/
  bin dev home lib64 mnt proc run serial-getty@hvc0.service sys usr
  boot etc lib media opt root sbin srv tmp var
  

  • Now, you can modify the files on the vg_system-root logical volume.

Red Hat Enterprise Linux 6

To enter a rescue environment, initiate an interactive Linux installation. Perform the following steps to enter a rescue environment on the RH64GOLD virtual machine:

1. Log on to LNXMAINT. Copy the RHEL64 EXEC file to a new file named RESCUE EXEC, and copy the user’s PARM-RH6 file to a new file (RH64GOLD RESCUE in this example):

   ==> copy rhel64 exec d rescue = =
   ==> copy rh64gold parm-rh6 d = rescue =
   

2. Edit RESCUE EXEC to point to the new RESCUE file:

   ==> x rescue exec d
   ...
   Address 'COMMAND'
    'CP SPOOL PUN *'
    'CP CLOSE RDR'
    'CP PURGE RDR ALL'
    'PUNCH RHEL64 KERNEL * (NOHEADER'
    'PUNCH' 'USERID'() 'RESCUE * (NOHEADER'
    'PUNCH RHEL64 INITRD * (NOHEADER'
    'CP CHANGE RDR ALL KEEP'
    'CP IPL 00C CLEAR'
    Exit
   

3. Edit the RH64GOLD RESCUE file, replacing any kickstart or VNC lines with the rescue command-line option:

   ==> x rh64gold rescue d
   root=/dev/ram0 ro ip=off ramdisk_size=40000
   CMSDASD=191 CMSCONFFILE=RH64GOLD.CONF-RH6
   rescue
   

4. Log off from LNXMAINT.

5. Log on to RH64GOLD and answer no to IPL from 100 question.

6. Define storage to 1 GB.

   ==> def stor 1g
   00: STORAGE = 1G
   00: Storage cleared - system reset.
   Enter
   

7. IPL CMS:

   ==> ipl cms
   

8. Run the RESCUE EXEC:

   ==> rescue
   00: NO FILES PURGED
   00: RDR FILE 0001 SENT FROM RH62GOLD PUN WAS 0001 RECS 113K CPY 001 A NOHOLD NO
   KEEP
   00: RDR FILE 0005 SENT FROM RH62GOLD PUN WAS 0005 RECS 0003 CPY 001 A NOHOLD NO
   KEEP
   00: RDR FILE 0009 SENT FROM RH62GOLD PUN WAS 0009 RECS 224K CPY 001 A NOHOLD NO
   KEEP
   00: 0000003 FILES CHANGED
   00: 0000003 FILES CHANGED
   ...
   Kernel command line: root=/dev/ram0 ro ip=off ramdisk_size=40000
    CMSDASD=191 CMSCONFFILE=RH64GOLD.CONF-RH6
    rescue
   ...
   Starting sshd to allow login over the network.
   Connect now to 9.60.18.144 and log in as user 'install' to start the installatio
   n.
   E.g. using: ssh -x install@9.60.18.144
   For VNC or text mode, disable X11 forwarding (recommended) with 'ssh -x'.
   For X11, enable X11 forwarding with 'ssh -X'.
   You may log in as the root user to start an interactive shell.
   

9. The installation process directs you to Telnet or Secure Shell (SSH) to the IP address of your Linux server to begin the first stage of the installation. Use SSH to connect to the IP address and log in as install.

10. You should get an SSH session and see a Choose a Language panel. Choose your language.

11. The rescue environment will prompt you for the location of the rescue image, which is located in the installation tree on the Linux administration system. Choose NFS directory, then enter the IP address of the Linux administration system and the path /var/nfs/rhel64.

    +------------------------------¦ NFS Setup +------------------------------+
    ¦                                                                         ¦
    ¦ Please enter the server and NFSv3 path to your Red Hat Enterprise Linux ¦
    ¦ installation image and optionally additional NFS mount options.         ¦
    ¦                                                                         ¦
    ¦ NFS server name: X.X.X.X_____________                                   ¦
    ¦ Red Hat Enterprise Linux directory: /var/nfs/rhel64_________            ¦
    

12. The Rescue window appears. Choose Continue. The rescue image will search for your Linux installation.

13. Hopefully, it will prompt you to mount the partitions that it finds:

    +--------------¦ Rescue +---------------+
    ¦                                       ¦
    ¦ Your system has been mounted under    ¦
    ¦ /mnt/sysimage.                        ¦
    ¦                                       ¦
    ¦ Press <return> to get a shell. If you ¦
    ¦ would like to make your system the    ¦
    ¦ root environment, run the command:    ¦
    ¦                                       ¦
    ¦        chroot /mnt/sysimage           ¦
    ¦                                       ¦
    ¦ The system will reboot automatically  ¦
    ¦ when you exit from the shell.         ¦
    

• Note Type exit to leave the shell and exit rescue mode.

• Note If the rescue image cannot find your partition, you can try to mount it yourself with the mount command. For example:

  # mount /dev/dasda1 /mnt/runtime/
  # ls /mnt/runtime/
  bin home media root sys
  boot lib mnt sbin tmp
  dev lib64 opt selinux usr
  etc lost+found proc srv var
  

Red Hat Enterprise Linux 5

To enter a rescue environment, initiate an interactive Linux installation. Perform the following steps to enter a rescue environment on the LINUX03 user ID;

1. Logon to LNXMAINT. Copy the RHEL5 EXEC file to a new file named RESCUE EXEC, and copy the user’s PARM-RH5 file to a new file (LINUX03 RESCUE, in this example):

   ==> copy rhel52 exec d rescue = =
   ==> copy linux03 parm-rh5 d = rescue =
   

2. Next, edit RESCUE EXEC to point to the new LINUX03 RESCUE file:

   ==> xedit RESCUE EXEC
   ...
   00005 'PUN RHEL52 KERNEL * (NOH'
   00006 'PUN' userid() ' RESCUE * (NOH'
   00007 'PUN RHEL52 INITRD * (NOH'
   ...
   ==> file
   

3. Edit the LINUX03 RESCUE file, replacing any kickstart or VNC lines with the rescue command line option:

   ==> xedit LINUX03 RESCUE
   ramdisk_size=40000 root=/dev/ram0 ro ip=off
   CMSDASD=191 CMSCONFFILE= LINUX03.CONF-RH5
   rescue
   

4. Now logoff from LNXMAINT and logon to LINUX03. Respond no to the IPL from DASD question, and run RESCUE EXEC;

   ==> rescue
   Linux version 2.6.18-92.el5 (brewbuilder@spark.z900.redhat.com) (gcc version 4.1
   .2 20071124 (Red Hat 4.1.2-41)) #1 SMP Tue Apr 29 13:16:58 EDT 2008
   We are running under VM (64 bit mode)
   Detected 4 CPU's
   Boot cpu address 0
   Built 1 zonelists. Total pages: 524288
   Kernel command line: ramdisk_size=40000 root=/dev/ram0 ro ip=off
    CMSDASD=191 CMSCONFFILE=Z12.conf
    rescue
   ...
   

5. The install process directs you to telnet or SSH to the IP address of your Linux server to begin the first stage of the installation. Use SSH to connect and log in as root. After choosing the language, the rescue environment will prompt you for the location of the rescue image, which is located in the install tree on the controller.

6. Choose NFS image, then enter the IP address or host name of the controller and the path /nfs/rhel5.2

7. The rescue image will search for your Linux installation and will prompt you to mount the partitions it finds. When prompted, select Continue to mount the partitions. After this, you will be presented with a shell prompt. Because you chose to mount your root partition, it will be automatically mounted to the /mnt/sysimage directory.

   • You can use the chroot command to run an interactive bash shell from the /mnt/sysimage directory. To do this, run the following command:

     sh-3.2# chroot /mnt/sysimage /bin/bash
     

   • The /bin/bash argument instructs chroot to run the bash shell after changing the root directory to /mnt/sysimage. From the chroot shell, the system appears almost exactly as it would if it were running (the only difference is that, because udev is not running, there may be no device nodes under /dev). If you need to rewrite the master boot record (MBR), run zipl from the chroot shell. To exit the shell, type: exit.

   • If the rescue image cannot find your partition, or if you chose Skip when prompted to search for the existing file systems, you can mount the file systems manually.

   • To manually mount the filesystems, first, bring the LVM logical volumes online with the following two commands:

     sh-3.2# lvm vgscan
     Reading all physical volumes. This may take a while...
       Found volume group "system_vg" using metadata type lvm2
     sh-3.2# lvm vgchange -a y
       5 logical volume(s) in volume group "system_vg" now active
     

   • When the LVM is online, create a mount point, and then mount the file systems.

     sh-3.2# mkdir /mnt/sysimage
     sh-3.2# mount /dev/system_vg/root_lv /mnt/sysimage/
     sh-3.2# mount /dev/system_vg/tmp_lv /mnt/sysimage/tmp
     sh-3.2# mount /dev/system_vg/opt_lv /mnt/sysimage/opt
     sh-3.2# mount /dev/system_vg/var_lv /mnt/sysimage/var
     sh-3.2# mount /dev/system_vg/usr_lv /mnt/sysimage/usr
     

   • Lastly, mount the /boot file system located on dasda1:

     sh-3.2# mount /dev/dasda1 /mnt/sysimage/boot
     

8. When finished, type: exit to leave the chroot shell, then type: exit again to leave rescue mode.

Red Hat Enterprise Linux 4

To enter a rescue environment, initiate an interactive Linux installation.

1. Run the rh4u3x exec. The install process directs you to telnet or SSH to the IP address of your Linux server to begin the first stage of the installation.
2. Use SSH to connect to the IP address and log in as root.

3. Before answering any questions in this window, use SSH in a new window to connect to
   the same IP address. This second SSH session brings you to a shell prompt:

   Welcome to the Red Hat Linux install environment 1.1 for zSeries
   loader has already been run. Starting shell.
   -/bin/sh-3.00#
   

4. Return to the first SSH session, and proceed through stage one of the interactive installation until the Virtual Network Computing (VNC) server is up and displays something similar to the following;

   Welcome to the Red Hat Linux install environment 1.1 for zSeries
   Running anaconda, the Red Hat Enterprise Linux system installer - please wait...
   Starting VNC...
   The VNC server is now running.
   Please connect to 10.1.40.99:1 to begin the install...
   Starting graphical installation...
   XKB extension not present on :1
   

5. Do not use VNC to connect to the IP address. Instead, return to the shell prompt of the second SSH session. At this point, you can access your root file system by its device node (/dev/dasda1 for example). To scan the root file system for errors, use the e2fsck command similar to the following;

   -/bin/sh-3.00# e2fsck /dev/dasda1
   e2fsck 1.35 (28-Feb-2004)
   /: recovering journal
   /: clean, 24005/142560 files, 171477/284928 blocks
   

6. To mount the root file system, first create the directory /mnt/sysimage, then use the mount command:

   -/bin/sh-3.00# mkdir /mnt/sysimage
   -/bin/sh-3.00# mount /dev/dasda1 /mnt/sysimage/
   

You can now access your root file system through this mount point. If additional DASD need to be onlined, Please refer to IBM Redbooks and/or your respective IBM support representative for additional assistance.