How to generate sosreport from the rescue environment

Resolution

To generate the sosreport output from the rescue environment:

1. Boot : Boot the system with the installation DVD disc of the corresponding version of Red Hat Enterprise Linux

   NOTE: Starting with RHEL7, there are two "rescue modes" that are often confused with each other:

   • Rescue image: this is the initramfs-0-rescue-*.img boot image file available from the grub menu, and
   • Anaconda rescue mode: this is booting into rescue mode using the RHEL installation DVD media.

   For obtaining a sosreport in rescue mode the practice is to boot off of the installation DVD media for the kernel version being run on the system. Attempting to use the *rescue*.img grub menu item available within RHEL 7 or later will typically fail to produce a usable environment for the purposes of gathering a sosreport.

2. Mode:

   • In RHEL7 or later, select Troubleshooting, then Rescue a Red Hat Enterprise Linux system.
   • In RHEL6, select Rescue installed system in RHEL6.
   • In RHEL5 or earlier, enter linux rescue into the boot prompt. If the system is multipathed, enter linux rescue mpath

3. Language: (RHEL6 or earlier) Once the rescue environment finishes booting, choose a language to use, if prompted.

4. Keyboard: (RHEL6 or earlier) Choose a keyboard layout to use, if prompted.

5. Network: (RHEL6 or earlier) Wait for network interfaces to be located, and activate them, so that requested data can be transferred to another host using scp command.

6. The rescue environment will try to find the current Red Hat Enterprise Linux installation on the system. You will be prompted with the following options:

   • Continue: continue mounting all of its partitions under /mnt/sysimage/ in Read & Write mode in RHEL7 and /mnt/sysroot/ in RHEL8/9.
   • Read-Only: continue mounting all of its partitions under /mnt/sysimage/ in Read Only mode in RHEL7 and /mnt/sysroot/ in RHEL8/9.
   • Skip: skip the mounting of the discovered Red Hat Enterprise Linux installation and proceed with manual mounting

   Select Continue to generate sosreport. If you select Skip, you will have to manually mount your filesystem before performing the next step.

7. Sosreport: Run the following commands to continue generating sosreport:

   In RHEL7 or earlier:

   # chroot /mnt/sysimage
   # sosreport
   

   In RHEL8 or later:

   #  chroot /mnt/sysroot
   # sos report
   

   The sosreport or sos report command can take some time to generate a report. It collects a significant amount of information that may help Red Hat technicians resolve your issue.

   NOTE: If you are running RHEL 4.6 or higher and the sosreport command fails, you can try the sysreport command, as it should still be available. Note that this command will not be available on RHEL 5 systems or later.
   NOTE: In RHEL 8/9, Please note the sosreport command has been deprecated in favor of the new sos command, e.g. sos report. Currently sosreport is redirecting to sos report.

   While rescue mode will attempt to automatically bind all necessary mount points to /mnt/sysimage, sometimes this fails with the follwing error seen when attempting to run the sosreport command.

   error on sosreport: no such file or directory /dev/urandom
   

   If this error occurs, then type the following commands. This will first exit the chroot'ed environment and mount the missing mount points:

   # exit   <-- this will exit the chroot'ed environment
   # mount -o bind /dev /mnt/sysimage/dev
   # mount -o bind /sys /mnt/sysimage/sys
   # mount -o bind /proc /mnt/sysimage/proc
   # chroot /mnt/sysimage
   # sosreport
   

   NOTE: If df command does not show dev, sys and proc and/or other expected filesystems as present and mounted, and the above steps have been tried but failed to mount all necessary filesystems, then likely an advanced storage option needs to be activated before proceeding. See one of the following resources for additional information on resolving. The specific steps are highly dependent on the type of storage environment present.

   • RHEL 7: Anaconda Rescue Mode
   • RHEL 6: Basic System Recovery

8. Results: Once sosreport generation completes, it will provide the output in the following directory whlie the user is in the chroot environment.

   • For RHEL 6 or earlier in directory: /var/tmp directory
   • For RHEL 7 or later in directory: /mnt/sysimage/var/tmp directory while in the chroot environment.

   Warning: During the running of the command sosreport you will be prompted for your name and case number. Use only letters and/or numbers when filling out this field. Adding other characters could damage the system or render the report unusable.

9. Copy: Copy the resulting file from step 8 either over the network or on to a USB drive so that it can be uploaded to the support case.

   • Network: Use the scp command to copy the local file to a remote system.

     # scp /mnt/sysimage/var/tmp/sosreport username@hostname:sosreport
     

   • USB drive: Using a preformatted USB drive (in the following example the whole USB drive has been formatted with an ext3 filesystem), plug the drive into the system.

     • if the USB drive auto-mounts on /media/{name}, then:

       # cp /mnt/sysimage/var/tmp/*sosreport* /media/{name}
       # umount /media/{name}
       

     • if it doesn't auto-mount under the /media directory, then manually mount it:

       # fdisk -l                      <== locate the USB drive's /dev/sdX device name
       # mkdir /mnt/usbdrive
       # mount -t ext3 /dev/sdX /mnt/usbdrive
       # cp /mnt/sysimage/var/tmp/*sosreport* /mnt/usbdrive
       # umount /mnt/usbdrive
       

   NOTE: In RHEL7 or later, enable networking before executing scp.

10. Provide: "How to provide files to Red Hat Support (vmcore, rhev logcollector, sosreports, heap dumps, log files, etc.)"

More Info

Related documentation
For Red Hat Enterprise Linux 9: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/performing_a_standard_rhel_installation/troubleshooting-after-installation_installing-rhel#booting-into-rescue-mode_using-rescue-mode
For Red Hat Enterprise Linux 8: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/performing_a_standard_rhel_installation/troubleshooting-after-installation_installing-rhel#using-rescue-mode_troubleshooting-after-installation
For Red Hat Enterprise Linux 7: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/installation_guide/sect-rescue-mode
For Red Hat Enterprise Linux 6: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/installation_guide/ap-rescuemode#Rescue_Mode-x86