How to generate sosreport from the rescue environment

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 9
  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 4.6 and later 4.x

Issue

  • How to run sosreport in rescue mode?
  • How to generate sosreport for system that can not boot?
  • Red Hat Enterprise Linux system does not boot.
  • How to collect system information and logs for Technical Support to troubleshoot?
  • System hung or had a kernel panic and now it hangs or gives me an error on reboot, how can I create a sosreport?

Resolution

To generate the sosreport output from the rescue environment:

  1. Boot : Boot the system with the installation DVD disc of the corresponding version of Red Hat Enterprise Linux

    NOTE: Starting with RHEL7, there are two "rescue modes" that are often confused with each other:

    • Rescue image: this is the initramfs-0-rescue-*.img boot image file available from the grub menu, and
    • Anaconda rescue mode: this is booting into rescue mode using the RHEL installation DVD media.

    For obtaining a sosreport in rescue mode the practice is to boot off of the installation DVD media for the kernel version being run on the system. Attempting to use the *rescue*.img grub menu item available within RHEL 7 or later will typically fail to produce a usable environment for the purposes of gathering a sosreport.

  2. Mode:

    • In RHEL7 or later, select Troubleshooting, then Rescue a Red Hat Enterprise Linux system.
    • In RHEL6, select Rescue installed system in RHEL6.
    • In RHEL5 or earlier, enter linux rescue into the boot prompt. If the system is multipathed, enter linux rescue mpath
  3. Language: (RHEL6 or earlier) Once the rescue environment finishes booting, choose a language to use, if prompted.

  4. Keyboard: (RHEL6 or earlier) Choose a keyboard layout to use, if prompted.

  5. Network: (RHEL6 or erlier) Wait for network interfaces to be located, and activate them, so that requested data can be transferred to another host using scp command.

  6. The rescue environment will try to find the current Red Hat Enterprise Linux installation on the system. You will be prompted with the following options:

    • Continue: continue mounting all of its partitions under /mnt/sysimage/ in Read & Write mode in RHEL7 and /mnt/sysroot/ in RHEL8/9.
    • Read-Only: continue mounting all of its partitions under /mnt/sysimage/ in Read Only mode in RHEL7 and /mnt/sysroot/ in RHEL8/9.
    • Skip: skip the mounting of the discovered Red Hat Enterprise Linux installation and proceed with manual mounting

    Select Continue to generate sosreport. If you select Skip, you will have to manually mount your filesystem before performing the next step.

  7. Sosreport: Run the following commands to continue generating sosreport:

    In RHEL7 or earlier:

    # chroot /mnt/sysimage
    # sosreport
    

    In RHEL8 or later:

    #  chroot /mnt/sysroot
    # sos report
    

    The sosreport or sos report command can take some time to generate a report. It collects a significant amount of information that may help Red Hat technicians resolve your issue.

    NOTE: If you are running RHEL 4.6 or higher and the sosreport command fails, you can try the sysreport command, as it should still be available. Note that this command will not be available on RHEL 5 systems or later.
    NOTE: In RHEL 8/9, Please note the sosreport command has been deprecated in favor of the new sos command, e.g. sos report. Currently sosreport is redirecting to sos report.

    While rescue mode will attempt to automatically bind all necessary mount points to /mnt/sysimage, sometimes this fails with the follwing error seen when attempting to run the sosreport command.

    error on sosreport: no such file or directory /dev/urandom
    

    If this error occurs, then type the following commands. This will first exit the chroot'ed environment and mount the missing mount points:

    # exit   <-- this will exit the chroot'ed environment
    # mount -o bind /dev /mnt/sysimage/dev
    # mount -o bind /sys /mnt/sysimage/sys
    # mount -o bind /proc /mnt/sysimage/proc
    # chroot /mnt/sysimage
    # sosreport
    

    NOTE: If df command does not show dev, sys and proc and/or other expected filesystems as present and mounted, and the above steps have been tried but failed to mount all necessary filesystems, then likely an advanced storage option needs to be activated before proceeding. See one of the following resources for additional information on resolving. The specific steps are highly dependent on the type of storage environment present.

  8. Results: Once sosreport generation completes, it will provide the output in the following directory whlie the user is in the chroot environment.

    • For RHEL 6 or earlier in directory: /var/tmp directory
    • For RHEL 7 or later in directory: /mnt/sysimage/var/tmp directory while in the chroot environment.

    Warning: During the running of the command sosreport you will be prompted for your name and case number. Use only letters and/or numbers when filling out this field. Adding other characters could damage the system or render the report unusable.

  9. Copy: Copy the resulting file from step 8 either over the network or on to a USB drive so that it can be uploaded to the support case.

    • Network: Use the scp command to copy the local file to a remote system.

      # scp /mnt/sysimage/var/tmp/sosreport username@hostname:sosreport
      
    • USB drive: Using a preformatted USB drive (in the following example the whole USB drive has been formatted with an ext3 filesystem), plug the drive into the system.

      • if the USB drive auto-mounts on /media/{name}, then:

        # cp /mnt/sysimage/var/tmp/*sosreport* /media/{name}
        # umount /media/{name}
        
      • if it doesn't auto-mount under the /media directory, then manually mount it:

        # fdisk -l                      <== locate the USB drive's /dev/sdX device name
        # mkdir /mnt/usbdrive
        # mount -t ext3 /dev/sdX /mnt/usbdrive
        # cp /mnt/sysimage/var/tmp/*sosreport* /mnt/usbdrive
        # umount /mnt/usbdrive
        

    NOTE: In RHEL7 or later, enable networking before executing scp.

  10. Provide: "How to provide files to Red Hat Support (vmcore, rhev logcollector, sosreports, heap dumps, log files, etc.)"

More Info

Related documentation
For Red Hat Enterprise Linux 9: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/performing_a_standard_rhel_installation/troubleshooting-after-installation_installing-rhel#booting-into-rescue-mode_using-rescue-mode
For Red Hat Enterprise Linux 8: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/performing_a_standard_rhel_installation/troubleshooting-after-installation_installing-rhel#using-rescue-mode_troubleshooting-after-installation
For Red Hat Enterprise Linux 7: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/installation_guide/sect-rescue-mode
For Red Hat Enterprise Linux 6: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/installation_guide/ap-rescuemode#Rescue_Mode-x86

Root Cause

  • Need to collect a sosreport and my system will not boot.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

14 Comments

I have tried to search this kb using search option "sosreport in rescue mode" but in result is was not available. Kindly make search engine in better way or make changes in title of this kb. For testing please use customer portal ID

Thanks for the feedback. We'll be updating the article to make it easier to find.

I have tried to search this kb using search option "sosreport in rescue mode" but in result is was not available. Kindly make search engine in better way or make changes in title of this kb. For testing please use customer portal ID

This is true I have also checked the same.

Thank you for pointing this out. I've made several changes to the contents of this Solution and it is now the first result for your "sosreport in rescue mode" query. Additionally, I have separated sosreport and sysreport contents into their own separate Solutions. You can find instructions for sysreport by searching "sysreport rescue" or by navigating to https://access.redhat.com/knowledge/solutions/320313

Unable to mount /dev into /mnt/sysimage/dev

Getting error that mount point /mnt/sysimage/dev does not exists

exit <-- this will exit the chroot'ed environment

mount -o bind /dev /mnt/sysimage/dev

mount -o bind /sys /mnt/sysimage/sys

mount -o bind /proc /mnt/sysimage/proc

chroot /mnt/sysimage

and then run the sosreport as stated above.

when chroot /mnt/sysimage
is not working in rescue mode. then how to reslove it

You should move the following warning so it comes before the command to run the report... what's the use of mentioning it after they already ran it?

Warning: During the running of the command sosreport you will be prompted for your name and case number. Use only letters and/or numbers when filling out this field. Adding other characters could damage the system or render the report unusable.

Also.. how do you " Exit the chroot environment..."

And how are you supposed to get that report to tech support when the OS won't boot successfully?

  • sometimes for environments really hosed (e.g. partitions missing) sosreport is not possible
  • to help share though customers should be able to bring up network and scp data off

  • Reload into rescue

  • "Skip": skip the mounting of the discovered Red Hat Enterprise Linux installation and proceed with manual mounting
fdisk -l > fdisk.txt
lsmod -l > lsmod.txt
dmidecode > dmidecode.txt
lspci -vvvv > lspci.txt
  • then bring up network
  • find a valid eth#
[root@seaking ~]# ls /sys/class/net/
br0  eth0  lo  vnet0  vnet1

ifconfig eth0 192.168.0.1 netmask 255.255.255.0 up 
route add default gw GATEWAY-IP eth0
  • if you have problems with the eth# above move to the next one
    ifconfig eth0 down

pick different from ls /sys/class/net and repeat ifconfig and route command above

After you bring up the network, ping the gateway as a test, then ping system you want to scp to.

Assuming the pings works

scp *.txt <someuser>@<somesystem>:<somefslocation>

And add those files to the case

Tech support pointed me to this article for a kernel panic. We just completed patching the system and it causes a kernel panic even when trying to boot into rescue mode. Would be nice to have some alternate guidance if the boot to rescue mode doesn't work.

Very useful!

How to run the chroot command if /mnt/sysimage is empty, due to mount failure for the root file system device. /dev/vgroot/lvroot does not contain a "clean filesystem"