What is an sosreport and how to create one in Red Hat Enterprise Linux?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 4.6+, 5, 6, 7, 8
  • Red Hat Enterprise MRG

Issue

Resolution

What is an sosreport?

The sosreport command is a tool that collects configuration details, system information and diagnostic information from a Red Hat Enterprise Linux system. For instance: the running kernel version, loaded modules, and system and service configuration files. The command also runs external programs to collect further information, and stores this output in the resulting archive.

Why am I being asked to provide an sosreport?

The output of sosreport is the common starting point for Red Hat support engineers when performing an initial analysis of a service request for a Red Hat Enterprise Linux system.

The utility provides a standardized way to collect diagnostic information that your Red Hat support engineer will reference throughout their investigation of issues reported in support cases.

The use of sosreport helps to ensure that you are not being continually asked for piecemeal data outputs.


How can I generate an sosreport?

Once the sos package has been installed, issue the following command to run sosreport:

# sosreport

Note that sosreport requires root permissions to gather data correctly. There is no current mechanism to allow non-root executions of sosreport.

The command will normally complete within a few minutes on Red Hat Enterprise Linux 6. Older versions may take longer to complete. Depending on local configuration and the options specified in some cases the command may take longer to finish. If you are concerned about the run time of the sosreport command contact your Red Hat support representative for assistance.

Once completed, sosreport will generate a compressed file under /tmp (for RHEL6 and earlier) or under /var/tmp (for RHEL7 and later). The file should be provided to your support representative (normally as an attachment to an open support case).

» Running sosreport results in "no valid plugins were enabled" message

  • Typically the "no valid plugins" message is output then the sosreport command run without full root privileges. The sosreport command requires root permissions to gather data from the system correctly. Many of the files, commands, and tools that it runs to gather information required root access privilege. For example, if multipath is configured on the system then sosreport will gather information from multipath subsystem on the current configuration settings, which it cannot do as a normal user.

    $ multipathd -k"show config"
    need to be root
    
  • Please try running again as root, and if you are still experiencing issues, open a support case so that we may investigate.*

» Running sosreport fills my available disk space

  • The size of the archive varies depending on system configuration and any optional sosreport features that are enabled. For example --all-logs will greatly increase the size of the archive as it removes size-limiting features for command output and log file collection.

  • If /tmp/ or /var/tmp is too small to hold an sosreport archive, use the --tmp-dir option to specify an alternate location with sufficient available space.

» How do I provide an sosreport to Red Hat?

  • To post sosreport, or any other file, to an existing support case you can use the redhat-support-tool command line option, the Red Hat Portal UI or several different methods using FTP.
  • If the collected sosreport file is too big to upload to the case, it may be uploaded to the Red Hat FTP dropbox.
    • If you use the ftp option, update your support case with the exact filename as this is the only way your support engineer will be able to retrieve it.

» I do not have a case number yet, do I need to provide one for sosreport to run?

  • The case number input prompt is optional, and if provided the case number will be part of the archive's file name. Omitting the case number will not have any negative effect on running the sosreport command.

  • Also, the command may be run in batch mode using the --batch option in order to avoid the need to enter user and account information interactively.


How can I control how the sosreport command runs?

The sosreport command has a plugin structure and allows the user to enable and disable plugins and specify plugin options via the command line. Available plugins and their options may be listed by running the following command:

# sosreport -l

How can I disable or enable specific plugins?

Users may selectively enable or disable plugins using the -e/--enable-plugins and -n/--skip-plugins options respectively. These options take comma-delimited lists of plugin names or may be specified multiple times. For example, to disable the amd and kvm plugins, use the following:

# sosreport -n kvm,amd

How do I use plugin options?

Individual plugins may provide additional options that may be specified via the -k option. These options are listed in the same output as the listing of available plugins (sosreport -l). The format this option takes is plugin_name.option_name=value. For example, to enable collection of container logs, use the following:

# sosreport -k podman.logs=on

Options will specify whether they are boolean toggles or expect a string or integer. For boolean toggles, users may use True, on, or yes and False, off, or no interchangeably.

Why does sosreport sometimes skip collecting certain command output?

An sosreport execution might print to console a message like the following:

[plugin:networking] skipped command 'nft list ruleset': required kernel modules or services not present (kmods=[nf_tables] services=[]).

sosreport aims to not alter the system it runs on in any way. Some commands called by certain plugins may automatically trigger a change (load a kernel module in the example above) and are thus gated by default. If making these changes is acceptable for your environment, and you would like to have sosreport make them in order to collect the skipped commands, run sosreport like so:

sosreport --allow-system-changes

Running sosreport on my system uses too much CPU time or memory (it is supported with sos version 3.6 or later)

By default sosreport will run up to 4 plugins in parallel in an effort to reduce total runtime. This may cause memory contention or high CPU usage on certain systems depending on which plugins are being run (and particularly if each of those plugins collects journal output). To reduce the number of simultaneous collections use the --threads option. For example, to run plugins one at a time, use the following:

# sosreport --threads=1

Installing sosreport:

The sos package must be installed in order to run the sosreport command. You can check to see if the sos package is installed and whether there is any problems with the installation using the following command:

# rpm -qa | grep sos
sos-3.2-35.el7_2.3.noarch                  << sos package is installed
# rpm -V sos                               << run verification on installed package
  • Red Hat Enterprise Linux 5 and later

    • If the system is registered with RHSM, use the yum command:

      # yum install sos
      
    • If the system is not registered with RHSM, the sos package can be downloaded from the RHN website or found on the installation CDs or DVD. The rpm command may be used to install the package on any version of Red Hat Enterprise Linux:

      # rpm -Uvh sos-<version>.noarch.rpm
      
  • Red Hat Enterprise Linux 4 Update 6 or later

    • If the system is registered with Red Hat Subscription Manager (RHSM), sos can be installed using the up2date command:

      # up2date sos
      

What to do if sosreport hangs

First, verify that there is enough free space available in either /tmp (RHEL 6) or /var/tmp (RHEL7 and later), and if not, use the --tmp-dir option as detailed earlier.

If sosreport is hanging/stalling on a specific plugin (currently running plugins are displayed during execution), try running sosreport with the problem plugin(s) disabled using the -n/--skip-plugins option as detailed earlier.

Note that each plugin has a default timeout of 5 minutes (controllable by the timeout plugin option available to all plugins). Please allow enough time for this timeout threshold to be hit, and sosreport should automatically terminate that plugin's execution. In the event that this termination does not happen, use the option highlighted above.

If an sosreport can not be completed at all for one reason or another, please see How to gather data from a Red Hat Enterprise Linux system for troubleshooting if the sosreport process goes to hung state? for further guidance.

I cannot run sosreport at all and would like to collect a bare minimum of data to provide

Please see sosreport fails. What data should I provide in its place? for an alternative bare-minimum data collection script.

For issue-specific guidance in the case of sosreport failure, please see Additional and alternative steps for gathering an sosreport for a Red Hat Enterprise Linux support case

What commands a particular sosreport plugin runs in the background

There are following ways to understand what a particular plugin does when its is executed through the sosreport command.

  • Check the source of sos package that provides sosreport command.
    For a specific version of source code, see downloads section on Customer Portal.

  • Check sos.log file in the sosreport. For example, for plugin scsi there would be something similar to as shown below

-- directories and files collected --

2020-07-30 11:04:53,632 INFO: [plugin:scsi] collecting path '/sys/bus/scsi'
2020-07-30 11:04:53,656 INFO: [plugin:scsi] collecting path '/sys/class/scsi_disk'
2020-07-30 11:04:53,663 INFO: [plugin:scsi] collecting path '/proc/scsi'
2020-07-30 11:04:53,666 INFO: [plugin:scsi] collecting path '/sys/class/scsi_generic'
2020-07-30 11:04:53,675 INFO: [plugin:scsi] collecting path '/sys/class/scsi_host'
2020-07-30 11:04:53,677 INFO: [plugin:scsi] collecting path '/sys/class/scsi_device'

-- these were the commands executed in the background --

2020-07-30 11:04:53,686 INFO: [plugin:scsi] collecting output of 'lsscsi -i'
2020-07-30 11:04:53,710 INFO: [plugin:scsi] collecting output of 'sg_map -x'
2020-07-30 11:04:53,765 INFO: [plugin:scsi] collecting output of 'udevadm info -a /sys/class/scsi_host/host3'
2020-07-30 11:04:53,789 INFO: [plugin:scsi] collecting output of 'udevadm info -a /sys/class/scsi_host/host1'
  • Generate the sosreport only with that particular plugin and check the files and directories and the output of the commands in the collected archive.
    To generate the sosreport with particular plugin, do
#sosreport -o plugin-name

To see the list of plugins, do

#sosreport -l

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

39 Comments

Add a final step in the end of the article about "where to send the sosreport" or a link to the article that has this info.

How long does this normally take?

sosreport is not completing:
-----------------------------------------------------------------------------------------------------
This utility will collect some detailed  information about the
hardware and  setup of your  Red Hat Enterprise Linux  system.
The information is collected and an archive is  packaged under
/tmp, which you can send to a support representative.
Red Hat will use this information for diagnostic purposes ONLY
and it will be considered confidential information.

This process may take a while to complete.
No changes will be made to your system.

Press ENTER to continue, or CTRL-C to quit.

Please enter your first initial and last name [fwnode5]: safaricom.fwnode5
Please enter the case number that you are generating this report for: 00437567

plugin ssh finished ...
--------------------------------------------------------------------------------------------------------
Then its stops.

These instructions almost, but not quite work. I encountered two wrinkles. There wrinkles weren't too hard to sort out, but I shouldn't have had to in the first place. It would be simple to edit the article in such a way that future users will know exactly what to do in the first place.

1. At least for my installation, the sosreport executable is in /usr/sbin . I don't happen to have /use/sbin in my path, and I suspect mine is the most common situation. Therefore, the command isn't
# sosreport

It's
# /usr/sbin/sosreport

2. The file permissions for the report file sosreport generated were _rw______ (600), with user root. When I attempted to upload the report file into the report case, I was logged in as myself and not root, and so the upload failed with no error indication other than that the uploaded file size was 0. I solved the problem by going to a terminal in which I was logged in as root and then changing the file permissions to rw__r__r (644), which is what they should have been in the first place.

You need to run sosreport as root, and if root doesn't have /usr/sbin in it's path, then fix that. I agree this should be updated to be more complete (as it is part of the cut/paste response when requesting a sosreport).

AND?????? Where do you upload the sosreport to?????

You'll be able to attach a sosreport to an existing Red Hat support case, here on the customer portal.

While taking the sosreport, there is this warning:
One or more plugins have detected a problem in your configuration.

Please review the following messages:

cluster:
* GFS2 is being used via weak-updates, kmod-gfs2 should be uninstalled and system rebooted to allow for kernel provided gfs2 module to be used.

Does this indicate that GFS is misconfigured on the cluster (use the built-in kernel module rather than kmod-gfs2)?

Hi Ed. This is probably a question to raise in your support case, or alternatively you could try the RHEL subscriber community here: https://access.redhat.com/groups/red-hat-enterprise-linux

The browse function to attach files to this ticket does not work. It continues to complain that I need to use a file name less than 80 characters. The file name was only about 53. I have dropped it multiple times until now it is only 'sosreport.tar.bz2' and it still complains that the file name is greater than 80 characters. What would you suggest. It would also be nice to just FTP the file directly to you folks since I could do that from the troubled system, but I had to move it two additional times so I could be told my 17 character long file name is longer than 80 characters!

Thanks for the feedback, Kyle. We'll investigate this issue.

While I'm opening a new ticket, I clicked on the link that took me HERE... and now I cannot seem to find my way back to the ticket I was creating. Hitting the back button DOES take me back to the beginning and I see all my entered description, however, I cannot click next or anything! Clicking the little "sosreport" hot link from inside a ticket creation should instead open a new Window or Tab! IMO

Great feedback, Dennis - thanks.

Just wondering about the two sentences which tell us that ...

"The command will normally complete within a few moments on Red Hat Enterprise Linux 6. Older versions may take a few minutes to complete."

Perhaps somebody from RedHat could explain exactly what the difference is between "a few moments" and "a few minutes". ;-)

Ha. Thanks for pointing that out, Mark.

Do you have a secure site to upload reports to?

You can securely attach the reports to your support case here on the customer portal.

please help me with the issue. I need to know why the rman command is not populated

Ok.. I got the SOS report. But I can not copy the same or attached to the upload support page

There is one small flaw in the alternate procedure. If run as is it will attempt to tar up the copy of /var/log/lastlog. This is pointless and can take virtually forever. The tar command at the end should look more like this:

tar -cvjf ${host}_sosreport.tar.bz2 --exclude=*/var/log/lastlog $sos_dir

What does the acronym sos in sosreport stand for?

I don't believe it stands for anything. It's just "SOS report", like the distress signal: http://en.wikipedia.org/wiki/Sos

I may be wrong, but I have been told that "sos" stands for "Son of Sys". RHEL 3 and RHEL4 originally used sysreport, the predecessor of sosreport.

Hi Imogen,
You are correct according to these documents:
Getting in the Fast Lane with Red Hat Support
Jeff Bastian & Guy Streeter, Red Hat
http://people.redhat.com/streeter/summit08-support-fast-lane.pdf
snip---------------
Opening a Ticket (continued)
 Always include a sosreport (or sysreport)
● sos = son of sysreport (RHEL 4.6, 5.0 and newer)
unsnip----------

and
Support data gathering tool
https://lists.ubuntu.com/archives/ubuntu-devel-discuss/2009-May/008278.html

snip-------------
Red Hat has something similar, though not as developed as Explorer.
Their sos (son of sysreport) tool is GPLed and could be tweaked to run
on Debian/Ubuntu.
unsnip-----------

A quick Google search for "son of sysreport" found these plus more (I had tried just "sos")

Mystery solved!

Any methods to disable EMC plugin and run the SOS report as I have a system running on RHEL 4 and the SOS report is hanging at EMC plugin?

the tar file is too big, when I upload from the Linux box to my pc, it always error, could not do it. please advise
thank you.

How long should the sosreport stay at this message?
plugin ssh finished ...

It's been there 45 minutes or so..

rpm -qa | grep cyrus

cyrus-sasl-2.1.19-5.EL4
cyrus-imapd-utils-2.2.12-3.RHEL4.1
cyrus-sasl-ntlm-2.1.19-5.EL4
cyrus-imapd-devel-2.2.12-3.RHEL4.1
cyrus-sasl-sql-2.1.19-5.EL4
cyrus-imapd-2.2.12-3.RHEL4.1
cyrus-sasl-md5-2.1.19-5.EL4
cyrus-sasl-plain-2.1.19-5.EL4
cyrus-imapd-murder-2.2.12-3.RHEL4.1
cyrus-sasl-devel-2.1.19-5.EL4
cyrus-sasl-gssapi-2.1.19-5.EL4
cyrus-imapd-nntp-2.2.12-3.RHEL4.1

Can anyone tell me that redhat openstack plateform needs a vdi broker like that with simple openstack plateform??

It say to generate an sosreport when submitting a new case to get faster results. The sosreport asks for a case number. A case number is not given until after the case is submitted. This should be cleaned up.

It's a very good technical proceedings !

This has not worked for awhile. I would've put Red Hat updates on since it started. On the actual backup software, I just updated it last week thinking that might correct the problem.

This tool is amazing, why there is no plugin to aws like there is for azure? Lets create this feature!

where is the file?

# sosreport
:
Creating compressed archive...

Your sosreport has been generated and saved in:
  /tmp/sosreport-mysos-20170928165743.tar.xz           <=============== created sosreport file is here.

The checksum is: a90fb5a48e8b32e4ac14f97cf183b907

Please send this file to your support representative.

If client is on a production server and can't install strace, another easy way to figure out which plugin is hanging is to pick the plugin listed alphabetically after the last plugin that is displayed in the sosreport -vvv output before it hangs. To get the list of plugins enabled alphabetically do sosreport -l. (ie - if sosreport -vvv output ends with ipmi-tools before it hangs then disable the plugin after ipmi-tools in the list to see if it gets the client past the hang. (sosreport -n jar for example. ))

In this section "Other: Otherwise, instead of collecting the failure sosreport, a manual report may be created by running the following script. Note this will only be a portion of what is collected in a standard sosreport." This command "uname -a &> uname" is duplicated; execute one time is enough.

Regards, Sam

You need to work on improving the structure of this page. It is impossible to follow and should be made easier given that the automatic response from support is to request a sosreport regardless of the logic or necessity of such a request.

hello

Pages