What is an sosreport and how to create one in Red Hat Enterprise Linux?
Environment
- Red Hat Enterprise Linux (RHEL) 4.6+, 5, 6, 7, 8
- Red Hat Enterprise MRG
- Exceptions:
- For System Is Not Booting cases
- Follow the directions in: "How to generate sosreport from the rescue environment" for cases where the system is not booting or does not stay running long enough to generate an sosreport via normal means.
- For Azure Virtual Guests
- Follow the directions in: "How to Create a Microsoft Azure Virtual Machine sosreport and Submit the Report to Red Hat" for Azure virtual guests.
Issue
- What is an
sosreport
?- How do I collect system log files, configuration details and system information from a Red Hat Enterprise Linux system to provide to Red Hat Technical Support for analysis?
- How can I generate an
sosreport
?- I am opening a service request with Red Hat Technical Support for technical assistance. What data I have to provide for initial analysis of my system?
- How can I control how the
sosreport
command runs? - How to install the package containing sosreport
- What to do if an sosreport hangs
- What commands a particular sosreport plugin runs in the background
Resolution
What is an sosreport?
The sosreport
command is a tool that collects configuration details, system information and diagnostic information from a Red Hat Enterprise Linux system. For instance: the running kernel version, loaded modules, and system and service configuration files. The command also runs external programs to collect further information, and stores this output in the resulting archive.
Why am I being asked to provide an sosreport?
The output of sosreport
is the common starting point for Red Hat support engineers when performing an initial analysis of a service request for a Red Hat Enterprise Linux system.
The utility provides a standardized way to collect diagnostic information that your Red Hat support engineer will reference throughout their investigation of issues reported in support cases.
The use of sosreport
helps to ensure that you are not being continually asked for piecemeal data outputs.
How can I generate an sosreport?
Once the sos
package has been installed, issue the following command to run sosreport:
# sosreport
Note that sosreport
requires root permissions to gather data correctly. There is no current mechanism to allow non-root executions of sosreport
.
The command will normally complete within a few minutes on Red Hat Enterprise Linux 6. Older versions may take longer to complete. Depending on local configuration and the options specified in some cases the command may take longer to finish. If you are concerned about the run time of the sosreport
command contact your Red Hat support representative for assistance.
Once completed, sosreport
will generate a compressed file under /tmp
(for RHEL6 and earlier) or under /var/tmp
(for RHEL7 and later). The file should be provided to your support representative (normally as an attachment to an open support case).
» Running sosreport
results in "no valid plugins were enabled" message
-
Typically the "no valid plugins" message is output then the
sosreport
command run without full root privileges. Thesosreport
command requires root permissions to gather data from the system correctly. Many of the files, commands, and tools that it runs to gather information required root access privilege. For example, if multipath is configured on the system then sosreport will gather information from multipath subsystem on the current configuration settings, which it cannot do as a normal user.$ multipathd -k"show config" need to be root
-
Please try running again as root, and if you are still experiencing issues, open a support case so that we may investigate.*
» Running sosreport
fills my available disk space
-
The size of the archive varies depending on system configuration and any optional
sosreport
features that are enabled. For example--all-logs
will greatly increase the size of the archive as it removes size-limiting features for command output and log file collection. -
If
/tmp/
or/var/tmp
is too small to hold an sosreport archive, use the--tmp-dir
option to specify an alternate location with sufficient available space.
» How do I provide an sosreport
to Red Hat?
- To post
sosreport
, or any other file, to an existing support case you can use the redhat-support-tool command line option, the Red Hat Portal UI or several different methods using FTP. - If the collected
sosreport
file is too big to upload to the case, it may be uploaded to the Red Hat FTP dropbox.- If you use the ftp option, update your support case with the exact filename as this is the only way your support engineer will be able to retrieve it.
» I do not have a case number yet, do I need to provide one for sosreport
to run?
-
The case number input prompt is optional, and if provided the case number will be part of the archive's file name. Omitting the case number will not have any negative effect on running the
sosreport
command. -
Also, the command may be run in batch mode using the
--batch
option in order to avoid the need to enter user and account information interactively.
How can I control how the sosreport
command runs?
The sosreport command has a plugin structure and allows the user to enable and disable plugins and specify plugin options via the command line. Available plugins and their options may be listed by running the following command:
# sosreport -l
How can I disable or enable specific plugins?
Users may selectively enable or disable plugins using the -e/--enable-plugins
and -n/--skip-plugins
options respectively. These options take comma-delimited lists of plugin names or may be specified multiple times. For example, to disable the amd
and kvm
plugins, use the following:
# sosreport -n kvm,amd
How do I use plugin options?
Individual plugins may provide additional options that may be specified via the -k
option. These options are listed in the same output as the listing of available plugins (sosreport -l
). The format this option takes is plugin_name.option_name=value
. For example, to enable collection of container logs, use the following:
# sosreport -k podman.logs=on
Options will specify whether they are boolean toggles or expect a string or integer. For boolean toggles, users may use True
, on
, or yes
and False
, off
, or no
interchangeably.
Why does sosreport
sometimes skip collecting certain command output?
An sosreport execution might print to console a message like the following:
[plugin:networking] skipped command 'nft list ruleset': required kernel modules or services not present (kmods=[nf_tables] services=[]).
sosreport
aims to not alter the system it runs on in any way. Some commands called by certain plugins may automatically trigger a change (load a kernel module in the example above) and are thus gated by default. If making these changes is acceptable for your environment, and you would like to have sosreport
make them in order to collect the skipped commands, run sosreport
like so:
sosreport --allow-system-changes
Running sosreport
on my system uses too much CPU time or memory (it is supported with sos version 3.6 or later)
By default sosreport
will run up to 4 plugins in parallel in an effort to reduce total runtime. This may cause memory contention or high CPU usage on certain systems depending on which plugins are being run (and particularly if each of those plugins collects journal output). To reduce the number of simultaneous collections use the --threads
option. For example, to run plugins one at a time, use the following:
# sosreport --threads=1
Installing sosreport:
The sos
package must be installed in order to run the sosreport
command. You can check to see if the sos
package is installed and whether there is any problems with the installation using the following command:
# rpm -qa | grep sos
sos-3.2-35.el7_2.3.noarch << sos package is installed
# rpm -V sos << run verification on installed package
-
Red Hat Enterprise Linux 5 and later
-
If the system is registered with RHSM, use the
yum
command:# yum install sos
-
If the system is not registered with RHSM, the
sos
package can be downloaded from the RHN website or found on the installation CDs or DVD. The rpm command may be used to install the package on any version of Red Hat Enterprise Linux:# rpm -Uvh sos-<version>.noarch.rpm
-
-
Red Hat Enterprise Linux 4 Update 6 or later
-
If the system is registered with Red Hat Subscription Manager (RHSM),
sos
can be installed using theup2date
command:# up2date sos
-
What to do if sosreport hangs
First, verify that there is enough free space available in either /tmp
(RHEL 6) or /var/tmp
(RHEL7 and later), and if not, use the --tmp-dir
option as detailed earlier.
If sosreport
is hanging/stalling on a specific plugin (currently running plugins are displayed during execution), try running sosreport
with the problem plugin(s) disabled using the -n/--skip-plugins
option as detailed earlier.
Note that each plugin has a default timeout of 5 minutes (controllable by the timeout
plugin option available to all plugins). Please allow enough time for this timeout threshold to be hit, and sosreport
should automatically terminate that plugin's execution. In the event that this termination does not happen, use the option highlighted above.
If an sosreport
can not be completed at all for one reason or another, please see How to gather data from a Red Hat Enterprise Linux system for troubleshooting if the sosreport
process goes to hung state? for further guidance.
I cannot run sosreport
at all and would like to collect a bare minimum of data to provide
Please see sosreport fails. What data should I provide in its place? for an alternative bare-minimum data collection script.
For issue-specific guidance in the case of sosreport
failure, please see Additional and alternative steps for gathering an sosreport for a Red Hat Enterprise Linux support case
What commands a particular sosreport plugin runs in the background
There are following ways to understand what a particular plugin does when its is executed through the sosreport
command.
-
Check the source of
sos
package that providessosreport
command.
For a specific version of source code, see downloads section on Customer Portal. -
Check
sos.log
file in the sosreport. For example, for pluginscsi
there would be something similar to as shown below
-- directories and files collected --
2020-07-30 11:04:53,632 INFO: [plugin:scsi] collecting path '/sys/bus/scsi'
2020-07-30 11:04:53,656 INFO: [plugin:scsi] collecting path '/sys/class/scsi_disk'
2020-07-30 11:04:53,663 INFO: [plugin:scsi] collecting path '/proc/scsi'
2020-07-30 11:04:53,666 INFO: [plugin:scsi] collecting path '/sys/class/scsi_generic'
2020-07-30 11:04:53,675 INFO: [plugin:scsi] collecting path '/sys/class/scsi_host'
2020-07-30 11:04:53,677 INFO: [plugin:scsi] collecting path '/sys/class/scsi_device'
-- these were the commands executed in the background --
2020-07-30 11:04:53,686 INFO: [plugin:scsi] collecting output of 'lsscsi -i'
2020-07-30 11:04:53,710 INFO: [plugin:scsi] collecting output of 'sg_map -x'
2020-07-30 11:04:53,765 INFO: [plugin:scsi] collecting output of 'udevadm info -a /sys/class/scsi_host/host3'
2020-07-30 11:04:53,789 INFO: [plugin:scsi] collecting output of 'udevadm info -a /sys/class/scsi_host/host1'
- Generate the
sosreport
only with that particular plugin and check the files and directories and the output of the commands in the collected archive.
To generate the sosreport with particular plugin, do
#sosreport -o plugin-name
To see the list of plugins, do
#sosreport -l
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments
Pages