xsos -- a tool for sysadmins and support techs

Latest response

My public yum repo now includes an rpm for xsos.

For those unfamiliar with xsos, well... a picture is worth a hundred words in this case, so I'll give you 1000+ words covering some of what xsos can do. Note that some of these screenshots were generated by querying sosreports and some of them were generated by executing xsos live on running systems.

xsos (run with no options)

xsos --os --mem --scrub-ip

xsos --bios --ip --scrub-mac

xsos --cpu --lspci --netdev

xsos --ethtool

xsos --bonding

xsos --ps

xsos --kdump

xsos --sysctl

This next one illustrates something special ...

xsos --cpu --intrupt

xsos --softirq
(This one does some diagnosing based on the file /proc/net/softnet_stat. See RFE #108 for details.)

Finally, here's the verbose version of the xsos help page (there's a shorter -h as well)
xsos --help

Usage: xsos [DISPLAY OPTIONS] [-6abokcfmdtlerngisp] [SOSREPORT ROOT]
  or:  xsos [DISPLAY OPTIONS] {--B|--C|--F|--M|--D|--T|--L|--R|--N|--G|--I|--P FILE}...
  or:  xsos [-?|-h|--help]
  or:  xsos [-U|--update]

Display system info from localhost or extracted sosreport
Run with "-h" to see simplified help page

Content options:
 -a, --all      show everything
 -b, --bios     show info from dmidecode
 -o, --os       show hostname, distro, SELinux, kernel info, uptime, etc
 -k, --kdump    inspect kdump configuration
 -c, --cpu      show info from /proc/cpuinfo
 -f, --intrupt  show info from /proc/interrupts
 -m, --mem      show info from /proc/meminfo
 -d, --disks    show info from /proc/partitions + dm-multipath synopsis
 -t, --mpath    show info from dm-multipath
 -l, --lspci    show info from lspci
 -e, --ethtool  show info from ethtool
 -r, --softirq  show info from /proc/net/softnet_stat
 -n, --netdev   show info from /proc/net/dev
 -g, --bonding  show info from /proc/net/bonding
 -i, --ip       show info from ip addr (BASH v4+ required)
     --net      alias for: --lspci --ethtool --softirq --netdev --bonding --ip
 -s, --sysctl   show important kernel sysctls
 -p, --ps       inspect running processes via ps

Display options:
     --scrub-ip     remove IP addresses & hostnames from output
     --scrub-mac    remove HW MAC addresses from output
                    see XSOS_SCRUB_IP_HN & XSOS_SCRUB_MACADDR env vars
 -6, --ipv6         parse ip addr output for IPv6 addresses instead of IPv4
 -q, --wwid=ID      restrict dm-multipath output to a particular mpath device,
                    where ID is a wwid, friendly name, or LUN identifier
 -u, --unit=P       change byte display for /proc/meminfo & /proc/net/dev,
                    where P is "b" for byte, or else "k", "m", "g", or "t"
 -v, --verbose=NUM  specify ps verbosity level (0-4, default: 1)
 -w, --width=NUM    change fold-width, in columns (positive number, e.g., 80)
                    "0" disables wrapping, "w" autodetects width (default)
 -x, --nocolor      disable output colorization
 -y, --less         send output to `less -SR`
 -z, --more         send output to `more`

If no content options are specified, xsos parses the environment variable
XSOS_DEFAULT_VIEW to figure out what information to display. If this variable
is unset at runtime, it is initialized internally as follows:


Tweak it to preference by adding additional space-separated MODULE statements,
where MODULE is the same as the long option (e.g. mem, ethtool, netdev). Note
that the --net alias option cannot be used for this purpose. Also note that the
-a / --all option has it's own environment variable: XSOS_ALL_VIEW

If SOSREPORT ROOT isn't provided, the data will be gathered from the localhost;
however, bios, multipath, and ethtool output will only be displayed if running
as root (UID 0). When executing in this manner as non-root, those modules will
be skipped, and a warning printed to stderr.

Sometimes a full sosreport isn't available; sometimes you simply have a
dmidecode-dump or the contents of /proc/meminfo and you'd like a summary...

Special options (BASH v4+ required):
 --B=FILE  read from FILE containing `dmidecode` dump
 --C=FILE  read from FILE containing /proc/cpuinfo dump
 --F=FILE  read from FILE containing /proc/interrupts dump
 --M=FILE  read from FILE containing /proc/meminfo dump
 --D=FILE  read from FILE containing /proc/partitions dump
 --T=FILE  read from FILE containing `multipath -v4 -ll` dump
 --L=FILE  read from FILE containing `lspci` dump
 --R=FILE  read from FILE containing /proc/net/softnet_stat dump
 --N=FILE  read from FILE containing /proc/net/dev dump
 --G=FILE  read from FILE containing /proc/net/bonding/xxx dump
 --I=FILE  read from FILE containing `ip addr` dump
 --P=FILE  read from FILE containing `ps aux` dump

As is hopefully clear, each of these options requires a filename as an
argument. These options can be used together, but cannot be used in concert
with regular "Content options" -- Content opts are ignored if Special options
are detected. Also note: the "=" can be replaced with a space if desired.

Re BASH v4+:
 BASH associative arrays are used for various things. In short, if running
 xsos on earlier BASH versions (e.g. RHEL5), you get ...
  * No output colorization
  * No -i/--ip
  * No parsing of "Special options"

Environment variables:
 For details of all configurable env variables, view first page of xsos
 source. There are vars to change default colors as well as other settings.
 Each variable name is prefixed with "XSOS_" and the important ones follow.

Version info: xsos v0.5.17 last mod 2015/04/12
See <github.com/ryran/xsos> to report bugs or suggestions

Necessary disclaimer: xsos is an open-source utility developed by me in my free time. I happen to be a Red Hat employee; that does not mean that xsos is shipped, supported, or officially condoned by Red Hat. If you can keep that in mind, you are free to post problems or questions here OR on the github issue tracker.


Very nice!!!

Added screenshot for new --bonding option and updated --ethtool screenshot, as the latest version added parsing of eth device rx ring buffer settings to the output. Changelog is on github or of course in the xsos rpm changelog @ people.redhat.com/rsawhill/rpms.

Really Usefull Tool!

I'm glad you're finding a use for it! Let me know if you have any suggestions.

Updated with new screenshot for --kdump. Since my last comment I also added a --scrub option and probably lots of other things.

See rpm or git changelog for changes.

Excellent work Ryan!

+1 - great stuff. I look forward to seeing more awesomeness!!

Added a new screenshot for --sysctl which has always been there, but which was given an overhaul today.

Great looking tool, thanks for posting (and keeping it updated)!

Will be sure to add it to my next customer build.

Really appreciate your ongoing work with this, Ryan, and the updates here!

This is such a fantastic tool. I use xsos most days in my work here at Red Hat.

It's useful for quickly pulling relevant information out of a sosreport which speeds our fault analysis and case turnaround times, as well as displaying that information in a clear and presentable way to you, our customers.

I use it mostly for viewing hardware platform and RHEL/kernel versions, for viewing network interface configuration, and looking for errors on the network interfaces - this actually uses an egrep which I contributed to xsos and which has been improved upon since then, making it even more useful for me, a true open source development result!

I'd encourage all customers to try running an xsos -a over your own uncompressed sosreports to see how the information on your system is represented. You may find it a useful addition to your own reporting and analysis.

Ryan, Great work. Ryan++
You have created very useful tool. Thanks.

New v0.4.3 rpm addressing RHEL7 issues pushed to the repo.

* Sun Jun 22 2014 Ryan Sawhill Aroha <rsaw@redhat.com> 0.4.3-1
- Modified -b / --bios display to show more info, per issue 98
- Added RHEL7 to supported systems regex due to GA release this month
- Tweaked the way /etc/os-release is parsed & displayed
- Tweaked the way /proc/cmdline is parsed: "BOOT_IMAGE=/vmlinuz..." is now
  automatically removed from the beginning of the kernel cmdline before display
  making it easier to compare default & current kernel cmdline
- See issue 99 for details on the RHEL7-specific kdump fixes

New v0.4.3 screenshot showing RHEL7


Since the last comment, most of the changes to xsos have tweaked the way xsos --ip works:

  • fixed a few bugs
  • changed the order of the columns to be [hopefully] more logical
  • completely changed the way --scrub-ip and --scrub-mac are handled
  • added a "Scope" column for IPv6

Updated screenshot in the original post.

Some new networking funness for the new year:

* Thu Jan 01 2015 Ryan Sawhill Aroha <rsaw@redhat.com> 0.5.5-2
- Implemented new module based on #108 RFE: Alert on network SoftIRQ backlog or
  budget drops..added new options (`-r`, `--R`, `--softirq`) to bash-completion
- Added 2 new sysctls to --sysctl output:
  net.ipv4.tcp_sack and net.ipv4.tcp_timestamps
- Replaced all references to awk with gawk since some gawk-specific features
  are used
- Fixed #109 (awk division by zero error in weird NETDEV output)

Screenshot showing Jamie's new feature from #108

Added updated screenshot above for new bonding layout. Enhancements for troubleshooting mode4 bonding, per RFE: more details on bonding mode 4. Thanks to PTalbert of GSS for his awesome suggestions.

Super cool! Like it a lot.

New rpm hit the yum repo today. I didn't update any of the screenshots, but there are a couple medium-sized changes in some of the modules' output, along with a handful of other lil fixes.

[rsawhill]$ rpm -qp --changelog latest-xsos.rpm | head
* Thu Feb 26 2015 Ryan Sawhill Aroha <rsaw@redhat.com> 0.5.12-1
- Add pause to the ethtool -S regex
- New features added to bonding mode 4 (#115)
- Add PCI address information in ethtool output (#114)
- Fix "no df_output .. shouldn't happen" (#119)
- Fix lspci missing VGA with really long pci ids (#121)
- lspci overhaul: split up net/storage output, show more storage stuff (what
  lspci module shows is now configurable by environment variables -- #122)

* Thu Jan 01 2015 Ryan Sawhill Aroha <rsaw@redhat.com> 0.5.5-2

Pushed new v0.5.17-1 rpm to the yum repo today.

[rsaw:noarch]$ rpm -qp --changelog xsos-0.5.17-1.noarch.rpm|head -12
* Sun Apr 12 2015 Ryan Sawhill Aroha <rsaw@redhat.com> 0.5.17-1
- Add RHSM details to os module, along with additional yum plugins funness and
  proxy-parsing for both RHN & RHSM (#127)
- Add new -f/--F/--intrupt options to parse /proc/interrupts data into a binary
  graph (#131)
- Notably, the softirq module (-r/--softirq) is now included by xsos -a/--all,
  as is the new -f/--intrupt module
- Make kdump svc-enablement status w/systemd (e.g., RHEL7) more clear (#129)
- Make memgraph (--mem) characters simpler & more legible -- especially
  important on virtual consoles (#130)

* Thu Feb 26 2015 Ryan Sawhill Aroha <rsaw@redhat.com> 0.5.12-1

Updated original post with new screenshots for --softirq and the new --intrupt option.

The other big change is the addition of yum-plugins and RHSM to the os module, though that's likely only interesting to Red Hat support techs.

Thanks for keeping this updated, Ryan!

Hi Rayn,

Thanks for xsos script, its really very useful, when i ran xsos --ps, i see the %cpu is showing as 1000% in a server. please check the below details.

Total number of threads/processes:
6906 / 3037
Top users of CPU & MEM:
oracle 1058.7% 30.0% 137.80 GiB --> could you explain what does it mean and how is it being calculated.
root 31.2% 0.2% 2.17 GiB
68 1.9% 0.0% 0.04 GiB
dk016140 0.2% 0.0% 0.01 GiB


Hi Veera. In short: for single-threaded processes, ps reports "%CPU" as a percentage of ONE logical processor; not the total available system processing power. If root has multiple processes that are pegging their respective CPUs (as in the following screenshot), then root's total "%CPU" will be above 100%. Check it out:


Note also that a single process can show more than 100% if it's multi-threaded.

Great job Ryan!!! I love it! Such a useful tool!!

Thanks for bumping the post, I discovered a great tool :-)

Excellent tool!

yes, this is excellent and useful. Kudos!!!

Haven't been keeping this thread updated. Since the last time I posted, there have been quite a few releases (github.com/ryran/xsos/releases) including a release today to enhance dmidecode-parsing.

screenshot of old-buggy vs new hotness

Hi Ryan,

The "--scrub-ip" & "--scrub-mac" options are not working, only "--scrub" works which scrubs ip, mac,hostname etc,. but option of scrubbing only ip or mac doesn't work. Please check on this:

[root@localhost ~]# xsos --ip --scrub-ip
xsos: unrecognized option '--scrub-ip'
Usage: xsos [DISPLAY OPTIONS] [-6abokcfmdtlerngisp] [SOSREPORT ROOT]
  or:  xsos [DISPLAY OPTIONS] {--B|--C|--F|--M|--D|--T|--L|--R|--N|--G|--I|--P FILE}...
  or:  xsos [-?|-h|--help]

Display system info from localhost or extracted sosreport
[root@localhost ~]# xsos --ip --scrub-mac
xsos: unrecognized option '--scrub-mac'
Usage: xsos [DISPLAY OPTIONS] [-6abokcfmdtlerngisp] [SOSREPORT ROOT]
  or:  xsos [DISPLAY OPTIONS] {--B|--C|--F|--M|--D|--T|--L|--R|--N|--G|--I|--P FILE}...
  or:  xsos [-?|-h|--help]

Display system info from localhost or extracted sosreport

I'm using the latest one even I tried the previous version as well:

[root@localhost ~]# rpm -q xsos

Other parameters and options are working as expected.

From the changelog I could see this :-

* Sat Jul 11 2015 Ryan Sawhill Aroha <rsaw@redhat.com> 0.7.0-1
- Remove --scrub-ip and --scrub-mac
- Add XSOS_SCRUB_SERIAL env variable (to the already-existing XSOS_SCRUB_IP_HN
- Add --scrub option that does all 3

Does this mean "--scrub-mac" & "--scrub-ip" doesn't work, however, "--scrub" does it all? But mac/ip scrubbing independently would be a nice option.

Hi Sadashiva,

I can confirm the same : the options --scrub-mac and --scrub-ip do not work ... but the option --scrub does.

Waves :)

Indeed. Back in 2005 I removed the two independent options. See also changelogs here: https://github.com/ryran/xsos/releases

Okay, then you might consider editing the screen shots those talks about using --scrub-ip and --scrub-mac options.

Also, nice if this gets added to epel repo...

Hi Sadashiva,

Well, something even better : I found out that xsos is completely 'portable', simply download the source code from https://github.com/ryran/xsos/releases, place it where ever you want, make the file executable and launch the tool with ./xsos. I've already added this useful application to my toolbox and put it on all machines and systems today.

Waves :)

Yes, you are right Christian.

Hi Ryan,

I accidentally came across your post ... and I want to say : Thank you very much for your work ... very well done ... what a great tool !

Cheers :)

Awesome, Great tool !!

Thanks Ryan,

very nice tool.

I like it, the philosophy behind it is simple, I tried it very quickly on my ubuntu 16.04 notebook, works, few warnings, however mostly works, will let my colleagues know about it tomorrow. thank you for developing it and sharing it. awesome

Hi, I have odd scenario where a few servers are reporting clean(hardly 50% memory usage) via "free -m" but when I run "xsos -ay" it tells me its 99% used. Our monitoring system has been reporting high memory usage on server. Why does free and xsos report differently?

Hi Ryan Verry nice! Thanks! I am missing disk partitioning and usage infos, but never mind I will file a pull request today (accidently got few frew hours :-)) Please look into the code of my pull request (it took me few minutes to find out how you organized the script) and drop me a line if something is wrong or missing because I would like to write few other "enrichments"

[root@ansible xsos]# bash xsos -d /root/tmp/sosreport-clue1-2019-05-13-nqxiqua
  Whole Disks from /proc/partitions:
    5 disks, totaling 10000 GiB (9.77 TiB)
    (3 multipath/mdraid components hidden)
    - - - - - - - - - - - - - - - - - - - - -
    Disk    Size in GiB
    ----    -----------
    md127   3726
    sda     447
    sdb     238
    sde     3726
    sdg     1863

  lsblk disk layout information:
    sda             8:0    0 447.1G  0 disk  
    `-ssd-ssd     253:1    0   400G  0 lvm   /srv/vm/images-ssd
    sdb             8:16   0 238.5G  0 disk  
    |-sdb1          8:17   0     1G  0 part  /boot
    |-sdb2          8:18   0  15.7G  0 part  [SWAP]
    `-sdb3          8:19   0 221.8G  0 part  /
    sdc             8:32   0   1.8T  0 disk  
    `-sdc1          8:33   0   1.8T  0 part  
      `-md127       9:127  0   3.7T  0 raid5 
        `-raid-vm 253:0    0   2.9T  0 lvm   /srv
    sdd             8:48   0   1.8T  0 disk  
    `-sdd1          8:49   0   1.8T  0 part  
      `-md127       9:127  0   3.7T  0 raid5 
        `-raid-vm 253:0    0   2.9T  0 lvm   /srv
    sde             8:64   0   3.7T  0 disk  
    sdf             8:80   0   1.8T  0 disk  
    `-sdf1          8:81   0   1.8T  0 part  
      `-md127       9:127  0   3.7T  0 raid5 
        `-raid-vm 253:0    0   2.9T  0 lvm   /srv
    sdg             8:96   0   1.8T  0 disk  /srv/backup

Probably cfg2html project may give you more ideas to enrich your/our project Have a great time and thanks again! Chris

Excellent tool! I have just missed "df -h" for File System info/usage. But it's really awesome!

Well done!

Can I use this tool to analyze abrt crashes? If the answer is "yes" what options do I have to use?


No, xsos does not look at data caught by abrt.

Presuming you mean an application core file, you would first install the debuginfo package for whatever crashed. In yum-utils there is debuginfo-install which makes this easier. Then open the core file in gdb debugger.

Between sosreport: 3.2 and version 3.9 the uptime file has moved from sos_commands/general/uptime to sos_commands/host/uptime and xsos just dies. This tool is quite useful and seems to be all that exists to help in this space, but github has been very quiet. I hope it will be updated. The actual error is: gawk: fatal: cannot open file./sosreport/SOSNAME//uptime' for reading (No such file or directory)`

Thanks for this report and the BIOS fix below.

Unfortunately both Ryan and Patrick have moved to other areas of the company so I guess xsos is just me now. I will try get to these soon.

BIOS also has a problem and can be fixed with:

<     SOS_CHECKFILE bios    {,sos_commands/kernel.}dmidecode     && DMIDECODE "$sosroot"
>     SOS_CHECKFILE bios    {,sos_commands/hardware/,sos_commands/kernel.}dmidecode     && DMIDECODE "$sosroot"

uptime doesn't use the same routine to locate the file, so I had to change the actual command and it probably breaks on older sosreports.

Hi BIll, I got a chance to look at these. In both cases it is expected that sos puts a symlink in the root directory for the sosreport, and that's what xsos looks for first:

$ grep sos installed-rpms 

$ readlink dmidecode 

$ readlink uptime 

It sounds like you might have had the symlink fail in the sosreport for some reason, perhaps an incomplete sos collection? Please let me know if there's a way to reproduce this.

Jamie - thanks for looking... doh! Apparently the recreation scenario is 1) untar sosreports on your mac 2) scp/move them to RHEL and loose all your symlinks (not quite sure what I did) 3) wrongly assume sosreports changed.

Aha :) I have been thinking about this in the meantime, and I think we could add support for situations where the symlink hasn't worked for whatever reason. Like you showed with BIOS above, it's not a very difficult condition for xsos to handle. I will try get to this soon.