xsos -- a tool for sysadmins and support techs

Latest response

My public yum repo now includes an rpm for xsos.

For those unfamiliar with xsos, well... a picture is worth a hundred words in this case, so I'll give you 1000+ words covering some of what xsos can do. Note that some of these screenshots were generated by querying sosreports and some of them were generated by executing xsos live on running systems.

xsos (run with no options)

xsos --os --mem --scrub-ip

xsos --bios --ip --scrub-mac


xsos --cpu --lspci --netdev


xsos --ethtool


xsos --bonding


xsos --ps


xsos --kdump


xsos --sysctl


This next one illustrates something special ...


xsos --cpu --intrupt


xsos --softirq
(This one does some diagnosing based on the file /proc/net/softnet_stat. See RFE #108 for details.)


Finally, here's the verbose version of the xsos help page (there's a shorter -h as well)
xsos --help

Usage: xsos [DISPLAY OPTIONS] [-6abokcfmdtlerngisp] [SOSREPORT ROOT]
  or:  xsos [DISPLAY OPTIONS] {--B|--C|--F|--M|--D|--T|--L|--R|--N|--G|--I|--P FILE}...
  or:  xsos [-?|-h|--help]
  or:  xsos [-U|--update]

Display system info from localhost or extracted sosreport
Run with "-h" to see simplified help page

Content options:
 -a, --all      show everything
 -b, --bios     show info from dmidecode
 -o, --os       show hostname, distro, SELinux, kernel info, uptime, etc
 -k, --kdump    inspect kdump configuration
 -c, --cpu      show info from /proc/cpuinfo
 -f, --intrupt  show info from /proc/interrupts
 -m, --mem      show info from /proc/meminfo
 -d, --disks    show info from /proc/partitions + dm-multipath synopsis
 -t, --mpath    show info from dm-multipath
 -l, --lspci    show info from lspci
 -e, --ethtool  show info from ethtool
 -r, --softirq  show info from /proc/net/softnet_stat
 -n, --netdev   show info from /proc/net/dev
 -g, --bonding  show info from /proc/net/bonding
 -i, --ip       show info from ip addr (BASH v4+ required)
     --net      alias for: --lspci --ethtool --softirq --netdev --bonding --ip
 -s, --sysctl   show important kernel sysctls
 -p, --ps       inspect running processes via ps

Display options:
     --scrub-ip     remove IP addresses & hostnames from output
     --scrub-mac    remove HW MAC addresses from output
                    see XSOS_SCRUB_IP_HN & XSOS_SCRUB_MACADDR env vars
 -6, --ipv6         parse ip addr output for IPv6 addresses instead of IPv4
 -q, --wwid=ID      restrict dm-multipath output to a particular mpath device,
                    where ID is a wwid, friendly name, or LUN identifier
 -u, --unit=P       change byte display for /proc/meminfo & /proc/net/dev,
                    where P is "b" for byte, or else "k", "m", "g", or "t"
 -v, --verbose=NUM  specify ps verbosity level (0-4, default: 1)
 -w, --width=NUM    change fold-width, in columns (positive number, e.g., 80)
                    "0" disables wrapping, "w" autodetects width (default)
 -x, --nocolor      disable output colorization
 -y, --less         send output to `less -SR`
 -z, --more         send output to `more`

If no content options are specified, xsos parses the environment variable
XSOS_DEFAULT_VIEW to figure out what information to display. If this variable
is unset at runtime, it is initialized internally as follows:

   XSOS_DEFAULT_VIEW='os'

Tweak it to preference by adding additional space-separated MODULE statements,
where MODULE is the same as the long option (e.g. mem, ethtool, netdev). Note
that the --net alias option cannot be used for this purpose. Also note that the
-a / --all option has it's own environment variable: XSOS_ALL_VIEW

If SOSREPORT ROOT isn't provided, the data will be gathered from the localhost;
however, bios, multipath, and ethtool output will only be displayed if running
as root (UID 0). When executing in this manner as non-root, those modules will
be skipped, and a warning printed to stderr.

Sometimes a full sosreport isn't available; sometimes you simply have a
dmidecode-dump or the contents of /proc/meminfo and you'd like a summary...

Special options (BASH v4+ required):
 --B=FILE  read from FILE containing `dmidecode` dump
 --C=FILE  read from FILE containing /proc/cpuinfo dump
 --F=FILE  read from FILE containing /proc/interrupts dump
 --M=FILE  read from FILE containing /proc/meminfo dump
 --D=FILE  read from FILE containing /proc/partitions dump
 --T=FILE  read from FILE containing `multipath -v4 -ll` dump
 --L=FILE  read from FILE containing `lspci` dump
 --R=FILE  read from FILE containing /proc/net/softnet_stat dump
 --N=FILE  read from FILE containing /proc/net/dev dump
 --G=FILE  read from FILE containing /proc/net/bonding/xxx dump
 --I=FILE  read from FILE containing `ip addr` dump
 --P=FILE  read from FILE containing `ps aux` dump

As is hopefully clear, each of these options requires a filename as an
argument. These options can be used together, but cannot be used in concert
with regular "Content options" -- Content opts are ignored if Special options
are detected. Also note: the "=" can be replaced with a space if desired.

Re BASH v4+:
 BASH associative arrays are used for various things. In short, if running
 xsos on earlier BASH versions (e.g. RHEL5), you get ...
  * No output colorization
  * No -i/--ip
  * No parsing of "Special options"

Environment variables:
 For details of all configurable env variables, view first page of xsos
 source. There are vars to change default colors as well as other settings.
 Each variable name is prefixed with "XSOS_" and the important ones follow.
  COLORS  FOLD_WIDTH  ALL_VIEW  DEFAULT_VIEW  HEADING_SEPARATOR  IP_VERSION
  MEM_UNIT  NET_UNIT  PS_UNIT  PS_LEVEL  UPDATE_CONFIRM  OUTPUT_HANDLER
  SCRUB_IP_HN  SCRUB_MACADDR  ETHTOOL_ERR_REGEX

Version info: xsos v0.5.17 last mod 2015/04/12
See <github.com/ryran/xsos> to report bugs or suggestions

Necessary disclaimer: xsos is an open-source utility developed by me in my free time. I happen to be a Red Hat employee; that does not mean that xsos is shipped, supported, or officially condoned by Red Hat. If you can keep that in mind, you are free to post problems or questions here OR on the github issue tracker.

Responses

Very nice!!!

Added screenshot for new --bonding option and updated --ethtool screenshot, as the latest version added parsing of eth device rx ring buffer settings to the output. Changelog is on github or of course in the xsos rpm changelog @ people.redhat.com/rsawhill/rpms.

Really Usefull Tool!

I'm glad you're finding a use for it! Let me know if you have any suggestions.

Updated with new screenshot for --kdump. Since my last comment I also added a --scrub option and probably lots of other things.

See rpm or git changelog for changes.

Excellent work Ryan!

+1 - great stuff. I look forward to seeing more awesomeness!!

Added a new screenshot for --sysctl which has always been there, but which was given an overhaul today.

Great looking tool, thanks for posting (and keeping it updated)!

Will be sure to add it to my next customer build.

Really appreciate your ongoing work with this, Ryan, and the updates here!

This is such a fantastic tool. I use xsos most days in my work here at Red Hat.

It's useful for quickly pulling relevant information out of a sosreport which speeds our fault analysis and case turnaround times, as well as displaying that information in a clear and presentable way to you, our customers.

I use it mostly for viewing hardware platform and RHEL/kernel versions, for viewing network interface configuration, and looking for errors on the network interfaces - this actually uses an egrep which I contributed to xsos and which has been improved upon since then, making it even more useful for me, a true open source development result!

I'd encourage all customers to try running an xsos -a over your own uncompressed sosreports to see how the information on your system is represented. You may find it a useful addition to your own reporting and analysis.

Ryan, Great work. Ryan++
You have created very useful tool. Thanks.

New v0.4.3 rpm addressing RHEL7 issues pushed to the repo.

* Sun Jun 22 2014 Ryan Sawhill Aroha <rsaw@redhat.com> 0.4.3-1
- Modified -b / --bios display to show more info, per issue 98
- Added RHEL7 to supported systems regex due to GA release this month
- Tweaked the way /etc/os-release is parsed & displayed
- Tweaked the way /proc/cmdline is parsed: "BOOT_IMAGE=/vmlinuz..." is now
  automatically removed from the beginning of the kernel cmdline before display
  making it easier to compare default & current kernel cmdline
- See issue 99 for details on the RHEL7-specific kdump fixes

New v0.4.3 screenshot showing RHEL7

(changelog)

Since the last comment, most of the changes to xsos have tweaked the way xsos --ip works:

  • fixed a few bugs
  • changed the order of the columns to be [hopefully] more logical
  • completely changed the way --scrub-ip and --scrub-mac are handled
  • added a "Scope" column for IPv6

Updated screenshot in the original post.

Some new networking funness for the new year:

* Thu Jan 01 2015 Ryan Sawhill Aroha <rsaw@redhat.com> 0.5.5-2
- Implemented new module based on #108 RFE: Alert on network SoftIRQ backlog or
  budget drops..added new options (`-r`, `--R`, `--softirq`) to bash-completion
- Added 2 new sysctls to --sysctl output:
  net.ipv4.tcp_sack and net.ipv4.tcp_timestamps
- Replaced all references to awk with gawk since some gawk-specific features
  are used
- Fixed #109 (awk division by zero error in weird NETDEV output)

Screenshot showing Jamie's new feature from #108

Added updated screenshot above for new bonding layout. Enhancements for troubleshooting mode4 bonding, per RFE: more details on bonding mode 4. Thanks to PTalbert of GSS for his awesome suggestions.

Super cool! Like it a lot.

New rpm hit the yum repo today. I didn't update any of the screenshots, but there are a couple medium-sized changes in some of the modules' output, along with a handful of other lil fixes.

[rsawhill]$ rpm -qp --changelog latest-xsos.rpm | head
* Thu Feb 26 2015 Ryan Sawhill Aroha <rsaw@redhat.com> 0.5.12-1
- Add pause to the ethtool -S regex
- New features added to bonding mode 4 (#115)
- Add PCI address information in ethtool output (#114)
- Fix "no df_output .. shouldn't happen" (#119)
- Fix lspci missing VGA with really long pci ids (#121)
- lspci overhaul: split up net/storage output, show more storage stuff (what
  lspci module shows is now configurable by environment variables -- #122)

* Thu Jan 01 2015 Ryan Sawhill Aroha <rsaw@redhat.com> 0.5.5-2

Pushed new v0.5.17-1 rpm to the yum repo today.

[rsaw:noarch]$ rpm -qp --changelog xsos-0.5.17-1.noarch.rpm|head -12
* Sun Apr 12 2015 Ryan Sawhill Aroha <rsaw@redhat.com> 0.5.17-1
- Add RHSM details to os module, along with additional yum plugins funness and
  proxy-parsing for both RHN & RHSM (#127)
- Add new -f/--F/--intrupt options to parse /proc/interrupts data into a binary
  graph (#131)
- Notably, the softirq module (-r/--softirq) is now included by xsos -a/--all,
  as is the new -f/--intrupt module
- Make kdump svc-enablement status w/systemd (e.g., RHEL7) more clear (#129)
- Make memgraph (--mem) characters simpler & more legible -- especially
  important on virtual consoles (#130)

* Thu Feb 26 2015 Ryan Sawhill Aroha <rsaw@redhat.com> 0.5.12-1

Updated original post with new screenshots for --softirq and the new --intrupt option.

The other big change is the addition of yum-plugins and RHSM to the os module, though that's likely only interesting to Red Hat support techs.

Thanks for keeping this updated, Ryan!

Hi Rayn,

Thanks for xsos script, its really very useful, when i ran xsos --ps, i see the %cpu is showing as 1000% in a server. please check the below details.

PS CHECK
Total number of threads/processes:
6906 / 3037
Top users of CPU & MEM:
USER %CPU %MEM RSS
oracle 1058.7% 30.0% 137.80 GiB --> could you explain what does it mean and how is it being calculated.
root 31.2% 0.2% 2.17 GiB
68 1.9% 0.0% 0.04 GiB
dk016140 0.2% 0.0% 0.01 GiB

Thanks
Veera.

Hi Veera. In short: for single-threaded processes, ps reports "%CPU" as a percentage of ONE logical processor; not the total available system processing power. If root has multiple processes that are pegging their respective CPUs (as in the following screenshot), then root's total "%CPU" will be above 100%. Check it out:

people.redhat.com/rsawhill/scratch/xsos-ps-100+.png

Note also that a single process can show more than 100% if it's multi-threaded.

Great job Ryan!!! I love it! Such a useful tool!!

Thanks for bumping the post, I discovered a great tool :-)

Excellent tool!

yes, this is excellent and useful. Kudos!!!

Haven't been keeping this thread updated. Since the last time I posted, there have been quite a few releases (github.com/ryran/xsos/releases) including a release today to enhance dmidecode-parsing.

screenshot of old-buggy vs new hotness

Hi Ryan,

The "--scrub-ip" & "--scrub-mac" options are not working, only "--scrub" works which scrubs ip, mac,hostname etc,. but option of scrubbing only ip or mac doesn't work. Please check on this:

[root@localhost ~]# xsos --ip --scrub-ip
xsos: unrecognized option '--scrub-ip'
Usage: xsos [DISPLAY OPTIONS] [-6abokcfmdtlerngisp] [SOSREPORT ROOT]
  or:  xsos [DISPLAY OPTIONS] {--B|--C|--F|--M|--D|--T|--L|--R|--N|--G|--I|--P FILE}...
  or:  xsos [-?|-h|--help]

Display system info from localhost or extracted sosreport
[root@localhost ~]# xsos --ip --scrub-mac
xsos: unrecognized option '--scrub-mac'
Usage: xsos [DISPLAY OPTIONS] [-6abokcfmdtlerngisp] [SOSREPORT ROOT]
  or:  xsos [DISPLAY OPTIONS] {--B|--C|--F|--M|--D|--T|--L|--R|--N|--G|--I|--P FILE}...
  or:  xsos [-?|-h|--help]

Display system info from localhost or extracted sosreport

I'm using the latest one even I tried the previous version as well:

[root@localhost ~]# rpm -q xsos
xsos-0.7.11-1.noarch

Other parameters and options are working as expected.

From the changelog I could see this :-

* Sat Jul 11 2015 Ryan Sawhill Aroha <rsaw@redhat.com> 0.7.0-1
- Remove --scrub-ip and --scrub-mac
- Add XSOS_SCRUB_SERIAL env variable (to the already-existing XSOS_SCRUB_IP_HN
  and XSOS_SCRUB_MACADDR vars)
- Add --scrub option that does all 3

Does this mean "--scrub-mac" & "--scrub-ip" doesn't work, however, "--scrub" does it all? But mac/ip scrubbing independently would be a nice option.

Hi Sadashiva,

I can confirm the same : the options --scrub-mac and --scrub-ip do not work ... but the option --scrub does.

Waves :)
Christian

Indeed. Back in 2005 I removed the two independent options. See also changelogs here: https://github.com/ryran/xsos/releases

Okay, then you might consider editing the screen shots those talks about using --scrub-ip and --scrub-mac options.

Also, nice if this gets added to epel repo...

Hi Sadashiva,

Well, something even better : I found out that xsos is completely 'portable', simply download the source code from https://github.com/ryran/xsos/releases, place it where ever you want, make the file executable and launch the tool with ./xsos. I've already added this useful application to my toolbox and put it on all machines and systems today.

Waves :)
Christian

Yes, you are right Christian.

Hi Ryan,

I accidentally came across your post ... and I want to say : Thank you very much for your work ... very well done ... what a great tool !

Cheers :)
Christian

Awesome, Great tool !!

Thanks Ryan,

very nice tool.

I like it, the philosophy behind it is simple, I tried it very quickly on my ubuntu 16.04 notebook, works, few warnings, however mostly works, will let my colleagues know about it tomorrow. thank you for developing it and sharing it. awesome

Hi, I have odd scenario where a few servers are reporting clean(hardly 50% memory usage) via "free -m" but when I run "xsos -ay" it tells me its 99% used. Our monitoring system has been reporting high memory usage on server. Why does free and xsos report differently?

Hi Ryan Verry nice! Thanks! I am missing disk partitioning and usage infos, but never mind I will file a pull request today (accidently got few frew hours :-)) Please look into the code of my pull request (it took me few minutes to find out how you organized the script) and drop me a line if something is wrong or missing because I would like to write few other "enrichments"

[root@ansible xsos]# bash xsos -d /root/tmp/sosreport-clue1-2019-05-13-nqxiqua
STORAGE
  Whole Disks from /proc/partitions:
    5 disks, totaling 10000 GiB (9.77 TiB)
    (3 multipath/mdraid components hidden)
    - - - - - - - - - - - - - - - - - - - - -
    Disk    Size in GiB
    ----    -----------
    md127   3726
    sda     447
    sdb     238
    sde     3726
    sdg     1863

  lsblk disk layout information:
    NAME          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
    sda             8:0    0 447.1G  0 disk  
    `-ssd-ssd     253:1    0   400G  0 lvm   /srv/vm/images-ssd
    sdb             8:16   0 238.5G  0 disk  
    |-sdb1          8:17   0     1G  0 part  /boot
    |-sdb2          8:18   0  15.7G  0 part  [SWAP]
    `-sdb3          8:19   0 221.8G  0 part  /
    sdc             8:32   0   1.8T  0 disk  
    `-sdc1          8:33   0   1.8T  0 part  
      `-md127       9:127  0   3.7T  0 raid5 
        `-raid-vm 253:0    0   2.9T  0 lvm   /srv
    sdd             8:48   0   1.8T  0 disk  
    `-sdd1          8:49   0   1.8T  0 part  
      `-md127       9:127  0   3.7T  0 raid5 
        `-raid-vm 253:0    0   2.9T  0 lvm   /srv
    sde             8:64   0   3.7T  0 disk  
    sdf             8:80   0   1.8T  0 disk  
    `-sdf1          8:81   0   1.8T  0 part  
      `-md127       9:127  0   3.7T  0 raid5 
        `-raid-vm 253:0    0   2.9T  0 lvm   /srv
    sdg             8:96   0   1.8T  0 disk  /srv/backup

Probably cfg2html project may give you more ideas to enrich your/our project Have a great time and thanks again! Chris