xsos -- a tool for sysadmins and support techs
My public yum repo now includes an rpm for xsos.
-
To get the rpm that will configure yum for you, see:
How to install xsos or rsar via yum? -
To install the xsos rpm without making it possible to get updates via yum, install:
people.redhat.com/rsawhill/rpms/latest-xsos.rpm -
To go directly to the xsos project page on github, see:
github.com/ryran/xsos -
To try out the latest version of xsos direct from github without installing bash tab-completion, run:
curl -Lo ./xsos bit.ly/xsos-direct; chmod +x ./xsos; ./xsos -ya
For those unfamiliar with xsos, well... a picture is worth a hundred words in this case, so I'll give you 1000+ words covering some of what xsos can do. Note that some of these screenshots were generated by querying sosreports and some of them were generated by executing xsos live on running systems.
xsos (run with no options)
xsos --os --mem --scrub-ip
xsos --bios --ip --scrub-mac
xsos --cpu --lspci --netdev
xsos --ethtool
xsos --bonding
xsos --ps
xsos --kdump
xsos --sysctl
This next one illustrates something special ...
xsos --cpu --intrupt
xsos --softirq
(This one does some diagnosing based on the file /proc/net/softnet_stat. See RFE #108 for details.)
Finally, here's the verbose version of the xsos help page (there's a shorter -h as well)
xsos --help
Usage: xsos [DISPLAY OPTIONS] [-6abokcfmdtlerngisp] [SOSREPORT ROOT]
or: xsos [DISPLAY OPTIONS] {--B|--C|--F|--M|--D|--T|--L|--R|--N|--G|--I|--P FILE}...
or: xsos [-?|-h|--help]
or: xsos [-U|--update]
Display system info from localhost or extracted sosreport
Run with "-h" to see simplified help page
Content options:
-a, --all show everything
-b, --bios show info from dmidecode
-o, --os show hostname, distro, SELinux, kernel info, uptime, etc
-k, --kdump inspect kdump configuration
-c, --cpu show info from /proc/cpuinfo
-f, --intrupt show info from /proc/interrupts
-m, --mem show info from /proc/meminfo
-d, --disks show info from /proc/partitions + dm-multipath synopsis
-t, --mpath show info from dm-multipath
-l, --lspci show info from lspci
-e, --ethtool show info from ethtool
-r, --softirq show info from /proc/net/softnet_stat
-n, --netdev show info from /proc/net/dev
-g, --bonding show info from /proc/net/bonding
-i, --ip show info from ip addr (BASH v4+ required)
--net alias for: --lspci --ethtool --softirq --netdev --bonding --ip
-s, --sysctl show important kernel sysctls
-p, --ps inspect running processes via ps
Display options:
--scrub-ip remove IP addresses & hostnames from output
--scrub-mac remove HW MAC addresses from output
see XSOS_SCRUB_IP_HN & XSOS_SCRUB_MACADDR env vars
-6, --ipv6 parse ip addr output for IPv6 addresses instead of IPv4
-q, --wwid=ID restrict dm-multipath output to a particular mpath device,
where ID is a wwid, friendly name, or LUN identifier
-u, --unit=P change byte display for /proc/meminfo & /proc/net/dev,
where P is "b" for byte, or else "k", "m", "g", or "t"
-v, --verbose=NUM specify ps verbosity level (0-4, default: 1)
-w, --width=NUM change fold-width, in columns (positive number, e.g., 80)
"0" disables wrapping, "w" autodetects width (default)
-x, --nocolor disable output colorization
-y, --less send output to `less -SR`
-z, --more send output to `more`
If no content options are specified, xsos parses the environment variable
XSOS_DEFAULT_VIEW to figure out what information to display. If this variable
is unset at runtime, it is initialized internally as follows:
XSOS_DEFAULT_VIEW='os'
Tweak it to preference by adding additional space-separated MODULE statements,
where MODULE is the same as the long option (e.g. mem, ethtool, netdev). Note
that the --net alias option cannot be used for this purpose. Also note that the
-a / --all option has it's own environment variable: XSOS_ALL_VIEW
If SOSREPORT ROOT isn't provided, the data will be gathered from the localhost;
however, bios, multipath, and ethtool output will only be displayed if running
as root (UID 0). When executing in this manner as non-root, those modules will
be skipped, and a warning printed to stderr.
Sometimes a full sosreport isn't available; sometimes you simply have a
dmidecode-dump or the contents of /proc/meminfo and you'd like a summary...
Special options (BASH v4+ required):
--B=FILE read from FILE containing `dmidecode` dump
--C=FILE read from FILE containing /proc/cpuinfo dump
--F=FILE read from FILE containing /proc/interrupts dump
--M=FILE read from FILE containing /proc/meminfo dump
--D=FILE read from FILE containing /proc/partitions dump
--T=FILE read from FILE containing `multipath -v4 -ll` dump
--L=FILE read from FILE containing `lspci` dump
--R=FILE read from FILE containing /proc/net/softnet_stat dump
--N=FILE read from FILE containing /proc/net/dev dump
--G=FILE read from FILE containing /proc/net/bonding/xxx dump
--I=FILE read from FILE containing `ip addr` dump
--P=FILE read from FILE containing `ps aux` dump
As is hopefully clear, each of these options requires a filename as an
argument. These options can be used together, but cannot be used in concert
with regular "Content options" -- Content opts are ignored if Special options
are detected. Also note: the "=" can be replaced with a space if desired.
Re BASH v4+:
BASH associative arrays are used for various things. In short, if running
xsos on earlier BASH versions (e.g. RHEL5), you get ...
* No output colorization
* No -i/--ip
* No parsing of "Special options"
Environment variables:
For details of all configurable env variables, view first page of xsos
source. There are vars to change default colors as well as other settings.
Each variable name is prefixed with "XSOS_" and the important ones follow.
COLORS FOLD_WIDTH ALL_VIEW DEFAULT_VIEW HEADING_SEPARATOR IP_VERSION
MEM_UNIT NET_UNIT PS_UNIT PS_LEVEL UPDATE_CONFIRM OUTPUT_HANDLER
SCRUB_IP_HN SCRUB_MACADDR ETHTOOL_ERR_REGEX
Version info: xsos v0.5.17 last mod 2015/04/12
See <github.com/ryran/xsos> to report bugs or suggestions
Necessary disclaimer: xsos is an open-source utility developed by me in my free time. I happen to be a Red Hat employee; that does not mean that xsos is shipped, supported, or officially condoned by Red Hat. If you can keep that in mind, you are free to post problems or questions here OR on the github issue tracker.
Responses
Red Hat
Guru
3863 points
Added screenshot for new --bonding option and updated --ethtool screenshot, as the latest version added parsing of eth device rx ring buffer settings to the output. Changelog is on github or of course in the xsos rpm changelog @ people.redhat.com/rsawhill/rpms.
Red Hat
Guru
3863 points
Updated with new screenshot for --kdump. Since my last comment I also added a --scrub option and probably lots of other things.
See rpm or git changelog for changes.
Red Hat
Guru
3863 points
Added a new screenshot for --sysctl which has always been there, but which was given an overhaul today.
Guru
6643 points
Great looking tool, thanks for posting (and keeping it updated)!
Will be sure to add it to my next customer build.
Red Hat
Guru
25394 points
Really appreciate your ongoing work with this, Ryan, and the updates here!
Red Hat
Guru
10579 points
This is such a fantastic tool. I use xsos most days in my work here at Red Hat.
It's useful for quickly pulling relevant information out of a sosreport which speeds our fault analysis and case turnaround times, as well as displaying that information in a clear and presentable way to you, our customers.
I use it mostly for viewing hardware platform and RHEL/kernel versions, for viewing network interface configuration, and looking for errors on the network interfaces - this actually uses an egrep which I contributed to xsos and which has been improved upon since then, making it even more useful for me, a true open source development result!
I'd encourage all customers to try running an xsos -a over your own uncompressed sosreports to see how the information on your system is represented. You may find it a useful addition to your own reporting and analysis.
Red Hat
Guru
3863 points
New v0.4.3 rpm addressing RHEL7 issues pushed to the repo.
* Sun Jun 22 2014 Ryan Sawhill Aroha <rsaw@redhat.com> 0.4.3-1
- Modified -b / --bios display to show more info, per issue 98
- Added RHEL7 to supported systems regex due to GA release this month
- Tweaked the way /etc/os-release is parsed & displayed
- Tweaked the way /proc/cmdline is parsed: "BOOT_IMAGE=/vmlinuz..." is now
automatically removed from the beginning of the kernel cmdline before display
making it easier to compare default & current kernel cmdline
- See issue 99 for details on the RHEL7-specific kdump fixes
Since the last comment, most of the changes to xsos have tweaked the way xsos --ip works:
- fixed a few bugs
- changed the order of the columns to be [hopefully] more logical
- completely changed the way
--scrub-ipand--scrub-macare handled - added a "Scope" column for IPv6
Updated screenshot in the original post.
Red Hat
Guru
3863 points
Some new networking funness for the new year:
* Thu Jan 01 2015 Ryan Sawhill Aroha <rsaw@redhat.com> 0.5.5-2
- Implemented new module based on #108 RFE: Alert on network SoftIRQ backlog or
budget drops..added new options (`-r`, `--R`, `--softirq`) to bash-completion
- Added 2 new sysctls to --sysctl output:
net.ipv4.tcp_sack and net.ipv4.tcp_timestamps
- Replaced all references to awk with gawk since some gawk-specific features
are used
- Fixed #109 (awk division by zero error in weird NETDEV output)
Added updated screenshot above for new bonding layout. Enhancements for troubleshooting mode4 bonding, per RFE: more details on bonding mode 4. Thanks to PTalbert of GSS for his awesome suggestions.
Red Hat
Guru
3863 points
New rpm hit the yum repo today. I didn't update any of the screenshots, but there are a couple medium-sized changes in some of the modules' output, along with a handful of other lil fixes.
[rsawhill]$ rpm -qp --changelog latest-xsos.rpm | head
* Thu Feb 26 2015 Ryan Sawhill Aroha <rsaw@redhat.com> 0.5.12-1
- Add pause to the ethtool -S regex
- New features added to bonding mode 4 (#115)
- Add PCI address information in ethtool output (#114)
- Fix "no df_output .. shouldn't happen" (#119)
- Fix lspci missing VGA with really long pci ids (#121)
- lspci overhaul: split up net/storage output, show more storage stuff (what
lspci module shows is now configurable by environment variables -- #122)
* Thu Jan 01 2015 Ryan Sawhill Aroha <rsaw@redhat.com> 0.5.5-2
Pushed new v0.5.17-1 rpm to the yum repo today.
[rsaw:noarch]$ rpm -qp --changelog xsos-0.5.17-1.noarch.rpm|head -12
* Sun Apr 12 2015 Ryan Sawhill Aroha <rsaw@redhat.com> 0.5.17-1
- Add RHSM details to os module, along with additional yum plugins funness and
proxy-parsing for both RHN & RHSM (#127)
- Add new -f/--F/--intrupt options to parse /proc/interrupts data into a binary
graph (#131)
- Notably, the softirq module (-r/--softirq) is now included by xsos -a/--all,
as is the new -f/--intrupt module
- Make kdump svc-enablement status w/systemd (e.g., RHEL7) more clear (#129)
- Make memgraph (--mem) characters simpler & more legible -- especially
important on virtual consoles (#130)
* Thu Feb 26 2015 Ryan Sawhill Aroha <rsaw@redhat.com> 0.5.12-1
Updated original post with new screenshots for --softirq and the new --intrupt option.
The other big change is the addition of yum-plugins and RHSM to the os module, though that's likely only interesting to Red Hat support techs.
Hi Rayn,
Thanks for xsos script, its really very useful, when i ran xsos --ps, i see the %cpu is showing as 1000% in a server. please check the below details.
PS CHECK
Total number of threads/processes:
6906 / 3037
Top users of CPU & MEM:
USER %CPU %MEM RSS
oracle 1058.7% 30.0% 137.80 GiB --> could you explain what does it mean and how is it being calculated.
root 31.2% 0.2% 2.17 GiB
68 1.9% 0.0% 0.04 GiB
dk016140 0.2% 0.0% 0.01 GiB
Thanks
Veera.
Red Hat
Guru
3863 points
Hi Veera. In short: for single-threaded processes, ps reports "%CPU" as a percentage of ONE logical processor; not the total available system processing power. If root has multiple processes that are pegging their respective CPUs (as in the following screenshot), then root's total "%CPU" will be above 100%. Check it out:
people.redhat.com/rsawhill/scratch/xsos-ps-100+.png
Note also that a single process can show more than 100% if it's multi-threaded.
Red Hat
Guru
3863 points
Haven't been keeping this thread updated. Since the last time I posted, there have been quite a few releases (github.com/ryran/xsos/releases) including a release today to enhance dmidecode-parsing.
Hi Ryan,
The "--scrub-ip" & "--scrub-mac" options are not working, only "--scrub" works which scrubs ip, mac,hostname etc,. but option of scrubbing only ip or mac doesn't work. Please check on this:
[root@localhost ~]# xsos --ip --scrub-ip
xsos: unrecognized option '--scrub-ip'
Usage: xsos [DISPLAY OPTIONS] [-6abokcfmdtlerngisp] [SOSREPORT ROOT]
or: xsos [DISPLAY OPTIONS] {--B|--C|--F|--M|--D|--T|--L|--R|--N|--G|--I|--P FILE}...
or: xsos [-?|-h|--help]
Display system info from localhost or extracted sosreport
[root@localhost ~]# xsos --ip --scrub-mac
xsos: unrecognized option '--scrub-mac'
Usage: xsos [DISPLAY OPTIONS] [-6abokcfmdtlerngisp] [SOSREPORT ROOT]
or: xsos [DISPLAY OPTIONS] {--B|--C|--F|--M|--D|--T|--L|--R|--N|--G|--I|--P FILE}...
or: xsos [-?|-h|--help]
Display system info from localhost or extracted sosreport
I'm using the latest one even I tried the previous version as well:
[root@localhost ~]# rpm -q xsos
xsos-0.7.11-1.noarch
Other parameters and options are working as expected.
From the changelog I could see this :-
* Sat Jul 11 2015 Ryan Sawhill Aroha <rsaw@redhat.com> 0.7.0-1
- Remove --scrub-ip and --scrub-mac
- Add XSOS_SCRUB_SERIAL env variable (to the already-existing XSOS_SCRUB_IP_HN
and XSOS_SCRUB_MACADDR vars)
- Add --scrub option that does all 3
Does this mean "--scrub-mac" & "--scrub-ip" doesn't work, however, "--scrub" does it all? But mac/ip scrubbing independently would be a nice option.
Guru
27750 points
Hi Sadashiva,
I can confirm the same : the options --scrub-mac and --scrub-ip do not work ... but the option --scrub does.
Waves :)
Christian
Also, nice if this gets added to epel repo...
Guru
27750 points
Hi Sadashiva,
Well, something even better : I found out that xsos is completely 'portable', simply download the source code from https://github.com/ryran/xsos/releases, place it where ever you want, make the file executable and launch the tool with ./xsos. I've already added this useful application to my toolbox and put it on all machines and systems today.
Waves :)
Christian
Guru
27750 points
Hi Ryan,
I accidentally came across your post ... and I want to say : Thank you very much for your work ... very well done ... what a great tool !
Cheers :)
Christian
Expert
1111 points
I like it, the philosophy behind it is simple, I tried it very quickly on my ubuntu 16.04 notebook, works, few warnings, however mostly works, will let my colleagues know about it tomorrow. thank you for developing it and sharing it. awesome
Hi, I have odd scenario where a few servers are reporting clean(hardly 50% memory usage) via "free -m" but when I run "xsos -ay" it tells me its 99% used. Our monitoring system has been reporting high memory usage on server. Why does free and xsos report differently?
Community Member
46 points
Hi Ryan Verry nice! Thanks! I am missing disk partitioning and usage infos, but never mind I will file a pull request today (accidently got few frew hours :-)) Please look into the code of my pull request (it took me few minutes to find out how you organized the script) and drop me a line if something is wrong or missing because I would like to write few other "enrichments"
[root@ansible xsos]# bash xsos -d /root/tmp/sosreport-clue1-2019-05-13-nqxiqua
STORAGE
Whole Disks from /proc/partitions:
5 disks, totaling 10000 GiB (9.77 TiB)
(3 multipath/mdraid components hidden)
- - - - - - - - - - - - - - - - - - - - -
Disk Size in GiB
---- -----------
md127 3726
sda 447
sdb 238
sde 3726
sdg 1863
lsblk disk layout information:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 447.1G 0 disk
`-ssd-ssd 253:1 0 400G 0 lvm /srv/vm/images-ssd
sdb 8:16 0 238.5G 0 disk
|-sdb1 8:17 0 1G 0 part /boot
|-sdb2 8:18 0 15.7G 0 part [SWAP]
`-sdb3 8:19 0 221.8G 0 part /
sdc 8:32 0 1.8T 0 disk
`-sdc1 8:33 0 1.8T 0 part
`-md127 9:127 0 3.7T 0 raid5
`-raid-vm 253:0 0 2.9T 0 lvm /srv
sdd 8:48 0 1.8T 0 disk
`-sdd1 8:49 0 1.8T 0 part
`-md127 9:127 0 3.7T 0 raid5
`-raid-vm 253:0 0 2.9T 0 lvm /srv
sde 8:64 0 3.7T 0 disk
sdf 8:80 0 1.8T 0 disk
`-sdf1 8:81 0 1.8T 0 part
`-md127 9:127 0 3.7T 0 raid5
`-raid-vm 253:0 0 2.9T 0 lvm /srv
sdg 8:96 0 1.8T 0 disk /srv/backup
Probably cfg2html project may give you more ideas to enrich your/our project Have a great time and thanks again! Chris
Excellent tool! I have just missed "df -h" for File System info/usage. But it's really awesome!
Well done!
Can I use this tool to analyze abrt crashes? If the answer is "yes" what options do I have to use?
Thanks
Red Hat
Guru
10579 points
No, xsos does not look at data caught by abrt.
Presuming you mean an application core file, you would first install the debuginfo package for whatever crashed. In yum-utils there is debuginfo-install which makes this easier. Then open the core file in gdb debugger.
Between sosreport: 3.2 and version 3.9 the uptime file has moved from sos_commands/general/uptime to sos_commands/host/uptime and xsos just dies.
This tool is quite useful and seems to be all that exists to help in this space, but github has been very quiet. I hope it will be updated.
The actual error is:
gawk: fatal: cannot open file./sosreport/SOSNAME//uptime' for reading (No such file or directory)`
BIOS also has a problem and can be fixed with:
3422c3422
< SOS_CHECKFILE bios {,sos_commands/kernel.}dmidecode && DMIDECODE "$sosroot"
---
> SOS_CHECKFILE bios {,sos_commands/hardware/,sos_commands/kernel.}dmidecode && DMIDECODE "$sosroot"
uptime doesn't use the same routine to locate the file, so I had to change the actual command and it probably breaks on older sosreports.
Red Hat
Guru
10579 points
Hi BIll, I got a chance to look at these. In both cases it is expected that sos puts a symlink in the root directory for the sosreport, and that's what xsos looks for first:
$ grep sos installed-rpms
sos-3.9.1-6.el8.noarch
$ readlink dmidecode
sos_commands/hardware/dmidecode
$ readlink uptime
sos_commands/host/uptime
It sounds like you might have had the symlink fail in the sosreport for some reason, perhaps an incomplete sos collection? Please let me know if there's a way to reproduce this.
Jamie - thanks for looking... doh! Apparently the recreation scenario is 1) untar sosreports on your mac 2) scp/move them to RHEL and loose all your symlinks (not quite sure what I did) 3) wrongly assume sosreports changed.
Red Hat
Guru
10579 points
Aha :) I have been thinking about this in the meantime, and I think we could add support for situations where the symlink hasn't worked for whatever reason. Like you showed with BIOS above, it's not a very difficult condition for xsos to handle. I will try get to this soon.