xsos -- a tool for sysadmins and support techs
My public yum repo now includes an rpm for xsos.
-
To get the rpm that will configure yum for you, see:
How to install xsos or rsar via yum? -
To install the xsos rpm without making it possible to get updates via yum, install:
people.redhat.com/rsawhill/rpms/latest-xsos.rpm -
To go directly to the xsos project page on github, see:
github.com/ryran/xsos -
To try out the latest version of xsos direct from github without installing bash tab-completion, run:
curl -Lo ./xsos bit.ly/xsos-direct; chmod +x ./xsos; ./xsos -ya
For those unfamiliar with xsos, well... a picture is worth a hundred words in this case, so I'll give you 1000+ words covering some of what xsos can do. Note that some of these screenshots were generated by querying sosreports and some of them were generated by executing xsos
live on running systems.
xsos --bios --ip --scrub-mac
xsos --cpu --lspci --netdev
xsos --ethtool
xsos --bonding
xsos --ps
xsos --kdump
xsos --sysctl
This next one illustrates something special ...
xsos --cpu --intrupt
xsos --softirq
(This one does some diagnosing based on the file /proc/net/softnet_stat. See RFE #108 for details.)
Finally, here's the verbose version of the xsos help page (there's a shorter -h
as well)
xsos --help
Usage: xsos [DISPLAY OPTIONS] [-6abokcfmdtlerngisp] [SOSREPORT ROOT]
or: xsos [DISPLAY OPTIONS] {--B|--C|--F|--M|--D|--T|--L|--R|--N|--G|--I|--P FILE}...
or: xsos [-?|-h|--help]
or: xsos [-U|--update]
Display system info from localhost or extracted sosreport
Run with "-h" to see simplified help page
Content options:
-a, --all show everything
-b, --bios show info from dmidecode
-o, --os show hostname, distro, SELinux, kernel info, uptime, etc
-k, --kdump inspect kdump configuration
-c, --cpu show info from /proc/cpuinfo
-f, --intrupt show info from /proc/interrupts
-m, --mem show info from /proc/meminfo
-d, --disks show info from /proc/partitions + dm-multipath synopsis
-t, --mpath show info from dm-multipath
-l, --lspci show info from lspci
-e, --ethtool show info from ethtool
-r, --softirq show info from /proc/net/softnet_stat
-n, --netdev show info from /proc/net/dev
-g, --bonding show info from /proc/net/bonding
-i, --ip show info from ip addr (BASH v4+ required)
--net alias for: --lspci --ethtool --softirq --netdev --bonding --ip
-s, --sysctl show important kernel sysctls
-p, --ps inspect running processes via ps
Display options:
--scrub-ip remove IP addresses & hostnames from output
--scrub-mac remove HW MAC addresses from output
see XSOS_SCRUB_IP_HN & XSOS_SCRUB_MACADDR env vars
-6, --ipv6 parse ip addr output for IPv6 addresses instead of IPv4
-q, --wwid=ID restrict dm-multipath output to a particular mpath device,
where ID is a wwid, friendly name, or LUN identifier
-u, --unit=P change byte display for /proc/meminfo & /proc/net/dev,
where P is "b" for byte, or else "k", "m", "g", or "t"
-v, --verbose=NUM specify ps verbosity level (0-4, default: 1)
-w, --width=NUM change fold-width, in columns (positive number, e.g., 80)
"0" disables wrapping, "w" autodetects width (default)
-x, --nocolor disable output colorization
-y, --less send output to `less -SR`
-z, --more send output to `more`
If no content options are specified, xsos parses the environment variable
XSOS_DEFAULT_VIEW to figure out what information to display. If this variable
is unset at runtime, it is initialized internally as follows:
XSOS_DEFAULT_VIEW='os'
Tweak it to preference by adding additional space-separated MODULE statements,
where MODULE is the same as the long option (e.g. mem, ethtool, netdev). Note
that the --net alias option cannot be used for this purpose. Also note that the
-a / --all option has it's own environment variable: XSOS_ALL_VIEW
If SOSREPORT ROOT isn't provided, the data will be gathered from the localhost;
however, bios, multipath, and ethtool output will only be displayed if running
as root (UID 0). When executing in this manner as non-root, those modules will
be skipped, and a warning printed to stderr.
Sometimes a full sosreport isn't available; sometimes you simply have a
dmidecode-dump or the contents of /proc/meminfo and you'd like a summary...
Special options (BASH v4+ required):
--B=FILE read from FILE containing `dmidecode` dump
--C=FILE read from FILE containing /proc/cpuinfo dump
--F=FILE read from FILE containing /proc/interrupts dump
--M=FILE read from FILE containing /proc/meminfo dump
--D=FILE read from FILE containing /proc/partitions dump
--T=FILE read from FILE containing `multipath -v4 -ll` dump
--L=FILE read from FILE containing `lspci` dump
--R=FILE read from FILE containing /proc/net/softnet_stat dump
--N=FILE read from FILE containing /proc/net/dev dump
--G=FILE read from FILE containing /proc/net/bonding/xxx dump
--I=FILE read from FILE containing `ip addr` dump
--P=FILE read from FILE containing `ps aux` dump
As is hopefully clear, each of these options requires a filename as an
argument. These options can be used together, but cannot be used in concert
with regular "Content options" -- Content opts are ignored if Special options
are detected. Also note: the "=" can be replaced with a space if desired.
Re BASH v4+:
BASH associative arrays are used for various things. In short, if running
xsos on earlier BASH versions (e.g. RHEL5), you get ...
* No output colorization
* No -i/--ip
* No parsing of "Special options"
Environment variables:
For details of all configurable env variables, view first page of xsos
source. There are vars to change default colors as well as other settings.
Each variable name is prefixed with "XSOS_" and the important ones follow.
COLORS FOLD_WIDTH ALL_VIEW DEFAULT_VIEW HEADING_SEPARATOR IP_VERSION
MEM_UNIT NET_UNIT PS_UNIT PS_LEVEL UPDATE_CONFIRM OUTPUT_HANDLER
SCRUB_IP_HN SCRUB_MACADDR ETHTOOL_ERR_REGEX
Version info: xsos v0.5.17 last mod 2015/04/12
See <github.com/ryran/xsos> to report bugs or suggestions
Necessary disclaimer: xsos is an open-source utility developed by me in my free time. I happen to be a Red Hat employee; that does not mean that xsos is shipped, supported, or officially condoned by Red Hat. If you can keep that in mind, you are free to post problems or questions here OR on the github issue tracker.
Responses
This is such a fantastic tool. I use xsos most days in my work here at Red Hat.
It's useful for quickly pulling relevant information out of a sosreport which speeds our fault analysis and case turnaround times, as well as displaying that information in a clear and presentable way to you, our customers.
I use it mostly for viewing hardware platform and RHEL/kernel versions, for viewing network interface configuration, and looking for errors on the network interfaces - this actually uses an egrep which I contributed to xsos and which has been improved upon since then, making it even more useful for me, a true open source development result!
I'd encourage all customers to try running an xsos -a
over your own uncompressed sosreports to see how the information on your system is represented. You may find it a useful addition to your own reporting and analysis.
Hi Rayn,
Thanks for xsos script, its really very useful, when i ran xsos --ps, i see the %cpu is showing as 1000% in a server. please check the below details.
PS CHECK
Total number of threads/processes:
6906 / 3037
Top users of CPU & MEM:
USER %CPU %MEM RSS
oracle 1058.7% 30.0% 137.80 GiB --> could you explain what does it mean and how is it being calculated.
root 31.2% 0.2% 2.17 GiB
68 1.9% 0.0% 0.04 GiB
dk016140 0.2% 0.0% 0.01 GiB
Thanks
Veera.
Hi Ryan,
The "--scrub-ip" & "--scrub-mac" options are not working, only "--scrub" works which scrubs ip, mac,hostname etc,. but option of scrubbing only ip or mac doesn't work. Please check on this:
[root@localhost ~]# xsos --ip --scrub-ip
xsos: unrecognized option '--scrub-ip'
Usage: xsos [DISPLAY OPTIONS] [-6abokcfmdtlerngisp] [SOSREPORT ROOT]
or: xsos [DISPLAY OPTIONS] {--B|--C|--F|--M|--D|--T|--L|--R|--N|--G|--I|--P FILE}...
or: xsos [-?|-h|--help]
Display system info from localhost or extracted sosreport
[root@localhost ~]# xsos --ip --scrub-mac
xsos: unrecognized option '--scrub-mac'
Usage: xsos [DISPLAY OPTIONS] [-6abokcfmdtlerngisp] [SOSREPORT ROOT]
or: xsos [DISPLAY OPTIONS] {--B|--C|--F|--M|--D|--T|--L|--R|--N|--G|--I|--P FILE}...
or: xsos [-?|-h|--help]
Display system info from localhost or extracted sosreport
I'm using the latest one even I tried the previous version as well:
[root@localhost ~]# rpm -q xsos
xsos-0.7.11-1.noarch
Other parameters and options are working as expected.
From the changelog I could see this :-
* Sat Jul 11 2015 Ryan Sawhill Aroha <rsaw@redhat.com> 0.7.0-1
- Remove --scrub-ip and --scrub-mac
- Add XSOS_SCRUB_SERIAL env variable (to the already-existing XSOS_SCRUB_IP_HN
and XSOS_SCRUB_MACADDR vars)
- Add --scrub option that does all 3
Does this mean "--scrub-mac" & "--scrub-ip" doesn't work, however, "--scrub" does it all? But mac/ip scrubbing independently would be a nice option.
Also, nice if this gets added to epel repo...
Hi Sadashiva,
Well, something even better : I found out that xsos
is completely 'portable', simply download the source code from https://github.com/ryran/xsos/releases, place it where ever you want, make the file executable and launch the tool with ./xsos
. I've already added this useful application to my toolbox and put it on all machines and systems today.
Waves :)
Christian
Hi, I have odd scenario where a few servers are reporting clean(hardly 50% memory usage) via "free -m" but when I run "xsos -ay" it tells me its 99% used. Our monitoring system has been reporting high memory usage on server. Why does free and xsos report differently?
Hi Ryan Verry nice! Thanks! I am missing disk partitioning and usage infos, but never mind I will file a pull request today (accidently got few frew hours :-)) Please look into the code of my pull request (it took me few minutes to find out how you organized the script) and drop me a line if something is wrong or missing because I would like to write few other "enrichments"
[root@ansible xsos]# bash xsos -d /root/tmp/sosreport-clue1-2019-05-13-nqxiqua
STORAGE
Whole Disks from /proc/partitions:
5 disks, totaling 10000 GiB (9.77 TiB)
(3 multipath/mdraid components hidden)
- - - - - - - - - - - - - - - - - - - - -
Disk Size in GiB
---- -----------
md127 3726
sda 447
sdb 238
sde 3726
sdg 1863
lsblk disk layout information:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 447.1G 0 disk
`-ssd-ssd 253:1 0 400G 0 lvm /srv/vm/images-ssd
sdb 8:16 0 238.5G 0 disk
|-sdb1 8:17 0 1G 0 part /boot
|-sdb2 8:18 0 15.7G 0 part [SWAP]
`-sdb3 8:19 0 221.8G 0 part /
sdc 8:32 0 1.8T 0 disk
`-sdc1 8:33 0 1.8T 0 part
`-md127 9:127 0 3.7T 0 raid5
`-raid-vm 253:0 0 2.9T 0 lvm /srv
sdd 8:48 0 1.8T 0 disk
`-sdd1 8:49 0 1.8T 0 part
`-md127 9:127 0 3.7T 0 raid5
`-raid-vm 253:0 0 2.9T 0 lvm /srv
sde 8:64 0 3.7T 0 disk
sdf 8:80 0 1.8T 0 disk
`-sdf1 8:81 0 1.8T 0 part
`-md127 9:127 0 3.7T 0 raid5
`-raid-vm 253:0 0 2.9T 0 lvm /srv
sdg 8:96 0 1.8T 0 disk /srv/backup
Probably cfg2html project may give you more ideas to enrich your/our project Have a great time and thanks again! Chris
Excellent tool! I have just missed "df -h" for File System info/usage. But it's really awesome!
Well done!
Can I use this tool to analyze abrt crashes? If the answer is "yes" what options do I have to use?
Thanks
No, xsos does not look at data caught by abrt.
Presuming you mean an application core file, you would first install the debuginfo package for whatever crashed. In yum-utils
there is debuginfo-install
which makes this easier. Then open the core file in gdb
debugger.
Between sosreport: 3.2 and version 3.9 the uptime
file has moved from sos_commands/general/uptime to sos_commands/host/uptime and xsos just dies.
This tool is quite useful and seems to be all that exists to help in this space, but github has been very quiet. I hope it will be updated.
The actual error is:
gawk: fatal: cannot open file
./sosreport/SOSNAME//uptime' for reading (No such file or directory)`
BIOS also has a problem and can be fixed with:
3422c3422
< SOS_CHECKFILE bios {,sos_commands/kernel.}dmidecode && DMIDECODE "$sosroot"
---
> SOS_CHECKFILE bios {,sos_commands/hardware/,sos_commands/kernel.}dmidecode && DMIDECODE "$sosroot"
uptime doesn't use the same routine to locate the file, so I had to change the actual command and it probably breaks on older sosreports.
Hi BIll, I got a chance to look at these. In both cases it is expected that sos puts a symlink in the root directory for the sosreport, and that's what xsos looks for first:
$ grep sos installed-rpms
sos-3.9.1-6.el8.noarch
$ readlink dmidecode
sos_commands/hardware/dmidecode
$ readlink uptime
sos_commands/host/uptime
It sounds like you might have had the symlink fail in the sosreport for some reason, perhaps an incomplete sos collection? Please let me know if there's a way to reproduce this.
Jamie - thanks for looking... doh! Apparently the recreation scenario is 1) untar sosreports on your mac 2) scp/move them to RHEL and loose all your symlinks (not quite sure what I did) 3) wrongly assume sosreports changed.