Censoring sosreport output

Latest response

I need to send Red Hat a sosreport but it has sensitive information that is not allowed offsite (IP addresses, hostnames etc).

I could knock up a script that unpacks the sosreport files and sed's all the relevant data out of the files but I was wondering if there was an "official" way of censoring the data? Red Hat knowledge base is not showing anything obvious that could help.

Many thanks

Aidan

Responses

Hi Aidan,

Thanks for posting the discussion. I'm the lead developer for sos and maintain the Red Hat and Fedora packages.

Right now there isn't anything included in the distribution to automate this task. There's a third-party tool maintained by Jamie Duncan (another Red Hatter) that's hosted on github.

You can download and use this to post-process sosreport tarballs today and both Jamie and I are working towards having this more tightly-integrated in future releases.

If you'd like to provide suggestions or follow the progress please feel free to join us on GitHub or to file a support case if you'd like to receive more information on future product updates in this area.

The main sos project pages are here: https://github.com/sosreport/sos

Regards,
Bryn.

Bryn

Thank you for the pointer to SOSCleaner, it looks exactly what we require. I'll have a play with it and if there are any other features we'd find useful I'll raise them through GitHub or a support case.

Many thanks

Aidan

Bryn,

Just a quick follow on question - soscleaner needs python 2.7. RHEL 6 only ships with python 2.6. Various things I have read online indicate that upgrading python on a RHEL release, beyond that which is provided natively, is considered a bad thing as large parts of the os (e.g. Yum) rely on a set version.

What's the general consensus within Red Hat on upgrading python to 2.7 on RHEL 6?

Thanks

Aidan

Aidan,

The standard response from Red Hat support: Can you reproduce the error on a supported release of the failing product.
Well if you break yum this is very hard.

If you have the resources, build a small management virtual machine running RHEL 7. RHEL7 comes with python 2.7.
Upload the sosreport to this "management VM" as run the soscleaner before uploading it to acces.redhat.com

Kind regards,

Jan Gerrit Kootstra

Aidan,

Good point - I tend to use it on my development systems that run Fedora (which is currently shipping python 2.7 and 3.3.2 runtimes). I wouldn't recommend attempting to update the system version of Python on a RHEL host as it's likely to lead to some unpredictable effects on system tools. One option would be to install a later python2 version into your home directory - the python source distribution supports this using --prefix. There's an example in a response to a question on stackoverflow].

That said 2.6 vs. 2.7 differences are often quite straightforward to work around. I'll get a clone on a RHEL6 box today and see what problems there are and how easy it'd be to get this to work on the earlier runtime.

Regards,
Bryn.

Thanks Bryn,

I'll try the home directory install. Unfortunately RHEL 7 is not possible at the moment - it's taken long enough to (almost) get RHEL 6.5 through security accreditation!

Aidan

I've just pulled the current soscleaner master onto a RHEL6 box and ran a few tests. So far it all seems to work as expected:

# soscleaner sosreport-rhn-support-bmr-20140626132918-6719.tar.xz
06-26 13:30:03 SOSCleaner CONSOLE: Log File Created at /tmp/soscleaner-20140626123003.log
06-26 13:30:03 SOSCleaner CONSOLE: Beginning SOSReport Extraction
06-26 13:30:04 SOSCleaner CONSOLE: soscleaner - 0.1
06-26 13:30:04 SOSCleaner CONSOLE: soscleaner is a tool to help obfuscate sensitive information from an existing sosreport.
06-26 13:30:04 SOSCleaner CONSOLE: Please review the content before passing it along to any third party.
06-26 13:30:06 SOSCleaner CONSOLE: The Hostname Does Not Appear to be an FQDN - Limited Cleaning Available
06-26 13:30:06 SOSCleaner CONSOLE: SOSCleaner Started
06-26 13:30:06 SOSCleaner CONSOLE: Working Directory - /tmp/soscleaner-20140626123003
06-26 13:30:06 SOSCleaner CONSOLE: IP Substitution Start Address - 10.230.230.1
06-26 13:30:06 SOSCleaner CONSOLE: Domain Name Substitution - example.com
06-26 13:30:18 SOSCleaner CONSOLE: SOSCleaner Completed
06-26 13:30:18 SOSCleaner CONSOLE: IP Addresses Obfuscated - 314
06-26 13:30:18 SOSCleaner CONSOLE: Hostnames Obfuscated - 1
06-26 13:30:18 SOSCleaner CONSOLE: Files Cleaned - 2322
06-26 13:30:18 SOSCleaner CONSOLE: Creating IP Report - /tmp/soscleaner-20140626123003-ip.csv
06-26 13:30:18 SOSCleaner CONSOLE: Creating Hostname Report - /tmp/soscleaner-20140626123003-hostname.csv
06-26 13:30:18 SOSCleaner CONSOLE: Starting Archiving Process - Creating /tmp/soscleaner-20140626123003.tar.gz
06-26 13:30:28 SOSCleaner CONSOLE: Archiving Complete
06-26 13:30:28 SOSCleaner CONSOLE: SOSCleaner Complete

You'll need to install the python-magic package (likewise on RHEL7/Fedora).

Here's the steps I used:

# git clone https://github.com/jduncan-rva/soscleaner.git
Initialized empty Git repository in /root/src/git/soscleaner/.git/
remote: Reusing existing pack: 391, done.
remote: Counting objects: 25, done.
remote: Compressing objects: 100% (21/21), done.
remote: Total 416 (delta 9), reused 0 (delta 0)
Receiving objects: 100% (416/416), 383.89 KiB | 235 KiB/s, done.
Resolving deltas: 100% (218/218), done.

# yum -y install python-magic
Loaded plugins: product-id, rhnplugin, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
This system is receiving updates from RHN Classic or RHN Satellite.
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package python-magic.x86_64 0:5.04-15.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================================================================
 Package                                   Arch                                Version                                  Repository                                         Size
================================================================================================================================================================================
Installing:
 python-magic                              x86_64                              5.04-15.el6                              rhel-x86_64-server-6                               26 k

Transaction Summary
================================================================================================================================================================================
Install       1 Package(s)

Total download size: 26 k
Installed size: 0  
Downloading Packages:
python-magic-5.04-15.el6.x86_64.rpm                                                                                                                      |  26 kB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : python-magic-5.04-15.el6.x86_64                                                                                                                              1/1 
  Verifying  : python-magic-5.04-15.el6.x86_64                                                                                                                              1/1 

Installed:
  python-magic.x86_64 0:5.04-15.el6                                                                                                                                             

Complete!

# cd soscleaner
# python setup.py install
/usr/lib64/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'platform'
  warnings.warn(msg)
running install
running build
running build_py
running build_scripts
running install_lib
copying build/lib/SOSCleaner.py -> /usr/lib/python2.6/site-packages
byte-compiling /usr/lib/python2.6/site-packages/SOSCleaner.py to SOSCleaner.pyc
running install_scripts
copying build/scripts-2.6/soscleaner -> /usr/bin
changing mode of /usr/bin/soscleaner to 755
running install_data
creating /usr/share/doc/soscleaner-0.1
copying doc/LICENSE -> /usr/share/doc/soscleaner-0.1
copying doc/soscleaner.8.gz -> /usr/share/man/man8
running install_egg_info
Writing /usr/lib/python2.6/site-packages/soscleaner-0.1-py2.6.egg-info

I'm working on a scratch VM here so I don't mind installing stuff into /usr/lib - if you prefer an RPM package there's a spec file included in the soscleaner source tree that you can use with Python distutils to build an RPM:

# python setup.py bdist_rpm
/usr/lib64/python2.6/distutils/dist.py:266: UserWarning: Unknown distribution option: 'platform'
  warnings.warn(msg)
running bdist_rpm
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/rpm
creating build/bdist.linux-x86_64/rpm/SOURCES
creating build/bdist.linux-x86_64/rpm/SPECS
creating build/bdist.linux-x86_64/rpm/BUILD
creating build/bdist.linux-x86_64/rpm/RPMS
creating build/bdist.linux-x86_64/rpm/SRPMS
writing 'build/bdist.linux-x86_64/rpm/SPECS/soscleaner.spec'
[...]
Executing(--clean): /bin/sh -e /var/tmp/rpm-tmp.zFnC8c
+ umask 022
+ cd /root/src/git/soscleaner/build/bdist.linux-x86_64/rpm/BUILD
+ rm -rf soscleaner-0.1
+ exit 0
moving build/bdist.linux-x86_64/rpm/SRPMS/soscleaner-0.1-1.src.rpm -> dist
moving build/bdist.linux-x86_64/rpm/RPMS/noarch/soscleaner-0.1-1.noarch.rpm -> dist

Regards,
Bryn.

Bryn,

That worked a treat - thank you.

I'll be getting soscleaner integrated into our RHEL platform!

Aidan

Hi Aidan.

Thanks for taking a look at SOSCleaner, and I'm glad it looks like it might help you out.

I've built out RHEL6/RHEL7 and Fedora 20 RPM's and uploaded them to my Red Hat page:

http://people.redhat.com/jduncan/soscleaner/soscleaner-0.1-14.el6.noarch.rpm
http://people.redhat.com/jduncan/soscleaner/soscleaner-0.1-14.el7.noarch.rpm
http://people.redhat.com/jduncan/soscleaner/soscleaner-0.1-14.fc20.src.rpm

These include fixes that went in just this morning that Bryn pointed out to me. Fresh off the presses!

If you'd like to test it out and kick its tires / find its bugs we'd love the help. There is currently a request to have it added to Fedora so we can get it into EPEL and even into RHEL. If it's helpful to you please comment that you'd like to see it in https://bugzilla.redhat.com/show_bug.cgi?id=1104746.

I will now conclude my shameless plug. :)

Thanks again.

Jamie Duncan
Senior Technical Account Manager

Jamie,

Thanks for the rpms - that's one less job for me to do today!

Where is the best place to suggest enhancements/report bugs etc? GitHub or on Bugzilla?

Aidan

Bugzilla is always a good bet!

EDIT: Actually... Use GitHub for this one.

I am getting this error while running the soscleaner. Any idea why it's displaying this error?

/usr/bin/soscleaner /home/guragain/sosreport-sguragain-20180425101235.tar.xz Traceback (most recent call last): File "/usr/bin/soscleaner", line 5, in from soscleaner import SOSCleaner File "build/bdist.linux-x86_64/egg/soscleaner.py", line 35, in from ipaddr import IPv4Network, IPv4Address, IPv6Network, IPv6Address ImportError: No module named ipaddr

You'll need the python-ipaddr package installed in order to use soscleaner - the error is because the script attempts to import a python module named 'ipaddr' which is not installed on your system.

Alternately, you can use the RPM builds - these should include dependency information and yum (or DNF) will automatically install any required packages for you. If you are using the RPM builds, and you still get this error without installing python-ipaddr, it's a bug in the RPM package and you should report it to the maintainer.

Bryn, Thank you. I was able to install that on one of my dev machines and ran from there. It was successful.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.