Back in the Hood!!

After a crazy and exciting week of innovations in San Francisco, here I am again to tell you a bit more on how to customize Red Hat Insights to your needs!

To blacklist or not to blacklist, that is the question

As explained in my previous post Red Hat Insights 102, you can control the data Red Hat Insights sends to Red Hat servers, how data is sent, and when it is sent. But deviating from the default has its drawbacks too.

We want to provide our customers with the necessary options and controls to the data that Insights collects. However, with each modification to the default payload sent to Red Hat, you may adversely change the level of analysis Insights provides about your environment. Since Insights collects what is minimally needed for analysis. Each removal of data from the collection payload can impact our rules capabilities to detect the issue within your infrastructure.

NOTE:

Before we continue, I want to add a few words about the differences in commands required for RHEL 6 vs. RHEL 7 systems. This will affect many of the commands shared below. The name of the Red Hat Insights client has changed for RHEL 7.5+ but has not yet done so for RHEL 6.

For RHEL 7.5+ users, the name of the client (and all commands and locations that use the name of the client) is insights-client. For older RHEL users, the name and other changes will come to RHEL 6 with the release of RHEL 6.10. For now though, RHEL 6 users should continue to use the legacy client name: redhat-access-insights.

Blacklisting information, or how to deny access to certain data

Red Hat Insights collects metadata about the runtime configuration of a system (approximately 1% of what would be collected via sosreport during a support case) but this data can be customized, anonymized, blacklisted and obfuscated in the way that you need.

You can decide to exclude entire files, specific commands, specific patterns, and specific keywords from the data that is sent to Red Hat. To enable these exclusions, you simply need to create a file called /etc/redhat-access-insights/remove.conf (or /etc/insights-client/remove.conf if you are already using RHEL 7.5) and specify this file in the remove_file line of /etc/redhat-access-insights/redhat-access-insights.conf (or /etc/insights-client/insights-client.conf if you are already using RHEL 7.5), as in the following example:

[remove]
files=/etc/cluster/cluster.conf,/etc/hosts
commands=/bin/dmesg
patterns=password,username
keywords=super$ecret,ultra$ecret

Let's dig into the directives from the example above.

• files: A comma-separated list of files to be excluded. Each element in the list of files must be the absolute path to the file. To ensure exclusion, file names listed here must match exactly what is shown in the collection rules.
• commands: A comma-separated list of commands that should not be executed and whose output should not be sent. To ensure exclusion, command names listed here must match exactly what is shown in the collection rules.
• patterns: A comma-separated list of patterns that should not be sent.
• keywords: A comma-separated list of keywords that should not be sent. Matching keywords will be replaced with the literal keyword. For this option to take effect, the obfuscate option must be set to True in the /etc/redhat-access-insights/redhat-access-insights.conf file.

Important!

Patterns affect entire lines so any line that includes a matching pattern will not be sent.

As explained in my previous blog post, you can validate data to be sent with the redhat-access-insights --no-upload command.

Let's get our hands dirty!

Let's take a look at what's collected and sent to Red Hat by the Red Hat Insights client using the default configuration.

To see what’s collected, without actually taking the action of collecting the data, we simply execute the agent with the --no-upload option as follows:

[root@gherkin ~]# redhat-access-insights --no-upload
Starting to collect Insights data
See Insights data in /var/tmp/oLUbKq/insights-gherkin-20180521110933.tar.gz

To inspect what's being collected, we simply have to unzip the generated tar.gz file and dig into it, through commands and files:

[root@gherkin ~]# tar xvfz /var/tmp/oLUbKq/insights-gherkin-20180521110933.tar.gz
[root@gherkin ~]# cd ./insights-gherkin-20180521110933/

We can see the hostname being collected:

[root@gherkin insights-gherkin-20180521110933]# cat insights_commands/hostname
gherkin

NIC information is also collected:

[root@gherkin insights-gherkin-20180521110933]# cat insights_commands/ethtool_eno1
Settings for eno1:
    Supported ports: [ TP ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: off (auto)
    Supports Wake-on: pumbg
    Wake-on: g
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes    

Kernel version information, architecture, and OS version is collected too:

[root@gherkin insights-gherkin-20180521110933]# cat insights_commands/uname_-a
Linux gherkin 3.10.0-693.11.1.el7.x86_64 #1 SMP Fri Oct 27 05:39:05 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@gherkin insights-gherkin-20180521110933]# cat insights_commands/uptime
 11:09:39 up 53 days, 10:27,  3 users,  load average: 0.40, 1.24, 0.90

Filesystems currently mounted on our machine:

[root@gherkin insights-gherkin-20180521110933]# cat insights_commands/mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=16323324k,nr_inodes=4080831,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
[...]

Insights also collects the hostnames and IPs our system knows about (these are my home LAN's machines):

[root@gherkin insights-gherkin-20180521110933]# cat etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
# Máquinas virtuales residentes en gherkin
192.168.100.250 ansible-tower    ansible-tower.tortilla.org
192.168.100.189    ansible-node1    ansible-node1.tortilla.org
192.168.100.199    icg1        icg1.example.com
192.168.100.202    icg2        icg2.example.com
192.168.100.147    icg3        icg3.example.com
192.168.100.194    icg4        icg4.example.com

But what if I don't want to send all that data to Red Hat? Well, let's configure our host so it won't send the previous data.

First of all, we need to configure /etc/redhat-access-insights/redhat-access-insights.conf to tell the Insights agent where the remove.conf file to be used is located:

[root@gherkin ~]# vi /etc/redhat-access-insights/redhat-access-insights.conf
[redhat-access-insights]
remove_file=/etc/redhat-access-insights/remove.conf

Since we don't want to send our own machine name or IP to Red Hat, in the very same file, we need to specify to obfuscate them as follows:

# Obfuscate IP addresses
obfuscate=True

# Obfuscate hostname
obfuscate_hostname=True

Another thing Red Hat Insights collects is the sshd configuration, for example, the access type:

[root@gherkin insights-gherkin-20180521110933]# cd etc/ssh/
[root@gherkin ssh]# grep -E '[P|p]assword' *
#PermitEmptyPasswords no
# the setting of "PermitRootLogin without-password".

Now it's the time to tune the remove.conf file to our needs:

[root@gherkin ~]# vi /etc/redhat-access-insights/remove.conf
[remove]
files=/etc/hosts,/proc/cpuinfo
commands=/bin/dmesg,/bin/mount,/bin/mount,/bin/uname -a,/sbin/ethtool,/usr/bin/uptime
patterns=password,username

In this way, all the previous information won't be collected and sent. To verify it, we just need to re run the agent with --no-upload option, and see its results:

[root@gherkin ~]# redhat-access-insights --no-upload
WARNING: Excluding data from files
Starting to collect Insights data
WARNING: Skipping command /bin/dmesg
WARNING: Skipping command /sbin/ethtool
WARNING: Skipping command /bin/mount
WARNING: Skipping command /bin/uname -a
WARNING: Skipping command /usr/bin/uptime
WARNING: Skipping file /proc/cpuinfo
WARNING: Skipping file /etc/hosts
See Insights data in /var/tmp/eJ9LoU/soscleaner-3125813513575232.tar.gz

By executing the agent, you can see the list of commands that will be skipped, as specified in the remove.conf file).

In addition, we'll inspect each of the previous commands and files from the tar.gz file:

[root@gherkin soscleaner-3125813513575232]# cat insights_commands/hostname
host0

[root@gherkin soscleaner-3125813513575232]# cat proc/cpuinfo
cat: proc/cpuinfo: No such file or directory

[root@gherkin soscleaner-3125813513575232]# cat etc/hosts
cat: etc/hosts: No such file or directory

[root@gherkin soscleaner-3125813513575232]# cat insights_commands/ethtool_eno1
cat: insights_commands/ethtool_eno1: No such file or directory

[root@gherkin soscleaner-3125813513575232]# cat insights_commands/mount
cat: insights_commands/mount: No such file or directory

[root@gherkin soscleaner-3125813513575232]# cat insights_commands/uptime
cat: insights_commands/uptime: No such file or directory

And finally, let's verify how patterns also work with the example of the sshd configuration:

[root@gherkin soscleaner-3125813513575232]# cd etc/ssh/
[root@gherkin ssh]# ls
sshd_config
[root@gherkin ~]# grep -E '[P|p]assword' sshd_config

Wrapping up

Red Hat Insights has been designed to ensure that minimal, specific data is collected.

We encourage customers to check what is collected with the --no-upload option prior to making modifications.

Find out more

If you want to know about the patterns and files and commands Insights collects data from as well as the the whole list of files and commands that are used to collect data from, you can take a look at this document.