Insights guidelines for deployment at scale

Insights usage varies from customer to customer so there is no real "one size fits all" template. However it is worth highlighting some of the features Red Hat has in place to assist with large sized deployments. This is not intended to be a best-practices guide, just some things to consider.

Deployment

I typically emphasize how easy it is to deploy insights - with its minimal steps, because it is a SaaS solution. See the getting started guide. If a customer happens to be also managing these systems via a version of Satellite with Insights integration, mass registration of Insights is built in (via the bootstrap script provided with Satellite).

Proxy

A proxy is not required to be used for Insights, but typically the systems of customers with larger scale deployments are already set up to communicate with Red Hat via a proxy, rather than being individually connected (or in disconnected environments). Insights also supports this infrastructure model via proxy support as an option within our client, so there are no changes that you as the customer need to make to how your systems communicate with our servers.
If you are using a version of Satellite with Insights integration, it can be used as the proxy.

Frequency

By default, Insights sends information once daily. We find that this default frequency fits the needs of most accounts.
However, one may have concerns about the impact, at scale, of all these clients checking in and transmitting a payload on your network. To help with this, we have implemented a default feature to make the Insights client automatically stagger its check-in time; so rather than all checking in at once, they'll check in individually or in random groups. All out of the box!

And if you want to take it a step further and have full control of your own schedule, the Insights client can be customized via cron (when using RHEL 6) or via systemd (when using RHEL 7.5+).

Remediation

Setting up the systems is one thing to consider at scale, but once Insights is setup and detecting all those problems now on 1000+ systems, how do you take action at scale?

A few recommendations in this regard would be:
* Use Ansible remediation with Insights-generated playbooks to coordinate a fix for one or multiple issues impacting your infrastructure.

• Use Remediations to help coordinate planned remediations for actions and systems.

• Address by severity - Insights applies one of 4 severity (risk) levels to all actions to help you focus on the most important ones.

• Attack the low hanging fruit - Insights also provides a "Risk of Change" rating for actions depending on the impact of the change on the environment. Remediating actions with the lowest Risk of Change generally won't require any downtime or coordination.

Back in the Hood!!

After a crazy and exciting week of innovations in San Francisco, here I am again to tell you a bit more on how to customize Red Hat Insights to your needs!

To blacklist or not to blacklist, that is the question

As explained in my previous post Red Hat Insights 102, you can control the data Red Hat Insights sends to Red Hat servers, how data is sent, and when it is sent. But deviating from the default has its drawbacks too.

We want to provide our customers with the necessary options and controls to the data that Insights collects. However, with each modification to the default payload sent to Red Hat, you may adversely change the level of analysis Insights provides about your environment. Since Insights collects what is minimally needed for analysis. Each removal of data from the collection payload can impact our rules capabilities to detect the issue within your infrastructure.

NOTE:

Before we continue, I want to add a few words about the differences in commands required for RHEL 6 vs. RHEL 7 systems. This will affect many of the commands shared below. The name of the Red Hat Insights client has changed for RHEL 7.5+ but has not yet done so for RHEL 6.

For RHEL 7.5+ users, the name of the client (and all commands and locations that use the name of the client) is insights-client. For older RHEL users, the name and other changes will come to RHEL 6 with the release of RHEL 6.10. For now though, RHEL 6 users should continue to use the legacy client name: redhat-access-insights.

Blacklisting information, or how to deny access to certain data

Red Hat Insights collects metadata about the runtime configuration of a system (approximately 1% of what would be collected via sosreport during a support case) but this data can be customized, anonymized, blacklisted and obfuscated in the way that you need.

You can decide to exclude entire files, specific commands, specific patterns, and specific keywords from the data that is sent to Red Hat. To enable these exclusions, you simply need to create a file called /etc/redhat-access-insights/remove.conf (or /etc/insights-client/remove.conf if you are already using RHEL 7.5) and specify this file in the remove_file line of /etc/redhat-access-insights/redhat-access-insights.conf (or /etc/insights-client/insights-client.conf if you are already using RHEL 7.5), as in the following example:

[remove]
files=/etc/cluster/cluster.conf,/etc/hosts
commands=/bin/dmesg
patterns=password,username
keywords=super$ecret,ultra$ecret

Let's dig into the directives from the example above.

• files: A comma-separated list of files to be excluded. Each element in the list of files must be the absolute path to the file. To ensure exclusion, file names listed here must match exactly what is shown in the collection rules.
• commands: A comma-separated list of commands that should not be executed and whose output should not be sent. To ensure exclusion, command names listed here must match exactly what is shown in the collection rules.
• patterns: A comma-separated list of patterns that should not be sent.
• keywords: A comma-separated list of keywords that should not be sent. Matching keywords will be replaced with the literal keyword. For this option to take effect, the obfuscate option must be set to True in the /etc/redhat-access-insights/redhat-access-insights.conf file.

Important!

Patterns affect entire lines so any line that includes a matching pattern will not be sent.

As explained in my previous blog post, you can validate data to be sent with the redhat-access-insights --no-upload command.

Let's get our hands dirty!

Let's take a look at what's collected and sent to Red Hat by the Red Hat Insights client using the default configuration.

To see what’s collected, without actually taking the action of collecting the data, we simply execute the agent with the --no-upload option as follows:

[root@gherkin ~]# redhat-access-insights --no-upload
Starting to collect Insights data
See Insights data in /var/tmp/oLUbKq/insights-gherkin-20180521110933.tar.gz

To inspect what's being collected, we simply have to unzip the generated tar.gz file and dig into it, through commands and files:

[root@gherkin ~]# tar xvfz /var/tmp/oLUbKq/insights-gherkin-20180521110933.tar.gz
[root@gherkin ~]# cd ./insights-gherkin-20180521110933/

We can see the hostname being collected:

[root@gherkin insights-gherkin-20180521110933]# cat insights_commands/hostname
gherkin

NIC information is also collected:

[root@gherkin insights-gherkin-20180521110933]# cat insights_commands/ethtool_eno1
Settings for eno1:
    Supported ports: [ TP ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Supported pause frame use: No
    Supports auto-negotiation: Yes
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Speed: 1000Mb/s
    Duplex: Full
    Port: Twisted Pair
    PHYAD: 1
    Transceiver: internal
    Auto-negotiation: on
    MDI-X: off (auto)
    Supports Wake-on: pumbg
    Wake-on: g
    Current message level: 0x00000007 (7)
                   drv probe link
    Link detected: yes    

Kernel version information, architecture, and OS version is collected too:

[root@gherkin insights-gherkin-20180521110933]# cat insights_commands/uname_-a
Linux gherkin 3.10.0-693.11.1.el7.x86_64 #1 SMP Fri Oct 27 05:39:05 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@gherkin insights-gherkin-20180521110933]# cat insights_commands/uptime
 11:09:39 up 53 days, 10:27,  3 users,  load average: 0.40, 1.24, 0.90

Filesystems currently mounted on our machine:

[root@gherkin insights-gherkin-20180521110933]# cat insights_commands/mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=16323324k,nr_inodes=4080831,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
[...]

Insights also collects the hostnames and IPs our system knows about (these are my home LAN's machines):

[root@gherkin insights-gherkin-20180521110933]# cat etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
# Máquinas virtuales residentes en gherkin
192.168.100.250 ansible-tower    ansible-tower.tortilla.org
192.168.100.189    ansible-node1    ansible-node1.tortilla.org
192.168.100.199    icg1        icg1.example.com
192.168.100.202    icg2        icg2.example.com
192.168.100.147    icg3        icg3.example.com
192.168.100.194    icg4        icg4.example.com

But what if I don't want to send all that data to Red Hat? Well, let's configure our host so it won't send the previous data.

First of all, we need to configure /etc/redhat-access-insights/redhat-access-insights.conf to tell the Insights agent where the remove.conf file to be used is located:

[root@gherkin ~]# vi /etc/redhat-access-insights/redhat-access-insights.conf
[redhat-access-insights]
remove_file=/etc/redhat-access-insights/remove.conf

Since we don't want to send our own machine name or IP to Red Hat, in the very same file, we need to specify to obfuscate them as follows:

# Obfuscate IP addresses
obfuscate=True

# Obfuscate hostname
obfuscate_hostname=True

Another thing Red Hat Insights collects is the sshd configuration, for example, the access type:

[root@gherkin insights-gherkin-20180521110933]# cd etc/ssh/
[root@gherkin ssh]# grep -E '[P|p]assword' *
#PermitEmptyPasswords no
# the setting of "PermitRootLogin without-password".

Now it's the time to tune the remove.conf file to our needs:

[root@gherkin ~]# vi /etc/redhat-access-insights/remove.conf
[remove]
files=/etc/hosts,/proc/cpuinfo
commands=/bin/dmesg,/bin/mount,/bin/mount,/bin/uname -a,/sbin/ethtool,/usr/bin/uptime
patterns=password,username

In this way, all the previous information won't be collected and sent. To verify it, we just need to re run the agent with --no-upload option, and see its results:

[root@gherkin ~]# redhat-access-insights --no-upload
WARNING: Excluding data from files
Starting to collect Insights data
WARNING: Skipping command /bin/dmesg
WARNING: Skipping command /sbin/ethtool
WARNING: Skipping command /bin/mount
WARNING: Skipping command /bin/uname -a
WARNING: Skipping command /usr/bin/uptime
WARNING: Skipping file /proc/cpuinfo
WARNING: Skipping file /etc/hosts
See Insights data in /var/tmp/eJ9LoU/soscleaner-3125813513575232.tar.gz

By executing the agent, you can see the list of commands that will be skipped, as specified in the remove.conf file).

In addition, we'll inspect each of the previous commands and files from the tar.gz file:

[root@gherkin soscleaner-3125813513575232]# cat insights_commands/hostname
host0

[root@gherkin soscleaner-3125813513575232]# cat proc/cpuinfo
cat: proc/cpuinfo: No such file or directory

[root@gherkin soscleaner-3125813513575232]# cat etc/hosts
cat: etc/hosts: No such file or directory

[root@gherkin soscleaner-3125813513575232]# cat insights_commands/ethtool_eno1
cat: insights_commands/ethtool_eno1: No such file or directory

[root@gherkin soscleaner-3125813513575232]# cat insights_commands/mount
cat: insights_commands/mount: No such file or directory

[root@gherkin soscleaner-3125813513575232]# cat insights_commands/uptime
cat: insights_commands/uptime: No such file or directory

And finally, let's verify how patterns also work with the example of the sshd configuration:

[root@gherkin soscleaner-3125813513575232]# cd etc/ssh/
[root@gherkin ssh]# ls
sshd_config
[root@gherkin ~]# grep -E '[P|p]assword' sshd_config

Wrapping up

Red Hat Insights has been designed to ensure that minimal, specific data is collected.

We encourage customers to check what is collected with the --no-upload option prior to making modifications.

Find out more

If you want to know about the patterns and files and commands Insights collects data from as well as the the whole list of files and commands that are used to collect data from, you can take a look at this document.

Before we begin...

Before we begin with how to configure Red Hat Insights to be tailored to your needs (in terms of controlling what is sent to Red Hat servers and how it is sent) let me please remind you of the very basics of Red Hat Insights…

Can I control what Red Hat Insights is doing behind the curtains?

Absolutely!

Red Hat Insights collects metadata about the runtime configuration of a system. The data collected is 1% of what would be collected via sosreport during a support case. The data collected is a subset of an sosreport, so if a sosreport has been approved for usage, Insights data collection should also be acceptable.

The Red Hat Insights tool allows customers to review the data being collected by use of a --no-upload parameter. This runs the Insights client & collection, but does not transmit it to Red Hat for analysis. This collection is stored locally in a temporary directory where it can be inspected.

# ls -lh /var/tmp/TAFHhW/insights-amaya-insights2-20180129165816.tar.gz
-rw-r--r--. 1 root root 138K Jan 29 16:58 /var/tmp/TAFHhW/insights-amaya-insights2-20180129165816.tar.gz
# ls -lh /var/tmp/sosreport-amaya-insights2-20180129165924.tar.xz
-rw-------. 1 root root 12M Jan 29 16:59 /var/tmp/sosreport-amaya-insights2-20180129165924.tar.xz

That data is sent to Red Hat’s servers over SSL and compared to our Support Knowledge Database, looking for matches, and results sent back to customer, in the form of actions, where they are displayed.

Insights on Red Hat Insights

Red Hat Insights requires Python-2.6.6-64 or later, being its main configuration file: /etc/redhat-access-insights/redhat-access-insights.conf

Red Hat Insights registration will auto-detect how the system is registered for software updates and can auto-configure the client based on that information. For auto-configuration, CERT is the default authmethod. Otherwise, authmethod can be set to BASIC, requiring a username and password for the target Insights server (Customer Portal or Satellite).

Red Hat Insights uses Satellite server as a Proxy to send diagnostic data to the Customer Portal so requires a connected environment.

Log files

The log file can be found at /var/log/redhat-access-insights/redhat-access-insights.log. The logs rotate each time data is successfully collected, to .log.1,.2,.n, so be aware that if an upload has occurred since the case was opened, relevant logs might now be in a different file.
The log file records the process of collecting data and uploading that data to the Insights server.

I still want to control more of it

Well, you can!

Insights can be configured by the customer to further restrict what's collected / sent, and optionally to obfuscate hostname and / or IP addresses from reports if desired. Customers can always look at the source code directly from the rpm - everything is made available for their perusal.

All data is trimmed down to the minimal necessary facts before being uploaded and encrypted both in transit and at rest. The customer may also choose to alter the name chosen to represent the system in the UI (eg,apache01.prod instead of a fully qualified domain name).

Customers can opt-out of sending any data they wish to the service via a configuration blacklist. The service will continue to function, and only health checks which depend on that specific piece of data will be impacted.

The Insights client will enable customers to ignore any specific file, keyword or pattern, making data redaction easy to use.

The data collected is sent over a secure TLS / https connection. It's encrypted at rest on Red Hat's systems using LUKS encryption, and is kept only until the next report is received, which by default is 24 hours. If another report doesn't arrive in the scan interval period, the data on file (encrypted) is kept for a maximum of 2 weeks and then deleted from our systems.

The Red Hat Insights client also provides an easy parameter to obfuscate hostname and IP information. The actual hostname and IP information is replaced with consistent obfuscated names sufficient for rule analysis.

How is data collected?

As new Insights rules are identified, there may be a need for additional metadata collection for analysis and detection. The list of System Information collected by Red Hat Insights is updated on an as-needed basis. The Red Hat Insights client, upon running, pings Red Hat to determine if any additional metadata is needed for rules which have been introduced since the last run. For example, if a new malware check is added, Insights may need to inspect new data sources to determine if a system is impacted.

This automatic check is enabled by default to ensure customers get all new rules and proactive alerts for their system. This ping to Red Hat can be disabled and manually updated via rpm version; however, this may cause customers to miss out on new health checks which depend on new information.

When Red Hat updates the collection rules, the rules are GPG signed by the redhat-tools GPG key. The Insights client will immediately abort if this signature cannot be verified. This file is also manually inspected carefully before each update is released.

Some examples

These are some of the files Red Hat Insights collects and sends to be processed:

/etc/redhat-release
/proc/meminfo
/var/log/messages

Do not worry! we do not collect the entire messages file, but rather the lines that match a potential rule (i.e. page allocation failure).

Or some of the commands we run:

Commands:

/bin/rpm -qa                        
/bin/uname -a
/usr/sbin/dmidecode

If you want to know the whole list of commands run and data collected you can take a look at this document.

As said, main configuration file is /etc/redhat-access-insights/redhat-access-insights.conf, and it’s a very usual ini type of file with # delimited comments, let’s take a brief look at it:

[root@server ~]# cat /etc/redhat-access-insights/redhat-access-insights.conf

# Example options in this file are the defaults

# Change log level, valid options DEBUG, INFO, WARNING, ERROR, CRITICAL. Default DEBUG
#loglevel=DEBUG

# Log each line executed
#trace=False

# Attempt to auto configure with Satellite server
#auto_config=True

# Change authentication method, valid options BASIC, CERT. Default BASIC
#authmethod=BASIC

# username to use when authmethod is BASIC
#username=
# password to use when authmethod is BASIC
#password=
[...]

To obfuscate your IP addresses, simply add the line:

obfuscate=True

Or to obfuscate hostnames, simply add the line:

obfuscate_hostname=True Blacklist

And addding items to the blacklist is as simple as using /etc/redhat-access-insights/remove.conf

Wrapping up...

We know you are concerned about the security of your data, yet there are times when it needs to be shared to provide the best capabilities for optimization and management. For this reason I wanted to let you know that the team here at Red Hat understands, and has worked hard to provide you with powerful tools that keep your data safe.
Wanna know more?? Find more info here.

Happy New Year! One way to get this new year started off right is to get started preventing some of the problems and downtime you may have experienced over the holidays. Using Insights can help future proof your infrastructure with integrated Ansible automation and a report on which systems you still need to patch for vulnerabilities like Meltdown and Spectre. Click here to see if you have systems that are missing the latest patches for these critical vulnerabilities.

Latest release

We're pleased to announce the latest service release of Red Hat Insights.
Read below for more information or check out the new features and let us know your thoughts by using the "Provide Feedback" button from within the Insights UI in the top right.

For more information about the latest Insights release, refer to our Red Hat Insights Release Notes.

Newest features:

User Interface Layout

In an effort to ease workflow and visibility throughout the Insights application we are shifting to a full screen application mode. This both frees up valuable screen space to more efficiently bring you the latest information on critical threats to your infrastructure, and gives us some more room to continue to add features that you request. Remember, keep the feedback coming, there's a provide feedback button at the top of the user interface.

Webhooks for integration

Insights has now enabled our Webhook integration for custom alerts and notifications. Many of the Incident Management and response services that are used today allow for Webhook consumption which allows for Insights notifications in the tools and workflows you already use. For more information on getting Webhooks configured, please visit Understanding Red Hat Insights - Webhook Integration.

Our engineers, for internal demo purposes, integrated Insights Webhooks functionality with the Slack messaging service. This example allowed for our teams to be notified in chat immediately when new critical issues had been identified on our registered hosts with Insights. For those interested our example code for this integration is online at:
https://github.com/jharting/insights-slackbot/blob/tutorial/tutorial/Tutorial.md
Please note, this is example code and is not officially supported by Red Hat.

CI/CD workflow examples

We previously added support for Jenkins and CI/CD workflow integration and now we have published some quick start example code. This code is available at https://github.com/RedHatInsights/insights-CI-examples/tree/master/examples/jenkins and we will have additional related materials coming soon right here on our blog.
Please note, this is example code and is not officially supported by Red Hat.

Newest rules widget

And to wrap up what's new in this service release, we've added a new widget to the UI with the latest rule additions to the Insights service. Since rules are constantly being added to Red Hat Insights on a regular basis you can now stay up to date with the latest additions on the Overview page. Additional filtering controls around when a rule was added have also been added to our rules page.

Thanks for your feedback and helping continue to improve the Insights service. Have a great new year!

Will