YAML-style denylist configuration for Red Hat Insights Client

Traditionally, the Insights Client utilizes an INI-style configuration, located at /etc/insights-client/remove.conf, with comma separated entries, for defining which files/commands/patterns to optionally omit in collection, like so:

[remove]
commands=/bin/date,/bin/df -al
files=/etc/hosts,/var/log/httpd/error_log
patterns=localhost,abc

With such a config, the full output of the commands and files defined would be omitted, and any lines containing the strings localhost or abc would be omitted from other command and file outputs.

This style of configuration is limited in its ability and expandability, so a new YAML configuration style has been devised.

Firstly, the functionality of remove.conf is split into two new files:
- /etc/insights-client/file-redaction.yaml -- containing thefiles, commands, and components directives
- /etc/insights-client/file-content-redaction.yaml -- containing the patterns and keywords directives

Secondly, as of insights-core version TBD, the preferred method of indicating which data to omit is by specifying components, or datasources that are part of insights-core. The .cache.json and .fallback.json files that have been provided as part of the insights-client installation and update flow are deprecated and will be provided for compatibility purposes only for some time.

Going forward, insights-core, which uses components as its granular functions, will be the main collection medium.

As to not break compatibility with any currently existing denylist configurations based on .cache.json entries, any commands and files specified in the denylist that match an available component will be converted to components automatically.

A list of components, and the files/commands they correspond to, can be found here: Datasource Catalog
The denylist requires the fully-qualified component name, so components from this list must be prefixed with insights.specs.default.DefaultSpecs. when specified in file-redaction.yaml.

This configuration also allows egrep-style regex matching for the patterns section, if desired. Documentation on how to use the format is included inline.

Example configuration of of /etc/insights-client/file-redaction.yaml

# file-redaction.yaml
---
# Omit entire output of commands
# Commands can be specified either by full command or
#   by the "symbolic_name" listed in /etc/insights-client/.cache.json
commands:
  - /bin/rpm -qa
  - /bin/ls
  - ethtool_i

# Omit entire output of files
# Files can be specified either by full filename or
#   by the "symbolic_name" listed in .cache.json
files:
  - /etc/audit/auditd.conf
  - cluster_conf

# Omit insights-core components
# Refer to the Datasource Catalog here for a full list of available insights-core components,
#   and the commands/files they correspond to.
#   See items listed under "General Datasources":
#   https://insights-core.readthedocs.io/en/latest/specs_catalog.html
#
# Components specified here must be listed with the fully qualified name, i.e.
#   must be prefixed with "insights.specs.default.DefaultSpecs."
components:
  - insights.specs.default.DefaultSpecs.httpd_V
  - insights.specs.default.DefaultSpecs.mysql_log
  - insights.specs.default.DefaultSpecs.ifconfig

Example configuration of of /etc/insights-client/file-content-redaction.yaml

# file-content-redaction.yaml
---
# Omit lines from files and command output using parameters listed here.
# Lines matching the parameters specified will be omitted
#   in the order that the parameters are given, e.g.,
#
# patterns:
#   - "example_string_1"
#   - "example_string_2"
#
# Lines containing "example_string_1" or "example_string_2" will be 
# omitted from output.
#
# To use regular expressions, wrap the array with "regex" like the following example:
#
# patterns:
#   regex:
#     - abc.*
#     - localhost[[:digit:]]
#
# Lines matching these regular expressions will be omitted
# from output.
patterns:
  regex:
    - "abc.*"

# Replace keywords in files and command output with generic identifiers by the soscleaner module
keywords:
  - 1.1.1.1
  - keyword_example

Related Resources

Red Hat Insights security information
System information collected by Red Hat Insights
Obfuscating IP addresses