Red Hat Training

A Red Hat training course is available for RHEL 8

Chapter 38. Using the Logging System Role

As a system administrator, you can use the Logging System Role to configure a RHEL host as a logging server to collect logs from many client systems.

38.1. The Logging System Role

With the Logging System Role, you can deploy logging configurations on local and remote hosts.

To apply a Logging System Role on one or more systems, you define the logging configuration in a playbook. A playbook is a list of one or more plays. Playbooks are human-readable, and they are written in the YAML format. For more information about playbooks, see Working with playbooks in Ansible documentation.

The set of systems that you want Ansible to configure according to the playbook is defined in an inventory file. For more information on creating and using inventories, see How to build your inventory in Ansible documentation.

Logging solutions provide multiple ways of reading logs and multiple logging outputs.

For example, a logging system can receive the following inputs:

  • local files,
  • systemd/journal,
  • another logging system over the network.

In addition, a logging system can have the following outputs:

  • logs are stored in the local files in the /var/log directory,
  • logs are sent to Elasticsearch,
  • logs are forwarded to another logging system.

With the logging system role, you can combine the inputs and outputs to fit your needs. For example, you can configure a logging solution that stores inputs from journal in a local file, whereas inputs read from files are both forwarded to another logging system and stored in the local log files.

38.2. Logging System Role parameters

In a Logging System Role playbook, you define the inputs in the logging_inputs parameter, outputs in the logging_outputs parameter, and the relationships between the inputs and outputs in the logging_flows parameter. The Logging System Role processes these variables with additional options to configure the logging system. You can also enable encryption.

Note

Currently, the only available logging system in the Logging System Role is Rsyslog.

  • logging_inputs - List of inputs for the logging solution.

    • name - Unique name of the input. Used in the logging_flows inputs list and a part of the generated config file name.
    • type - Type of the input element. The type specifies a task type which corresponds to a directory name in roles/rsyslog/{tasks,vars}/inputs/.

      • basics - Inputs configuring inputs from systemd journal or unix socket.

        • kernel_message - Load imklog if set to true. Default to false.
        • use_imuxsock - Use imuxsock instead of imjournal. Default to false.
        • ratelimit_burst - Maximum number of messages that can be emitted within ratelimit_interval. Default to 20000 if use_imuxsock is false. Default to 200 if use_imuxsock is true.
        • ratelimit_interval - Interval to evaluate ratelimit_burst. Default to 600 seconds if use_imuxsock is false. Default to 0 if use_imuxsock is true. 0 indicates rate limiting is turned off.
        • persist_state_interval - Journal state is persisted every value messages. Default to 10. Effective only when use_imuxsock is false.
      • files - Inputs configuring inputs from local files.
      • remote - Inputs configuring inputs from the other logging system over network.
    • state - State of the configuration file. present or absent. Default to present.
  • logging_outputs - List of outputs for the logging solution.

    • files - Outputs configuring outputs to local files.
    • forwards - Outputs configuring outputs to another logging system.
    • remote_files - Outputs configuring outputs from another logging system to local files.
  • logging_flows - List of flows that define relationships between logging_inputs and logging_outputs. The logging_flows variable has the following keys:

    • name - Unique name of the flow
    • inputs - List of logging_inputs name values
    • outputs - List of logging_outputs name values.

Additional resources

  • Documentation installed with the rhel-system-roles package in /usr/share/ansible/roles/rhel-system-roles.logging/README.html

38.3. Applying a local Logging System Role

Follow these steps to prepare and apply a Red Hat Ansible Engine playbook to configure a logging solution on a set of separate machines. Each machine will record logs locally.

Prerequisites

  • You have Red Hat Ansible Engine installed on the system from which you want to run the playbook.

    Note

    You do not have to have Red Hat Ansible Engine installed on the systems on which you want to deploy the logging solution.

  • You have the rhel-system-roles package on the system from which you want to run the playbook.

    Note

    You do not have to have rsyslog installed, because the system role installs rsyslog when deployed.

  • You have an inventory file listing the systems on which you want to configure the logging solution.

Procedure

  1. Create a playbook that defines the required role:

    1. Create a new YAML file and open it in a text editor, for example:

      # vi logging-playbook.yml
    2. Insert the following content:

      ---
      - name: Deploying basics input and implicit files output
        hosts: all
        roles:
          - linux-system-roles.logging
        vars:
          logging_inputs:
            - name: system_input
              type: basics
          logging_outputs:
            - name: files_output
              type: files
          logging_flows:
            - name: flow1
              inputs: [system_input]
              outputs: [files_output]
  2. Execute the playbook on a specific inventory:

    # ansible-playbook -i inventory-file /path/to/file/logging-playbook.yml

    Where:

    • inventory-file is the inventory file.
    • logging-playbook.yml is the playbook you use.

Verification

  1. Test the syntax of the /etc/rsyslog.conf file:

    # rsyslogd -N 1
    rsyslogd: version 8.1911.0-6.el8, config validation run (level 1), master config /etc/rsyslog.conf
    rsyslogd: End of config validation run. Bye.
  2. Verify that the system sends messages to the log:

    1. Send a test message:

      # logger test
    2. View the /var/log/messages log, for example:

      # cat /var/log/messages
      Aug  5 13:48:31 hostname root[6778]: test

      Where `hostname` is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

38.4. Filtering logs in a local Logging System Role

You can deploy a logging solution which filters the logs based on the rsyslog property-based filter.

Prerequisites

  • Access and permissions to one or more managed nodes, which are systems you want to configure with the Logging System Role.
  • Access and permissions to a control node, which is a system from which Red Hat Ansible Engine configures other systems.

    On the control node:

    • Red Hat Ansible Engine is installed
    • The rhel-system-roles package is installed
    • An inventory file which lists the managed nodes.

Procedure

  1. Create a new playbook.yml file with the following content:

    ---
    - name: Deploying files input and configured files output
      hosts: all
      roles:
        - linux-system-roles.logging
      vars:
        logging_inputs:
          - name: files_input0
            type: files
            input_log_path: /var/log/containerA/*.log
          - name: files_input1
            type: files
            input_log_path: /var/log/containerB/*.log
        logging_outputs:
          - name: files_output0
            type: files
            property: msg
            property_op: contains
            property_value: error
            path: /var/log/errors.log
          - name: files_output1
            type: files
            property: msg
            property_op: "!contains"
            property_value: error
            path: /var/log/others.log
        logging_flows:
          - name: flow0
            inputs: [files_input0, files_input1]
            outputs: [files_output0, files_output1]

    Using this configuration, all messages that contain the error string are logged in /var/log/errors.log, and all other messages are logged in /var/log/others.log.

    You can replace the error property value with the string by which you want to filter.

    You can modify the variables according to your preferences.

  2. Optional: Verify playbook syntax.

    # ansible-playbook --syntax-check playbook.yml
  3. Run the playbook on your inventory file:

    # ansible-playbook -i inventory_file /path/to/file/playbook.yml

Verification

  1. Test the syntax of the /etc/rsyslog.conf file:

    # rsyslogd -N 1
    rsyslogd: version 8.1911.0-6.el8, config validation run (level 1), master config /etc/rsyslog.conf
    rsyslogd: End of config validation run. Bye.
  2. Verify that the system sends messages that contain the error string to the log:

    1. Send a test message:

      # logger error
    2. View the /var/log/errors.log log, for example:

      # cat /var/log/errors.log
      Aug  5 13:48:31 hostname root[6778]: error

      Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Additional resources

  • Documentation installed with the rhel-system-roles package in /usr/share/ansible/roles/rhel-system-roles.logging/README.html

38.5. Applying a remote logging solution using the Logging System Role

Follow these steps to prepare and apply a Red Hat Ansible Engine playbook to configure a remote logging solution. In this playbook, one or more clients take logs from systemd-journal and forward them to a remote server. The server receives remote input from remote_rsyslog and remote_files and outputs the logs to local files in directories named by remote host names.

Prerequisites

  • You have Red Hat Ansible Engine installed on the system from which you want to run the playbook.

    Note

    You do not have to have Red Hat Ansible Engine installed on the systems on which you want to deploy the logging solution.

  • You have the rhel-system-roles package on the system from which you want to run the playbook.

    Note

    You do not have to have rsyslog installed, because the system role installs rsyslog when deployed.

  • You have at least two systems:

    • At least one will be the logging server.
    • At least one will be the logging client.

Procedure

  1. Create a playbook that defines the required role:

    1. Create a new YAML file and open it in a text editor, for example:

      # vi logging-playbook.yml
    2. Insert the following content into the file:

      ---
      - name: Deploying remote input and remote_files output
        hosts: server
        roles:
          - linux-system-roles.logging
        vars:
          logging_inputs:
            - name: remote_udp_input
              type: remote
              udp_ports: [ 601 ]
            - name: remote_tcp_input
              type: remote
              tcp_ports: [ 601 ]
          logging_outputs:
            - name: remote_files_output
              type: remote_files
          logging_flows:
            - name: flow_0
              inputs: [remote_udp_input, remote_tcp_input]
              outputs: [remote_files_output]
      
      - name: Deploying basics input and forwards output
        hosts: clients
        roles:
          - linux-system-roles.logging
        vars:
          logging_inputs:
            - name: basic_input
              type: basics
          logging_outputs:
            - name: forward_output0
              type: forwards
              severity: info
              target: host1.example.com
              udp_port: 601
            - name: forward_output1
              type: forwards
              facility: mail
              target: host1.example.com
              tcp_port: 601
          logging_flows:
            - name: flows0
              inputs: [basic_input]
              outputs: [forward_output0, forward_output1]
      
      [basic_input]
      [forward_output0, forward_output1]

      Where host1.example.com is the logging server.

      Note

      You can modify the parameters in the playbook to fit your needs.

      Warning

      The logging solution works only with the ports defined in the SELinux policy of the server or client system and open in the firewall. The default SELinux policy includes ports 601, 514, 6514, 10514, and 20514. To use a different port, modify the SELinux policy on the client and server systems . Configuring the firewall through system roles is not yet supported.

  2. Create an inventory file that lists your servers and clients:

    1. Create a new file and open it in a text editor, for example:

      # vi inventory.ini
    2. Insert the following content into the inventory file:

      [servers]
      server ansible_host=host1.example.com
      [clients]
      client ansible_host=host2.example.com

      Where: * host1.example.com is the logging server. * host2.example.com is the logging client.

  3. Execute the playbook on your inventory.

    # ansible-playbook -i /path/to/file/inventory.ini /path/to/file/_logging-playbook.yml

    Where:

    • inventory.ini is the inventory file.
    • logging-playbook.yml is the playbook you created.

Verification

  1. On both the client and the server system, test the syntax of the /etc/rsyslog.conf file:

    # rsyslogd -N 1
    rsyslogd: version 8.1911.0-6.el8, config validation run (level 1), master config /etc/rsyslog.conf
    rsyslogd: End of config validation run. Bye.
  2. Verify that the client system sends messages to the server:

    1. On the client system, send a test message:

      # logger test
    2. On the server system, view the /var/log/messages log, for example:

      # cat /var/log/messages
      Aug  5 13:48:31 host2.example.com root[6778]: test

      Where host2.example.com is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Additional resources

38.6. Using the logging system roles with RELP

Reliable Event Logging Protocol (RELP) is a networking protocol for data and message logging over the TCP network. It ensures reliable delivery of event messages and you can use it in environments that do not tolerate any message loss.

The RELP sender transfers log entries in form of commands and the receiver acknowledges them once they are processed. To ensure consistency, RELP stores the transaction number to each transferred command for any kind of message recovery.

You can consider a remote logging system in between the RELP Client and RELP Server. the RELP Client transfers the logs to the remote logging system and the RELP Server receives all the logs sent by the remote logging system.

Administrators can use the Logging System Role to configure the logging system to reliably send and receive log entries.

38.6.1. Configuring client logging with RELP

You can use the Logging System Role to configure logging in RHEL systems that are logged on a local machine and can transfer logs to the remote logging system with RELP by running an Ansible playbook.

This procedure configures RELP on all hosts in the clients group in the Ansible inventory. The RELP configuration uses Transport Layer Security (TLS) to encrypt the message transmission for secure transfer of logs over the network.

Prerequisites

  • You have permissions to run playbooks on managed nodes on which you want to configure RELP.
  • The managed nodes are listed in the inventory file on the control node.
  • The ansible and rhel-system-roles packages are installed on the control node.

Procedure

  1. Create a playbook.yml file with the following content:

    ---
    - name: Deploying basic input and relp output
      hosts: clients
      roles:
        - rhel-system-roles.logging
      vars:
        logging_inputs:
          - name: basic_input
            type: basics
        logging_outputs:
          - name: relp_client
            type: relp
            target: logging.server.com
            port: 20514
            tls: true
            ca_cert: /etc/pki/tls/certs/ca.pem
            cert: /etc/pki/tls/certs/client-cert.pem
            private_key: /etc/pki/tls/private/client-key.pem
            pki_authmode: name
            permitted_servers:
              - '*.server.example.com'
        logging_flows:
          - name: example_flow
            inputs: [basic_input]
            outputs: [relp_client]

    The playbooks uses following settings:

    • target: This is a required parameter that specifies the host name where the remote logging system is running.
    • port: Port number the remote logging system is listening.
    • tls: Ensures secure transfer of logs over the network. If you do not want a secure wrapper you can set the tls variable to false. By default tls parameter is set to true while working with RELP and requires key/certificates and triplets {ca_cert, cert, private_key} and/or {ca_cert_src, cert_src, private_key_src}.

      • If {ca_cert_src, cert_src, private_key_src} triplet is set, the default locations /etc/pki/tls/certs and /etc/pki/tls/private are used as the destination on the managed node to transfer files from control node. In this case, the file names are identical to the original ones in the triplet
      • If {ca_cert, cert, private_key} triplet is set, files are expected to be on the default path before the logging configuration.
      • If both the triplets are set, files are transfered from local path from control node to specific path of the managed node.
    • ca_cert: Represents the path to CA certificate. Default path is /etc/pki/tls/certs/ca.pem and the file name is set by the user.
    • cert: Represents the path to cert. Default path is /etc/pki/tls/certs/server-cert.pem and the file name is set by the user.
    • private_key: Represents the path to private key. Default path is /etc/pki/tls/private/server-key.pem and the file name is set by the user.
    • ca_cert_src: Reprents local CA cert file path which is copied to the target host. If ca_cert is specified, it is copied to the location.
    • cert_src: Reprents the local cert file path which is copied to the target host. If cert is specified, it is copied to the location.
    • private_key_src: Represents the local key file path which is copied to the target host. If private_key is specified, it is copied to the location.
    • pki_authmode: Accepts the authentication mode as name or fingerprint.
    • permitted_servers: List of servers that will be allowed by the logging client to connect and send logs over TLS.
    • inputs: List of logging input dictionary.
    • outputs: List of logging output dictionary.
  2. Optional: Verify playbook syntax.

    # ansible-playbook --syntax-check playbook.yml
  3. Run the playbook:

    # ansible-playbook -i inventory_file playbook.yml

38.6.2. Configuring server logging with RELP

You can use the Logging System Role to configure logging in RHEL systems as a server and can receive logs from the remote logging system with RELP by running an Ansible playbook.

This procedure configures RELP on all hosts in the server group in the Ansible inventory. The RELP configuration uses TLS to encrypt the message transmission for secure transfer of logs over the network.

Prerequisites

  • You have permissions to run playbooks on managed nodes on which you want to configure RELP.
  • The managed nodes are listed in the inventory file on the control node.
  • The ansible and rhel-system-roles packages are installed on the control node.

Procedure

  1. Create a playbook.yml file with the following content:

    ---
    - name: Deploying remote input and remote_files output
      hosts: server
      roles:
        - rhel-system-roles.logging
      vars:
        logging_inputs:
          - name: relp_server
            type: relp
            port: 20514
            tls: true
            ca_cert: /etc/pki/tls/certs/ca.pem
            cert: /etc/pki/tls/certs/server-cert.pem
            private_key: /etc/pki/tls/private/server-key.pem
            pki_authmode: name
            permitted_clients:
              - '*example.client.com'
        logging_outputs:
          - name: remote_files_output
            type: remote_files
        logging_flows:
          - name: example_flow
            inputs: relp_server
            outputs: remote_files_output

    The playbooks uses following settings:

    • port: Port number the remote logging system is listening.
    • tls: Ensures secure transfer of logs over the network. If you do not want a secure wrapper you can set the tls variable to false. By default tls parameter is set to true while working with RELP and requires key/certificates and triplets {ca_cert, cert, private_key} and/or {ca_cert_src, cert_src, private_key_src}.

      • If {ca_cert_src, cert_src, private_key_src} triplet is set, the default locations /etc/pki/tls/certs and /etc/pki/tls/private are used as the destination on the managed node to transfer files from control node. In this case, the file names are identical to the original ones in the triplet
      • If {ca_cert, cert, private_key} triplet is set, files are expected to be on the default path before the logging configuration.
      • If both the triplets are set, files are transfered from local path from control node to specific path of the managed node.
    • ca_cert: Represents the path to CA certificate. Default path is /etc/pki/tls/certs/ca.pem and the file name is set by the user.
    • cert: Represents the path to cert. Default path is /etc/pki/tls/certs/server-cert.pem and the file name is set by the user.
    • private_key: Represents the path to private key. Default path is /etc/pki/tls/private/server-key.pem and the file name is set by the user.
    • ca_cert_src: Reprents local CA cert file path which is copied to the target host. If ca_cert is specified, it is copied to the location.
    • cert_src: Reprents the local cert file path which is copied to the target host. If cert is specified, it is copied to the location.
    • private_key_src: Represents the local key file path which is copied to the target host. If private_key is specified, it is copied to the location.
    • pki_authmode: Accepts the authentication mode as name or fingerprint.
    • permitted_clients: List of clients that will be allowed by the logging server to connect and send logs over TLS.
    • inputs: List of logging input dictionary.
    • outputs: List of logging output dictionary.
  2. Optional: Verify playbook syntax.

    # ansible-playbook --syntax-check playbook.yml
  3. Run the playbook:

    # ansible-playbook -i inventory_file playbook.yml

38.7. Using logging system role with TLS

Transport Layer Security (TLS) is a cryptographic protocol designed to securely communicate over the computer network.

As an administrator, you can use the Logging System Role on RHEL to configure secure transfer of logs using Red Hat Ansible Automation Platform.

38.7.1. Configuring client logging with TLS

You can use the Logging System Role to configure logging in RHEL systems that are logged on a local machine and can transfer logs to the remote logging system with TLS by running an Ansible playbook.

This procedure configures TLS on all hosts in the clients group in the Ansible inventory. The TLS protocol encrypts the message transmission for secure transfer of logs over the network.

Prerequisites

  • You have permissions to run playbooks on managed nodes on which you want to configure TLS.
  • The managed nodes are listed in the inventory file on the control node.
  • The ansible and rhel-system-roles packages are installed on the control node.

Procedure

  1. Create a playbook.yml file with the following content:

    ---
    - name: Deploying files input and forwards output with certs
      hosts: clients
      roles:
        - rhel-system-roles.logging
      vars:
        logging_pki_files:
          - ca_cert_src: /local/path/to/ca_cert.pem
            cert_src: /local/path/to/cert.pem
            private_key_src: /local/path/to/key.pem
        logging_inputs:
          - name: input_name
            type: files
            input_log_path: /var/log/containers/*.log
        logging_outputs:
          - name: output_name
            type: forwards
            target: your_target_host
            tcp_port: 514
            tls: true
            pki_authmode: x509/name
            permitted_server: 'server.example.com'
        logging_flows:
          - name: flow_name
            inputs: [input_name]
            outputs: [output_name]

    The playbook uses the following parameters:

    logging_pki_files
    Using this parameter you can configure TLS and has to pass ca_cert_src, cert_src, and private_key_src parameters.
    ca_cert
    Represents the path to CA certificate. Default path is /etc/pki/tls/certs/ca.pem and the file name is set by the user.
    cert
    Represents the path to cert. Default path is /etc/pki/tls/certs/server-cert.pem and the file name is set by the user.
    private_key
    Represents the path to private key. Default path is /etc/pki/tls/private/server-key.pem and the file name is set by the user.
    ca_cert_src
    Reprents local CA cert file path which is copied to the target host. If ca_cert is specified, it is copied to the location.
    cert_src
    Reprents the local cert file path which is copied to the target host. If cert is specified, it is copied to the location.
    private_key_src
    Represents the local key file path which is copied to the target host. If private_key is specified, it is copied to the location.
    tls
    Using this parameter ensures secure transfer of logs over the network. If you do not want a secure wrapper, you can set tls: true.
  2. Verify playbook syntax:

    # ansible-playbook --syntax-check playbook.yml
  3. Run the playbook on your inventory file:

    # ansible-playbook -i inventory_file playbook.yml

38.7.2. Configuring server logging with TLS

You can use the Logging System Role to configure logging in RHEL systems as a server and can receive logs from the remote logging system with TLS by running an Ansible playbook.

This procedure configures TLS on all hosts in the server group in the Ansible inventory.

Prerequisites

  • You have permissions to run playbooks on managed nodes on which you want to configure TLS.
  • The managed nodes are listed in the inventory file on the control node.
  • The ansible and rhel-system-roles packages are installed on the control node.

Procedure

  1. Create a playbook.yml file with the following content:

    ---
    - name: Deploying remote input and remote_files output with certs
      hosts: server
      roles:
        - rhel-system-roles.logging
      vars:
        logging_pki_files:
          - ca_cert_src: /local/path/to/ca_cert.pem
            cert_src: /local/path/to/cert.pem
            private_key_src: /local/path/to/key.pem
        logging_inputs:
          - name: input_name
            type: remote
            tcp_ports: 514
            tls: true
            permitted_clients: ['clients.example.com']
        logging_outputs:
          - name: output_name
            type: remote_files
            remote_log_path: /var/log/remote/%FROMHOST%/%PROGRAMNAME:::secpath-replace%.log
            async_writing: true
            client_count: 20
            io_buffer_size: 8192
        logging_flows:
          - name: flow_name
            inputs: [input_name]
            outputs: [output_name]

    The playbook uses the following parameters:

    logging_pki_files
    Using this parameter you can configure TLS and has to pass ca_cert_src, cert_src, and private_key_src parameters.
    ca_cert
    Represents the path to CA certificate. Default path is /etc/pki/tls/certs/ca.pem and the file name is set by the user.
    cert
    Represents the path to cert. Default path is /etc/pki/tls/certs/server-cert.pem and the file name is set by the user.
    private_key
    Represents the path to private key. Default path is /etc/pki/tls/private/server-key.pem and the file name is set by the user.
    ca_cert_src
    Reprents local CA cert file path which is copied to the target host. If ca_cert is specified, it is copied to the location.
    cert_src
    Reprents the local cert file path which is copied to the target host. If cert is specified, it is copied to the location.
    private_key_src
    Represents the local key file path which is copied to the target host. If private_key is specified, it is copied to the location.
    tls
    Using this parameter ensures secure transfer of logs over the network. If you do not want a secure wrapper, you can set tls: true.
  2. Verify playbook syntax:

    # ansible-playbook --syntax-check playbook.yml
  3. Run the playbook on your inventory file:

    # ansible-playbook -i inventory_file playbook.yml

38.8. Additional resources

  • Getting started with RHEL System Roles
  • Documentation installed with the rhel-system-roles package in /usr/share/ansible/roles/rhel-system-roles.logging/README.html
  • RHEL System Roles KB article
  • For details about the ansible-playbook command, see the ansible-playbook(1) man page.