Chapter 15. Configuring a remote logging solution

To ensure that logs from various machines in your environment are recorded centrally on a logging server, you can configure the Rsyslog application to record logs that fit specific criteria from the client system to the server.

15.1. The Rsyslog logging service

The Rsyslog application, in combination with the systemd-journald service, provides local and remote logging support in Red Hat Enterprise Linux. The rsyslogd daemon continuously reads syslog messages received by the systemd-journald service from the Journal. rsyslogd then filters and processes these syslog events and records them to rsyslog log files or forwards them to other services according to its configuration.

The rsyslogd daemon also provides extended filtering, encryption protected relaying of messages, input and output modules, and support for transportation using the TCP and UDP protocols.

In /etc/rsyslog.conf, which is the main configuration file for rsyslog, you can specify the rules according to which rsyslogd handles the messages. Generally, you can classify messages by their source and topic (facility) and urgency (priority), and then assign an action that should be performed when a message fits these criteria.

In /etc/rsyslog.conf, you can also see a list of log files maintained by rsyslogd. Most log files are located in the /var/log/ directory. Some applications, such as httpd and samba, store their log files in a subdirectory within /var/log/.

Additional resources

  • The rsyslogd(8) and rsyslog.conf(5) man pages.
  • Documentation installed with the rsyslog-doc package in the /usr/share/doc/rsyslog/html/index.html file.

15.2. Installing Rsyslog documentation

The Rsyslog application has extensive online documentation that is available at https://www.rsyslog.com/doc/, but you can also install the rsyslog-doc documentation package locally.

Prerequisites

  • You have activated the AppStream repository on your system.
  • You are authorized to install new packages using sudo.

Procedure

  • Install the rsyslog-doc package: 

    # dnf install rsyslog-doc

Verification

  • Open the /usr/share/doc/rsyslog/html/index.html file in a browser of your choice, for example: 

    $ firefox /usr/share/doc/rsyslog/html/index.html &

15.3. Configuring a server for remote logging over TCP

The Rsyslog application enables you to both run a logging server and configure individual systems to send their log files to the logging server. To use remote logging through TCP, configure both the server and the client. The server collects and analyzes the logs sent by one or more client systems.

With the Rsyslog application, you can maintain a centralized logging system where log messages are forwarded to a server over the network. To avoid message loss when the server is not available, you can configure an action queue for the forwarding action. This way, messages that failed to be sent are stored locally until the server is reachable again. Note that such queues cannot be configured for connections using the UDP protocol.

The omfwd plug-in provides forwarding over UDP or TCP. The default protocol is UDP. Because the plug-in is built in, it does not have to be loaded.

By default, rsyslog uses TCP on port 514.

Prerequisites

  • Rsyslog is installed on the server system.
  • You are logged in as root on the server.
  • The policycoreutils-python-utils package is installed for the optional step using the semanage command.
  • The firewalld service is running.

Procedure

  1. Optional: To use a different port for rsyslog traffic, add the syslogd_port_t SELinux type to port. For example, enable port 30514: 

    # semanage port -a -t syslogd_port_t -p tcp 30514

  2. Optional: To use a different port for rsyslog traffic, configure firewalld to allow incoming rsyslog traffic on that port. For example, allow TCP traffic on port 30514: 

    # firewall-cmd --zone=<zone-name> --permanent --add-port=30514/tcp
success
# firewall-cmd --reload

  3. Create a new file in the /etc/rsyslog.d/ directory named, for example, remotelog.conf, and insert the following content: 

    # Define templates before the rules that use them
# Per-Host templates for remote systems
template(name="TmplAuthpriv" type="list") {
    constant(value="/var/log/remote/auth/")
    property(name="hostname")
    constant(value="/")
    property(name="programname" SecurePath="replace")
    constant(value=".log")
    }

template(name="TmplMsg" type="list") {
    constant(value="/var/log/remote/msg/")
    property(name="hostname")
    constant(value="/")
    property(name="programname" SecurePath="replace")
    constant(value=".log")
    }

# Provides TCP syslog reception
module(load="imtcp")

# Adding this ruleset to process remote messages
ruleset(name="remote1"){
     authpriv.*   action(type="omfile" DynaFile="TmplAuthpriv")
      *.info;mail.none;authpriv.none;cron.none
action(type="omfile" DynaFile="TmplMsg")
}

input(type="imtcp" port="30514" ruleset="remote1")
  4. Save the changes to the /etc/rsyslog.d/remotelog.conf file.

  5. Test the syntax of the /etc/rsyslog.conf file: 

    # rsyslogd -N 1
rsyslogd: version 8.1911.0-2.el8, config validation run...
rsyslogd: End of config validation run. Bye.

  6. Make sure the rsyslog service is running and enabled on the logging server: 

    # systemctl status rsyslog

  7. Restart the rsyslog service. 

    # systemctl restart rsyslog

  8. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot: 

    # systemctl enable rsyslog

Your log server is now configured to receive and store log files from the other systems in your environment.

Additional resources

  • rsyslogd(8), rsyslog.conf(5), semanage(8), and firewall-cmd(1) man pages.
  • Documentation installed with the rsyslog-doc package in the /usr/share/doc/rsyslog/html/index.html file.

15.4. Configuring remote logging to a server over TCP

Follow this procedure to configure a system for forwarding log messages to a server over the TCP protocol. The omfwd plug-in provides forwarding over UDP or TCP. The default protocol is UDP. Because the plug-in is built in, you do not have to load it.

Prerequisites

  • The rsyslog package is installed on the client systems that should report to the server.
  • You have configured the server for remote logging.
  • The specified port is permitted in SELinux and open in firewall.
  • The system contains the policycoreutils-python-utils package, which provides the semanage command for adding a non-standard port to the SELinux configuration.

Procedure

  1. Create a new file in the /etc/rsyslog.d/ directory named, for example, 10-remotelog.conf, and insert the following content: 

    *.* action(type="omfwd"
      queue.type="linkedlist"
      queue.filename="example_fwd"
      action.resumeRetryCount="-1"
      queue.saveOnShutdown="on"
      target="example.com" port="30514" protocol="tcp"
     )

    Where:

    • The queue.type="linkedlist" setting enables a LinkedList in-memory queue,
    • The queue.filename setting defines a disk storage. The backup files are created with the example_fwd prefix in the working directory specified by the preceding global workDirectory directive.
    • The action.resumeRetryCount -1 setting prevents rsyslog from dropping messages when retrying to connect if server is not responding,
    • The queue.saveOnShutdown="on" setting saves in-memory data if rsyslog shuts down.

    • The last line forwards all received messages to the logging server. Port specification is optional.

      With this configuration, rsyslog sends messages to the server but keeps messages in memory if the remote server is not reachable. A file on disk is created only if rsyslog runs out of the configured memory queue space or needs to shut down, which benefits the system performance.

    Note

    Rsyslog processes configuration files /etc/rsyslog.d/ in the lexical order.

  2. Restart the rsyslog service. 

    # systemctl restart rsyslog

Verification

To verify that the client system sends messages to the server, follow these steps:

  1. On the client system, send a test message: 

    # logger test

  2. On the server system, view the /var/log/messages log, for example: 

    # cat /var/log/remote/msg/hostname/root.log
Feb 25 03:53:17 hostname root[6064]: test

    Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Additional resources

  • rsyslogd(8) and rsyslog.conf(5) man pages.
  • Documentation installed with the rsyslog-doc package in the /usr/share/doc/rsyslog/html/index.html file.

15.5. Configuring TLS-encrypted remote logging

By default, Rsyslog sends remote-logging communication in the plain text format. If your scenario requires to secure this communication channel, you can encrypt it using TLS.

To use encrypted transport through TLS, configure both the server and the client. The server collects and analyzes the logs sent by one or more client systems.

You can use either the ossl network stream driver (OpenSSL) or the gtls stream driver (GnuTLS).

Note

If you have a separate system with higher security, for example, a system that is not connected to any network or has stricter authorizations, use the separate system as the certifying authority (CA).

Prerequisites

  • You have root access to both the client and server systems.
  • The rsyslog and rsyslog-openssl packages are installed on the server and the client systems.
  • If you use the gtls network stream driver, install the rsyslog-gnutls package instead of rsyslog-openssl.
  • If you generate certificates using the certtool command, install the gnutls-utils package.

  • On your logging server, the following certificates are in the /etc/pki/ca-trust/source/anchors/ directory and your system configuration is updated by using the update-ca-trust command:

    • ca-cert.pem - a CA certificate that can verify keys and certificates on logging servers and clients.
    • server-cert.pem - a public key of the logging server.
    • server-key.pem - a private key of the logging server.

  • On your logging clients, the following certificates are in the /etc/pki/ca-trust/source/anchors/ directory and your system configuration is updated by using update-ca-trust:

    • ca-cert.pem - a CA certificate that can verify keys and certificates on logging servers and clients.
    • client-cert.pem - a public key of a client.
    • client-key.pem - a private key of a client.
    • If the server runs RHEL 9.2 or later and the FIPS mode is enabled, clients must either support the Extended Master Secret (EMS) extension or use TLS 1.3. TLS 1.2 connections without EMS fail. For more information, see the TLS extension "Extended Master Secret" enforced Knowledgebase article.

Procedure

  1. Configure the server for receiving encrypted logs from your client systems:

    1. Create a new file in the /etc/rsyslog.d/ directory named, for example, securelogser.conf.

    2. To encrypt the communication, the configuration file must contain paths to certificate files on your server, a selected authentication method, and a stream driver that supports TLS encryption. Add the following lines to the /etc/rsyslog.d/securelogser.conf file: 

      # Set certificate files
global(
  DefaultNetstreamDriverCAFile="/etc/pki/ca-trust/source/anchors/ca-cert.pem"
  DefaultNetstreamDriverCertFile="/etc/pki/ca-trust/source/anchors/server-cert.pem"
  DefaultNetstreamDriverKeyFile="/etc/pki/ca-trust/source/anchors/server-key.pem"
)

# TCP listener
module(
  load="imtcp"
  PermittedPeer=["client1.example.com", "client2.example.com"]
  StreamDriver.AuthMode="x509/name"
  StreamDriver.Mode="1"
  StreamDriver.Name="ossl"
)

# Start up listener at port 514
input(
  type="imtcp"
  port="514"
)
      Note

      If you prefer the GnuTLS driver, use the StreamDriver.Name="gtls" configuration option. See the documentation installed with the rsyslog-doc package for more information about less strict authentication modes than x509/name.

    3. Save the changes to the /etc/rsyslog.d/securelogser.conf file.

    4. Verify the syntax of the /etc/rsyslog.conf file and any files in the /etc/rsyslog.d/ directory: 

      # rsyslogd -N 1
rsyslogd: version 8.1911.0-2.el8, config validation run (level 1)...
rsyslogd: End of config validation run. Bye.

    5. Make sure the rsyslog service is running and enabled on the logging server: 

      # systemctl status rsyslog

    6. Restart the rsyslog service: 

      # systemctl restart rsyslog

    7. Optional: If Rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot: 

      # systemctl enable rsyslog

  2. Configure clients for sending encrypted logs to the server:

    1. On a client system, create a new file in the /etc/rsyslog.d/ directory named, for example, securelogcli.conf.

    2. Add the following lines to the /etc/rsyslog.d/securelogcli.conf file: 

      # Set certificate files
global(
  DefaultNetstreamDriverCAFile="/etc/pki/ca-trust/source/anchors/ca-cert.pem"
  DefaultNetstreamDriverCertFile="/etc/pki/ca-trust/source/anchors/client-cert.pem"
  DefaultNetstreamDriverKeyFile="/etc/pki/ca-trust/source/anchors/client-key.pem"
)


# Set up the action for all messages
*.* action(
  type="omfwd"
  StreamDriver="ossl"
  StreamDriverMode="1"
  StreamDriverPermittedPeers="server.example.com"
  StreamDriverAuthMode="x509/name"
  target="server.example.com" port="514" protocol="tcp"
)
      Note

      If you prefer the GnuTLS driver, use the StreamDriver.Name="gtls" configuration option.

    3. Save the changes to the /etc/rsyslog.d/securelogser.conf file.

    4. Verify the syntax of the /etc/rsyslog.conf file and other files in the /etc/rsyslog.d/ directory: 

      # rsyslogd -N 1
rsyslogd: version 8.1911.0-2.el8, config validation run (level 1)...
rsyslogd: End of config validation run. Bye.

    5. Make sure the rsyslog service is running and enabled on the logging server: 

      # systemctl status rsyslog

    6. Restart the rsyslog service: 

      # systemctl restart rsyslog

    7. Optional: If Rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot: 

      # systemctl enable rsyslog

Verification

To verify that the client system sends messages to the server, follow these steps:

  1. On the client system, send a test message: 

    # logger test

  2. On the server system, view the /var/log/messages log, for example: 

    # cat /var/log/remote/msg/hostname/root.log
Feb 25 03:53:17 hostname root[6064]: test

    Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Additional resources

  • certtool(1), openssl(1), update-ca-trust(8), rsyslogd(8), and rsyslog.conf(5) man pages.
  • Documentation installed with the rsyslog-doc package at /usr/share/doc/rsyslog/html/index.html.
  • Using the logging System Role with TLS.

15.6. Configuring a server for receiving remote logging information over UDP

The Rsyslog application enables you to configure a system to receive logging information from remote systems. To use remote logging through UDP, configure both the server and the client. The receiving server collects and analyzes the logs sent by one or more client systems. By default, rsyslog uses UDP on port 514 to receive log information from remote systems.

Follow this procedure to configure a server for collecting and analyzing logs sent by one or more client systems over the UDP protocol.

Prerequisites

  • Rsyslog is installed on the server system.
  • You are logged in as root on the server.
  • The policycoreutils-python-utils package is installed for the optional step using the semanage command.
  • The firewalld service is running.

Procedure

  1. Optional: To use a different port for rsyslog traffic than the default port 514:

    1. Add the syslogd_port_t SELinux type to the SELinux policy configuration, replacing portno with the port number you want rsyslog to use: 

      # semanage port -a -t syslogd_port_t -p udp portno

    2. Configure firewalld to allow incoming rsyslog traffic, replacing portno with the port number and zone with the zone you want rsyslog to use: 

      # firewall-cmd --zone=zone --permanent --add-port=portno/udp
success
# firewall-cmd --reload

    3. Reload the firewall rules: 

      # firewall-cmd --reload

  2. Create a new .conf file in the /etc/rsyslog.d/ directory, for example, remotelogserv.conf, and insert the following content: 

    # Define templates before the rules that use them
# Per-Host templates for remote systems
template(name="TmplAuthpriv" type="list") {
    constant(value="/var/log/remote/auth/")
    property(name="hostname")
    constant(value="/")
    property(name="programname" SecurePath="replace")
    constant(value=".log")
    }

template(name="TmplMsg" type="list") {
    constant(value="/var/log/remote/msg/")
    property(name="hostname")
    constant(value="/")
    property(name="programname" SecurePath="replace")
    constant(value=".log")
    }

# Provides UDP syslog reception
module(load="imudp")

# This ruleset processes remote messages
ruleset(name="remote1"){
     authpriv.*   action(type="omfile" DynaFile="TmplAuthpriv")
      *.info;mail.none;authpriv.none;cron.none
action(type="omfile" DynaFile="TmplMsg")
}

input(type="imudp" port="514" ruleset="remote1")

    Where 514 is the port number rsyslog uses by default. You can specify a different port instead.

  3. Verify the syntax of the /etc/rsyslog.conf file and all .conf files in the /etc/rsyslog.d/ directory: 

    # rsyslogd -N 1
rsyslogd: version 8.1911.0-2.el8, config validation run...

  4. Restart the rsyslog service. 

    # systemctl restart rsyslog

  5. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot: 

    # systemctl enable rsyslog

Additional resources

  • rsyslogd(8) , rsyslog.conf(5), semanage(8), and firewall-cmd(1) man pages.
  • Documentation installed with the rsyslog-doc package in the /usr/share/doc/rsyslog/html/index.html file.

15.7. Configuring remote logging to a server over UDP

Follow this procedure to configure a system for forwarding log messages to a server over the UDP protocol. The omfwd plug-in provides forwarding over UDP or TCP. The default protocol is UDP. Because the plug-in is built in, you do not have to load it.

Prerequisites

Procedure

  1. Create a new .conf file in the /etc/rsyslog.d/ directory, for example, 10-remotelogcli.conf, and insert the following content: 

    *.* action(type="omfwd"
      queue.type="linkedlist"
      queue.filename="example_fwd"
      action.resumeRetryCount="-1"
      queue.saveOnShutdown="on"
      target="example.com" port="portno" protocol="udp"
     )

    Where:

    • The queue.type="linkedlist" setting enables a LinkedList in-memory queue.
    • The queue.filename setting defines a disk storage. The backup files are created with the example_fwd prefix in the working directory specified by the preceding global workDirectory directive.
    • The action.resumeRetryCount -1 setting prevents rsyslog from dropping messages when retrying to connect if the server is not responding.
    • The enabled queue.saveOnShutdown="on" setting saves in-memory data if rsyslog shuts down.
    • The portno value is the port number you want rsyslog to use. The default value is 514.

    • The last line forwards all received messages to the logging server, port specification is optional.

      With this configuration, rsyslog sends messages to the server but keeps messages in memory if the remote server is not reachable. A file on disk is created only if rsyslog runs out of the configured memory queue space or needs to shut down, which benefits the system performance.

    Note

    Rsyslog processes configuration files /etc/rsyslog.d/ in the lexical order.

  2. Restart the rsyslog service. 

    # systemctl restart rsyslog

  3. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot: 

    # systemctl enable rsyslog

Verification

To verify that the client system sends messages to the server, follow these steps:

  1. On the client system, send a test message: 

    # logger test

  2. On the server system, view the /var/log/remote/msg/hostname/root.log log, for example: 

    # cat /var/log/remote/msg/hostname/root.log
Feb 25 03:53:17 hostname root[6064]: test

    Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Additional resources

  • rsyslogd(8) and rsyslog.conf(5) man pages.
  • Documentation installed with the rsyslog-doc package at /usr/share/doc/rsyslog/html/index.html.

15.8. Load balancing helper in Rsyslog

The RebindInterval setting specifies an interval at which the current connection is broken and is re-established. This setting applies to TCP, UDP, and RELP traffic. The load balancers perceive it as a new connection and forward the messages to another physical target system.

The RebindInterval setting proves to be helpful in scenarios when a target system has changed its IP address. The Rsyslog application caches the IP address when the connection establishes, therefore, the messages are sent to the same server. If the IP address changes, the UDP packets will be lost until the Rsyslog service restarts. Re-establishing the connection will ensure the IP to be resolved by DNS again. 

action(type=”omfwd” protocol=”tcp” RebindInterval=”250” target=”example.com” port=”514” …)

action(type=”omfwd” protocol=”udp” RebindInterval=”250” target=”example.com” port=”514” …)

action(type=”omrelp” RebindInterval=”250” target=”example.com” port=”6514” …)

15.9. Configuring reliable remote logging

With the Reliable Event Logging Protocol (RELP), you can send and receive syslog messages over TCP with a much reduced risk of message loss. RELP provides reliable delivery of event messages, which makes it useful in environments where message loss is not acceptable. To use RELP, configure the imrelp input module, which runs on the server and receives the logs, and the omrelp output module, which runs on the client and sends logs to the logging server.

Prerequisites

  • You have installed the rsyslog, librelp, and rsyslog-relp packages on the server and the client systems.
  • The specified port is permitted in SELinux and open in the firewall.

Procedure

  1. Configure the client system for reliable remote logging:

    1. On the client system, create a new .conf file in the /etc/rsyslog.d/ directory named, for example, relpclient.conf, and insert the following content: 

      module(load="omrelp")
*.* action(type="omrelp" target="_target_IP_" port="_target_port_")

      Where:

      • target_IP is the IP address of the logging server.
      • target_port is the port of the logging server.
    2. Save the changes to the /etc/rsyslog.d/relpclient.conf file.

    3. Restart the rsyslog service. 

      # systemctl restart rsyslog

    4. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot: 

      # systemctl enable rsyslog

  2. Configure the server system for reliable remote logging:

    1. On the server system, create a new .conf file in the /etc/rsyslog.d/ directory named, for example, relpserv.conf, and insert the following content: 

      ruleset(name="relp"){
*.* action(type="omfile" file="_log_path_")
}


module(load="imrelp")
input(type="imrelp" port="_target_port_" ruleset="relp")

      Where:

      • log_path specifies the path for storing messages.
      • target_port is the port of the logging server. Use the same value as in the client configuration file.
    2. Save the changes to the /etc/rsyslog.d/relpserv.conf file.

    3. Restart the rsyslog service. 

      # systemctl restart rsyslog

    4. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot: 

      # systemctl enable rsyslog

Verification

To verify that the client system sends messages to the server, follow these steps:

  1. On the client system, send a test message: 

    # logger test

  2. On the server system, view the log at the specified log_path, for example: 

    # cat /var/log/remote/msg/hostname/root.log
Feb 25 03:53:17 hostname root[6064]: test

    Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Additional resources

  • rsyslogd(8) and rsyslog.conf(5) man pages.
  • Documentation installed with the rsyslog-doc package in the /usr/share/doc/rsyslog/html/index.html file.

15.10. Supported Rsyslog modules

To expand the functionality of the Rsyslog application, you can use specific modules. Modules provide additional inputs (Input Modules), outputs (Output Modules), and other functionalities. A module can also provide additional configuration directives that become available after you load the module.

You can list the input and output modules installed on your system by entering the following command: 

# ls /usr/lib64/rsyslog/{i,o}m*

You can view the list of all available rsyslog modules in the /usr/share/doc/rsyslog/html/configuration/modules/idx_output.html file after you install the rsyslog-doc package.

15.11. Configuring the netconsole service to log kernel messages to a remote host

When logging to disk or using a serial console is not possible, you can use the netconsole kernel module and the same-named service to log kernel messages over a network to a remote rsyslog service.

Prerequisites

  • A system log service, such as rsyslog is installed on the remote host.
  • The remote system log service is configured to receive incoming log entries from this host.

Procedure

  1. Install the netconsole-service package: 

    # dnf install netconsole-service

  2. Edit the /etc/sysconfig/netconsole file and set the SYSLOGADDR parameter to the IP address of the remote host: 

    # SYSLOGADDR=192.0.2.1

  3. Enable and start the netconsole service: 

    # systemctl enable --now netconsole

Verification steps

  • Display the /var/log/messages file on the remote system log server.

Additional resources

Configuring a remote logging solution

15.12. Additional resources