Chapter 8. Configuring a remote logging solution

To ensure that logs from various machines in your environment are recorded centrally on a logging server, you can configure the Rsyslog application to record logs that fit specific criteria from the client system to the server.

8.1. The Rsyslog logging service

The Rsyslog application, in combination with the systemd-journald service, provides local and remote logging support in Red Hat Enterprise Linux. The rsyslogd daemon continuously reads syslog messages received by the systemd-journald service from the journal. rsyslogd then filters and processes these syslog events and records them to rsyslog log files or forwards them to other services according to its configuration.

The rsyslogd daemon also provides extended filtering, encryption protected relaying of messages, input and output modules, and support for transportation using the TCP and UDP protocols.

In /etc/rsyslog.conf, which is the main configuration file for rsyslog, you can specify the rules according to which rsyslogd handles the messages. Generally, you can classify messages by their source and topic (facility) and urgency (priority), and then assign an action that should be performed when a message fits these criteria.

In /etc/rsyslog.conf, you can also see a list of log files maintained by rsyslogd. Most log files are located in the /var/log/ directory. Some applications, such as httpd and samba, store their log files in a subdirectory within /var/log/.

Additional resources

8.2. Installing Rsyslog documentation

The Rsyslog application has extensive documentation that is available at https://www.rsyslog.com/doc/, but you can also install the rsyslog-doc documentation package locally by following this procedure.

Prerequisites

  • You have activated the AppStream repository on your system
  • You are authorized to install new packages using sudo

Procedure

  • Install the rsyslog-doc package:

    $ sudo yum install rsyslog-doc

Verification

8.3. Configuring remote logging over TCP

The Rsyslog application enables you to both run a logging server and configure individual systems to send their log files to the logging server. To use remote logging through TCP, configure both the server and the client. The server collects and analyzes the logs sent by one or more client systems.

With the Rsyslog application, you can maintain a centralized logging system where log messages are forwarded to a server over the network. To avoid message loss when the server is not available, you can configure an action queue for the forwarding action. This way, messages that failed to be sent are stored locally until the server is reachable again. Note that such queues cannot be configured for connections using the UDP protocol.

The omfwd plug-in provides forwarding over UDP or TCP. The default protocol is UDP. Because the plug-in is built in, it does not have to be loaded.

8.3.1. Configuring a server for remote logging over TCP

Follow this procedure to configure a server for collecting and analyzing logs sent by one or more client systems.

By default, rsyslog uses TCP on port 514.

Prerequisites

  • rsyslog is installed on the server system
  • You are logged in as root on the server

Procedure

  1. Optional: To use a different port for rsyslog traffic, add the syslogd_port_t SELinux type to port. For example, enable port 30514:

    # semanage port -a -t syslogd_port_t -p tcp 30514
  2. Optional: To use a different port for rsyslog traffic, configure firewalld to allow incoming rsyslog traffic on that port. For example, allow TCP traffic on port 30514 in zone zone:

    # firewall-cmd --zone=zone --permanent --add-port=30514/tcp
    success
  3. Create a new file in the /etc/rsyslog.d/ directory named, for example, remotelog.conf, and insert the following content:

    # Define templates before the rules that use them
    ### Per-Host Templates for Remote Systems ###
    template(name="TmplAuthpriv" type="list") {
        constant(value="/var/log/remote/auth/")
        property(name="hostname")
        constant(value="/")
        property(name="programname" SecurePath="replace")
        constant(value=".log")
        }
    
    template(name="TmplMsg" type="list") {
        constant(value="/var/log/remote/msg/")
        property(name="hostname")
        constant(value="/")
        property(name="programname" SecurePath="replace")
        constant(value=".log")
        }
    
    # Provides TCP syslog reception
    module(load="imtcp")
    # Adding this ruleset to process remote messages
    ruleset(name="remote1"){
         authpriv.*   action(type="omfile" DynaFile="TmplAuthpriv")
          *.info;mail.none;authpriv.none;cron.none
    action(type="omfile" DynaFile="TmplMsg")
    }
    
    input(type="imtcp" port="30514" ruleset="remote1")
  4. Save the changes to the /etc/rsyslog.d/remotelog.conf file.
  5. Make sure the rsyslog service is running and enabled on the logging server:

    # systemctl status rsyslog
  6. Restart the rsyslog service.

    # systemctl restart rsyslog
  7. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

    # systemctl enable rsyslog

Your log server is now configured to receive and store log files from the other systems in your environment.

Verification

  • Test the syntax of the /etc/rsyslog.conf file:

    # rsyslogd -N 1
    rsyslogd: version 8.1911.0-2.el8, config validation run (level 1), master config /etc/rsyslog.conf
    rsyslogd: End of config validation run. Bye.

Additional resources

8.3.2. Configuring remote logging to a server over TCP

Follow this procedure to configure a system for forwarding log messages to a server over the TCP protocol. The omfwd plug-in provides forwarding over UDP or TCP. The default protocol is UDP. Because the plug-in is built in, you do not have to load it.

Prerequisites

  • The rsyslog package is installed on the client systems that should report to the server.
  • You have configured the server for remote logging.
  • The specified port is permitted in SELinux and open in firewall.

Procedure

  1. Create a new file in the /etc/rsyslog.d/ directory named, for example, remotelog.conf, and insert the following content:

    *.* action(type="omfwd"
          queue.type="linkedlist"
          queue.filename="example_fwd"
          action.resumeRetryCount="-1"
          queue.saveOnShutdown="on"
          target="example.com" port="30514" protocol="tcp"
         )

    Where:

    • queue.type="linkedlist" enables a LinkedList in-memory queue,
    • queue.filename defines a disk storage. The backup files are created with the example_fwd prefix in the working directory specified by the preceding global workDirectory directive,
    • the action.resumeRetryCount -1 setting prevents rsyslog from dropping messages when retrying to connect if server is not responding,
    • enabled queue.saveOnShutdown="on" saves in-memory data if rsyslog shuts down,
    • the last line forwards all received messages to the logging server, port specification is optional.

    With this configuration, rsyslog sends messages to the server but keeps messages in memory if the remote server is not reachable. A file on disk is created only if rsyslog runs out of the configured memory queue space or needs to shut down, which benefits the system performance.

  2. Restart the rsyslog service.

    # systemctl restart rsyslog

Verification

To verify that the client system sends messages to the server, follow these steps:

  1. On the client system, send a test message:

    # logger test
  2. On the server system, view the /var/log/messages log, for example:

    # cat /var/log/remote/msg/hostname/root.log
    Feb 25 03:53:17 hostname root[6064]: test

    Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Additional resources

8.4. Configuring remote logging over UDP

The Rsyslog application enables you to configure a system to receive logging information from remote systems. To use remote logging through UDP, configure both the server and the client. The receiving server collects and analyzes the logs sent by one or more client systems. By default, rsyslog uses UDP on port 514 to receive log information from remote systems.

8.4.1. Configuring a server for receiving remote logging information over UDP

Follow this procedure to configure a server for collecting and analyzing logs sent by one or more client systems over the UDP protocol.

Prerequisites

  • The rsyslog utility is installed.

Procedure

  1. Optional: To use a different port for rsyslog traffic than the default port 514:

    1. Add the syslogd_port_t SELinux type to the SELinux policy configuration, replacing portno with the port number you want rsyslog to use:

      # semanage port -a -t syslogd_port_t -p udp portno
    2. Configure firewalld to allow incoming rsyslog traffic, replacing portno with the port number and zone with the zone you want rsyslog to use:

      # firewall-cmd --zone=zone --permanent --add-port=portno/udp
      success
    3. Reload the firewall rules:

      # firewall-cmd --reload
  2. Create a new .conf file in the /etc/rsyslog.d/ directory, for example, remotelogserv.conf, and insert the following content:

    # Define templates before the rules that use them
    ### Per-Host Templates for Remote Systems ###
    template(name="TmplAuthpriv" type="list") {
        constant(value="/var/log/remote/auth/")
        property(name="hostname")
        constant(value="/")
        property(name="programname" SecurePath="replace")
        constant(value=".log")
        }
    
    template(name="TmplMsg" type="list") {
        constant(value="/var/log/remote/msg/")
        property(name="hostname")
        constant(value="/")
        property(name="programname" SecurePath="replace")
        constant(value=".log")
        }
    
    # Provides UDP syslog reception
    module(load="imudp")
    
    # This ruleset processes remote messages
    ruleset(name="remote1"){
         authpriv.*   action(type="omfile" DynaFile="TmplAuthpriv")
          *.info;mail.none;authpriv.none;cron.none
    action(type="omfile" DynaFile="TmplMsg")
    }
    
    input(type="imudp" port="514" ruleset="remote1")

    Where 514 is the port number rsyslog uses by default. You can specify a different port instead.

  3. Restart the rsyslog service.

    # systemctl restart rsyslog
  4. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

    # systemctl enable rsyslog

Verification

  1. Verify the syntax of the /etc/rsyslog.conf file and all .conf files in the /etc/rsyslog.d/ directory:

    # rsyslogd -N 1
    rsyslogd: version 8.1911.0-2.el8, config validation run (level 1), master config /etc/rsyslog.conf
    rsyslogd: End of config validation run. Bye.

Additional resources

8.4.2. Configuring remote logging to a server over UDP

Follow this procedure to configure a system for forwarding log messages to a server over the UDP protocol. The omfwd plug-in provides forwarding over UDP or TCP. The default protocol is UDP. Because the plug-in is built in, you do not have to load it.

Prerequisites

Procedure

  1. Create a new .conf file in the /etc/rsyslog.d/ directory, for example, remotelogcli.conf, and insert the following content:

    *.* action(type="omfwd"
          queue.type="linkedlist"
          queue.filename="example_fwd"
          action.resumeRetryCount="-1"
          queue.saveOnShutdown="on"
          target="example.com" port="portno" protocol="udp"
         )

    Where:

    • queue.type="linkedlist" enables a LinkedList in-memory queue.
    • queue.filename defines a disk storage. The backup files are created with the example_fwd prefix in the working directory specified by the preceding global workDirectory directive.
    • The action.resumeRetryCount -1 setting prevents rsyslog from dropping messages when retrying to connect if the server is not responding.
    • enabled queue.saveOnShutdown="on" saves in-memory data if rsyslog shuts down.
    • portno is the port number you want rsyslog to use. The default value is 514.
    • The last line forwards all received messages to the logging server, port specification is optional.

      With this configuration, rsyslog sends messages to the server but keeps messages in memory if the remote server is not reachable. A file on disk is created only if rsyslog runs out of the configured memory queue space or needs to shut down, which benefits the system performance.

  2. Restart the rsyslog service.

    # systemctl restart rsyslog
  3. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

    # systemctl enable rsyslog

Verification

To verify that the client system sends messages to the server, follow these steps:

  1. On the client system, send a test message:

    # logger test
  2. On the server system, view the /var/log/remote/msg/hostname/root.log log, for example:

    # cat /var/log/remote/msg/hostname/root.log
    Feb 25 03:53:17 hostname root[6064]: test

    Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Additional resources

8.5. Configuring reliable remote logging

With the Reliable Event Logging Protocol (RELP), you can send and receive syslog messages over TCP with a much reduced risk of message loss. RELP provides reliable delivery of event messages, which makes it useful in environments where message loss is not acceptable. To use RELP, configure the imrelp input module, which runs on the server and receives the logs, and the omrelp output module, which runs on the client and sends logs to the logging server.

Prerequisites

  • You have installed the rsyslog, librelp, and rsyslog-relp packages on the server and the client systems.
  • The specified port is permitted in SELinux and open in the firewall.

Procedure

  1. Configure the client system for reliable remote logging:

    1. On the client system, create a new .conf file in the /etc/rsyslog.d/ directory named, for example, relpcli.conf, and insert the following content:

      module(load="omrelp")
      *.* action(type="omrelp" target="target_IP" port="target_port")

      Where:

      • target_IP is the IP address of the logging server.
      • target_port is the port of the logging server.
    2. Save the changes to the /etc/rsyslog.d/relpserv.conf file.
    3. Restart the rsyslog service.

      # systemctl restart rsyslog
    4. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog
  2. Configure the server system for reliable remote logging:

    1. On the server system, create a new .conf file in the /etc/rsyslog.d/ directory named, for example, relpserv.conf, and insert the following content:

      ruleset(name="relp"){
      *.* action(type="omfile" file="log_path")
      }
      
      
      module(load="imrelp")
      input(type="imrelp" port="target_port" ruleset="relp")

      Where:

      • log_path specifies the path for storing messages.
      • target_port is the port of the logging server. Use the same value as in the client configuration file.
    2. Save the changes to the /etc/rsyslog.d/relpserv.conf file.
    3. Restart the rsyslog service.

      # systemctl restart rsyslog
    4. Optional: If rsyslog is not enabled, ensure the rsyslog service starts automatically after reboot:

      # systemctl enable rsyslog

Verification

To verify that the client system sends messages to the server, follow these steps:

  1. On the client system, send a test message:

    # logger test
  2. On the server system, view the log at the specified log_path, for example:

    # cat /var/log/remote/msg/hostname/root.log
    Feb 25 03:53:17 hostname root[6064]: test

    Where hostname is the host name of the client system. Note that the log contains the user name of the user that entered the logger command, in this case root.

Additional resources

8.6. Supported Rsyslog modules

To expand the functionality of the Rsyslog utility, you can use specific additional modules. Modules provide additional inputs (Input Modules), outputs (Output Modules), and other specific functionalities. A module may also provide additional configuration directives that become available after you load that module.

List the input and output modules installed on your system with the following command:

# ls /usr/lib64/rsyslog/{i,o}m*

To view the list of all available rsyslog modules, open the following page from documentation installed from the rsyslog-doc package.

$ firefox file:///usr/share/doc/rsyslog/html/configuration/modules/idx_output.html

8.7. Additional resources