22.4. Working with Queues in Rsyslog

Queues are used to pass content, mostly syslog messages, between components of rsyslog. With queues, rsyslog is capable of processing multiple messages simultaneously and to apply several actions to a single message at once. The data flow inside rsyslog can be illustrated as follows:
Message Flow in Rsyslog

Figure 22.1. Message Flow in Rsyslog

Whenever rsyslog receives a message, it passes this message to the preprocessor and then places it into the main message queue. Messages wait there to be dequeued and passed to the rule processor.
The rule processor is a parsing and filtering engine. Here, the rules defined in /etc/rsyslog.conf are applied. Based on these rules, the rule processor evaluates which actions are to be performed. Each action has its own action queue. Messages are passed through this queue to the respective action processor which creates the final output. Note that at this point, several actions can run simultaneously on one message. For this purpose, a message is duplicated and passed to multiple action processors.
Only one queue per action is possible. Depending on configuration, the messages can be sent right to the action processor without action queuing. This is the behavior of direct queues (see below). In case the output action fails, the action processor notifies the action queue, which then takes an unprocessed element back and after some time interval, the action is attempted again.
To sum up, there are two positions where queues stand in rsyslog: either in front of the rule processor as a single main message queue or in front of various types of output actions as action queues. Queues provide two main advantages that both lead to increased performance of message processing:
  • they serve as buffers that decouple producers and consumers in the structure of rsyslog
  • they allow for parallelization of actions performed on messages
Apart from this, queues can be configured with several directives to provide optimal performance for your system. These configuration options are covered in the following sections.

Warning

If an output plug-in is unable to deliver a message, it is stored in the preceding message queue. If the queue fills, the inputs block until it is no longer full. This will prevent new messages from being logged via the blocked queue. In the absence of separate action queues this can have severe consequences, such as preventing SSH logging, which in turn can prevent SSH access. Therefore it is advised to use dedicated action queues for outputs which are forwarded over a network or to a database.

22.4.1. Defining Queues

Based on where the messages are stored, there are several types of queues: direct, in-memory, disk, and disk-assisted in-memory queues that are most widely used. You can choose one of these types for the main message queue and also for action queues. Add the following into /etc/rsyslog.conf: 
object(queue.type=”queue_type”)
By adding this you can apply the setting for:
  • main message queue: replace object with main_queue
  • an action queue: replace object with action
  • ruleset: replace object with ruleset
Replace queue_type with one of direct, linkedlist or fixedarray (which are in-memory queues), or disk.
The default setting for a main message queue is the FixedArray queue with a limit of 10,000 messages. Action queues are by default set as Direct queues.

Direct Queues

For many simple operations, such as when writing output to a local file, building a queue in front of an action is not needed. To avoid queuing, use: 
object(queue.type=”Direct”)
Replace object with main_queue, action or ruleset to use this option to the main message queue, an action queue or for the ruleset respectively. With direct queue, messages are passed directly and immediately from the producer to the consumer.

Disk Queues

Disk queues store messages strictly on a hard drive, which makes them highly reliable but also the slowest of all possible queuing modes. This mode can be used to prevent the loss of highly important log data. However, disk queues are not recommended in most use cases. To set a disk queue, type the following into /etc/rsyslog.conf: 
object(queue.type=”Disk”)
Replace object with main_queue, action or ruleset to use this option to the main message queue, an action queue or for the ruleset respectively. The default size of a queue can be modified with the following configuration directive: 
object(queue.size=”size”)
where size represents the specified size of disk queue part. The defined size limit is not restrictive, rsyslog always writes one complete queue entry, even if it violates the size limit. Each part of a disk queue matches with an individual file. The naming directive for these files looks as follows: 
object(queue.filename=”name”)
This sets a name prefix for the file followed by a 7-digit number starting at one and incremented for each file. 
object(queue.maxfilesize=”size”)
Disk queues are written in parts, with a default size 1 MB. Specify size to use a different value.

In-memory Queues

With in-memory queue, the enqueued messages are held in memory which makes the process very fast. The queued data is lost if the computer is power cycled or shut down. However, you can use the action (queue.saveonshutdown=”on”) setting to save the data before shutdown. There are two types of in-memory queues:
  • FixedArray queue — the default mode for the main message queue, with a limit of 10,000 elements. This type of queue uses a fixed, pre-allocated array that holds pointers to queue elements. Due to these pointers, even if the queue is empty a certain amount of memory is consumed. However, FixedArray offers the best run time performance and is optimal when you expect a relatively low number of queued messages and high performance.
  • LinkedList queue — here, all structures are dynamically allocated in a linked list, thus the memory is allocated only when needed. LinkedList queues handle occasional message bursts very well.
In general, use LinkedList queues when in doubt. Compared to FixedArray, it consumes less memory and lowers the processing overhead.
Use the following syntax to configure in-memory queues: 
object(queue.type=”LinkedList”)
object(queue.type=”FixedArray”)
Replace object with main_queue, action or ruleset to use this option to the main message queue, an action queue or for the ruleset respectively.

Disk-Assisted In-memory Queues

Both disk and in-memory queues have their advantages and rsyslog lets you combine them in disk-assisted in-memory queues. To do so, configure a normal in-memory queue and then add the queue.filename=”file_name” directive to its block to define a file name for disk assistance. This queue then becomes disk-assisted, which means it couples an in-memory queue with a disk queue to work in tandem.
The disk queue is activated if the in-memory queue is full or needs to persist after shutdown. With a disk-assisted queue, you can set both disk-specific and in-memory specific configuration parameters. This type of queue is probably the most commonly used, it is especially useful for potentially long-running and unreliable actions.
To specify the functioning of a disk-assisted in-memory queue, use the so-called watermarks: 
object(queue.highwatermark=”number”)
object(queue.lowwatermark=”number”)
Replace object with main_queue, action or ruleset to use this option to the main message queue, an action queue or for the ruleset respectively. Replace number with a number of enqueued messages. When an in-memory queue reaches the number defined by the high watermark, it starts writing messages to disk and continues until the in-memory queue size drops to the number defined with the low watermark. Correctly set watermarks minimize unnecessary disk writes, but also leave memory space for message bursts since writing to disk files is rather lengthy. Therefore, the high watermark must be lower than the whole queue capacity set with queue.size. The difference between the high watermark and the overall queue size is a spare memory buffer reserved for message bursts. On the other hand, setting the high watermark too low will turn on disk assistance unnecessarily often.

Example 22.12. Reliable Forwarding of Log Messages to a Server

Rsyslog is often used to maintain a centralized logging system, where log messages are forwarded to a server over the network. To avoid message loss when the server is not available, it is advisable to configure an action queue for the forwarding action. This way, messages that failed to be sent are stored locally until the server is reachable again. Note that such queues are not configurable for connections using the UDP protocol.

Procedure 22.1. Forwarding To a Single Server

Suppose the task is to forward log messages from the system to a server with host name example.com, and to configure an action queue to buffer the messages in case of a server outage. To do so, perform the following steps:
  • Use the following configuration in /etc/rsyslog.conf or create a file with the following content in the /etc/rsyslog.d/ directory: 
    *.* action(type=”omfwd”
queue.type=”LinkedList”
queue.filename=”example_fwd”
action.resumeRetryCount="-1"
queue.saveonshutdown="on"
arget="example.com" Port="6514" Protocol="tcp")
    Where:
    • queue.type enables a LinkedList in-memory queue,
    • queue.filename defines a disk storage, in this case the backup files are created in the /var/lib/rsyslog/ directory with the example_fwd prefix,
    • the action.resumeRetryCount= “-1” setting prevents rsyslog from dropping messages when retrying to connect if server is not responding,
    • enabled queue.saveonshutdown saves in-memory data if rsyslog shuts down,
    • the last line forwards all received messages to the logging server using reliable TCP delivery, port specification is optional.
    With the above configuration, rsyslog keeps messages in memory if the remote server is not reachable. A file on disk is created only if rsyslog runs out of the configured memory queue space or needs to shut down, which benefits the system performance.

Procedure 22.2. Forwarding To Multiple Servers

The process of forwarding log messages to multiple servers is similar to the previous procedure:
  • Each destination server requires a separate forwarding rule, action queue specification, and backup file on disk. For example, use the following configuration in /etc/rsyslog.conf or create a file with the following content in the /etc/rsyslog.d/ directory: 
    *.* action(type=”omfwd”
        queue.type=”LinkedList”
        queue.filename=”example_fwd1”
        action.resumeRetryCount="-1"
        queue.saveonshutdown="on"
        Target="example1.com" Protocol="tcp")
*.* action(type=”omfwd”
        queue.type=”LinkedList”
        queue.filename=”example_fwd2”
        action.resumeRetryCount="-1"
        queue.saveonshutdown="on"
        Target="example2.com" Protocol="tcp")

22.4.2. Creating a New Directory for rsyslog Log Files

Rsyslog runs as the syslogd daemon and is managed by SELinux. Therefore all files to which rsyslog is required to write to, must have the appropriate SELinux file context.

Procedure 22.3. Creating a New Working Directory

  1. If required to use a different directory to store working files, create a directory as follows: 
    ~]# mkdir /rsyslog
  2. Install utilities to manage SELinux policy: 
    ~]# yum install policycoreutils-python
  3. Set the SELinux directory context type to be the same as the /var/lib/rsyslog/ directory: 
    ~]# semanage fcontext -a -t syslogd_var_lib_t /rsyslog
  4. Apply the SELinux context: 
    ~]# restorecon -R -v /rsyslog
restorecon reset /rsyslog context unconfined_u:object_r:default_t:s0->unconfined_u:object_r:syslogd_var_lib_t:s0
  5. If required, check the SELinux context as follows: 
    ~]# ls -Zd /rsyslog
drwxr-xr-x. root root system_u:object_r:syslogd_var_lib_t:s0   /rsyslog
  6. Create subdirectories as required. For example: 
    ~]# mkdir /rsyslog/work/
    The subdirectories will be created with the same SELinux context as the parent directory.
  7. Add the following line in /etc/rsyslog.conf immediately before it is required to take effect: 
    global(workDirectory=”/rsyslog/work”)
    This setting will remain in effect until the next WorkDirectory directive is encountered while parsing the configuration files.

22.4.3. Managing Queues

All types of queues can be further configured to match your requirements. You can use several directives to modify both action queues and the main message queue. Currently, there are more than 20 queue parameters available, see the section called “Online Documentation”. Some of these settings are used commonly, others, such as worker thread management, provide closer control over the queue behavior and are reserved for advanced users. With advanced settings, you can optimize rsyslog's performance, schedule queuing, or modify the behavior of a queue on system shutdown.

Limiting Queue Size

You can limit the number of messages that queue can contain with the following setting: 
object(queue.highwatermark=”number”)
Replace object with main_queue, action or ruleset to use this option to the main message queue, an action queue or for the ruleset respectively. Replace number with a number of enqueued messages. You can set the queue size only as the number of messages, not as their actual memory size. The default queue size is 10,000 messages for the main message queue and ruleset queues, and 1000 for action queues.
Disk assisted queues are unlimited by default and cannot be restricted with this directive, but you can reserve them physical disk space in bytes with the following settings: 
object(queue.maxdiskspace=”number”)
Replace object with main_queue, action or ruleset. When the size limit specified by number is hit, messages are discarded until sufficient amount of space is freed by dequeued messages.

Discarding Messages

When a queue reaches a certain number of messages, you can discard less important messages in order to save space in the queue for entries of higher priority. The threshold that launches the discarding process can be set with the so-called discard mark: 
object(queue.discardmark=”number”)
Replace object with MainMsg or with Action to use this option to the main message queue or for an action queue respectively. Here, number stands for a number of messages that have to be in the queue to start the discarding process. To define which messages to discard, use: 
object(queue.discardseverity=”number”)
Replace number with one of the following numbers for respective priorities: 7 (debug), 6 (info), 5 (notice), 4 (warning), 3 (err), 2 (crit), 1 (alert), or 0 (emerg). With this setting, both newly incoming and already queued messages with lower than defined priority are erased from the queue immediately after the discard mark is reached.

Using Timeframes

You can configure rsyslog to process queues during a specific time period. With this option you can, for example, transfer some processing into off-peak hours. To define a time frame, use the following syntax: 
object(queue.dequeuetimebegin=”hour”)
object(queue.dequeuetimeend=”hour”)
With hour you can specify hours that bound your time frame. Use the 24-hour format without minutes.

Configuring Worker Threads

A worker thread performs a specified action on the enqueued message. For example, in the main message queue, a worker task is to apply filter logic to each incoming message and enqueue them to the relevant action queues. When a message arrives, a worker thread is started automatically. When the number of messages reaches a certain number, another worker thread is turned on. To specify this number, use: 
object(queue.workerthreadminimummessages=”number”)
Replace number with a number of messages that will trigger a supplemental worker thread. For example, with number set to 100, a new worker thread is started when more than 100 messages arrive. When more than 200 messages arrive, the third worker thread starts and so on. However, too many working threads running in parallel becomes ineffective, so you can limit the maximum number of them by using: 
object(queue.workerthreads=”number”)
where number stands for a maximum number of working threads that can run in parallel. For the main message queue, the default limit is 1 thread. Once a working thread has been started, it keeps running until an inactivity timeout appears. To set the length of timeout, type: 
object(queue.timeoutworkerthreadshutdown=”time”)
Replace time with the duration set in milliseconds. Specifies time without new messages after which the worker thread will be closed. Default setting is one minute.

Batch Dequeuing

To increase performance, you can configure rsyslog to dequeue multiple messages at once. To set the upper limit for such dequeueing, use: 
$object(queue.DequeueBatchSize= ”number”)
Replace number with the maximum number of messages that can be dequeued at once. Note that a higher setting combined with a higher number of permitted working threads results in greater memory consumption.

Terminating Queues

When terminating a queue that still contains messages, you can try to minimize the data loss by specifying a time interval for worker threads to finish the queue processing: 
object(queue.timeoutshutdown=”time”)
Specify time in milliseconds. If after that period there are still some enqueued messages, workers finish the current data element and then terminate. Unprocessed messages are therefore lost. Another time interval can be set for workers to finish the final element: 
object(queue.timeoutactioncompletion=”time”)
In case this timeout expires, any remaining workers are shut down. To save data at shutdown, use: 
object(queue.saveonshutdown=”on”)
If set, all queue elements are saved to disk before rsyslog terminates.

22.4.4. Using the New Syntax for rsyslog queues

In the new syntax available in rsyslog 7, queues are defined inside the action() object that can be used both separately or inside a ruleset in /etc/rsyslog.conf. The format of an action queue is as follows: 
action(type="action_type "queue.size="queue_size" queue.type="queue_type" queue.filename="file_name"
Replace action_type with the name of the module that is to perform the action and replace queue_size with a maximum number of messages the queue can contain. For queue_type, choose disk or select from one of the in-memory queues: direct, linkedlist or fixedarray. For file_name specify only a file name, not a path. Note that if creating a new directory to hold log files, the SELinux context must be set. See Section 22.4.2, “Creating a New Directory for rsyslog Log Files” for an example.

Example 22.13. Defining an Action Queue

To configure the output action with an asynchronous linked-list based action queue which can hold a maximum of 10,000 messages, enter a command as follows: 
action(type="omfile" queue.size="10000" queue.type="linkedlist" queue.filename="logfile")
The rsyslog 7 syntax for a direct action queues is as follows: 
*.* action(type="omfile" file="/var/lib/rsyslog/log_file
     )
The rsyslog 7 syntax for an action queue with multiple parameters can be written as follows: 
*.* action(type="omfile"
              queue.filename="log_file"
              queue.type="linkedlist"
              queue.size="10000"
     )
The default work directory, or the last work directory to be set, will be used. If required to use a different work directory, add a line as follows before the action queue: 
global(workDirectory="/directory")

Example 22.14. Forwarding To a Single Server Using the New Syntax

The following example is based on the procedure Procedure 22.1, “Forwarding To a Single Server” in order to show the difference between the traditional sysntax and the rsyslog 7 syntax. The omfwd plug-in is used to provide forwarding over UDP or TCP. The default is UDP. As the plug-in is built in it does not have to be loaded.
Use the following configuration in /etc/rsyslog.conf or create a file with the following content in the /etc/rsyslog.d/ directory: 
*.* action(type="omfwd"
      queue.type="linkedlist"
      queue.filename="example_fwd"
      action.resumeRetryCount="-1"
      queue.saveOnShutdown="on"
      target="example.com" port="6514" protocol="tcp"
     )
Where:
  • queue.type="linkedlist" enables a LinkedList in-memory queue,
  • queue.filename defines a disk storage. The backup files are created with the example_fwd prefix, in the working directory specified by the preceding global workDirectory directive,
  • the action.resumeRetryCount -1 setting prevents rsyslog from dropping messages when retrying to connect if server is not responding,
  • enabled queue.saveOnShutdown="on" saves in-memory data if rsyslog shuts down,
  • the last line forwards all received messages to the logging server, port specification is optional.