How to configure failover for rsyslog in Red Hat Enterprise Linux 6?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 6

Issue

  • How to configure a failover rsyslog environment in Red Hat Enterprise Linux

Resolution

Note that code below is for rsyslog client, not rsyslog server

Rsyslog has the capability to work with failover servers to prevent messages loss if primary rsyslog went down. A pre-requisite for that is TCP based syslog forwarding . The reason is that with UDP there is no reliable way to detect the remote system has gone away.

1) Configure TCP based rsyslog server. Steps on how to configure TCP based rsyslog server are available in following kbase article

How to configure system to accept remote log messages in Red Hat Enterprise Linux 6?

2) After Configuring tcp based rsyslog forwarding to primary server.  Use '$ActionExecOnlyWhenPreviousIsSuspended on' directive to to automatically switch destination rsyslog server if primary server fails.

*.info;mail.none;authpriv.none;cron.none                @@<Primary rsyslog server>
$ActionExecOnlyWhenPreviousIsSuspended on
&@@<Address of secondary rsyslog server >

Example of Failover Syslog Configuration on rsyslog Client machine

Let's assume you have a primary(target) and secondary (backup) central servers. Then, you can use the following config file excerpt to send data to them:

/etc/rsyslog.conf

*.info;mail.none;authpriv.none;cron.none                @@192.168.1.2
$ActionExecOnlyWhenPreviousIsSuspended on 
&@@192.168.1.3       (& is used to have more than one action for)
& /var/log/localbuffer (forward to localbuffer )
$ActionExecOnlyWhenPreviousIsSuspended off (to re-set it for the next selector )

*192.168.1.2 is a primaray rsyslog server
*192.168.1.3 is a secondary rsyslog server

This selector processes all messages it receives (*.info;mail.none;authpriv.none;cron.none  ). It tries to forward every message to 192.168.1.2

(via TCP). If it can not reach that server,it tries 192.168.1.3 ,and If neither of these servers can be connected, the data is stored in /var/log/localbuffer.

Please note that the secondary and the local log buffer are only used if  the one before them does not work. So ideally, /var/log/localbuffer will never receive a message. If one of the servers resumes operation, it automatically takes over processing again.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.