rhel6: Why rsyslog imptcp module generates hung TCP connections ?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 6
  • rsyslog

Issue

  • Rsyslog is configured to use TCP (imptcp) module
  • Hung rsyslog on established TCP connections between production servers and the central rsyslog server.

Resolution

The rsyslog7 package is available on RHEL6, it contains fixes for this issue.

Root Cause

  • A client is establishing a TCP channel to rsyslog
  • When the client now disappears without properly tearing down the connection (i.e. network interruption, or kill -9 <client>), then rsyslog keeps the incoming connection up, so is not detecting the dead tcp peer.

Newer versions of rsyslog contain fixes to detect this situation and tear down the TCP channel.

Diagnostic Steps

/etc/rsyslog.conf

# Provides TCP syslog reception                                                                                                                                                                                
$ModLoad imptcp
$InputTCPServerRun 514
# Provides TCP syslog reception                                                                                                                                                                                
$ModLoad imptcp
$InputTCPServerRun 514
$InputPTCPServerKeepAlive on  <=== Keep alive not supported

rsyslog debug log on versions including the patch, i.e. rsyslog7:

1309.441560958:7fb009c3d700: --------imuxsock calling select, active file descriptors (max 3): 3 
1312.047322939:7fb003fff700: imptcp: new connection on listen socket 6
1312.047370787:7fb003fff700: KEEPALIVE enabled for socket 9
1312.123092204:7fb003fff700: imptcp: added socket 9 to epoll[5] set
1312.123140824:7fb003fff700: imptcp going on epoll_wait

netstat:

tcp        0      0 a.b.c.f:514           r.e.w.d:55695         ESTABLISHED -                   keepalive (6781.40/0/0)

conf:

$ModLoad imptcp
$InputPTCPServerRun 514
$InputPTCPServerKeepalive on

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.