Rsyslog Client Stops Forwarding Logs to Remote Rsyslog Server Over TCP

Issue

• The rsyslog service intermittently stops forwarding logs to the configured remote syslog server over TCP.

• On affected systems, especially those running Red Hat Enterprise Linux 9, the following log entries may appear in the system journal (journalctl -u rsyslog):

  Jun 26 01:50:04 hostname rsyslogd[33148]: omfwd: [wrkr 0/140686041015872] we had a generic or IO error with the remote server. The actual error message should already have been provided.  Server is 10.a.b.c.. This can be caused by the remote server or an interim system like a load balancer or firewall. Rsyslog will re-open the connection if configured to do so. [v8.2412.0-1.el9 try https://www.rsyslog.com/e/2027 ]
  Jun 26 01:50:04 hostname rsyslogd[33148]: omfwd: [wrkr 0] target 10.a.b.c:514 became unavailable during buffer flush. Remaining messages will be sent when it is online again. [v8.2412.0-1.el9 try https://www.rsyslog.com/e/2007 ]
  Jun 26 01:50:04 hostname rsyslogd[33148]: omfwd: [wrkr 0/140686041015872] no working target servers in pool available, suspending action [v8.2412.0-1.el9 try https://www.rsyslog.com/e/2007 ]
  Jun 26 01:50:04 hostname rsyslogd[33148]: action 'action-8-builtin:omfwd' suspended (module 'builtin:omfwd'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2412.0-1.el9 try https://www.rsyslog.com/e/2007 ]
  

Note : Depending on the Red Hat Enterprise Linux version, we might see only a subset of above logs.