The rsyslogd daemon was unable to connect to the remote rsyslog system

Diagnostic Steps

System Information:

crash> sys | grep -e NODE -e RELEASE
    NODENAME:  *****
     RELEASE: 2.6.32-754.36.1.el6.x86_64

crash> sys -i | grep DMI_SYS_VENDOR: -A 1
         DMI_SYS_VENDOR: VMware, Inc.
       DMI_PRODUCT_NAME: VMware Virtual Platform

• There is a network connectivity issue with the remote rsyslog system (HOSTNAME: Example@server.com IP: 192.168.XX.XX).

crash> xportshow --pid 2160 -vv
-----PID=2160  COMM=rs:action 1 que   (8 threads)
 fd     file              socket
 --     ----              ------
  3  0xffff880c26d17080  0xffff880c2b31bc00  PF_FILE  SOCK_DGRAM   UNIX 
     +-----------------------------------------------------------------
     |      state          i_ino   Path
     +-----------------------------------------------------------------
     |sock  CLOSE          14679   /dev/log
     +-----------------------------------------------------------------
  4  0xffff880c2b77aec0  0xffff880c29148b80  PF_INET  SOCK_DGRAM   UDP  
      udp   127.x.x.1:xxx             0.0.0.0:*                  st=7
  6  0xffff88078662a680  0xffff880823937700  PF_INET  SOCK_STREAM  TCP  
      tcp   10.x.xx.12:47306        192.xxx.xx.48:689           SYN_SENT  <<<- - -

• The network packet is also re-transmitted.

crash> xportshow --retrans -vv --port 514
------------------------------------------------------------------------------
<struct tcp_sock 0xffff880c1ee8d080>        TCP
tcp   10.x.xx.12:47306        192.168.xx.xx:514           SYN_SENT
    windows: rcv=14600, snd=0  advmss=1460 rcv_ws=9 snd_ws=0
        --- Emulating __tcp_select_window ---
          rcv_mss=88 free_space=65535 allowed_space=65535 full_space=65535
          rcv_ssthresh=14600, so free_space->14600 
          rcv_wscale=9
          window is not changed
    nonagle=0 sack_ok=0 tstamp_ok=0
    rmem_alloc=0, wmem_alloc=1
    rx_queue=0, tx_queue=1
    rcvbuf=87380, sndbuf=16384
    rcv_tstamp=n/a, lsndtime=n/a,  RTO=32000 ms
 **  Write Queue (skbuff, data length)
                 <struct sk_buff 0xffff880a293b39c0> 0
                 <struct skb_shared_info 0xffff880c277fb540>
    -- Retransmissions --
       retransmits=5, ca_state=TCP_CA_Disorder, 42.72 s since first retransmission

• The 1g limit of the rsyslogd spool space has reached its upper limit on the rsyslog client system (Example@server.com3786/192.168.XX.XX).

From: /etc/rsyslog.d/client.conf

# An "In-Memory Queue" is created for remote logging.
$WorkDirectory /var/lib/rsyslog # where to place spool files                  <<< - - - 
$ActionQueueFileName queue      # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g     # spool space limit (use as much as possible) <<< - - -
$ActionQueueSaveOnShutdown on   # save messages to disk on shutdown
$ActionQueueType LinkedList     # run asynchronously
$ActionResumeRetryCount -1      # infinety retries if host is down

crash> crashinfo --ls=/var/lib/rsyslog | grep -v /var/lib/rsyslog | grep -v Directory | head
 -rw-------     0     0    1049019 2022-10-14 06:32 queue.00000001
 -rw-------     0     0    1048827 2022-10-14 06:52 queue.00000002
 -rw-------     0     0    1049230 2022-10-14 07:10 queue.00000003
 -rw-------     0     0    1049158 2022-10-14 07:32 queue.00000004
 -rw-------     0     0    1048926 2022-10-14 07:55 queue.00000005
 -rw-------     0     0    1049225 2022-10-14 08:17 queue.00000006
 -rw-------     0     0    1049042 2022-10-14 08:38 queue.00000007
 -rw-------     0     0    1048949 2022-10-14 09:02 queue.00000008
 -rw-------     0     0    1049199 2022-10-14 09:22 queue.00000009
 -rw-------     0     0    1049092 2022-10-14 09:47 queue.00000010

crash> crashinfo --ls=/var/lib/rsyslog | grep -v /var/lib/rsyslog | grep -v Directory | grep -v 0.10s | wc -l
2062

crash> crashinfo --ls=/var/lib/rsyslog | grep -v /var/lib/rsyslog | grep -v Directory | grep -v 0.10s | awk '{ total += $4} END { printf " Size of the rsyslog spool files: %.02f MiB\n", total/2^20 }'
Size of the rsyslog spool files: 1083.70 MiB

• The disk usage of the mount point "/var/log" has reached 100% on the rsyslog client system (serverName/10.xx.xx.161).

crash> mount | grep -e NAME -e log
    VFSMOUNT         SUPERBLK     TYPE   DEVNAME   DIRNAME
ffff880c27488bc0 ffff880c25ad5c00 ext4   /dev/mapper/vg_redhat6x8664-logs /var/log  

crash> dmshow --pvs | grep -e NAME -e  logs
PV NAME                                         PV's MAPPED_DEVICE      PV STATE        PV SIZE (MiB)  VG NAME                                   LV NAME
sdb                                             [PV not DM dev]         online                2048.00  vg_redhat6x8664                           logs