rhel5 troubleshooting packet loss with rsyslog remote logging

Issue

• We have a bare metal system which is serving as the loghost for all PCI compliance systems.

• We did a packet trace against a staged push of log entries and were able to account for all of the packets - however not all of the packets appeared in the log file after being received by rsyslogd.

• It is suspected that rsyslogd may not be able to keep up with the volume of packets that this host receives.

• probably 50+ clients send syslog data to this loghost on a daily basis.

• In the previous occurrence of this test, they compared the number of entries sent by the script, the number of UDP syslog packets sent, and the number of log entries written by rsyslogd and found that all of the packets were accounted for in tcpdump, but packets were missing in the rsyslogd logfiles.