rhel 5.7: snmpd segfaults, syslog/logrotate config
Hi
I have 8 rhel 5.7 servers where snmpd segfaults once a week.
It might be caused by a combination of my custzomizations and /etc/logrotate.d/snmpd
Errors look like this:
snmpd[21870]: segfault at 0000000000010202 rip 0000000000010202 rsp 00007fffe9f0f568 error 14
snmpd[31982]: segfault at 00002aaaae730d67 rip 00002aaaae730d67 rsp 00007fffc374dbe8 error 14
snmpd[1461]: segfault at 0000000000010202 rip 0000000000010202 rsp 00007fff67b5f798 error 14
I made some customizations to the snmp configuration because I did not want snmpd to flood /var/log/messages.
I changed the snmpd configuration to log to the local6 facility.
# cat /etc/sysconfig/snmpd.options
# snmpd command line options
# OPTIONS="-Lsd -Lf /dev/null -p /var/run/snmpd.pid -a"
# log to syslog local6 faciltiy, see /etc/syslog.conf
OPTIONS="-Ls6 -Lf /dev/null -p /var/run/snmpd.pid -a"
And I redirect snmp messages to /var/log/snmpd.log
# grep local6 /etc/syslog.conf
*.info;mail.none;news.none;authpriv.none;cron.none;local6.none /var/log/messages
local6.* /var/log/snmpd.log
Versions:
# rpm -q net-snmp
net-snmp-5.3.2.2-14.el5_7.1
# rpm -qf /etc/logrotate.d/snmpd
net-snmp-5.3.2.2-14.el5_7.1
Logrotate:
# cat /etc/logrotate.d/snmpd
/var/log/snmpd.log {
notifempty
missingok
postrotate
/bin/kill -HUP `cat /var/run/snmpd.pid 2> /dev/null` 2> /dev/null || true
endscript
}
/etc/logrotate.d/snmpd seams to be configured for snmpd native logging (without syslog).
I think the postrotate with syslog should be like this:
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
Anyway if I run
/bin/kill -HUP `cat /var/run/snmpd.pid 2> /dev/null` 2> /dev/null || true
manually, snmpd segfaults.
I changed /etc/logrotate.d/snmpd to:
/var/log/snmpd.log {
notifempty
missingok
postrotate
if /bin/grep -v '^#' /var/log/snmpd.log | /bin/grep /etc/syslog.conf > /dev/null
then
/bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
else
/bin/kill -HUP `cat /var/run/snmpd.pid 2> /dev/null` 2> /dev/null || true
fi
endscript
}
Can anyone confirm the segfault?
Should /etc/logrotate.d/snmpd be changed as described?
Regards
Kolja von Sawilski
Responses
Hello,
While I have not attempted to reproduce this issue myself, if the application is segfaulting then it would be best for us to get a bug report open and attempt to track down the problem. As far as I can tell, the changes you've made are fine and should be allowed, so even if they are the trigger for the issue its still something that Red Hat would be interested in fixing.
Would you be able to open a support case so we can gather some information from you in order to troubleshoot further? If you want to provide the following information when you open the case, it will help speed things up:
* A sosreport from the system in question
* An application core from when the segfault occurs
* A list of changes you've made to trigger the issue (basically what you've stated in this thread)
* Any commands or steps necessary to reproduce the issue
Thanks,
John Ruemker, RHCA
Red Hat Software Maintenance Engineer
Online User Groups Moderator
I'm seeing the same thing on a bunch of my RHEL 5.7 servers.
I'm also NOT seeing it on a bunch of my other RHEL 5.7 servers - and they're servers NOT using hp-snmp-agents.
Still investigating whether this is related to the hp stuff Are you using hp-snmp-agents?
Apr 28 11:34:39 clunky_server_name kernel: snmpd[18777]: segfault at 00002aaaae525d67 rip 00002aaaae525d67 rsp 00007fff0c1abbd8 error 14
There have been a few crash fixes in the latest net-snmp in RHEL-5:
https://rhn.redhat.com/network/software/packages/change_log.pxt?pid=692755
You might want to try the latest package and then file a report if you continue seeing crashes.
Siddhesh
Software Maintenance Engineer
Red Hat
FWIW:
As per case CASE:4640078035 opened with HP, snmpd segfaulting on HUP signals (with "dlmod cmaX /usr/lib64/libcmaX64.so" entered in /etc/snmp/snmpd.conf via hp-snmp-agents) is an issue "with hp-snmp-agents". They're planning on fixing this, but do not have an ETA.
They also pointed out the fact that /etc/logrotate.d/snmpd changed (somewhere between RHEL 5.3 & RHEL 5.7) from:
/sbin/service snmpd condrestart 2> /dev/null > /dev/null || true
to:
/bin/kill -HUP `cat /var/run/snmpd.pid 2> /dev/null` 2> /dev/null || true
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
