rhel 5.7: snmpd segfaults, syslog/logrotate config

Latest response

Hi

 

I have 8 rhel 5.7 servers where snmpd segfaults once a week.

It might be caused by a combination of my custzomizations and /etc/logrotate.d/snmpd

 

Errors look like this:

snmpd[21870]: segfault at 0000000000010202 rip 0000000000010202 rsp 00007fffe9f0f568 error 14
snmpd[31982]: segfault at 00002aaaae730d67 rip 00002aaaae730d67 rsp 00007fffc374dbe8 error 14
snmpd[1461]: segfault at 0000000000010202 rip 0000000000010202 rsp 00007fff67b5f798 error 14

 

 

I made some customizations to the snmp configuration because I did not want snmpd to flood /var/log/messages.

 

I changed the snmpd configuration to log to the local6 facility.

# cat /etc/sysconfig/snmpd.options

 

# snmpd command line options
# OPTIONS="-Lsd -Lf /dev/null -p /var/run/snmpd.pid -a"
# log to syslog local6 faciltiy, see /etc/syslog.conf
OPTIONS="-Ls6 -Lf /dev/null -p /var/run/snmpd.pid -a"

 

 

And I redirect snmp messages to /var/log/snmpd.log

 

# grep local6 /etc/syslog.conf

*.info;mail.none;news.none;authpriv.none;cron.none;local6.none          /var/log/messages

local6.*                                                                                             /var/log/snmpd.log

 

 

 

Versions:

# rpm -q net-snmp
net-snmp-5.3.2.2-14.el5_7.1

# rpm -qf /etc/logrotate.d/snmpd
net-snmp-5.3.2.2-14.el5_7.1

 

Logrotate:

# cat /etc/logrotate.d/snmpd

 

/var/log/snmpd.log {
    notifempty
    missingok
    postrotate
            /bin/kill -HUP `cat /var/run/snmpd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

 

/etc/logrotate.d/snmpd seams to be configured for snmpd native logging (without syslog).

I think the postrotate with syslog should be like this:

        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true

 

Anyway if I run

/bin/kill -HUP `cat /var/run/snmpd.pid 2> /dev/null` 2> /dev/null || true

manually, snmpd segfaults.

 

I changed /etc/logrotate.d/snmpd to:

/var/log/snmpd.log {
    notifempty
    missingok
    postrotate
        if /bin/grep -v '^#' /var/log/snmpd.log | /bin/grep /etc/syslog.conf > /dev/null
        then
            /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
        else
            /bin/kill -HUP `cat /var/run/snmpd.pid 2> /dev/null` 2> /dev/null || true
        fi
    endscript
}
 

Can anyone confirm the segfault?

Should /etc/logrotate.d/snmpd be changed as described?

 

 

 

Regards

Kolja von Sawilski
 

Responses

Hello,

While I have not attempted to reproduce this issue myself, if the application is segfaulting then it would be best for us to get a bug report open and attempt to track down the problem.  As far as I can tell, the changes you've made are fine and should be allowed, so even if they are the trigger for the issue its still something that Red Hat would be interested in fixing.

Would you be able to open a support case so we can gather some information from you in order to troubleshoot further?  If you want to provide the following information when you open the case, it will help speed things up:

 

* A sosreport from the system in question

* An application core from when the segfault occurs

* A list of changes you've made to trigger the issue (basically what you've stated in this thread)

* Any commands or steps necessary to reproduce the issue

 

Thanks,

John Ruemker, RHCA

Red Hat Software Maintenance Engineer

Online User Groups Moderator

I'm seeing the same thing on a bunch of my RHEL 5.7 servers.

I'm also NOT seeing it on a bunch of my other RHEL 5.7 servers -  and they're servers NOT using hp-snmp-agents.

Still investigating whether this is related to the hp stuff  Are you using hp-snmp-agents?

 

Apr 28 11:34:39 clunky_server_name kernel: snmpd[18777]: segfault at 00002aaaae525d67 rip 00002aaaae525d67 rsp 00007fff0c1abbd8 error 14

There have been a few crash fixes in the latest net-snmp in RHEL-5:

 

https://rhn.redhat.com/network/software/packages/change_log.pxt?pid=692755

 

You might want to try the latest package and then file a report if you continue seeing crashes.

 

Siddhesh

Software Maintenance Engineer

Red Hat

FWIW:

As per case CASE:4640078035 opened with HP, snmpd segfaulting on HUP signals (with "dlmod cmaX /usr/lib64/libcmaX64.so" entered in /etc/snmp/snmpd.conf via hp-snmp-agents) is an issue "with hp-snmp-agents".  They're planning on fixing this, but do not have an ETA. 

 

They also pointed out the fact that /etc/logrotate.d/snmpd changed (somewhere between RHEL 5.3 & RHEL 5.7) from:

/sbin/service snmpd condrestart 2> /dev/null > /dev/null || true

to:

 /bin/kill -HUP `cat /var/run/snmpd.pid 2> /dev/null` 2> /dev/null || true