snmpd issue - could not create socket

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 5.7
  • net-snmp-5.3.2.2-14.el5

Issue

  • I have two questios about the snmpd issues which occurred at our customer's site.

    • Problem 1:
      • Following snmpd messages were logged heavily. What is the cause of this issue?
    Oct 15 00:00:17 server51 snmpd[17250]: could not create socket 
Oct 15 00:00:17 server51 snmpd[17250]: could not open /proc/net/if_inet6

    • Problem 2:

      • The customer reloaded snmpd service at 00:00, Oct 16 to restore to normal situation. Just after that, however, snmpd crashed with segfault.

      • It looks that this issue is related to 'problem .1' from the messages. Perhaps it will recover without any problems if restarting snmpd service instead of reloading snmpd. Is there other way (except restarting snmpd) to recover if 'problem .1' occurs?

      • /var/log/messages-20131016

      Oct 16 00:00:15 server51 snmpd[17250]: Reconfiguring daemon  <= reload snmpd
      • /var/log/messages-20131017
      Oct 16 00:00:15 server51 syslogd 1.4.1: restart (remote reception).
Oct 16 00:00:15 server51 snmpd[17250]: NET-SNMP version 5.3.2.2 restarted
Oct 16 00:00:15 server51 snmpd[17250]: getaddrinfo: localhost No address associated with hostname
Oct 16 00:00:15 server51 snmpd[17250]: snmpd: create_trap_session: Unknown host (udp:localhost:162) (Too many open files)
Oct 16 00:00:15 server51 snmpd[17250]: /etc/snmp/snmpd.conf: line 107: Error: cannot create trap2sink: localhost
Oct 16 00:00:15 server51 snmpd[17250]: /etc/snmp/snmpd.conf: line 327: Error: Can't open /etc/mtab (setmntent)
Oct 16 00:00:15 server51 kernel: snmpd[17250]: segfault at 0000000000000000 rip 00002ae5db2b8a4b rsp 00007fff1d7509c0 error 4

Resolution

  • Modify -- /etc/sysconfig/snmpd.options
 # OPTIONS="(..snip..)"
  ulimit -n 4096    <-------------------- add

Root Cause

  • The RHEL per user maximum open fd limit is 1024. It's hitting this limit. Increseing the open fd limit should solve this problem.
  • It's not a BUG . just hitting maximum number fd limit .

but , if you use net-snmp-5.3.2.2-9.el5 or less , you need to update net-snmp to net-snmp-5.3.2.2-14.el5 or more . because we have the following issue

https://access.redhat.com/errata/RHBA-2010:0422

* snmpd, the SNMP daemon, could have leaked file descriptors periodically when
retrieving data for RMON-MIB. This leak could have exhausted file descriptors
for the snmpd process and rendered the daemon unable to respond to SNMP
requests. With this update, all unnecessary file descriptors are appropriately
closed so that snmpd thus no longer leaks file descriptors. (BZ#585202)

version history

5.3.2.2-25.el5_11
5.3.2.2-22.el5_10.1
5.3.2.2-20.el5
5.3.2.2-17.el5_8.1
5.3.2.2-17.el5
5.3.2.2-14.el5_7.1
5.3.2.2-14.el5
5.3.2.2-9.el5_5.1
5.3.2.2-9.el5
5.3.2.2-7.el5_4.2
5.3.2.2-7.el5
5.3.2.2-5.el5_3.2
5.3.2.2-5.el5_3.1
5.3.2.2-5.el5
5.3.1-24.el5_2.2
5.3.1-24.el5_2.1
5.3.1-24.el5
5.3.1-19.el5_1.4
5.3.1-19.el5_1.3
5.3.1-19.el5_1.2
5.3.1-19.el5_1.1
5.3.1-19.el5
5.3.1-14.0.1.el5
5.3.1-14.el5

#ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 62679
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024 <================================ FD limit
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 62679
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Diagnostic Steps

  • snmpd logs too many open files
Oct 16 00:00:15 server51 snmpd[17250]: getaddrinfo: localhost No address associated with hostname
Oct 16 00:00:15 server51 snmpd[17250]: snmpd: create_trap_session: Unknown host (udp:localhost:162) (Too many open files)
Oct 16 00:00:15 server51 snmpd[17250]: /etc/snmp/snmpd.conf: line 107: Error: cannot create trap2sink: localhost
Oct 16 00:00:15 server51 snmpd[17250]: /etc/snmp/snmpd.conf: line 327: Error: Can't open /etc/mtab (setmntent)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments