How to setup a cluster node to send SNMP traps for clustering on RHEL 6

Issue

• Introduction
• Environment
• What SNMP traps does Foghorn provide?
• Configure the SNMP Clients
  • Start the Services at Boot Time for the SNMP Client
  • Start the Services for the SNMP Client
• Configure the SNMPTrapd Server
  • Start the Services at Boot Time for the SNMPTrapd Server
  • Start the Services for the SNMPTrapd Server
• Verify that the SNMP Traps are Caught
• What happens when the SNMP traps are Caught?

Introduction

High Availability clustering has had SNMP MIBs since Red Hat Enterprise Linux(RHEL) 5, but the SNMP traps that were sent did not cover fencing, cluster services state change status, etc. In RHEL 6.1 a new package was introduced called foghorn which will send those traps. The foghorn service is a daemon that listens for specific D-Bus signals and translates those signals into SNMPv2 traps. Currently, foghorn listens for exactly five D-Bus signals and generates five SNMPv2 traps. The foghorn service requires messagebus (dbus-daemon) service and snmpd service to be running. Foghorn will require a snmp server listening for traps.

Ideally you do not want it to be another cluster node that you are monitoring, but another host that will catch the traps that is not apart of the cluster.This article covers sending and receiving the SNMP traps that are in the package cluster-snmp and foghorn.

For information about SNMP and foghorn see the articles below:

• 10.2. Configuring SNMP with the Red Hat High Availability Add-On
• How do I configure an SNMP server on Red Hat Enterprise Linux?
• How do I monitor my Red Hat Enterprise Linux 4, 5 and 6 Clustering with net-snmp?

This article will setup an snmptrapd server(snmptrap server) that will catch the traps that are sent to the server. If you are already have a preconfigured snmptrapd server or a network management device that collects the SNMP traps then the snmptrapd service is not needed.

Environment

• Red Hat Enterprise Linux Server 6.1 (with the High Availability Add on)

What SNMP traps does Foghorn provide?

Foghorn adds SNMP traps that are not apart of the cluster-snmp package. It adds trap for fenced, rgmanager, and corosync.

Configure the SNMP Clients

In this article I will try and document all the steps that need to be performed to correct setup a server that will listen for certain traps and write the trap to a text file. The cluster nodes that will be used are in this 3 node cluster are: rh6node1.examplerh.com, rh6node2.examplerh.com, rh6node3,examplerh.com. The host snmptrapserver.examplerh.com will be snmp server that will catch the traps from all the cluster nodes in the cluster.

Install all the packages that are required. Do the following on all the nodes in the cluster:

# yum -y install foghorn cluster-snmp net-snmp net-snmp-utils

Execute the following command on all the cluster nodes so that SNMP traps will be generated for corosync:

# echo "OPTIONS=\"-d\" " > /etc/sysconfig/corosync-notifyd

The service foghorn uses AgentX to communicate with snmpd (the master agent). Edit the /etc/snmp/snmpd.conf file on all the nodes. Add the following to the bottom of the file:

###############################################################################                                                                                                     
# Cluster SNMP Traps                                                                                                                                                                
###############################################################################                                                                                                     
dlmod RedHatCluster    /usr/lib64/cluster-snmp/libClusterMonitorSnmp.so                                                                                                             
view    systemview    included    REDHAT-CLUSTER-MIB:redhatCluster                                                                                                                  

###############################################################################                                                                                                     
# Foghorn                                                                                                                                                                           
###############################################################################                                                                                                     
master agentx                                                                                                                                                                       
trap2sink snmptrapserver.examplerh.com 

Start the Services at Boot Time for the SNMP Client

If you want to be able to send SNMP traps after the machine has booted up then you will need to enable the services to start at boot time:

# chkconfig messagebus on
# chkconfig snmpd on
# chkconfig corosync-notifyd on
# chkconfig foghorn on

Start the Services for the SNMP Client

Start all the services in this particular order because each service relies on the previous service. The service cman needs to be started before corosync-notifyd will start.

# service messagebus restart
# service snmpd start
# service foghorn start
# service corosync-notifyd start

Configure the SNMPTrapd Server

The snmptrapd service is only needed if there is no other service that can catch the snmp traps. Some environments may already have a network management device or SNMP service that is listening for SNMP traps. The following describes how to install and configure a snmptrapd service which will catch the SNMP traps that are sent to the service.

Install all the packages that are required for the server to catch the snmp traps that are sent to it. The snmpd and foghorn services do not need to be started. The packages are installed because they contain the MIBs that are required so that the SNMP traps can be translate to human readable form:

# yum -y install foghorn cluster-snmp net-snmp

Edit the configuration file /etc/snmp/snmptrapd.conf to accept the traps. In this configuration we have configured a traphandle to do some action when this trap is encountered:

# Authorises traps with the specified community to trigger the types of processing listed.
authCommunity log,execute,net public
# Add a trap handler for fenced,rgmanager notification.
traphandle REDHAT-FENCE-MIB::fenceNotifyFenceNode /root/bin/fenced-FenceNotifyFenceNode.sh
traphandle REDHAT-RGMANAGER-MIB::rgmanagerServiceStateChange /root/bin/rgmanager-ServiceStateChange.sh      

Edit the /etc/sysconfig/snmptrapd configuration file so that /var/log/messages file does not contain all the SNMP traps. All traps will be logged to the /var/log/snmptrapd.log file:

# snmptrapd command line options
# OPTIONS="-Lsd -p /var/run/snmptrapd.pid"
OPTIONS="-Lf /var/log/snmptrapd.log -p /var/run/snmptrapd.pid"

Install the binaries that are attached to the document. There are 2 binaries that will be installed into the /root/bin directory:

# mkdir /root/bin
# tar jxvf snmptrapd-scripts.tar.bz2
# cp /root/snmptrapd-scripts/* /root/bin
# chmod 700 /root/bin/fenced-FenceNotifyFenceNode.sh
# chmod 700 /root/bin/rgmanager-ServiceStateChange.sh

Start the Services at Boot Time for the SNMPTrapd Server

If you want to be able to receive SNMP traps after the machine has booted up then you will need to enable the services to start at boot time and this should be enabled for the host that is listening for snmp traps. In our case that would be snmptrapserver.examplerh.com:

# chkconfig snmptrapd on

Start the Services for the SNMPTrapd Server

Only start the snmptrapd service on the snmp server, which in our case is snmptrapserver.examplerh.com:

# service snmptrapd start

Verify that the SNMP Traps are Caught

Use this command to simulate an SNMP trap being sent for a fence event. dbus will send the message which will be caught by foghorn that will turn the dbus message into an SNMP trap. The message is stating that rh6node3.examplerh.com(nodeID: 3) had an unsuccesfully(-1) fence. If the message is caught then a message should be sent to root's mailbox and if logging to a file is enabled then a log entry should be written to /var/log/snmptrapd-traps/fenced.log:

# dbus-send --system --type=signal / com.redhat.cluster.fence.FenceNode string:rh6node3.examplerh.com int32:3 int32:-1
# dbus-send --system --type=signal / com.redhat.cluster.rgmanager.ServiceStateChange  string:script1 string:started string:some_flags string:rhel6-2.examplerh.com string:rhel6-1.examplerh.com

Verify that snmpd is working correctly by getting the cluster name:

# snmpwalk -v 1 -c public localhost REDHAT-CLUSTER-MIB:rhcClusterName.0
REDHAT-CLUSTER-MIB::rhcClusterName.0 = STRING: "rh6nodesThree"

This command will send a regular SNMP trap and this SNMP trap should be logged to /var/log/snmptrapd.log:

# snmptrap -v 2c -c public localhost  '' .1.3.6.1.4.1.116.33960.0.10002 .1.3.6.1.4.1.116.33960.0.1000 s "TEST"

What happens when the SNMP traps are Caught?

All the SNMP traps will be logged to /var/log/snmptrapd.log instead of /var/log/messages. In our configuration we have defined a couple of trap handlers that will execute a binary when the SNMP traps are caught that we have defined.

• REDHAT-FENCE-MIB::fenceNotifyFenceNode: /root/bin/fenced-FenceNotifyFenceNode.sh
• REDHAT-RGMANAGER-MIB::rgmanagerServiceStateChange: /root/bin/rgmanager-ServiceStateChange.sh

When the binary is executed it will send a message to root@localhost and if logging to a file has not been disabled then it will write the SNMP trap to a log file in the directory /var/log/snmptrapd-traps/.

If you are being flooded with snmp traps then you maybe experiencing the following issue: Receiving "Connected" and "Disconnected" snmp traps from corosync too often on Red Hat Enterprise Linux.