Chapter 13. Configuration of SNMP traps

As a storage administrator, you can deploy and configure the simple network management protocol (SNMP) gateway in a Red Hat Ceph Storage cluster to receive alerts from the Prometheus Alertmanager and route them as SNMP traps to the cluster.

13.1. Simple network management protocol

Simple network management protocol (SNMP) is one of the most widely used open protocols, to monitor distributed systems and devices across a variety of hardware and software platforms. Ceph’s SNMP integration focuses on forwarding alerts from its Prometheus Alertmanager cluster to a gateway daemon. The gateway daemon transforms the alert into an SNMP Notification and sends it on to a designated SNMP management platform. The gateway daemon is from the snmp_notifier_project, which provides SNMP V2c and V3 support with authentication and encryption.

The Red Hat Ceph Storage SNMP gateway service deploys one instance of the gateway by default. You can increase this by providing placement information. However, if you enable multiple SNMP gateway daemons, your SNMP management platform receives multiple notifications for the same event.

The SNMP traps are alert messages and the Prometheus Alertmanager sends these alerts to the SNMP notifier which then looks for object identifier (OID) in the given alerts’ labels. Each SNMP trap has a unique ID which allows it to send additional traps with updated status to a given SNMP poller. SNMP hooks into the Ceph health checks so that every health warning generates a specific SNMP trap.

In order to work correctly and transfer information on device status to the user to monitor, SNMP relies on several components. There are four main components that makeup SNMP:

  • SNMP Manager- The SNMP manager, also called a management station, is a computer that runs network monitoring platforms. A platform that has the job of polling SNMP-enabled devices and retrieving data from them. An SNMP Manager queries agents, receives responses from agents and acknowledges asynchronous events from agents.
  • SNMP Agent - An SNMP agent is a program that runs on a system to be managed and contains the MIB database for the system. These collect data like bandwidth and disk space, aggregates it, and sends it to the management information base (MIB).
  • Management information base (MIB) - These are components contained within the SNMP agents. The SNMP manager uses this as a database and asks the agent for access to particular information. This information is needed for the network management systems (NMS). The NMS polls the agent to take information from these files and then proceeds to translate it into graphs and displays that can be viewed by the user. MIBs contain statistical and control values that are determined by the network device.
  • SNMP Devices

The following versions of SNMP are compatible and supported for gateway implementation:

  • V2c - Uses an community string without any authentication and is vulnerable to outside attacks.
  • V3 authNoPriv - Uses the username and password authentication without encryption.
  • V3 authPriv - Uses the username and password authentication with encryption to the SNMP management platform.
Important

When using SNMP traps, ensure that you have the correct security configuration for your version number to minimize the vulnerabilities that are inherent to SNMP and keep your network protected from unauthorized users.

13.2. Configuring snmptrapd

It is important to configure the simple network management protocol (SNMP) target before deploying the snmp-gateway because the snmptrapd daemon contains the auth settings that you need to specify when creating the snmp-gateway service.

The SNMP gateway feature provides a means of exposing the alerts that are generated in the Prometheus stack to an SNMP management platform. You can configure the SNMP traps to the destination based on the snmptrapd tool. This tool allows you to establish one or more SNMP trap listeners.

The following parameters are important for configuration:

  • The engine-id is a unique identifier for the device, in hex, and required for SNMPV3 gateway. Red Hat recommends to use `8000C53F_CLUSTER_FSID_WITHOUT_DASHES_`for this parameter.
  • The snmp-community, which is the SNMP_COMMUNITY_FOR_SNMPV2 parameter, is public for SNMPV2c gateway.
  • The auth-protocol which is the AUTH_PROTOCOL, is mandatory for SNMPV3 gateway and is SHA by default.
  • The privacy-protocol, which is the PRIVACY_PROTOCOL, is mandatory for SNMPV3 gateway.
  • The PRIVACY_PASSWORD is mandatory for SNMPV3 gateway with encryption.
  • The SNMP_V3_AUTH_USER_NAME is the user name and is mandatory for SNMPV3 gateway.
  • The SNMP_V3_AUTH_PASSWORD is the password and is mandatory for SNMPV3 gateway.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to the nodes.
  • Install firewalld on Red Hat Enterprise Linux system.

Procedure

  1. On the SNMP management host, install the SNMP packages:

    Example

    [root@host01 ~]# dnf install -y net-snmp-utils net-snmp

  2. Open the port 162 for SNMP to receive alerts:

    Example

    [root@host01 ~]# firewall-cmd --zone=public --add-port=162/udp
[root@host01 ~]# firewall-cmd --zone=public --add-port=162/udp --permanent

  3. Implement the management information base (MIB) to make sense of the SNMP notification and enhance SNMP support on the destination host. Copy the raw file from the main repository: https://github.com/ceph/ceph/blob/master/monitoring/snmp/CEPH-MIB.txt

    Example

    [root@host01 ~]# curl -o CEPH_MIB.txt -L https://raw.githubusercontent.com/ceph/ceph/master/monitoring/snmp/CEPH-MIB.txt
[root@host01 ~]# scp CEPH_MIB.txt root@host02:/usr/share/snmp/mibs

  4. Create the snmptrapd directory.

    Example

    [root@host01 ~]# mkdir /root/snmptrapd/

  5. Create the configuration files in snmptrapd directory for each protocol based on the SNMP version:

    Syntax

    format2 %V\n% Agent Address: %A \n Agent Hostname: %B \n Date: %H - %J - %K - %L - %M - %Y \n Enterprise OID: %N \n Trap Type: %W \n Trap Sub-Type: %q \n Community/Infosec Context: %P \n Uptime: %T \n Description: %W \n PDU Attribute/Value Pair Array:\n%v \n -------------- \n
createuser -e 0x_ENGINE_ID_ SNMPV3_AUTH_USER_NAME AUTH_PROTOCOL SNMP_V3_AUTH_PASSWORD PRIVACY_PROTOCOL PRIVACY_PASSWORD
authuser log,execute SNMP_V3_AUTH_USER_NAME
authCommunity   log,execute,net SNMP_COMMUNITY_FOR_SNMPV2

    • For SNMPV2c, create the snmptrapd_public.conf file as follows:

      Example

      format2 %V\n% Agent Address: %A \n Agent Hostname: %B \n Date: %H - %J - %K - %L - %M - %Y \n Enterprise OID: %N \n Trap Type: %W \n Trap Sub-Type: %q \n Community/Infosec Context: %P \n Uptime: %T \n Description: %W \n PDU Attribute/Value Pair Array:\n%v \n -------------- \n

authCommunity   log,execute,net public

      The public setting here must match the snmp_community setting used when deploying the snmp-gateway service.

    • For SNMPV3 with authentication only, create the snmptrapd_auth.conf file as follows:

      Example

      format2 %V\n% Agent Address: %A \n Agent Hostname: %B \n Date: %H - %J - %K - %L - %M - %Y \n Enterprise OID: %N \n Trap Type: %W \n Trap Sub-Type: %q \n Community/Infosec Context: %P \n Uptime: %T \n Description: %W \n PDU Attribute/Value Pair Array:\n%v \n -------------- \n
createuser -e 0x8000C53Ff64f341c655d11eb8778fa163e914bcc myuser SHA mypassword
authuser log,execute myuser

      The 0x8000C53Ff64f341c655d11eb8778fa163e914bcc string is the engine_id, and myuser and mypassword are the credentials. The password security is defined by the SHA algorithm.

      This corresponds to the settings for deploying the snmp-gateway daemon.

      Example

      snmp_v3_auth_username: myuser
snmp_v3_auth_password: mypassword

    • For SNMPV3 with authentication and encryption, create the snmptrapd_authpriv.conf file as follows:

      Example

      format2 %V\n% Agent Address: %A \n Agent Hostname: %B \n Date: %H - %J - %K - %L - %M - %Y \n Enterprise OID: %N \n Trap Type: %W \n Trap Sub-Type: %q \n Community/Infosec Context: %P \n Uptime: %T \n Description: %W \n PDU Attribute/Value Pair Array:\n%v \n -------------- \n
createuser -e 0x8000C53Ff64f341c655d11eb8778fa163e914bcc myuser SHA mypassword DES mysecret
authuser log,execute myuser

      The 0x8000C53Ff64f341c655d11eb8778fa163e914bcc string is the engine_id, and myuser and mypassword are the credentials. The password security is defined by the SHA algorithm and DES is the type of privacy encryption.

      This corresponds to the settings for deploying the snmp-gateway daemon.

      Example

      snmp_v3_auth_username: myuser
snmp_v3_auth_password: mypassword
snmp_v3_priv_password: mysecret

  6. Run the daemon on the SNMP management host:

    Syntax

    /usr/sbin/snmptrapd -M /usr/share/snmp/mibs -m CEPH-MIB.txt -f -C -c /root/snmptrapd/CONFIGURATION_FILE -Of -Lo :162

    Example

    [root@host01 ~]# /usr/sbin/snmptrapd -M /usr/share/snmp/mibs -m CEPH-MIB.txt -f -C -c /root/snmptrapd/snmptrapd_auth.conf -Of -Lo :162

  7. If any alert is triggered on the storage cluster, you can monitor the output on the SNMP management host. Verify the SNMP traps and also the traps decoded by MIB.

    Example

    NET-SNMP version 5.8
 Agent Address: 0.0.0.0
 Agent Hostname: <UNKNOWN>
 Date: 15 - 5 - 12 - 8 - 10 - 4461391
 Enterprise OID: .
 Trap Type: Cold Start
 Trap Sub-Type: 0
 Community/Infosec Context: TRAP2, SNMP v3, user myuser, context
 Uptime: 0
 Description: Cold Start
 PDU Attribute/Value Pair Array:
.iso.org.dod.internet.mgmt.mib-2.1.3.0 = Timeticks: (292276100) 3 days, 19:52:41.00
.iso.org.dod.internet.snmpV2.snmpModules.1.1.4.1.0 = OID: .iso.org.dod.internet.private.enterprises.ceph.cephCluster.cephNotifications.prometheus.promMgr.promMgrPrometheusInactive
.iso.org.dod.internet.private.enterprises.ceph.cephCluster.cephNotifications.prometheus.promMgr.promMgrPrometheusInactive.1 = STRING: "1.3.6.1.4.1.50495.1.2.1.6.2[alertname=CephMgrPrometheusModuleInactive]"
.iso.org.dod.internet.private.enterprises.ceph.cephCluster.cephNotifications.prometheus.promMgr.promMgrPrometheusInactive.2 = STRING: "critical"
.iso.org.dod.internet.private.enterprises.ceph.cephCluster.cephNotifications.prometheus.promMgr.promMgrPrometheusInactive.3 = STRING: "Status: critical
- Alert: CephMgrPrometheusModuleInactive
  Summary: Ceph's mgr/prometheus module is not available
  Description: The mgr/prometheus module at 10.70.39.243:9283 is unreachable. This could mean that the module has been disabled or the mgr itself is down.
Without the mgr/prometheus module metrics and alerts will no longer function. Open a shell to ceph and use 'ceph -s' to determine whether the mgr is active. If the mgr is not active, restart it, otherwise you can check the mgr/prometheus module is loaded with 'ceph mgr module ls'  and if it's not listed as enabled, enable it with 'ceph mgr module enable prometheus'"

    In the above example, an alert is generated after the Prometheus module is disabled.

Additional Resources

13.3. Deploying the SNMP gateway

You can deploy the simple network management protocol (SNMP) gateway using either SNMPV2c or SNMPV3. There are two methods to deploy the SNMP gateway:

  1. By creating a credentials file.
  2. By creating one service configuration yaml file with all the details.

You can use the following parameters to deploy the SNMP gateway based on the versions:

  • The service_type is the snmp-gateway.
  • The service_name is any user-defined string.
  • The count is the number of SNMP gateways to be deployed in a storage cluster.
  • The snmp_destination parameter must be of the format hostname:port.
  • The engine-id is a unique identifier for the device, in hex, and required for SNMPV3 gateway. Red Hat recommends to use `8000C53F_CLUSTER_FSID_WITHOUT_DASHES_`for this parameter.
  • The snmp_community parameter is public for SNMPV2c gateway.
  • The auth-protocol is mandatory for SNMPV3 gateway and is SHA by default.
  • The privacy-protocol is mandatory for SNMPV3 gateway with authentication and encryption.
  • The port is 9464 by default.
  • You must provide a -i FILENAME to pass the secrets and passwords to the orchestrator.

Once the SNMP gateway service is deployed or updated, the Prometheus Alertmanager configuration is automatically updated to forward any alert that has an objectidentifier to the SNMP gateway daemon for further processing.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to the nodes.
  • Configuring snmptrapd on the destination host, which is the SNMP management host.

Procedure

  1. Log into the Cephadm shell:

    Example

    [root@host01 ~]# cephadm shell

  2. Create a label for the host on which SNMP gateway needs to be deployed:

    Syntax

    ceph orch host label add HOSTNAME snmp-gateway

    Example

    [ceph: root@host01 /]# ceph orch host label add host02 snmp-gateway

  3. Create a credentials file or a service configuration file based on the SNMP version:

    • For SNMPV2c, create the file as follows:

      Example

      [ceph: root@host01 /]# cat snmp_creds.yml

snmp_community: public

      OR

      Example

      [ceph: root@host01 /]# cat snmp-gateway.yml

service_type: snmp-gateway
service_name: snmp-gateway
placement:
  count: 1
spec:
  credentials:
    snmp_community: public
  port: 9464
  snmp_destination: 192.168.122.73:162
  snmp_version: V2c

    • For SNMPV3 with authentication only, create the file as follows:

      Example

      [ceph: root@host01 /]# cat snmp_creds.yml

snmp_v3_auth_username: myuser
snmp_v3_auth_password: mypassword

      OR

      Example

      [ceph: root@host01 /]# cat snmp-gateway.yml

service_type: snmp-gateway
service_name: snmp-gateway
placement:
  count: 1
spec:
  credentials:
    snmp_v3_auth_password: mypassword
    snmp_v3_auth_username: myuser
  engine_id: 8000C53Ff64f341c655d11eb8778fa163e914bcc
  port: 9464
  snmp_destination: 192.168.122.1:162
  snmp_version: V3

    • For SNMPV3 with authentication and encryption, create the file as follows:

      Example

      [ceph: root@host01 /]# cat snmp_creds.yml

snmp_v3_auth_username: myuser
snmp_v3_auth_password: mypassword
snmp_v3_priv_password: mysecret

      OR

      Example

      [ceph: root@host01 /]# cat snmp-gateway.yml

service_type: snmp-gateway
service_name: snmp-gateway
placement:
  count: 1
spec:
  credentials:
    snmp_v3_auth_password: mypassword
    snmp_v3_auth_username: myuser
    snmp_v3_priv_password: mysecret
  engine_id: 8000C53Ff64f341c655d11eb8778fa163e914bcc
  port: 9464
  snmp_destination: 192.168.122.1:162
  snmp_version: V3

  4. Run the ceph orch command:

    Syntax

    ceph orch apply snmp-gateway --snmp_version=V2c_OR_V3 --destination=SNMP_DESTINATION [--port=PORT_NUMBER]\
[--engine-id=8000C53F_CLUSTER_FSID_WITHOUT_DASHES_] [--auth-protocol=MDS_OR_SHA] [--privacy_protocol=DES_OR_AES] -i FILENAME

    OR

    Syntax

    ceph orch apply -i FILENAME.yml

    • For SNMPV2c, with the snmp_creds file, run the ceph orch command with the snmp-version as V2c:

      Example

      [ceph: root@host01 /]# ceph orch apply snmp-gateway --snmp-version=V2c --destination=192.168.122.73:162 --port=9464 -i snmp_creds.yml

    • For SNMPV3 with authentication only, with the snmp_creds file, run the ceph orch command with the snmp-version as V3 and engine-id:

      Example

      [ceph: root@host01 /]# ceph orch apply snmp-gateway --snmp-version=V3 --engine-id=8000C53Ff64f341c655d11eb8778fa163e914bcc--destination=192.168.122.73:162 -i snmp_creds.yml

    • For SNMPV3 with authentication and encryption, with the snmp_creds file, run the ceph orch command with the snmp-version as V3, privacy-protocol, and engine-id:

      Example

      [ceph: root@host01 /]# ceph orch apply snmp-gateway --snmp-version=V3 --engine-id=8000C53Ff64f341c655d11eb8778fa163e914bcc--destination=192.168.122.73:162 --privacy-protocol=AES -i snmp_creds.yml

      OR

    • For all the SNMP versions, with the snmp-gateway file, run the following command:

      Example

      [ceph: root@host01 /]# ceph orch apply -i snmp-gateway.yml

Additional Resources