Chapter 10. Alerts and Notifications

Alerts are current problems and critical conditions that occur in the system and notified to the user. The Grafana monitoring platform generates alerts based on severity levels.

You can configure alerts via SMTP and SNMP protocols. SMTP configuration will send email alerts to users that have email notifications enabled. SNMPv3 configuration will send SNMP trap alerts to the Alerts notifications drawer of the Web Administration environment.

10.1. Types of Alerts

The alerts triggered by the dashboard are classified in the following categories:

  • Status alerts : Alerts arising when a cluster resource undergoes a change of state. For example, Healthy to Unhealthy.
  • Utilization alerts: Alerts arising after a cluster resource exceed the set threshold and after it reverts to the normal state. For example, when the Host CPU utilization is breached, an alert is triggered notifying the user about the event.

10.2. List of Alerts

The list of Web Administration alerts are given in the tables below.

Status Alerts

Table 10.1. Status Alerts

AlertSystem Resource(s)

volume status

Volume and Cluster

volume state

Volume and Cluster

brick status

Volume, Host, and Cluster

peer status

Cluster

rebalance status

Volume and Cluster

Geo-Replication status

Cluster

quorum of volume lost

Volume and Cluster

quorum of volume regained

Volume and Cluster

svc connected

Cluster

svc disconnected

Cluster

minimum number of bricks not up in EC subvolume

Volume and Cluster

minimum number of bricks up in EC subvolume

Volume and Cluster

afr quorum met for subvolume

Volume and Cluster

afr quorum fail for subvolume

Volume and Cluster

afr subvolume up

Volume and Cluster

afr subvolume down

Volume and Cluster

Utilization Alerts

Table 10.2. Utilization Alerts

AlertSystem Resource

cpu utilization

Host

memory utilization

host

swap utilization

host

volume utilization

Volume and Cluster

brick utilization

Volume and Cluster

10.3. Alerts Notifications Drawer

Alerts drawer is a notification indicator embedded in the Web Administration interface to display the system wide alerts.

Accessing Alerts Drawer

  1. To access the Alerts drawer, log in the Web Administration interface. In the default landing interface, locate and click on the interactive bell icon on the header bar at the top right-hand side.

    headerbar
  2. The drawer is opened displaying the number of alerts generated.
alert drw

To filter alerts, click on the status icon at the right.

alerts filter

10.4. SMTP Notifications Configuration

Tendrl-ansible installs and configures tendrl-notifier. After the tendrl-notifier file is configured, configure SMTP email notifications:

  1. Open the /etc/tendrl/notifier/email.conf.yaml file.

  2. Update the parameters: 

    email_id = <The sender email id>

email_smtp_server = <The smtp server>

email_smtp_port = <The smtp port>

  3. If the SMTP server supports only authenticated email, follow the template in the /etc/tendrl/notifier/email_auth.conf.yaml file and accordingly enable the following: 

        auth = <ssl/tls>

    email_pass = <password corresponding to email_id for authenticating to smtp server>

  4. Restart the tendrl-notifier service: 

    systemctl restart tendrl-notifier

10.5. SNMPv3 Notification Configuration

Configure SNMP

Tendrl-ansible installs and configures tendrl-notifier. After the tendrl-notifier file is configured, configure SNMPv3 trap notifications:

  1. Open the tendrl-notifier configuration file: 

    # cat /etc/tendrl/notifier/snmp.conf.yaml

  2. Update the parameters in the file for v3 trap alerts: 

    For v3_endpoint:
# For more hosts you can add more entry with endpoint2, endpoint3, etc
endpoint1:

# Name or IP address of the remote SNMP host.
        host_ip: <Receiving machine ip>
# Name of the user on the host that connects to the agent.
        username: <Username of receiver>
# Enables the agent to receive packets from the host.
        auth_key: <md5 password>
# The private user password
        priv_key: <des password>


# For v2_endpoint:

# For more hosts you can add more entry with endpoint2, endpoint3, etc

endpoint1:
# Name or IP address of the remote SNMP host.
     host_ip: <Receiving machine ip>
     community: <community name>

  3. Restart the tendrl-notifier service: 

    systemctl restart tendrl-notifier