Chapter 6. Creating alerts in Datadog

Administrators can create monitors that track the metrics of the Red Hat Ceph Storage cluster and generate alerts. For example, if an OSD is down, Datadog can alert an administrator that one or more OSDs are down.

Prerequisites

  • Root-level access to the Ceph Monitor node.
  • Appropriate Ceph key providing access to the Red Hat Ceph Storage cluster.
  • Internet access.

Procedure

  1. Click Monitors to see an overview of the Datadog monitors.

    datadog manage monitors
  2. To create a monitor, select Monitors→New Monitor.
  3. Select the detection method. For example, "Threshold Alert."

    datadog new monitor
  4. Define the metric. To create an advanced alert, click on the Advanced…​ link. Then, select a metric from the combo box. For example, select the ceph.num_in_osds Ceph metric.
  5. Click Add Query+ to add another query.

    datadog monitor ceph metric 1
  6. Select another metric from the combo box. For example, select the ceph.num_up_osds Ceph metric.

    datadog monitor ceph metric 2
  7. In the Express these queries as: field, enter a-b, where a is the value of ceph.num_in_osds and b is the value of ceph.num_up_osds. When the difference is 1 or greater, there is at least one OSD down.
  8. Set the alert conditions. For example, set the trigger to be above or equal to, the threshold to in total and the time elapsed to 1 minute.
  9. Set the Alert threshold field to 1. When at least one OSD is in the cluster and it is not up and running, the monitor will alert the user.
  10. Give the monitor a title in the input field below Preview and Edit. This is required to save the monitor.
  11. Enter a description of the alert in the text field.

    datadog monitor ceph metric 3
    Note

    The text field supports metric variables and Markdown syntax.

  12. Add the recipients of the alert. This will add an email address to the text field. When the alert gets triggered, the recipients will receive the alert.