Chapter 6. Management of Alerts on the Ceph dashboard

As a storage administrator, you can see the details of alerts and create silences for them on the Red Hat Ceph Storage dashboard. This includes the following pre-defined alerts:

  • 10% OSDs Down
  • Flapping OSD
  • Health Error
  • Health Warn
  • High PG count deviation
  • Low monitor quorum count
  • MTU Mismatch
  • Network packets errors
  • Network packets dropped
  • OSD down
  • OSDs near full
  • PGs inactive
  • PGs unclean
  • Pool filling up
  • Pool full
  • Root volume full
  • Slow OSD Ops
  • Storage filling up

Figure 6.1. Pre-defined alerts

Pre-defined alerts

6.1. Enabling monitoring stack

You can manually enable the monitoring stack of the Red Hat Ceph Storage cluster, such as Prometheus, Alertmanager, and Grafana, using the command-line interface.

You can use the Prometheus and Alertmanager API to manage alerts and silences.

Prerequisite

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to all the nodes.

Procedure

  1. Log into the cephadm shell:

    Example

    [root@host01 ~]# cephadm shell

  2. Set the APIs for the monitoring stack:

    • Specify the host and port of the Alertmanager server:

      Syntax

      ceph dashboard set-alertmanager-api-host 'ALERTMANAGER_API__HOST: PORT'

      Example

      [ceph: root@host01 /]# ceph dashboard set-alertmanager-api-host 'http://10.0.0.101:9093'
      Option ALERTMANAGER_API_HOST updated

    • To see the configured alerts, configure the URL to the Prometheus API. Using this API, the Ceph Dashboard UI verifies that a new silence matches a corresponding alert.

      Syntax

      ceph dashboard set-prometheus-api-host 'PROMETHEUS_API_HOST: PORT'

      Example

      [ceph: root@host01 /]# ceph dashboard set-prometheus-api-host 'http://10.0.0.101:9095'
      Option PROMETHEUS_API_HOST updated

      After setting up the hosts, refresh your browser’s dashboard window.

    • Specify the host and port of the Grafana server:

      Syntax

      ceph dashboard set-grafana-api-url 'GRAFANA_API__URL: PORT'

      Example

      [ceph: root@host01 /]# ceph dashboard set-grafana-api-url 'http://10.0.0.101:3000'
      Option GRAFANA_API_URL updated

  3. Get the Prometheus, Alertmanager, and Grafana API host details:

    Example

    [ceph: root@host01 /]# ceph dashboard get-alertmanager-api-host
    http://10.0.0.101:9093
    [ceph: root@host01 /]# ceph dashboard get-prometheus-api-host
    http://10.0.0.101:9095
    [ceph: root@host01 /]# ceph dashboard get-grafana-api-url
    http://10.0.0.101:3000

  4. Optional: If you are using a self-signed certificate in your Prometheus, Alertmanager, or Grafana setup, disable the certificate verification in the dashboard This avoids refused connections caused by certificates signed by an unknown Certificate Authority (CA) or that do not match the hostname.

    • For Prometheus:

      Example

      [ceph: root@host01 /]# ceph dashboard set-prometheus-api-ssl-verify False

    • For Alertmanager:

      Example

      [ceph: root@host01 /]# ceph dashboard set-alertmanager-api-ssl-verify False

    • For Grafana:

      Example

      [ceph: root@host01 /]# ceph dashboard set-grafana-api-ssl-verify False

  5. Get the details of the self-signed certificate verification setting for Prometheus, Alertmanager, and Grafana:

    Example

    [ceph: root@host01 /]# ceph dashboard get-prometheus-api-ssl-verify
    [ceph: root@host01 /]# ceph dashboard get-alertmanager-api-ssl-verify
    [ceph: root@host01 /]# ceph dashboard get-grafana-api-ssl-verify

  6. Optional: If the dashboard does not reflect the changes, you have to disable and then enable the dashboard:

    Example

    [ceph: root@host01 /]# ceph mgr module disable dashboard
    [ceph: root@host01 /]# ceph mgr module enable dashboard

Additional Resources

6.2. Viewing alerts on the Ceph dashboard

After an alert has fired, you can view it on the Red Hat Ceph Storage Dashboard. You can edit the Manager module settings to trigger a mail when an alert is fired.

Prerequisite

  • A running Red Hat Ceph Storage cluster.
  • Dashboard is installed.
  • An alert fired.

Procedure

  1. Log in to the Dashboard.
  2. Customize the alerts module on the dashboard to get an email alert for the storage cluster:

    1. On the navigation menu, click Cluster.
    2. Select Manager modules.
    3. Select alerts module.
    4. In the Edit drop-down menu, select Edit.
    5. In the Edit Manager module window, update the required parameters and click Update.

      Figure 6.2. Edit Manager module for alerts

      Edit Manager module for alerts
  3. On the navigation menu, click Cluster.
  4. Select Monitoring from the drop-down menu.
  5. To view details of the alert, click the Expand/Collapse icon on it’s row.

    Figure 6.3. Viewing alerts

    Viewing alerts
  6. To view the source of an alert, click on its row, and then click Source.

6.3. Creating a silence on the Ceph dashboard

You can create a silence for an alert for a specified amount of time on the Red Hat Ceph Storage Dashboard.

Prerequisite

  • A running Red Hat Ceph Storage cluster.
  • Dashboard is installed.
  • An alert fired.

Procedure

  1. Log in to the Dashboard.
  2. On the navigation menu, click Cluster.
  3. Select Monitoring from the drop-down menu.
  4. To create silence for an alert, select it’s row.
  5. Click +Create Silence.
  6. In the Create Silence window, Add the details for the Duration and click Create Silence.

    Figure 6.4. Create Silence

    Create Silence
  7. You get a notification that the silence was created successfully.

6.4. Re-creating a silence on the Ceph dashboard

You can re-create a silence from an expired silence on the Red Hat Ceph Storage Dashboard.

Prerequisite

  • A running Red Hat Ceph Storage cluster.
  • Dashboard is installed.
  • An alert fired.
  • A silence created for the alert.

Procedure

  1. Log in to the Dashboard.
  2. On the navigation menu, click Cluster.
  3. Select Monitoring from the drop-down menu.
  4. Click the Silences tab.
  5. To recreate an expired silence, click it’s row.
  6. Click the Recreate button.
  7. In the Recreate Silence window, add the details and click Recreate Silence.

    Figure 6.5. Recreate silence

    Re-create Silence
  8. You get a notification that the silence was recreated successfully.

6.5. Editing a silence on the Ceph dashboard

You can edit an active silence, for example, to extend the time it is active on the Red Hat Ceph Storage Dashboard. If the silence has expired, you can either recreate a silence or create a new silence for the alert.

Prerequisite

  • A running Red Hat Ceph Storage cluster.
  • Dashboard is installed.
  • An alert fired.
  • A silence created for the alert.

Procedure

  1. Log in to the Dashboard.
  2. On the navigation menu, click Cluster.
  3. Select Monitoring from the drop-down menu.
  4. Click the Silences tab.
  5. To edit the silence, click it’s row.
  6. In the Edit drop-down menu, select Edit.
  7. In the Edit Silence window, update the details and click Edit Silence.

    Figure 6.6. Edit silence

    Edit Silence
  8. You get a notification that the silence was updated successfully.

6.6. Expiring a silence on the Ceph dashboard

You can expire a silence so any matched alerts will not be suppressed on the Red Hat Ceph Storage Dashboard.

Prerequisite

  • A running Red Hat Ceph Storage cluster.
  • Dashboard is installed.
  • An alert fired.
  • A silence created for the alert.

Procedure

  1. Log in to the Dashboard.
  2. On the navigation menu, click Cluster.
  3. Select Monitoring from the drop-down menu.
  4. Click the Silences tab.
  5. To expire a silence, click it’s row.
  6. In the Edit drop-down menu, select Expire.
  7. In the Expire Silence dialog box, select Yes, I am sure, and then click Expire Silence.

    Figure 6.7. Expire Silence

    Expire Silence
  8. You get a notification that the silence was expired successfully.

6.7. Additional Resources