An alert can have several annotations, although the following three should always be defined:

An alert can have several labels, although the following should always be defined:

Once the ConfigMap has been edited, the prometheus process in the Prometheus container must be triggered with a SIGHUP to read the configuration changes. This can be done using the following commands:

oc rsh -c prometheus prometheus-0 bash
bash-4.2$ kill -SIGHUP 1
bash-4.2$ exit

The following alert definition is useful to test whether the alerting functionality has been configured correctly. It should raise an alert if any of the OpenShift Container Platform nodes are up:

- alert: "Node up" # helpful for testing
  expr: up{job="kubernetes-nodes"} == 1
  annotations:
    miqTarget: "ContainerNode"
    url: "https://www.example.com/fixing_instructions"
    description: "ContainerNode {{ $labels.instance }} is up"
  labels:
    severity: "info"

The following alert definition should raise an alert if the number of pods running on any node exceeds the threshold defined in the promQL expression:

- alert: "Too Many Pods"
  expr: sum(kubelet_running_pod_count) > 50
  annotations:
    miqTarget: "ExtManagementSystem"
    url: "https://www.example.com/too_many_pods_fixing_instructions"
    description: "Too many running pods"
  labels:
    severity: "error"

The following alert definition should raise an alert if the number of authenticated logins to the OpenShift Container Platform console exceeds the threshold defined in the promQL expression:

- alert: "Too Many Requests"
  expr: rate(authenticated_user_requests[2m]) > 20
  annotations:
    miqTarget: "ExtManagementSystem"
    url: "https://www.example.com/too_many_requests_fixing_instructions"
    description: "Too many authenticated login requests"
  labels:
    severity: "warning"

The following alert definition uses metrics returned by the node-exporter, and should raise an alert if node CPU utilization exceeds the threshold defined in the promQL expression:

- alert: "Node CPU Usage"
  expr: (100 - (avg by (instance) ⏎
        (irate(node_cpu{app="prometheus-node-exporter", mode="idle"}[5m])) ⏎
        * 100)) > 70
  for: 30s
  annotations:
    miqTarget: "ExtManagementSystem"
    url: "https://www.example.com/high_node_cpu_fixing_instructions"
    description: "{{ $labels.instance }}: CPU usage is above 70% ⏎
          (current value is: {{ $value }})"
  labels:
    severity: "warning"

Each alert type is configurable to send an email, an SNMP trap, or run an automate instance (see Figure 9.5, “Defining a "Server Stopped" Alert”).

Figure 9.5. Defining a "Server Stopped" Alert

​

An email sent by one of these alerts will have a subject such as:

Alert Triggered: EVM Worker Killed, for (MIQSERVER) cfmesrv06.

The email body will contain text such as the following:

Alert 'EVM Worker Killed', triggered

Event:  Alert condition met
Entity: (MiqServer) cfmesrv06

To determine more information - such as the actual worker type that was killed - it may be necessary to search evm.log on the appliance mentioned.