TargetDown alerts for Cluster Logging Operator

Solution Verified - Updated -

Environment

  • RedHat OpenShift Service on AWS [ROSA]
    • 4.x
  • OpenShift Container Platform [OCP]
    • 4.x

Issue

  • Alert is seen in the Openshift console.
TargetDown      Warning
50% of the cluster-logging-operator-metrics/cluster-logging-operator-metrics targets in Namespace
NS                       openshift-logging
 namespace have been unreachable for more than 15 minutes. This may be a symptom of network connectivity issues, down nodes, or failures within these components. Assess the health of the infrastructure and nodes running these targets and then contact support.

Resolution

  • Delete the cluster-logging-operator pod.
$ oc delete pod cluster-logging-operator-xxxx  -n openshift-logging 

Root Cause

  • Metrics from the cluster-logging-operator pod were not getting scraped due to connectivity issues.

Diagnostic Steps

  • To check the Alerts which are firing in the OpenShift console, In Administrative view, navigate to:

    • Observe > Alerting
  • To check the Targets which are down, navigate to :

    • Observe > Targets

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments