Alert "elasticsearch-metrics targets in Namespace openshift-logging namespace have been unreachable" is firing in OpenShift Container Platform

Solution Verified - Updated -

Issue

  • The following alert is firing in the OpenShift Container Platform cluster:

    x% of the elasticsearch-metrics/elasticsearch-metrics targets in Namespace openshift-logging namespace have been unreachable
    
  • The Cluster Logging Operator is showing the following messages in the status field:

    $ oc get ClusterLogging instance -o yaml
    [..]
        nodeConditions:
        elasticsearch-cdm-2wn4b6sn-1:
        - lastTransitionTime: "2023-05-08T07:06:32Z"
          message: Disk storage usage for node is 1.18Tb (87.59966182226196%). Shards will be not be allocated on this node.
          reason: Disk Watermark Low
          status: "True"
          type: NodeStorage
        elasticsearch-cdm-2wn4b6sn-2:
        - lastTransitionTime: "2023-05-08T07:06:32Z"
          message: Disk storage usage for node is 1.17Tb (87.17254571557135%). Shards will be not be allocated on this node.
          reason: Disk Watermark Low
          status: "True"
          type: NodeStorage
        elasticsearch-cdm-2wn4b6sn-3:
        - lastTransitionTime: "2023-05-08T07:06:32Z"
          message: Disk storage usage for node is 1.16Tb (86.54566843101432%). Shards will be not be allocated on this node.
          reason: Disk Watermark Low
          status: "True"
          type: NodeStorage
    
    [..]
    

Environment

  • Red Hat OpenShift Container Platform (RHOCP) 4.11
  • OpenShift Logging 5.6

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content