Alert "elasticsearch-metrics targets in Namespace openshift-logging namespace have been unreachable" is firing in OpenShift Container Platform
Issue
-
The following alert is firing in the OpenShift Container Platform cluster:
x% of the elasticsearch-metrics/elasticsearch-metrics targets in Namespace openshift-logging namespace have been unreachable -
The Cluster Logging Operator is showing the following messages in the
statusfield:$ oc get ClusterLogging instance -o yaml [..] nodeConditions: elasticsearch-cdm-2wn4b6sn-1: - lastTransitionTime: "2023-05-08T07:06:32Z" message: Disk storage usage for node is 1.18Tb (87.59966182226196%). Shards will be not be allocated on this node. reason: Disk Watermark Low status: "True" type: NodeStorage elasticsearch-cdm-2wn4b6sn-2: - lastTransitionTime: "2023-05-08T07:06:32Z" message: Disk storage usage for node is 1.17Tb (87.17254571557135%). Shards will be not be allocated on this node. reason: Disk Watermark Low status: "True" type: NodeStorage elasticsearch-cdm-2wn4b6sn-3: - lastTransitionTime: "2023-05-08T07:06:32Z" message: Disk storage usage for node is 1.16Tb (86.54566843101432%). Shards will be not be allocated on this node. reason: Disk Watermark Low status: "True" type: NodeStorage [..]
Environment
- Red Hat OpenShift Container Platform (RHOCP) 4.11
- OpenShift Logging 5.6
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.