OpenShift Logging dashboards shows "No datapoints found."

Solution Unverified - Updated -

Issue

  • The OpenShift Logging dashboards in the "Monitoring" view show a "No datapoints found". error.
  • The ElasticSearch Pod is showing the following no permissions error:

    [2022-02-07T11:11:13,001][WARN ][r.suppressed             ] [elasticsearch-cdm-cll12fnj-1] path: /_prometheus/metrics, params: {}
    org.elasticsearch.ElasticsearchException: Search request failed
    [..]
    Caused by: org.elasticsearch.ElasticsearchSecurityException: no permissions for [indices:data/read/search] and User [name=system:serviceaccount:openshift-monitoring:prometheus-k8s, roles=[prometheus], requestedTenant=null]
    
  • When querying the /_prometheus/metrics endpoint for ElasticSearch, the following no permissions error is shown:

    sh-4.4$   curl -H"Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" -s -k https://elasticsearch-metrics.openshift-logging.svc:60001/_prometheus/metrics
    {"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [indices:data/read/search] and User [name=system:serviceaccount:openshift-monitoring:prometheus-k8s, roles=[prometheus], requestedTenant=null]"}],"type":"exception","reason":"Search request failed","caused_by":{"type":"security_exception","reason":"no permissions for [indices:data/read/search] and User [name=system:serviceaccount:openshift-monitoring:prometheus-k8s, roles=[prometheus], requestedTenant=null]"}},"status":500}
    

Environment

  • Red Hat OpenShift Container Platform (RHOCP) 4.9
  • cluster-logging.5.3.4-13 / elasticsearch-operator.5.3.4-13

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content