How to capture Elasticsearch metrics with curl?

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform
    • 4
  • Red Hat OpenShift Logging
    • 4
    • 5

Issue

  • How to capture Elasticsearch metrics with curl?

Resolution

Elasticsearch metrics used to scrape via token generated by prometheus-k8s serviceaccount but new approach is to use the token generated by elasticsearch-metrics serviceaccount. Both the methods are explained below:

Old Method (till OpenShift Elasticsearch 5.3 version):

  • From prometheus-k8s pod in openshift-monitoring namespace run below commands:
# oc rsh prometheus-k8s-0
$ token=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

$ curl -H "Authorization: Bearer $token" --cacert /etc/prometheus/configmaps/serving-certs-ca-bundle/service-ca.crt https://elasticsearch-metrics.openshift-logging.svc.cluster.local:60001/_prometheus/metrics

New Method (from OpenShift Elasticsearch 5.4+ version):

  1. Generate token from elasticsearch-metrics serviceaccount:
For RHOCP 4.10 and older versions:
$ oc serviceaccounts get-token elasticsearch-metrics -n openshift-logging

For RHOCP 4.11 and newer versions:
$ oc create token elasticsearch-metrics -n openshift-logging
  1. From prometheus-k8s pod in openshift-monitoring project, run below commands:
$ oc rsh prometheus-k8s-0
$ token=<token-obtained-in-step1>
$ curl -k -H "Authorization: Bearer $token" https://elasticsearch-metrics.openshift-logging.svc:60001/_prometheus/metrics

Below is the snippet of elasticsearch metrics obtained via above commands:

--//--snippet--//--
# HELP es_indices_get_exists_time_seconds Time spent while existing documents get command
# TYPE es_indices_get_exists_time_seconds gauge
es_indices_get_exists_time_seconds{cluster="elasticsearch",node="elasticsearch-cdm-pk7bx5zr-3",nodeid="TDEMXjJ6TFyZzA0iGybi3w",} 0.21
# HELP es_indices_fielddata_evictions_count Count of evictions in field data cache
# TYPE es_indices_fielddata_evictions_count gauge
es_indices_fielddata_evictions_count{cluster="elasticsearch",node="elasticsearch-cdm-pk7bx5zr-3",nodeid="TDEMXjJ6TFyZzA0iGybi3w",} 0.0
.......
--//--snippet--//--

Note: As metrics are captured using endpoints of elasticsearch-metrics service, the above output will give metrics of any one elasticsearch-cdm pod . Hence to capture metrics of specific pod run the command few more times and check node value that will appear in various metrics example in es_indices_querycache_hit_count.

es_indices_querycache_hit_count{cluster="elasticsearch",node="elasticsearch-cdm-pk7bx5zr-3",nodeid="TDEMXjJ6TFyZzA0iGybi3w",} 19.0

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments