High response times when querying Loki series having cluster-reader clusterrole and namespace access granted in RHOL
Issue
- As a platform developer with cluster-reader Role in OpenShift Container Platform, when querying the Loki series takes up to 23 seconds when using Loki via Explore or Grafana dashboard with Loki functionality.
-
As an user granted cluster-reader clusterrole and namespace access to the Loki logs, when it's queried the Loki series, the time is high. The same behaviour is not observed when querying the application logs
$ time logcli -o raw --tls-skip-verify --bearer-token="$(oc whoami -t)" --addr "https://${lokiroute}/api/logs/v1/application" series --analyze-labels --retries=2 --since=6h '{log_type="application"}' [...] Total Streams: 251 Unique Labels: 10 Label Name Unique Values Found In Streams k8s_pod_name 251 251 kubernetes_pod_name 251 251 k8s_namespace_name 202 251 kubernetes_namespace_name 202 251 kubernetes_host 3 251 k8s_node_name 3 251 kubernetes_container_name 1 251 openshift_log_type 1 251 log_type 1 251 k8s_container_name 1 251 real 0m53.729s user 0m0.117s sys 0m0.031s
Environment
- Red Hat OpenShift Container Platform (RHOCP)
- 4
- Red Hat OpenShift Logging (RHOL)
- 5
- 6
- LokiStack
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.