High response times when querying Loki series having cluster-reader clusterrole and namespace access granted in RHOL

Solution Verified - Updated -

Issue

  • As a platform developer with cluster-reader Role in OpenShift Container Platform, when querying the Loki series takes up to 23 seconds when using Loki via Explore or Grafana dashboard with Loki functionality.
  • As an user granted cluster-reader clusterrole and namespace access to the Loki logs, when it's queried the Loki series, the time is high. The same behaviour is not observed when querying the application logs

    $ time logcli -o raw --tls-skip-verify --bearer-token="$(oc whoami -t)" --addr "https://${lokiroute}/api/logs/v1/application" series --analyze-labels  --retries=2 --since=6h '{log_type="application"}'
    [...]
    Total Streams:  251
    Unique Labels:  10
    
    Label Name                 Unique Values  Found In Streams
    k8s_pod_name               251            251
    kubernetes_pod_name        251            251
    k8s_namespace_name         202            251
    kubernetes_namespace_name  202            251
    kubernetes_host            3              251
    k8s_node_name              3              251
    kubernetes_container_name  1              251
    openshift_log_type         1              251
    log_type                   1              251
    k8s_container_name         1              251
    
    real    0m53.729s
    user    0m0.117s
    sys 0m0.031s
    

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Red Hat OpenShift Logging (RHOL)
    • 5
    • 6
  • LokiStack

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content