Kibana dashboard is showing blank

Solution Verified - Updated -

Environment

  • RedHat OpenShift Container Platform (RHOCP)
    • 3.11

Issue

  • All pages on the Kibana dashboard is blank after login

  • Kibana (either Timelion or Kibana) page is showing errors like:

    Timelion: Error: in cell #1: [security_exception] no permissions for [indices:data/read/search] and User [name=<USER>, roles [gen_user_<UUID>, gen_kibana_<UUID>]]

    Error: no permissions for [indices:data/read/search] and User [name=<USER>, roles=[gen_user_<UUID>, gen_kibana_<UUID>]]: [security_exception] no permissions for [indices:data/read/search] and User [name=<USER>, roles=[gen_user_<UUID>, gen_kibana_<UUID>]]

Resolution

  • Delete the .kibana indices for the user-facing issue - NOTE: Deletion is an irreversible action and will cause data loss on Kibana for any custom Visualization, Dashboard, etc. See the Diagnostic Steps section to overcome this PRIOR the steps shown here

    $ printf "%s" <USERNAME> | sha1sum | awk '{print $1}' <-- Save HASH result here

$ oc exec -c elasticsearch <ES-MASTER-POD> -- curl -sfL --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert --cacert /etc/elasticsearch/secret/admin-ca -XDELETE https://localhost:9200/.kibana.<HASH> <--put HASH value here

  • Restart the Kibana pod

    $ oc delete pod <KIBANA-POD>

Root Cause

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or its entities, products, or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

  • This issue is being tracked in Bugzilla #1719733

  • Please note that RHOCP 3.11.135 addressed this through RHBA-2019:2352

    • RHBA actually just delivered the two additional ES utils scripts for importing/exporting data from Kibana to overcome the data loss of the actions in resolution, and not a fix for the issue itself

    • See PR#1688

    • See README

Diagnostic Steps

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or its entities, products, or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

  • Export the kibana object (like views, dashboards, etc.) using the export-kibana-objects

    $ oc get po -n openshift-logging <-- retrieve both ES-MASTER-POD and KIBANA-POD names
$ oc -n openshift-logging exec -c elasticsearch <ES-MASTER-POD> -- export-kibana-objects <USER> > <USER-DUMP>.json

  • Check the exported JSON whether it contains actual data. Here's an empty one for reference, for which you would skip the import step

    $ jq -S . testuser01.json
{
"_shards": {
  "failed": 0,
  "skipped": 0,
  "successful": 1,
  "total": 1
},
"hits": {
  "hits": [],
  "max_score": null,
  "total": 0
},
"timed_out": false,
"took": 0
}

  • Import the previously exported kibana object using the import-kibana-objects

    $ cat <USER-DUMP>.json | oc exec -i -c elasticsearch <ES-MASTER-POD> -- import-kibana-objects

  • Trying to import an empty JSON would return the error below

    $ jq -S . <USERNAME>.json
{ 
"_shards": {
  "failed": 0,
  "skipped": 0, 
  "successful": 1,
  "total": 1
},
"hits": { 
  "hits": [],
  "max_score": null,
  "total": 0
},
"timed_out": false,
"took": 0
} 

$ cat <USERNAME>.json | oc exec -i -c elasticsearch <ES-MASTER-POD> -- import-kibana-objects
Traceback (most recent call last):
File "<string>", line 7, in <module>
IndexError: list index out of range
command terminated with exit code 1

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments