Kibana "Tenant indices migration failed" issue after migration in RHOCP 4

Solution Verified - Updated -

Issue

  • After upgrading to RHOL 5.4 Kibana is in Yellow status throwing error plugin:opendistro_security@6.8.1 Tenant indices migration failed. Workaround in articles Kibana status is Yellow in OpenShift 4 and Kibana in yellow state with error "plugin:opendistro_security@6.8.1 Setting up index template" in OpenShift Container Platform 4 don't work
  • Kibana UI shows:

    Kibana status is `Yellow`
    
    Plugin Status:
    plugin:opendistro_security@6.8.1       Tenant indices migration failed
    
  • Kibana pod logs show Authorization exception error with statusCode: 403

    $ oc -n openshift-logging logs $(oc -n openshift-logging get pods -l component=kibana -o NAME) -c kibana|grep "Authorization Exception:.*statusCode.*403"
    {"type":"error","@timestamp":"2022-04-26T19:04:42Z","tags":["error","migration"],"pid":117,"level":"error","error":{"message":"Authorization Exception: Authorization Exception","name":"Error","stack":"Authorization Exception :: {\"path\":\"/_opendistro/_security/tenantinfo\",\"query\":{},\"statusCode\":403,\"response\":\"\"}\n    at respond (/opt/app-root/src/node_modules/elasticsearch/src/lib/transport.js:308:15)\n    at checkRespForFailure (/opt/app-root/src/node_modules/elasticsearch/src/lib/transport.js:267:7)\n    at HttpConnector.<anonymous> (/opt/app-root/src/node_modules/elasticsearch/src/lib/connectors/http.js:166:7)\n    at IncomingMessage.wrapper (/opt/app-root/src/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)\n    at IncomingMessage.emit (events.js:203:15)\n    at endReadableNT (_stream_readable.js:1145:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"message":"Authorization Exception: Authorization Exception"}
    
  • Kibana UI shows Kibana server is not ready yet
  • Not able to view logs in kibana as kibana shows status is Yellow.
  • Kibana is in Yellow state but elasticsearch cluster is healthy.

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Red Hat OpenShift Logging (RHOL)
    • 5.4

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content