Error in Kibana UI after upgrading to OpenShift 3.6

Solution Unverified - Updated -

Issue

  • After upgrading to OpenShift 3.6, there is an error in the Kibana UI and logs are not getting sent to Elastic search properly. The following error appears in the elasticsearch pod logs:

    Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
    Cannot retrieve cluster state due to: ClusterService was close during health call. This is not an error, will keep on trying ...
       * Try running sgadmin.sh with -icl and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
       * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
    Clustername: logging-es
    Clusterstate: YELLOW
    Number of nodes: 3
    Number of data nodes: 3
    .searchguard.logging-es-data-master-51abcde9x index already exists, so we do not need to create one.
    INFO: .searchguard.logging-es-data-master-51abcde9x index state is YELLOW, it seems you miss some replicas
    
  • The following error appeared in the Kibana UI after upgrading to OCP 3.6:

    Discover: [exception] The index returned an empty result. You can use the Time Picker to change the time filter or select a higher time interval
    
  • In version 3.5 and prior, there was an index which contained logs of all projects called .all. After upgrading to the Aggregated Logging Stack v3.6, this index seems to be gone. It looks like it has been replaced with an index called .operations but this index is empty. We're trying to troubleshoot platform-wide issues and need to search logs of all projects at once. This seems to be impossible now.

Environment

  • OpenShift Container Platform (OCP) 3.6

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content