Elasticsearch pods in error state when using glusterfs storage

Solution Verified - Updated -

Issue

  • When deployed the logging stack on OCP cluster, the elasticsearch pod never comes online and fails with the below stack trace (on all deployment configs).
[2018-01-25 01:41:02,934][INFO ][container.run            ] Seeding the searchguard ACL index.  Will wait up to 604800 seconds.
/usr/share/java/elasticsearch/config
Will connect to localhost:9300 ... done
2018-01-25 01:41:03 INFO  SearchGuardSSLPlugin:84 - Search Guard 2 plugin not available
2018-01-25 01:41:03 INFO  SearchGuardPlugin:58 - Clustername: elasticsearch
2018-01-25 01:41:03 INFO  SearchGuardPlugin:70 - Node [null] is a transportClient: true/tribeNode: false/tribeNodeClient: false
2018-01-25 01:41:03 INFO  plugins:180 - [Pyre] modules [], plugins [search-guard-ssl, search-guard2], sites []
2018-01-25 01:41:03 INFO  DefaultSearchGuardKeyStore:423 - Open SSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: org.apache.tomcat.jni.SSL
2018-01-25 01:41:04 INFO  DefaultSearchGuardKeyStore:146 - sslHTTPProvider:null with ciphers []
2018-01-25 01:41:04 INFO  DefaultSearchGuardKeyStore:148 - sslTransport protocols [TLSv1.2, TLSv1.1]
2018-01-25 01:41:04 INFO  DefaultSearchGuardKeyStore:149 - sslHTTP protocols [TLSv1.2, TLSv1.1]
2018-01-25 01:41:04 INFO  transport:99 - [Pyre] Using [com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] as transport, overridden by [search-guard-ssl]
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
ERR: Timed out while waiting for a green or yellow cluster state.
   * Try running sgadmin.sh with -icl and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
0
[2018-01-25 01:46:05,504][WARN ][container.run            ] Error seeding the searchguard ACL index... retrying in 10 seconds - 0 retries so far
[2018-01-25 01:46:05,506][WARN ][container.run            ] Seeding will continue to fail until the cluster status is YELLOW
[2018-01-25 01:46:05,722][INFO ][container.run            ] Remaining red indices: 1
/usr/share/java/elasticsearch/config
Will connect to localhost:9300 ... done
2018-01-25 01:46:16 INFO  SearchGuardSSLPlugin:84 - Search Guard 2 plugin not available
2018-01-25 01:46:16 INFO  SearchGuardPlugin:58 - Clustername: elasticsearch
2018-01-25 01:46:16 INFO  SearchGuardPlugin:70 - Node [null] is a transportClient: true/tribeNo
  • Every time the deployment fails when the backend storage used is glusterfs storage class.
  • The deployment never comes up and fails on timeout error.

Environment

  • Openshift Container Platform (OCP)
    • 3.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content