Elasticsearch pods in error state when using glusterfs storage
Issue
- When deployed the logging stack on OCP cluster, the
elasticsearch
pod never comes online and fails with the below stack trace (on all deployment configs).
[2018-01-25 01:41:02,934][INFO ][container.run ] Seeding the searchguard ACL index. Will wait up to 604800 seconds.
/usr/share/java/elasticsearch/config
Will connect to localhost:9300 ... done
2018-01-25 01:41:03 INFO SearchGuardSSLPlugin:84 - Search Guard 2 plugin not available
2018-01-25 01:41:03 INFO SearchGuardPlugin:58 - Clustername: elasticsearch
2018-01-25 01:41:03 INFO SearchGuardPlugin:70 - Node [null] is a transportClient: true/tribeNode: false/tribeNodeClient: false
2018-01-25 01:41:03 INFO plugins:180 - [Pyre] modules [], plugins [search-guard-ssl, search-guard2], sites []
2018-01-25 01:41:03 INFO DefaultSearchGuardKeyStore:423 - Open SSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: org.apache.tomcat.jni.SSL
2018-01-25 01:41:04 INFO DefaultSearchGuardKeyStore:146 - sslHTTPProvider:null with ciphers []
2018-01-25 01:41:04 INFO DefaultSearchGuardKeyStore:148 - sslTransport protocols [TLSv1.2, TLSv1.1]
2018-01-25 01:41:04 INFO DefaultSearchGuardKeyStore:149 - sslHTTP protocols [TLSv1.2, TLSv1.1]
2018-01-25 01:41:04 INFO transport:99 - [Pyre] Using [com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] as transport, overridden by [search-guard-ssl]
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
ERR: Timed out while waiting for a green or yellow cluster state.
* Try running sgadmin.sh with -icl and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
0
[2018-01-25 01:46:05,504][WARN ][container.run ] Error seeding the searchguard ACL index... retrying in 10 seconds - 0 retries so far
[2018-01-25 01:46:05,506][WARN ][container.run ] Seeding will continue to fail until the cluster status is YELLOW
[2018-01-25 01:46:05,722][INFO ][container.run ] Remaining red indices: 1
/usr/share/java/elasticsearch/config
Will connect to localhost:9300 ... done
2018-01-25 01:46:16 INFO SearchGuardSSLPlugin:84 - Search Guard 2 plugin not available
2018-01-25 01:46:16 INFO SearchGuardPlugin:58 - Clustername: elasticsearch
2018-01-25 01:46:16 INFO SearchGuardPlugin:70 - Node [null] is a transportClient: true/tribeNo
- Every time the deployment fails when the backend storage used is
glusterfs
storage class. - The deployment never comes up and fails on
timeout error
.
Environment
- Openshift Container Platform (OCP)
- 3.x
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.