Elasticsearch pods in error state when using glusterfs storage
Issue
- When deployed the logging stack on OCP 3.6 cluster, the
elasticsearchpod never comes online and fails with the below stack trace (on all deployment configs).
[2018-01-25 01:41:02,934][INFO ][container.run ] Seeding the searchguard ACL index. Will wait up to 604800 seconds.
/usr/share/java/elasticsearch/config
Will connect to localhost:9300 ... done
2018-01-25 01:41:03 INFO SearchGuardSSLPlugin:84 - Search Guard 2 plugin not available
2018-01-25 01:41:03 INFO SearchGuardPlugin:58 - Clustername: elasticsearch
2018-01-25 01:41:03 INFO SearchGuardPlugin:70 - Node [null] is a transportClient: true/tribeNode: false/tribeNodeClient: false
2018-01-25 01:41:03 INFO plugins:180 - [Pyre] modules [], plugins [search-guard-ssl, search-guard2], sites []
2018-01-25 01:41:03 INFO DefaultSearchGuardKeyStore:423 - Open SSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: org.apache.tomcat.jni.SSL
2018-01-25 01:41:04 INFO DefaultSearchGuardKeyStore:146 - sslHTTPProvider:null with ciphers []
2018-01-25 01:41:04 INFO DefaultSearchGuardKeyStore:148 - sslTransport protocols [TLSv1.2, TLSv1.1]
2018-01-25 01:41:04 INFO DefaultSearchGuardKeyStore:149 - sslHTTP protocols [TLSv1.2, TLSv1.1]
2018-01-25 01:41:04 INFO transport:99 - [Pyre] Using [com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] as transport, overridden by [search-guard-ssl]
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
ERR: Timed out while waiting for a green or yellow cluster state.
* Try running sgadmin.sh with -icl and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
0
[2018-01-25 01:46:05,504][WARN ][container.run ] Error seeding the searchguard ACL index... retrying in 10 seconds - 0 retries so far
[2018-01-25 01:46:05,506][WARN ][container.run ] Seeding will continue to fail until the cluster status is YELLOW
[2018-01-25 01:46:05,722][INFO ][container.run ] Remaining red indices: 1
/usr/share/java/elasticsearch/config
Will connect to localhost:9300 ... done
2018-01-25 01:46:16 INFO SearchGuardSSLPlugin:84 - Search Guard 2 plugin not available
2018-01-25 01:46:16 INFO SearchGuardPlugin:58 - Clustername: elasticsearch
2018-01-25 01:46:16 INFO SearchGuardPlugin:70 - Node [null] is a transportClient: true/tribeNo
- Every time the deployment fails when the backend storage used is
glusterfsstorage class. - The deployment never comes up and fails on
timeout error.
Environment
- Openshift Container Platform (OCP)
- 3.6
- 3.7
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
