NullPointerException in SearchGuard in OpenShift Enterprise version 3.2 and 3.3
Issue
OpenShift Enterprise (OSE) version 3.2 comes with ElasticSearch (ES) version 1.5.2 as part of the ElasticSearch, Fluentd and Kibana (EFK) logging stacks.
ES cluster status health turns to red based on this solution #3139821 and this case #01796385 are due to UNASSIGNED shards.
Analyzing which ES indices that cause the problem by invoking ES /_cat/indices and /_cat/shards APIs return nothing. There is no output/result being displayed.
#oc exec <pod> -- curl -s --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert --cacert /etc/elasticsearch/secret/admin-ca 'https://localhost:9200/_cat/indices?v'
#oc exec <pod> -- curl -s --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert --cacert /etc/elasticsearch/secret/admin-ca 'https://localhost:9200/_cat/shards?v'
Found NullPointerException error inside ES log.
[2017-06-28 05:49:38,052][ERROR][com.floragunn.searchguard.filter.SearchGuardActionFilter] Error while apply() due to java.lang.NullPointerException for action indices:monitor/stats
java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at java.util.Arrays$ArrayList.<init>(Arrays.java:3813)
at java.util.Arrays.asList(Arrays.java:3800)
at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply0(SearchGuardActionFilter.java:191)
at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply(SearchGuardActionFilter.java:90)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.FLSActionFilter.applySecure(FLSActionFilter.java:76)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:97)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.DLSActionFilter.applySecure(DLSActionFilter.java:73)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:97)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.RequestActionFilter.applySecure(RequestActionFilter.java:94)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:97)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:64)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at io.fabric8.elasticsearch.plugin.ActionForbiddenActionFilter.apply(ActionForbiddenActionFilter.java:48)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)
at org.elasticsearch.client.node.NodeIndicesAdminClient.execute(NodeIndicesAdminClient.java:77)
at org.elasticsearch.client.FilterClient$IndicesAdmin.execute(FilterClient.java:120)
at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient$IndicesAdmin.execute(BaseRestHandler.java:149)
at org.elasticsearch.client.support.AbstractIndicesAdminClient.stats(AbstractIndicesAdminClient.java:524)
at org.elasticsearch.rest.action.cat.RestIndicesAction$1$1.processResponse(RestIndicesAction.java:83)
at org.elasticsearch.rest.action.cat.RestIndicesAction$1$1.processResponse(RestIndicesAction.java:78)
at org.elasticsearch.rest.action.support.RestActionListener.onResponse(RestActionListener.java:49)
at org.elasticsearch.action.support.TransportAction$ThreadedActionListener$1.run(TransportAction.java:113)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
[2017-06-28 05:49:53,571][ERROR][com.floragunn.searchguard.filter.SearchGuardActionFilter] Error while apply() due to java.lang.NullPointerException for action indices:monitor/stats
java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at java.util.Arrays$ArrayList.<init>(Arrays.java:3813)
at java.util.Arrays.asList(Arrays.java:3800)
at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply0(SearchGuardActionFilter.java:191)
at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply(SearchGuardActionFilter.java:90)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.FLSActionFilter.applySecure(FLSActionFilter.java:76)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:97)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.DLSActionFilter.applySecure(DLSActionFilter.java:73)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:97)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.RequestActionFilter.applySecure(RequestActionFilter.java:94)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:97)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:64)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at io.fabric8.elasticsearch.plugin.ActionForbiddenActionFilter.apply(ActionForbiddenActionFilter.java:48)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)
at org.elasticsearch.client.node.NodeIndicesAdminClient.execute(NodeIndicesAdminClient.java:77)
at org.elasticsearch.client.FilterClient$IndicesAdmin.execute(FilterClient.java:120)
at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient$IndicesAdmin.execute(BaseRestHandler.java:149)
at org.elasticsearch.client.support.AbstractIndicesAdminClient.stats(AbstractIndicesAdminClient.java:524)
at org.elasticsearch.rest.action.cat.RestShardsAction$1.processResponse(RestShardsAction.java:68)
at org.elasticsearch.rest.action.cat.RestShardsAction$1.processResponse(RestShardsAction.java:63)
at org.elasticsearch.rest.action.support.RestActionListener.onResponse(RestActionListener.java:49)
at org.elasticsearch.action.support.TransportAction$ThreadedActionListener$1.run(TransportAction.java:113)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
[2017-06-28 05:53:20,960][ERROR][com.floragunn.searchguard.filter.SearchGuardActionFilter] Error while apply() due to java.lang.NullPointerException for action indices:monitor/stats
java.lang.NullPointerException
at java.util.Objects.requireNonNull(Objects.java:203)
at java.util.Arrays$ArrayList.<init>(Arrays.java:3813)
at java.util.Arrays.asList(Arrays.java:3800)
at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply0(SearchGuardActionFilter.java:191)
at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply(SearchGuardActionFilter.java:90)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.FLSActionFilter.applySecure(FLSActionFilter.java:76)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:97)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.DLSActionFilter.applySecure(DLSActionFilter.java:73)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:97)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at com.floragunn.searchguard.filter.RequestActionFilter.applySecure(RequestActionFilter.java:94)
at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:97)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:64)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at io.fabric8.elasticsearch.plugin.ActionForbiddenActionFilter.apply(ActionForbiddenActionFilter.java:48)
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)
at org.elasticsearch.client.node.NodeIndicesAdminClient.execute(NodeIndicesAdminClient.java:77)
at org.elasticsearch.client.FilterClient$IndicesAdmin.execute(FilterClient.java:120)
at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient$IndicesAdmin.execute(BaseRestHandler.java:149)
at org.elasticsearch.client.support.AbstractIndicesAdminClient.stats(AbstractIndicesAdminClient.java:524)
at org.elasticsearch.rest.action.cat.RestShardsAction$1.processResponse(RestShardsAction.java:68)
at org.elasticsearch.rest.action.cat.RestShardsAction$1.processResponse(RestShardsAction.java:63)
at org.elasticsearch.rest.action.support.RestActionListener.onResponse(RestActionListener.java:49)
at org.elasticsearch.action.support.TransportAction$ThreadedActionListener$1.run(TransportAction.java:113)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
It seems to hit Bug 1369924.
Environment
- OpenShift Enterprise 3.2
- ElasticSearch 1.5.2
- Installed 3 nodes ES cluster which consist of 1 ES master and 2 ES data nodes.
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.