NullPointerException in SearchGuard in OpenShift Enterprise version 3.2 and 3.3

Solution Verified - Updated -

Issue

OpenShift Enterprise (OSE) version 3.2 comes with ElasticSearch (ES) version 1.5.2 as part of the ElasticSearch, Fluentd and Kibana (EFK) logging stacks.

ES cluster status health turns to red based on this solution #3139821 and this case #01796385 are due to UNASSIGNED shards.

Analyzing which ES indices that cause the problem by invoking ES /_cat/indices and /_cat/shards APIs return nothing. There is no output/result being displayed.

#oc exec <pod> -- curl -s --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert --cacert /etc/elasticsearch/secret/admin-ca 'https://localhost:9200/_cat/indices?v'

#oc exec <pod> -- curl -s --key /etc/elasticsearch/secret/admin-key --cert /etc/elasticsearch/secret/admin-cert --cacert /etc/elasticsearch/secret/admin-ca 'https://localhost:9200/_cat/shards?v'

Found NullPointerException error inside ES log.

[2017-06-28 05:49:38,052][ERROR][com.floragunn.searchguard.filter.SearchGuardActionFilter] Error while apply() due to java.lang.NullPointerException for action indices:monitor/stats
java.lang.NullPointerException
    at java.util.Objects.requireNonNull(Objects.java:203)
    at java.util.Arrays$ArrayList.<init>(Arrays.java:3813)
    at java.util.Arrays.asList(Arrays.java:3800)
    at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply0(SearchGuardActionFilter.java:191)
    at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply(SearchGuardActionFilter.java:90)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
    at com.floragunn.searchguard.filter.FLSActionFilter.applySecure(FLSActionFilter.java:76)
    at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:97)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
    at com.floragunn.searchguard.filter.DLSActionFilter.applySecure(DLSActionFilter.java:73)
    at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:97)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
    at com.floragunn.searchguard.filter.RequestActionFilter.applySecure(RequestActionFilter.java:94)
    at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:97)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
    at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:64)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
    at io.fabric8.elasticsearch.plugin.ActionForbiddenActionFilter.apply(ActionForbiddenActionFilter.java:48)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)
    at org.elasticsearch.client.node.NodeIndicesAdminClient.execute(NodeIndicesAdminClient.java:77)
    at org.elasticsearch.client.FilterClient$IndicesAdmin.execute(FilterClient.java:120)
    at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient$IndicesAdmin.execute(BaseRestHandler.java:149)
    at org.elasticsearch.client.support.AbstractIndicesAdminClient.stats(AbstractIndicesAdminClient.java:524)
    at org.elasticsearch.rest.action.cat.RestIndicesAction$1$1.processResponse(RestIndicesAction.java:83)
    at org.elasticsearch.rest.action.cat.RestIndicesAction$1$1.processResponse(RestIndicesAction.java:78)
    at org.elasticsearch.rest.action.support.RestActionListener.onResponse(RestActionListener.java:49)
    at org.elasticsearch.action.support.TransportAction$ThreadedActionListener$1.run(TransportAction.java:113)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
[2017-06-28 05:49:53,571][ERROR][com.floragunn.searchguard.filter.SearchGuardActionFilter] Error while apply() due to java.lang.NullPointerException for action indices:monitor/stats
java.lang.NullPointerException
    at java.util.Objects.requireNonNull(Objects.java:203)
    at java.util.Arrays$ArrayList.<init>(Arrays.java:3813)
    at java.util.Arrays.asList(Arrays.java:3800)
    at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply0(SearchGuardActionFilter.java:191)
    at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply(SearchGuardActionFilter.java:90)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
    at com.floragunn.searchguard.filter.FLSActionFilter.applySecure(FLSActionFilter.java:76)
    at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:97)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
    at com.floragunn.searchguard.filter.DLSActionFilter.applySecure(DLSActionFilter.java:73)
    at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:97)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
    at com.floragunn.searchguard.filter.RequestActionFilter.applySecure(RequestActionFilter.java:94)
    at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:97)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
    at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:64)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
    at io.fabric8.elasticsearch.plugin.ActionForbiddenActionFilter.apply(ActionForbiddenActionFilter.java:48)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)
    at org.elasticsearch.client.node.NodeIndicesAdminClient.execute(NodeIndicesAdminClient.java:77)
    at org.elasticsearch.client.FilterClient$IndicesAdmin.execute(FilterClient.java:120)
    at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient$IndicesAdmin.execute(BaseRestHandler.java:149)
    at org.elasticsearch.client.support.AbstractIndicesAdminClient.stats(AbstractIndicesAdminClient.java:524)
    at org.elasticsearch.rest.action.cat.RestShardsAction$1.processResponse(RestShardsAction.java:68)
    at org.elasticsearch.rest.action.cat.RestShardsAction$1.processResponse(RestShardsAction.java:63)
    at org.elasticsearch.rest.action.support.RestActionListener.onResponse(RestActionListener.java:49)
    at org.elasticsearch.action.support.TransportAction$ThreadedActionListener$1.run(TransportAction.java:113)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)
[2017-06-28 05:53:20,960][ERROR][com.floragunn.searchguard.filter.SearchGuardActionFilter] Error while apply() due to java.lang.NullPointerException for action indices:monitor/stats
java.lang.NullPointerException
    at java.util.Objects.requireNonNull(Objects.java:203)
    at java.util.Arrays$ArrayList.<init>(Arrays.java:3813)
    at java.util.Arrays.asList(Arrays.java:3800)
    at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply0(SearchGuardActionFilter.java:191)
    at com.floragunn.searchguard.filter.SearchGuardActionFilter.apply(SearchGuardActionFilter.java:90)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
    at com.floragunn.searchguard.filter.FLSActionFilter.applySecure(FLSActionFilter.java:76)
    at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:97)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
    at com.floragunn.searchguard.filter.DLSActionFilter.applySecure(DLSActionFilter.java:73)
    at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:97)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
    at com.floragunn.searchguard.filter.RequestActionFilter.applySecure(RequestActionFilter.java:94)
    at com.floragunn.searchguard.filter.AbstractActionFilter.apply(AbstractActionFilter.java:97)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
    at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:64)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
    at io.fabric8.elasticsearch.plugin.ActionForbiddenActionFilter.apply(ActionForbiddenActionFilter.java:48)
    at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)
    at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82)
    at org.elasticsearch.client.node.NodeIndicesAdminClient.execute(NodeIndicesAdminClient.java:77)
    at org.elasticsearch.client.FilterClient$IndicesAdmin.execute(FilterClient.java:120)
    at org.elasticsearch.rest.BaseRestHandler$HeadersAndContextCopyClient$IndicesAdmin.execute(BaseRestHandler.java:149)
    at org.elasticsearch.client.support.AbstractIndicesAdminClient.stats(AbstractIndicesAdminClient.java:524)
    at org.elasticsearch.rest.action.cat.RestShardsAction$1.processResponse(RestShardsAction.java:68)
    at org.elasticsearch.rest.action.cat.RestShardsAction$1.processResponse(RestShardsAction.java:63)
    at org.elasticsearch.rest.action.support.RestActionListener.onResponse(RestActionListener.java:49)
    at org.elasticsearch.action.support.TransportAction$ThreadedActionListener$1.run(TransportAction.java:113)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:748)

It seems to hit Bug 1369924.

Environment

  • OpenShift Enterprise 3.2
  • ElasticSearch 1.5.2
  • Installed 3 nodes ES cluster which consist of 1 ES master and 2 ES data nodes.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content