OpenShift logging fails to start after upgrade to 3.11.51 due to "node settings must not contain any index level settings"

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform
    • 3.11.51

Issue

OpenShift Logging ElasticSearch pod fails to start due to the following Exception after upgrading to OpenShift 3.11.51:

[2019-01-08T22:20:24,106][WARN ][o.e.c.s.SettingsModule   ] [logging-es-data-master-d468a043]
*************************************************************************************
Found index level settings on node level configuration.

Since elasticsearch 5.x index level settings can NOT be set on the nodes
configuration like the elasticsearch.yaml, in system properties or command line
arguments.In order to upgrade all indices the settings must be updated via the
/${index}/_settings API. Unless all settings are dynamic all indices must be closed
in order to apply the upgradeIndices created in the future should use index templates
to set default values.

Please ensure all required values are updated on all indices by executing:

curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
  "index.number_of_replicas" : "0",
  "index.number_of_shards" : "1",
  "index.translog.flush_threshold_period" : "5m",
  "index.translog.flush_threshold_size" : "256mb",
  "index.unassigned.node_left.delayed_timeout" : "2m"
}'
*************************************************************************************

[2019-01-08T22:20:24,116][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [logging-es-data-master-d468a043] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: node settings must not contain any index level settings
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
Caused by: java.lang.IllegalArgumentException: node settings must not contain any index level settings
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:132) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
        at org.elasticsearch.node.Node.<init>(Node.java:344) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
        at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:233) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
        ... 6 more

Resolution

  • Ensure the Aggregated Logging upgrade was successful as part of 3.11 upgrade. This issue occurs when an old ElasticSearch 2.x configuration is used with the ElasticSearch 5.6 version shipped with OpenShift 3.11.
  • If the playbook fails and elasticsearch pods are in crashloopbackoff. Please take a backup of the configmap and deployment config of elasticsearch and remove it.
$ oc get cm logging-elasticsearch -n openshift-logging -o yaml > logging-elasticsearch_backup.txt
$ oc get dc logging-es-data-master-xxxx -n openshift-logging -o yaml > logging-es-data-master-xxxx_dc_backup.txt
  • Once the backups are created, delete the configmap and deploymentconfig so that playbook can create a new configmap based on Elasticsearch 5.x config.
  • Finally run the ansible-playbook and ensure it ends properly without errors.

Root Cause

ElasticSearch was updated to version 5.6 in OpenShift 3.11. OpenShift 3.10 and earlier were using 2.x or older. The error on this page can occur if the Aggregated Logging upgrade did not complete successfully or if the upgrade was not performed and the ElasticSearch image in DeploymentConfig was updated by hand.

https://docs.openshift.com/container-platform/3.11/upgrading/automated_upgrades.html#upgrading-efk-logging-stack

Diagnostic Steps

  • Confirm Aggregated Logging upgrade playbook was executed
  • Check logging-es DeploymentConfig image version:
# oc get dc/logging-es-.... -o yaml

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments