OpenShift logging fails to start after upgrade to 3.11.51 due to "node settings must not contain any index level settings"
Environment
- Red Hat OpenShift Container Platform
- 3.11.51
Issue
OpenShift Logging ElasticSearch pod fails to start due to the following Exception after upgrading to OpenShift 3.11.51:
[2019-01-08T22:20:24,106][WARN ][o.e.c.s.SettingsModule ] [logging-es-data-master-d468a043]
*************************************************************************************
Found index level settings on node level configuration.
Since elasticsearch 5.x index level settings can NOT be set on the nodes
configuration like the elasticsearch.yaml, in system properties or command line
arguments.In order to upgrade all indices the settings must be updated via the
/${index}/_settings API. Unless all settings are dynamic all indices must be closed
in order to apply the upgradeIndices created in the future should use index templates
to set default values.
Please ensure all required values are updated on all indices by executing:
curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
"index.number_of_replicas" : "0",
"index.number_of_shards" : "1",
"index.translog.flush_threshold_period" : "5m",
"index.translog.flush_threshold_size" : "256mb",
"index.unassigned.node_left.delayed_timeout" : "2m"
}'
*************************************************************************************
[2019-01-08T22:20:24,116][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [logging-es-data-master-d468a043] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: node settings must not contain any index level settings
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
Caused by: java.lang.IllegalArgumentException: node settings must not contain any index level settings
at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:132) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
at org.elasticsearch.node.Node.<init>(Node.java:344) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:233) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:233) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.13.redhat-1.jar:5.6.13.redhat-1]
... 6 more
Resolution
- Ensure the Aggregated Logging upgrade was successful as part of 3.11 upgrade. This issue occurs when an old ElasticSearch 2.x configuration is used with the ElasticSearch 5.6 version shipped with OpenShift 3.11.
- If the playbook fails and elasticsearch pods are in
crashloopbackoff
. Please take a backup of the configmap and deployment config of elasticsearch and remove it.
$ oc get cm logging-elasticsearch -n openshift-logging -o yaml > logging-elasticsearch_backup.txt
$ oc get dc logging-es-data-master-xxxx -n openshift-logging -o yaml > logging-es-data-master-xxxx_dc_backup.txt
- Once the backups are created, delete the configmap and deploymentconfig so that playbook can create a new configmap based on Elasticsearch 5.x config.
- Finally run the ansible-playbook and ensure it ends properly without errors.
Root Cause
ElasticSearch was updated to version 5.6 in OpenShift 3.11. OpenShift 3.10 and earlier were using 2.x or older. The error on this page can occur if the Aggregated Logging upgrade did not complete successfully or if the upgrade was not performed and the ElasticSearch image in DeploymentConfig was updated by hand.
Diagnostic Steps
- Confirm Aggregated Logging upgrade playbook was executed
- Check
logging-es
DeploymentConfig image version:
# oc get dc/logging-es-.... -o yaml
- Check
logging-es
ConfigMap for anyindex
configurations if also running ElasticSearch 5.x. Do not remove theseindex
configurations by hand but make sure you are running upgrade playbook: https://docs.openshift.com/container-platform/3.11/upgrading/automated_upgrades.html#upgrading-efk-logging-stack
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments