Fluentd is unable to send logs to the internal Elasticsearch with an Internal Error and Code 500
Issue
-
Fluentd
is unable to send the logs to the internalElasticsearch
delivered with theLogging
stack with the error:2021-02-09T01:28:40.005392772+00:00 stdout F 2021-02-09 01:28:40 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluent-plugin-elasticsearch-4.1.1/lib/fluent/plugin/out_elasticsearch.rb:1015:in `rescue in send_bulk' 2021-02-09T01:28:40.103228631+00:00 stdout F 2021-02-09 01:28:40 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluent-plugin-elasticsearch-4.1.1/lib/fluent/plugin/out_elasticsearch.rb:977:in `send_bulk' 2021-02-09T01:28:40.166087695+00:00 stdout F 2021-02-09 01:28:40 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluent-plugin-elasticsearch-4.1.1/lib/fluent/plugin/out_elasticsearch.rb:804:in `block in write' 2021-02-09T01:28:40.166195367+00:00 stdout F 2021-02-09 01:28:40 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluent-plugin-elasticsearch-4.1.1/lib/fluent/plugin/out_elasticsearch.rb:803:in `each' 2021-02-09T01:28:40.166219345+00:00 stdout F 2021-02-09 01:28:40 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluent-plugin-elasticsearch-4.1.1/lib/fluent/plugin/out_elasticsearch.rb:803:in `write' 2021-02-09T01:28:40.166219345+00:00 stdout F 2021-02-09 01:28:40 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:1125:in `try_flush' 2021-02-09T01:28:40.166227211+00:00 stdout F 2021-02-09 01:28:40 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:1431:in `flush_thread_run' 2021-02-09T01:28:40.166234132+00:00 stdout F 2021-02-09 01:28:40 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:461:in `block (2 levels) in start' 2021-02-09T01:28:40.166241125+00:00 stdout F 2021-02-09 01:28:40 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create' 2021-02-09T01:28:41.070095031+00:00 stdout F 2021-02-09 01:28:41 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=2 next_retry_seconds=2021-02-09 01:28:43 +0000 chunk="5badd34aa36abda87b94450afd6aa79d" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): [500] {\"code\":500,\"message\":\"Internal Error\",\"error\":{}}\n"
-
Fluentd
is not able to send logs to the internalElasticsearch
and theElasticsearch proxy
is throwing errors like:time="2021-02-09T01:28:38Z" level=info msg="Handling request \"authorization\"" time="2021-02-09T01:28:38Z" level=info msg="Error processing request in handler authorization: Unable to determine username"
- On OCP4.5 we are observing
fluentd
pods, which are buffering a huge amount of logs and are not able anymore to push them to the ES cluster. fluentd
is unable to process buffered logs due to authentication error againstelasticsearch
in OpenShift 4.5.- Occasionally,
fluentd
is unable to send logs toelasticsearch
because it is unable to authenticate to theelasticsearch-proxy
.
Environment
- Red Hat OpenShift Container Platform (RHOCP) 4.5
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.