Fluentd is unable to send logs to the internal Elasticsearch with an Internal Error and Code 500

Issue

• Fluentd is unable to send the logs to the internal Elasticsearch delivered with the Logging stack with the error:

  2021-02-09T01:28:40.005392772+00:00 stdout F   2021-02-09 01:28:40 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluent-plugin-elasticsearch-4.1.1/lib/fluent/plugin/out_elasticsearch.rb:1015:in `rescue in send_bulk'
  2021-02-09T01:28:40.103228631+00:00 stdout F   2021-02-09 01:28:40 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluent-plugin-elasticsearch-4.1.1/lib/fluent/plugin/out_elasticsearch.rb:977:in `send_bulk'
  2021-02-09T01:28:40.166087695+00:00 stdout F   2021-02-09 01:28:40 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluent-plugin-elasticsearch-4.1.1/lib/fluent/plugin/out_elasticsearch.rb:804:in `block in write'
  2021-02-09T01:28:40.166195367+00:00 stdout F   2021-02-09 01:28:40 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluent-plugin-elasticsearch-4.1.1/lib/fluent/plugin/out_elasticsearch.rb:803:in `each'
  2021-02-09T01:28:40.166219345+00:00 stdout F   2021-02-09 01:28:40 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluent-plugin-elasticsearch-4.1.1/lib/fluent/plugin/out_elasticsearch.rb:803:in `write'
  2021-02-09T01:28:40.166219345+00:00 stdout F   2021-02-09 01:28:40 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:1125:in `try_flush'
  2021-02-09T01:28:40.166227211+00:00 stdout F   2021-02-09 01:28:40 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:1431:in `flush_thread_run'
  2021-02-09T01:28:40.166234132+00:00 stdout F   2021-02-09 01:28:40 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin/output.rb:461:in `block (2 levels) in start'
  2021-02-09T01:28:40.166241125+00:00 stdout F   2021-02-09 01:28:40 +0000 [warn]: /opt/rh/rh-ruby25/root/usr/local/share/gems/gems/fluentd-1.7.4/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
  2021-02-09T01:28:41.070095031+00:00 stdout F 2021-02-09 01:28:41 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=2 next_retry_seconds=2021-02-09 01:28:43 +0000 chunk="5badd34aa36abda87b94450afd6aa79d" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): [500] {\"code\":500,\"message\":\"Internal Error\",\"error\":{}}\n"
  

• Fluentd is not able to send logs to the internal Elasticsearch and the Elasticsearch proxy is throwing errors like:

  time="2021-02-09T01:28:38Z" level=info msg="Handling request \"authorization\""
  time="2021-02-09T01:28:38Z" level=info msg="Error processing request in handler authorization: Unable to determine username"
  

• On OCP4.5 we are observing fluentd pods, which are buffering a huge amount of logs and are not able anymore to push them to the ES cluster.
• fluentd is unable to process buffered logs due to authentication error against elasticsearch in OpenShift 4.5.
• Occasionally, fluentd is unable to send logs to elasticsearch because it is unable to authenticate to the elasticsearch-proxy.