Fluentd error end of file reached EOFError

Solution In Progress - Updated -

Issue

Kibana fails to contact ES:

2020-12-22T08:42:38.728054792Z Elasticsearch ERROR: 2020-12-22T08:42:38Z
2020-12-22T08:42:38.728054792Z   Error: Request error, retrying
2020-12-22T08:42:38.728054792Z   GET https://elasticsearch.openshift-logging.svc.cluster.local:9200/_opendistro/_security/authinfo => socket hang up
2020-12-22T08:42:38.728054792Z       at Log.error (/opt/app-root/src/node_modules/elasticsearch/src/lib/log.js:226:56)
2020-12-22T08:42:38.728054792Z       at checkRespForFailure (/opt/app-root/src/node_modules/elasticsearch/src/lib/transport.js:259:18)
2020-12-22T08:42:38.728054792Z       at HttpConnector.<anonymous> (/opt/app-root/src/node_modules/elasticsearch/src/lib/connectors/http.js:164:7)
2020-12-22T08:42:38.728054792Z       at ClientRequest.wrapper (/opt/app-root/src/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)
2020-12-22T08:42:38.728054792Z       at ClientRequest.emit (events.js:198:13)
2020-12-22T08:42:38.728054792Z       at TLSSocket.socketCloseListener (_http_client.js:373:11)
2020-12-22T08:42:38.728054792Z       at TLSSocket.emit (events.js:203:15)
2020-12-22T08:42:38.728054792Z       at _handle.close (net.js:607:12)
2020-12-22T08:42:38.728054792Z       at TCP.done (_tls_wrap.js:400:7)

Fluentd fails too:

2020-12-22T13:27:42.393114119Z 2020-12-22 13:27:42 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=15 next_retry_seconds=2020-12-22 13:32:45 +0000 chunk="5b70bc1ab64131268e616c47c423b18c" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): end of file reached (EOFError)"

Environment

  • Red Hat OpenShift Container Platform
    • 4.5

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In