Fluentd error end of file reached EOFError

Solution Verified - Updated -

Issue

  • Kibana fails to contact Elasticsearch:

    2020-12-22T08:42:38.728054792Z Elasticsearch ERROR: 2020-12-22T08:42:38Z 
    2020-12-22T08:42:38.728054792Z   Error: Request error, retrying
    2020-12-22T08:42:38.728054792Z   GET https://elasticsearch.openshift-logging.svc.cluster.local:9200/_opendistro/_security/authinfo => socket hang up
    2020-12-22T08:42:38.728054792Z       at Log.error (/opt/app-root/src/node_modules/elasticsearch/src/lib/log.js:226:56)
    2020-12-22T08:42:38.728054792Z       at checkRespForFailure (/opt/app-root/src/node_modules/elasticsearch/src/lib/transport.js:259:18)
    2020-12-22T08:42:38.728054792Z       at HttpConnector.<anonymous> (/opt/app-root/src/node_modules/elasticsearch/src/lib/connectors/http.js:164:7)
    2020-12-22T08:42:38.728054792Z       at ClientRequest.wrapper (/opt/app-root/src/node_modules/elasticsearch/node_modules/lodash/lodash.js:4935:19)
    2020-12-22T08:42:38.728054792Z       at ClientRequest.emit (events.js:198:13)
    2020-12-22T08:42:38.728054792Z       at TLSSocket.socketCloseListener (_http_client.js:373:11)
    2020-12-22T08:42:38.728054792Z       at TLSSocket.emit (events.js:203:15)
    2020-12-22T08:42:38.728054792Z       at _handle.close (net.js:607:12)
    2020-12-22T08:42:38.728054792Z       at TCP.done (_tls_wrap.js:400:7)
    
  • Fluentd fails giving the error:

    2020-12-22T13:27:42.393114119Z 2020-12-22 13:27:42 +0000 [warn]: [clo_default_output_es] failed to flush the buffer. retry_time=15 next_retry_seconds=2020-12-22 13:32:45 +0000 chunk="5b70bc1ab64131268e616c47c423b18c" error_class=Fluent::Plugin::ElasticsearchOutput::RecoverableRequestFailure error="could not push logs to Elasticsearch cluster ({:host=>\"elasticsearch.openshift-logging.svc.cluster.local\", :port=>9200, :scheme=>\"https\", :user=>\"fluentd\", :password=>\"obfuscated\"}): end of file reached (EOFError)"
    

Environment

  • Red Hat OpenShift Container Platform
    • 4.5

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content