Fluentd pod keeps restarting and stuck in CrashLoopBackOff

Solution Verified - Updated -

Issue

  • All the logging-fluentd-xxxxx pods in openshift-logging namespace keeps crashing.
logging-fluentd-jj77j                         0/1       CrashLoopBackOff    1131       4d        10.0.0.1     node02.example.com    <none>
logging-fluentd-lwp6z                         0/1       CrashLoopBackOff    549        4d      10.0.0.2    node01.example.com    <none>
logging-fluentd-s9nsp                         0/1       CrashLoopBackOff    834        4d        10.0.0.3    node03.example.com    <none>
  • Docker service logs on the node show below error messages:
Aug 22 02:03:17 node1.example.com dockerd-current[17000]: time="2020-08-22T02:03:17.722980022-04:00" level=error msg="Handler for POST /v1.26/containers/create?name=k8s_fluentd-elasticsearch_logging-fluentd-mksxs_openshift-logging_bc70e819-ec22-11e9-b59b-566f2fcb01c7_6332 returned error: Conflict. The container name \"/k8s_fluentd-elasticsearch_logging-fluentd-mksxs_openshift-logging_bc70e819-ec22-11e9-b59b-566f2fcb01c7_6332\" is already in use by container 05a8e05ccbac065ff191ae840f07a21918b26afe30047008eb7dea4d8feef06c. You have to remove (or rename) that container to be able to reuse that name."
  • Fluentd pod logs show below error messages:
2020-08-23 11:14:43 -0400 [error]: unexpected error error="getaddrinfo: Name or service not known"
  2020-08-23 11:14:43 -0400 [error]: /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/plugin/out_forward.rb:508:in `getaddrinfo'
  2020-08-23 11:14:43 -0400 [error]: /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/plugin/out_forward.rb:508:in `resolve_dns!'
  2020-08-23 11:14:43 -0400 [error]: /usr/share/gems/gems/fluentd-0.12.43/lib/fluent/plugin/out_forward.rb:494:in `resolved_host'
  • Restarting of fluentd pods manually does not help.

Environment

  • Red Hat OpenShift Container Platform
    • 3.11

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content