Number of `FluentD` or `Collector` pods are showing `failed to flush the buffer` error message

Solution Verified - Updated -

Issue

  • FluentD or Collector pods are throwing errors similar to the following:

    2022-01-28T05:59:48.087126221Z 2022-01-28 05:59:48 +0000 : [retry_default] failed to flush the buffer. retry_time=5929 next_retry_seconds=2022-01-28 06:00:44 +0000 chunk="5d66d22fbc13bf20a5da35252471a8f3" error_class=Fluent::Plugin::Elast
    icsearchOutput::RetryStreamEmitFailure error="buffer is full." 
    
  • FluentD or Collector pods are consuming high filesystem space under /var/lib/fluentd/
  • After getting the alert Elasticsearch Node Disk Flood Watermark Reached, fluentd will fail to deliver logs to Elasticsearch

Environment

  • OpenShift Container Platform (RHOCP)
  • Red Hat OpenShift Logging

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content