Number of `FluentD` or `Collector` pods are showing `failed to flush the buffer` error message
Issue
-
FluentD
orCollector
pods are throwing errors similar to the following:2022-01-28T05:59:48.087126221Z 2022-01-28 05:59:48 +0000 : [retry_default] failed to flush the buffer. retry_time=5929 next_retry_seconds=2022-01-28 06:00:44 +0000 chunk="5d66d22fbc13bf20a5da35252471a8f3" error_class=Fluent::Plugin::Elast icsearchOutput::RetryStreamEmitFailure error="buffer is full."
FluentD
orCollector
pods are consuming high filesystem space under/var/lib/fluentd/
- After getting the alert
Elasticsearch Node Disk Flood Watermark Reached
, fluentd will fail to deliver logs to Elasticsearch
Environment
- OpenShift Container Platform (RHOCP)
- Red Hat OpenShift Logging
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.