MessageSizeTooLarge when log forwarding to Kafka in RHOCP 4
Issue
-
After configuring the
clusterlogforwarder
to forward logs to external Kafka, the below logs are visible in fluentd pod:2021-07-14 17:32:45 +0000 [warn]: Send exception occurred: Kafka::MessageSizeTooLarge 2021-07-14 17:32:45 +0000 [warn]: Exception Backtrace : /usr/local/share/gems/gems/ruby-kafka-1.1.0/lib/kafka/protocol.rb:160:in `handle_error' 2021-07-14 17:32:45 +0000 [warn]: failed to flush the buffer. retry_time=20 next_retry_seconds=2021-07-14 17:33:48 +0000 chunk="5c542ac822c737b83fe4376514eca5b8" error_class=Kafka::MessageSizeTooLarge error="Kafka::MessageSizeTooLarge"
Environment
- Red Hat OpenShift Container Platform (RHOCP)
- 4.6 and newer versions
- Red Hat OpenShift Logging (RHOL)
- 5
- Fluentd
- Vector
- Kafka
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.