Too far behind messages when log forwarding to Loki in RHOCP 4
Issue
-
Rejected messages in the collector when log forwarding to Loki with the error
entry too far behind
:openshift-logging/logging-loki-ingester-1[loki-ingester]: level=warn ts=2023-10-04T09:17:18.665393472Z caller=grpc_logging.go:43 method=/logproto.Pusher/Push duration=2.273379ms err="rpc error: code = Code(400) desc = entry with timestamp 2023-10-04 07:36:24.87037 +0000 UTC ignored, reason: 'entry too far behind, oldest acceptable timestamp is: 2023-10-04T08:17:16Z' for stream: {fluentd_thread=\"flush_thread_1\", kubernetes_host=\"woker-1.example.com\", log_type=\"audit\"},\nuser 'audit', total ignored: 1 out of 122" msg=gRPC
Environment
- Red Hat OpenShift Container Platform (RHOCP)
- 4
- Red Hat OpenShift Logging (RHOL)
- 5.7
- 5.8
- Loki
- Fluentd
- Vector
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.