Too far behind messages when log forwarding to Loki in RHOCP 4

Solution Verified - Updated -

Issue

  • Rejected messages in the collector when log forwarding to Loki with the error entry too far behind:

    openshift-logging/logging-loki-ingester-1[loki-ingester]: level=warn ts=2023-10-04T09:17:18.665393472Z caller=grpc_logging.go:43 method=/logproto.Pusher/Push duration=2.273379ms err="rpc error: code = Code(400) desc = entry with timestamp 2023-10-04 07:36:24.87037 +0000 UTC ignored, reason: 'entry too far behind, oldest acceptable timestamp is: 2023-10-04T08:17:16Z' for stream: {fluentd_thread=\"flush_thread_1\", kubernetes_host=\"woker-1.example.com\", log_type=\"audit\"},\nuser 'audit', total ignored: 1 out of 122" msg=gRPC
    

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Red Hat OpenShift Logging (RHOL)
    • 5.7
    • 5.8
  • Loki
  • Fluentd
  • Vector

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content