LokiStack per stream rate limited in RHOCP 4

Issue

• Loki Internal log is being stream rate limited.
• An alert reporting X% of records have resulted in an error by openshift-logging/collector-xxx collector component is active
• How to minimize the effort to continuously tune the ingestion per stream rate limit when forwarding logs to Loki.
• Automatically control the "per stream ingestion rate" in Loki.

• Loki Ingester pods are producing error messages similar to the following:

  level=warn ts=2024-08-07T15:38:10.37249059Z caller=grpc_logging.go:57 method=/logproto.Pusher/Push duration=749.707µs err="rpc error: code = Code(429) desc = entry with timestamp 2024-08-07 15:38:10.26457417 +0000 UTC ignored, reason: 'Per stream rate limit exceeded (limit: 5MB/sec) while attempting to ingest for stream '{kubernetes_container_name=\"<container name>\", kubernetes_host=\"<hostname>\", kubernetes_namespace_name=\"<namespace>\", kubernetes_pod_name=\"<pod name>\", log_type=\"<log type>\"}' totaling 4305B, consider splitting a stream via additional labels or contact your Loki administrator to see if the limit can be increased'