Loki fails with "failed to create stream: no schema config found for time" in RHOCP 4

Issue

• Collector pods are reporting 500 internal server error connecting to Loki:

  2025-04-17T13:43:03.727880623Z 2025-04-17T13:43:03.727782Z  WARN sink{component_kind="sink" component_id=output_default_lokistack_infrastructure component_type=loki}: vector::sinks::util::retries: Retrying after error. error=Server responded with an error: 500 Internal Server Error internal_log_rate_limit=true
  

• Gateway pods show errors as:

  2025-04-17T13:43:09.634143531Z level=warn name=lokistack-gateway ts=2025-04-17T13:43:09.633998931Z caller=instrumentation.go:33 request=logging-loki-gateway-5d8c4976c4-jcf8h/vuX5TRGQpf-184838 proto=HTTP/1.1 method=POST status=500 content=application/x-protobuf path=/api/logs/v1/infrastructure/loki/api/v1/push duration=3.502279ms bytes=105
  

• Seeing the below warnings in the Loki Ingester pods:

  2025-04-17T13:43:10.279241716Z level=warn ts=2025-04-17T13:43:10.224285391Z caller=grpc_logging.go:76 method=/logproto.Pusher/Push duration=211.623µs msg=gRPC err="failed to create stream: no schema config found for time 1744596016.775"
  2025-04-17T13:43:10.479416586Z level=warn ts=2025-04-17T13:43:10.414651114Z caller=grpc_logging.go:76 method=/logproto.Pusher/Push duration=173.517µs msg=gRPC err="failed to create stream: no schema config found for time 1744593316.327"
  

• And in the Loki distributor pods:

  2025-04-17T13:43:07.380734570Z level=warn ts=2025-04-17T13:43:07.367436332Z caller=logging.go:128 orgID=infrastructure msg="POST /loki/api/v1/push (500) 6.791464ms Response: \"rpc error: code = Unknown desc = failed to create stream: no schema config found for time 1744594336.5\\n\" ws: false; Accept-Encoding: identity; Content-Encoding: snappy; Content-Length: 65086; Content-Type: application/x-protobuf; User-Agent: Vector/0.37.1 (x86_64-unknown-linux-gnu); X-Forwarded-For: 10.128.4.143; X-Forwarded-Prefix: /api/logs/v1/infrastructure; X-Scope-Orgid: infrastructure; "
  [...]
  level=error ts=2025-06-09T20:14:46.860228734Z caller=manager.go:50 component=distributor path=write msg="write operation failed" details="entry for stream '{k8s_container_name=\"node-exporter\", k8s_namespace_name=\"openshift-monitoring\", k8s_node_name=\"<node name>\", k8s_pod_name=\"node-exporter-wt7tn\", kubernetes_container_name=\"node-exporter\", kubernetes_host=\"<node name>\", kubernetes_namespace_name=\"openshift-monitoring\", kubernetes_pod_name=\"node-exporter-wt7tn\", log_type=\"infrastructure\", openshift_log_type=\"infrastructure\"}' has timestamp too old: 2025-04-08T23:53:42Z, oldest acceptable timestamp is: 2025-06-02T20:14:46Z" org_id=infrastructure