Loki discards logs silently in RHOCP 4

Issue

• Logs are missed from Loki, but not errors observed in the Loki pods

• In the Vector pods are visible errors with Server responded with an error: 400 Bad Request when log forwarding to Loki, but no errors are visible in the Loki pods. An example of the logs in the Vector side are:

  $ oc -n openshift-logging logs collector-pxjbn
  ...
  2024-06-28T13:55:35.875322Z ERROR sink{component_kind="sink" component_id=output_default_test component_type=loki}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] has been suppressed 1 times.
  2024-06-28T13:55:35.875450Z ERROR sink{component_kind="sink" component_id=output_default_test component_type=loki}: vector::sinks::util::retries: Non-retriable error; dropping the request. error=Server responded with an error: 400 Bad Request internal_log_rate_limit=true
  2024-06-28T13:55:35.875523Z ERROR sink{component_kind="sink" component_id=output_default_test component_type=loki}: vector_common::internal_event::service: Internal log [Service call failed. No retries or retries exhausted.] has been suppressed 1 times.
  2024-06-28T13:55:35.875615Z ERROR sink{component_kind="sink" component_id=output_default_test component_type=loki}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=Some(ServerError { code: 400 }) request_id=4 error_type="request_failed" stage="sending" internal_log_rate_limit=true
  

• When querying the metric loki_discarded_samples_total is observed that logs are discarded by Loki with the reason line_too_long