Loki Ingester pods failing to flush chunks with status 0/1 in RHOCP 4

Issue

• Loki logging pods are not functioning in the Production environment.
• Attempts to restart the Loki Ingester pods but unsuccessful being with STATUS: 0/1
• Logging data is unavailable due to this issue.

• Loki ingester pods are in STATUS: 0/1 failing to write the chunks to the storage with the below errors:

  2025-06-12T06:47:31.609274594Z level=error ts=2025-06-12T06:47:31.588812771Z caller=flush.go:143 org_id=application msg="failed to flush" err="failed to flush chunks: store put chunk: InternalError: We encountered an internal error. Please try again.\n\tstatus code: 500, request id: mbt0nxm7-9uyf8d-2g8, host id: worker.example.com, num_chunks: 1, labels: {kubernetes_container_name=\"compliance\", kubernetes_host=\"ingress-2.worker.example.com\", kubernetes_namespace_name=\"rhacs-operator\", kubernetes_pod_name=\"collector-hvkqc\", log_type=\"application\"}"
  

  level=error ts=2025-09-13T00:45:25.417652578Z caller=flush.go:143 org_id=infrastructure msg="failed to flush" err="failed to flush chunks: store put chunk: SerializationError: failed to unmarshal error message\n\tstatus code: 504, request id: , host id: \ncaused by: UnmarshalError: failed to unmarshal error