Loki Ingester pod complain InvalidBucketState in RHOCP4

Issue

• Loki ingester pod has error related to the state of the ODF bucket.

  level=error ts=2024-09-02T06:56:22.272728198Z caller=flush.go:178 component=ingester loop=26 org_id=application msg=“failed to flush” retries=2 err=“failed to flush chunks: store put chunk: InvalidBucketState: The request is not valid with the current state of the bucket.\n\tstatus code: 409, request id: node-abc.xyz, host id: node-abc.xyz, num_chunks: 1, labels: {kubernetes_container_name=\“abc-xyz\“, kubernetes_host=\“xxxx-xxx-xx-xxx-infra\“, kubernetes_namespace_name=\“xxxx-xxxxxxx-xxxxxxxxxx-xxxxx-xxxx\“, kubernetes_pod_name=\“xxxx-xxxxxxx-xxxxxxxxxx-xxxxx-xxxx\“, log_type=\“application\“, service_name=\“unknown_service\“}”
  

• noobaa-default-backing-store status is fluctuating between Ready and Rejected.

  $ oc get backingstore -n openshift-storage
  
  NAME                                                TYPE        PHASE   AGE
  noobaa-default-backing-store                        pv-pool     Ready   17d
  
  After a while
  
  $ oc get backingstore -n openshift-storage
  
  NAME                                                TYPE        PHASE      AGE
  noobaa-default-backing-store                        pv-pool     Rejected   17d