Thanos storage crashlooping showing the message `invalid magic number`

Issue

• The observability-thanos-compact-0 is crashlooping in open-cluster-management-observability namespace.

• Thanos storage invalid magic number in the system logs:

  err="create index header reader: write index header: new index reader: invalid magic number 31303030 for XABCXXXXXXXXXXXXXXXXXXX123/index"
  
  component=receive msg="changing probe status" status=not-ready reason="hashring has changed; server is not ready to receive requests"
  

• The logs indicate that while the requests are error-free, the uploads are being ignored. Examples, block XABCXXXXXXXXXXXXXXXXXXX123 successfully uploaded some files, but the meta file is missing. There are no error logs suggesting it couldn't be written to disk:

  YYYY-MM-DDT13:01:06.902945204Z level=info ts=YYYY-MM-DDT13:01:06.902818718Z caller=shipper.go:334 component=receive component=multi-tsdb tenant=abcd1234-efg5-678h-ijk9-1011lmno1213 msg="upload new block" id=XABCXXXXXXXXXXXXXXXXXXX123
  YYYY-MM-DDT13:01:07.609613205Z level=debug ts=YYYY-MM-DDT13:01:07.609515432Z caller=objstore.go:288 component=receive component=multi-tsdb tenant=abcd1234-efg5-678h-ijk9-1011lmno1213 msg="uploaded file" from=/var/thanos/receive/abcd1234-efg5-678h-ijk9-1011lmno1213/thanos/upload/XABCXXXXXXXXXXXXXXXXXXX123/chunks/000001 dst=XABCXXXXXXXXXXXXXXXXXXX123/chunks/000001 bucket="tracing: pdcocps3"
  YYYY-MM-DDT13:01:08.259098367Z level=debug ts=YYYY-MM-DDT13:01:08.258983054Z caller=objstore.go:288 component=receive component=multi-tsdb tenant=abcd1234-efg5-678h-ijk9-1011lmno1213 msg="uploaded file" from=/var/thanos/receive/abcd1234-efg5-678h-ijk9-1011lmno1213/thanos/upload/XABCXXXXXXXXXXXXXXXXXXX123/index dst=XABCXXXXXXXXXXXXXXXXXXX123/index bucket="tracing: pdcocps3"
  YYYY-MM-DDT13:01:08.276492603Z level=debug ts=YYYY-MM-DDT13:01:08.276374784Z caller=receive.go:637 component=receive component=uploader msg="upload phase done" uploaded=1 elapsed=1.408070757