RGW fails with 404 NoSuchKey ERROR and unable to download existing objects from the cluster

Issue

• s3cmd client get request is failing with 404 (NoSuchKey) Error when we try to get the object from the cluster:

  s3cmd -c s3cfg.admin get s3://<file>
  download: 's3://<file>' -> '<file>'  [1 of 1]
     524288 of 471561718     0% in    0s     6.78 MB/s  failed
  WARNING: Retrying failed request: <file> (EOF from S3!)
  WARNING: Waiting 3 sec...
  download: 's3://<file>' -> '<file>'  [1 of 1]
  ERROR: S3 error: 404 (NoSuchKey)
  

• The object is known to exist in the cluster but can't be retrieved or accessed from the client side on a healthy cluster.