Quay UI shows "Manifest not found" and a key error exception is generated in logs

Issue

• Quay user interface generates a Manifest not found error whenever any manifest details are requested. At the same time, the Tags view shows all tags and the image sizes.

• For each of the Manifest not found errors the following exception is generated in Quay logs:

  2024-07-01T15:45:57.616902350Z gunicorn-web stdout | 2024-07-01 15:45:57,615 [243] [ERROR] [data.registry_model.registry_oci_model] Could not list manifest layers `71335`
  2024-07-01T15:45:57.616902350Z gunicorn-web stdout | Traceback (most recent call last):
  2024-07-01T15:45:57.616902350Z gunicorn-web stdout |   File "/quay-registry/data/registry_model/registry_oci_model.py", line 601, in list_manifest_layers
  2024-07-01T15:45:57.616902350Z gunicorn-web stdout |     layers = self._list_manifest_layers(
  2024-07-01T15:45:57.616902350Z gunicorn-web stdout |   File "/quay-registry/data/registry_model/registry_oci_model.py", line 1175, in _list_manifest_layers
  2024-07-01T15:45:57.616902350Z gunicorn-web stdout |     requires_empty_blob = parsed.get_requires_empty_layer_blob(retriever)
  2024-07-01T15:45:57.616902350Z gunicorn-web stdout |   File "/quay-registry/image/oci/manifest.py", line 469, in get_requires_empty_layer_blob
  2024-07-01T15:45:57.616902350Z gunicorn-web stdout |     schema2_config = self._get_built_config(content_retriever)
  2024-07-01T15:45:57.616902350Z gunicorn-web stdout |   File "/quay-registry/image/oci/manifest.py", line 499, in _get_built_config
  2024-07-01T15:45:57.616902350Z gunicorn-web stdout |     config_bytes = content_retriever.get_blob_bytes_with_digest(self.config.digest)
  2024-07-01T15:45:57.616902350Z gunicorn-web stdout |   File "/quay-registry/data/model/oci/retriever.py", line 57, in get_blob_bytes_with_digest
  2024-07-01T15:45:57.616902350Z gunicorn-web stdout |     return self.storage.get_content(blob.locations, get_layer_path(blob))
  2024-07-01T15:45:57.616902350Z gunicorn-web stdout |   File "/quay-registry/storage/distributedstorage.py", line 23, in wrapper
  2024-07-01T15:45:57.616902350Z gunicorn-web stdout |     storage = self._storages[random.sample(locations, 1)[0]]
  2024-07-01T15:45:57.616902350Z gunicorn-web stdout | KeyError: 'local_us'
  2024-07-01T15:45:57.617067440Z gunicorn-web stdout | 2024-07-01 15:45:57,616 [243] [DEBUG] [endpoints.api.manifest] Missing layers for manifest `sha256:f60ecf8c4b17bfba5d70841e2df44d2e853269ed7c8b4c752285494070dcf8f1`