Private Automation Hub container pull fails with "received unexpected HTTP status: 500 Internal Server Error" or digest mismatch
Issue
- Automation Controller project sync or manual container pull from Private Automation Hub fails when fetching an execution environment or container image layer
-
Pull fails with a digest mismatch when the layer file on disk does not match the expected checksum:
Error: unable to copy from source docker://<hub-host>/<image>:<tag>: writing blob: storing blob to file "...": happened during read: Digest did not match, expected sha256:<expected>, got sha256:<actual> -
Pull fails with HTTP 500 when the layer file is missing from storage but still referenced in the Hub database:
Error: unable to copy from source docker://<hub-host>/<image>:<tag>: reading blob sha256:<digest>: fetching blob: received unexpected HTTP status: 500 Internal Server Error -
Private Automation Hub content service logs show a missing artifact path:
Exception: Expected path '/var/lib/pulp/media/artifact/<xx>/<rest-of-sha256>' is not found -
Re-pushing the same image or re-running Hub repair alone does not restore the layer while the database record for that digest remains
Environment
- Red Hat Ansible Automation Platform 2.6 (RPM, Containerized, or Operator)
- Hub/Pulp artifact media under
/var/lib/pulp(or the container volume that mounts it):- Shared storage: multi-node / HA Hub where each node mounts the same filesystem (commonly NFS) at
/var/lib/pulp - Local storage: single-node Hub where
/var/lib/pulpis on the host disk or a node-local volume (not shared with other Hub nodes)
- Shared storage: multi-node / HA Hub where each node mounts the same filesystem (commonly NFS) at
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.