Access to Mirror Registry impossible with 503 forbidden

Posted on

I successfully installed the Mirror Registry for OpenShift respecting all the requirements explained in the documentation (VM with RHEL 8.5 with podman 3.4.2). At the end of the installation, I got the init user and its password and the URL to get access to the Mirror Registry. I can then login locally (in the VM) to the Mirror Registry with the following command : podman login -u init -p eDtjn627luMIFJ5g01xmYkAp3Ws8i4S8 mirror.ibm-technology-garage.cloud:8443 --tls-verify=false.

I received the following answer :

podman login -u init -p eDtjn627luMIFJ5g01xmYkAp3Ws8i4S8 mirror.ibm-technology-garage.cloud:8443 --tls-verify=false
Error: authenticating creds for "mirror.ibm-technology-garage.cloud:8443": pinging container registry mirror.ibm-technology-garage.cloud:8443: invalid status code from registry 503 (Service Unavailable)
I then decided to add a log level to the command to better understand this 503 return code.

Here is the answer:

podman login -u init -p eDtjn627luMIFJ5g01xmYkAp3Ws8i4S8 mirror.ibm-technology-garage.cloud:8443 --tls-verify=false --log-level debug
INFO[0000] podman filtering at log level debug
DEBU[0000] Called login.PersistentPreRunE(podman login -u init -p eDtjn627luMIFJ5g01xmYkAp3Ws8i4S9 mirror.ibm-technology-garage.cloud:8443 --tls-verify=false --log-level debug)
DEBU[0000] Merged system config "/usr/share/containers/containers.conf"
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/lib/containers/storage
DEBU[0000] Using run root /run/containers/storage
DEBU[0000] Using static dir /var/lib/containers/storage/libpod
DEBU[0000] Using tmp dir /run/libpod
DEBU[0000] Using volume path /var/lib/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] cached value indicated that overlay is supported
DEBU[0000] cached value indicated that metacopy is being used
DEBU[0000] cached value indicated that native-diff is not being used
INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=true
DEBU[0000] Initializing event backend file
DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument
DEBU[0000] Using OCI runtime "/usr/bin/runc"
INFO[0000] Found CNI network podman (type=bridge) at /etc/cni/net.d/87-podman.conflist
DEBU[0000] Default CNI network name podman is unchangeable
INFO[0000] Setting parallel job count to 7
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/001-rhel-shortnames.conf"
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf"
DEBU[0000] No credentials for mirror.ibm-technology-garage.cloud:8443 found
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/mirror.ibm-technology-garage.cloud:8443
DEBU[0000] GET https://mirror.ibm-technology-garage.cloud:8443/v2/
DEBU[0000] Ping https://mirror.ibm-technology-garage.cloud:8443/v2/ err Get "https://mirror.ibm-technology-garage.cloud:8443/v2/": Forbidden (&url.Error{Op:"Get", URL:"https://mirror.ibm-technology-garage.cloud:8443/v2/", Err:(errors.errorString)(0xc0004c87d0)})
DEBU[0000] GET http://mirror.ibm-technology-garage.cloud:8443/v2/
DEBU[0060] Ping http://mirror.ibm-technology-garage.cloud:8443/v2/ status 503
DEBU[0060] GET https://mirror.ibm-technology-garage.cloud:8443/v1/_ping
DEBU[0060] Ping https://mirror.ibm-technology-garage.cloud:8443/v1/_ping err Get "https://mirror.ibm-technology-garage.cloud:8443/v1/_ping": Forbidden (&url.Error{Op:"Get", URL:"https://mirror.ibm-technology-garage.cloud:8443/v1/_ping", Err:(errors.errorString)(0xc0003c47b0)})
DEBU[0060] GET http://mirror.ibm-technology-garage.cloud:8443/v1/_ping
DEBU[0120] Ping http://mirror.ibm-technology-garage.cloud:8443/v1/_ping status 503
Error: authenticating creds for "mirror.ibm-technology-garage.cloud:8443": pinging container registry mirror.ibm-technology-garage.cloud:8443: invalid status code from registry 503 (Service Unavailable)

Then it is not simple to go ahead with that issue. I also get connect remotely successfully from my MacBook using a podman 3.4.4. And from another VM with a podman 3.4.2 ... to the same host where I installed Mirror Registry. But when I try locally on the same VM running the Mirror Registry, then it fails with code 503 and the Forbidden return message. If you have any idea to troubleshot this ? Thanks.

NB I think I get something wrong here in the RHEL8 VM, may be this is related to /etc/hosts or some internal security.

PT

Responses