podman pull intermittent failures (RHEL 9)

Posted on

Has anyone seen this? I'm trying to "openstack tripleo deploy" and keep getting intermittent failures. When I try to pull the docker image manually, it also fails intermittently with the following:

2024-02-04 01:22:56.267819 | 58ac789e-b5ea-544a-5a27-000000001701 | TASK | Pull registry.redhat.io/rhosp-rhel9/openstack-haproxy:17.0 image
2024-02-04 01:22:57.023842 | 58ac789e-b5ea-544a-5a27-000000001701 | FATAL | Pull registry.redhat.io/rhosp-rhel9/openstack-haproxy:17.0 image | all-in-one | error={"changed": true, "cmd": "podman pull registry.redhat.io/rhosp-rhel9/openstack-haproxy:17.0", "delta": "0:00:00.526133", "end": "2024-02-04 01:22:57.000332", "msg": "non-zero return code", "rc": 125, "start": "2024-02-04 01:22:56.474199", "stderr": "Trying to pull registry.redhat.io/rhosp-rhel9/openstack-haproxy:17.0...\nError: initializing source docker://registry.redhat.io/rhosp-rhel9/openstack-haproxy:17.0: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/articles/3399531", "stderr_lines": ["Trying to pull registry.redhat.io/rhosp-rhel9/openstack-haproxy:17.0...", "Error: initializing source docker://registry.redhat.io/rhosp-rhel9/openstack-haproxy:17.0: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/articles/3399531"], "stdout": "", "stdout_lines": []}

Here are 4 consecutive attempt pulling cinder-volume:17.0 with the 4th one failing (I cut out much of the output for consiseness):

[stack@localhost ~]$ podman pull registry.redhat.io/rhosp-rhel9/openstack-cinder-volume:17.0
Trying to pull registry.redhat.io/rhosp-rhel9/openstack-cinder-volume:17.0...
Getting image source signatures
Checking if image destination supports signatures
...
Copying config 6f7c2c25a9 done
Writing manifest to image destination
Storing signatures
6f7c2c25a95d385aaa550f82cc6542dc0c14b7202c4c09e80acc5eb477505541

[stack@localhost ~]$ pull registry.redhat.io/rhosp-rhel9/openstack-cinder-volume:17.0g.json registry.redhat.io
Trying to pull registry.redhat.io/rhosp-rhel9/openstack-cinder-volume:17.0...
Getting image source signatures
...
Copying config 6f7c2c25a9 done
Writing manifest to image destination
Storing signatures
6f7c2c25a95d385aaa550f82cc6542dc0c14b7202c4c09e80acc5eb477505541

[stack@localhost ~]$ podman pull registry.redhat.io/rhosp-rhel9/openstack-cinder-volume:17.0
Trying to pull registry.redhat.io/rhosp-rhel9/openstack-cinder-volume:17.0...
Getting image source signatures
Checking if image destination supports signatures
...
Copying config 6f7c2c25a9 done
Writing manifest to image destination
Storing signatures
6f7c2c25a95d385aaa550f82cc6542dc0c14b7202c4c09e80acc5eb477505541

[stack@localhost ~]$ podman pull registry.redhat.io/rhosp-rhel9/openstack-cinder-volume:17.0
Trying to pull registry.redhat.io/rhosp-rhel9/openstack-cinder-volume:17.0...
Error: initializing source docker://registry.redhat.io/rhosp-rhel9/openstack-cinder-volume:17.0: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/articles/3399531

I tried all the tricks I could find (adding [all:vars] to /etc/ansible/hosts, adding my RH credentials to $HOME/.docker/config.json, etc. Nothing works. It just keeps failing intermittently.

Thanks

DM

Responses