unable to sync container repositories

Issue

• We already had Openstack 16.0 repos enabled and synced and with 16.1 we would have expected to receive new blobs and tags.

• Instead it looks like we cannot sync anything, it looks like a auth issue, but we are using the same token used since the beginning of activities.

• It looks like we can fetch and download tasks but not containers blob

• On one repo we even tried to reset the password, same result. Could you please check if the redhat registry is correcly working?

Jul 30 15:54:48 satellite.localdomain pulp[2568]: urllib3.connectionpool:INFO: [188914a1] Starting new HTTPS connection (1): registry.redhat.io
Jul 30 15:54:48 satellite.localdomain pulp[2558]: nectar.downloaders.threaded:INFO: Download succeeded: https://registry.redhat.io/v2/rhosp-rhel8/openstack-barbican-keystone-listener/blobs/sha256:9762e1a78efd0afe031f97761621e22bd0128d3bfa3e7fc846d7daeb493602dd.
Jul 30 15:54:49 satellite.localdomain pulp[2558]: nectar.downloaders.threaded:INFO: Download succeeded: https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?scope=repository%3Arhosp-rhel8%2Fopenstack-barbican-keystone-listener%3Apull&service=docker-registry.
Jul 30 15:54:49 satellite.localdomain pulp[2568]: nectar.downloaders.threaded:INFO: [188914a1] Download failed: Download of https://registry.redhat.io/v2/ failed with code 401: Unauthorized
Jul 30 15:54:49 satellite.localdomain pulp[2568]: urllib3.connectionpool:INFO: [188914a1] Starting new HTTPS connection (1): registry.redhat.io
Jul 30 15:54:50 satellite.localdomain pulp[2568]: nectar.downloaders.threaded:INFO: [188914a1] Download succeeded: https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth?scope=repository%3Arhosp-rhel8%2Fopenstack-heat-api-cfn%3Apull&service=docker-registry.
Jul 30 15:54:50 satellite.localdomain pulp[2568]: urllib3.connectionpool:INFO: [188914a1] Starting new HTTPS connection (2): registry.redhat.io
Jul 30 15:54:51 satellite.localdomain pulp[2568]: nectar.downloaders.threaded:INFO: [188914a1] Download succeeded: https://registry.redhat.io/v2/.
Jul 30 15:54:52 satellite.localdomain pulp[2568]: nectar.downloaders.threaded:INFO: [188914a1] Download succeeded: https://registry.redhat.io/v2/rhosp-rhel8/openstack-heat-api-cfn/tags/list.
Jul 30 15:54:52 satellite.localdomain pulp[2568]: nectar.downloaders.threaded:INFO: [188914a1] Download succeeded: https://registry.redhat.io/v2/rhosp-rhel8/openstack-heat-api-cfn/manifests/16.0-93.
Jul 30 15:54:53 satellite.localdomain pulp[2559]: nectar.downloaders.threaded:INFO: Download succeeded: https://registry.redhat.io/v2/openshift4/ose-prometheus-node-exporter/blobs/sha256:e2db33997bdc8f74e3e21704c1c339b59c9fb3e7d7fa928e9a516af0882f9807.
Jul 30 15:54:53 satellite.localdomain pulp[2559]: nectar.downloaders.threaded:INFO: Download failed: Download of https://registry.redhat.io/v2/openshift4/ose-prometheus-node-exporter/blobs/sha256:ece1755091b846f110f60ce2669880ccd7626619eea765ac4326c30297c910e8 failed with code 401: Unauthorized
Jul 30 15:54:53 satellite.localdomain pulp[2559]: nectar.downloaders.threaded:INFO: Download failed: Download of https://registry.redhat.io/v2/openshift4/ose-prometheus-node-exporter/blobs/sha256:51090ebd7c598469d65c48c2ef26cf4b0cd09623f385e51b2efd88da24c8eda4 failed with code 401: Unauthorized
Jul 30 15:54:53 satellite.localdomain pulp[2559]: nectar.downloaders.threaded:INFO: Download failed: Download of https://registry.redhat.io/v2/openshift4/ose-prometheus-node-exporter/blobs/sha256:e07433601712a092b0f8945a5cf9165ee973ca43f790cccb13e745ae8c0ad177 failed with code 401: Unauthorized
Jul 30 15:54:53 satellite.localdomain pulp[2559]: nectar.downloaders.threaded:INFO: Download failed: Download of https://registry.redhat.io/v2/openshift4/ose-prometheus-node-exporter/blobs/sha256:910f3f3e2e177d27fc6e8f3049e60d0cd1c59c2eb9bf0f22bc2c29f38c140588 failed with code 401: Unauthorized