Troubleshooting Authentication Issues with registry.redhat.io

Updated -

Red Hat-supported container images are moving from the existing Red Hat Registry (registry.access.redhat.com) to a new one (registry.redhat.io). With that move will come a change in the authentication needed to pull those container images. This article describes how to troubleshoot authentication issues.

Testing Authentication

Testing basic user authentication can be accomplished with the following command:

curl -Lv -u <username>:<password> "https://sso.redhat.com/auth/realms/rhcc/protocol/redhat-docker-v2/auth?service=docker-registry&client_id=curl&scope=repository:rhel:pull"

where <username> and <password> are the credentials you would normally use to log into the Red Hat Customer Portal. The <password> can be omitted, and curl will prompt for it interactively.

Testing basic auth with a user created token can be accomplished with the following command:

# curl -u $TOKENID:$SECRET "https://sso.redhat.com/auth/realms/rhcc/protocol/redhat-docker-v2/auth?service=docker-registry&client_id=curl&scope=repository:rhel:pull"

where

  • $TOKENID is a shell variable that contains the token name shown in the token management interface on the Customer Portal. This will be in the form of <account_number>|<name>. The <account_number> is your Red Hat Account number, and <name> is the name you gave to the token.
  • $SECRET is a shell variable that contains the very long token value

A successful authentication attempt will result in HTTP 200 OK and a JSON object like the following:

{"token":"<access_token>","access_token":"<access_token>","expires_in":300,"issued_at":"2018-08-13T21:28:03Z"}

where <access_token> is a very long access token value.

The following HTTP 401 Unauthorized error indicates that you may be attempting to use the wrong credentials to log in to the registry:

{"errors":[{"code":"UNAUTHORIZED","message":"Invalid username or password","detail":[{"type":"repository","name":"rhel","actions":["pull"]}]}]}

If using user credentials please ensure they are correct by attempting a new login to the Red Hat Customer Portal. If using a token please ensure that the correct ID and token value is being set. The secret value may not be entered correctly if you try to paste it into curl's interactive password prompt. Errors or responses other than 200 OK and 401 Unauthorized could indicate a network issue, such as a firewall, proxy, or other general network connectivity problems.

Proxies / Firewalls

Some systems may require or may already be configured to utilize a web (HTTP/HTTPS) proxy to access the Internet.

If your system requires the use of a web proxy to access external sites (like registry.redhat.io) please ensure the following:

  1. registry.redhat.io and sso.redhat.com should be whitelisted by your proxy and/or network firewall
  2. Either configure system-wide proxy settings or configure docker directly to use the proxy

If proxy settings are configured system-wide, then docker, skopeo (used by atomic, podman, and buildah), and curl will use these settings automatically. Otherwise, for testing purposes, you will need to explicitly tell curl to use your proxy by adding --proxy <proxy_address>:<proxy_port> to the test command. To check that curl is using your proxy you can look for the following in the output:

* About to connect() to proxy proxy.example.com port 8080 (#0)
*   Trying 192.168.2.100...
* Connected to proxy.example.com (192.168.2.100) port 8080 (#0)
* Establish HTTP proxy tunnel to sso.redhat.com:443
> CONNECT registry.redhat.io:443 HTTP/1.1
> Host: registry.redhat.io:443
> User-Agent: curl/7.29.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established

If curl fails to connect to your proxy or fails to tunnel to redhat.com this may indicate a problem with your proxy. If there is no proxy in use but curl still fails to connect to sso.redhat.com there may be a firewall on your system or network that is actively blocking access. In either case please first check with your network team to ensure that connections are allowed.

Using a registry auth access token to test connectivity to the registry

Once authentication is successful the access token can be used to test connectivity to the registry itself. This is an example of using a user-generated auth token to retrieve the container registry access token

curl -Lv -u $TOKENID:$SECRET "https://sso.redhat.com/auth/realms/rhcc/protocol/redhat-docker-v2/auth?service=docker-registry&client_id=curl&scope=repository:rhel:pull" | python -mjson.tool

The response from the SSO server will look similar to:

{
    "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwiandrIjogeyJrdHkiOiJSU0EiLCJhbGciOiJSUzI1NiIsInVzZSI6InNpZyIsIm4iOiIwZjd1Rk5CWW1wbE52S1ZES1I3cUtQVkZoWmg5ZllPQ1JkeWxxSi1jM0xSaEhXUGFDOU1ucjR6eUVXY25zaUxuTFcydmVicDFBTWpNWlVNMXhNeWQ4Z0pVdmE2YjgxUXlTcWpndDl5aWNhem5fZkY0M2VHcnNkYm5xZFpsQmYxUUl5LWhsd3hvdTNXUzhVVkJha3E4dHFCdDlISGc3VFRONFR1VHl2c18tODhFT3hZNklWeTI4Y3RxTTE2VzBXQmxFdU1yd1BPdGRXVGtEUkNjUzV0ZDhBa0Y4RlpMWV9nQ2M0WG5oVTI1Nnd3Y25MTUpGcGdaZjNMZ0k1NEJIMkR6LWVnSUc1RDNmQ0NLX0FOU1llNGZuZDRoM2hEQjBCWndhdzlYM0VyZkJHZjdGX0FQMlpnN1lKcUNhS0d0RGExNzBjb2J6MGxyRTI5Mjljd2p5dm9HbElfd2syWXlPMzJiZ3prZUJ4NndxVk4tNVFmV0syUjF5MFJFaWJGc3c2dGhQQkg5Q3lMR2I4VjdPdUdZZ0JIWkw3Q09ZdF9WdWZCRHRmMzlFamZOZ3FpdFZFelRQcWtWdldYQ1NOb2xUakk1RnJ3Wkh4VVJDQkpmUldYb3BKeE16Q3VDblMzM2ZSUG5FSm53WG84anZfTFA2NEhNcE93X0x5dEhKM0puZ1p0a3dtVzJ1TnM5c3pDOFcyUDNfOWFXWVhTMGl4YUlrMU5oSnBWZmRPeGlOb3lVSmhZYXl0THZKT2xSamgwZ3FtNm1UaFZXM3dkUzV1Qzh4TnB0LTE1b1BscWpHWEFfZXNRdENaendZdE5lUXdsTUJqOG1OTDNOMXB2bnZYd0NGTG1DOWpDcVRhNGhGdlpBUVU1c0RYM1VxU1JJZW1ZWk1iSEJpaXVsNnpLdXVXcyIsImUiOiJBUUFCIn19.eyJqdGkiOiJkNTc1MWE1NC02MTFkLTQyYzYtODIzNi03NzQ5NjE3ZjU1NTciLCJleHAiOjE1MzQxOTMwOTEsIm5iZiI6MTUzNDE5Mjc5MSwiaWF0IjoxNTM0MTkyNzkxLCJpc3MiOiJodHRwczovL3Nzby5zdGFnZS5yZWRoYXQuY29tL2F1dGgvcmVhbG1zL3JoY2MiLCJhdWQiOiJkb2NrZXItcmVnaXN0cnkiLCJzdWIiOiJ0ZXN0dXNlciIsInR5cCI6IkJlYXJlciIsImF6cCI6ImRvY2tlci1yZWdpc3RyeSIsImFjY2VzcyI6W119.wS0ytJ5ov0HB72Av8PwK74Ntwa6bDKFT_wqTbTVdYb_GOYSwE8WtQOECBavuKXLWfb3_mqhh7qroXHdDMMZhsqjgs8dNSD-mg2vv",
    "expires_in": 300,
    "issued_at": "2018-08-13T20:32:49Z",
    "token": "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwiandrIjogeyJrdHkiOiJSU0EiLCJhbGciOiJSUzI1NiIsInVzZSI6InNpZyIsIm4iOiIwZjd1Rk5CWW1wbE52S1ZES1I3cUtQVkZoWmg5ZllPQ1JkeWxxSi1jM0xSaEhXUGFDOU1ucjR6eUVXY25zaUxuTFcydmVicDFBTWpNWlVNMXhNeWQ4Z0pVdmE2YjgxUXlTcWpndDl5aWNhem5fZkY0M2VHcnNkYm5xZFpsQmYxUUl5LWhsd3hvdTNXUzhVVkJha3E4dHFCdDlISGc3VFRONFR1VHl2c18tODhFT3hZNklWeTI4Y3RxTTE2VzBXQmxFdU1yd1BPdGRXVGtEUkNjUzV0ZDhBa0Y4RlpMWV9nQ2M0WG5oVTI1Nnd3Y25MTUpGcGdaZjNMZ0k1NEJIMkR6LWVnSUc1RDNmQ0NLX0FOU1llNGZuZDRoM2hEQjBCWndhdzlYM0VyZkJHZjdGX0FQMlpnN1lKcUNhS0d0RGExNzBjb2J6MGxyRTI5Mjljd2p5dm9HbElfd2syWXlPMzJiZ3prZUJ4NndxVk4tNVFmV0syUjF5MFJFaWJGc3c2dGhQQkg5Q3lMR2I4VjdPdUdZZ0JIWkw3Q09ZdF9WdWZCRHRmMzlFamZOZ3FpdFZFelRQcWtWdldYQ1NOb2xUakk1RnJ3Wkh4VVJDQkpmUldYb3BKeE16Q3VDblMzM2ZSUG5FSm53WG84anZfTFA2NEhNcE93X0x5dEhKM0puZ1p0a3dtVzJ1TnM5c3pDOFcyUDNfOWFXWVhTMGl4YUlrMU5oSnBWZmRPeGlOb3lVSmhZYXl0THZKT2xSamgwZ3FtNm1UaFZXM3dkUzV1Qzh4TnB0LTE1b1BscWpHWEFfZXNRdENaendZdE5lUXdsTUJqOG1OTDNOMXB2bnZYd0NGTG1DOWpDcVRhNGhGdlpBUVU1c0RYM1VxU1JJZW1ZWk1iSEJpaXVsNnpLdXVXcyIsImUiOiJBUUFCIn19.eyJqdGkiOiJkNTc1MWE1NC02MTFkLTQyYzYtODIzNi03NzQ5NjE3ZjU1NTciLCJleHAiOjE1MzQxOTMwOTEsIm5iZiI6MTUzNDE5Mjc5MSwiaWF0IjoxNTM0MTkyNzkxLCJpc3MiOiJodHRwczovL3Nzby5zdGFnZS5yZWRoYXQuY29tL2F1dGgvcmVhbG1zL3JoY2MiLCJhdWQiOiJkb2NrZXItcmVnaXN0cnkiLCJzdWIiOiJ0ZXN0dXNlciIsInR5cCI6IkJlYXJlciIsImF6cCI6ImRvY2tlci1yZWdpc3RyeSIsImFjY2VzcyI6W119.wS0ytJ5ov0HB72Av8PwK74Ntwa6bDKFT_wqTbTVdYb_GOYSwE8WtQOECBavuKXLWfb3_mqhh7qroXHdDMMZhsqjgs8dNSD-mg2vv"
}

Take the access_token value, and pass it via the Authorization: Bearer <access_token> header like so:

curl -Lv -H "Authorization: Bearer $ACCESS_TOKEN" https://registry.redhat.io/v2/

A successful test will result in a HTTP 200 OK and an empty JSON object. This verifies that the generated access token is valid and, more importantly, that your system can access the registry endpoint.

3 Comments

It should be noted somewhere that capital letters in the service account name do not work.

I'm happy to report that this issue has been resolved. Capital/uppercase letters are allowed, but the service account name is now case-insensitive. For example, '1234|my-Token' and '1234|my-token' can be used interchangeably.

Should add the following to the troubleshooting steps:

HTTP Basic Auth with User Credentials

# curl -u $USERNAME:$PASSWORD "https://sso.redhat.com/auth/realms/rhcc/protocol/redhat-docker-v2/auth?service=docker-registry&client_id=curl&scope=repository:rhel:pull"

HTTP Basic Auth with Authentication Token

# curl -u $TOKENID:$SECRET "https://sso.redhat.com/auth/realms/rhcc/protocol/redhat-docker-v2/auth?service=docker-registry&client_id=curl&scope=repository:rhel:pull"

OAuth Direct Access Grant with User Credentials

# curl -X POST --data "username=$USERNAME&password=$PASSWORD&grant_type=password&service=docker-registry&client_id=curl-test&scope=repository:rhel:pull" https://sso.redhat.com/auth/realms/rhcc/protocol/redhat-docker-v2/auth

OAuth Direct Access Grant with Authentication Token

# curl -X POST --data "username=$TOKENID&password=$SECRET&grant_type=password&service=docker-registry&client_id=curl-test&scope=repository:rhel:pull" https://sso.redhat.com/auth/realms/rhcc/protocol/redhat-docker-v2/auth