Unable to register system to Redhat Subscription Management (RHSM): errors out with "SSLError: certificate verify failed"

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL)
    • 9
    • 8
    • 7
    • 6
    • 5.7 onwards
  • Red Hat Subscription Manager (RHSM)

Issue

  • Unable to register system to Redhat Subscription Managment (RHSM), errors out with "SSLError: certificate verify failed"
  • SSLError: certificate verify failed error while registering the system with RHSM.

Resolution

NOTE: The steps below require signing in as root and using a command prompt. If you are booted to a GUI interface, select the terminal application from your system menus to access a command prompt.

Basic testing

Before and between each troubleshooting step below, you can check basic connectivity with SSL using the openssl and curl commands as follows. For the openssl command, use Control+C to close the connection when done:

# openssl s_client -connect subscription.rhn.redhat.com:443 -CAfile /etc/rhsm/ca/redhat-uep.pem
# curl -v -u [rhnusername] --cacert /etc/rhsm/ca/redhat-uep.pem https://subscription.rhn.redhat.com/subscription/users/[rhnusername]/owners

or

# openssl s_client -connect subscription.rhsm.redhat.com:443 -CAfile /etc/rhsm/ca/redhat-uep.pem
# curl -v -u [rhnusername] --cacert /etc/rhsm/ca/redhat-uep.pem https://subscription.rhsm.redhat.com/subscription/users/[rhnusername]/owners

1. Check the firewall rules (iptables or firewalld)

Check for any internal firewall rules blocking the access to Red Hat sites or not allowing traffic over port 443. For a quick check to determine if this is a firewall issue, disable the local service:

RHEL5 and RHEL6:

# service iptables stop

RHEL7 and later:

# systemctl stop firewalld

If subscription-manager register works after this, we know that your firewall settings need to be updated to allow for TCP traffic over port 443. See the following solutions for more information:
How do I access RHSM (yum/up2date) through a firewall?

For RHSM, you will need to allow TCP traffic over port 443 with the following Internet resources:

  • For registration (RHSM, subscription-manager): subscription.rhn.redhat.com or subscription.rhsm.redhat.com
  • For updates (yum): cdn.redhat.com

2. Check the time setting on the system

SSL depends on appropriate date and time ranges. Make sure your system has the current time and date.

# grep ZONE /etc/sysconfig/clock

The time should match between the TZ time and the current date/time

# date

RHEL5 and RHEL6:

# ntpq -p

RHEL7 and later:

# chronyc sources
# chronyc tracking 

Last command help checking for any jitter.

3. Check intranet and proxy configuration

Be sure your local network has the routes and SSL proxy rules it needs to connect.

4. Check with firewall/proxy administrators to see if any HTTPS inspection is being performed

HTTPS inspection by firewalls/proxies is known to cause these sorts of problems with subscription-manager. As can re-signing SSL communications (similar to man-in-the-middle attack).
Further steps for troubleshooting RHSM firewall/proxy issues can be found here and here.

If the corporate firewall is performing SSL inspection, the following changes will be needed:


  • Disable SSL inspections on *.redhat.com.
  • Whitelist Red Hat's CA Certificate redhat-uep.pem in Firewall's Certificate so these CAs will not get conflicted while receiving traffic via *.redhat.com.

    See the following solution for more information: Common networking issues while accessing Red Hat Subscription Management (RHSM)

    See the article for understanding the use of Red Hat's CA certificate to establish HTTPS connections with Red Hat's CDN

5. Reinstall the root certificate

You can reinstall the root certificate by reinstalling the python-rhsm package on your system. Since updates must be done offline until your registration is successful, you can do the following:

  • Search for and download python-rhsm from the package downloads page at the Customer Portal: https://access.redhat.com/downloads/content/package-browser

  • Copy the RPM package file to the RHEL server.

  • Install the package using rpm as described here:
    How do I install or upgrade an RPM package?

Note
If you have another server running RHEL 6 or higher you can use yumdownloader.
Before you can do this, on a similar system running RHEL 6 run the yumdownloader command.
The example below shows a certain version, the package number will increase based on time.

# mkdir /tmp/python-rhsm;cd /tmp/python-rhsm
# yumdownloader --resolve subscription-manager\*

Then scp the folder:

tar -czvf /tmp/python-rhsm/python-rhsm.tar.gz /tmp/python-rhsm
scp /tmp/python-rhsm.tar.gz root@xxx.xxx.xxx.xxx:/tmp/

Log into the server, where you will remove the python-rhsm package

# yum remove python-rhsm
# ls -la /tmp/python-rhsm.tar.gz
# mkdir -p /tmp/python-rhsm
# tar -zxvf python-rhsm.tar.gz
# cd python-rhsm
# yum install python-rhsm-1.14.3-1.el6.x86_64 subscription-manager-1.14.10-1.el6.x86_64.rpm subscription-manager-firstboot-1.14.10-1.el6.x86_64.rpm subscription-manager-gui-1.14.10-1.el6.x86_64.rpm

Root Cause

  • SSL failures can be caused by multiple issues, such as firewall being present between the systems that cause certificates to fail, or time inaccuracies on the systems in question.
  • For example, firewalls performing HTTPS inspection can cause the certificate verify failed error with subscription-manager. Check the firewall and disable https inspection for the RHSM client if that is the case.

Diagnostic Steps

Results from curl command:

# curl -v -u rhnusername --cacert /etc/rhsm/ca/redhat-uep.pem https://subscription.rhn.redhat.com/subscription/users/rhnusername/owners
Enter host password for user 'rhnusername':
* About to connect() to subscription.rhn.redhat.com port 443 (#0)
*   Trying xx.xx.xx.xx... connected
* Connected to subscription.rhn.redhat.com (xx.xx.xx.xx) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/rhsm/ca/redhat-uep.pem
  CApath: none
* Issuer certificate is invalid: 'E=ca-support@redhat.com,CN=subscription.rhn.redhat.com,OU=Red Hat Network,O="Red Hat, Inc.",ST=North Carolina,C=US'
* NSS error -8156
* Closing connection #0
* Peer certificate cannot be authenticated with known CA certificates
curl: (60) Peer certificate cannot be authenticated with known CA certificates
More details here: http://curl.haxx.se/docs/sslcerts.html
...

Logs from /var/log/rhsm/rhsm.log

    2011-12-27 08:41:07,641 [INFO]  @connection.py:350 - Using certificate authentication: key = /etc/pki/consumer/key.pem, cert = /etc/pki/consumer/cert.pem, ca = /etc/rhsm/ca/, insecure = False
    2011-12-27 08:41:07,641 [DEBUG] @connection.py:188 - Loading CA PEM certificates from: /etc/rhsm/ca/
    2011-12-27 08:41:07,641 [DEBUG] @connection.py:171 - Loading CA certificate: '/etc/rhsm/ca/redhat-uep.pem'
    2011-12-27 08:41:07,642 [DEBUG] @connection.py:171 - Loading CA certificate: '/etc/rhsm/ca/candlepin-stage.pem'
    2011-12-27 08:41:07,642 [WARNING]  @connection.py:383 - Error fetching supported resources, this UEPConnection is likely not usable:
    2011-12-27 08:41:07,642 [ERROR]  @connection.py:385 - No such file or directory
    Traceback (most recent call last):
      File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 376, in _load_supported_resources
        resources_list = self.conn.request_get("/")
      File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 252, in request_get
        return self._request("GET", method)
      File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 191, in _request
        context.load_cert(self.cert_file, keyfile=self.key_file)
      File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Context.py", line 76, in load_cert
        m2.ssl_ctx_use_cert(self.ctx, certfile)
    SSLError: No such file or directory
    2011-12-27 08:41:07,643 [INFO]  @connection.py:362 - Connection Established: host: subscription.rhn.redhat.com, port: 443, handler: /subscription
    2011-12-27 08:41:11,713 [INFO]  @connection.py:339 - Using basic authentication as: isavia_sysdep
    2011-12-27 08:41:11,713 [DEBUG]  @connection.py:188 - Loading CA PEM certificates from: /etc/rhsm/ca/
    2011-12-27 08:41:11,713 [DEBUG]  @connection.py:171 - Loading CA certificate: '/etc/rhsm/ca/redhat-uep.pem'
    2011-12-27 08:41:11,714 [DEBUG]  @connection.py:171 - Loading CA certificate: '/etc/rhsm/ca/candlepin-stage.pem'
    2011-12-27 08:41:11,714 [DEBUG]  @connection.py:209 - Making request: GET /subscription/
    2011-12-27 08:41:12,456 [WARNING]  @connection.py:383 - Error fetching supported resources, this UEPConnection is likely not usable:
    2011-12-27 08:41:12,456 [ERROR]  @connection.py:385 - certificate verify failed
    Traceback (most recent call last):
      File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 376, in _load_supported_resources
        resources_list = self.conn.request_get("/")
      File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 252, in request_get
        return self._request("GET", method)
      File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 212, in _request
        conn.request(request_type, handler, body=body, headers=headers)
      File "/usr/lib64/python2.6/httplib.py", line 914, in request
        self._send_request(method, url, body, headers)
      File "/usr/lib64/python2.6/httplib.py", line 951, in _send_request
        self.endheaders()
      File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
        self._send_output()
      File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
        self.send(msg)
      File "/usr/lib64/python2.6/httplib.py", line 739, in send
        self.connect()
      File "/usr/lib64/python2.6/site-packages/M2Crypto/httpslib.py", line 50, in connect
        self.sock.connect((self.host, self.port))
      File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 185, in connect
        ret = self.connect_ssl()
      File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 178, in connect_ssl
        return m2.ssl_connect(self.ssl, self._timeout)
    SSLError: certificate verify failed
    2011-12-27 08:41:12,457 [INFO]  @connection.py:362 - Connection Established: host: subscription.rhn.redhat.com, port: 443, handler: /subscription
    2011-12-27 08:41:12,458 [DEBUG]  @connection.py:188 - Loading CA PEM certificates from: /etc/rhsm/ca/
    2011-12-27 08:41:12,459 [DEBUG]  @connection.py:171 - Loading CA certificate: '/etc/rhsm/ca/redhat-uep.pem'
    2011-12-27 08:41:12,459 [DEBUG]  @connection.py:171 - Loading CA certificate: '/etc/rhsm/ca/candlepin-stage.pem'
    2011-12-27 08:41:12,460 [DEBUG]  @connection.py:209 - Making request: GET /subscription/users/isavia_sysdep/owners
    2011-12-27 08:41:13,104 [ERROR]  @managercli.py:65 - Error during registration: certificate verify failed
    2011-12-27 08:41:13,104 [ERROR]  @managercli.py:66 - certificate verify failed
    Traceback (most recent call last):
      File "/usr/share/rhsm/subscription_manager/managercli.py", line 600, in _do_command
        owner_key = self._determine_owner_key(admin_cp)
      File "/usr/share/rhsm/subscription_manager/managercli.py", line 678, in _determine_owner_key
        owners = cp.getOwnerList(self.username)
      File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 488, in getOwnerList
        return self.conn.request_get(method)
      File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 252, in request_get
        return self._request("GET", method)
      File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 212, in _request
        conn.request(request_type, handler, body=body, headers=headers)
      File "/usr/lib64/python2.6/httplib.py", line 914, in request
        self._send_request(method, url, body, headers)
      File "/usr/lib64/python2.6/httplib.py", line 951, in _send_request
        self.endheaders()
      File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
        self._send_output()
      File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
        self.send(msg)
      File "/usr/lib64/python2.6/httplib.py", line 739, in send
        self.connect()
      File "/usr/lib64/python2.6/site-packages/M2Crypto/httpslib.py", line 50, in connect
        self.sock.connect((self.host, self.port))
      File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 185, in connect
        ret = self.connect_ssl()
      File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 178, in connect_ssl
        return m2.ssl_connect(self.ssl, self._timeout)
    SSLError: certificate verify failed

Verify there aren't certificates installed locally that might be causing this:

# certutil -L -d sql:/etc/pki/nssdb/

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments