Unable to register system to Redhat Subscription Management (RHSM): errors out with "SSLError: certificate verify failed"

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 5.7 onwards
  • Red Hat Subscription Manager (RHSM)

Issue

  • Unable to register system to Redhat Subscription Managment (RHSM), errors out with "SSLError: certificate verify failed"
  • SSLError: certificate verify failed error while registering the system with RHSM.

Resolution

NOTE: The steps below require signing in as root and using a command prompt. If you are booted to a GUI interface, select the terminal application from your system menus to access a command prompt.

Basic testing

Before and between each troubleshooting step below, you can check basic connectivity with SSL using the openssl and curl commands as follows. For the openssl command, use Control+C to close the connection when done:

# openssl s_client -connect subscription.rhn.redhat.com:443 -CAfile /etc/rhsm/ca/redhat-uep.pem
# curl -v -u [rhnusername] --cacert /etc/rhsm/ca/redhat-uep.pem https://subscription.rhn.redhat.com/subscription/users/[rhnusername]/owners

or

# openssl s_client -connect subscription.rhsm.redhat.com:443 -CAfile /etc/rhsm/ca/redhat-uep.pem
# curl -v -u [rhnusername] --cacert /etc/rhsm/ca/redhat-uep.pem https://subscription.rhsm.redhat.com/subscription/users/[rhnusername]/owners

1. Check the firewall rules (iptables or firewalld)

Check for any internal firewall rules blocking the access to Red Hat sites or not allowing traffic over port 443. For a quick check to determine if this is a firewall issue, disable the iptables service:

RHEL5 and RHEL6
# service iptables stop

RHEL7:
# systemctl stop firewalld

If subscription-manager register works after this, we know that your firewall settings need to be updated to allow for TCP traffic over port 443. See the following solutions for more information:
How do I access RHSM (yum/up2date) through a firewall?

For RHSM, you will need to allow TCP traffic over port 443 with the following Internet resources:

  • For registration (RHSM, subscription-manager): subscription.rhn.redhat.com or subscription.rhsm.redhat.com
  • For updates (yum): cdn.redhat.com

2. Check the time setting on the system

SSL depends on appropriate date and time ranges. Make sure your system has the current time and date.

# grep ZONE /etc/sysconfig/clock
The time should match between the TZ time and the current date/time
# date
RHEL5 and RHEL6:
# ntpq -p

RHEL7:
# chronyd sources
or
# chronyd tracking
(To check for any jitter)

3. Check intranet and proxy configuration

Be sure your local network has the routes and SSL proxy rules it needs to connect.

4. Check with firewall/proxy administrators to see if any HTTPS inspection is being performed

HTTPS inspection by firewalls/proxies is known to cause these sorts of problems with subscription-manager. As can re-signing SSL communications (similar to man-in-the-middle attack).
Further steps for troubleshooting RHSM firewall/proxy issues can be found here and here.

5. Reinstall the root certificate

You can reinstall the root certificate by reinstalling the python-rhsm package on your system. Since updates must be done offline until your registration is successful, you can do the following:

  • Search for and download python-rhsm from the package downloads page at the Customer Portal: https://access.redhat.com/downloads/content/package-browser

  • Copy the RPM package file to the RHEL server.

  • Install the package using rpm as described here:
    How do I install or upgrade an RPM package?

Note
If you have another server running rhel 6 or higher you can use yumdownloader
Before you can do this, on a similar system running Red Hat 6 run the yumdownloader command.
The example below shows a certain version, the package number will increase based on time.

# mkdir /tmp/python-rhsm;cd /tmp/python-rhsm
# yumdownloader --resolve subscription-manager\*

Then scp the folder 
tar -czvf /tmp/python-rhsm/python-rhsm.tar.gz /tmp/python-rhsm
scp /tmp/python-rhsm.tar.gz root@xxx.xxx.xxx.xxx:/tmp/

Log into the server, where you will remove the python-rhsm package

# yum remove python-rhsm
# ls -la /tmp/python-rhsm.tar.gz
# mkdir -p /tmp/python-rhsm
# tar -zxvf python-rhsm.tar.gz
cd into this directory 
# yum install python-rhsm-1.14.3-1.el6.x86_64 subscription-manager-1.14.10-1.el6.x86_64.rpm subscription-manager-firstboot-1.14.10-1.el6.x86_64.rpm subscription-manager-gui-1.14.10-1.el6.x86_64.rpm

Root Cause

  • SSL failures can be caused by multiple issues, such as firewall being present between the systems that cause certificates to fail, or time inaccuracies on the systems in question.
  • For example, firewalls performing HTTPS inspection can cause the certificate verify failed error with subscription-manager. Check the firewall and disable https inspection for the RHSM client if that is the case.

Diagnostic Steps

  • Results from OpenSSL test:
# openssl s_client -connect xmlrpc.rhn.redhat.com:443 -CAfile /usr/share/rhn/RHNS-CA-CERT
CONNECTED(00000003)
139883445217096:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 309 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---
  • Results from curl command:
# curl -v -u rhnusername --cacert /etc/rhsm/ca/redhat-uep.pem https://subscription.rhn.redhat.com/subscription/users/rhnusername/owners
Enter host password for user 'rhnusername':
* About to connect() to subscription.rhn.redhat.com port 443 (#0)
*   Trying 209.132.183.49... connected
* Connected to subscription.rhn.redhat.com (209.132.183.49) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/rhsm/ca/redhat-uep.pem
  CApath: none
* Issuer certificate is invalid: 'E=ca-support@redhat.com,CN=subscription.rhn.redhat.com,OU=Red Hat Network,O="Red Hat, Inc.",ST=North Carolina,C=US'
* NSS error -8156
* Closing connection #0
* Peer certificate cannot be authenticated with known CA certificates
curl: (60) Peer certificate cannot be authenticated with known CA certificates
More details here: http://curl.haxx.se/docs/sslcerts.html
...
  • Logs from /var/log/rhsm/rhsm.log
        2011-12-27 08:41:07,641 [INFO]  @connection.py:350 - Using certificate authentication: key = /etc/pki/consumer/key.pem, cert = /etc/pki/consumer/cert.pem, ca = /etc/rhsm/ca/, insecure = False
        2011-12-27 08:41:07,641 [DEBUG] @connection.py:188 - Loading CA PEM certificates from: /etc/rhsm/ca/
        2011-12-27 08:41:07,641 [DEBUG] @connection.py:171 - Loading CA certificate: '/etc/rhsm/ca/redhat-uep.pem'
        2011-12-27 08:41:07,642 [DEBUG] @connection.py:171 - Loading CA certificate: '/etc/rhsm/ca/candlepin-stage.pem'
        2011-12-27 08:41:07,642 [WARNING]  @connection.py:383 - Error fetching supported resources, this UEPConnection is likely not usable:
        2011-12-27 08:41:07,642 [ERROR]  @connection.py:385 - No such file or directory
        Traceback (most recent call last):
          File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 376, in _load_supported_resources
            resources_list = self.conn.request_get("/")
          File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 252, in request_get
            return self._request("GET", method)
          File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 191, in _request
            context.load_cert(self.cert_file, keyfile=self.key_file)
          File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Context.py", line 76, in load_cert
            m2.ssl_ctx_use_cert(self.ctx, certfile)
        SSLError: No such file or directory
        2011-12-27 08:41:07,643 [INFO]  @connection.py:362 - Connection Established: host: subscription.rhn.redhat.com, port: 443, handler: /subscription
        2011-12-27 08:41:11,713 [INFO]  @connection.py:339 - Using basic authentication as: isavia_sysdep
        2011-12-27 08:41:11,713 [DEBUG]  @connection.py:188 - Loading CA PEM certificates from: /etc/rhsm/ca/
        2011-12-27 08:41:11,713 [DEBUG]  @connection.py:171 - Loading CA certificate: '/etc/rhsm/ca/redhat-uep.pem'
        2011-12-27 08:41:11,714 [DEBUG]  @connection.py:171 - Loading CA certificate: '/etc/rhsm/ca/candlepin-stage.pem'
        2011-12-27 08:41:11,714 [DEBUG]  @connection.py:209 - Making request: GET /subscription/
        2011-12-27 08:41:12,456 [WARNING]  @connection.py:383 - Error fetching supported resources, this UEPConnection is likely not usable:
        2011-12-27 08:41:12,456 [ERROR]  @connection.py:385 - certificate verify failed
        Traceback (most recent call last):
          File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 376, in _load_supported_resources
            resources_list = self.conn.request_get("/")
          File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 252, in request_get
            return self._request("GET", method)
          File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 212, in _request
            conn.request(request_type, handler, body=body, headers=headers)
          File "/usr/lib64/python2.6/httplib.py", line 914, in request
            self._send_request(method, url, body, headers)
          File "/usr/lib64/python2.6/httplib.py", line 951, in _send_request
            self.endheaders()
          File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
            self._send_output()
          File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
            self.send(msg)
          File "/usr/lib64/python2.6/httplib.py", line 739, in send
            self.connect()
          File "/usr/lib64/python2.6/site-packages/M2Crypto/httpslib.py", line 50, in connect
            self.sock.connect((self.host, self.port))
          File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 185, in connect
            ret = self.connect_ssl()
          File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 178, in connect_ssl
            return m2.ssl_connect(self.ssl, self._timeout)
        SSLError: certificate verify failed
        2011-12-27 08:41:12,457 [INFO]  @connection.py:362 - Connection Established: host: subscription.rhn.redhat.com, port: 443, handler: /subscription
        2011-12-27 08:41:12,458 [DEBUG]  @connection.py:188 - Loading CA PEM certificates from: /etc/rhsm/ca/
        2011-12-27 08:41:12,459 [DEBUG]  @connection.py:171 - Loading CA certificate: '/etc/rhsm/ca/redhat-uep.pem'
        2011-12-27 08:41:12,459 [DEBUG]  @connection.py:171 - Loading CA certificate: '/etc/rhsm/ca/candlepin-stage.pem'
        2011-12-27 08:41:12,460 [DEBUG]  @connection.py:209 - Making request: GET /subscription/users/isavia_sysdep/owners
        2011-12-27 08:41:13,104 [ERROR]  @managercli.py:65 - Error during registration: certificate verify failed
        2011-12-27 08:41:13,104 [ERROR]  @managercli.py:66 - certificate verify failed
        Traceback (most recent call last):
          File "/usr/share/rhsm/subscription_manager/managercli.py", line 600, in _do_command
            owner_key = self._determine_owner_key(admin_cp)
          File "/usr/share/rhsm/subscription_manager/managercli.py", line 678, in _determine_owner_key
            owners = cp.getOwnerList(self.username)
          File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 488, in getOwnerList
            return self.conn.request_get(method)
          File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 252, in request_get
            return self._request("GET", method)
          File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 212, in _request
            conn.request(request_type, handler, body=body, headers=headers)
          File "/usr/lib64/python2.6/httplib.py", line 914, in request
            self._send_request(method, url, body, headers)
          File "/usr/lib64/python2.6/httplib.py", line 951, in _send_request
            self.endheaders()
          File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
            self._send_output()
          File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
            self.send(msg)
          File "/usr/lib64/python2.6/httplib.py", line 739, in send
            self.connect()
          File "/usr/lib64/python2.6/site-packages/M2Crypto/httpslib.py", line 50, in connect
            self.sock.connect((self.host, self.port))
          File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 185, in connect
            ret = self.connect_ssl()
          File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 178, in connect_ssl
            return m2.ssl_connect(self.ssl, self._timeout)
        SSLError: certificate verify failed
  • Verify there aren't certificates installed locally that might be causing this:
# certutil -L -d sql:/etc/pki/nssdb/

Certificate Nickname                                         Trust Attributes
                                                             SSL,S/MIME,JAR/XPI

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

7 Comments

I was assigned a new TCPIP address. I reconfigured and rebooted the system. After rebooting, I was able to register the system. I made no firewall changes but perhaps there were firewall rules associated with the other IP address, though this seems unlikely, since I just set that one up.

[root@lblxcbo1 ~]# subscription-manager register
Username: spectrum_health
Password:
The system has been registered with id: 6bdfe1ba-2948-4d81-a920-87a8ffc5e8e5

Heh, same here. A reboot fixed it :(

subscription-manager register --auto-attach --insecure

We do have commercial certificates form digicert for Satellite, now without --insecure I cannot register a host, I need to bring CA into the host before issuing subscription-manager resgister, any hint ?

errata: Step 5

Then scp the folder 
tar -czvf /tmp/python-rhsm/python-rhsm.tar.gz /tmp/python-rhsm

replace with:

Then scp the folder 
tar -czvf /tmp/python-rhsm.tar.gz /tmp/python-rhsm

I think the simplest way to fail is to not have a proxy defined. I think I would move that check up to first. Check /etc/rhsm/rhsm.conf for proxy settings. Most corporate users will be behind a firewall.

In the case # 02385906, I searched Red hat knowledge base and did not see any solution for error - gaierror: [Errno -2] Name or service not known. What I found this was due to wrong satellite FQDN in rhsm.conf file.

Editing that part was not enough. Since later host was giving another error as -

[ERROR] subscription-manager:30464:MainThread @managercli.py:174 Unable to perform refresh due to the following exception: certificate verify failed [ERROR] subscription-manager:30464:MainThread @managercli.py:175 - certificate verify failed ... ..... return m2.ssl_connect(self.ssl, self._timeout) SSLError: certificate verify failed

This error was due to katello-ca-consumer package pointing to wrong satellite host. Installing with katello-ca-consumer correct package resolved the issue.

Can we have knowledge base # 68657 diagnostic section edited with above certificate error too. And may we have a new solution for error gaierror: [Errno -2] Name or service not known mentioning that satellite name may not be correct?