Unable to register system to Redhat Subscription Management (RHSM): errors out with "SSLError: certificate verify failed"
Environment
- Red Hat Enterprise Linux (RHEL)
- 9
- 8
- 7
- 6
- 5.7 onwards
- Red Hat Subscription Manager (RHSM)
Issue
- Unable to register system to Redhat Subscription Managment (RHSM), errors out with "SSLError: certificate verify failed"
SSLError: certificate verify failed
error while registering the system with RHSM.
Resolution
NOTE: The steps below require signing in as root and using a command prompt. If you are booted to a GUI interface, select the terminal application from your system menus to access a command prompt.
Basic testing
Before and between each troubleshooting step below, you can check basic connectivity with SSL using the openssl
and curl
commands as follows. For the openssl
command, use Control+C to close the connection when done:
# openssl s_client -connect subscription.rhn.redhat.com:443 -CAfile /etc/rhsm/ca/redhat-uep.pem
# curl -v -u [rhnusername] --cacert /etc/rhsm/ca/redhat-uep.pem https://subscription.rhn.redhat.com/subscription/users/[rhnusername]/owners
or
# openssl s_client -connect subscription.rhsm.redhat.com:443 -CAfile /etc/rhsm/ca/redhat-uep.pem
# curl -v -u [rhnusername] --cacert /etc/rhsm/ca/redhat-uep.pem https://subscription.rhsm.redhat.com/subscription/users/[rhnusername]/owners
1. Check the firewall rules (iptables or firewalld)
Check for any internal firewall rules blocking the access to Red Hat sites or not allowing traffic over port 443. For a quick check to determine if this is a firewall issue, disable the local service:
RHEL5 and RHEL6:
# service iptables stop
RHEL7 and later:
# systemctl stop firewalld
If subscription-manager register
works after this, we know that your firewall settings need to be updated to allow for TCP traffic over port 443. See the following solutions for more information:
How do I access RHSM (yum/up2date) through a firewall?
For RHSM, you will need to allow TCP traffic over port 443 with the following Internet resources:
- For registration (RHSM, subscription-manager): subscription.rhn.redhat.com or subscription.rhsm.redhat.com
- For updates (yum): cdn.redhat.com
2. Check the time setting on the system
SSL depends on appropriate date and time ranges. Make sure your system has the current time and date.
# grep ZONE /etc/sysconfig/clock
The time should match between the TZ time and the current date/time
# date
RHEL5 and RHEL6:
# ntpq -p
RHEL7 and later:
# chronyc sources
# chronyc tracking
Last command help checking for any jitter.
3. Check intranet and proxy configuration
Be sure your local network has the routes and SSL proxy rules it needs to connect.
4. Check with firewall/proxy administrators to see if any HTTPS inspection is being performed
HTTPS inspection by firewalls/proxies is known to cause these sorts of problems with subscription-manager. As can re-signing SSL communications (similar to man-in-the-middle attack).
Further steps for troubleshooting RHSM firewall/proxy issues can be found here and here.
If the corporate firewall is performing SSL inspection, the following changes will be needed:
- Disable SSL inspections on
*.redhat.com
. - Whitelist Red Hat's CA Certificate
redhat-uep.pem
in Firewall's Certificate so these CAs will not get conflicted while receiving traffic via*.redhat.com
.
See the following solution for more information: Common networking issues while accessing Red Hat Subscription Management (RHSM)
See the article for understanding the use of Red Hat's CA certificate to establish HTTPS connections with Red Hat's CDN
5. Reinstall the root certificate
You can reinstall the root certificate by reinstalling the python-rhsm
package on your system. Since updates must be done offline until your registration is successful, you can do the following:
-
Search for and download
python-rhsm
from the package downloads page at the Customer Portal: https://access.redhat.com/downloads/content/package-browser -
Copy the RPM package file to the RHEL server.
-
Install the package using rpm as described here:
How do I install or upgrade an RPM package?
Note
If you have another server running RHEL 6 or higher you can use yumdownloader
.
Before you can do this, on a similar system running RHEL 6 run the yumdownloader
command.
The example below shows a certain version, the package number will increase based on time.
# mkdir /tmp/python-rhsm;cd /tmp/python-rhsm
# yumdownloader --resolve subscription-manager\*
Then scp the folder:
tar -czvf /tmp/python-rhsm/python-rhsm.tar.gz /tmp/python-rhsm
scp /tmp/python-rhsm.tar.gz root@xxx.xxx.xxx.xxx:/tmp/
Log into the server, where you will remove the python-rhsm
package
# yum remove python-rhsm
# ls -la /tmp/python-rhsm.tar.gz
# mkdir -p /tmp/python-rhsm
# tar -zxvf python-rhsm.tar.gz
# cd python-rhsm
# yum install python-rhsm-1.14.3-1.el6.x86_64 subscription-manager-1.14.10-1.el6.x86_64.rpm subscription-manager-firstboot-1.14.10-1.el6.x86_64.rpm subscription-manager-gui-1.14.10-1.el6.x86_64.rpm
Root Cause
- SSL failures can be caused by multiple issues, such as firewall being present between the systems that cause certificates to fail, or time inaccuracies on the systems in question.
- For example, firewalls performing HTTPS inspection can cause the certificate verify failed error with subscription-manager. Check the firewall and disable https inspection for the RHSM client if that is the case.
Diagnostic Steps
Results from curl
command:
# curl -v -u rhnusername --cacert /etc/rhsm/ca/redhat-uep.pem https://subscription.rhn.redhat.com/subscription/users/rhnusername/owners
Enter host password for user 'rhnusername':
* About to connect() to subscription.rhn.redhat.com port 443 (#0)
* Trying xx.xx.xx.xx... connected
* Connected to subscription.rhn.redhat.com (xx.xx.xx.xx) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/rhsm/ca/redhat-uep.pem
CApath: none
* Issuer certificate is invalid: 'E=ca-support@redhat.com,CN=subscription.rhn.redhat.com,OU=Red Hat Network,O="Red Hat, Inc.",ST=North Carolina,C=US'
* NSS error -8156
* Closing connection #0
* Peer certificate cannot be authenticated with known CA certificates
curl: (60) Peer certificate cannot be authenticated with known CA certificates
More details here: http://curl.haxx.se/docs/sslcerts.html
...
Logs from /var/log/rhsm/rhsm.log
2011-12-27 08:41:07,641 [INFO] @connection.py:350 - Using certificate authentication: key = /etc/pki/consumer/key.pem, cert = /etc/pki/consumer/cert.pem, ca = /etc/rhsm/ca/, insecure = False
2011-12-27 08:41:07,641 [DEBUG] @connection.py:188 - Loading CA PEM certificates from: /etc/rhsm/ca/
2011-12-27 08:41:07,641 [DEBUG] @connection.py:171 - Loading CA certificate: '/etc/rhsm/ca/redhat-uep.pem'
2011-12-27 08:41:07,642 [DEBUG] @connection.py:171 - Loading CA certificate: '/etc/rhsm/ca/candlepin-stage.pem'
2011-12-27 08:41:07,642 [WARNING] @connection.py:383 - Error fetching supported resources, this UEPConnection is likely not usable:
2011-12-27 08:41:07,642 [ERROR] @connection.py:385 - No such file or directory
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 376, in _load_supported_resources
resources_list = self.conn.request_get("/")
File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 252, in request_get
return self._request("GET", method)
File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 191, in _request
context.load_cert(self.cert_file, keyfile=self.key_file)
File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Context.py", line 76, in load_cert
m2.ssl_ctx_use_cert(self.ctx, certfile)
SSLError: No such file or directory
2011-12-27 08:41:07,643 [INFO] @connection.py:362 - Connection Established: host: subscription.rhn.redhat.com, port: 443, handler: /subscription
2011-12-27 08:41:11,713 [INFO] @connection.py:339 - Using basic authentication as: isavia_sysdep
2011-12-27 08:41:11,713 [DEBUG] @connection.py:188 - Loading CA PEM certificates from: /etc/rhsm/ca/
2011-12-27 08:41:11,713 [DEBUG] @connection.py:171 - Loading CA certificate: '/etc/rhsm/ca/redhat-uep.pem'
2011-12-27 08:41:11,714 [DEBUG] @connection.py:171 - Loading CA certificate: '/etc/rhsm/ca/candlepin-stage.pem'
2011-12-27 08:41:11,714 [DEBUG] @connection.py:209 - Making request: GET /subscription/
2011-12-27 08:41:12,456 [WARNING] @connection.py:383 - Error fetching supported resources, this UEPConnection is likely not usable:
2011-12-27 08:41:12,456 [ERROR] @connection.py:385 - certificate verify failed
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 376, in _load_supported_resources
resources_list = self.conn.request_get("/")
File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 252, in request_get
return self._request("GET", method)
File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 212, in _request
conn.request(request_type, handler, body=body, headers=headers)
File "/usr/lib64/python2.6/httplib.py", line 914, in request
self._send_request(method, url, body, headers)
File "/usr/lib64/python2.6/httplib.py", line 951, in _send_request
self.endheaders()
File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
self._send_output()
File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
self.send(msg)
File "/usr/lib64/python2.6/httplib.py", line 739, in send
self.connect()
File "/usr/lib64/python2.6/site-packages/M2Crypto/httpslib.py", line 50, in connect
self.sock.connect((self.host, self.port))
File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 185, in connect
ret = self.connect_ssl()
File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 178, in connect_ssl
return m2.ssl_connect(self.ssl, self._timeout)
SSLError: certificate verify failed
2011-12-27 08:41:12,457 [INFO] @connection.py:362 - Connection Established: host: subscription.rhn.redhat.com, port: 443, handler: /subscription
2011-12-27 08:41:12,458 [DEBUG] @connection.py:188 - Loading CA PEM certificates from: /etc/rhsm/ca/
2011-12-27 08:41:12,459 [DEBUG] @connection.py:171 - Loading CA certificate: '/etc/rhsm/ca/redhat-uep.pem'
2011-12-27 08:41:12,459 [DEBUG] @connection.py:171 - Loading CA certificate: '/etc/rhsm/ca/candlepin-stage.pem'
2011-12-27 08:41:12,460 [DEBUG] @connection.py:209 - Making request: GET /subscription/users/isavia_sysdep/owners
2011-12-27 08:41:13,104 [ERROR] @managercli.py:65 - Error during registration: certificate verify failed
2011-12-27 08:41:13,104 [ERROR] @managercli.py:66 - certificate verify failed
Traceback (most recent call last):
File "/usr/share/rhsm/subscription_manager/managercli.py", line 600, in _do_command
owner_key = self._determine_owner_key(admin_cp)
File "/usr/share/rhsm/subscription_manager/managercli.py", line 678, in _determine_owner_key
owners = cp.getOwnerList(self.username)
File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 488, in getOwnerList
return self.conn.request_get(method)
File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 252, in request_get
return self._request("GET", method)
File "/usr/lib/python2.6/site-packages/rhsm/connection.py", line 212, in _request
conn.request(request_type, handler, body=body, headers=headers)
File "/usr/lib64/python2.6/httplib.py", line 914, in request
self._send_request(method, url, body, headers)
File "/usr/lib64/python2.6/httplib.py", line 951, in _send_request
self.endheaders()
File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
self._send_output()
File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
self.send(msg)
File "/usr/lib64/python2.6/httplib.py", line 739, in send
self.connect()
File "/usr/lib64/python2.6/site-packages/M2Crypto/httpslib.py", line 50, in connect
self.sock.connect((self.host, self.port))
File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 185, in connect
ret = self.connect_ssl()
File "/usr/lib64/python2.6/site-packages/M2Crypto/SSL/Connection.py", line 178, in connect_ssl
return m2.ssl_connect(self.ssl, self._timeout)
SSLError: certificate verify failed
Verify there aren't certificates installed locally that might be causing this:
# certutil -L -d sql:/etc/pki/nssdb/
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments