Red Hat Satellite Capsule installation or upgrade fails with ssl error: "SSL_connect returned=1 errno=0 state=error: certificate verify failed) for Capsule https://capsule.lab.example.com:9090/features"

Solution Verified - Updated -

Environment

  • Red Hat Satellite 6
  • Red Hat Satellite Capsule 6

Issue

  • Red Hat Satellite Capsule installation or upgrade fails with ssl error: SSL_connect returned=1 errno=0 state=error: certificate verify failed) for Capsule https://capsule.lab.example.com:9090/features Please check the Capsule is configured and running on the host.
  • While trying to install Red Hat Satellite Capsule, it fails with following error messages:

    [DEBUG 2018-03-23 08:17:20 main] Exit with status code: 6 (signal was 6)
    [ERROR 2018-03-23 08:17:20 main] Errors encountered during run:
    [ERROR 2018-03-23 08:17:20 main]  Proxy capsule.lab.example.com cannot be registered: Unable to communicate with the Capsule: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed) for Capsule https://capsule.lab.example.com:9090/features Please check the Capsule is configured and running on the host.
    [ERROR 2018-03-23 08:17:20 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:23:in `create'
    [ERROR 2018-03-23 08:17:20 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:16:in `block in defaultvalues'
    [ERROR 2018-03-23 08:17:20 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property.rb:488:in `set'
    [ERROR 2018-03-23 08:17:20 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property.rb:564:in `sync'
    [ERROR 2018-03-23 08:17:20 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:236:in `sync'
    [ERROR 2018-03-23 08:17:20 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:134:in `sync_if_neede
    

Resolution

No custom certificates in use

  • Make sure /root/ssl-build directory is not deleted on satellite.
  • Verify that Satellite and Capsule server are on the same version.
  • Generate a new Capsule certificate on Red Hat Satellite as follows:

    [root@satellite ~]# capsule-certs-generate --foreman-proxy-fqdn capsule.lab.example.com \
    --certs-tar capsule.lab.example.com-certs.tar --certs-update-all
    
  • Copy the archive file to the Red Hat Satellite Capsule.

  • Re-run the satellite-installer on the Red Hat Satellite Capsule.

Custom certificates are used:

In case custom certificates are used and the Capsule should use a newly introduced sub-CA, this additional sub-CA will be required to be added to Satellite's CAs as well.

  • Add the sub-CA to the already existing CA-bundle by simple concatenation of the all CAs into one file of new CA-bundle.
  • Update Satellite's CA by:

    • running katello-certs-checkwith the Satellite's cert/key and the new CA-bundle,
    • and then executing satellite-installer command as shown in the output of katello-certs-check

      [root@satellite ~]# katello-certs-check -c -k -b
      ...

      [root@satellite ~]# satellite-installer --scenario satellite ....

  • Re-run katello-certs-check with Capsule's cert/key and the new CA-chain, followed by capsule-certs-generate as shown in the output of katello-certs-check

    [root@satellite ~]# katello-certs-check -c <capsule-cert> -k <capsule-key> -b <ca-bundle>
    ...
    capsule-certs-generate --foreman-proxy-fqdn "$CAPSULE"....
    
  • Copy the archive to the Red Hat Satellite Capsule.

  • Re-run the generated satellite-installer command from katello-certs-check on the Red Hat Satellite Capsule
  • For an existing capsule, use the output command below To use them inside an EXISTING $CAPSULE, run this command INSTEAD:
  • Make sure the ISSUER is the same for both Red Hat Satellite and Capsule in the output of the below command:

    # openssl crl2pkcs7 -nocrl -certfile  /etc/foreman-proxy/foreman_ssl_ca.pem  | openssl pkcs7 -text -print_certs |egrep '(Issuer:|Subject:|CA:|DNS:|Digital|Not Before|Not After|keyid|serial:|TLS)'
    
    # openssl crl2pkcs7 -nocrl -certfile  /etc/foreman-proxy/ssl_cert.pem  | openssl pkcs7 -text -print_certs |egrep '(Issuer:|Subject:|CA:|DNS:|Digital|Not Before|Not After|keyid|serial:|TLS)'
    
  • For more KB articles/solutions related to Red Hat Satellite 6.x SSL Certificates Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x SSL Certificates Issues

  • For more KB articles/solutions related to Red Hat Satellite 6.x Installation/Upgrade/Update Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Installation/Upgrade/Update Issues.

Root Cause

  • Satellite and Capsule are not on the same version.
  • Capsule Certificates tar is not correctly created from Satellite server or capsule certificates are corrupted.
  • Additional sub-CA, missing on Satellite used for Capsule Certificate.

Diagnostic Steps

  • satellite-installer fails on Capsule host as follows:

     Proxy capsule.lab.example.com cannot be registered: Unable to communicate with the Capsule: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed) for Capsule https://capsule.lab.example.com:9090/features Please check the Capsule is configured and running on the host.
    /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:23:in `create'
    /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:16:in `block in defaultvalues
    
  • Besides the message shown above, a connection to the foreman-proxy service, using openssl will reveal information of the installed certificate and the possible lack of trust:

    [root@capsule ~]# openssl s_client -connect capsule.lab.example.com:9090
    CONNECTED(00000003)
    depth=2 O = EXAMPLE.COM, CN = Certificate Authority
    verify return:1
    depth=1 O = EXAMPLE.COM, CN = Sub Certificate Authority
    verify return:1
    depth=0 O = EXAMPLE.COM, CN = capsule.lab.example.com
    verify return:1
    ---
    Certificate chain
     0 s:/O=EXAMPLE.COM/CN=capsule.lab.example.com
       i:/O=EXAMPLE.COM/CN=Sub Certificate Authority
     1 s:/O=EXAMPLE.COM/CN=Sub Certificate Authority
       i:/O=EXAMPLE.COM/CN=Certificate Authority
     2 s:/O=EXAMPLE.COM/CN=Certificate Authority
       i:/O=EXAMPLE.COM/CN=Certificate Authority
    ---
    
  • All certificates of a Certificate Authority will be required to be included into the ca-bundle - including all sub-CA. satellite-installer will make sure that those are trusted system-wide. If the openssl s_client command is failing, this is an indicator that the Satellite provided list of CAs is not complete and would need to be adjusted.

  • If Satellite's configuration is missing the additional sub-CA, the resulting files /etc/pki/katello/certs/katello-server-ca.crt and /etc/pki/ca-trust/source/anchors/katello_server-host-cert.crt found on the capsule will be lacking the sub-CA entry.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments