Red Hat Satellite Capsule installation fails with ssl error: "SSL_connect returned=1 errno=0 state=error: certificate verify failed) for Capsule https://capsule.lab.example.com:9090/features"

Solution Verified - Updated -

Environment

  • Red Hat Satellite 6.*
  • Red Hat Satellite Capsule 6.*

Issue

  • Red Hat Satellite Capsule installtation fails with ssl error: SSL_connect returned=1 errno=0 state=error: certificate verify failed) for Capsule https://capsule.lab.example.com:9090/features Please check the Capsule is configured and running on the host.
  • While trying to install Red Hat Satellite Capsule, it fails with following error messages:

    [DEBUG 2018-03-23 08:17:20 main] Exit with status code: 6 (signal was 6)
    [ERROR 2018-03-23 08:17:20 main] Errors encountered during run:
    [ERROR 2018-03-23 08:17:20 main]  Proxy capsule.lab.example.com cannot be registered: Unable to communicate with the Capsule: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed) for Capsule https://capsule.lab.example.com:9090/features Please check the Capsule is configured and running on the host.
    [ERROR 2018-03-23 08:17:20 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:23:in `create'
    [ERROR 2018-03-23 08:17:20 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:16:in `block in defaultvalues'
    [ERROR 2018-03-23 08:17:20 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property.rb:488:in `set'
    [ERROR 2018-03-23 08:17:20 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property.rb:564:in `sync'
    [ERROR 2018-03-23 08:17:20 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:236:in `sync'
    [ERROR 2018-03-23 08:17:20 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:134:in `sync_if_neede
    

Resolution

No custom certificates in use

  • Verify that Satellite and Capsule server are on the same version.
  • Generate a new Capsule certificate on Red Hat Satellite as follows:

    [root@satellite ~]# capsule-certs-generate --foreman-proxy-fqdn capsule.lab.example.com \
    --certs-tar capsule.lab.example.com-certs.tar --certs-update-all
    
  • Copy the archive file to the Red Hat Satellite Capsule.

  • Re-run the satellite-installer on the Red Hat Satellite Capsule.

Custom certificates are used:

In case custom certificates are used and the Capsule should use a newly introduced sub-CA, this additional sub-CA will be required to be added to Satellite's CAs as well.

  • Add the sub-CA to the already existing CA-bundle by simple concatenation of the all CAs into one file of new CA-bundle.
  • Update Satellite's CA by:

    • running katello-certs-checkwith the Satellite's cert/key and the new CA-bundle,
    • and then executing satellite-installer command as shown in the output of katello-certs-check

      [root@satellite ~]# katello-certs-check -c -k -b
      ...

      [root@satellite ~]# satellite-installer --scenario satellite ....

  • Re-run katello-certs-check with Capsule's cert/key and the new CA-chain, followed by capsule-certs-generate as shown in the output of katello-certs-check

    [root@satellite ~]# katello-certs-check -c <capsule-cert> -k <capsule-key> -b <ca-bundle>
    ...
    capsule-certs-generate --foreman-proxy-fqdn "$CAPSULE"....
    
  • Copy the archive to the Red Hat Satellite Capsule.

  • Re-run the satellite-installer on the Red Hat Satellite Capsule

Root Cause

  • Satellite and Capsule are not on the same version.
  • Capsule Certificates tar is not correctly created from Satellite server or capsule certificates are corrupted.
  • Additional sub-CA, missing on Satellite used for Capsule Certificate.

Diagnostic Steps

  • satellite-installer fails on Capsule host as follows:

     Proxy capsule.lab.example.com cannot be registered: Unable to communicate with the Capsule: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed) for Capsule https://capsule.lab.example.com:9090/features Please check the Capsule is configured and running on the host.
    /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:23:in `create'
    /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:16:in `block in defaultvalues
    
  • Besides the message shown above, a connection to the foreman-proxy service, using openssl will reveal information of the installed certificate and the possible lack of trust:

    [root@capsule ~]# openssl s_client -connect capsule.lab.example.com:9090
    CONNECTED(00000003)
    depth=2 O = EXAMPLE.COM, CN = Certificate Authority
    verify return:1
    depth=1 O = EXAMPLE.COM, CN = Sub Certificate Authority
    verify return:1
    depth=0 O = EXAMPLE.COM, CN = capsule.lab.example.com
    verify return:1
    ---
    Certificate chain
     0 s:/O=EXAMPLE.COM/CN=capsule.lab.example.com
       i:/O=EXAMPLE.COM/CN=Sub Certificate Authority
     1 s:/O=EXAMPLE.COM/CN=Sub Certificate Authority
       i:/O=EXAMPLE.COM/CN=Certificate Authority
     2 s:/O=EXAMPLE.COM/CN=Certificate Authority
       i:/O=EXAMPLE.COM/CN=Certificate Authority
    ---
    
  • All certificates of a Certificate Authority will be required to be included into the ca-bundle - including all sub-CA. satellite-installer will make sure that those are trusted system-wide. If the openssl s_client command is failing, this is an indicator that the Satellite provided list of CAs is not complete and would need to be adjusted.

If Satellite's configuration is missing the additional sub-CA, the resulting files /etc/pki/katello/certs/katello-server-ca.crt and /etc/pki/ca-trust/source/anchors/katello_server-host-cert.crt found on the capsule will be lacking the sub-CA entry.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

8 Comments

Hello,

I have tried the solution you provided above but I still get the same error. Is there any other way to fix this error?

Thanks, -Mahdi

No one at RedHat has a solution for the issue?

In our case the redo on the certs file did not fix the problem. It turns out that, despite assurances to the contrary, port 9090 was not open between the Satellite and the Capsule.

I got "SyntaxError: JSON.parse: unexpected character at line 1 column 1 of the JSON data" on Web UI

Hello,

I have tried the solution provided here but I'm still getting the same error. Does anybody has another solution to fix this error? Thanks.

Have the same Issue with above Self-Signed Capsule Cert. Port 9090 is open from Satellite and Cert Trust is also sucessfull when testing with curl to curl https://:9090/features Satellite and Capsule are Version 6.5.2.1-1

  1. The guide should be a bit more clear as what does it mean to: Add the sub-CA to the already existing CA-bundle? there are no instructions on this

  2. What does the following mean? "Update Satellite's ca by running katello-certs-check" . Is this the smart-proxy or the main Satellite server?

My error in question: Command run: foreman-installer \ --scenario foreman-proxy-content \ --certs-tar-file "/root/smart-proxy.domain.com-certs.tar"\ --foreman-proxy-content-parent-fqdn "foreman.domain.com"\ --foreman-proxy-register-in-foreman "true"\ --foreman-proxy-foreman-base-url "https://foreman.domain.com"\ --foreman-proxy-trusted-hosts "foreman.domain.com"\ --foreman-proxy-trusted-hosts "wlb01620.domain.com"\ --foreman-proxy-oauth-consumer-key "STRING_HERE"\ --foreman-proxy-oauth-consumer-secret "OTHER_STRING_HERE"\ --puppet-server-foreman-url "https://smart-proxy.domain.com"

Error: [ERROR 2020-03-06T21:32:22 main] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[smart-proxy.domain.com]/ensure: change from 'absent' to 'present' failed: Proxy smart-proxy.domain.com cannot be registered: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException] : Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed (s elf signed certificate in certificate chain)) for proxy https://smart-proxy.domain.com:9090/v2/features Please check the proxy is configured and running on the host.

curl https://smart-proxy.domain.com:9090/v2/features -k

could not read client cert from environment

Hello, thanks for the feedback. I clarified both the points 1. and 2. in the solution text.

From the self signed certificate in certificate chain error, I suspect the CA bundle has a root self-signed certificate, that is not trusted by either Satellite or Capsule (I think the former is right). To let Satellite or Capsule to trust that CA, add the CA (either whole bundle, or at least the root, self-signed one) to system trusted CAs:

cp CA.crt /etc/pki/ca-trust/source/anchors
sudo update-ca-trust extract