Satellite 6: candlepin and candlepin_auth fail with response "404 Resource Not Found" and "Katello::Resources::Candlepin::CandlepinPing: 404 Resource Not Found" (SSLEngine problem)

Solution Verified - Updated -

Environment

  • Red Hat Satellite 6

Resolution

  • Remove all Candlepin certificates:
# rm -f /etc/pki/katello/keystore
# rm -f /etc/candlepin/certs/amqp/*
  • Remove files in /etc/pki/katello/nssdb directory:
# rm -rf /etc/pki/katello/nssdb/*
  • Run satellite-installer
# satellite-installer --scenario satellite

Root Cause

  • Candlepin certificates are not signed by the current certification authority "/etc/pki/katello/certs/katello-default-ca.crt"
Candlepin component:
  /etc/candlepin/certs/amqp/candlepin.truststore
  /etc/candlepin/certs/amqp/candlepin.jks
Tomcat:
  /etc/pki/katello/keystore

Diagnostic Steps

"hammer ping" shows the following status:

# hammer ping
candlepin:      
    Status:          FAIL
    Server Response: Message: 404 Resource Not Found
candlepin_auth: 
    Status:          FAIL
    Server Response: Message: Katello::Resources::Candlepin::CandlepinPing: 404 Resource Not Found  (GET /candlepin/status)
pulp:           
    Status:          ok
    Server Response: Duration: 32ms
pulp_auth:      
    Status:          ok
    Server Response: Duration: 16ms
elasticsearch:  
    Status:          ok
    Server Response: Duration: 16ms
foreman_tasks:  
    Status:          ok
    Server Response: Duration: 0ms

Discrepancy in "keyid" between certification authority "/etc/pki/katello/certs/katello-default-ca.crt" and "keystore":

# openssl x509 -text -in katello-default-ca.crt |grep keyid
                keyid:2A:75:E8:8B:21:43:A1:39:B5:C4:CB:6D:51:0D:1F:53:B6:0A:F6:BF

# keytool -list -v -keystore /etc/candlepin/certs/amqp/candlepin.truststore
...
#2: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 2A 75 E8 8B 21 43 A1 39   B5 C4 CB 6D 51 0D 1F 53  *u..!C.9...mQ..S
0010: B6 0A F6 BF
...

# keytool -v -list -keystore /etc/pki/katello/keystore --storetype PKCS12 -storepass $(sed -e '/keystorePass/!d' /etc/tomcat/server.xml -e 's/\s*keystorePass=//' -e "s/\"//g")
...
#2: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 2A 75 E8 8B 21 43 A1 39   B5 C4 CB 6D 51 0D 1F 53  *u..!C.9...mQ..S
0010: B6 0A F6 BF  
...

The following messages are logged to /var/log/candlepin/candlepin.log:

Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1348) ~[na:1.7.0_99]
        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:519) ~[na:1.7.0_99]
        at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1200) ~[na:1.7.0_99]
        at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1172) ~[na:1.7.0_99]
        at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) ~[na:1.7.0_99]
        at org.apache.qpid.transport.network.security.ssl.SSLSender.send(SSLSender.java:157) ~[qpid-common-0.30.redhat-1.jar:0.30.redhat-1]
        ... 48 common frames omitted
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[na:1.7.0_99]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1714) ~[na:1.7.0_99]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:281) ~[na:1.7.0_99]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:273) ~[na:1.7.0_99]
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1472) ~[na:1.7.0_99]
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:213) ~[na:1.7.0_99]
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:913) ~[na:1.7.0_99]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:853) ~[na:1.7.0_99]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:851) ~[na:1.7.0_99]
        at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_99]
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1285) ~[na:1.7.0_99]
        at org.apache.qpid.transport.network.security.ssl.SSLReceiver.doTasks(SSLReceiver.java:206) ~[qpid-common-0.30.redhat-1.jar:0.30.redhat-1]
        at org.apache.qpid.transport.network.security.ssl.SSLReceiver.received(SSLReceiver.java:165) ~[qpid-common-0.30.redhat-1.jar:0.30.redhat-1]
        at org.apache.qpid.transport.network.security.ssl.SSLReceiver.received(SSLReceiver.java:36) ~[qpid-common-0.30.redhat-1.jar:0.30.redhat-1]
        at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:161) ~[qpid-common-0.30.redhat-1.jar:0.30.redhat-1]
        ... 1 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:350) ~[na:1.7.0_99]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:260) ~[na:1.7.0_99]
        at sun.security.validator.Validator.validate(Validator.java:260) ~[na:1.7.0_99]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326) ~[na:1.7.0_99]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:283) ~[na:1.7.0_99]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:138) ~[na:1.7.0_99]
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1459) ~[na:1.7.0_99]

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.