Satellite 6: candlepin and candlepin_auth fail with response "404 Resource Not Found" and "Katello::Resources::Candlepin::CandlepinPing: 404 Resource Not Found" (SSLEngine problem)

Solution Verified - Updated -

Environment

  • Red Hat Satellite 6

Issue

  • candlepin and candlepin_auth fail on the Satellite server
# hammer ping
candlepin:      
    Status:          FAIL
    Server Response: Message: 404 Resource Not Found
candlepin_auth: 
    Status:          FAIL
    Server Response: Message: Katello::Resources::Candlepin::CandlepinPing: 404 Resource Not Found  (GET /candlepin/status)

Resolution

Satellite 6.6 and newer

  • Remove all Candlepin certificates:
# rm -f /etc/candlepin/certs/keystore
# rm -f /etc/candlepin/certs/amqp/*
  • Remove files in /etc/pki/katello/nssdb directory:
# rm -rf /etc/pki/katello/nssdb/*
  • Run satellite-installer
# satellite-installer --scenario satellite

Satellite 6.5 and older

  • Remove all Candlepin certificates:
# rm -f /etc/pki/katello/keystore
# rm -f /etc/candlepin/certs/amqp/*
  • Remove files in /etc/pki/katello/nssdb directory:
# rm -rf /etc/pki/katello/nssdb/*
  • Run satellite-installer
# satellite-installer --scenario satellite

For more KB articles/solutions related to Red Hat Satellite 6.x Candlepin Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Candlepin Issues

Root Cause

  • Candlepin certificates are not signed by the current certification authority "/etc/pki/katello/certs/katello-default-ca.crt"
Candlepin component:
  /etc/candlepin/certs/amqp/candlepin.truststore
  /etc/candlepin/certs/amqp/candlepin.jks
Tomcat:
Satellite 6.5 and older
  /etc/pki/katello/keystore
Satellite 6.6. and newer
  /etc/candlepin/certs/keystore

Diagnostic Steps

"hammer ping" shows the following status:

# hammer ping
candlepin:      
    Status:          FAIL
    Server Response: Message: 404 Resource Not Found
candlepin_auth: 
    Status:          FAIL
    Server Response: Message: Katello::Resources::Candlepin::CandlepinPing: 404 Resource Not Found  (GET /candlepin/status)
pulp:           
    Status:          ok
    Server Response: Duration: 32ms
pulp_auth:      
    Status:          ok
    Server Response: Duration: 16ms
elasticsearch:  
    Status:          ok
    Server Response: Duration: 16ms
foreman_tasks:  
    Status:          ok
    Server Response: Duration: 0ms

Discrepancy in "keyid" between certification authority "/etc/pki/katello/certs/katello-default-ca.crt" and "keystore":

# openssl x509 -text -in katello-default-ca.crt |grep keyid
                keyid:2A:75:E8:8B:21:43:A1:39:B5:C4:CB:6D:51:0D:1F:53:B6:0A:F6:BF

# keytool -list -v -keystore /etc/candlepin/certs/amqp/candlepin.truststore
...
#2: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 2A 75 E8 8B 21 43 A1 39   B5 C4 CB 6D 51 0D 1F 53  *u..!C.9...mQ..S
0010: B6 0A F6 BF
...

# keytool -v -list -keystore /etc/pki/katello/keystore --storetype PKCS12 -storepass $(sed -e '/keystorePass/!d' /etc/tomcat/server.xml -e 's/\s*keystorePass=//' -e "s/\"//g")
...
#2: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 2A 75 E8 8B 21 43 A1 39   B5 C4 CB 6D 51 0D 1F 53  *u..!C.9...mQ..S
0010: B6 0A F6 BF  
...

The following messages are logged to /var/log/candlepin/candlepin.log:

Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1348) ~[na:1.7.0_99]
        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:519) ~[na:1.7.0_99]
        at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1200) ~[na:1.7.0_99]
        at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1172) ~[na:1.7.0_99]
        at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) ~[na:1.7.0_99]
        at org.apache.qpid.transport.network.security.ssl.SSLSender.send(SSLSender.java:157) ~[qpid-common-0.30.redhat-1.jar:0.30.redhat-1]
        ... 48 common frames omitted
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[na:1.7.0_99]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1714) ~[na:1.7.0_99]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:281) ~[na:1.7.0_99]
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:273) ~[na:1.7.0_99]
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1472) ~[na:1.7.0_99]
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:213) ~[na:1.7.0_99]
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:913) ~[na:1.7.0_99]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:853) ~[na:1.7.0_99]
        at sun.security.ssl.Handshaker$1.run(Handshaker.java:851) ~[na:1.7.0_99]
        at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_99]
        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1285) ~[na:1.7.0_99]
        at org.apache.qpid.transport.network.security.ssl.SSLReceiver.doTasks(SSLReceiver.java:206) ~[qpid-common-0.30.redhat-1.jar:0.30.redhat-1]
        at org.apache.qpid.transport.network.security.ssl.SSLReceiver.received(SSLReceiver.java:165) ~[qpid-common-0.30.redhat-1.jar:0.30.redhat-1]
        at org.apache.qpid.transport.network.security.ssl.SSLReceiver.received(SSLReceiver.java:36) ~[qpid-common-0.30.redhat-1.jar:0.30.redhat-1]
        at org.apache.qpid.transport.network.io.IoReceiver.run(IoReceiver.java:161) ~[qpid-common-0.30.redhat-1.jar:0.30.redhat-1]
        ... 1 common frames omitted
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:350) ~[na:1.7.0_99]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:260) ~[na:1.7.0_99]
        at sun.security.validator.Validator.validate(Validator.java:260) ~[na:1.7.0_99]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326) ~[na:1.7.0_99]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:283) ~[na:1.7.0_99]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:138) ~[na:1.7.0_99]
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1459) ~[na:1.7.0_99]

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments