How to renew and redeploy internal logging certificates in OpenShift 3

Solution Verified - Updated -

Issue

  • fluentd is not sending any container logs to ElasticSearch;
  • Internal logging certificates expired.
  • Elasticsearch pods have readinessProbe errors:

    Warning  Unhealthy  16s (x59 over 5m)  kubelet, infra1.example.com  Readiness probe failed: Elasticsearch node is not ready to accept HTTP requests yet [response code: 000]
    

    Logs show:

    [2019-12-01 00:00:00,000][ERROR][container.run            ] Timed out waiting for Elasticsearch to be ready
    cat: elasticsearch_connect_log.txt: No such file or directory
    

    Elasticsearch logs (logging-es.log) reports failed certificate with errors:

    javax.net.ssl.SSLHandshakeException: General SSLEngine problem
    Caused by: java.security.cert.CertPathValidatorException: validity check failed
    Caused by: java.security.cert.CertificateExpiredException: NotAfter: Sun Dec 08 16:51:12 UTC 2019
    
  • ES seems to be stuck at early boot, only few logs and CrashLoop

    [INFO ][container.run            ] Setting heap dump location /elasticsearch/persistent/heapdump.hprof
    [INFO ][container.run            ] Checking if Elasticsearch is ready on https://localhost:9200
    

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 3.11
    • 3.10
    • 3.9
    • 4.x

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content