How to renew and redeploy internal logging certificates in OpenShift

Issue

• fluentd is not sending any container logs to ElasticSearch.
• Internal logging certificates expired.

• Elasticsearch pods have readinessProbe errors:

  Warning  Unhealthy  16s (x59 over 5m)  kubelet, infra1.example.com  Readiness probe failed: Elasticsearch node is not ready to accept HTTP requests yet [response code: 000]
  

  Logs show:

  [2019-12-01 00:00:00,000][ERROR][container.run            ] Timed out waiting for Elasticsearch to be ready
  cat: elasticsearch_connect_log.txt: No such file or directory
  

  Elasticsearch logs (logging-es.log) reports failed certificate with errors:

  javax.net.ssl.SSLHandshakeException: General SSLEngine problem
  Caused by: java.security.cert.CertPathValidatorException: validity check failed
  Caused by: java.security.cert.CertificateExpiredException: NotAfter: Sun Dec 08 16:51:12 UTC 2019
  

• ES seems to be stuck at early boot, with only a few logs and CrashLoop

  [INFO ][container.run            ] Setting heap dump location /elasticsearch/persistent/heapdump.hprof
  [INFO ][container.run            ] Checking if Elasticsearch is ready on https://localhost:9200