x509: certificate signed by unknown authority error in monitoring components fail after redeploying new certificates

Solution Verified - Updated -

Issue

  • After running redeploy-certificates.yml playbook monitoring components have started to fail and show errors about invalid certificates in their logs (similar to below).

    $ oc logs -n openshift-monitoring grafana-xxx -c grafana-proxy
    ...
    server.go:2923: http: TLS handshake error from 10.47.4.1:40868: remote error: tls: unknown certificate authority
    ...
    $ oc logs -n openshift-monitoring alertmanager-main-0 -c alertmanager
    ...
    server.go:2923: http: TLS handshake error from 10.47.8.50:36282: remote error: tls: bad certificate
    server.go:2923: http: TLS handshake error from 10.47.4.1:57474: remote error: tls: unknown certificate authority
    ...
    $oc logs -n openshift-monitoring prometheus-k8s-0 -c prometheus
    ...
    level=error ts=2018-11-09T15:27:01.075454778Z caller=notifier.go:473 component=notifier alertmanager=https://10.47.4.81:9094/api/v1/alerts count=0 msg="Error sending alert" err="Post https://10.47.4.81:9094/api/v1/alerts: x509: certificate signed by unknown authority"
    
  • Routes in the openshift-monitoring namespace (such as grafana) fail to resolve and return a 503 "Application is not available" error

Environment

  • OpenShift Container Platform 3.11

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In