Openshift router certificate error: unable to load SSL private key from PEM file

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform
  • Red Hat OpenShift Container Platform 4.x
    • v3.11
    • v4.16

Issue

  • While deploying a router, getting below error;
E0812 15:21:03.359503       1 limiter.go:137] error reloading router: exit status 1
[ALERT] 224/152103 (29) : parsing [/var/lib/haproxy/conf/haproxy.config:116] : 'bind 127.0.0.1:10xxx' : inconsistencies between private key and certificate loaded from PEM file '/etc/pki/tls/private/tls.crt'.

[ALERT] 004/174016 (31) : parsing [/var/lib/haproxy/conf/haproxy.config:116] : 'bind 127.0.0.1:10444' : unable to load SSL private key from PEM file '/etc/pki/tls/private/tls.crt'.
[ALERT] 004/174016 (31) : parsing [/var/lib/haproxy/conf/haproxy.config:153] : 'bind 127.0.0.1:10443' : unable to load SSL private key from PEM file '/etc/pki/tls/private/tls.crt'.

Resolution

  • If you are using the internal OpenShift Container Platform CA to sign certificates, follow the steps given here to redeploy the router certificates manually.
  • If you are using an external Certificate Authority (CA) to sign your certificates, create a new CSR and Private key.

  • If you are facing an issue in OpenShift 4.x check the article for renewing the certificates How to redeploy/renew an expired default ingress certificate in RHOCP4?

  • Follow the below steps for v3.11

  • Step 1: Generate a unique key:

# openssl genrsa -out router.key 2048
  • Step 2: Generate a Certificate Signing Request (CSR):
# openssl req -new -key router.key -out router.csr
  • Step 3: Create a Self-Signed Certificate (CRT):
# openssl x509 -req -days 365 -in router.csr -signkey router.key -out router.crt
  • Step 4: Append the newly generated certificate and key in the .pem file along with root chain certificates.
# cat router.crt router.key <CA_chain.crt>  > router.pem
  • Step 5: Delete all the previous router resources:
# oc delete all -l router=router
# oc delete secrets <secret_name>
# oc delete sa router
  • Step 6: Create a new router with new .pem file:
# oc adm router router \
  --default-cert=<file_name.pem> \
  --replicas=3 \
  --service-account=router \
  --ciphers=modern \
  --selector='node-role.kubernetes.io/infra=true' \
  --images='openshift3/ose-haproxy-router:v3.11.232' -n default

Root Cause

  • The server certificate does not contains private key in correct order.
  • The private key can be corrupted or not match with the server certificate.

Diagnostic Steps

  • Check the logs of the router v3.11:
E0817 10:54:51.383548       1 limiter.go:137] error reloading router: exit status 1
[ALERT] 229/105451 (25) : parsing [/var/lib/haproxy/conf/haproxy.config:116] : 'bind 127.0.0.1:10444' : inconsistencies between private key and certificate loaded from PEM file '/etc/pki/tls/private/tls.crt'.
[ALERT] 229/105451 (25) : parsing [/var/lib/haproxy/conf/haproxy.config:153] : 'bind 127.0.0.1:10443' : inconsistencies between private key and certificate loaded from PEM file '/etc/pki/tls/private/tls.crt'.
[ALERT] 229/105451 (25) : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT] 229/105451 (25) : Fatal errors found in configuration.
  • Check the logs of the Ingress v4.16:
haproxy.go:418] can't scrape HAProxy: dial unix /var/lib/haproxy/run/haproxy.sock: connect: no such file or directory
 E0528 11:04:34.570921      1 limiter.go:165] error reloading router: exit status 1
 [NOTICE]  (34) : haproxy version is 2.8.5-aaba8d0
 [NOTICE]  (34) : path to executable is /usr/sbin/haproxy
 [ALERT]   (34) : config : parsing [/var/lib/haproxy/conf/haproxy.config:131] : 'bind unix@/var/lib/haproxy/run/haproxy-sni.sock' in section 'frontend' : No Private Key found in '/var/lib/haproxy/router/certs/default.pem.key'.
 [ALERT]   (34) : config : parsing [/var/lib/haproxy/conf/haproxy.config:173] : 'bind unix@/var/lib/haproxy/run/haproxy-no-sni.sock' in section 'frontend' : No Private Key found in '/var/lib/haproxy/router/certs/default.pem.key'.
 [ALERT]   (34) : config : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
  • IMP: Verify the md5 hash values of the certificate and private key in PEM file, both the value should be same:
# openssl x509 -noout -modulus -in <file_name.pem> | openssl md5
# openssl rsa -noout -modulus -in <file_name.pem> | openssl md5

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments