OpenShift 4: Ingress to route config with invalid cert config leads to haproxy reload failure

Issue

• An ingress object was created with an invalid certificate configuration [1]. This ingress object lead to a route creation which was automatically ingested by the router default pods (expected). However, the route itself was invalid as a result of the problematic certificate. This caused the router pods to fail reloads [2], leading to inconsistent backend state and a failure to update the haproxy.config in response to backend changes - leading to a partial or total outage.

[1] certificate details - redacted

tls.crt:
-----BEGIN CERTIFICATE-----
"redacted"
-----END CERTIFICATE-----


tls.key
Bag Attributes
    friendlyName: 2024_ocp4_prod_<redacted>
    localKeyID: 54 69 6D <redacted> 37 34 34
    1.3.18.0.2.28.24: IBM_SDK_JAVA_8_PKCS12
Key Attributes: <No Attributes>
-----BEGIN PRIVATE KEY-----
REDACTED
-----END PRIVATE KEY-----
Bag Attributes
    friendlyName: <redacted>ca1
    2.16.840.1.113894.746875.1.1: <Unsupported tag 6>
    1.3.18.0.2.28.24: IBM_SDK_JAVA_8_PKCS12
subject=/DC=ca/DC=<redacted>/DC=<redacted>/DC=<redacted>/CN=<redacted> Root CA 1
issuer=/DC=ca/DC=<redacted>/DC=<redacted>/DC=<redacted>/CN=<redacted> Root CA 1
-----BEGIN CERTIFICATE-----
REDACTED
-----END CERTIFICATE-----
Bag Attributes
    friendlyName: <redacted>ca8
    2.16.840.1.113894.746875.1.1: <Unsupported tag 6>
    1.3.18.0.2.28.24: IBM_SDK_JAVA_8_PKCS12
subject=/DC=ca/DC=<redacted>/DC=<redacted>/DC=<redacted>/CN=<redacted> Issuing CA 8
issuer=/DC=ca/DC=<redacted>/DC=<redacted>/DC=<redacted>/CN=<redacted> Root CA 1
-----BEGIN CERTIFICATE-----
REDACTED
-----END CERTIFICATE-----

[2] haproxy logs:

[NOTICE]   (90908) : path to executable is /usr/sbin/haproxy
[ALERT]    (90908) : config : parsing [/var/lib/haproxy/conf/haproxy.config:132] : 'bind unix@/var/lib/haproxy/run/haproxy-sni.sock' in section 'frontend' : 'crt-list' : inconsistencies between private key and certificate loaded '/var/lib/haproxy/router/certs/eside:broken-config.pem'.
[ALERT]    (90908) : config : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT]    (90908) : config : Fatal errors found in configuration.
E0201 00:36:22.933698       1 limiter.go:165] error reloading router: exit status 1
[NOTICE]   (90912) : haproxy version is 2.6.13-234aa6d
[NOTICE]   (90912) : path to executable is /usr/sbin/haproxy
[ALERT]    (90912) : config : parsing [/var/lib/haproxy/conf/haproxy.config:132] : 'bind unix@/var/lib/haproxy/run/haproxy-sni.sock' in section 'frontend' : 'crt-list' : inconsistencies between private key and certificate loaded '/var/lib/haproxy/router/certs/eside:broken-config.pem'.
[ALERT]    (90912) : config : Error(s) found in configuration file : /var/lib/haproxy/conf/haproxy.config
[ALERT]    (90912) : config : Fatal errors found in configuration.