Working of TLS Ciphers in OpenShift 4

Updated -

TLS handshake issues could arise due to various reasons:
1. tls/ssl version mismatch
2. cipher mismatch
3. certificate issue

Generally, the error string or error code that is seen during TLS handshake failure would help investigate this type of issues. During TLS handshake, Client(hello) is the first entity that initiates the handshake by exchanging certificates and ciphers. In OpenShift 4, it depends from where the connection is being invoked, at what layer or endpoint connection is getting terminated. Accordingly we can check TLS version and desired list of ciphers for the component.

General troubleshooting guidelines with respect to ciphers compatibility issues:

  • Get the exact request flow, understand the source and destination system types.
  • Check baseOS type and version of the container
  • Check whether FIPS is enabled or not. FIPS enabled cluster comes with reduced set of ciphers and only offer the stronger ones.
  • Container ciphers (when it initiate the connection) will not check host for ciphers/ssl.

TLS inspection about OpenShift 4 Ingress layer

  • TLS termination could happen at the ingress controller pods (router pods) or at the application pod. In the second case, the ciphers config from the app itself is what matters. In the first case, strong ciphers from the cluster config protect the application pods - but traffic going to those pods from inside the cluster, directly to its service, would not be protected. You can prevent that with network policies.
  • If the Route has a passthrough TLS termination (Route.spec.tls.termination) then the connection will be passed directly to the Pod for the TLS negotiation. In this case, the client can use the ciphers that are already available in the application Pod.

  • If the termination is not passthrough then the TLS negotiation will be handled by the OpenShift Ingress itself, in this case the client needs to support the ciphers presented by the OpenShift Ingress regardless of what's in the Pod.

  • One can configure the ciphers for the OCP Router via this spec in IngressController

Please refer TLS configuration in OpenShift 4 to know how and where it is supported to set ciphers and TLS version of our own choice.

  • Here is how curl get ciphers inside http container (oc new-app httpd)
sh-4.4$ type curl
curl is /usr/bin/curl
sh-4.4$ 
sh-4.4$ ldd /usr/bin/curl | grep cryp
    libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f10e721f000)
    libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f10e5a05000)
    libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f10e37c6000)
sh-4.4$ 
sh-4.4$ rpm -qf /lib64/libcrypto.so.1.1
openssl-libs-1.1.1k-9.el8_7.x86_64
sh-4.4$ 
sh-4.4$ rpm -qa | grep ssl
openssl-1.1.1k-9.el8_7.x86_64
mod_ssl-2.4.37-56.module+el8.8.0+19808+379766d6.7.x86_64
openssl-libs-1.1.1k-9.el8_7.x86_64
apr-util-openssl-1.6.1-6.el8_8.1.x86_64
xmlsec1-openssl-1.2.25-4.el8.x86_64
sh-4.4$ 
  • Similarly, if the issue arises for a connection between container and external host, then it is essential to check ciphers / tls version loaded by the application stack.
  • Some middleware application provide their own configuration to load the ciphers and make use of them while contacting to any internal/external system.
  • If its not provisioned explicitly by the application, then check baseOS and curl/openssl module to know set of available ciphers.
1. To check whether a specific cipher exist or not.
$ openssl ciphers -v | grep "ECDHE-RSA-AES128-SHA"
ECDHE-RSA-AES128-SHA256        TLSv1.2 Kx=ECDH     Au=RSA   Enc=AES(128)               Mac=SHA256
ECDHE-RSA-AES128-SHA           TLSv1   Kx=ECDH     Au=RSA   Enc=AES(128)               Mac=SHA1

2. Enable detailed tracing for checking tls version and cipher compatibility towards an endpoint in question. 
$ echo Q| openssl s_client -connect <target_endpoint>:<port> -tls1_2 -cipher ECDHE-RSA-AES128-SHA -status -msg -debug

Comments