TLS configuration in OpenShift Container Platform

Introduction

This doc is intended as a comparison and overview of TLS configuration options in OpenShift Container Platform 3 and 4. Where possible, it is advised to use the latest version of TLS, 1.3.

TLS 1.3 is a significant rewrite of the TLS specification including substantial changes to the handshake protocol, with several performance and security improvements. For some OpenShift versions and components, TLS 1.3 is not yet a supported option. For these components one can alternatively configure OpenShift components to use the most secure TLS 1.2 options available.Note that a single TLS connection requires both a compatible client AND server. By disabling older cipher suites in server side components, one risks preventing older clients from connecting. All configuration options described below should be tested thoroughly before applying in production.

TLS 1.2 Cipher Suites

A single cipher suite defines the Key Exchange (Kx) method, Key Exchange Authentication (Au) method (i.e. Certificate key type) , bulk Encryption (Enc) method and Message Authentication (Mac) method. Limiting the set to TLS 1.2 cipher suites is a good starting point as it already removes cipher suites from earlier versions with known weak cryptographic primitives like CBC (vulnerable to Lucky13/CVE-2013-0169) and DES/3DES (vulnerable to Sweet32/CVE-2016-2183).

However there are a number of TLS 1.2 cipher suites that include methods that are no longer supported in TLS 1.3. Namely, RSA for Key Exchange and non-AEAD methods for Message Authentication e.g.

TLS 1.2 Cipher Suites

ECDHE-ECDSA-AES256-GCM-SHA384  Kx=ECDH  Au=ECDSA Enc=AESGCM(256) Mac=AEAD
ECDHE-RSA-AES256-GCM-SHA384  Kx=ECDH    Au=RSA  Enc=AESGCM(256) Mac=AEAD
ECDHE-ECDSA-CHACHA20-POLY1305  Kx=ECDH  Au=ECDSA Enc=CHACHA20/POLY1305(256) Mac=AEAD
ECDHE-RSA-CHACHA20-POLY1305  Kx=ECDH    Au=RSA  Enc=CHACHA20/POLY1305(256) Mac=AEAD
ECDHE-ECDSA-AES128-GCM-SHA256  Kx=ECDH  Au=ECDSA Enc=AESGCM(128) Mac=AEAD
ECDHE-RSA-AES128-GCM-SHA256  Kx=ECDH    Au=RSA  Enc=AESGCM(128) Mac=AEAD
ECDHE-ECDSA-AES128-SHA256  Kx=ECDH     Au=ECDSA Enc=AES(128)  Mac=SHA256
ECDHE-RSA-AES128-SHA256  Kx=ECDH   Au=RSA  Enc=AES(128)  Mac=SHA256
AES256-GCM-SHA384      Kx=RSA   Au=RSA  Enc=AESGCM(256) Mac=AEAD
AES128-GCM-SHA256      Kx=RSA   Au=RSA  Enc=AESGCM(128) Mac=AEAD
AES256-SHA256          Kx=RSA   Au=RSA  Enc=AES(256)  Mac=SHA256
AES128-SHA256          Kx=RSA   Au=RSA  Enc=AES(128)  Mac=SHA256
DHE-RSA-AES256-GCM-SHA384  Kx=DH    Au=RSA  Enc=AESGCM(256) Mac=AEAD
DHE-RSA-CHACHA20-POLY1305  Kx=DH    Au=RSA  Enc=CHACHA20/POLY1305(256) Mac=AEAD
DHE-RSA-AES128-GCM-SHA256  Kx=DH    Au=RSA  Enc=AESGCM(128) Mac=AEAD
DHE-RSA-AES256-SHA256  Kx=DH       Au=RSA  Enc=AES(256)  Mac=SHA256
DHE-RSA-AES128-SHA256  Kx=DH       Au=RSA  Enc=AES(128)  Mac=SHA256

(Note that cipher suites using AES CCM for bulk encryption have been removed from the above, as they are not supported by Go crypto/tls or OpenSSL in RHEL7)

The RSA method for key exchange has been removed from TLS 1.3 as it does not provide Forward Secrecy. Cipher suites that use RSA for key exchange should therefore be avoided.

Cipher suites using non-AEAD algorithms for message integrity are no longer supported in TLS 1.3 and should also be avoided.

The only TLS 1.2 supported cipher suites not using the above deprecated methods are:

ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-ECDSA-CHACHA20-POLY1305
ECDHE-RSA-CHACHA20-POLY1305
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-GCM-SHA256
DHE-RSA-AES256-GCM-SHA384
DHE-RSA-CHACHA20-POLY1305
DHE-RSA-AES128-GCM-SHA256

However there are some caveats:

• CHACHA20/POLY1305 for bulk encryption is not supported by OpenSSL in RHEL7

• DHE for key exchange (i.e non Elliptic Curve DH) is not support by Go crypto/tls

Removing those cipher suites we are left with:

ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-GCM-SHA256

This list is equivalent to the Intermediate Mozilla list, with TLS 1.3 and unsupported cipher suites removed.

The above cipher suites use the same primitives available in TLS 1.3 and therefore provide perfect forward secrecy, strong bulk encryption and use AEAD for message integrity. By limiting the enabled cipher suites to those in the above set, one can prevent the establishment of TLS connections with known vulnerabilities (e.g. Sweet32, Lucky13). However there are several other security enhancements available in TLS 1.3 (see Appendix), using the above TLS 1.2 cipher suites should not be considered as offering equivalent security to TLS 1.3.

The following cipher suites are new in TLS 1.3 and are the only supported options under the protocol specification.

TLS 1.3 Cipher Suites

TLS_AES_256_GCM_SHA384         Kx=any   Au=any  Enc=AESGCM(256)          Mac=AEAD
TLS_CHACHA20_POLY1305_SHA256 Kx=any     Au=any  Enc=CHACHA20/POLY1305(256) Mac=AEAD
TLS_AES_128_GCM_SHA256       Kx=any     Au=any  Enc=AESGCM(128)          Mac=AEAD
TLS_AES_128_CCM_SHA256       Kx=any     Au=any  Enc=AESCCM(128)          Mac=AEAD

The new cipher suites in TLS 1.3 are defined differently and do not specify the certificate type (e.g. RSA) or the key exchange mechanism (e.g. DHE or ECHDE), despite those algorithms being supported[0][1].

TLS Configuration

Note that in the below steps the cipher suite names needed to be translated from the OpenSSL name to the IANA name (e.g. ECDHE-RSA-AES256-GCM-SHA384becomesTLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384).

OpenShift 3.11

OpenShift 3.11 does not support TLS 1.3 which was added to Go in version 1.12 while core OpenShift 3.11 components are built with Go version 1.10. The HAProxy router used in OpenShift 3.11 is not written in Go, however uses the version of OpenSSL shipped in RHEL7, which also does not support TLS 1.3.

Master

Edit /etc/origin/master/master-config.yaml with the desired cipher suites:

 servingInfo:
    …
    minTLSVersion: VersionTLS12
    cipherSuites:  
    - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

Then restart the master service:

$ /usr/local/bin/master-restart api api && /usr/local/bin/master-restart controllers controllers

TLS changes made here will also be applied to Etcd instances.

https://access.redhat.com/solutions/3374601

Web Console

$ oc edit configmap webconsole-config -n  openshift-web-console
  ...
 servingInfo:
    …
    minTLSVersion: VersionTLS12
    cipherSuites:  
    - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

Node

Kubelet (ports 10248, 10250)

$ oc edit configmap node-config-infra -n  openshift-node
  ...
 servingInfo:
    …
    minTLSVersion: VersionTLS12
    cipherSuites:  
    - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

Master, Infra and Compute nodes each have ConfigMaps that require separate edits.

Router

For the router, we need to use the OpenSSL cipher suite names. We can also (optionally) re-add the two cipher suites not supported by the Go crypto/tls package as HAProxy in the router is not written in Go and uses OpenSSL.

The router in OCP 3.11, like all RHEL7 based products, uses a version of OpenSSL that does not support TLS 1.3.

$ oc edit deploymentconfig router -n default
...
    - name: ROUTER_CIPHERS
      value: ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256
    - name: SSL_MIN_VERSION
      value: TLSv1.2
...

The above configuration applies only to the cluster router. Individual services should also be checked for compatibility with any router configuration changes as backend server components may also require similar changes (depending on whether they are edge terminated or passthrough).

https://docs.openshift.com/container-platform/3.11/architecture/networking/routes.html#ciphers

https://docs.openshift.com/container-platform/3.11/install_config/router/default_haproxy_router.html#bind-ciphers

Note that router TLS configuration changes do not currently affect port 1936, which is used for router metrics.

Etcd

Etcd (ports 2379, 9977-9) will inherit the Master configuration changes:

https://docs.openshift.com/container-platform/3.11/install_config/master_node_configuration.html#master-config-tls-cipher

OpenShift 4

OpenShift 4 has been built with Go 1.12 since version 4.2 and thus supports TLS 1.3 in most components. Before Openshift 4.6, the router used HAProxy and OpenSSL from RHEL7 does not yet support TLS 1.3. In later versions, the router is based on RHEL and does support TLS 1.3.

OCP 4 - Master

At time of writing, the latest version of OpenShift 4 (4.7) supports TLS 1.3 and the API server (port 6443) is configured to use the “Intermediate” TLS security profile, which includes a reduced cipher suites from both TLS 1.3 and 1.2, without any known vulnerabilities. No further configuration of the API server should be necessary, however one can do so by using the configuration described here:

https://docs.openshift.com/container-platform/4.7/rest_api/config_apis/apiserver-config-openshift-io-v1.html

TLS 1.2:
    ECDHE-RSA-AES256-GCM-SHA384
    ECDHE-RSA-CHACHA20-POLY1305
    ECDHE-RSA-AES128-GCM-SHA256
TLS 1.3:
    TLS_AES_256_GCM_SHA384
    TLS_CHACHA20_POLY1305_SHA256
    TLS_AES_128_GCM_SHA256

Some other internal services, namely kube-controller (port 10257) and kube-scheduler (port 10259) use a slightly expanded set of cipher suites however these are not configurable in OpenShift Container Platform versions before 4.8.2. Version 4.8.2 introduced a change that allows for cipher suite configuration and only enables known secure cipher suites by default.

OCP 4 - Web Console

TLS is not configurable in the Web Console in OpenShift Container Platform 4.7 and earlier. Version 4.5.13 introduced a change that deprecates older vulnerable cipher suites and uses a secure set from common OpenShift defaults.

OCP 4 - Node

Kubelet (ports 10248, 10250)

$ oc label machineconfigpool worker custom-kubelet=tls-ciphers
$ oc label machineconfigpool master custom-kubelet=tls-ciphers
$ oc create -f kubeletconfig.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
  name: tls-ciphers
spec:
  machineConfigPoolSelector:
    matchLabels:
      custom-kubelet: tls-ciphers
  kubeletConfig:
    tlsCipherSuites:
    - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
    - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
    - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

https://docs.openshift.com/container-platform/4.7/scalability_and_performance/recommended-host-practices.html#create-a-kubeletconfig-crd-to-edit-kubelet-parameters_

For the kubelet, TLS 1.3 cipher suites cannot be declared via configuration in Kubernetes versions before 1.19 and OpenShift 4.6. Despite this, TLS 1.3 cipher suites are enabled for the kubelet in addition to any TLS 1.2 cipher suites declared in configuration.

OCP 4 - Router

Before OpenShift 4.6, the router uses a hardened set of TLS 1.2 cipher suites (TLS 1.3 was not supported as the HAProxy router depended on OpenSSL from RHEL7):

ECDHE-RSA-AES128-GCM-SHA256              
ECDHE-RSA-AES256-GCM-SHA384          
DHE-RSA-AES128-GCM-SHA256                
DHE-RSA-AES256-GCM-SHA384

However further customization is available via the below method (we’ve removed two cipher suites only for illustrative purposes):

$ oc edit ingresscontroller default -n openshift-ingress-operator
spec:
  tlsSecurityProfile:
    type: Custom
    custom:
    ciphers:
    - ECDHE-ECDSA-AES128-GCM-SHA256
    - ECDHE-RSA-AES128-GCM-SHA256
    minTLSVersion: VersionTLS12

In OpenShift 4.6 and later, the router is based on RHEL8 and supports TLS 1.3.

https://docs.openshift.com/container-platform/4.6/networking/ingress-operator.html#nw-ingress-controller-tls-profiles_configuring-ingress

OCP 4 - Etcd

Ports 9977-9, 2379, 2380

TLS is not configurable for Etcd in OpenShift Container Platform 4.6 and earlier. Version 4.6.16 introduced a change that deprecates older vulnerable cipher suites and uses a secure set from common OpenShift defaults.

OCP 4 - Machine Config Server

Ports 22623-4

TLS is not configurable for the Machine Config Server in OpenShift Container Platform versions before 4.8.2. Version 4.8.2 introduced a change that only enables known secure cipher suites by default.

OCP 4 - Node

Exporter

Port 9100

TLS is not configurable for the Node Exporter in OpenShift Container Platform versions before 4.6.1. Version 4.6.1 introduced a change that only enables known secure cipher suites by default.

OCP 4 - Kube RBAC Proxy

Port 9192

TLS is not configurable for the Kube RBAC Proxy in OpenShift Container Platform versions before 4.8.2. Version 4.8.2 introduced a change that only enables known secure cipher suites by default.

Appendix

Other TLS 1.3 Security Improvements

• Weak primitives removed in TLS 1.3: eg RC4, SHA1, MD5.

  Already not supported in OpenShift

• Removal of CBC mode cipher suites

  e.g. TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA ,TLS_RSA_WITH_AES_128_CBC_SHA, DES-CBC3-SHA

• No ChangeCipherSpec messages used in handshake:

  Not needed in TLS 1.3 (except for compat with earlier TLS versions). No known security issues with these messages however previously allowed for CVE-2014-0224.

• No negotiation of compression

• Re-key mechanism

  Replaces session renegotiation, client session renegotiation is disabled by default in OpenShift

• Removing PKCS#1 v1.5 padding

  This only applies to RSA key exchange, disabling cipher suites that use RSA key exchange makes this N/A.

• Removed some DHE groups

  Custom groups are no longer supported, the following curves are supported under TLS 1.3:

  X25519, prime256v1, secp384r1 https://wiki.mozilla.org/Security/Server_Side_TLS#Modern_compatibility

• DSA certificates no longer supported

  Already not supported by default in OpenShift

• Anti-downgrade feature

• Encrypted extensions during handshake

• New improved session resumption features

• New ECC curves: Curve 25519 and Curve 448

Other references

https://access.redhat.com/blogs/766093/posts/2975791

https://access.redhat.com/blogs/766093/posts/2978671