TLS configuration in OpenShift Container Platform
Table of Contents
Introduction
This doc is intended as a comparison and overview of TLS configuration options in OpenShift Container Platform 3 and 4. Where possible, it is advised to use the latest version of TLS, 1.3.
TLS 1.3 is a significant rewrite of the TLS specification including substantial changes to the handshake protocol, with several performance and security improvements. For some OpenShift versions and components, TLS 1.3 is not yet a supported option. For these components one can alternatively configure OpenShift components to use the most secure TLS 1.2 options available.Note that a single TLS connection requires both a compatible client AND server. By disabling older cipher suites in server side components, one risks preventing older clients from connecting. All configuration options described below should be tested thoroughly before applying in production.
TLS 1.2 Cipher Suites
A single cipher suite defines the Key Exchange (Kx) method, Key Exchange Authentication (Au) method (i.e. Certificate key type) , bulk Encryption (Enc) method and Message Authentication (Mac) method. Limiting the set to TLS 1.2 cipher suites is a good starting point as it already removes cipher suites from earlier versions with known weak cryptographic primitives like CBC (vulnerable to Lucky13/CVE-2013-0169) and DES/3DES (vulnerable to Sweet32/CVE-2016-2183).
However there are a number of TLS 1.2 cipher suites that include methods that are no longer supported in TLS 1.3. Namely, RSA for Key Exchange and non-AEAD methods for Message Authentication e.g.
TLS 1.2 Cipher Suites ECDHE-ECDSA-AES256-GCM-SHA384 Kx=ECDH Au=ECDSA Enc=AESGCM(256) Mac=AEAD ECDHE-RSA-AES256-GCM-SHA384 Kx=ECDH Au=RSA Enc=AESGCM(256) Mac=AEAD ECDHE-ECDSA-CHACHA20-POLY1305 Kx=ECDH Au=ECDSA Enc=CHACHA20/POLY1305(256) Mac=AEAD ECDHE-RSA-CHACHA20-POLY1305 Kx=ECDH Au=RSA Enc=CHACHA20/POLY1305(256) Mac=AEAD ECDHE-ECDSA-AES128-GCM-SHA256 Kx=ECDH Au=ECDSA Enc=AESGCM(128) Mac=AEAD ECDHE-RSA-AES128-GCM-SHA256 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD ECDHE-ECDSA-AES128-SHA256 Kx=ECDH Au=ECDSA Enc=AES(128) Mac=SHA256 ECDHE-RSA-AES128-SHA256 Kx=ECDH Au=RSA Enc=AES(128) Mac=SHA256 AES256-GCM-SHA384 Kx=RSA Au=RSA Enc=AESGCM(256) Mac=AEAD AES128-GCM-SHA256 Kx=RSA Au=RSA Enc=AESGCM(128) Mac=AEAD AES256-SHA256 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA256 AES128-SHA256 Kx=RSA Au=RSA Enc=AES(128) Mac=SHA256 DHE-RSA-AES256-GCM-SHA384 Kx=DH Au=RSA Enc=AESGCM(256) Mac=AEAD DHE-RSA-CHACHA20-POLY1305 Kx=DH Au=RSA Enc=CHACHA20/POLY1305(256) Mac=AEAD DHE-RSA-AES128-GCM-SHA256 Kx=DH Au=RSA Enc=AESGCM(128) Mac=AEAD DHE-RSA-AES256-SHA256 Kx=DH Au=RSA Enc=AES(256) Mac=SHA256 DHE-RSA-AES128-SHA256 Kx=DH Au=RSA Enc=AES(128) Mac=SHA256
(Note that cipher suites using AES CCM for bulk encryption have been removed from the above, as they are not supported by Go crypto/tls or OpenSSL in RHEL7)
The RSA method for key exchange has been removed from TLS 1.3 as it does not provide Forward Secrecy. Cipher suites that use RSA for key exchange should therefore be avoided.
Cipher suites using non-AEAD algorithms for message integrity are no longer supported in TLS 1.3 and should also be avoided.
The only TLS 1.2 supported cipher suites not using the above deprecated methods are:
ECDHE-ECDSA-AES256-GCM-SHA384 ECDHE-RSA-AES256-GCM-SHA384 ECDHE-ECDSA-CHACHA20-POLY1305 ECDHE-RSA-CHACHA20-POLY1305 ECDHE-ECDSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES256-GCM-SHA384 DHE-RSA-CHACHA20-POLY1305 DHE-RSA-AES128-GCM-SHA256
However there are some caveats:
-
CHACHA20/POLY1305 for bulk encryption is not supported by OpenSSL in RHEL7
-
DHE for key exchange (i.e non Elliptic Curve DH) is not support by Go crypto/tls
Removing those cipher suites we are left with:
ECDHE-ECDSA-AES256-GCM-SHA384 ECDHE-RSA-AES256-GCM-SHA384 ECDHE-ECDSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-GCM-SHA256
This list is equivalent to the Intermediate Mozilla list, with TLS 1.3 and unsupported cipher suites removed.
The above cipher suites use the same primitives available in TLS 1.3 and therefore provide perfect forward secrecy, strong bulk encryption and use AEAD for message integrity. By limiting the enabled cipher suites to those in the above set, one can prevent the establishment of TLS connections with known vulnerabilities (e.g. Sweet32, Lucky13). However there are several other security enhancements available in TLS 1.3 (see Appendix), using the above TLS 1.2 cipher suites should not be considered as offering equivalent security to TLS 1.3.
The following cipher suites are new in TLS 1.3 and are the only supported options under the protocol specification.
TLS 1.3 Cipher Suites TLS_AES_256_GCM_SHA384 Kx=any Au=any Enc=AESGCM(256) Mac=AEAD TLS_CHACHA20_POLY1305_SHA256 Kx=any Au=any Enc=CHACHA20/POLY1305(256) Mac=AEAD TLS_AES_128_GCM_SHA256 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD TLS_AES_128_CCM_SHA256 Kx=any Au=any Enc=AESCCM(128) Mac=AEAD
The new cipher suites in TLS 1.3 are defined differently and do not specify the certificate type (e.g. RSA) or the key exchange mechanism (e.g. DHE or ECHDE), despite those algorithms being supported[0][1].
TLS Configuration
Note that in the below steps the cipher suite names needed to be translated from the OpenSSL name to the IANA name (e.g. ECDHE-RSA-AES256-GCM-SHA384becomesTLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384).
OpenShift 3.11
OpenShift 3.11 does not support TLS 1.3 which was added to Go in version 1.12 while core OpenShift 3.11 components are built with Go version 1.10. The HAProxy router used in OpenShift 3.11 is not written in Go, however uses the version of OpenSSL shipped in RHEL7, which also does not support TLS 1.3.
Master
Edit /etc/origin/master/master-config.yaml with the desired cipher suites:
servingInfo:
…
minTLSVersion: VersionTLS12
cipherSuites:
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
Then restart the master service:
$ /usr/local/bin/master-restart api api && /usr/local/bin/master-restart controllers controllers
TLS changes made here will also be applied to Etcd instances.
https://access.redhat.com/solutions/3374601
Web Console
$ oc edit configmap webconsole-config -n openshift-web-console
...
servingInfo:
…
minTLSVersion: VersionTLS12
cipherSuites:
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
Node
Kubelet (ports 10248, 10250)
$ oc edit configmap node-config-infra -n openshift-node
...
servingInfo:
…
minTLSVersion: VersionTLS12
cipherSuites:
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
Master, Infra and Compute nodes each have ConfigMaps that require separate edits.
Router
For the router, we need to use the OpenSSL cipher suite names. We can also (optionally) re-add the two cipher suites not supported by the Go crypto/tls package as HAProxy in the router is not written in Go and uses OpenSSL.
The router in OCP 3.11, like all RHEL7 based products, uses a version of OpenSSL that does not support TLS 1.3.
$ oc edit deploymentconfig router -n default
...
- name: ROUTER_CIPHERS
value: ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256
- name: SSL_MIN_VERSION
value: TLSv1.2
...
The above configuration applies only to the cluster router. Individual services should also be checked for compatibility with any router configuration changes as backend server components may also require similar changes (depending on whether they are edge terminated or passthrough).
https://docs.openshift.com/container-platform/3.11/architecture/networking/routes.html#ciphers
Note that router TLS configuration changes do not currently affect port 1936, which is used for router metrics.
Etcd
Etcd (ports 2379, 9977-9) will inherit the Master configuration changes:
OpenShift 4
OpenShift 4 has been built with Go 1.12 since version 4.2 and thus supports TLS 1.3 in most components. Before Openshift 4.6, the router used HAProxy and OpenSSL from RHEL7 does not yet support TLS 1.3. In later versions, the router is based on RHEL and does support TLS 1.3.
OCP 4 - Master
At time of writing, the latest version of OpenShift 4 (4.7) supports TLS 1.3 and the API server (port 6443) is configured to use the “Intermediate” TLS security profile, which includes a reduced cipher suites from both TLS 1.3 and 1.2, without any known vulnerabilities. No further configuration of the API server should be necessary, however one can do so by using the configuration described here:
TLS 1.2:
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-CHACHA20-POLY1305
ECDHE-RSA-AES128-GCM-SHA256
TLS 1.3:
TLS_AES_256_GCM_SHA384
TLS_CHACHA20_POLY1305_SHA256
TLS_AES_128_GCM_SHA256
Some other internal services, namely kube-controller (port 10257) and kube-scheduler (port 10259) use a slightly expanded set of cipher suites however these are not configurable in OpenShift Container Platform versions before 4.8.2. Version 4.8.2 introduced a change that allows for cipher suite configuration and only enables known secure cipher suites by default.
OCP 4 - Web Console
TLS is not configurable in the Web Console in OpenShift Container Platform 4.7 and earlier. Version 4.5.13 introduced a change that deprecates older vulnerable cipher suites and uses a secure set from common OpenShift defaults.
OCP 4 - Node
Kubelet (ports 10248, 10250)
$ oc label machineconfigpool worker custom-kubelet=tls-ciphers
$ oc label machineconfigpool master custom-kubelet=tls-ciphers
$ oc create -f kubeletconfig.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: KubeletConfig
metadata:
name: tls-ciphers
spec:
machineConfigPoolSelector:
matchLabels:
custom-kubelet: tls-ciphers
kubeletConfig:
tlsCipherSuites:
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
For the kubelet, TLS 1.3 cipher suites cannot be declared via configuration in Kubernetes versions before 1.19 and OpenShift 4.6. Despite this, TLS 1.3 cipher suites are enabled for the kubelet in addition to any TLS 1.2 cipher suites declared in configuration.
OCP 4 - Router
Before OpenShift 4.6, the router uses a hardened set of TLS 1.2 cipher suites (TLS 1.3 was not supported as the HAProxy router depended on OpenSSL from RHEL7):
ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES256-GCM-SHA384
However further customization is available via the below method (we’ve removed two cipher suites only for illustrative purposes):
$ oc edit ingresscontroller default -n openshift-ingress-operator
spec:
tlsSecurityProfile:
type: Custom
custom:
ciphers:
- ECDHE-ECDSA-AES128-GCM-SHA256
- ECDHE-RSA-AES128-GCM-SHA256
minTLSVersion: VersionTLS12
In OpenShift 4.6 and later, the router is based on RHEL8 and supports TLS 1.3.
OCP 4 - Etcd
Ports 9977-9, 2379, 2380
TLS is not configurable for Etcd in OpenShift Container Platform 4.6 and earlier. Version 4.6.16 introduced a change that deprecates older vulnerable cipher suites and uses a secure set from common OpenShift defaults.
OCP 4 - Machine Config Server
Ports 22623-4
TLS is not configurable for the Machine Config Server in OpenShift Container Platform versions before 4.8.2. Version 4.8.2 introduced a change that only enables known secure cipher suites by default.
OCP 4 - Node
Exporter
Port 9100
TLS is not configurable for the Node Exporter in OpenShift Container Platform versions before 4.6.1. Version 4.6.1 introduced a change that only enables known secure cipher suites by default.
OCP 4 - Kube RBAC Proxy
Port 9192
TLS is not configurable for the Kube RBAC Proxy in OpenShift Container Platform versions before 4.8.2. Version 4.8.2 introduced a change that only enables known secure cipher suites by default.
Appendix
Other TLS 1.3 Security Improvements
-
Weak primitives removed in TLS 1.3: eg RC4, SHA1, MD5.
Already not supported in OpenShift
-
Removal of CBC mode cipher suites
e.g. TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA ,TLS_RSA_WITH_AES_128_CBC_SHA, DES-CBC3-SHA
-
No ChangeCipherSpec messages used in handshake:
Not needed in TLS 1.3 (except for compat with earlier TLS versions). No known security issues with these messages however previously allowed for CVE-2014-0224.
-
No negotiation of compression
-
Re-key mechanism
Replaces session renegotiation, client session renegotiation is disabled by default in OpenShift
-
Removing PKCS#1 v1.5 padding
This only applies to RSA key exchange, disabling cipher suites that use RSA key exchange makes this N/A.
-
Removed some DHE groups
Custom groups are no longer supported, the following curves are supported under TLS 1.3:
X25519, prime256v1, secp384r1 https://wiki.mozilla.org/Security/Server_Side_TLS#Modern_compatibility
-
DSA certificates no longer supported
Already not supported by default in OpenShift
-
Anti-downgrade feature
-
Encrypted extensions during handshake
-
New improved session resumption features
-
New ECC curves: Curve 25519 and Curve 448
Comments