RHSB-2023-001 OpenShift misconfiguration of FIPS cryptographic library

Executive summary

This notification is for customers who run Red Hat OpenShift in FIPS mode. Red Hat identified an issue where changes in OpenShift are required to maintain FIPS compliance, using validated cryptography. Red Hat provides FIPS Validated cryptographic modules to OpenShift customers who are required or prefer to use FIPS-validated cryptography.

Customers must choose to use the FIPS-validated cryptographic modules by enabling FIPS mode within OpenShift. Red Hat discovered that, when FIPS mode was enabled, not all of the cryptographic modules in use were FIPS-validated. Instead, some cryptographic modules from the standard Golang library were used. Our commercial OpenShift offerings include standard cryptographic modules that provide strong encryption, but have not been validated for FIPS compliance. At all times, platform data in transit and at rest was encrypted.

This issue is assigned CVE-2023-3089, rated with a Moderate severity impact. There are three options for remediation of this compliance issue. Customers are required to apply a software update, regenerate some certificates, or in the most regulated environments, require a reinstall of OpenShift.

Technical summary

Red Hat discovered that some Openshift components written in Go were not configured to use the FIPS certified cryptographic module OpenSSL when running on clusters booted in FIPS mode. Instead, OpenShift was using the default cryptographic modules of Golang. We are not aware of a flaw in Go's standard crypto module implementation, the problem is that the Golang modules have not been FIPS-validated, meaning all unpatched versions of OpenShift cannot be considered FIPS compliant. To correct this issue, the impacted components have been rebuilt to ensure that the FIPS-validated OpenSSL module is used in all required places when OpenShift is operating in FIPS mode.

This issue only relates to certification. Red Hat is not aware of any security issues with the Go Crypto library. Our customers successfully use it across many products, and it implements the same standard algorithms implemented by OpenSSL. However, this library has not been validated through NIST’s Cryptographic Module Validation Program.

FIPS mode is supported on OpenShift versions 4.10 through 4.12. The only components that claim to use FIPS-validated cryptography are the Release Payload Components (core platform of OpenShift); along with certain optional operators and layered products. See the CVE page for a detailed breakdown of affected components.

Updates for affected products and services

Unlike most fixes, remediation for this issue will not be released in order from the newest to oldest version of OpenShift. Fixed versions of OpenShift 4.10 and 4.11 are generally available now. Additional builds will follow as soon as possible. Customers must wait for their target minor version to be fixed before upgrading, and should follow recommendations from the OpenShift update service to ensure that only FIPS-validated cryptography is used.

Fixes will come out in two phases: core OpenShift, followed by impacted operators and layered products. Customers using impacted offerings can choose one of the following two strategies as fixes roll out:

• Wait to begin updates until all layers have been fixed.

• Temporarily disable any impacted layered products and optional operators, update OpenShift, wait for additional layers to be fixed, apply updates for additional layers, then re-enable them.

Noncompliant cryptographic modules were used to generate a subset of platform keys and certificates.

Customers have two in-place remediation routes available and should choose one based on their operational requirements and risk tolerance:

1. Only apply updates. This is for customers not required to use FIPS mode. The integrity of existing certificates is not in question, so rotating them is unnecessary.

2. Apply updates, then rotate certificates. This is for customers who need to maintain FIPS compliance. Approximately 50% of certificates were produced by non-compliant modules, so rotation is required. Follow the Regenerating cluster certificates procedure.

Alternatively, you may choose to reinstall OpenShift.

Customers operating FedRAMP environments have 90 days to apply fixes. Red Hat is treating this problem with a high priority and intends to remediate all affected components with ample time for customers to plan, test, and deploy fixes. Follow progress on the CVE page for further details on what components are affected and the status of fixes.

For customers of RHACM, we are working in conjunction to remediate those environments. You can find the work and procedures in this KCS article.

Validation

Red Hat is developing a tool to confirm that the fix worked, verifying that your system is once again compliant with FIPS. It will run a static scan of container images to check for incorrectly built OpenShift components written in Go. Once ready, the tool will be published on the CVE page.

FAQ

Q. How do I know if my OpenShift cluster is affected?

A. There are two options:

1.     For connected clusters, use Red Hat Insights to understand which clusters may be affected. For more information on Red Hat Insights in an OpenShift environment, visit the Customer Portal. 

2.    If you are running a disconnected cluster, your only option will be to manually check if your cluster is running in FIPS mode. See if the "fips" parameter is set to "true" in the install-config.yaml file. This is decided at install time; it cannot be changed during cluster operation. See this solution for a command to quickly extract this information from your cluster.  

Q: How do I know that I’m running in FIPS mode?

A: Check if the “fips” parameter is set to “true” in the install-config.yaml file. This is decided at install time; it cannot be changed during cluster operation. See this solution for a command to quickly extract this information from your cluster.

Q: Where can I get more general information about FIPS?

A: For more information, read the General FAQ for OpenShift and FIPS compliance.

Q: What is the expected downtime required to take corrective action?

A: For remediation route #1, downtime will be limited to the upgrade process. For remediation route #2, to rotate the certificates you need to bounce the nodes, which will cause additional downtime after the upgrade process. Clusters can remain in operation during certificate rotation. If you choose to reinstall Openshift, downtime will vary depending on the complexity of your environment.

Q: If I am running a self-managed OCP, will Red Hat provide support beyond written documentation to address the issue?

A: Yes, reach out to support. Information on how to contact Support is available here. https://access.redhat.com/support/contact/technicalSupport

Q. What if I decide to reinstall instead of applying patches?

To ensure that all key material is generated with FIPS compliant cryptography, installation must be run from a FIPS compliant system. For example, a RHEL server booted in FIPS mode.

Q: How has Red Hat reduced the risk of this kind of problem happening again?

A: Red Hat now builds OpenShift Go based components with the correct configuration. Red Hat has added a policy to enforce the correct build flags for components written in Go to be FIPS compliant, and has created and used a scanning tool to ensure container images are built correctly. Deviation from this policy will result in an error terminating the process. This ensures that processes operating on a FIPS enabled node can only use FIPS compliant OpenSSL libraries.