CVE-2020-11100 haproxy: malformed HTTP/2 requests can lead to out-of-bounds writes

Public Date:
Updated -
Status
Resolved
Impact
Critical

Insights vulnerability analysis

View exposed systems

Executive Summary

Red Hat is aware of a flaw in the way HAProxy processed certain HTTP/2 request packets.  An attacker could send crafted HTTP/2 request packets which cause memory corruption, leading to a crash or remote arbitrary code execution with the permissions of the user running HAProxy. 

This issue has been assigned CVE-2020-11100 with a severity impact of Critical.

Affected Products

Red Hat Products Affected by CVE-2020-11100

Product/PackageAffected State
Red Hat Enterprise Linux 6 (haproxy)Not Affected
Red Hat Enterprise Linux 7 (haproxy)Not Affected
Red Hat Enterprise Linux 7 Software Collections (rh-haproxy18-haproxy)Affected - Will Fix active streams
Red Hat Enterprise Linux 8 (haproxy)Affected - Will Fix active streams
Red Hat OpenStack Platform 10 (openstack-haproxy-container)Not Affected - uses RHEL 7 package
Red Hat OpenStack Platform 13 (openstack-haproxy-container)Not Affected - uses RHEL 7 package
Red Hat OpenStack Platform 15 (openstack-haproxy-container)Affected - uses RH:EL 8 package
Red Hat OpenStack Platform 15 (openstack-neutron-base-container)Affected - uses RH:EL 8 package
Red Hat OpenStack Platform 16 (openstack-haproxy-container)Affected - uses RH:EL 8 package
Red Hat OpenStack Platform 16 (openstack-neutron-base-container)Affected - uses RH:EL 8 package
Red Hat OpenShift Container Platform 3.11Affected - Will Fix
Red Hat OpenShift Container Platform 4Affected - Low Severity

Updates for Affected Products

ProductPackageAdvisory/Update
Red Hat Enterprise Linux 8haproxyRHSA-2020:1288
Red Hat Enterprise Linux 8 Update Services for SAP SolutionshaproxyRHSA-2020:1289
Red Hat Software Collectionsrh-haproxy18-haproxyRHSA-2020:1290
Red Hat OpenShift Container Platform 3.11haproxyRHSA-2020:1287
Red Hat OpenStack Platform 15openstack-haproxy-containerRHBA-2020:1328
Red Hat OpenStack Platform 15openstack-neutron-base-containerUpdates available in the Container Catalog
Red Hat OpenStack Platform 16openstack-haproxy-containerRHBA-2020:1327
Red Hat OpenStack Platform 16openstack-neutron-base-containerUpdates available in the Container Catalog


Technical Details and Background

HAProxy is a TCP/HTTP reverse proxy which is designed for high availability environments.  HAProxy is typically deployed in front of a cluster of application servers and dispatches incoming requests to one of the servers, resulting in increased performance and high availability.  When deployed with a website, HAProxy can optionally use the HTTP/2 protocol for more efficient use of network resources.

This is a flaw in the way HTTP/2 request packets are handled by HAProxy. 

HAProxy packages shipped with Red Hat Enterprise Linux 6 and 7 do not contain support for HTTP/2; therefore, they are not affected by this flaw.

All OpenShift Container Platform 4 versions through 4.3 contain the vulnerable code. However, exploitation requires setting ROUTER_USE_HTTP2 in the OpenShift Ingress Operator, which is not currently possible. The impact of this vulnerability, is therefore, reduced in OCP 4.x before version 4.4 to low.

OpenShift Container Platform 3.11 added a configuration option to ose-haproxy-router that made enabling HTTP/2 support easy. However, it is not enabled by default on that version.

Mitigation

Security errata for affected products are released as early as possible; please consult the table above for availability details. Customers requiring a solution before fixes are available, or seeking an alternative solution, should assess the mitigation described below.

On Red Hat Enterprise Linux 8, HAProxy is confined by SELinux, which should mitigate remote arbitrary code execution.  

This issue can be mitigated by not enabling support for HTTP/2 protocol. The default HAProxy configurations shipped with Red Hat products have HTTP/2 disabled, but it is common to alter these defaults in a customer environment.

You can check if HTTP/2 is enabled by searching your HAProxy configuration files for a line containing 'h2'.  Typically this looks something like:
    bind :443 ssl crt pub.pem alpn h2,http/1.1

To mitigate this vulnerability in OpenShift Container Platform 3.11, keep HTTP/2 disabled as it is by default, or disable if you have it previously enabled.

You can also verify if HTTP/2 support is enabled on a given server using the curl command line tool as follows.  Note that in both cases, the client offers HTTP/2, but the server only accepts it in the first case.  Replace the hostname with your own.

$ curl -v --http2 https://h2-enabled.example.com/ 2>&1 >/dev/null | grep ALPN 
* ALPN, offering h2
* ALPN, offering http/1.1
* ALPN, server accepted to use h2

$ curl -v --http2 https://h2-not-enabled.example.com/ 2>&1 >/dev/null | grep ALPN
* ALPN, offering h2
* ALPN, offering http/1.1
* ALPN, server did not agree to a protocol

** Note that if the backend server is serving HTTP/2 responses and HAProxy is using passthrough mode, this test will show HTTP/2 in use.  A negative result from this test can be relied on.  A positive test may need to be confirmed by ensuring a closer examination of the configuration to confirm more specifics of where the response is being received.  If you get a positive result and you think you’re not using HTTP/2, check the route configuration for passthrough mode.

A good URL to test for OpenShift Container Platform 3.11 is the “Cluster Console” (in the openshift-console project) because it doesn’t use a passthrough route. Note that this route is distinct from the “Application Console” (in the openshift-web-console project), which isn't proxied by the router pod.

$ curl -v --http2 https://console.apps.example.com 2>&1 >/dev/null | grep ALPN
* ALPN, offering h2
* ALPN, offering http/1.1
* ALPN, server did not agree to a protocol

The API server in OpenShift Container Platform 3.11 is an example of a pass through route.  Even though this is behind haproxy, the fact it’s a passthrough means the connection terminates inside the pod in a golang binary that has HTTP/2 enabled.  The server detected in this case is inside the apiserver pod, and not the vulnerable haproxy in the router pod.

$ curl -v --http2 https://apiserver-kube-service-catalog.apps.example.com 2>&1 >/dev/null | grep ALPN
* ALPN, offering h2
* ALPN, offering http/1.1
* ALPN, server did not agree to a protocol

Acknowledgements

Red Hat thanks the HAProxy project for reporting this issue. Upstream acknowledges Felix Wilhelm from Google Project Zero as the original reporter of this issue.

References

Google Project Zero Disclosure

Google Project Zero 

HAProxy team blog

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In

11 Comments

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In

Kinda hard to test this with the latest RHEL7 version of curl being 7.29. The --http2 option is not valid for that version. Any other way to check?

On RHEL8 you can use curl 7.61.1 which has the --http2 option aboard.

You can also use a number of online tools to help determine whether HTTP/2 is supported by your server.

One example is https://www.ssllabs.com/ssltest/analyze.html. This will only work for servers accessible on the internet, and we recommend that you select "Do not show the results on the boards". In this example, once the results are available you will look for the "ALPN" (or NPN) result and see if "h2" is present.

This test only demonstrates whether HTTP/2 is supported by the server. You still need to confirm whether it's being served by a vulnerable version of haproxy.

Hi Gene Siepka,

You can check the protocol in many ways:

If the SSL/TLS engine supports the Next Protocol Negotiation extension, you can use the openssl:

openssl s_client -connect www.example.com:443 -nextprotoneg ''

Another way is running is-http2 (external tool, needs to be installed)

[quicklab@master-2 ~]$ is-http2 www.example.com
✓ HTTP/2 supported by www.example.com
Supported protocols: h2 http/1.1

You can also use a number of online tools to help determine whether HTTP/2 is supported by your server.

One example is https://www.ssllabs.com/ssltest/analyze.html. This will only work for servers accessible on the internet, and we recommend that you select "Do not show the results on the boards". In this example, once the results are available you will look for the "ALPN" (or NPN) result and see if "h2" is present.

This test only demonstrates whether HTTP/2 is supported by the server. You still need to confirm whether it's being served by a vulnerable version of haproxy.

Also is there any info on OpenShift versions prior to 3.11? We have upgraded some of our clusters to 3.11 but others are stil on 3.9 and are in the pipeline to be upgraded.

Prior to 3.11, in versions OpenShift 3.9 and 3.10 the haproxy version is affected. However HTTP/2 is not enabled by default, an administrator had to customize the haproxy router configuration to add HTTP/2 support: https://docs.openshift.com/container-platform/3.10/install_config/router/customized_haproxy_router.html

Use the information in the mitigation section to diagnose and mitigate the problem in OpenShift 3.9 and later. Both OpenShift 3.9, and 3.10 are out of support scope according to: https://access.redhat.com/support/policy/updates/openshift_noncurrent

The webconsole (openshift3/ose-console pods of the openshift-console project) seems to be affected:

$ curl -vvv --http2 https://openshift.example.com:8443 2>&1 >/dev/null | grep ALPN * ALPN, offering h2 * ALPN, offering http/1.1 * ALPN, server accepted to use h2

Or does this is a false positive too?

Hi Thomas,

Yes, this is a false positive.

There are actually two consoles in 3.11 -

  • cluster console; the one in the doc above, and which has a reencrypt route configured in haproxy

  • application console; this one you mention - it's not behind haproxy in the router pod, and is a golang binary that supports http/2 natively

You can see the details of that first one with, "oc get route -n openshift-console".

You can also check IPs of the hostnames for each console to confirm that they're different. One is handled directly and one via the router pods.

We'll update the docs to make this clearer.

As i dont have curl version installed to support http2 . So I checked the haproxy configuration running for our routers and it dont have bind stanza with http2 support . Is this a valid check ? we are running openshift 3.11

cat /var/lib/haproxy/conf/haproxy.config | grep -i bind ssl-default-bind-options no-sslv3 ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS bind :80 bind :443 bind 127.0.0.1:10444 ssl no-sslv3 crt /etc/pki/tls/private/tls.crt crt-list /var/lib/haproxy/conf/cert_config.map accept-proxy bind 127.0.0.1:10443 ssl no-sslv3 crt /etc/pki/tls/private/tls.crt accept-proxy

I also check the router configuration for ROUTER_ENABLE_HTTP2 and we didnt set that .

Please confirm if the above are valid checks.

Yes, this is the check noted above as "You can check if HTTP/2 is enabled by searching your HAProxy configuration files for a line containing 'h2'."

I'll reformat the output you provided to split by lines to make it clearer, and also restrict to "bind " (the trailing space filters out the bind-options and bind-ciphers that make it harder to read).

grep -i 'bind ' /var/lib/haproxy/conf/haproxy.config

bind :80

bind :443

bind 127.0.0.1:10444 ssl no-sslv3 crt /etc/pki/tls/private/tls.crt crt-list /var/lib/haproxy/conf/cert_config.map accept-proxy

bind 127.0.0.1:10443 ssl no-sslv3 crt /etc/pki/tls/private/tls.crt accept-proxy

In conclusion, this test is good and it's now easy to observe that "h2" does not appear in any bind statement.