HTTP/2.0 and SNI causing incorrect responses

Root Cause

The HTTP/2.0 Specification

The HTTP/2.0 specification was designed to speed up the HTTP/1.0 specification by defining 2 key parts of the protocol

• Enable multiple HTTP(S)) requests, to the same origin server, to re-use the same connection
• Enable connection coalescing where URIs with different SNI endpoints can share the same connection as long as:
  • the origin server is authoritative for both server names. That is, both server names resolve to the same IP address
  • and for HTTPS connections, the URI (provided in the SNI) resolves to the same TLS certificate

This means that a client only needs to establish one TLS connection to be able to access multiple resources in parallel (scripts.myserver.com; images.myserver.com; .....), when the resources share the same TLS certificate. This Speeds up access to all of a server's content.

Unexpected Consequence for Proxy Servers

Unexpected consequences are documented in the HTTP/2.0 specification

In some deployments, reusing a connection for multiple origins can result in requests being directed to the wrong origin server. For example, TLS termination might be performed by a middlebox that uses the TLS Server Name Indication [TLS-EXT] extension to select an origin server. This means that it is possible for clients to send requests to servers that might not be the intended target for the request, even though the server is otherwise authoritative.

HTTP/2.0 RFC - 9.1.1. Connection Reuse

Example Replication of Issue

A client can re-create the issue discussed in this knowledge article by:

• The client requests a new connection to a.mycluster.com (serverA) by:
  • Requesting the IP via DNS lookup;
  • ... Other steps discussed later;
  • Sending a connection request to the IP, as this is the first time the client is communicating with server A.
• The Ingress Router (HAProxy) sees it as a new connection and a HTTPS request so it:
  • Checks for a Route defined for the SNI (provided in the request) and identifies:
    • Service A as the app / API endpoint;
    • and EDGE as the TLS termination type (the default termination type).
  • Terminates the connection at the Ingress Router by establishing a TLS connection with the client and establishing a HTTP connection with Service A;
  • Stores a reference to this information so that future HTTP(S) requests can re-use it.
• Once the connection is established the client can send multiple requests through the connection in parallel. Speeding up the communication with the server by not needing to re-establish TLS connections for each request;
• The client now wishes to also connect with b.mycluster.com (serverB) by:
  • Requesting the IP via DNS lookup - It gets the same IP;
  • Since the client already has a connection to that IP it checks the new server certificate and confirms it resolves to the same TLS certificate (see specification above). The easiest and default setup in OpenShift is to use a wildcard certificate;
  • Since the IP and certificates match, the client will re-use the connection and TLS session;
  • The client sends a HTTP request for b.mycluster.com through the already established connection.
• The Ingress Route receives the request and identifies it as an established connection to server A and forwards the request onto Service A;
• The application / API behind Service A will either:
  • Check the SNI, see the URI is not for itself and return a 421 (Misdirected Request);
  • Check for the resource that is requested from Service B see that it does not exist on its server and returns a HTTP status code of 404 (Not Found);
  • Return an incorrect result because the resource existed on both ServerA and ServerB but the information is different.

Example Expected Result

• The client requests a new connection to a.mycluster.com (serverA) by:
  • Requesting the IP via DNS lookup;
  • ... Other steps discussed later;
  • Sending a connection request to the IP, as this is the first time the client is communicating with server A.
• The Ingress Router (HAProxy) sees it as a new connection and a HTTPS request so it:
  • Checks for a Route defined for the SNI (provided in the request) and identifies:
    • Service A as the app / API endpoint;
    • and EDGE as the TLS termination type (the default termination type).
  • Terminates the connection at the Ingress Router by establishing a TLS connection with the client and establishing a HTTP connection with Service A;
  • Stores a reference to this information so that future HTTP(S) requests can re-use it.
• Once the connection is established the client can send multiple requests through the connection in parallel. Speeding up the communication with the server by not needing to re-establish TLS connections for each request;
• The client now wishes to also connect with b.mycluster.com (serverB) by:
  • Requesting the IP via DNS lookup - It gets the same IP;
  • Since the client already has a connection to that IP it checks the new server certificate and confirms it resolves to the same TLS certificate (see specification above). The easiest and default setup in OpenShift is to use a wildcard certificate;
  • Since the IP and certificates match, the client will re-use the connection and TLS session;
  • The client sends a HTTP request for b.mycluster.com through the already established connection.
• The Ingress Route receives the request and identifies it as an established connection to server A and forwards the request onto Service A;
• The application / API behind Service A checks the SNI, sees the URI is not for itself and return a 421 (Misdirected Request);
• The client responds to the 421 (Misdirected Request) by requesting a new connection;
• The Ingress Router (HAProxy) sees it as a new connection and a HTTPS request so it:
  • Checks for a Route defined for the SNI (provided in the request) and identifies:
  • Service B as the app / API endpoint;
  • and EDGE as the TLS termination type (the default termination type).
  • Terminates the connection at the Ingress Router by establishing a TLS connection with the client and establishing a HTTP connection with Service B;
  • Stores a reference to this information so that future HTTP(S) requests can re-use it.
• Once the connection is established the client can send multiple requests through the connection in parallel. Speeding up the communication with the server by not needing to re-establish TLS connections for each request;

Note: There would be no issues with connecting to the correct application / API, if the web applications/APIs can send 421 (Misdirected Request) and the client can handle this status code correctly.