OpenShift HTTP connection closed between HAProxy and client, but not between HAProxy and application pod

Solution Verified - Updated -

Environment

  • OpenShift 4 (RHOCP)

Issue

There is an HTTP connection established by a client to an application in an OpenShift cluster through a secured route1. When the client closes the connection, the connection between the client and the HAProxy is closed, but the same does not happen between the HAProxy and the application pod.

  1. Secured routes ↩︎

Resolution

In order to have a one to one client server connection, where the connection closure in the client closes the connection on the application side, there are two options:

  1. Use NodePort to talk directly to the service, skipping the HAProxy. ExternalIP needs to be bound to a NIC on one of the nodes in the cluster. External routing must be set to get the traffic to the node on which the IP is bound.

  2. Use a Passthrough route. In this case, the pod needs to handle the certificate, and the TLS is handled also by the pod.

Root Cause

When a re-encrypt or edge secure route is used, the HAProxy acts as an HTTP proxy. In this mode, HAProxy receives an HTTP request from a client on one TCP connection, forwards the HTTP request to a server on another TCP connection, and then forwards the HTTP response from the server to the client. The client controls the connection to HAProxy, but HAProxy controls the connection to the server. There is no promise that HAProxy will forward a FIN packet or close the server connection when the client closes its connection. HAProxy keeps its connection to the server open in order to be re-used in case further client requests come in.

Diagnostic Steps

  1. Get a tcpdump in the router pod, in order to get the traffic between the client and the router. Follow the process described in the article How to use tcpdump inside OpenShift v4 Pod (with ssh). Capture the traffic coming from the client IP.

  2. Get a tcpdump inside the application pod, following the same process as for the router pod. In this case, the traffic coming from the router IP needs to be captured.

  3. Open a connection on the client side, and close it.

  4. Check the tcpdump on the HAProxy. A FIN packet can be found coming from the client IP to the router IP.

TCPDUMP FIN

  1. Check the tcpdump on the pod. The FIN packet is not sent from the router IP to the pod IP.

TCPDUMP No FIN

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments