OpenShift router 1936:/healthz healthcheck failing.

Solution In Progress - Updated -

Environment

OpenShift Enterprise Container Platform 3.7+

Issue

  • I make a "Get" request from an external client to the OpenShift router and the connection fails.
  • My health check for the OpenShift router worked in 3.6 hitting the router stats port but in 3.7+ its failing.

Resolution

Open up an iptables rule for port 1936 and make sure the HTTP request method is all capital letters.

The following works:

# telnet ROUTER_IP  1936

GET /healthz HTTP/1.1
host:router.apps.ocp.example.com
connection:close

Root Cause

In 3.6 the 1936:/healthz endpoint would send back a 200 response for the following request method.

# telnet ROUTER_IP  1936

Get /healthz HTTP/1.1
host:router.apps.ocp.example.com
connection:close

In 3.7 and above if this is sent the router will reset the connection, only a HTTP request method with all capital letters will work. According to rfc2616 a method is case-sensitive.

Diagnostic Steps

The packet trace will look like

$ tshark -r trace.pcap  -Y "tcp.stream == 0"
    1   0.000000 10.0.0..8 → 10.0.0..59 TCP 74 59507 → 1936 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=2070243190 TSecr=0 WS=128
    2   0.000069 10.0.0..59 → 10.0.0..8 TCP 74 1936 → 59507 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=222353196 TSecr=2070243190 WS=128
    3   0.000277 10.0.0..8 → 10.0.0..59 TCP 66 59507 → 1936 [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSval=2070243191 TSecr=222353196
    4   0.000438 10.0.0..8 → 10.0.0..59 TCP 144 59507 → 1936 [PSH, ACK] Seq=1 Ack=1 Win=14720 Len=78 TSval=2070243191 TSecr=222353196
    5   0.000452 10.0.0..59 → 10.0.0..8 TCP 66 1936 → 59507 [ACK] Seq=1 Ack=79 Win=29056 Len=0 TSval=222353196 TSecr=2070243191
    6   0.000515 10.0.0..59 → 10.0.0..8 TCP 66 1936 → 59507 [RST, ACK] Seq=1 Ack=79 Win=29056 Len=0 TSval=0 TSecr=2070243191

The data packet sent in frame 4 by the client has the following data. Note that the HTTP method is spelled with an initial upper case "G" and lower case "et"

$ tshark -r port1936.pcap  -Y "frame.number == 4" -x | tail -6 | head -3
0040  d7 2c 47 65 74 20 2f 68 65 61 6c 74 68 7a 20 48   .,Get /healthz H
0050  54 54 50 2f 31 2e 31 0d 0a 43 6f 6e 6e 65 63 74   TTP/1.1..Connect
0060  69 6f 6e 3a 43 6c 6f 73 65 0d 0a 48 6f 73 74 3a   ion:Close..Host:

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments