Repeating "http2: server: error reading preface from client" logs in OpenShift

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Red Hat OpenShift Service on AWS (ROSA)
    • 4
  • Red Hat OpenShift Dedicated (OSD)
    • 4
  • Go application
  • HTTP/2

Issue

  • The following messages are flooding application logs in OpenShift:

    http2: server connection from 10.x.y.z:44444 on 0xc000a20e00
http2: server: error reading preface from client 10.x.y.z:44444: read tcp 10.v.z.n:8443->10.x.y.z:44444: read: connection reset by peer

Resolution

Check if the application running is a Go application and remove the GODEBUG=http2debug=1 configuration parameter.

Root Cause

This is typically a health check TCP connection from the OpenShift ingress, if the HTTP/2 Ingress connectivity is enabled, and in most cases this message is harmless.

A HTTP/2 enabled server endpoint running a Go application with the parameter GODEBUG=http2debug=1 configured produces this message when a client (like the OpenShift ingress) didn't send HTTP/2 preface message after establishing a TCP connection and disconnected.

Diagnostic Steps

Check if the HTTP/2 Ingress connectivity is enabled in the cluster, searching for the annotation ingress.operator.openshift.io/default-enable-http2: "true" in the ingress configuration (for the whole cluster) or in a specific ingresscontroller:

$ oc get ingresses.config cluster -o yaml

$ oc get ingresscontroller -n openshift-ingress-operator -o yaml

In some cases, the configuration parameter in the Go application is set in the application pod:

$ oc get pod [pod_name] -n [namespace_name] -o yaml
[...]
  containers:
  - env:
  [...]
    - name: GODEBUG
      value: http2debug=1
[...]

But it could be configured directly in the application.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments