API Priority and Fairness Alpha featureGate throttles requests in OpenShift v4.5

Solution Verified - Updated -

Issue

Certain workloads can trigger Priority and Fairness (or P&F) to throttle requets from OAS which can cause an OpenShift 4.5 cluster to become unresponsive

E0929 06:01:07.495783       1 errors.go:77] the server has received too many requests and has asked us to try again later (post subjectaccessreviews.authorization.k8s.io)
E0929 06:01:08.387416       1 webhook.go:199] Failed to make webhook authorizer request: Post https://172.30.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0929 06:01:08.387716       1 errors.go:77] Post https://172.30.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0929 06:01:08.387975       1 runtime.go:78] Observed a panic: &errors.errorString{s:"killing connection/stream because serving request timed out and response had been started"} (killing con
nection/stream because serving request timed out and response had been started)
goroutine 11240557 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x27ebb80, 0xc0004c6ed0)
        @/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0xc004fadbd8, 0x1, 0x1)
        @/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82
panic(0x27ebb80, 0xc0004c6ed0)
        runtime/panic.go:679 +0x1b2

Environment

  • Red Hat OpenShift Container Platform (OCP) 4.5.0 -> 4.5.15

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content