API Priority and Fairness Alpha featureGate throttles requests in OpenShift v4.5
Issue
Certain workloads can trigger Priority and Fairness (or P&F) to throttle requets from OAS which can cause an OpenShift 4.5 cluster to become unresponsive
E0929 06:01:07.495783 1 errors.go:77] the server has received too many requests and has asked us to try again later (post subjectaccessreviews.authorization.k8s.io)
E0929 06:01:08.387416 1 webhook.go:199] Failed to make webhook authorizer request: Post https://172.30.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0929 06:01:08.387716 1 errors.go:77] Post https://172.30.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews: context canceled
E0929 06:01:08.387975 1 runtime.go:78] Observed a panic: &errors.errorString{s:"killing connection/stream because serving request timed out and response had been started"} (killing con
nection/stream because serving request timed out and response had been started)
goroutine 11240557 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x27ebb80, 0xc0004c6ed0)
@/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0xc004fadbd8, 0x1, 0x1)
@/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82
panic(0x27ebb80, 0xc0004c6ed0)
runtime/panic.go:679 +0x1b2
Environment
- Red Hat OpenShift Container Platform (OCP) 4.5.0 -> 4.5.15
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.