Operators fail due to leader election lost
Issue
-
APIcast operator fails at random intervals and is subsequently restarted with an error like this
E0221 12:52:25.963270 1 leaderelection.go:361] Failed to update lock: resource name may not be empty I0221 12:52:26.589096 1 leaderelection.go:278] failed to renew lease apicast-operator/988b4062.3scale.net: timed out waiting for the condition {"level":"info","ts":1645447946.986577,"logger":"controller-runtime.manager.controller.apicast","msg":"Stopping workers","reconciler group":"apps.3scale.net","reconciler kind":"APIcast"} {"level":"error","ts":1645447947.510864,"logger":"setup","msg":"problem running manager","error":"leader election lost","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/remote-source/deps/gomod/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132\nmain.main\n\t/remote-source/app/main.go:102\nruntime.main\n\t/opt/rh/go-toolset-1.13/root/usr/lib/go-toolset-1.13-golang/src/runtime/proc.go:203"}
-
3scale operator thrown this error
E1215 07:53:08.028116 1 leaderelection.go:320] error retrieving resource lock 3scale/12345678.3scale.net: Get https://10.246.0.1:443/api/v1/namespaces/3scale/configmaps/12345678.3scale.net: context deadline exceeded {"level":"error","ts":1671090788.0282373,"logger":"setup","msg":"problem running manager","error":"leader election lost","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Errornt/remote-source/deps/gomod/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128nmain.mainnt/remote-source/app/main.go:318nruntime.mainnt/opt/rh/go-toolset-1.13/root/usr/lib/go-toolset-1.13-golang/src/runtime/proc.go:203"}
Environment
- Red Hat 3scale API Management
- Apicast operator
- 3scale operator
- 2.11.0 On-Premise
- 2.12.0 On-Premise
- 2.13.0 On-Premise
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.