Operators fail due to leader election lost

Solution Verified - Updated -

Issue

  • APIcast operator fails at random intervals and is subsequently restarted with an error like this

    E0221 12:52:25.963270       1 leaderelection.go:361] Failed to update lock: resource name may not be empty
    I0221 12:52:26.589096       1 leaderelection.go:278] failed to renew lease apicast-operator/988b4062.3scale.net: timed out waiting for the condition
    {"level":"info","ts":1645447946.986577,"logger":"controller-runtime.manager.controller.apicast","msg":"Stopping workers","reconciler group":"apps.3scale.net","reconciler kind":"APIcast"}
    {"level":"error","ts":1645447947.510864,"logger":"setup","msg":"problem running manager","error":"leader election lost","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/remote-source/deps/gomod/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132\nmain.main\n\t/remote-source/app/main.go:102\nruntime.main\n\t/opt/rh/go-toolset-1.13/root/usr/lib/go-toolset-1.13-golang/src/runtime/proc.go:203"}
    
  • 3scale operator thrown this error

    E1215 07:53:08.028116       1 leaderelection.go:320] error retrieving resource lock 3scale/12345678.3scale.net: Get https://10.246.0.1:443/api/v1/namespaces/3scale/configmaps/12345678.3scale.net: context deadline exceeded
    {"level":"error","ts":1671090788.0282373,"logger":"setup","msg":"problem running manager","error":"leader election lost","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Errornt/remote-source/deps/gomod/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128nmain.mainnt/remote-source/app/main.go:318nruntime.mainnt/opt/rh/go-toolset-1.13/root/usr/lib/go-toolset-1.13-golang/src/runtime/proc.go:203"}
    

Environment

  • Red Hat 3scale API Management
    • Apicast operator
    • 3scale operator
    • 2.11.0 On-Premise
    • 2.12.0 On-Premise
    • 2.13.0 On-Premise

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content