Autoscaler pod in CrashLoopBackOff state after recent OpenShift Container Platform update

Solution Verified - Updated -

Issue

  • After updating OpenShift to 4.14.17 we are observing autoscaler-default pod in CrashLoopBackOff state with the below error being reported.

    I0327 10:37:39.166179 1 node_instances_cache.go:168] Refresh cloud provider node instances cache finished, refresh took 31.131576ms
    panic: interface conversion: interface {} is nil, not string
    goroutine 128 [running]:
    k8s.io/autoscaler/cluster-autoscaler/cloudprovider/clusterapi.unstructuredToTaint(...)
    /go/src/k8s.io/autoscaler/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_unstructured.go:246
    k8s.io/autoscaler/cluster-autoscaler/cloudprovider/clusterapi.unstructuredScalableResource.Taints({0xc000ba8b60?, 0xc000126f00?, 0xc0009daf80?, 0x2?})
    /go/src/k8s.io/autoscaler/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_unstructured.go:214 +0x8a5
    k8s.io/autoscaler/cluster-autoscaler/cloudprovider/clusterapi.(*nodegroup).TemplateNodeInfo(0xc00074be20)
    /go/src/k8s.io/autoscaler/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup.go:266 +0x2ea
    k8s.io/autoscaler/cluster-autoscaler/core/utils.GetNodeInfoFromTemplate({0x26db450, 0xc00074be20}, {0xc0047e2100, 0x16, 0x20}, {0xc003b1d2f0?, 0xc003b1d320?})
    /go/src/k8s.io/autoscaler/cluster-autoscaler/core/utils/utils.go:41 +0x9d
    k8s.io/autoscaler/cluster-autoscaler/processors/nodeinfosprovider.(*MixedTemplateNodeInfoProvider).Process(0xc000959440, 0xc003cd8400, {0xc0065b1740, 0x8, 0x69561a?}, {0xc0047e2100, 0x16, 0x20}, {0xc003b1d2f0, 0xc003b1d320}, ...)
    /go/src/k8s.io/autoscaler/cluster-autoscaler/processors/nodeinfosprovider/mixed_nodeinfos_processor.go:155 +0x599
    k8s.io/autoscaler/cluster-autoscaler/core.(*StaticAutoscaler).RunOnce(0xc0006c6160, {0x4?, 0x0?, 0x3b8baa0?})
    /go/src/k8s.io/autoscaler/cluster-autoscaler/core/static_autoscaler.go:352 +0xc8a
    main.run(0xc0000d7000?, {0x26d1f18, 0xc000742270})
    /go/src/k8s.io/autoscaler/cluster-autoscaler/main.go:489 +0x2cd
    main.main.func2({0x931de6?, 0xc000137380?})
    /go/src/k8s.io/autoscaler/cluster-autoscaler/main.go:577 +0x25
    created by k8s.io/client-go/tools/leaderelection.(*LeaderElector).Run
    /go/src/k8s.io/autoscaler/cluster-autoscaler/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:208 +0x105
    
  • We are seeing autoscaler-default in CrashLoopBackOff state after recent update of OpenShift.

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4.13.38 and later
    • 4.14.17 and later
    • 4.15.0 and later

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content