OpenShift Container Platform - Node keeps crashing on Azure after upgarde to 3.11.153

Solution Verified - Updated -

Issue

  • After upgrading to 3.11.153, the atomic-openshift-node do not start on Azure. It keeps failing on querying the Azure API to get the hostname. As a result, node never becomes ready. We also see the following stack-trace in the logs:
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: I1028 12:24:57.084837   90588 kubelet_node_status.go:317] Adding node label from cloud provider: beta.kubernetes.io/instance-type=Standard_B2ms
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: E1028 12:24:57.084993   90588 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:573
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/panic.go:502
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/panic.go:63
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/signal_unix.go:388
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_standard.go:418
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_zones.go:51
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:323
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:73
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:376
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1360
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361

Environment

  • Red Hat OpenShift Container Platform 3.11

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content