OpenShift Container Platform - Node keeps crashing on Azure after upgarde to 3.11.153
Issue
- After upgrading to 3.11.153, the
atomic-openshift-node
do not start on Azure. It keeps failing on querying the Azure API to get thehostname
. As a result, node never becomes ready. We also see the following stack-trace in the logs:
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: I1028 12:24:57.084837 90588 kubelet_node_status.go:317] Adding node label from cloud provider: beta.kubernetes.io/instance-type=Standard_B2ms
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: E1028 12:24:57.084993 90588 runtime.go:66] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:72
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:65
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:51
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:573
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/panic.go:502
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/panic.go:63
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/signal_unix.go:388
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_standard.go:418
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/cloudprovider/providers/azure/azure_zones.go:51
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:323
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:73
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet_node_status.go:376
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/pkg/kubelet/kubelet.go:1360
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /builddir/build/BUILD/atomic-openshift-git-0.aaf3f71/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88
Oct 28 12:24:57 node.example.com atomic-openshift-node[90577]: /opt/rh/go-toolset-1.10/root/usr/lib/go-toolset-1.10-golang/src/runtime/asm_amd64.s:2361
Environment
- Red Hat OpenShift Container Platform 3.11
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.