ovnkube-master is in CrashloopBackOff state after upgrading cluster to OpenShift v4.13
Issue
ovnkube-master
POD stuck inCrashLoopBackOff
after upgrading cluster to 4.13.ovnkube-master
POD restarting continuously with below panic:
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1a80fa5]
goroutine 102 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x30?})
/go/src/github.com/openshift/ovn-kubernetes/go-controller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:56 +0xd7
panic({0x1ceffe0, 0x32652a0})
/usr/lib/golang/src/runtime/panic.go:884 +0x212
github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn.(*DefaultNetworkController).syncNetworkPolicies(0xc000c33680, {0xc001a26ce0, 0x2, 0x42a927?})
/go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/policy.go:351 +0xd65
- ovnkube-master PODs are going into
CrashLoopBackOff
due to presence of ACL with priority 1013 and without a name.
$ oc rsh -n openshift-ovn-kubernetes -c northd $(oc get po -n openshift-ovn-kubernetes -owide | grep ovnkube-master| grep $(oc get lease -n openshift-ovn-kubernetes ovn-kubernetes-master -o template --template={{.spec.holderIdentity}})| awk '{print $1}') ovn-nbctl --no-leader list acl |grep -A 2 -B 10 1013
_uuid : 14934bbb-0d23-4963-96ce-338be3c203f8
action : allow
direction : from-lport
external_ids : {default-deny-policy-type=Egress}
label : 0
log : false
match : "inport == @clusterRtrPortGroup && (ip4.mcast || mldv1 || mldv2 || (ip6.dst[120..127] == 0xff && ip6.dst[116] == 1))"
meter : []
name : []
options : {apply-after-lb="true"}
priority : 1013
severity : []
tier : 0
--
_uuid : 3d73790e-690c-4332-9c54-d729f23ec3a9
action : allow
direction : to-lport
external_ids : {}
label : 0
log : false
match : "outport == @clusterRtrPortGroup && (ip4.mcast || mldv1 || mldv2 || (ip6.dst[120..127] == 0xff && ip6.dst[116] == 1))"
meter : []
name : []
options : {}
priority : 1013
severity : []
tier : 0
Environment
- Red Hat OpenShift Container Platform
- 4.13
- OVNKubernetes
- Multicast enabled (by default it is enabled) while using OVNKubernetes CNI in OpenShift version 4.7 and later upgraded to 4.13
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.