ovnkube-master is in CrashloopBackOff state after upgrading cluster to OpenShift v4.13

Solution In Progress - Updated -

Issue

  • ovnkube-master POD stuck in CrashLoopBackOff after upgrading cluster to 4.13.
  • ovnkube-master POD restarting continuously with below panic:
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1a80fa5]

goroutine 102 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0x30?})
        /go/src/github.com/openshift/ovn-kubernetes/go-controller/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:56 +0xd7
panic({0x1ceffe0, 0x32652a0})
        /usr/lib/golang/src/runtime/panic.go:884 +0x212
github.com/ovn-org/ovn-kubernetes/go-controller/pkg/ovn.(*DefaultNetworkController).syncNetworkPolicies(0xc000c33680, {0xc001a26ce0, 0x2, 0x42a927?})
        /go/src/github.com/openshift/ovn-kubernetes/go-controller/pkg/ovn/policy.go:351 +0xd65
  • ovnkube-master PODs are going into CrashLoopBackOff due to presence of ACL with priority 1013 and without a name.
$ oc rsh -n openshift-ovn-kubernetes -c northd $(oc get po -n openshift-ovn-kubernetes -owide | grep ovnkube-master| grep $(oc get lease -n openshift-ovn-kubernetes ovn-kubernetes-master -o template --template={{.spec.holderIdentity}})| awk '{print $1}') ovn-nbctl --no-leader list acl |grep -A 2 -B 10 1013
_uuid               : 14934bbb-0d23-4963-96ce-338be3c203f8
action              : allow
direction           : from-lport
external_ids        : {default-deny-policy-type=Egress}
label               : 0
log                 : false
match               : "inport == @clusterRtrPortGroup && (ip4.mcast || mldv1 || mldv2 || (ip6.dst[120..127] == 0xff && ip6.dst[116] == 1))"
meter               : []
name                : []
options             : {apply-after-lb="true"}
priority            : 1013
severity            : []
tier                : 0
--
_uuid               : 3d73790e-690c-4332-9c54-d729f23ec3a9
action              : allow
direction           : to-lport
external_ids        : {}
label               : 0
log                 : false
match               : "outport == @clusterRtrPortGroup && (ip4.mcast || mldv1 || mldv2 || (ip6.dst[120..127] == 0xff && ip6.dst[116] == 1))"
meter               : []
name                : []
options             : {}
priority            : 1013
severity            : []
tier                : 0

Environment

  • Red Hat OpenShift Container Platform
    • 4.13
  • OVNKubernetes
  • Multicast enabled (by default it is enabled) while using OVNKubernetes CNI in OpenShift version 4.7 and later upgraded to 4.13

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content