Node is "NotReady" during upgrade to 4.8

Solution Verified - Updated -

Issue

  • When upgrading from OpenShift Container Platform 4.7 to 4.8.5, one of the Master Nodes is stuck in the "NotReady" state.
  • The logs are showing regular fatal errors ("panic: close of closed channel") from CRI-O:

    Aug 17 09:28:51 ip-10-0-139-77 crio[1298394]: panic: close of closed channel
    Aug 17 09:28:51 ip-10-0-139-77 crio[1298394]: goroutine 1323 [running]:
    Aug 17 09:28:51 ip-10-0-139-77 crio[1298394]: panic(0x5573ffea0e20, 0x55740013b840)
    Aug 17 09:28:51 ip-10-0-139-77 crio[1298394]:         /usr/lib/golang/src/runtime/panic.go:1065 +0x565 fp=0xc00063d5b0 sp=0xc00063d4e8 pc=0x5573fe02b4a5
    Aug 17 09:28:51 ip-10-0-139-77 crio[1298394]: runtime.closechan(0xc00185aba0)
    Aug 17 09:28:51 ip-10-0-139-77 crio[1298394]:         /usr/lib/golang/src/runtime/chan.go:363 +0x3f5 fp=0xc00063d5f0 sp=0xc00063d5b0 pc=0x5573fdff97b5
    Aug 17 09:28:51 ip-10-0-139-77 crio[1298394]: github.com/cri-o/cri-o/internal/oci.(*runtimeOCI).ReopenContainerLog(0xc0005cbe30, 0x5574001b0240, 0xc000d1fad0, 0xc000dee000, 0x55740015e5f8, 0xc002024d50)
    Aug 17 09:28:51 ip-10-0-139-77 crio[1298394]:         /builddir/build/BUILD/cri-o-3b87110e2208fd889e35829d92f4400742568fa6/_output/src/github.com/cri-o/cri-o/internal/oci/runtime_oci.go:1050 +0x7c6 fp=0xc00063d758 sp=0xc00063d5f0 pc=0x5573ff65e186
    Aug 17 09:28:51 ip-10-0-139-77 crio[1298394]: github.com/cri-o/cri-o/internal/oci.(*Runtime).ReopenContainerLog(0xc0006aff50, 0x5574001b0240, 0xc000d1fad0, 0xc000dee000, 0x0, 0x0)
    Aug 17 09:28:51 ip-10-0-139-77 crio[1298394]:         /builddir/build/BUILD/cri-o-3b87110e2208fd889e35829d92f4400742568fa6/_output/src/github.com/cri-o/cri-o/internal/oci/oci.go:433 +0x93 fp=0xc00063d798 sp=0xc00063d758 pc=0x5573ff654113
    
  • CRI-O is in a restart loop following the panics above.

Environment

  • Red Hat OpenShift Container Platform (OCP) 4.7.22
  • Red Hat OpenShift Container Platform (OCP) 4.8.5

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content