Crio panics with "panic: close of closed channel " after attempting to stop a container in OpenShift Container Platform 4

Solution Verified - Updated -

Issue

  • crio panics with panic: close of closed channel and the below stacktrace:

    Mar 14 15:23:27 worker-1 hyperkube[1692]: I0314 15:23:27.387723    1692 kubelet.go:1954] "SyncLoop REMOVE" source="api" pods=[debug/worker-1-debug]
    Mar 14 15:23:27 worker-1 hyperkube[1692]: I0314 15:23:27.387792    1692 kubelet_pods.go:1285] "Killing unwanted pod" podName="worker-1-debug"
    Mar 14 15:23:27 worker-1 hyperkube[1692]: I0314 15:23:27.387838    1692 kuberuntime_container.go:720] "Killing container with a grace period override" pod="debug/worker-1-debug" podUID=ed3fba93-12fd-40ba-af22-207bc2dd7ebd containerName="container-00" containerID="cri-o://5e4a5efe282d3f77fe472d8810fa9a8a61df545a6087a7e8ecaa9379b7f1fa5c" gracePeriod=2
    Mar 14 15:23:27 worker-1 systemd[1]: crio-conmon-5e4a5efe282d3f77fe472d8810fa9a8a61df545a6087a7e8ecaa9379b7f1fa5c.scope: Consumed 55ms CPU time
    Mar 14 15:23:27 worker-1 crio[1636]: panic: close of closed channel
    Mar 14 15:23:27 worker-1 crio[1636]: goroutine 5778599 [running]:
    Mar 14 15:23:27 worker-1 crio[1636]: panic(0x55c2b280a280, 0x55c2b2aa4f90)
    Mar 14 15:23:27 worker-1 crio[1636]:         /usr/lib/golang/src/runtime/panic.go:1065 +0x565 fp=0xc001827530 sp=0xc001827468 pc=0x55c2b098e8a5
    Mar 14 15:23:27 worker-1 crio[1636]: runtime.closechan(0xc00187e300)
    Mar 14 15:23:27 worker-1 crio[1636]:         /usr/lib/golang/src/runtime/chan.go:363 +0x3f5 fp=0xc001827570 sp=0xc001827530 pc=0x55c2b095cbb5
    Mar 14 15:23:27 worker-1 crio[1636]: github.com/cri-o/cri-o/internal/oci.(*runtimeOCI).StopContainer.func1(0xc001827678, 0xc0015e5080)
    Mar 14 15:23:27 worker-1 crio[1636]:         /builddir/build/BUILD/cri-o-c05847896bc721f6529b1ceb4bafaf6cfe523b5d/_output/src/github.com/cri-o/cri-o/internal/oci/runtime_oci.go:701 +0x49 fp=0xc001827588 sp=0xc001827570 pc=0x55c2b1fd0a29
    Mar 14 15:23:27 worker-1 crio[1636]: github.com/cri-o/cri-o/internal/oci.(*runtimeOCI).StopContainer(0xc001ad3530, 0x55c2b2b19a20, 0xc001a90c00, 0xc0015e5080, 0x2, 0x55c2b2ac7d68, 0xc0000da780)
    Mar 14 15:23:27 worker-1 crio[1636]:         /builddir/build/BUILD/cri-o-c05847896bc721f6529b1ceb4bafaf6cfe523b5d/_output/src/github.com/cri-o/cri-o/internal/oci/runtime_oci.go:710 +0x788 fp=0xc001827650 sp=0xc001827588 pc=0x55c2b1fc3948
    Mar 14 15:23:27 worker-1 crio[1636]: github.com/cri-o/cri-o/internal/oci.(*Runtime).StopContainer(0xc0005fe5d0, 0x55c2b2b19a20, 0xc001a90c00, 0xc0015e5080, 0x2, 0x0, 0x8000101)
    Mar 14 15:23:27 worker-1 crio[1636]:         /builddir/build/BUILD/cri-o-c05847896bc721f6529b1ceb4bafaf6cfe523b5d/_output/src/github.com/cri-o/cri-o/internal/oci/oci.go:323 +0x9d fp=0xc001827698 sp=0xc001827650 pc=0x55c2b1fba85d
    Mar 14 15:23:27 worker-1 crio[1636]: github.com/cri-o/cri-o/internal/lib.(*ContainerServer).StopContainer(0xc0006be180, 0x55c2b2b19a20, 0xc001a90c00, 0xc0015e5080, 0x2, 0xc000daebd0, 0x0)
    Mar 14 15:23:27 worker-1 crio[1636]:         /builddir/build/BUILD/cri-o-c05847896bc721f6529b1ceb4bafaf6cfe523b5d/_output/src/github.com/cri-o/cri-o/internal/lib/stop.go:14 +0x79 fp=0xc001827748 sp=0xc001827698 pc=0x55c2b2009159
    Mar 14 15:23:27 worker-1 crio[1636]: github.com/cri-o/cri-o/server.(*Server).StopContainer(0xc0001ba580, 0x55c2b2b19a20, 0xc001a90c00, 0xc001827840, 0x55c2b0962325, 0x55c2b2977e00)
    Mar 14 15:23:27 worker-1 crio[1636]:         /builddir/build/BUILD/cri-o-c05847896bc721f6529b1ceb4bafaf6cfe523b5d/_output/src/github.com/cri-o/cri-o/server/container_stop.go:34 +0x349 fp=0xc001827810 sp=0xc001827748 pc=0x55c2b2076b69
    Mar 14 15:23:27 worker-1 crio[1636]: github.com/cri-o/cri-o/server/cri/v1alpha2.(*service).StopContainer(0xc00059c080, 0x55c2b2b19a20, 0xc001a90c00, 0xc00195c220, 0xc00059c080, 0x1, 0x1)
    Mar 14 15:23:27 worker-1 crio[1636]:         /builddir/build/BUILD/cri-o-c05847896bc721f6529b1ceb4bafaf6cfe523b5d/_output/src/github.com/cri-o/cri-o/server/cri/v1alpha2/rpc_stop_container.go:17 +0x85 fp=0xc001827868 sp=0xc001827810 pc=0x55c2b2184625
    

Environment

  • Red Hat OpenShift Container Platform (OCP) 4

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content