Podsandbox creation failing after rebooting/upgrading hosts due to SElinux denials

Solution Verified - Updated -

Issue

  • After rebooting host, pods failed to start with series of below errors in node service logs:
$ journalctl  atomic-openshift-node.service
Jul 29 08:40:36 node1.test.example.com atomic-openshift-node[101620]: I0729 08:40:36.954916  101620 kuberuntime_manager.go:403] No ready sandbox for pod "apiserver-c72wp_kube-service-catalog(f6b0b99c-d161-11ea-896b-00505685765c)" can be found. Need to start a new one.
Jul 29 08:40:36 node1.test.example.com atomic-openshift-node[101620]: W0729 08:40:36.964780  101620 cni.go:243] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "6dcd888fe304c21d0df1e8743a25c70f9c4fc8dc5ad5c42d079e32312fb8fb1e"
  • Pods go in CrashLoopBackOff state with following events:
Events:
  Type     Reason                  Age                From                                                                     Message
  ----     ------                  ----               ----                                                                     -------
  Warning  FailedCreatePodSandBox  25m                kubelet, xxxx        Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6dcd888fe304c21d0df1e8743a25c70f9c4fc8dc5ad5c42d079e32312fb8fb1e" network for pod "apiserver-c72wp": NetworkPlugin cni failed to set up pod "apiserver-c72wp_kube-service-catalog" CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "6dcd888fe304c21d0df1e8743a25c70f9c4fc8dc5ad5c42d079e32312fb8fb1e"
  • SELinux denied transition logs observed on problematic host:
$ cat /var/log/audit/audit.log | grep AVC
type=AVC msg=audit(1596006887.677:146971): avc:  denied  { transition } for  pid=129441 comm="runc:[2:INIT]" path="/usr/bin/pod" dev="dm-17" ino=21055520 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:container_t:s0:c10,c21 tclass=process permissive=0
type=AVC msg=audit(1596006888.999:146982): avc:  denied  { transition } for  pid=129627 comm="runc:[2:INIT]" path="/usr/bin/pod" dev="dm-19" ino=21055520 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:container_t:s0:c616,c635 tclass=process permissive=0
  • Rebooting the node, restarting sdn and ovs pods, rebooting the node is not able to resolve the issue.

  • How to fix this?

Environment

  • Red Hat OpenShift Container Platform
    • 3.11

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content