Podsandbox creation failing after rebooting/upgrading hosts due to SElinux denials
Issue
- After rebooting host, pods failed to start with series of below errors in node service logs:
$ journalctl atomic-openshift-node.service
Jul 29 08:40:36 node1.test.example.com atomic-openshift-node[101620]: I0729 08:40:36.954916 101620 kuberuntime_manager.go:403] No ready sandbox for pod "apiserver-c72wp_kube-service-catalog(f6b0b99c-d161-11ea-896b-00505685765c)" can be found. Need to start a new one.
Jul 29 08:40:36 node1.test.example.com atomic-openshift-node[101620]: W0729 08:40:36.964780 101620 cni.go:243] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "6dcd888fe304c21d0df1e8743a25c70f9c4fc8dc5ad5c42d079e32312fb8fb1e"
- Pods go in
CrashLoopBackOff
state with following events:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreatePodSandBox 25m kubelet, xxxx Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "6dcd888fe304c21d0df1e8743a25c70f9c4fc8dc5ad5c42d079e32312fb8fb1e" network for pod "apiserver-c72wp": NetworkPlugin cni failed to set up pod "apiserver-c72wp_kube-service-catalog" CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "6dcd888fe304c21d0df1e8743a25c70f9c4fc8dc5ad5c42d079e32312fb8fb1e"
- SELinux denied transition logs observed on problematic host:
$ cat /var/log/audit/audit.log | grep AVC
type=AVC msg=audit(1596006887.677:146971): avc: denied { transition } for pid=129441 comm="runc:[2:INIT]" path="/usr/bin/pod" dev="dm-17" ino=21055520 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:container_t:s0:c10,c21 tclass=process permissive=0
type=AVC msg=audit(1596006888.999:146982): avc: denied { transition } for pid=129627 comm="runc:[2:INIT]" path="/usr/bin/pod" dev="dm-19" ino=21055520 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:system_r:container_t:s0:c616,c635 tclass=process permissive=0
-
Rebooting the node, restarting sdn and ovs pods, rebooting the node is not able to resolve the issue.
-
How to fix this?
Environment
- Red Hat OpenShift Container Platform
- 3.11
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.