Creating /etc/fstab manually with SELinux context unconfined_u:object_r:admin_home_t makes ostree fail and thus makes node uprades fail in Red Hat OpenShift Container Platform 4.x

Issue

• Creating /etc/fstab manually with SELinux context unconfined_u:object_r:admin_home_t makes ostree fail and thus makes node uprades fail in Red Hat OpenShift Container Platform 4.x

• Creating /etc/fstab manually with SELinux context unconfined_u:object_r:admin_home_t (or other labels that trigger an SELinux deny) makes ostree fail during upgrade processes.

• Created /etc/fstab with ansible to configure local ( hostpath ) volumes on all the workers. Upon upgrade of Red Hat OpenShift Container Platform, the nodes do not come up

• Nodes show up as NotReady and SSH to the nodes with the core user fails

• Going through the cluster oc adm must-gather logs, it is obvious that the cluster has never successfully upgraded. From an MCO perspective, the operator is not degraded and the master pool has been upgraded and the worker pool is not degraded but essentially stuck while upgrading:

 - lastTransitionTime: "2020-07-16T02:24:52Z"
    message: All nodes are updating to rendered-worker-160ab82951b16365d61ffdc01e5683f0
    reason: ""
    status: "True"
    type: Updating

With 1 unavailable node:
  unavailableMachineCount: 1
  updatedMachineCount: 11

• It is possible to boot into the previous ostree backup image via grub and then connect to the node via SSH.

• If SSH still does not work when booting into the ostree backup image, then it is at least possible to edit the Grub entry, remove all console=... entries and add rd.break to the kernel command line. Then, from the emergency console, one can analyze logs and files.

• From the node's log, one can see that ostree cannot access /etc/fstab due to an SELinux issue:

systemd-fstab-generator[1746]: Failed to open /etc/fstab: Permission denied
kernel: audit: type=1400 audit(1595448012.213:4): avc:  denied  { read } for  pid=1746 comm="systemd-fstab-g" name="fstab" dev="dm-0" ino=1015044194 scontext=system_u:system_r:init_t:s0 t
(...)
kernel: audit: type=1400 audit(1595448012.313:5): avc:  denied  { read } for  pid=1740 comm="ostree-system-g" name="fstab" dev="dm-0" ino=1015044194 scontext=system_u:system_r:init_t:s0 t
context=unconfined_u:object_r:admin_home_t:s0 tclass=file permissive=0
systemd[1734]: /usr/lib/systemd/system-generators/ostree-system-generator failed with exit status 1.

ostree shows this after logging into 4.3:

# ostree admin status
  rhcos d784acd0be69daccad169eb853dc70d77c5ed377a644b34243814e7ec7bc3c23.0 (pending)
    Version: 44.81.202006080130-0
    origin refspec: d784acd0be69daccad169eb853dc70d77c5ed377a644b34243814e7ec7bc3c23
* rhcos 40039646cc9b435addba7f2d768e9b67472acf0edc102686c7c7e1e6eb957d9d.0
    Version: 43.81.202005121559.0
    origin refspec: 40039646cc9b435addba7f2d768e9b67472acf0edc102686c7c7e1e6eb957d9d

And it shows strange files in the /ostree director for that deploy:

# pwd
/ostree/deploy/rhcos/deploy/d784acd0be69daccad169eb853dc70d77c5ed377a644b34243814e7ec7bc3c23.0

# ls var/
total 12K
-rw-r--r--.  1 root root  190 Jan  1  1970 .updated
drwxr-xr-x. 12 root root  253 Jun 18 19:30 ..
drwxr-xr-x.  2 root root    6 Jun 18 19:33 mnt
lrwxrwxrwx.  1 root root   11 Jul 16 02:28 lock -> ../run/lock
drwxr-xr-x.  3 root root   18 Jul 16 02:28 empty
drwxr-xr-x.  2 root root    6 Jul 16 02:28 adm
drwxr-xr-x.  2 root root    6 Jul 16 02:28 games
drwxr-xr-x.  2 root root    6 Jul 16 02:28 ftp
drwxr-xr-x.  2 root root    6 Jul 16 02:28 gopher
drwxr-xr-x.  2 root root    6 Jul 16 02:28 nis
lrwxrwxrwx.  1 root root   10 Jul 16 02:28 mail -> spool/mail
drwxr-xr-x.  2 root root    6 Jul 16 02:28 local
drwxr-xr-x.  2 root root    6 Jul 16 02:28 preserve
drwxr-xr-x.  2 root root    6 Jul 16 02:28 yp
drwxr-xr-x.  4 root root   29 Jul 16 02:28 spool
drwxr-xr-x.  3 root root   18 Jul 16 02:28 kerberos
drwxr-xr-x.  3 root root   18 Jul 16 02:28 db
drwxr-xr-x.  2 root root    6 Jul 16 02:28 home
drwxr-xr-x.  2 root root    6 Jul 16 02:28 srv
drwx------.  2 root root    6 Jul 16 02:28 roothome
drwxr-xr-x. 11 root root  114 Jul 16 02:28 usrlocal
drwxr-xr-x.  5 root root   55 Jul 16 02:28 cache
lrwxrwxrwx.  1 root root    6 Jul 16 02:28 run -> ../run
drwxr-xr-x. 24 root root 4.0K Jul 16 02:28 .
drwxr-xr-x. 30 root root 4.0K Jul 16 02:29 lib
drwxr-xr-x.  2 root root   17 Jul 22 14:45 opt
drwxrwxrwt.  2 root root    6 Jul 23 21:12 tmp
drwxr-xr-x. 13 root root  205 Jul 31 16:38 log