Security Context Constraint Issues Causing Rook-Ceph Pod Failures After ODF Upgrade

Issue

After OpenShift Data Foundation Operator upgrade from 4.15.x to 4.16.x or greater, various pods are failing to run due to a security context constraint issue.

Example:
$ oc get events -n openshift-storage shows errors regarding security context constraints when attempting to create ODF pods

53m         Warning   FailedCreate                   replicaset/rook-ceph-mon-c-d48797d7c                                           Error creating: pods "rook-ceph-mon-c-d48797d7c-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, spec.volumes[2]: Invalid value: "hostPath": hostPath volumes are not allowed to be used, spec.volumes[3]: Invalid value: "hostPath": hostPath volumes are not allowed to be used, spec.volumes[4]: Invalid value: "hostPath": hostPath volumes are not allowed to be used, provider restricted-v2: .initContainers[0].privileged: Invalid value: true: Privileged containers are not allowed, provider restricted-v2: .initContainers[1].privileged: Invalid value: true: Privileged containers are not allowed, provider restricted-v2: .containers[0].privileged: Invalid value: true: Privileged containers are not allowed, provider restricted-v2: .containers[1].privileged: Invalid value: true: Privileged containers are not allowed, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "noobaa-db": Forbidden: not usable by user or serviceaccount, provider "noobaa": Forbidden: not usable by user or serviceaccount, provider "noobaa-endpoint": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "rook-ceph": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "rook-ceph-csi": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]
7m          Warning   FailedCreate         replicaset/rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-688d67c884              Error creating: pods "rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-688d67c884-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, spec.volumes[2]: Invalid value: "hostPath": hostPath volumes are not allowed to be used, spec.volumes[3]: Invalid value: "hostPath": hostPath volumes are not allowed to be used, spec.volumes[4]: Invalid value: "hostPath": hostPath volumes are not allowed to be used, provider restricted-v2: .initContainers[0].privileged: Invalid value: true: Privileged containers are not allowed, provider restricted-v2: .containers[0].privileged: Invalid value: true: Privileged containers are not allowed, provider restricted-v2: .containers[1].privileged: Invalid value: true: Privileged containers are not allowed, provider "rootless-builds": Forbidden: not usable by user or serviceaccount, provider "restricted": Forbidden: not usable by user or serviceaccount, provider "nonroot-v2": Forbidden: not usable by user or serviceaccount, provider "nonroot": Forbidden: not usable by user or serviceaccount, provider "noobaa-db": Forbidden: not usable by user or serviceaccount, provider "noobaa": Forbidden: not usable by user or serviceaccount, provider "noobaa-endpoint": Forbidden: not usable by user or serviceaccount, provider "hostmount-anyuid": Forbidden: not usable by user or serviceaccount, provider "logging-scc": Forbidden: not usable by user or serviceaccount, provider "machine-api-termination-handler": Forbidden: not usable by user or serviceaccount, provider "hostnetwork-v2": Forbidden: not usable by user or serviceaccount, provider "hostnetwork": Forbidden: not usable by user or serviceaccount, provider "hostaccess": Forbidden: not usable by user or serviceaccount, provider "rook-ceph": Forbidden: not usable by user or serviceaccount, provider "node-exporter": Forbidden: not usable by user or serviceaccount, provider "rook-ceph-csi": Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]

$ oc get pods -l rook_cluster=openshift-storage shows pods in varying states and some completely missing

NAME                                                              READY   STATUS             RESTARTS   AGE
rook-ceph-crashcollector-[REDACTED]                               0/1     Pending            0          9h
rook-ceph-crashcollector-[REDACTED]                               0/1     Pending            0          9h
rook-ceph-crashcollector-[REDACTED]                               1/1     Running            0          9h
rook-ceph-exporter-[REDACTED]                                     0/1     Pending            0          9h
rook-ceph-exporter-[REDACTED]                                     1/1     Running            80         9h
rook-ceph-mgr-a-6494b4f789-7fhgs                                  2/3     CrashLoopBackOff   155        9h
rook-ceph-mgr-b-589f65cc64-hjh54                                  3/3     Running            0          9h
rook-ceph-osd-0-8649f8fc98-nrz8q                                  1/2     Running            81         9h
rook-ceph-osd-1-6979b954d8-95sfx                                  1/2     Running            85         9h
rook-ceph-osd-2-f4d998b5d-55gt5                                   2/2     Running            0          15h
rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-f9f6b59q7s5s   1/2     CrashLoopBackOff   121        9h