ODF operator pods failing with `CreateContainerConfigError` state

Issue

Various ODF operator pods are failing with CreateContainerConfigError state:
- odf-operator-controller-manager
- ocs-metrics-exporter
- csi-addons-controller-manager

This would cause the odd-console pod to fail to run with ContainerCreating state. This could happen within an upgrade context or it could happen randomly.

$ oc get pods -l app.kubernetes.io/name=ocs-metrics-exporter
NAME                                    READY   STATUS                       RESTARTS   AGE
ocs-metrics-exporter-6c7d46c667-v5q8l   1/1     Running                      0          47h
ocs-metrics-exporter-86669fdbdb-5n65s   0/1     CreateContainerConfigError   0          47h
$ 

$ oc get pods -l app.kubernetes.io/name=odf-operator
NAME                                               READY   STATUS                       RESTARTS   AGE
odf-operator-controller-manager-6795966b6f-lsqqh   1/2     CreateContainerConfigError   0          47h
odf-operator-controller-manager-864ddcf787-pkv25   2/2     Running                      0          47h
$ 


$ oc get pod -l app=odf-console
NAME                           READY   STATUS              RESTARTS   AGE
odf-console-75b67c6bc7-8rznj   0/1     ContainerCreating   0          47h
odf-console-7c8fcbdc86-wp24q   0/1     ContainerCreating   0          47h
$ 


$ oc get pod -l app.kubernetes.io/name=csi-addons
NAME                                             READY   STATUS                       RESTARTS     AGE
csi-addons-controller-manager-77d88ffdbc-wvn8j   0/1     CreateContainerConfigError   0            47h
$ 

The symptoms could also be coupled with a degraded service-ca clusterOperator:

$ oc get pods -n openshift-service-ca
NAME                          READY   STATUS                       RESTARTS   AGE
service-ca-54889f9bc8-xl79f   0/1     CreateContainerConfigError   0          35h


`service-ca` ClusterOperator shows `progressing`state:

$ oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE

service-ca                                 4.10.34   True        True          False      329d    Progressing: ...

The logs and events streamed using oc get events -w -n openshift-storage suggest that ODF operator pods are running with privileged or anyuid rights in the SCC instead of the expected restricted scc:

142m        Warning   Failed                            pod/odf-operator-controller-manager-6f7c4f6c49-6fkgc                         Error: container has runAsNonRoot and image will run as root (pod: "odf-operator-controller-manager-6f7c4f6c49-6fkgc_openshift-storage(1e58dd99-457b-4fbc-a400-aa2097b4890a)", container: manager)

When the must-gather archives are checked on the support shell, the default SCC checks fail:

[FAIL] ccx_rules_ocp.ocs.ocs_pods_without_default_scc.report
------------------------------------------------------------
Links:
    bz:
        https://bugzilla.redhat.com/show_bug.cgi?id=2078040


The following pods do not have the default SCC's defined by OCS Operator:

  NAME: csi-addons-controller-manager-77d88ffdbc-rlg8z
  SCC: privileged

  NAME: csi-cephfsplugin-72tsp
  SCC: privileged

  NAME: csi-cephfsplugin-7q2tc
  SCC: privileged

  NAME: csi-cephfsplugin-8db5c
  SCC: privileged

  NAME: csi-cephfsplugin-dcmfn
  SCC: privileged

  NAME: csi-cephfsplugin-j59kp
  SCC: privileged

  NAME: csi-cephfsplugin-provisioner-9b6db5df7-8h8gw
  SCC: privileged

  NAME: csi-cephfsplugin-provisioner-9b6db5df7-mx5cp
  SCC: privileged

  NAME: csi-cephfsplugin-rcbph
  SCC: privileged

  NAME: csi-cephfsplugin-s5gx9
  SCC: privileged

  NAME: csi-cephfsplugin-vzvld
  SCC: privileged

  NAME: csi-cephfsplugin-z67vd
  SCC: privileged

  NAME: csi-cephfsplugin-zbmlq
  SCC: privileged

  NAME: csi-rbdplugin-4k4x9
  SCC: privileged

  NAME: csi-rbdplugin-5qp2c
  SCC: privileged

  NAME: csi-rbdplugin-bcwn9
  SCC: privileged

  NAME: csi-rbdplugin-bkp9l
  SCC: privileged

  NAME: csi-rbdplugin-dv7fd
  SCC: privileged

  NAME: csi-rbdplugin-fcg6n
  SCC: privileged

  NAME: csi-rbdplugin-k9v8f
  SCC: privileged

  NAME: csi-rbdplugin-provisioner-767dd58965-65pqp
  SCC: privileged

  NAME: csi-rbdplugin-provisioner-767dd58965-g5bgk
  SCC: privileged

  NAME: csi-rbdplugin-rpvlk
  SCC: privileged

  NAME: csi-rbdplugin-v54kh
  SCC: privileged

  NAME: csi-rbdplugin-v5mjk
  SCC: privileged

  NAME: lab-ocp-i1ocplabtelcostc-debug
  SCC: privileged

  NAME: lab-ocp-i2ocplabtelcostc-debug
  SCC: privileged

  NAME: lab-ocp-i3ocplabtelcostc-debug
  SCC: privileged

  NAME: must-gather-6nvsf-helper
  SCC: privileged

  NAME: noobaa-core-0
  SCC: privileged

  NAME: noobaa-db-pg-0
  SCC: privileged

  NAME: noobaa-endpoint-65f7fb85f5-7gccr
  SCC: privileged

  NAME: noobaa-operator-b4c54cbd4-czrfl
  SCC: privileged

  NAME: ocs-operator-6b65fd78b7-d7kcd
  SCC: privileged

  NAME: odf-console-7c8fcbdc86-nqc9v
  SCC: privileged

  NAME: odf-operator-controller-manager-6795966b6f-28fs7
  SCC: privileged

  NAME: rook-ceph-crashcollector-lab-ocp-i1.ocp.labtelco.stc-6d9bddkl9x
  SCC: privileged

  NAME: rook-ceph-crashcollector-lab-ocp-i2.ocp.labtelco.stc-8669bbrkzb
  SCC: privileged

  NAME: rook-ceph-crashcollector-lab-ocp-i3.ocp.labtelco.stc-d7fcfq5k7h
  SCC: privileged

  NAME: rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-5f596b76jtmmf
  SCC: privileged

  NAME: rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-b87d44ddxcjq7
  SCC: privileged

  NAME: rook-ceph-mgr-a-7548d8cf9-jf9t2
  SCC: privileged

  NAME: rook-ceph-mon-a-7c946bb4dc-cn88p
  SCC: privileged

  NAME: rook-ceph-mon-e-bfc556d88-nd7pd
  SCC: privileged

  NAME: rook-ceph-mon-f-7f9568b897-kkm2t
  SCC: privileged

  NAME: rook-ceph-operator-7fc5d7d4f6-kch4j
  SCC: privileged

  NAME: rook-ceph-osd-0-9f7c9656b-rnlgl
  SCC: privileged

  NAME: rook-ceph-osd-10-847dcfb4c6-xjtxr
  SCC: privileged

  NAME: rook-ceph-osd-11-59b59cf5bd-htkwb
  SCC: privileged

  NAME: rook-ceph-osd-12-78c57dcd47-w6ftw
  SCC: privileged

  NAME: rook-ceph-osd-13-76dbc6dbb7-9x67b
  SCC: privileged

  NAME: rook-ceph-osd-14-65795d86b8-6kv8b
  SCC: privileged

  NAME: rook-ceph-osd-15-6ddfc665bd-45w8c
  SCC: privileged

  NAME: rook-ceph-osd-3-b94c749cc-fn2bb
  SCC: privileged

  NAME: rook-ceph-osd-4-659959988b-vf4jk
  SCC: privileged

  NAME: rook-ceph-osd-6-7b97f74546-jgkk2
  SCC: privileged

  NAME: rook-ceph-osd-7-647cd5b659-tnx95
  SCC: privileged

  NAME: rook-ceph-osd-8-847bf99d9c-9mpq9
  SCC: privileged

  NAME: rook-ceph-osd-prepare-42279230c70e646bbe8028f99dd5154d-m9ngk
  SCC: privileged

  NAME: rook-ceph-osd-prepare-cdadc593708b98993827f8d1064120a2-9scv9
  SCC: privileged

  NAME: rook-ceph-osd-prepare-d46bebde547ff8d8ba2627f8cb14fcef-kfzvk
  SCC: privileged

  NAME: rook-ceph-osd-prepare-e273bfcd6627166f3d1098a9bb0e0368-6lcmk
  SCC: privileged

  NAME: rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-86d5795gsc9t
  SCC: privileged

  NAME: rook-ceph-tools-6bc8c88fff-ngkpl
  SCC: privileged


The SCC (Security Context Constraints) for OCS pods should not be changed to default.
If the SCC's are changed, this can result in existing Ceph volumes giving access denied when trying to read/write.