ODF: rook-ceph-osd-prepare Pods Continue to Deploy in a "Pending" state after the Desired Ceph OSD State has been Attained

Issue

In some instances, ODF users may notice rook-ceph-osd-prepare pods appearing in their $ oc get pods -n openshift-storage output. These prepare pods are associated with OSD prepare jobs $ oc get jobs -n openshift-storage.

Usually, these jobs will be in a Completed state and can be cleared out when the user deletes the jobs. However, there may be instances when these pods/jobs are in a Pending status indefinitely, or more deployed than expected. If in Pending, these pods/jobs will also have a Pending PVC associated with them as well.

The function of the prepare pod/job is to deploy a Persistent Volume Claim (PVC) to bind with a Persistent Volume (PV). However, in the event there are no PVs to bind with (Local Storage), that job/PVC will remain in a Pending state.

A common way this occurs is that a user utilizes the "Add Capacity" function too many times, which increases the count in the storagecluster.yaml and in most cases won't match the amount of Available PVs.

$ oc get pods,jobs -n openshift-storage|grep prepare

pod/rook-ceph-osd-prepare-36569fd6d3212d7a37d3e5b465185e9c-4jq4d      0/1     Pending   0               3m51s
pod/rook-ceph-osd-prepare-840b1b664511e2febad3e19225298a3e-d4nc8      0/1     Pending   0               3m51s

job.batch/rook-ceph-osd-prepare-36569fd6d3212d7a37d3e5b465185e9c   0/1           3m51s      3m51s
job.batch/rook-ceph-osd-prepare-840b1b664511e2febad3e19225298a3e   0/1           3m51s      3m51s

$ oc get pvc -n openshift-storage | grep -i pending
NAME                                              STATUS   VOLUME         CAPACITY   ACCESS MODES   STORAGECLASS          AGE
ocs-deviceset-ocs-storagecluster-1-data-4l2b4w    Pending                                           ocs-storagecluster
ocs-deviceset-ocs-storagecluster-2-data-4f2967    Pending                                           ocs-storagecluster