ODF: Changes in crush rules due to device class changes results in PGs being in an unknown, misplaced, and/or remapped state

Issue

• Pools are utilizing a crush rule that references a device class that is not currently utilized in the cluster, this results in PGs being in an unknown, misplaced, and/or remapped state
• More than 100 percent of objects are misplaced in Ceph after upgrading the OpenShift Data Foundations operator to 4.16.x
• Rook created crush rules to utilize a device class that is not used in the cluster; these crush rules are now applied to pools, causing issues
• The defaultCephDeviceClass in the StorageCluster CR is incorrect
• Transitioned from unsupported HDDs to SSDs in ODF, resulting in data unavailability

Example:
Ceph status shows a large amount of objects misplaced

$ oc exec -it $(oc get pod -n openshift-storage -l app=rook-ceph-operator -o name) -n openshift-storage -- ceph status -c /var/lib/rook/openshift-storage/openshift-storage.config

  cluster:
    id:     [REDACTED]
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum a,b,d (age 7d)
    mgr: b(active, since 7d), standbys: a
    mds: 1/1 daemons up, 1 hot standby
    osd: 6 osds: 6 up (since 7d), 6 in (since 4M); 169 remapped pgs
    rgw: 1 daemon active (1 hosts, 1 zones)

  data:
    volumes: 0/1 healthy, 1 recovering
    pools:   12 pools, 281 pgs
    objects: 184.82k objects, 265 GiB
    usage:   1.2 TiB used, 17 TiB / 18 TiB avail
    pgs:     33.808% pgs unknown
             886324/554472 objects misplaced (159.850%)
             138 active+clean+remapped
             95  unknown
             48  active+undersized+remapped

All OSDs are utilizing the device class ssd

$ oc exec -it $(oc get pod -n openshift-storage -l app=rook-ceph-operator -o name) -n openshift-storage -- ceph osd tree -c /var/lib/rook/openshift-storage/openshift-storage.config

ID  CLASS  WEIGHT   TYPE NAME                                         STATUS  REWEIGHT  PRI-AFF
-1         5.85956  root default                                                               
-5         1.95319      host [REDACTED]                           
 1    ssd  0.97659          osd.1                                         up   1.00000  1.00000
 5    ssd  0.97659          osd.5                                         up   1.00000  1.00000
-7         1.95319      host [REDACTED]                           
 2    ssd  0.97659          osd.2                                         up   1.00000  1.00000
 4    ssd  0.97659          osd.4                                         up   1.00000  1.00000
-3         1.95319      host [REDACTED]                           
 0    ssd  0.97659          osd.0                                         up   1.00000  1.00000
 3    ssd  0.97659          osd.3                                         up   1.00000  1.00000

The pools are utilizing a crush rule that references hdd while there are only ssds in the cluster

$ oc exec -it $(oc get pod -n openshift-storage -l app=rook-ceph-operator -o name) -n openshift-storage -- ceph osd pool ls detail -c /var/lib/rook/openshift-storage/openshift-storage.config

pool 1 'ocs-storagecluster-cephblockpool' replicated size 3 min_size 2 crush_rule 25 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode on last_change 9437 lfor 0/0/30 flags hashpspool,selfmanaged_snaps stripe_width 0 target_size_ratio 0.49 application rbd

The crush rules being utilized references hdd while there are only ssds in the cluster

$ oc exec -it $(oc get pod -n openshift-storage -l app=rook-ceph-operator -o name) -n openshift-storage -- ceph osd crush rule dump -c /var/lib/rook/openshift-storage/openshift-storage.config

    {
        "rule_id": 25,
        "rule_name": "ocs-storagecluster-cephblockpool_host_hdd",
        "type": 1,
        "steps": [
            {
                "op": "take",
                "item": -2,
                "item_name": "default~hdd"
            },
            {
                "op": "chooseleaf_firstn",
                "num": 0,
                "type": "host"
            },
            {
                "op": "emit"
            }
        ]
    },