During upgrade Prometheus operator fails with Forbidden updates to statefulset

Issue

• During an upgrade from 4.8.29 to 4.9.19, the Prometheus operator failed to rollout updates to prometheus and alert manager. As a result, prometheus became unavailable.
• Prometheus and associated cluster metrics and performance dashboards became unavailable
• Prometheus operator log reported the failures below and the statefulsets were stuck at Prometheus(0/2 available) Alert Manager(0/3 available).

level=info ts=2022-02-08T17:19:39.289456243Z caller=operator.go:804 component=alertmanageroperator key=openshift-monitoring/main msg="recreating AlertManager StatefulSet because the update operation wasn't possible" reason="Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy' and 'minReadySeconds' are forbidden"
level=info ts=2022-02-08T17:19:39.295713026Z caller=operator.go:742 component=alertmanageroperator key=openshift-monitoring/main msg="sync alertmanager"
level=info ts=2022-02-08T17:19:39.330897841Z caller=operator.go:804 component=alertmanageroperator key=openshift-monitoring/main msg="recreating AlertManager StatefulSet because the update operation wasn't possible" reason="Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy' and 'minReadySeconds' are forbidden"
level=info ts=2022-02-08T17:19:39.42114441Z caller=operator.go:1306 component=prometheusoperator key=openshift-monitoring/k8s statefulset=prometheus-k8s shard=0 msg="recreating StatefulSet because the update operation wasn't possible" reason="Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy' and 'minReadySeconds' are forbidden"
level=info ts=2022-02-08T17:19:39.426608664Z caller=operator.go:1221 component=prometheusoperator key=openshift-monitoring/k8s msg="sync prometheus"
level=info ts=2022-02-08T17:19:39.590102973Z caller=operator.go:1306 component=prometheusoperator key=openshift-monitoring/k8s statefulset=prometheus-k8s shard=0 msg="recreating StatefulSet because the update operation wasn't possible" reason="Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy' and 'minReadySeconds' are forbidden"
level=info ts=2022-02-08T17:24:44.340355154Z caller=operator.go:1221 component=prometheusoperator key=openshift-monitoring/k8s msg="sync prometheus"