Storage CO is degraded after cluster upgrade to 4.12 due to invalid vSphere credentials.

Issue

• After upgrade to 4.12 Storage CO is degraded because of invalid Sphere credentials (cluster operator status for respective versions) .

NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
storage                                    4.12.19   True        True          False      22m
storage                                    4.11.41   True        False         False      19d

4.11: Problem detector running but broken, default vsphere storage class exists. vmware-vsphere-csi-driver-operator is also having issue but is running.
4.12: same as 4.11 plus the vmware csi driver having the below error:

2023-07-10T14:15:17.362780911+12:00 W0710 02:15:17.362776       1 vspherecontroller.go:457] Marking cluster upgrade status unknown because Failed to connect to vSphere: Post "https://unreachable/sdk": dial tcp: lookup unreachable on 172.xx.x.xx:xx: server misbehaving

• Storage cluster operator is "In progress" indefinitely

    Last Transition Time:  2023-06-07T01:56:55Z
    Message:               VSphereCSIDriverOperatorCRDegraded: All is well
    Reason:                AsExpected
    Status:                False
    Type:                  Degraded
    Last Transition Time:  2023-06-22T01:25:54Z
    Message:               VSphereCSIDriverOperatorCRProgressing: Waiting for VSphere operator to report status
    Reason:                VSphereCSIDriverOperatorCR_WaitForOperator
    Status:                True
    Type:                  Progressing
    Last Transition Time:  2023-06-22T02:23:27Z
    Message:               VSphereCSIDriverOperatorCRAvailable: All is well
VSphereProblemDetectorControllerAvailable: failed to connect to unreachable: Post "https://unreachable/sdk": dial tcp: lookup unreachable on 172.xx.xx:xx: server misbehaving

• Cluster firing an alert for the vsphere config

VSphereOpenshiftConnectionFailure          firing