OpenShift Data Foundation managed service StorageSystems instance ocs-storagecluster-storagesystem Condition:Progressing with event "InstallPlan not found for CSV mcg-operator.v4.10.9"
Environment
- Red Hat OpenShift on AWS (ROSA)
- 4.10
- OpenShift Data Foundation Managed Service
- 4.10
Issue
When installing Red Hat OpenShift Data Foundation Managed Service with Red Hat Hybrid Cloud Console (OCM web console) or via rosa cli on ROSA 4.10.x, the following event messages are shown:
- Storage System ocs-storagecluster-storagesystem in openshift-storage project Condition: Progressing
- Checking events and found "InstallPlan not found for CSV mcg-operator.v4.10.9"
Resolution
-
There was no action taken to resolve the request. When the cluster gets upgraded from 4.10 to 4.11, the error message will go away.
-
Workaround : to reinstall the ODF operator by following all steps from OCS/ODF Upgrade Not Progressing
Root Cause
-
The cluster is on ODF 4.10. odf-operator is creating subscription for mcg-operator(with provided version 4.10.9) and ocs-operator. OLM is complaining that mcg-operator(4.10.10) is available and logged out the error message. ocs-operator is up and running fine in the status. When the cluster will get upgraded to 4.11(which is not using the manual dependency version) the error message will not be there.
-
This is a minor issue , related to this old bug
Bug 2117521 - [Tracker for OCP BZ #2116291] Upgrade of encrypted BM cluster failed with "StorageSystem", "name": "ocs-storagecluster-storagesystem", "namespace": "openshift-storage", "error": "InstallPlan not found for CSV ocs-operator.v4.11.0
--> closed as duplicate of:...that says:
"OLM failed to update the ODF subscription to reflect that CSV for odf-operator 4.11 was installed. The ODF subscription still lists CSV 4.10.5 as its installed/currentCSV which leads the disconnection between subscription and CSV 4.11. As a result, OLM resolver complains that CSV 4.11 is not referenced in ODF subscription.""Sadly, the most common fix for this is to reinstall ODF operator."
--> Closed as Won't Fix, "Failure to update Subscription does happen occasionally but it shouldn't consistently happen."
Diagnostic Steps
- Check pods which running under the
openshift-storagenamespace
$ oc get pod -n openshift-storage
NAME READY STATUS RESTARTS AGE
4a1e9834bbe432a481b66d6217107741aa9a0df9214bdbe83057e110f26hcqp 0/1 Completed 0 17h
58170ab247ef84e66e772daea74a63fa253f7c254a2054c058200ba9aa5zgz8 0/1 Completed 0 17h
6f70c8b65e5a693e11613dd966e9a37bb81e3324323c2dfe14badc99e7z4gt6 0/1 Completed 0 17h
addon-ocs-consumer-catalog-xmd88 1/1 Running 0 17h
alertmanager-managed-ocs-alertmanager-0 2/2 Running 0 17h
csi-addons-controller-manager-759b488df-mf9rv 2/2 Running 0 17h
csi-cephfsplugin-mw4lg 2/2 Running 0 17h
csi-cephfsplugin-provisioner-5d6b768994-mqwgq 5/5 Running 0 17h
csi-cephfsplugin-provisioner-5d6b768994-sj9pj 5/5 Running 0 17h
csi-cephfsplugin-rb4fb 2/2 Running 0 17h
csi-cephfsplugin-sfdtl 2/2 Running 0 17h
csi-rbdplugin-74qjg 3/3 Running 0 17h
csi-rbdplugin-84dxn 3/3 Running 0 17h
csi-rbdplugin-mhtjq 3/3 Running 0 17h
csi-rbdplugin-provisioner-65477c4f5-9mpsh 6/6 Running 0 17h
csi-rbdplugin-provisioner-65477c4f5-xh7b9 6/6 Running 0 17h
ed18aed928969cb48886de78882f4750eb53bab63bc964c5896065287cptdzw 0/1 Completed 0 17h
f2178cee2a90b7d1e15890fa39c5398d06cad74733a4f20add7449bf48t2p6g 0/1 Completed 0 17h
fe2b32e19e80bf7e7f02ddf47f9646a513d4d3f587dc4bd7056c1a3186frns2 0/1 Completed 0 17h
ocs-metrics-exporter-5dd96c885b-8pw89 1/1 Running 0 17h
ocs-operator-6888799d6b-m8rf5 1/1 Running 0 17h
ocs-osd-aws-data-gather-78dd69c8dd-s47mn 1/1 Running 0 17h
ocs-osd-controller-manager-99658d756-4xz8c 3/3 Running 0 17h
odf-console-57b8476cd4-thxmk 1/1 Running 0 17h
odf-operator-controller-manager-6f44676f4f-h7fsr 2/2 Running 0 17h
prometheus-managed-ocs-prometheus-0 3/3 Running 0 17h
prometheus-operator-8547cc9f89-c67rx 1/1 Running 0 17h
redhat-operators-kpxms 1/1 Running 0 17h
rook-ceph-operator-548b87d44b-4pds7 1/1 Running 0 17h
rook-ceph-tools-7c8c77bd96-vkw78 1/1 Running 0 17h
- Check CSVs under the
openshift-storagenamespace
$ oc get csv -n openshift-storage
NAME DISPLAY VERSION REPLACES PHASE
mcg-operator.v4.10.10 NooBaa Operator 4.10.10 mcg-operator.v4.10.9 Succeeded
observability-operator.v0.0.20 Observability Operator 0.0.20 observability-operator.v0.0.19 Succeeded
ocs-operator.v4.10.9 OpenShift Container Storage 4.10.9 ocs-operator.v4.10.8 Succeeded
ocs-osd-deployer.v2.0.11 OCS OSD Deployer 2.0.11 ocs-osd-deployer.v2.0.10 Succeeded
odf-csi-addons-operator.v4.10.9 CSI Addons 4.10.9 odf-csi-addons-operator.v4.10.8 Succeeded
odf-operator.v4.10.9 OpenShift Data Foundation 4.10.9 odf-operator.v4.10.8 Succeeded
openshift-gitops-operator.v1.5.10 Red Hat OpenShift GitOps 1.5.10 openshift-gitops-operator.v1.5.9 Succeeded
openshift-pipelines-operator-rh.v1.7.3 Red Hat OpenShift Pipelines 1.7.3 openshift-pipelines-operator-rh.v1.7.2 Succeeded
ose-prometheus-operator.4.10.0 Prometheus Operator 4.10.0 ose-prometheus-operator.4.8.0 Succeeded
route-monitor-operator.v0.1.461-dbddf1f Route Monitor Operator 0.1.461-dbddf1f route-monitor-operator.v0.1.453-c29f30e Succeeded
- Check storage cluster is Ready phase
$ oc get storagecluster -n openshift-storage
NAME AGE PHASE EXTERNAL CREATED AT VERSION
storagecluster.ocs.openshift.io/ocs-storagecluster 17h Ready true 2023-03-08T00:00:48Z
- Check StorageSystems
$ oc get -n storagesystem openshift-storage
NAME STORAGE-SYSTEM-KIND STORAGE-SYSTEM-NAME
storagesystem.odf.openshift.io/ocs-storagecluster-storagesystem storagecluster.ocs.openshift.io/v1 ocs-storagecluster
- Check ocs-storagecluster-storagesystem status
$ oc describe storagesystem.odf.openshift.io/ocs-storagecluster-storagesystem -n storagesystem openshift-storage
...omitted...
Status:
Conditions:
Last Heartbeat Time: 2023-03-08T17:36:02Z
Last Transition Time: 2023-03-08T00:00:48Z
Message: Reconcile is in progress
Reason: Reconciling
Status: False
Type: Available
Last Heartbeat Time: 2023-03-08T17:36:02Z
Last Transition Time: 2023-03-08T00:00:48Z
Message: Reconcile is in progress
Reason: Reconciling
Status: True
Type: Progressing
Last Heartbeat Time: 2023-03-08T17:36:02Z
Last Transition Time: 2023-03-08T00:00:48Z
Message: StorageSystem CR is valid
Reason: Valid
Status: False
Type: StorageSystemInvalid
Last Heartbeat Time: 2023-03-08T17:36:02Z
Last Transition Time: 2023-03-08T00:00:48Z
Message: InstallPlan not found for CSV mcg-operator.v4.10.9
Reason: NotReady
Status: False
Type: VendorCsvReady
Last Heartbeat Time: 2023-03-08T00:00:48Z
Last Transition Time: 2023-03-08T00:00:48Z
Message: Initializing StorageSystem
Reason: Init
Status: Unknown
Type: VendorSystemPresent
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning ReconcileFailed 61m (x17 over 17h) StorageSystem controller InstallPlan not found for CSV mcg-operator.v4.10.9
Check from the https://console.redhat.com/openshift/ the addon shows successfully "Installed".
- Another example
% oc get storagesystem ocs-storagecluster-storagesystem -o yaml
apiVersion: odf.openshift.io/v1alpha1
kind: StorageSystem
metadata:
creationTimestamp: "2024-01-03T13:58:27Z"
finalizers:
- storagesystem.odf.openshift.io
generation: 1
name: ocs-storagecluster-storagesystem
namespace: openshift-storage
resourceVersion: "356694259"
uid: d39cdf37-5e9d-4ea6-918d-873faed152f5
spec:
kind: storagecluster.ocs.openshift.io/v1
name: ocs-storagecluster
namespace: openshift-storage
status:
conditions:
- lastHeartbeatTime: "2024-08-13T16:44:10Z"
lastTransitionTime: "2024-08-13T15:32:18Z"
message: Reconcile is in progress
reason: Reconciling
status: "False"
type: Available
- lastHeartbeatTime: "2024-08-13T16:44:10Z"
lastTransitionTime: "2024-08-13T15:32:18Z"
message: Reconcile is in progress
reason: Reconciling
status: "True"
type: Progressing
- lastHeartbeatTime: "2024-08-13T16:44:10Z"
lastTransitionTime: "2024-01-03T13:58:26Z"
message: StorageSystem CR is valid
reason: Valid
status: "False"
type: StorageSystemInvalid
- lastHeartbeatTime: "2024-08-13T16:44:10Z"
lastTransitionTime: "2024-08-13T15:32:19Z"
message: ClusterServiceVersion.operators.coreos.com "mcg-operator.v4.15.4-rhodf"
not found; ClusterServiceVersion.operators.coreos.com "ocs-operator.v4.15.4-rhodf"
not found; ClusterServiceVersion.operators.coreos.com "odf-csi-addons-operator.v4.15.4-rhodf"
not found
reason: NotReady
status: "False"
type: VendorCsvReady
- lastHeartbeatTime: "2024-08-13T15:31:36Z"
lastTransitionTime: "2024-01-03T13:58:26Z"
reason: Found
status: "True"
type: VendorSystemPresent
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments