Duplicate of the rbac-permissions-operator in new ROSA cluster.

Solution Verified - Updated -

Environment

  • Red Hat Openshift Service on AWS (ROSA)
    • 4.x
  • Red Hat OpenShift Dedicated (OSD)
    • 4.x

Issue

  • A newly installed ROSA cluster with version 4.10.13, In Installed Operators view, showing a duplicate of the same operator rbac-permissions-operator.
  • The cluster came with a broken configuration on start and operator was already available.
  • Operator Subscription and CSV can't bind each other.
  • The issue has been seen in both fresh install and upgrade:
    • A subscription is created for an Operator.
    • A operator is upgraded to a new version

Resolution

  • This is being tracked in the bug BZ1980755 preventing the cluster's ability to upgrade operators.
  • A full wipe of the CSV's and operator resources helped to resolve this issue.
oc delete -f <(oc get csv -A -l operators.coreos.com/managed-velero-operator.openshift-velero="" -o json)
oc delete -f <(oc get csv -A -l operators.coreos.com/route-monitor-operator.openshift-route-monitor-operator="" -o json)
oc delete -f <(oc get csv -A -l operators.coreos.com/splunk-forwarder-operator.openshift-splunk-forwarder-operator="" -o json)
oc delete -f <(oc get csv -A -l operators.coreos.com/addon-operator.openshift-addon-operator="" -o json)
oc delete -f <(oc get csv -A -l operators.coreos.com/ocm-agent-operator.openshift-ocm-agent-operator="" -o json)
oc delete -f <(oc get csv -A -l operators.coreos.com/rbac-permissions-operator.openshift-rbac-permissions="" -o json)
oc delete -f <(oc get csv -A -l operators.coreos.com/managed-upgrade-operator.openshift-managed-upgrade-operator="" -o json)
oc delete -f <(oc get csv -A -l operators.coreos.com/configure-alertmanager-operator.openshift-monitoring="" -o json)
oc delete -f <(oc get csv -A -l operators.coreos.com/must-gather-operator.openshift-must-gather-operator="" -o json)
oc delete -f <(oc get csv -A -l operators.coreos.com/custom-domains-operator.openshift-custom-domains-operator="" -o json)
oc delete -f <(oc get csv -A -l operators.coreos.com/osd-metrics-exporter.openshift-osd-metrics="" -o json)
oc delete -f <(oc get csv -A -l operators.coreos.com/cloud-ingress-operator.openshift-cloud-ingress-operator="" -o json)
# for some reason MNMO never has a CSV label?
oc delete csv -n openshift-managed-node-metadata-operator managed-node-metadata-operator.v0.1.69-f49190f 
oc delete -f <(oc get sub,catsrc,csv,og,operator,operatorconditions -A -l hive.openshift.io/managed -o json)

Root Cause

  • The CSV of the operator is created, but there is no update in the subscription status.
  • Even if the install plan is completed causing the subscription is in the unknown status and CSV is in the Cannot Update status.

Diagnostic Steps

(1) Check if duplicate operators are available in installed operators view and verify its status is unknown and Catalog Source not found
rbac-permissions-operators status

(2) Navigate to openshift-rbac-permissions namespace and check if multiple install plans are available for openshift-rbac-permissions.

$ oc project openshift-rbac-permissions

$ oc get ip
NAME            CSV                                          APPROVAL    APPROVED
install-4vtgk   rbac-permissions-operator.v0.1.206-3623c53   Automatic   true
install-9fndw   rbac-permissions-operator.v0.1.224-b589559   Automatic   true
install-fc2gv   rbac-permissions-operator.v0.1.206-3623c53   Automatic   true
install-ffjt5   rbac-permissions-operator.v0.1.234-cad2008   Automatic   true
install-zmw99   rbac-permissions-operator.v0.1.216-99c383f   Automatic   true

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments