ODF upgrade from 4.18 to 4.19 stuck with NooBaa in "Creating" phase and "dbClusterStatus" stuck in "Importing"
Issue
The ODF 4.19 upgrade migrates the NooBaa database from the existing single PostgreSQL instance to a highly available PostgreSQL cluster managed by CloudNativePG (CNPG). If you are hitting this issue, the following is observed:
-
ODF upgrade completed successfully; however, the StorageCluster CR remains in phase
Progressingdue to the NooBaa CR failing to reconcile with a message:Waiting on Nooba instance to finish initialization. -
NooBaa S3 services remained unavailable until manual intervention was taken.
-
The NooBaa CR remains in
phase Creatingwith NooBaa CR "message: cluster creation or import is still in progress, skipping update," and thedbClusterStatus: Importingindefinitely. -
An s3 outage remains as the
noobaa-coreandnoobaa-endpointpods fail to scale up due to the upgrade sequence requiring thedbClusterStatusto beReady, ** WARNING: DO NOT MANUALLY SCALE UP noobaa-core and noobaa-endpoint.** -
The CNPG Cluster reports
Phase: Cluster in healthy state -
The
noobaa-operatorlogs contain the following warnings and errors repeatedly: -
BackingStore/NamespaceStore remain in phase
Connectinglevel=info msg="cnpg:: the cluster spec was changed but the cluster creation or import is still in progress, skipping update" sys=openshift-storage/noobaa level=error msg="cnpg:: got error reconciling cluster. error: cluster creation or import is still in progress, skipping update" sys=openshift-storage/noobaa level=warning msg="⏳ Temporary Error: cluster creation or import is still in progress, skipping update" sys=openshift-storage/noobaa -
The
noobaa-db-pg-0pod remains alongsidenoobaa-db-pg-cluster-1andnoobaa-db-pg-cluster-2pods.
Environment
- Red Hat OpenShift Container Platform (RHOCP) 4.x
- Red Hat OpenShift Data Foundation (ODF) 4.19.x (upgrading from 4.18.x)
- Multi-Cloud Object Gateway (MCG) / NooBaa
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.