Investigating and Fixing Duplicates inside NooBaa-DB PostgreSQL DB Causing Backingstore Flapping or Performance Degradation in OpenShift Data Foundation (ODF)

Issue

• ODF object buckets are experiencing intermittent connectivity. Behaviors are commonly observed after the manual PostgreSQL v12-v15 upgrade was accomplished after upgrading ODF from ODF v4.14 to ODF v4.15+. Observations include longer queries shown in the noobaa-core pod, increased resource consumption on noobaa-db-pg-0, and timeouts on the noobaa-endpoint pod.

$ oc adm top pods | grep noobaa                                     CPU          Memory
noobaa-core-0                                                       349m         667Mi
noobaa-db-pg-0                                                      12188m       15199Mi <----
noobaa-endpoint-556cfc78cf-j84cb                                    150m         825Mi

noobaa-endpoint-1-logs.out:Mar-25 7:10:26.870 [Endpoint/13]  [WARN] core.sdk.object_io:: upload_object: failed upload { bucket: <omitted>, key: '<omitted>', md_conditions: undefined, obj_id: '67e256e275e186000df74245', size: 8605, num_parts: 1, md5_b64: 'd7+e/SPZ16Qp/oflPYGzTA==', sha256_b64: '<omitted>' } [RpcError: duplicate key value violates unique constraint "idx_btree_objectmds_latest_version_index"] { rpc_code: 'INTERNAL', rpc_data: { retryable: true } }

• The resolution steps can be applied also in ODF 4.19 or higher were noobaa-db is in HA mode with CNPG, with two noobaa db pods running, in this example noobaa-db-pg-cluster-1 is the primary , so commands will be performed only on the primary pod.

  % oc get cluster
  NAME                   AGE     INSTANCES   READY   STATUS                     PRIMARY
  noobaa-db-pg-cluster   2d10h   2           2       Cluster in healthy state   noobaa-db-pg-cluster-1
  
  noobaa-db-pg-cluster-1                                            1/1     Running     0               2d10h
  noobaa-db-pg-cluster-2                                            1/1     Running     0               2d10h
  

  ODF 4.19 or higher: On the Resolution section below , some commands need a minor change: Where it says:

  noobaa-db-pg-0 change to the primary pod, in this example: noobaa-db-pg-cluster-1

  /var/lib/pgsql/data/ change to /var/lib/postgres/data/