NooBaa creates empty buckets in OpenShift Container Storage 4.X cluster, why?

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Storage 4.X

Issue

  • NooBaa creates empty buckets in OpenShift Container Storage 4.X cluster, why?

  • NooBaa's Backingstore is flapping between ALL_NODES_OFFLINE and OPTIMAL

Resolution

  • This issue is targeted to be fixed in OCS 4.7.3.

  • To get rid of the stale empty buckets, one would need to delete them manually:

    # noobaa bucket delete <bucket-name>

    NOTE: The noobaa CLI can be obtained from the MCG package

Root Cause

  • Buckets are always deleted when cleanup is required by the library. However, on OBC creation failure, lib-bucket-provisioner sends a delete request to the provisioner for cleanup purposes before retrying. Noobaa provisioner would look at the reclaim policy of the object bucket but in some cases did not delete the underlying bucket.

  • This issue is been tracked in Bug #1947796

Diagnostic Steps

  • There would be only a few Object Bucket Claim(s):
NAMESPACE       NAME                      BUCKET-NAME   STORAGE-CLASS                 BUCKET-CLASS                  PHASE   
quay-registry   instance-quay-datastore                 openshift-storage.noobaa.io   noobaa-default-bucket-class   Bound
  • However, there would be many more Object Buckets created in NooBaa:
$ grep quay buckets-nooba.txt | wc -l
9589

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments