NooBaa "Error: Code=UNAUTHORIZED Message=account not found" - OpenShift Data Foundation (ODF)

Issue

Usually in this particular scenario, this very likely is data loss. The UNAUTHORIZED Message=account not found error can be attributed to the db-noobaa-db-pg-0 being accidentally deleted. The noobaa DB holds all of noobaa's metadata - system configurations, authentication info, buckets info, and objects metadata. Deleting the PV would mean losing data. The buckets would still be present, however, the keys to gain access to those buckets are no longer there.

There could be some misconception, as it may appear NooBaa is healthy with all pods up/running. However, NooBaa is a managed resource. If the db PVC gets deleted, the NooBaa Operator reconciles itself by creating a new PVC so the statefulset can become running again.

In the event, the above does not apply, there will be troubleshooting steps to perform.

time="2021-03-18T08:42:07Z" level=warning msg="RPC: GetConnection creating connection to wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/ 0xc00059acd0"
time="2021-03-18T08:42:07Z" level=info msg="RPC: Connecting websocket (0xc00059acd0) &{RPC:0xc0003fd860 Address:wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/ State:init WS:<nil> PendingRequests:
map[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s}"
time="2021-03-18T08:42:07Z" level=info msg="RPC: Connected websocket (0xc00059acd0) &{RPC:0xc0003fd860 Address:wss://noobaa-mgmt.openshift-storage.svc.cluster.local:443/rpc/ State:init WS:<nil> PendingRequests:m
ap[] NextRequestID:0 Lock:{state:1 sema:0} ReconnectDelay:0s}"
time="2021-03-18T08:42:07Z" level=error msg="⚠️  RPC: system.read_system() Response Error: Code=UNAUTHORIZED Message=account not found 5e96a72c498f080030a76337