OCS 4.X - File upload to S3 Noobaa bucket fails with : data AssertionError [ERR_ASSERTION]: _id must be unique. found 2 rows with _id=undefined in table mdsequences

Issue

After upgrading Red Hat OpenShift container platform from 4.7.2 to 4.8.17, and OCS operator v4.6.3 to v4.8.3

When doing an upload to S3: (in this case file name is "file1" , and the bucket name is "obc-test-noobaa-bd45a66d-045d-4cf1-a8aa-2c6644648924" )

$ > s3cmd put file1 s3://obc-test-noobaa-bd45a66d-045d-4cf1-a8aa-2c6644648924

... it fails, we can not upload anything to our s3 storage, the error is shown in the debug output from the s3cmd :

DEBUG: S3Error: 500 (Internal Server Error)
DEBUG: HttpHeader: x-amz-request-id: kvuo3ac5-28ielj-d8s
DEBUG: HttpHeader: x-amz-id-2: kvuo3ac5-28ielj-d8s
DEBUG: HttpHeader: access-control-allow-origin: *
DEBUG: HttpHeader: access-control-allow-credentials: true
DEBUG: HttpHeader: access-control-allow-methods: GET,POST,PUT,DELETE,OPTIONS
DEBUG: HttpHeader: access-control-allow-headers: Content-Type,Content-MD5,Authorization,X-Amz-User-Agent,X-Amz-Date,ETag,X-Amz-Content-Sha256
DEBUG: HttpHeader: access-control-expose-headers: ETag,X-Amz-Version-Id
DEBUG: HttpHeader: content-type: application/xml
DEBUG: HttpHeader: content-length: 267
DEBUG: HttpHeader: date: Thu, 11 Nov 2021 08:06:23 GMT
DEBUG: HttpHeader: keep-alive: timeout=5
DEBUG: HttpHeader: set-cookie: 1a4aa612fe797ac8466d7ee00e5520d5=126049b2ac2d6252558a0ba84e762875; path=/; HttpOnly; Secure; SameSite=None
DEBUG: HttpHeader: etag:
DEBUG: ErrorXML: Code: 'InternalError'
DEBUG: ErrorXML: Message: 'We encountered an internal error. Please try again.'
DEBUG: ErrorXML: Resource: '/obc-test-noobaa-bd45a66d-045d-4cf1-a8aa-2c6644648924/file1'
DEBUG: ErrorXML: RequestId: 'kvuo3ac5-28ielj-d8s'
WARNING: Upload failed: /file1 (500 (InternalError): We encountered an internal error. Please try again.)
WARNING: Waiting 3 sec...

ISSUE 2 : In ODF 4.10.6, after a backup and restore operation of noobaa DB , we can see a similar error on the noobaa-endpoint pod logs:

2022-11-07T22:51:09.731150489Z Nov-7 22:51:09.730 [Endpoint/12] [ERROR] core.util.postgres_client:: updateOneWithClient failed { system: 6335cd3ed98f8500293a0830, bucket: 6336d4827d328300283c9ba3, content_type: 'binary/octet-stream' } { '$set': { last_write: 1667861469712, system: 6335cd3ed98f8500293a0830, bucket: 6336d4827d328300283c9ba3, content_type: 'binary/octet-stream' }, '$inc': { writes: 67779, reads: 0 } } UPDATE bucketstats SET data = jsonb_set(jsonb_set(jsonb_set(jsonb_set(jsonb_set(jsonb_set(data,'{content_type}','"binary/octet-stream"'),'{bucket}','"6336d4827d328300283c9ba3"'),'{system}','"6335cd3ed98f8500293a0830"'),'{last_write}','1667861469712'::jsonb),'{reads}',to_jsonb(COALESCE(Cast(data->>'reads' as numeric),0)+0)),'{writes}',to_jsonb(COALESCE(Cast(data->>'writes' as numeric),0)+67779)) WHERE (data->>'system'='6335cd3ed98f8500293a0830' and data->>'bucket'='6336d4827d328300283c9ba3' and data->>'content_type'='binary/octet-stream') RETURNING _id, data AssertionError [ERR_ASSERTION]: _id must be unique. found 2 rows with _id=undefined in table bucketstats 

Somehow the restore caused duplicate id sequences found in bucketstats tables.