Quay Errors out while connecting to AWS S3 bucket

Solution Verified - Updated -

Environment

  • Red Hat Quay (Quay)
    • 3
  • Red Hat OpenShift Container Platform (RHOCP)
    • 4

Issue

  • Quay is not connecting to s3 bucket as we have described in config file:
DISTRIBUTED_STORAGE_CONFIG:
    s3Storage:
        - S3Storage
        - host: s3.ap-south-1.amazonaws.com
          s3_access_key: *****************************
          s3_bucket: quay-bucket-1
          s3_secret_key: *****************************
          storage_path: /datastorage/registry
DISTRIBUTED_STORAGE_DEFAULT_LOCATIONS: []
DISTRIBUTED_STORAGE_PREFERENCE:
    - s3Storage
  • It errors out with message:
gunicorn-registry stdout | 2022-10-21 14:38:36,892 [253] [DEBUG] [urllib3.connectionpool] https://s3.ap-south-1.amazonaws.com:443 "POST /quay-bucket-1/storage/quayregistrystorage/uploads/23cd6e62-264c-48e4-94a7-21061b0e4ef1?uploads HTTP/1.1" 400 None

Resolution

  • Add s3_ region parameter under AWS storage driver section in Quay config.yaml. The argument for the parameter should be the S3-bucket-region used as shown below:
DISTRIBUTED_STORAGE_CONFIG:
    default:
        - S3Storage
        - host: s3.ap-south-1.amazonaws.com
          s3_access_key: *****************
          s3_bucket: quay-bucket-1
          s3_secret_key: *********************************
          storage_path: /storage/quayregistrystorage
          s3_region: ap-south-1

Root Cause

  • An S3 bucket can be accessed through its URL. The URL format of a bucket is either of two options:

    • http://s3.amazonaws.com/[bucket_name]/
    • http://[bucket_name].s3.amazonaws.com/

  • When configuring Red Hat Quay AWS S3 Cloudfront, a new parameter, s3_region is required. Currently, the Red Hat Quay config editor does not include this field. As a temporary workaround, you must manually insert the s3_region parameter in your config.yaml. If it is not present, the Authorization header explicitly mentions different region and not the region set in hostname in Quay config yaml. This seems to create problems when uploading because AWS rejects requests like and Boto library complains about the authorization header being malformed.

Diagnostic Steps

  • Check Quay debug logs for requests from Quay to the backend s3 bucket and/or for additional information. The following request is being created by boto library using the parameters provided in Quay config file.
$ oc logs <quay_app_pod> -n <quay_namespace>

gunicorn-registry stdout | 2022-10-21 14:38:36,855 [253] [DEBUG] [botocore.endpoint] Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://s3.ap-south-1.amazonaws.com/prod-quaybucket-1/storage/quayregistrystorage/uploads/23cd6e62-264c-48e4-94a7-21061b0e4ef1?uploads, headers={'Content-Type': b'application/octet-stream', 'x-amz-server-sideencryption': b'AES256', 'User-Agent': b'Boto3/1.21.42 Python/3.8.12 Linux/4.18.0-372.26.1.el8_6.x86_64 Botocore/1.24.42 Resource', 'X-Amz-Date': b'20221021T143

836Z', 'X-Amz-Content-SHA256':b'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855', 'Authorization': b'AWS4-HMAC-SHA256 Credential=************/20221021/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date; x-amz-server-side-encryption, Signature=0cb9a0fb318504e8e0e141a650a2f7169cdcf84b80bd01e5e43519614a663ee5', 'amz-sdk-invocation-id': b'f3396002-ff29-406d-87f1-03518c8df072', 'amz-sdk-request': b'attempt=1', 'Content-Length': '0'}>
  • Boto library complains about the authorization header being malformed.
gunicorn-registry stdout | 2022-10-21 14:38:36,892 [253] [DEBUG] [urllib3.connectionpool] https://s3.ap-south-1.amazonaws.com:443 "POST /quay-bucket-1/storage/quayregistrystorage/uploads/23cd6e62-264c-48e4-94a7-21061b0e4ef1?uploads HTTP/1.1" 400 None

gunicorn-registry stdout | 2022-10-21 14:38:36,894 [253] [DEBUG] [botocore.parsers] Response headers: {'x-amz-request-id': 'HGYAYKZN5TRH0G9H', 'x-amz-id-2': 'IpNxTwbnB+8QdXOu44vymB8O4zs7thzzbdQV8FQ+5MiMoZn78O+W5J4fKGvkK+28waqZUZtmI8A=', 'Content-Type': 'application/xml', 'Transfer-Encoding': 'chunked', 'Date': 'Fri, 21 Oct 2022 14:38:36 GMT', 'Server': 'AmazonS3', 'Connection': 'close'}

gunicorn-registry stdout | 2022-10-21 14:38:36,894 [253] [DEBUG] [botocore.parsers] Response body:
gunicorn-registry stdout | b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region \'us-east-1\' is wrong; expecting \'ap-south-1\'</Message><Region>ap-south-1</Region><RequestId>HGYAYKZN5TRH0G9H</RequestId><HostId>IpNxTwbnB+8QdXOu44vymB8O4zs7thzzbdQV8FQ+5MiMoZn78O+W5J4fKGvkK+28waqZUZtmI8A=</HostId></Error>'

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments