Chapter 9. Multicloud Object Gateway

Procedure

1. From the MCG command-line interface, run the following command:

   noobaa backingstore create <backing-store-type> <backingstore_name> --access-key=<AWS ACCESS KEY> --secret-key=<AWS SECRET ACCESS KEY> --target-bucket <bucket-name>

   1. Replace <backing-store-type> with your relevant backing store type: aws-s3, google-cloud-store, azure-blob, s3-compatible, or ibm-cos.
   2. Replace <backingstore_name> with the name of the backingstore.
   3. Replace <AWS ACCESS KEY> and <AWS SECRET ACCESS KEY> with an AWS access key ID and secret access key you created for this purpose.

   4. Replace <bucket-name> with an existing AWS bucket name. This argument tells NooBaa which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.

      The output will be similar to the following:

      INFO[0001] ✅ Exists: NooBaa "noobaa"
      INFO[0002] ✅ Created: BackingStore "aws-resource"
      INFO[0002] ✅ Created: Secret "backing-store-secret-aws-resource"

1. Create a secret with the credentials:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <backingstore-secret-name>
   type: Opaque
   data:
     AWS_ACCESS_KEY_ID: <AWS ACCESS KEY ID ENCODED IN BASE64>
     AWS_SECRET_ACCESS_KEY: <AWS SECRET ACCESS KEY ENCODED IN BASE64>

   1. You must supply and encode your own AWS access key ID and secret access key using Base64, and use the results in place of <AWS ACCESS KEY ID ENCODED IN BASE64> and <AWS SECRET ACCESS KEY ENCODED IN BASE64>.
   2. Replace <backingstore-secret-name> with a unique name.

2. Apply the following YAML for a specific backing store:

   apiVersion: noobaa.io/v1alpha1
   kind: BackingStore
   metadata:
     finalizers:
     - noobaa.io/finalizer
     labels:
       app: noobaa
     name: bs
     namespace: noobaa
   spec:
     awsS3:
       secret:
         name: <backingstore-secret-name>
         namespace: noobaa
       targetBucket: <bucket-name>
     type: <backing-store-type>

   1. Replace <bucket-name> with an existing AWS bucket name. This argument tells NooBaa which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.
   2. Replace <backingstore-secret-name> with the name of the secret created in the previous step.
   3. Replace <backing-store-type> with your relevant backing store type: aws-s3, google-cloud-store, azure-blob, s3-compatible, or ibm-cos.

Procedure

1. From the Multicloud Object Gateway (MCG) command-line interface, run the following NooBaa command:

   noobaa backingstore create s3-compatible rgw-resource --access-key=<RGW ACCESS KEY> --secret-key=<RGW SECRET KEY> --target-bucket=<bucket-name> --endpoint=<RGW endpoint>

   1. To get the <RGW ACCESS KEY> and <RGW SECRET KEY>, run the following command using your RGW user secret name:

      oc get secret <RGW USER SECRET NAME> -o yaml

   2. Decode the access key ID and the access key from Base64 and keep them.
   3. Replace <RGW USER ACCESS KEY> and <RGW USER SECRET ACCESS KEY> with the appropriate, decoded data from the previous step.
   4. Replace <bucket-name> with an existing RGW bucket name. This argument tells Multicloud Object Gateway which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.

   5. To get the <RGW endpoint>, see Accessing the RADOS Object Gateway S3 endpoint.

      The output will be similar to the following:

      INFO[0001] ✅ Exists: NooBaa "noobaa"
      INFO[0002] ✅ Created: BackingStore "rgw-resource"
      INFO[0002] ✅ Created: Secret "backing-store-secret-rgw-resource"

1. Create a CephObjectStore user. This also creates a secret containing the RGW credentials:

   apiVersion: ceph.rook.io/v1
   kind: CephObjectStoreUser
   metadata:
     name: <RGW-Username>
     namespace: openshift-storage
   spec:
     store: ocs-storagecluster-cephobjectstore
     displayName: "<Display-name>"

   1. Replace <RGW-Username> and <Display-name> with a unique username and display name.

2. Apply the following YAML for an S3-Compatible backing store:

   apiVersion: noobaa.io/v1alpha1
   kind: BackingStore
   metadata:
     finalizers:
     - noobaa.io/finalizer
     labels:
       app: noobaa
     name: <backingstore-name>
     namespace: openshift-storage
   spec:
     s3Compatible:
       endpoint: <RGW endpoint>
       secret:
         name: <backingstore-secret-name>
         namespace: openshift-storage
       signatureVersion: v4
       targetBucket: <RGW-bucket-name>
     type: s3-compatible

   1. Replace <backingstore-secret-name> with the name of the secret that was created with CephObjectStore in the previous step.
   2. Replace <bucket-name> with an existing RGW bucket name. This argument tells Multicloud Object Gateway which bucket to use as a target bucket for its backing store, and subsequently, data storage and administration.
   3. To get the <RGW endpoint>, see Accessing the RADOS Object Gateway S3 endpoint.

Procedure

1. In your OpenShift Storage console, navigate to Overview → Object Service → select the Multicloud Object Gateway link:

   

2. Select the Resources tab in the left, highlighted below. From the list that populates, select Add Cloud Resource:

   

3. Select Add new connection:

   

4. Select the relevant native cloud provider or S3 compatible option and fill in the details:

   

5. Select the newly created connection and map it to the existing bucket:

   
6. Repeat these steps to create as many backing stores as needed.

Procedure

1. Click Operators → Installed Operators from the left pane of the OpenShift Web Console to view the installed operators.
2. Click OpenShift Container Storage Operator.

3. On the OpenShift Container Storage Operator page, scroll right and click the Bucket Class tab.

   Figure 9.1. OpenShift Container Storage Operator page with Bucket Class tab

   
4. Click Create Bucket Class.

5. On the Create new Bucket Class page, perform the following:

   1. Enter a Bucket Class Name and click Next.

      Figure 9.2. Create Bucket Class page

      

   2. In Placement Policy, select Tier 1 - Policy Type and click Next. You can choose either one of the options as per your requirements.

      • Spread allows spreading of the data across the chosen resources.
      • Mirror allows full duplication of the data across the chosen resources.

      • Click Add Tier to add another policy tier.

        Figure 9.3. Tier 1 - Policy Type selection page

        

   3. Select atleast one Backing Store resource from the available list if you have selected Tier 1 - Policy Type as Spread and click Next. Alternatively, you can also create a new backing store.

      Figure 9.4. Tier 1 - Backing Store selection page

      

1. Review and confirm Bucket Class settings.

   Figure 9.5. Bucket class settings review page

   
2. Click Create Bucket Class.

Verification steps

1. Click Operators → Installed Operators.
2. Click OpenShift Container Storage Operator.
3. Search for the new Bucket Class or click Bucket Class tab to view all the Bucket Classes.

Procedure

1. Click Operators → Installed Operators from the left pane of the OpenShift Web Console to view the installed operators.
2. Click OpenShift Container Storage Operator.

3. On the OpenShift Container Storage Operator page, scroll right and click the Backing Store tab.

   Figure 9.6. OpenShift Container Storage Operator page with backing store tab

   

4. Click Create Backing Store.

   Figure 9.7. Create Backing Store page

   

5. On the Create New Backing Store page, perform the following:

   1. Enter a Backing Store Name.
   2. Select a Provider.
   3. Select a Region.
   4. Enter an Endpoint. This is optional.

   5. Select a Secret from drop down list, or create your own secret. Optionally, you can Switch to Credentials view which lets you fill in the required secrets.

      For more information on creating an OCP secret, see the section Creating the secret in the Openshift Container Platform documentation.

      Each backingstore requires a different secret. For more information on creating the secret for a particular backingstore, see the Section 9.3.1, “Adding storage resources for hybrid or Multicloud using the MCG command line interface” and follow the procedure for the addition of storage resources using a YAML.

      Note

      This menu is relevant for all providers except Google Cloud and local PVC.

   6. Enter Target bucket. The target bucket is a container storage that is hosted on the remote cloud service. It allows you to create a connection that tells MCG that it can use this bucket for the system.
6. Click Create Backing Store.

Verification steps

1. Click Operators → Installed Operators.
2. Click OpenShift Container Storage Operator.
3. Search for the new backing store or click Backing Store tab to view all the backing stores.

1. From the MCG command-line interface, run the following command to create a bucket class with a mirroring policy:

   $ noobaa bucketclass create mirror-to-aws --backingstores=azure-resource,aws-resource --placement Mirror

2. Set the newly created bucket class to a new bucket claim, generating a new bucket that will be mirrored between two locations:

   $ noobaa obc create  mirrored-bucket --bucketclass=mirror-to-aws

1. Apply the following YAML. This YAML is a hybrid example that mirrors data between local Ceph storage and AWS:

   apiVersion: noobaa.io/v1alpha1
   kind: BucketClass
   metadata:
     name: hybrid-class
     labels:
       app: noobaa
   spec:
     placementPolicy:
       tiers:
         - tier:
             mirrors:
               - mirror:
                   spread:
                     - cos-east-us
               - mirror:
                   spread:
                     - noobaa-test-bucket-for-ocp201907291921-11247_resource

2. Add the following lines to your standard Object Bucket Claim (OBC):

   additionalConfig:
     bucketclass: mirror-to-aws

   For more information about OBCs, see Section 9.6, “Object Bucket Claim”.

1. In your OpenShift Storage console, navigate to Overview → Object Service → select the Multicloud Object Gateway link:

   

2. Click the buckets icon on the left side. You will see a list of your buckets:

   
3. Click the bucket you want to update.

4. Click Edit Tier 1 Resources:

   

5. Select Mirror and check the relevant resources you want to use for this bucket. In the following example, we mirror data between on prem Ceph RGW to AWS:

   
6. Click Save.

1. Create the bucket policy in JSON format. See the following example:

   {
       "Version": "NewVersion",
       "Statement": [
           {
               "Sid": "Example",
               "Effect": "Allow",
               "Principal": [
                       "john.doe@example.com"
               ],
               "Action": [
                   "s3:GetObject"
               ],
               "Resource": [
                   "arn:aws:s3:::john_bucket"
               ]
           }
       ]
   }

   There are many available elements for bucket policies. For details on these elements and examples of how they can be used, see AWS Access Policy Language Overview.

   For more examples of bucket policies, see AWS Bucket Policy Examples.

   Instructions for creating S3 users can be found in Section 9.5.3, “Creating an AWS S3 user in the Multicloud Object Gateway”.

2. Using AWS S3 client, use the put-bucket-policy command to apply the bucket policy to your S3 bucket:

   # aws --endpoint ENDPOINT --no-verify-ssl s3api put-bucket-policy --bucket MyBucket --policy BucketPolicy

   Replace ENDPOINT with the S3 endpoint

   Replace MyBucket with the bucket to set the policy on

   Replace BucketPolicy with the bucket policy JSON file

   Add --no-verify-ssl if you are using the default self signed certificates

   For example:

   # aws --endpoint https://s3-openshift-storage.apps.gogo44.noobaa.org --no-verify-ssl s3api put-bucket-policy -bucket MyBucket --policy file://BucketPolicy

   For more information on the put-bucket-policy command, see the AWS CLI Command Reference for put-bucket-policy.

Procedure

1. In your OpenShift Storage console, navigate to Overview → Object Service → select the Multicloud Object Gateway link:

   

2. Under the Accounts tab, click Create Account:

   

3. Select S3 Access Only, provide the Account Name, for example, john.doe@example.com. Click Next:

   

4. Select S3 default placement, for example, noobaa-default-backing-store. Select Buckets Permissions. A specific bucket or all buckets can be selected. Click Create:

   

Procedure

1. Add the following lines to your application YAML:

   apiVersion: objectbucket.io/v1alpha1
   kind: ObjectBucketClaim
   metadata:
     name: <obc-name>
   spec:
     generateBucketName: <obc-bucket-name>
     storageClassName: openshift-storage.noobaa.io

   These lines are the Object Bucket Claim itself.

   1. Replace <obc-name> with the a unique Object Bucket Claim name.
   2. Replace <obc-bucket-name> with a unique bucket name for your Object Bucket Claim.

2. You can add more lines to the YAML file to automate the use of the Object Bucket Claim. The example below is the mapping between the bucket claim result, which is a configuration map with data and a secret with the credentials. This specific job will claim the Object Bucket from NooBaa, which will create a bucket and an account.

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: testjob
   spec:
     template:
       spec:
         restartPolicy: OnFailure
         containers:
           - image: <your application image>
             name: test
             env:
               - name: BUCKET_NAME
                 valueFrom:
                   configMapKeyRef:
                     name: <obc-name>
                     key: BUCKET_NAME
               - name: BUCKET_HOST
                 valueFrom:
                   configMapKeyRef:
                     name: <obc-name>
                     key: BUCKET_HOST
               - name: BUCKET_PORT
                 valueFrom:
                   configMapKeyRef:
                     name: <obc-name>
                     key: BUCKET_PORT
               - name: AWS_ACCESS_KEY_ID
                 valueFrom:
                   secretKeyRef:
                     name: <obc-name>
                     key: AWS_ACCESS_KEY_ID
               - name: AWS_SECRET_ACCESS_KEY
                 valueFrom:
                   secretKeyRef:
                     name: <obc-name>
                     key: AWS_SECRET_ACCESS_KEY

   1. Replace all instances of <obc-name> with your Object Bucket Claim name.
   2. Replace <your application image> with your application image.

3. Apply the updated YAML file:

   # oc apply -f <yaml.file>

   1. Replace <yaml.file> with the name of your YAML file.

4. To view the new configuration map, run the following:

   # oc get cm <obc-name>

   1. Replace obc-name with the name of your Object Bucket Claim.

      You can expect the following environment variables in the output:

      • BUCKET_HOST - Endpoint to use in the application

      • BUCKET_PORT - The port available for the application

        • The port is related to the BUCKET_HOST. For example, if the BUCKET_HOST is https://my.example.com, and the BUCKET_PORT is 443, the endpoint for the object service would be https://my.example.com:443.
      • BUCKET_NAME - Requested or generated bucket name
      • AWS_ACCESS_KEY_ID - Access key that is part of the credentials
      • AWS_SECRET_ACCESS_KEY - Secret access key that is part of the credentials

Procedure

1. Use the command-line interface to generate the details of a new bucket and credentials. Run the following command:

   # noobaa obc create <obc-name> -n openshift-storage

   Replace <obc-name> with a unique Object Bucket Claim name, for example, myappobc.

   Additionally, you can use the --app-namespace option to specify the namespace where the Object Bucket Claim configuration map and secret will be created, for example, myapp-namespace.

   Example output:

   INFO[0001] ✅ Created: ObjectBucketClaim "test21obc"

   The MCG command-line-interface has created the necessary configuration and has informed OpenShift about the new OBC.

2. Run the following command to view the Object Bucket Claim:

   # oc get obc -n openshift-storage

   Example output:

   NAME        STORAGE-CLASS                 PHASE   AGE
   test21obc   openshift-storage.noobaa.io   Bound   38s

3. Run the following command to view the YAML file for the new Object Bucket Claim:

   # oc get obc test21obc -o yaml -n openshift-storage

   Example output:

   apiVersion: objectbucket.io/v1alpha1
   kind: ObjectBucketClaim
   metadata:
     creationTimestamp: "2019-10-24T13:30:07Z"
     finalizers:
     - objectbucket.io/finalizer
     generation: 2
     labels:
       app: noobaa
       bucket-provisioner: openshift-storage.noobaa.io-obc
       noobaa-domain: openshift-storage.noobaa.io
     name: test21obc
     namespace: openshift-storage
     resourceVersion: "40756"
     selfLink: /apis/objectbucket.io/v1alpha1/namespaces/openshift-storage/objectbucketclaims/test21obc
     uid: 64f04cba-f662-11e9-bc3c-0295250841af
   spec:
     ObjectBucketName: obc-openshift-storage-test21obc
     bucketName: test21obc-933348a6-e267-4f82-82f1-e59bf4fe3bb4
     generateBucketName: test21obc
     storageClassName: openshift-storage.noobaa.io
   status:
     phase: Bound

4. Inside of your openshift-storage namespace, you can find the configuration map and the secret to use this Object Bucket Claim. The CM and the secret have the same name as the Object Bucket Claim. To view the secret:

   # oc get -n openshift-storage secret test21obc -o yaml

   Example output:

   Example output:
   apiVersion: v1
   data:
     AWS_ACCESS_KEY_ID: c0M0R2xVanF3ODR3bHBkVW94cmY=
     AWS_SECRET_ACCESS_KEY: Wi9kcFluSWxHRzlWaFlzNk1hc0xma2JXcjM1MVhqa051SlBleXpmOQ==
   kind: Secret
   metadata:
     creationTimestamp: "2019-10-24T13:30:07Z"
     finalizers:
     - objectbucket.io/finalizer
     labels:
       app: noobaa
       bucket-provisioner: openshift-storage.noobaa.io-obc
       noobaa-domain: openshift-storage.noobaa.io
     name: test21obc
     namespace: openshift-storage
     ownerReferences:
     - apiVersion: objectbucket.io/v1alpha1
       blockOwnerDeletion: true
       controller: true
       kind: ObjectBucketClaim
       name: test21obc
       uid: 64f04cba-f662-11e9-bc3c-0295250841af
     resourceVersion: "40751"
     selfLink: /api/v1/namespaces/openshift-storage/secrets/test21obc
     uid: 65117c1c-f662-11e9-9094-0a5305de57bb
   type: Opaque

   The secret gives you the S3 access credentials.

5. To view the configuration map:

   # oc get -n openshift-storage cm test21obc -o yaml

   Example output:

   apiVersion: v1
   data:
     BUCKET_HOST: 10.0.171.35
     BUCKET_NAME: test21obc-933348a6-e267-4f82-82f1-e59bf4fe3bb4
     BUCKET_PORT: "31242"
     BUCKET_REGION: ""
     BUCKET_SUBREGION: ""
   kind: ConfigMap
   metadata:
     creationTimestamp: "2019-10-24T13:30:07Z"
     finalizers:
     - objectbucket.io/finalizer
     labels:
       app: noobaa
       bucket-provisioner: openshift-storage.noobaa.io-obc
       noobaa-domain: openshift-storage.noobaa.io
     name: test21obc
     namespace: openshift-storage
     ownerReferences:
     - apiVersion: objectbucket.io/v1alpha1
       blockOwnerDeletion: true
       controller: true
       kind: ObjectBucketClaim
       name: test21obc
       uid: 64f04cba-f662-11e9-bc3c-0295250841af
     resourceVersion: "40752"
     selfLink: /api/v1/namespaces/openshift-storage/configmaps/test21obc
     uid: 651c6501-f662-11e9-9094-0a5305de57bb

   The configuration map contains the S3 endpoint information for your application.

Procedure

1. Log into the OpenShift Web Console.
2. On the left navigation bar, click Storage → Object Bucket Claims.

3. Click Create Object Bucket Claim:

   

4. Enter a name for your object bucket claim and select the appropriate storage class based on your deployment, internal or external, from the dropdown menu:

   Internal mode

   

   The following storage classes, which were created after deployment, are available for use:

   • ocs-storagecluster-ceph-rgw uses the Ceph Object Gateway (RGW)
   • openshift-storage.noobaa.io uses the Multicloud Object Gateway

   External mode

   

   The following storage classes, which were created after deployment, are available for use:

   • ocs-external-storagecluster-ceph-rgw uses the Ceph Object Gateway (RGW)

   • openshift-storage.noobaa.io uses the Multicloud Object Gateway

     Note

     The RGW OBC storage class is only available with fresh installations of OpenShift Container Storage version 4.5. It does not apply to clusters upgraded from previous OpenShift Container Storage releases.

5. Click Create.

   Once you create the OBC, you are redirected to its detail page:

   

Procedure

1. In the Multicloud Object Gateway user interface, from the Overview page, click Add Storage Resources:

   

2. In the window, click Deploy Kubernetes Pool:

   

3. In the Create Pool step create the target pool for the future installed nodes.

   

4. In the Configure step, configure the number of requested pods and the size of each PV. For each new pod, one PV is be created.

   
5. In the Review step, you can find the details of the new pool and select the deployment method you wish to use: local or external deployment. If local deployment is selected, the Kubernetes nodes will deploy within the cluster. If external deployment is selected, you will be provided with a YAML file to run externally.

6. All nodes will be assigned to the pool you chose in the first step, and can be found under Resources → Storage resources → Resource name: