Chapter 7. Administration

Procedure

1. Calculate the recommended number of shards:

   number of objects expected in a bucket / 100,000

   Note

   The maximum number of bucket index shards currently supported is 65,521.

2. Set the rgw_override_bucket_index_max_shards parameter accordingly:

   Syntax

   ceph config set client.rgw rgw_override_bucket_index_max_shards VALUE

   Replace VALUE with the recommended number of shards calculated:

   Example

   [ceph: root@host01 /]# ceph config set client.rgw rgw_override_bucket_index_max_shards 12

   • To configure bucket index resharding for all instances of the Ceph Object Gateway, set the rgw_override_bucket_index_max_shards parameter with the global option.
   • To configure bucket index resharding only for a particular instance of the Ceph Object Gateway, add rgw_override_bucket_index_max_shards parameter under the instance.

3. Restart the Ceph Object Gateways on all nodes in the cluster to take effect:

   Syntax

   ceph orch restart SERVICE_TYPE

   Example

   [ceph: root#host01 /]# ceph orch restart rgw

Procedure

1. Calculate the recommended number of shards:

   number of objects expected in a bucket / 100,000

   Note

   The maximum number of bucket index shards currently supported is 65,521.

2. Extract the zone group configuration to the zonegroup.json file:

   Example

   [ceph: root@host01 /]# radosgw-admin zonegroup get > zonegroup.json

3. In the zonegroup.json file, set the bucket_index_max_shards parameter for each named zone:

   Syntax

   bucket_index_max_shards = VALUE

   Replace VALUE with the recommended number of shards calculated:

   Example

   bucket_index_max_shards = 12

4. Reset the zone group:

   Example

   [ceph: root@host01 /]# radosgw-admin zonegroup set < zonegroup.json

5. Update the period:

   Example

   [ceph: root@host01 /]# radosgw-admin period update --commit

6. Check if resharding is complete:

   Syntax

   radosgw-admin reshard status --bucket BUCKET_NAME

   Example

   [ceph: root@host01 /]# radosgw-admin reshard status --bucket data

Procedure

1. Set the rgw_dynamic_resharding parameter is set to true.

   Example

   [ceph: root@host01 /]# radosgw-admin period get

2. Optional: Customize Ceph configuration using the following command:

   Syntax

   ceph config set client.rgw OPTION VALUE

   Replace OPTION with the following options:

   • rgw_reshard_num_logs: The number of shards for the resharding log. The default value is 16.
   • rgw_reshard_bucket_lock_duration: The duration of the lock on a bucket during resharding. The default value is 360 seconds.
   • rgw_dynamic_resharding: Enables or disables dynamic resharding. The default value is true.
   • rgw_max_objs_per_shard: The maximum number of objects per shard. The default value is 100000 objects per shard.
   • rgw_reshard_thread_interval: The maximum time between rounds of reshard thread processing. The default value is 600 seconds.

   Example

   [ceph: root@host01 /]# ceph config set client.rgw rgw_reshard_num_logs 23

3. Add a bucket to the resharding queue:

   Syntax

   radosgw-admin reshard add --bucket BUCKET --num-shards NUMBER

   Example

   [ceph: root@host01 /]# radosgw-admin reshard add --bucket data --num-shards 10

4. List the resharding queue:

   Example

   [ceph: root@host01 /]# radosgw-admin reshard list

5. Check the bucket log generations and shards:

   Example

   [ceph: root@host01 /]# radosgw-admin bucket layout --bucket data
   {
       "layout": {
           "resharding": "None",
           "current_index": {
               "gen": 1,
               "layout": {
                   "type": "Normal",
                   "normal": {
                       "num_shards": 23,
                       "hash_type": "Mod"
                   }
               }
           },
           "logs": [
               {
                   "gen": 0,
                   "layout": {
                       "type": "InIndex",
                       "in_index": {
                           "gen": 0,
                           "layout": {
                               "num_shards": 11,
                               "hash_type": "Mod"
                           }
                       }
                   }
               },
               {
                   "gen": 1,
                   "layout": {
                       "type": "InIndex",
                       "in_index": {
                           "gen": 1,
                           "layout": {
                               "num_shards": 23,
                               "hash_type": "Mod"
                           }
                       }
                   }
               }
           ]
       }
   }

6. Check bucket resharding status:

   Syntax

   radosgw-admin reshard status --bucket BUCKET

   Example

   [ceph: root@host01 /]# radosgw-admin reshard status --bucket data

7. Process entries on the resharding queue immediately:

   [ceph: root@host01 /]# radosgw-admin reshard process

8. Cancel pending bucket resharding:

   Warning

   You can only cancel pending resharding operations. Do not cancel ongoing resharding operations.

   Syntax

   radosgw-admin reshard cancel --bucket BUCKET

   Example

   [ceph: root@host01 /]# radosgw-admin reshard cancel --bucket data

Procedure

1. Check if resharding is enabled on the zonegroup:

   Example

   [ceph: root@host01 /]# radosgw-admin sync status

   If zonegroup features enabled is not enabled for resharding on the zonegroup, then continue with the procedure.

2. Enable the resharding feature on all the zonegroups in the multi-site configuration where Ceph Object Gateway is installed:

   Syntax

   radosgw-admin zonegroup modify --rgw-zonegroup=ZONEGROUP_NAME --enable-feature=resharding

   Example

   [ceph: root@host01 /]# radosgw-admin zonegroup modify --rgw-zonegroup=us --enable-feature=resharding

3. Update the period and commit:

   Example

   [ceph: root@host01 /]# radosgw-admin period update --commit

4. Enable the resharding feature on all the zones in the multi-site configuration where Ceph Object Gateway is installed:

   Syntax

   radosgw-admin zone modify --rgw-zone=ZONE_NAME --enable-feature=resharding

   Example

   [ceph: root@host01 /]# radosgw-admin zone modify --rgw-zone=us-east --enable-feature=resharding

5. Update the period and commit:

   Example

   [ceph: root@host01 /]# radosgw-admin period update --commit

6. Verify the resharding feature is enabled on the zones and zonegroups. You can see that each zone lists its supported_features and the zonegroups lists its enabled_features

   Example

   [ceph: root@host01 /]# radosgw-admin period get
   
   "zones": [
                       {
                           "id": "505b48db-6de0-45d5-8208-8c98f7b1278d",
                           "name": "us_east",
                           "endpoints": [
                               "http://10.0.208.11:8080"
                           ],
                           "log_meta": "false",
                           "log_data": "true",
                           "bucket_index_max_shards": 11,
                           "read_only": "false",
                           "tier_type": "",
                           "sync_from_all": "true",
                           "sync_from": [],
                           "redirect_zone": "",
                           "supported_features": [
                               "resharding"
                           ]
   .
   .
                   "default_placement": "default-placement",
                   "realm_id": "26cf6f23-c3a0-4d57-aae4-9b0010ee55cc",
                   "sync_policy": {
                       "groups": []
                   },
                   "enabled_features": [
                       "resharding"
                   ]

7. Check the sync status:

   Example

   [ceph: root@host01 /]# radosgw-admin sync status
             realm 26cf6f23-c3a0-4d57-aae4-9b0010ee55cc (usa)
         zonegroup 33a17718-6c77-493e-99fe-048d3110a06e (us)
              zone 505b48db-6de0-45d5-8208-8c98f7b1278d (us_east)
   zonegroup features enabled: resharding

   In this example. you can see that the resharding feature is enabled for the us zonegroup.

8. Optional: You can disable the resharding feature for the zonegroups:

   1. Disable the feature on all the zonegroups in the multi-site where Ceph Object Gateway is installed:

      Syntax

      radosgw-admin zonegroup modify --rgw-zonegroup=ZONEGROUP_NAME --disable-feature=resharding

      Example

      [ceph: root@host01 /]# radosgw-admin zonegroup modify --rgw-zonegroup=us --disable-feature=resharding

   2. Update the period and commit:

      Example

      [ceph: root@host01 /]# radosgw-admin period update --commit

Procedure

1. Back up the original bucket index:

   Syntax

   radosgw-admin bi list --bucket=BUCKET > BUCKET.list.backup

   Example

   [ceph: root@host01 /]# radosgw-admin bi list --bucket=data > data.list.backup

2. Reshard the bucket index:

   Syntax

   radosgw-admin bucket reshard --bucket=BUCKET --num-shards=NUMBER

   Example

   [ceph: root@host01 /]# radosgw-admin bucket reshard --bucket=data --num-shards=100

Procedure

1. List stale instances:

   [ceph: root@host01 /]# radosgw-admin reshard stale-instances list

2. Clean the stale instances of the bucket entries:

   [ceph: root@host01 /]# radosgw-admin reshard stale-instances rm

Procedure

1. Rename a user:

   Syntax

   radosgw-admin user rename --uid=CURRENT_USER_NAME --new-uid=NEW_USER_NAME

   Example

   [ceph: root@host01 /]# radosgw-admin user rename --uid=user1 --new-uid=user2
   
   {
       "user_id": "user2",
       "display_name": "user 2",
       "email": "",
       "suspended": 0,
       "max_buckets": 1000,
       "auid": 0,
       "subusers": [],
       "keys": [
           {
               "user": "user2",
               "access_key": "59EKHI6AI9F8WOW8JQZJ",
               "secret_key": "XH0uY3rKCUcuL73X0ftjXbZqUbk0cavD11rD8MsA"
           }
       ],
       "swift_keys": [],
       "caps": [],
       "op_mask": "read, write, delete",
       "default_placement": "",
       "placement_tags": [],
       "bucket_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "user_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "temp_url_keys": [],
       "type": "rgw"
   }

   If a user is inside a tenant, specify both the user name and the tenant:

   Syntax

   radosgw-admin user rename --uid USER_NAME --new-uid NEW_USER_NAME --tenant TENANT

   Example

   [ceph: root@host01 /]# radosgw-admin user rename --uid=test$user1 --new-uid=test$user2 --tenant test
   
   1000 objects processed in tvtester1. Next marker 80_tVtester1_99
   2000 objects processed in tvtester1. Next marker 64_tVtester1_44
   3000 objects processed in tvtester1. Next marker 48_tVtester1_28
   4000 objects processed in tvtester1. Next marker 2_tVtester1_74
   5000 objects processed in tvtester1. Next marker 14_tVtester1_53
   6000 objects processed in tvtester1. Next marker 87_tVtester1_61
   7000 objects processed in tvtester1. Next marker 6_tVtester1_57
   8000 objects processed in tvtester1. Next marker 52_tVtester1_91
   9000 objects processed in tvtester1. Next marker 34_tVtester1_74
   9900 objects processed in tvtester1. Next marker 9_tVtester1_95
   1000 objects processed in tvtester2. Next marker 82_tVtester2_93
   2000 objects processed in tvtester2. Next marker 64_tVtester2_9
   3000 objects processed in tvtester2. Next marker 48_tVtester2_22
   4000 objects processed in tvtester2. Next marker 32_tVtester2_42
   5000 objects processed in tvtester2. Next marker 16_tVtester2_36
   6000 objects processed in tvtester2. Next marker 89_tVtester2_46
   7000 objects processed in tvtester2. Next marker 70_tVtester2_78
   8000 objects processed in tvtester2. Next marker 51_tVtester2_41
   9000 objects processed in tvtester2. Next marker 33_tVtester2_32
   9900 objects processed in tvtester2. Next marker 9_tVtester2_83
   {
       "user_id": "test$user2",
       "display_name": "User 2",
       "email": "",
       "suspended": 0,
       "max_buckets": 1000,
       "auid": 0,
       "subusers": [],
       "keys": [
           {
               "user": "test$user2",
               "access_key": "user2",
               "secret_key": "123456789"
           }
       ],
       "swift_keys": [],
       "caps": [],
       "op_mask": "read, write, delete",
       "default_placement": "",
       "placement_tags": [],
       "bucket_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "user_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "temp_url_keys": [],
       "type": "rgw"
   }

2. Verify that the user has been renamed successfully:

   Syntax

   radosgw-admin user info --uid=NEW_USER_NAME

   Example

   [ceph: root@host01 /]# radosgw-admin user info --uid=user2

   If a user is inside a tenant, use the TENANT$USER_NAME format:

   Syntax

   radosgw-admin user info --uid= TENANT$USER_NAME

   Example

   [ceph: root@host01 /]# radosgw-admin user info --uid=test$user2

1. Find the access key for the specific user:

   Example

   [root@host01 ~]# radosgw-admin user info --uid=johndoe

   The access key is the "access_key" value in the output:

   Example

   [root@host01 ~]# radosgw-admin user info --uid=johndoe
   {
       "user_id": "johndoe",
       ...
       "keys": [
           {
               "user": "johndoe",
               "access_key": "0555b35654ad1656d804",
               "secret_key": "h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q=="
           }
       ],
       ...
   }

2. Specify the user ID and the access key from the previous step to remove the access key:

   Syntax

   radosgw-admin key rm --uid=USER_ID --access-key ACCESS_KEY

   Example

   [root@host01 ~]# radosgw-admin key rm --uid=johndoe --access-key 0555b35654ad1656d804

Procedure

1. Delete the policy attached to the role:

   Syntax

   radosgw-admin role policy delete --role-name=ROLE_NAME --policy-name=POLICY_NAME

   Example

   [root@host01 ~]# radosgw-admin role policy delete --role-name=S3Access1 --policy-name=Policy1

2. Delete the role:

   Syntax

   radosgw-admin role delete --role-name=ROLE_NAME

   Example

   [root@host01 ~]# radosgw-admin role delete --role-name=S3Access1

Procedure

1. List the buckets:

   Example

   [ceph: root@host01 /]# radosgw-admin bucket list
   [
       "34150b2e9174475db8e191c188e920f6/swcontainer",
       "s3bucket1",
       "34150b2e9174475db8e191c188e920f6/swimpfalse",
       "c278edd68cfb4705bb3e07837c7ad1a8/ec2container",
       "c278edd68cfb4705bb3e07837c7ad1a8/demoten1",
       "c278edd68cfb4705bb3e07837c7ad1a8/demo-ct",
       "c278edd68cfb4705bb3e07837c7ad1a8/demopostup",
       "34150b2e9174475db8e191c188e920f6/postimpfalse",
       "c278edd68cfb4705bb3e07837c7ad1a8/demoten2",
       "c278edd68cfb4705bb3e07837c7ad1a8/postupsw"
   ]

2. Rename the bucket:

   Syntax

   radosgw-admin bucket link --bucket=ORIGINAL_NAME --bucket-new-name=NEW_NAME --uid=USER_ID

   Example

   [ceph: root@host01 /]# radosgw-admin bucket link --bucket=s3bucket1 --bucket-new-name=s3newb --uid=testuser

   If the bucket is inside a tenant, specify the tenant as well:

   Syntax

   radosgw-admin bucket link --bucket=tenant/ORIGINAL_NAME --bucket-new-name=NEW_NAME --uid=TENANT$USER_ID

   Example

   [ceph: root@host01 /]# radosgw-admin bucket link --bucket=test/s3bucket1 --bucket-new-name=s3newb --uid=test$testuser

3. Verify the bucket was renamed:

   Example

   [ceph: root@host01 /]# radosgw-admin bucket list
   [
       "34150b2e9174475db8e191c188e920f6/swcontainer",
       "34150b2e9174475db8e191c188e920f6/swimpfalse",
       "c278edd68cfb4705bb3e07837c7ad1a8/ec2container",
       "s3newb",
       "c278edd68cfb4705bb3e07837c7ad1a8/demoten1",
       "c278edd68cfb4705bb3e07837c7ad1a8/demo-ct",
       "c278edd68cfb4705bb3e07837c7ad1a8/demopostup",
       "34150b2e9174475db8e191c188e920f6/postimpfalse",
       "c278edd68cfb4705bb3e07837c7ad1a8/demoten2",
       "c278edd68cfb4705bb3e07837c7ad1a8/postupsw"
   ]

Procedure

1. Generate a list of objects that hold data within a bucket.

   Syntax

   radosgw-admin bucket radoslist --bucket BUCKET_NAME

   Example

   [root@host01 ~]# radosgw-admin bucket radoslist --bucket mybucket

   Note

   If the BUCKET_NAME is omitted, then all objects in all buckets are displayed.

2. Check the version of rgw-orphan-list.

   Example

   [root@host01 ~]# head /usr/bin/rgw-orphan-list

   The version should be 2023-01-11 or newer.

3. Create a directory where you need to generate the list of orphans.

   Example

   [root@host01 ~]# mkdir orphans

4. Navigate to the directory created earlier.

   Example

   [root@host01 ~]# cd orphans

5. From the pool list, select the pool in which you want to find orphans. This script might run for a long time depending on the objects in the cluster.

   Example

   [root@host01 orphans]# rgw-orphan-list

   Example

   Available pools:
       .rgw.root
       default.rgw.control
       default.rgw.meta
       default.rgw.log
       default.rgw.buckets.index
       default.rgw.buckets.data
       rbd
       default.rgw.buckets.non-ec
       ma.rgw.control
       ma.rgw.meta
       ma.rgw.log
       ma.rgw.buckets.index
       ma.rgw.buckets.data
       ma.rgw.buckets.non-ec
   Which pool do you want to search for orphans?

   Enter the pool name to search for orphans.

   Important

   A data pool must be specified when using the rgw-orphan-list command, and not a metadata pool.

6. View the details of the rgw-orphan-list tool usage. `

   Synatx

   rgw-orphan-list -h
   rgw-orphan-list POOL_NAME /DIRECTORY

   Example

   [root@host01 orphans]# rgw-orphan-list default.rgw.buckets.data /orphans
   
   2023-09-12 08:41:14 ceph-host01 Computing delta...
   2023-09-12 08:41:14 ceph-host01 Computing results...
   10 potential orphans found out of a possible 2412 (0%).         <<<<<<< orphans detected
   The results can be found in './orphan-list-20230912124113.out'.
       Intermediate files are './rados-20230912124113.intermediate' and './radosgw-admin-20230912124113.intermediate'.
   ***
   *** WARNING: This is EXPERIMENTAL code and the results should be used
   ***          only with CAUTION!
   ***
   Done at 2023-09-12 08:41:14.

7. Run the ls -l command to verify the files ending with error should be zero length indicating the script ran without any issues.

   Example

   [root@host01 orphans]# ls -l
   
   -rw-r--r--. 1 root root    770 Sep 12 03:59 orphan-list-20230912075939.out
   -rw-r--r--. 1 root root      0 Sep 12 03:59 rados-20230912075939.error
   -rw-r--r--. 1 root root 248508 Sep 12 03:59 rados-20230912075939.intermediate
   -rw-r--r--. 1 root root      0 Sep 12 03:59 rados-20230912075939.issues
   -rw-r--r--. 1 root root      0 Sep 12 03:59 radosgw-admin-20230912075939.error
   -rw-r--r--. 1 root root 247738 Sep 12 03:59 radosgw-admin-20230912075939.intermediate

8. Review the orphan objects listed.

   Example

   [root@host01 orphans]# cat ./orphan-list-20230912124113.out
   
   a9c042bc-be24-412c-9052-dda6b2f01f55.16749.1_key1.cherylf.433-bucky-4865-0.0
   a9c042bc-be24-412c-9052-dda6b2f01f55.16749.1_key1.cherylf.433-bucky-4865-0.1
   a9c042bc-be24-412c-9052-dda6b2f01f55.16749.1_key1.cherylf.433-bucky-4865-0.2
   a9c042bc-be24-412c-9052-dda6b2f01f55.16749.1_key1.cherylf.433-bucky-4865-0.3
   a9c042bc-be24-412c-9052-dda6b2f01f55.16749.1_key1.cherylf.433-bucky-4865-0.4
   a9c042bc-be24-412c-9052-dda6b2f01f55.16749.1_key1.cherylf.433-bucky-4865-0.5
   a9c042bc-be24-412c-9052-dda6b2f01f55.16749.1_key1.cherylf.433-bucky-4865-0.6
   a9c042bc-be24-412c-9052-dda6b2f01f55.16749.1_key1.cherylf.433-bucky-4865-0.7
   a9c042bc-be24-412c-9052-dda6b2f01f55.16749.1_key1.cherylf.433-bucky-4865-0.8
   a9c042bc-be24-412c-9052-dda6b2f01f55.16749.1_key1.cherylf.433-bucky-4865-0.9

9. Remove orphan objects:

   Syntax

   rados -p POOL_NAME rm OBJECT_NAME

   Example

   [root@host01 orphans]# rados -p default.rgw.buckets.data rm myobject

   Warning

   Verify you are removing the correct objects. Running the rados rm command removes data from the storage cluster.

Procedure

1. Check the bucket index of a specific bucket:

   Syntax

   radosgw-admin bucket check --bucket=BUCKET_NAME

   Example

   [root@rgw ~]# radosgw-admin bucket check --bucket=mybucket

2. Fix the inconsistencies in the bucket index, including removal of orphaned objects:

   Syntax

   radosgw-admin bucket check --fix --bucket=BUCKET_NAME

   Example

   [root@rgw ~]# radosgw-admin bucket check --fix --bucket=mybucket

Using the boto script

1. Install the python3-boto3 package:

   Example

   [user@client ~]$  dnf install python3-boto3

2. Create an S3 bucket.

3. Create a python script topic.py to create an SNS topic for http,amqp, or kafka protocol:

   Example

   import boto3
   from botocore.client import Config
   import sys
   
   # endpoint and keys from vstart
   endpoint = 'http://127.0.0.1:8000'
   access_key='0555b35654ad1656d804'
   secret_key='h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q=='
   
   client = boto3.client('sns',
       	endpoint_url=endpoint,
       	aws_access_key_id=access_key,
       	aws_secret_access_key=secret_key,
       	config=Config(signature_version='s3'))
   
   attributes = {"push-endpoint": "amqp://localhost:5672", "amqp-exchange": "ex1", "amqp-ack-level": "broker"}
   
   client.create_topic(topic_name="mytopic", Attributes=attributes)

4. Run the python script for creating topic:

   Example

   python3 topic.py

5. Create a python script notification.py to create S3 bucket notification for s3:objectCreate and s3:objectRemove events:

   Example

   import boto3
   import sys
   
   # bucket name as first argument
   bucketname = sys.argv[1]
   # topic ARN as second argument
   topic_arn = sys.argv[2]
   # notification id as third argument
   notification_id = sys.argv[3]
   
   # endpoint and keys from vstart
   endpoint = 'http://127.0.0.1:8000'
   access_key='0555b35654ad1656d804'
   secret_key='h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q=='
   
   client = boto3.client('s3',
           endpoint_url=endpoint,
           aws_access_key_id=access_key,
           aws_secret_access_key=secret_key)
   
   # regex filter on the object name and metadata based filtering are extension to AWS S3 API
   # bucket and topic should be created beforehand
   
   topic_conf_list = [{'Id': notification_id,
                       'TopicArn': topic_arn,
                       'Events': ['s3:ObjectCreated:*', 's3:ObjectRemoved:*'],
                       }]
   
   client.put_bucket_notification_configuration(
      Bucket=bucketname,
      NotificationConfiguration={
          'TopicConfigurations': [
              {
                  'Id': notification_name,
                  'TopicArn': topic_arn,
                  'Events': ['s3:ObjectCreated:*', 's3:ObjectRemoved:*']
              }]})

6. Run the python script for creating the bucket notification:

   Example

   python3 notification.py

7. Create S3 objects in the bucket.

8. Fetch the notification configuration:

   Example

   endpoint = 'http://127.0.0.1:8000'
   access_key='0555b35654ad1656d804'
   secret_key='h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q=='
   
   client = boto3.client('s3',
           endpoint_url=endpoint,
           aws_access_key_id=access_key,
           aws_secret_access_key=secret_key)
   
   # getting a specific notification configuration is an extension to AWS S3 API
   
   print(client.get_bucket_notification_configuration(Bucket=bucketname))

9. Optional: Delete the objects.

   1. Verify the object deletion events at the http, rabbitmq, or kafka receiver.

Using thr AWS CLI

1. Create topic:

   Syntax

   aws --endpoint=AWS_END_POINT sns create-topic --name NAME --attributes=ATTRIBUTES_FILE

   Example

   [user@client ~]$ aws --endpoint=http://localhost sns create-topic --name test-kafka --attributes=file://topic.json
   
   sample topic.json:
   {"push-endpoint": "kafka://localhost","verify-ssl": "False", "kafka-ack-level": "broker", "persistent":"true"}
   ref: https://docs.aws.amazon.com/cli/latest/reference/sns/create-topic.html

2. Create the bucket notification:

   Syntax

   aws s3api put-bucket-notification-configuration --bucket BUCKET_NAME --notification-configuration NOTIFICATION_FILE

   Example

   [user@client ~]$ aws s3api put-bucket-notification-configuration --bucket my-bucket --notification-configuration file://notification.json
   
   sample notification.json
   {
       "TopicConfigurations": [
           {
               "Id": "test_notification",
               "TopicArn": "arn:aws:sns:us-west-2:123456789012:test-kafka",
               "Events": [
                   "s3:ObjectCreated:*"
               ]
           }
       ]
   }

3. Fetch the notification configuration:

   Syntax

   aws s3api --endpoint=AWS_ENDPOINT get-bucket-notification-configuration --bucket BUCKET_NAME

   Example

   [user@client ~]$ aws s3api --endpoint=http://localhost get-bucket-notification-configuration --bucket my-bucket
   {
       "TopicConfigurations": [
           {
               "Id": "test_notification",
               "TopicArn": "arn:aws:sns:default::test-kafka",
               "Events": [
                   "s3:ObjectCreated:*"
               ]
           }
       ]
   }

Procedure

1. Create a JSON file for lifecycle configuration:

   Example

   [user@client ~]$ vi lifecycle.json

2. Add the specific lifecycle configuration rules in the file:

   Example

   {
   	"Rules": [
           {
   		    "Filter": {
   			    "Prefix": "images/"
   		    },
   		    "Status": "Enabled",
   		    "Expiration": {
   			    "Days": 1
   		    },
   		    "ID": "ImageExpiration"
   	    }
       ]
   }

   The lifecycle configuration example expires objects in the images directory that are more than 1 day old.

3. Set the lifecycle configuration on the bucket:

   Syntax

   aws --endpoint-url=RADOSGW_ENDPOINT_URL:PORT s3api put-bucket-lifecycle-configuration --bucket BUCKET_NAME --lifecycle-configuration file://PATH_TO_LIFECYCLE_CONFIGURATION_FILE/LIFECYCLE_CONFIGURATION_FILE.json

   Example

   [user@client ~]$ aws --endpoint-url=http://host01:80 s3api put-bucket-lifecycle-configuration --bucket testbucket --lifecycle-configuration file://lifecycle.json

   In this example, the lifecycle.json file exists in the current directory.

Procedure

1. Create a JSON file for the lifecycle configuration:

   Example

   [user@client ~]$ vi lifecycle.json

2. Add the specific lifecycle configuration rules to the file:

   Example

   {
   	"Rules": [
           {
   		    "Filter": {
   			    "Prefix": "images/"
   		    },
   		    "Status": "Enabled",
   		    "Expiration": {
   			    "Days": 1
   		    },
   		    "ID": "ImageExpiration"
   	    },
   		{
   			"Filter": {
   				"Prefix": "docs/"
   			},
   			"Status": "Enabled",
   			"Expiration": {
   				"Days": 30
   			},
   			"ID": "DocsExpiration"
   		}
   	]
   }

3. Update the lifecycle configuration on the bucket:

   Syntax

   aws --endpoint-url=RADOSGW_ENDPOINT_URL:PORT s3api put-bucket-lifecycle-configuration --bucket BUCKET_NAME --lifecycle-configuration file://PATH_TO_LIFECYCLE_CONFIGURATION_FILE/LIFECYCLE_CONFIGURATION_FILE.json

   Example

   [user@client ~]$ aws --endpoint-url=http://host01:80 s3api put-bucket-lifecycle-configuration --bucket testbucket --lifecycle-configuration file://lifecycle.json

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. List bucket lifecycle progress:

   Example

   [ceph: root@host01 /]# radosgw-admin lc list
   
   [
      {
            “bucket”: “:testbucket:8b63d584-9ea1-4cf3-8443-a6a15beca943.54187.1”,
            “started”: “Thu, 01 Jan 1970 00:00:00 GMT”,
            “status” : “UNINITIAL”
      },
      {
            “bucket”: “:testbucket1:8b635499-9e41-4cf3-8443-a6a15345943.54187.2”,
            “started”: “Thu, 01 Jan 1970 00:00:00 GMT”,
            “status” : “UNINITIAL”
      }
   ]

   The bucket lifecycle processing status can be one of the following:

   • UNINITIAL - The process has not run yet.
   • PROCESSING - The process is currently running.
   • COMPLETE - The process has completed.

3. Optional: You can manually process bucket lifecycle policies:

   1. Process the lifecycle policy for a single bucket:

      Syntax

      radosgw-admin lc process --bucket=BUCKET_NAME

      Example

      [ceph: root@host01 /]# radosgw-admin lc process --bucket=testbucket1

   2. Process all bucket lifecycle policies immediately:

      Example

      [ceph: root@host01 /]# radosgw-admin lc process

Procedure

1. Log into the Cephadm shell:

   Example

   [root@host01 ~]# cephadm shell

2. Set the lifecycle expiration time:

   Syntax

   ceph config set client.rgw rgw_lifecycle_work_time %D:%D-%D:%D

   Replace %d:%d-%d:%d with start_hour:start_minute-end_hour:end_minute.

   Example

   [ceph: root@host01 /]# ceph config set client.rgw rgw_lifecycle_work_time 06:00-08:00

Procedure

1. Create a new data pool:

   Syntax

   ceph osd pool create POOL_NAME

   Example

   [ceph: root@host01 /]# ceph osd pool create test.hot.data

2. Add a new storage class:

   Syntax

   radosgw-admin zonegroup placement add  --rgw-zonegroup default --placement-id PLACEMENT_TARGET --storage-class STORAGE_CLASS

   Example

   [ceph: root@host01 /]# radosgw-admin zonegroup placement add  --rgw-zonegroup default --placement-id default-placement --storage-class hot.test
   {
           "key": "default-placement",
           "val": {
               "name": "default-placement",
               "tags": [],
               "storage_classes": [
                   "STANDARD",
                   "hot.test"
               ]
           }
       }

3. Provide the zone placement information for the new storage class:

   Syntax

   radosgw-admin zone placement add --rgw-zone default --placement-id PLACEMENT_TARGET --storage-class STORAGE_CLASS --data-pool DATA_POOL

   Example

   [ceph: root@host01 /]# radosgw-admin zone placement add --rgw-zone default --placement-id default-placement --storage-class hot.test --data-pool test.hot.data
   {
              "key": "default-placement",
              "val": {
                  "index_pool": "test_zone.rgw.buckets.index",
                  "storage_classes": {
                      "STANDARD": {
                          "data_pool": "test.hot.data"
                      },
                      "hot.test": {
                          "data_pool": "test.hot.data",
                     }
                  },
                  "data_extra_pool": "",
                  "index_type": 0
              }

   Note

   Consider setting the compression_type when creating cold or archival data storage pools with write once.

4. Enable the rgw application on the data pool:

   Syntax

   ceph osd pool application enable POOL_NAME rgw

   Example

   [ceph: root@host01 /]# ceph osd pool application enable test.hot.data rgw
   enabled application 'rgw' on pool 'test.hot.data'

5. Restart all the rgw daemons.

6. Create a bucket:

   Example

   [ceph: root@host01 /]# aws s3api create-bucket --bucket testbucket10 --create-bucket-configuration LocationConstraint=default:default-placement --endpoint-url http://10.0.0.80:8080

7. Add the object:

   Example

   [ceph: root@host01 /]# aws --endpoint=http://10.0.0.80:8080 s3api put-object --bucket testbucket10  --key compliance-upload --body /root/test2.txt

8. Create a second data pool:

   Syntax

   ceph osd pool create POOL_NAME

   Example

   [ceph: root@host01 /]# ceph osd pool create test.cold.data

9. Add a new storage class:

   Syntax

   radosgw-admin zonegroup placement add  --rgw-zonegroup default --placement-id PLACEMENT_TARGET --storage-class STORAGE_CLASS

   Example

   [ceph: root@host01 /]# radosgw-admin zonegroup placement add  --rgw-zonegroup default --placement-id default-placement --storage-class cold.test
   {
           "key": "default-placement",
           "val": {
               "name": "default-placement",
               "tags": [],
               "storage_classes": [
                   "STANDARD",
                   "cold.test"
               ]
           }
       }

10. Provide the zone placement information for the new storage class:

    Syntax

    radosgw-admin zone placement add --rgw-zone default --placement-id PLACEMENT_TARGET --storage-class STORAGE_CLASS --data-pool DATA_POOL

    Example

    [ceph: root@host01 /]# radosgw-admin zone placement add --rgw-zone default --placement-id default-placement --storage-class cold.test --data-pool test.cold.data

11. Enable rgw application on the data pool:

    Syntax

    ceph osd pool application enable POOL_NAME rgw

    Example

    [ceph: root@host01 /]# ceph osd pool application enable test.cold.data rgw
    enabled application 'rgw' on pool 'test.cold.data'

12. Restart all the rgw daemons.

13. To view the zone group configuration, run the following command:

    Syntax

    radosgw-admin zonegroup get
    {
        "id": "3019de59-ddde-4c5c-b532-7cdd29de09a1",
        "name": "default",
        "api_name": "default",
        "is_master": "true",
        "endpoints": [],
        "hostnames": [],
        "hostnames_s3website": [],
        "master_zone": "adacbe1b-02b4-41b8-b11d-0d505b442ed4",
        "zones": [
            {
                "id": "adacbe1b-02b4-41b8-b11d-0d505b442ed4",
                "name": "default",
                "endpoints": [],
                "log_meta": "false",
                "log_data": "false",
                "bucket_index_max_shards": 11,
                "read_only": "false",
                "tier_type": "",
                "sync_from_all": "true",
                "sync_from": [],
                "redirect_zone": ""
            }
        ],
        "placement_targets": [
            {
                "name": "default-placement",
                "tags": [],
                "storage_classes": [
                    "hot.test",
                    "cold.test",
                    "STANDARD"
                ]
            }
        ],
        "default_placement": "default-placement",
        "realm_id": "",
        "sync_policy": {
            "groups": []
        }
    }

14. To view the zone configuration, run the following command:

    Syntax

    radosgw-admin zone get
    {
        "id": "adacbe1b-02b4-41b8-b11d-0d505b442ed4",
        "name": "default",
        "domain_root": "default.rgw.meta:root",
        "control_pool": "default.rgw.control",
        "gc_pool": "default.rgw.log:gc",
        "lc_pool": "default.rgw.log:lc",
        "log_pool": "default.rgw.log",
        "intent_log_pool": "default.rgw.log:intent",
        "usage_log_pool": "default.rgw.log:usage",
        "roles_pool": "default.rgw.meta:roles",
        "reshard_pool": "default.rgw.log:reshard",
        "user_keys_pool": "default.rgw.meta:users.keys",
        "user_email_pool": "default.rgw.meta:users.email",
        "user_swift_pool": "default.rgw.meta:users.swift",
        "user_uid_pool": "default.rgw.meta:users.uid",
        "otp_pool": "default.rgw.otp",
        "system_key": {
            "access_key": "",
            "secret_key": ""
        },
        "placement_pools": [
            {
                "key": "default-placement",
                "val": {
                    "index_pool": "default.rgw.buckets.index",
                    "storage_classes": {
                        "cold.test": {
                            "data_pool": "test.cold.data"
                        },
                        "hot.test": {
                            "data_pool": "test.hot.data"
                        },
                        "STANDARD": {
                            "data_pool": "default.rgw.buckets.data"
                        }
                    },
                    "data_extra_pool": "default.rgw.buckets.non-ec",
                    "index_type": 0
                }
            }
        ],
        "realm_id": "",
        "notif_pool": "default.rgw.log:notif"
    }

15. Create a bucket:

    Example

    [ceph: root@host01 /]# aws s3api create-bucket --bucket testbucket10 --create-bucket-configuration LocationConstraint=default:default-placement --endpoint-url http://10.0.0.80:8080

16. List the objects prior to transition:

    Example

    [ceph: root@host01 /]# radosgw-admin bucket list --bucket testbucket10
    
            {
                "ETag": "\"211599863395c832a3dfcba92c6a3b90\"",
                "Size": 540,
                "StorageClass": "STANDARD",
                "Key": "obj1",
                "VersionId": "W95teRsXPSJI4YWJwwSG30KxSCzSgk-",
                "IsLatest": true,
                "LastModified": "2023-11-23T10:38:07.214Z",
                "Owner": {
                    "DisplayName": "test-user",
                    "ID": "test-user"
                }
            }

17. Create a JSON file for lifecycle configuration:

    Example

    [ceph: root@host01 /]# vi lifecycle.json

18. Add the specific lifecycle configuration rule in the file:

    Example

    {
        "Rules": [
            {
                "Filter": {
                    "Prefix": ""
                },
                "Status": "Enabled",
                "Transitions": [
                    {
                        "Days": 5,
                        "StorageClass": "hot.test"
                    },
     {
                        "Days": 20,
                        "StorageClass": "cold.test"
                    }
                ],
                "Expiration": {
                    "Days": 365
                },
                "ID": "double transition and expiration"
            }
        ]
    }

    The lifecycle configuration example shows an object that will transition from the default STANDARD storage class to the hot.test storage class after 5 days, again transitions after 20 days to the cold.test storage class, and finally expires after 365 days in the cold.test storage class.

19. Set the lifecycle configuration on the bucket:

    Example

    [ceph: root@host01 /]# aws s3api put-bucket-lifecycle-configuration --bucket testbucket20 --lifecycle-configuration file://lifecycle.json

20. Retrieve the lifecycle configuration on the bucket:

    Example

    [ceph: root@host01 /]# aws s3api get-bucket-lifecycle-configuration --bucket testbucket20
    {
        "Rules": [
            {
                "Expiration": {
                    "Days": 365
                },
                "ID": "double transition and expiration",
                "Prefix": "",
                "Status": "Enabled",
                "Transitions": [
                    {
                        "Days": 20,
                        "StorageClass": "cold.test"
                    },
                    {
                        "Days": 5,
                        "StorageClass": "hot.test"
                    }
                ]
            }
        ]
    }

21. Verify that the object is transitioned to the given storage class:

    Example

    [ceph: root@host01 /]# radosgw-admin bucket list --bucket testbucket10
    
            {
                "ETag": "\"211599863395c832a3dfcba92c6a3b90\"",
                "Size": 540,
                "StorageClass": "cold.test",
                "Key": "obj1",
                "VersionId": "W95teRsXPSJI4YWJwwSG30KxSCzSgk-",
                "IsLatest": true,
                "LastModified": "2023-11-23T10:38:07.214Z",
                "Owner": {
                    "DisplayName": "test-user",
                    "ID": "test-user"
                }
            }

Procedure

1. Create a bucket with object lock enabled:

   Syntax

   aws --endpoint=http://RGW_PORT:8080 s3api create-bucket --bucket BUCKET_NAME --object-lock-enabled-for-bucket

   Example

   [root@rgw-2 ~]# aws --endpoint=http://rgw.ceph.com:8080 s3api create-bucket --bucket worm-bucket --object-lock-enabled-for-bucket

2. Set a retention period for the bucket:

   Syntax

   aws --endpoint=http://RGW_PORT:8080 s3api put-object-lock-configuration --bucket BUCKET_NAME --object-lock-configuration '{ "ObjectLockEnabled": "Enabled", "Rule": { "DefaultRetention": { "Mode": "RETENTION_MODE", "Days": NUMBER_OF_DAYS }}}'

   Example

   [root@rgw-2 ~]# aws --endpoint=http://rgw.ceph.com:8080 s3api put-object-lock-configuration --bucket worm-bucket --object-lock-configuration '{ "ObjectLockEnabled": "Enabled", "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE", "Days": 10 }}}'

   Note

   You can choose either the GOVERNANCE or COMPLIANCE mode for the RETENTION_MODE in S3 object lock, to apply different levels of protection to any object version that is protected by object lock.

   In GOVERNANCE mode, users cannot overwrite or delete an object version or alter its lock settings unless they have special permissions.

   In COMPLIANCE mode, a protected object version cannot be overwritten or deleted by any user, including the root user in your AWS account. When an object is locked in COMPLIANCE mode, its RETENTION_MODE cannot be changed, and its retention period cannot be shortened. COMPLIANCE mode helps ensure that an object version cannot be overwritten or deleted for the duration of the period.

3. Put the object into the bucket with a retention time set:

   Syntax

   aws --endpoint=http://RGW_PORT:8080 s3api put-object --bucket BUCKET_NAME --object-lock-mode RETENTION_MODE --object-lock-retain-until-date "DATE" --key compliance-upload --body TEST_FILE

   Example

   [root@rgw-2 ~]# aws --endpoint=http://rgw.ceph.com:8080 s3api put-object --bucket worm-bucket --object-lock-mode COMPLIANCE --object-lock-retain-until-date "2022-05-31" --key compliance-upload --body test.dd
   {
       "ETag": "\"d560ea5652951637ba9c594d8e6ea8c1\"",
       "VersionId": "Nhhk5kRS6Yp6dZXVWpZZdRcpSpBKToD"
   }

4. Upload a new object using the same key:

   Syntax

   aws --endpoint=http://RGW_PORT:8080 s3api put-object --bucket BUCKET_NAME --object-lock-mode RETENTION_MODE --object-lock-retain-until-date "DATE" --key compliance-upload --body PATH

   Example

   [root@rgw-2 ~]# aws --endpoint=http://rgw.ceph.com:8080 s3api put-object --bucket worm-bucket --object-lock-mode COMPLIANCE --object-lock-retain-until-date "2022-05-31" --key compliance-upload --body /etc/fstab
   {
       "ETag": "\"d560ea5652951637ba9c594d8e6ea8c1\"",
       "VersionId": "Nhhk5kRS6Yp6dZXVWpZZdRcpSpBKToD"
   }

Procedure

1. Set the value of rgw_gc_max_concurrent_io to 20, and the value of rgw_gc_max_trim_chunk to 64:

   Example

   [ceph: root@host01 /]# ceph config set client.rgw rgw_gc_max_concurrent_io 20
   [ceph: root@host01 /]# ceph config set client.rgw rgw_gc_max_trim_chunk 64

2. Restart the Ceph Object Gateway to allow the changed settings to take effect.
3. Monitor the storage cluster during GC activity to verify that the increased values do not adversely affect performance.

Procedure

1. To increase the number of threads to run in parallel, set the value of rgw_lc_max_worker to a value between 3 and 9:

   Example

   [ceph: root@host01 /]# ceph config set client.rgw rgw_lc_max_worker 7

2. To increase the number of threads in each thread’s work pool, set the value of rgw_lc_max_wp_worker to a value between 3 and 9:

   Example

   [ceph: root@host01 /]# ceph config set client.rgw rgw_lc_max_wp_worker 7

3. Restart the Ceph Object Gateway to allow the changed settings to take effect.
4. Monitor the storage cluster to verify that the increased values do not adversely affect performance.