Chapter 5. Advanced configuration

1. Create a realm. Replace NAME with the realm name.

   Syntax

   radosgw-admin realm create --rgw-realm=NAME --default

2. Rename the default zone and zonegroup. Replace <name> with the zonegroup or zone name.

   Syntax

   radosgw-admin zonegroup rename --rgw-zonegroup default --zonegroup-new-name=NEW_ZONE_GROUP_NAME
   radosgw-admin zone rename --rgw-zone default --zone-new-name us-east-1 --rgw-zonegroup=ZONE_GROUP_NAME

3. Configure the primary zonegroup. Replace NAME with the realm or zonegroup name. Replace FQDN with the fully qualified domain name(s) in the zonegroup.

   Syntax

   radosgw-admin zonegroup modify --rgw-realm=REALM_NAME --rgw-zonegroup=ZONE_GROUP_NAME --endpoints http://FQDN:80 --master --default

4. Create a system user. Replace USER_ID with the username. Replace DISPLAY_NAME with a display name. It can contain spaces.

   Syntax

   radosgw-admin user create --uid=USER_ID \
                               --display-name="DISPLAY_NAME" \
                               --access-key=ACCESS_KEY --secret=SECRET_KEY \ --system

5. Configure the primary zone. Replace NAME with the realm, zonegroup, or zone name. Replace FQDN with the fully qualified domain name(s) in the zonegroup.

   Syntax

   radosgw-admin zone modify --rgw-realm=REALM_NAME --rgw-zonegroup=ZONE_GROUP_NAME \
                               --rgw-zone=ZONE_NAME --endpoints http://FQDN:80 \
                               --access-key=ACCESS_KEY --secret=SECRET_KEY \
                               --master --default

6. Optional: If you specified the realm and zone in the service specification during the deployment of the Ceph Object Gateway, update the spec section of the specification file:

   Syntax

   spec:
     rgw_realm: REALM_NAME
     rgw_zone: ZONE_NAME

7. Update the Ceph configuration database:

   Syntax

   ceph config set client.rgw.SERVICE_NAME rgw_realm REALM_NAME
   ceph config set client.rgw.SERVICE_NAME rgw_zonegroup ZONE_GROUP_NAME
   ceph config set client.rgw.SERVICE_NAME rgw_zone PRIMARY_ZONE_NAME

   Example

   [ceph: root@host01 /]# ceph config set client.rgw.rgwsvcid.mons-1.jwgwwp rgw_realm test_realm
   [ceph: root@host01 /]# ceph config set client.rgw.rgwsvcid.mons-1.jwgwwp rgw_zonegroup us
   [ceph: root@host01 /]# ceph config set client.rgw.rgwsvcid.mons-1.jwgwwp rgw_zone us-east-1

8. Commit the updated configuration:

   Example

   [ceph: root@host01 /]# radosgw-admin period update --commit

9. Restart the Ceph Object Gateway:

   Note

   Use the output from the ceph orch ps command, under the NAME column, to get the SERVICE_TYPE.ID information.

   1. To restart the Ceph Object Gateway on an individual node in the storage cluster:

      Syntax

      systemctl restart ceph-CLUSTER_ID@SERVICE_TYPE.ID.service

      Example

      [root@host01 ~]# systemctl restart ceph-c4b34c6f-8365-11ba-dc31-529020a7702d@rgw.realm.zone.host01.gwasto.service

   2. To restart the Ceph Object Gateways on all nodes in the storage cluster:

      Syntax

      ceph orch restart SERVICE_TYPE

      Example

      [ceph: root@host01 /]# ceph orch restart rgw

10. Establish the secondary zone. See the Establishing a secondary zone section.

Procedure

1. Log into the cephadm shell:

   Example

   [root@host04 ~]# cephadm shell

2. Pull the primary realm configuration from the host:

   Syntax

   radosgw-admin realm pull --url=URL_TO_PRIMARY_ZONE_GATEWAY --access-key=ACCESS_KEY --secret-key=SECRET_KEY

   Example

   [ceph: root@host04 /]# radosgw-admin realm pull --url=http://10.74.249.26:80 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ

3. Pull the primary period configuration from the host:

   Syntax

   radosgw-admin period pull --url=URL_TO_PRIMARY_ZONE_GATEWAY --access-key=ACCESS_KEY --secret-key=SECRET_KEY

   Example

   [ceph: root@host04 /]# radosgw-admin period pull --url=http://10.74.249.26:80 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ

4. Configure a secondary zone:

   Note

   All zones run in an active-active configuration by default; that is, a gateway client might write data to any zone and the zone will replicate the data to all other zones within the zone group. If the secondary zone should not accept write operations, specify the `--read-only flag to create an active-passive configuration between the master zone and the secondary zone. Additionally, provide the access_key and secret_key of the generated system user stored in the master zone of the master zone group.

   Syntax

   radosgw-admin zone create --rgw-zonegroup=_ZONE_GROUP_NAME_ \
                --rgw-zone=_SECONDARY_ZONE_NAME_ --endpoints=http://_RGW_SECONDARY_HOSTNAME_:_RGW_PRIMARY_PORT_NUMBER_1_ \
                --access-key=_SYSTEM_ACCESS_KEY_ --secret=_SYSTEM_SECRET_KEY_ \
                [--read-only]

   Example

   [ceph: root@host04 /]# radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-2 --endpoints=http://rgw2:80 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ

5. Optional: Delete the default zone:

   Important

   Do not delete the default zone and its pools if you are using the default zone and zone group to store data.

   Example

   [ceph: root@host04 /]# radosgw-admin zone rm --rgw-zone=default
   [ceph: root@host04 /]# ceph osd pool rm default.rgw.log default.rgw.log --yes-i-really-really-mean-it
   [ceph: root@host04 /]# ceph osd pool rm default.rgw.meta default.rgw.meta --yes-i-really-really-mean-it
   [ceph: root@host04 /]# ceph osd pool rm default.rgw.control default.rgw.control --yes-i-really-really-mean-it
   [ceph: root@host04 /]# ceph osd pool rm default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it
   [ceph: root@host04 /]# ceph osd pool rm default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it

6. Optional: If you specified the realm and zone in the service specification during the deployment of the Ceph Object Gateway, update the spec section of the specification file:

   Syntax

   spec:
     rgw_realm: REALM_NAME
     rgw_zone: ZONE_NAME

7. Update the Ceph configuration database:

   Syntax

   ceph config set client.rgw.SERVICE_NAME rgw_realm REALM_NAME
   ceph config set client.rgw.SERVICE_NAME rgw_zonegroup ZONE_GROUP_NAME
   ceph config set client.rgw.SERVICE_NAME rgw_zone SECONDARY_ZONE_NAME

   Example

   [ceph: root@host04 /]# ceph config set client.rgw.rgwsvcid.mons-1.jwgwwp rgw_realm test_realm
   [ceph: root@host04 /]# ceph config set client.rgw.rgwsvcid.mons-1.jwgwwp rgw_zonegroup us
   [ceph: root@host04 /]# ceph config set client.rgw.rgwsvcid.mons-1.jwgwwp rgw_zone us-east-2

8. Commit the changes:

   Syntax

   radosgw-admin period update --commit

   Example

   [ceph: root@host04 /]# radosgw-admin period update --commit

9. Outside the cephadm shell, fetch the FSID of the storage cluster and the processes:

   Example

   [root@host04 ~]#  systemctl list-units | grep ceph

10. Start the Ceph Object Gateway daemon:

    Syntax

    systemctl start ceph-FSID@DAEMON_NAME
    systemctl enable ceph-FSID@DAEMON_NAME

    Example

    [root@host04 ~]# systemctl start ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-2.host04.ahdtsw.service
    [root@host04 ~]# systemctl enable ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-2.host04.ahdtsw.service

Procedure

1. Make the secondary zone the primary and default zone. For example:

   Syntax

   radosgw-admin zone modify --rgw-zone=ZONE_NAME --master --default

   By default, Ceph Object Gateway runs in an active-active configuration. If the cluster was configured to run in an active-passive configuration, the secondary zone is a read-only zone. Remove the --read-only status to allow the zone to receive write operations. For example:

   Syntax

   radosgw-admin zone modify --rgw-zone=ZONE_NAME --master --default --read-only=false

2. Update the period to make the changes take effect:

   Example

   [ceph: root@host01 /]# radosgw-admin period update --commit

3. Restart the Ceph Object Gateway.

   Note

   Use the output from the ceph orch ps command, under the NAME column, to get the SERVICE_TYPE.ID information.

   1. To restart the Ceph Object Gateway on an individual node in the storage cluster:

      Syntax

      systemctl restart ceph-CLUSTER_ID@SERVICE_TYPE.ID.service

      Example

      [root@host01 ~]# systemctl restart ceph-c4b34c6f-8365-11ba-dc31-529020a7702d@rgw.realm.zone.host01.gwasto.service

   2. To restart the Ceph Object Gateways on all nodes in the storage cluster:

      Syntax

      ceph orch restart SERVICE_TYPE

      Example

      [ceph: root@host01 /]# ceph orch restart rgw

1. From the recovered zone, pull the realm from the current primary zone:

   Syntax

   radosgw-admin realm pull --url=URL_TO_PRIMARY_ZONE_GATEWAY \
                               --access-key=ACCESS_KEY --secret=SECRET_KEY

2. Make the recovered zone the primary and default zone:

   Syntax

   radosgw-admin zone modify --rgw-zone=ZONE_NAME --master --default

3. Update the period to make the changes take effect:

   Example

   [ceph: root@host01 /]# radosgw-admin period update --commit

4. Restart the Ceph Object Gateway in the recovered zone:

   Syntax

   ceph orch restart SERVICE_TYPE

   Example

   [ceph: root@host01 /]# ceph orch restart rgw

5. If the secondary zone needs to be a read-only configuration, update the secondary zone:

   Syntax

   radosgw-admin zone modify --rgw-zone=ZONE_NAME --read-only
   radosgw-admin zone modify --rgw-zone=ZONE_NAME --read-only

6. Update the period to make the changes take effect:

   Example

   [ceph: root@host01 /]# radosgw-admin period update --commit

7. Restart the Ceph Object Gateway in the secondary zone:

   Syntax

   ceph orch restart SERVICE_TYPE

   Example

   [ceph: root@host01 /]# ceph orch restart rgw

Procedure

1. View the type of data log:

   Example

   [root@host01 ~]# radosgw-admin datalog status
       {
           "marker": "1_1657793517.559260_543389.1",
           "last_update": "2022-07-14 10:11:57.559260Z"
       },

   1_ in marker reflects OMAP data log type.

2. Change the data log type to FIFO:

   Note

   Configuration values are case-sensitive. Use fifo in lowercase to set configuration options.

   Note

   After upgrading from Red Hat Ceph Storage 4 to Red Hat Ceph Storage 5, change the default data log type to fifo.

   Example

   [root@host01 ~]# radosgw-admin --log-type fifo datalog type

3. Verify the changes:

   Example

   [root@host01 ~]# radosgw-admin datalog status
   	{
   	    "marker": "G00000000000000000001@00000000000000000037:00000000000003563105",
           "last_update": "2022-07-14T10:14:07.516629Z"
   	},

   : in marker reflects FIFO data log type.

Procedure

1. Create a new realm:

   Syntax

   radosgw-admin realm create --rgw-realm=REALM_NAME [--default]

   Example

   [ceph: root@host01 /]# radosgw-admin realm create --rgw-realm=test_realm --default
   {
       "id": "0956b174-fe14-4f97-8b50-bb7ec5e1cf62",
       "name": "test_realm",
       "current_period": "1950b710-3e63-4c41-a19e-46a715000980",
       "epoch": 1
   }

2. Create a new zone group:

   Syntax

   radosgw-admin zonegroup create --rgw-zonegroup=ZONE_GROUP_NAME --endpoints=FQDN:PORT --rgw-realm=REALM_NAME|--realm-id=REALM_ID --master --default

   Example

   [ceph: root@host01 /]# radosgw-admin zonegroup create --rgw-zonegroup=us --endpoints=http://rgw1:80 --rgw-realm=test_realm --master --default
   {
       "id": "f1a233f5-c354-4107-b36c-df66126475a6",
       "name": "us",
       "api_name": "us",
       "is_master": "true",
       "endpoints": [
           "http:\/\/rgw1:80"
       ],
       "hostnames": [],
       "hostnames_s3webzone": [],
       "master_zone": "",
       "zones": [],
       "placement_targets": [],
       "default_placement": "",
       "realm_id": "0956b174-fe14-4f97-8b50-bb7ec5e1cf62"
   }

3. Create one or more zones depending on the use case:

   Syntax

   radosgw-admin zone create --rgw-zonegroup=ZONE_GROUP_NAME --rgw-zone=ZONE_NAME --master --default --endpoints=FQDN:PORT,FQDN:PORT

   Example

   [ceph: root@host01 /]# radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east --master --default --endpoints=http://rgw1:80

4. Get the JSON file with the configuration of the zone group:

   Syntax

   radosgw-admin zonegroup get --rgw-zonegroup=ZONE_GROUP_NAME > JSON_FILE_NAME

   Example

   [ceph: root@host01 /]# radosgw-admin zonegroup get --rgw-zonegroup=us > zonegroup-us.json

   1. Open the file for editing, and set the log_meta, log_data, and sync_from_all fields to false:

      Example

          {
              "id": "72f3a886-4c70-420b-bc39-7687f072997d",
              "name": "default",
              "api_name": "",
              "is_master": "true",
              "endpoints": [],
              "hostnames": [],
              "hostnames_s3website": [],
              "master_zone": "a5e44ecd-7aae-4e39-b743-3a709acb60c5",
              "zones": [
                  {
                      "id": "975558e0-44d8-4866-a435-96d3e71041db",
                      "name": "testzone",
                      "endpoints": [],
                      "log_meta": "false",
                      "log_data": "false",
                      "bucket_index_max_shards": 11,
                      "read_only": "false",
                      "tier_type": "",
                      "sync_from_all": "false",
                      "sync_from": []
                  },
                  {
                      "id": "a5e44ecd-7aae-4e39-b743-3a709acb60c5",
                      "name": "default",
                      "endpoints": [],
                      "log_meta": "false",
                      "log_data": "false",
                      "bucket_index_max_shards": 11,
                      "read_only": "false",
                      "tier_type": "",
                      "sync_from_all": "false",
                      "sync_from": []
                  }
              ],
              "placement_targets": [
                  {
                      "name": "default-placement",
                      "tags": []
                  }
              ],
              "default_placement": "default-placement",
              "realm_id": "2d988e7d-917e-46e7-bb18-79350f6a5155"
          }

5. Use the updated JSON file to set the zone group:

   Syntax

   radosgw-admin zonegroup set --rgw-zonegroup=ZONE_GROUP_NAME --infile=JSON_FILE_NAME

   Example

   [ceph: root@host01 /]# radosgw-admin zonegroup set --rgw-zonegroup=us --infile=zonegroup-us.json

6. Update the period:

   Example

   [ceph: root@host01 /]# radosgw-admin period update --commit

7. Optional: If you specified the realm and zone in the service specification during the deployment of the Ceph Object Gateway, update the spec section of the specification file:

   Syntax

   spec:
     rgw_realm: REALM_NAME
     rgw_zone: ZONE_NAME

8. Validate if the two zones are successfully disabled:

   Example

   [root@ceph-ck-multi-pst82t-node5 ~]# radosgw-admin sync status
   
   realm 1e513df5-279b-4558-9dd0-3e50af411740 (india)
   zonegroup 09d320dd-d9f8-4fce-b951-8a59306a2d85 (south)
   zone a88defd9-84f4-4a4e-8b21-7f3fbc190005 (ka)
   current time 2024-03-15T05:01:18Z
   zonegroup features enabled: compress-encrypted,resharding
   metadata sync no sync (zone is master)
   data sync source: 7a1ad335-9e09-403a-879c-d29cd81e9c4d (tn)
   not syncing from zone

Procedure

1. Create one local realm on the first data center in the storage cluster:

   Syntax

   radosgw-admin realm create --rgw-realm=REALM_NAME --default

   Example

   [ceph: root@host01 /]# radosgw-admin realm create --rgw-realm=ldc1 --default

2. Create one local master zonegroup on the first data center:

   Syntax

   radosgw-admin zonegroup create --rgw-zonegroup=ZONE_GROUP_NAME --endpoints=http://RGW_NODE_NAME:80 --rgw-realm=REALM_NAME --master --default

   Example

   [ceph: root@host01 /]# radosgw-admin zonegroup create --rgw-zonegroup=ldc1zg --endpoints=http://rgw1:80 --rgw-realm=ldc1 --master --default

3. Create one local zone on the first data center:

   Syntax

   radosgw-admin zone create --rgw-zonegroup=ZONE_GROUP_NAME --rgw-zone=ZONE_NAME --master --default --endpoints=HTTP_FQDN[,HTTP_FQDN]

   Example

   [ceph: root@host01 /]# radosgw-admin zone create --rgw-zonegroup=ldc1zg --rgw-zone=ldc1z --master --default --endpoints=http://rgw.example.com

4. Commit the period:

   Example

   [ceph: root@host01 /]# radosgw-admin period update --commit

5. Optional: If you specified the realm and zone in the service specification during the deployment of the Ceph Object Gateway, update the spec section of the specification file:

   Syntax

   spec:
     rgw_realm: REALM_NAME
     rgw_zone: ZONE_NAME

6. You can either deploy the Ceph Object Gateway daemons with the appropriate realm and zone or update the configuration database:

   • Deploy the Ceph Object Gateway using placement specification:

     Syntax

     ceph orch apply rgw SERVICE_NAME --realm=REALM_NAME --zone=ZONE_NAME --placement="NUMBER_OF_DAEMONS HOST_NAME_1 HOST_NAME_2"

     Example

     [ceph: root@host01 /]# ceph orch apply rgw rgw --realm=ldc1 --zone=ldc1z --placement="1 host01"

   • Update the Ceph configuration database:

     Syntax

     ceph config set client.rgw.SERVICE_NAME rgw_realm REALM_NAME
     ceph config set client.rgw.SERVICE_NAME rgw_zonegroup ZONE_GROUP_NAME
     ceph config set client.rgw.SERVICE_NAME rgw_zone ZONE_NAME

     Example

     [ceph: root@host01 /]# ceph config set client.rgw.rgwsvcid.mons-1.jwgwwp rgw_realm ldc1
     [ceph: root@host01 /]# ceph config set client.rgw.rgwsvcid.mons-1.jwgwwp rgw_zonegroup ldc1zg
     [ceph: root@host01 /]# ceph config set client.rgw.rgwsvcid.mons-1.jwgwwp rgw_zone ldc1z

7. Restart the Ceph Object Gateway.

   Note

   Use the output from the ceph orch ps command, under the NAME column, to get the SERVICE_TYPE.ID information.

   1. To restart the Ceph Object Gateway on an individual node in the storage cluster:

      Syntax

      systemctl restart ceph-CLUSTER_ID@SERVICE_TYPE.ID.service

      Example

      [root@host01 ~]# systemctl restart ceph-c4b34c6f-8365-11ba-dc31-529020a7702d@rgw.realm.zone.host01.gwasto.service

   2. To restart the Ceph Object Gateways on all nodes in the storage cluster:

      Syntax

      ceph orch restart SERVICE_TYPE

      Example

      [ceph: root@host01 /]# ceph orch restart rgw

8. Create one local realm on the second data center in the storage cluster:

   Syntax

   radosgw-admin realm create --rgw-realm=REALM_NAME --default

   Example

   [ceph: root@host04 /]# radosgw-admin realm create --rgw-realm=ldc2 --default

9. Create one local master zonegroup on the second data center:

   Syntax

   radosgw-admin zonegroup create --rgw-zonegroup=ZONE_GROUP_NAME --endpoints=http://RGW_NODE_NAME:80 --rgw-realm=REALM_NAME --master --default

   Example

   [ceph: root@host04 /]# radosgw-admin zonegroup create --rgw-zonegroup=ldc2zg --endpoints=http://rgw2:80 --rgw-realm=ldc2 --master --default

10. Create one local zone on the second data center:

    Syntax

    radosgw-admin zone create --rgw-zonegroup=ZONE_GROUP_NAME --rgw-zone=ZONE_NAME --master --default --endpoints=HTTP_FQDN[, HTTP_FQDN]

    Example

    [ceph: root@host04 /]# radosgw-admin zone create --rgw-zonegroup=ldc2zg --rgw-zone=ldc2z --master --default --endpoints=http://rgw.example.com

11. Commit the period:

    Example

    [ceph: root@host04 /]# radosgw-admin period update --commit

12. Optional: If you specified the realm and zone in the service specification during the deployment of the Ceph Object Gateway, update the spec section of the specification file:

    Syntax

    spec:
      rgw_realm: REALM_NAME
      rgw_zone: ZONE_NAME

13. You can either deploy the Ceph Object Gateway daemons with the appropriate realm and zone or update the configuration database:

    • Deploy the Ceph Object Gateway using placement specification:

      Syntax

      ceph orch apply rgw SERVICE_NAME --realm=REALM_NAME --zone=ZONE_NAME --placement="NUMBER_OF_DAEMONS HOST_NAME_1 HOST_NAME_2"

      Example

      [ceph: root@host01 /]# ceph orch apply rgw rgw --realm=ldc2 --zone=ldc2z --placement="1 host01"

    • Update the Ceph configuration database:

      Syntax

      ceph config set client.rgw.SERVICE_NAME rgw_realm REALM_NAME
      ceph config set client.rgw.SERVICE_NAME rgw_zonegroup ZONE_GROUP_NAME
      ceph config set client.rgw.SERVICE_NAME rgw_zone ZONE_NAME

      Example

      [ceph: root@host01 /]# ceph config set client.rgw.rgwsvcid.mons-1.jwgwwp rgw_realm ldc2
      [ceph: root@host01 /]# ceph config set client.rgw.rgwsvcid.mons-1.jwgwwp rgw_zonegroup ldc2zg
      [ceph: root@host01 /]# ceph config set client.rgw.rgwsvcid.mons-1.jwgwwp rgw_zone ldc2z

14. Restart the Ceph Object Gateway.

    Note

    Use the output from the ceph orch ps command, under the NAME column, to get the SERVICE_TYPE.ID information.

    1. To restart the Ceph Object Gateway on individual node in the storage cluster:

       Syntax

       systemctl restart ceph-CLUSTER_ID@SERVICE_TYPE.ID.service

       Example

       [root@host04 ~]# systemctl restart ceph-c4b34c6f-8365-11ba-dc31-529020a7702d@rgw.realm.zone.host01.gwasto.service

    2. To restart the Ceph Object Gateways on all nodes in the storage cluster:

       Syntax

       ceph orch restart SERVICE_TYPE

       Example

       [ceph: root@host04 /]# ceph orch restart rgw

15. Create a replicated realm on the first data center in the storage cluster:

    Syntax

    radosgw-admin realm create --rgw-realm=REPLICATED_REALM_1 --default

    Example

    [ceph: root@host01 /]# radosgw-admin realm create --rgw-realm=rdc1 --default

    Use the --default flag to make the replicated realm default on the primary site.

16. Create a master zonegroup for the first data center:

    Syntax

    radosgw-admin zonegroup create --rgw-zonegroup=RGW_ZONE_GROUP --endpoints=http://_RGW_NODE_NAME:80 --rgw-realm=_RGW_REALM_NAME --master --default

    Example

    [ceph: root@host01 /]# radosgw-admin zonegroup create --rgw-zonegroup=rdc1zg --endpoints=http://rgw1:80 --rgw-realm=rdc1 --master --default

17. Create a master zone on the first data center:

    Syntax

    radosgw-admin zone create --rgw-zonegroup=RGW_ZONE_GROUP --rgw-zone=_MASTER_RGW_NODE_NAME --master --default --endpoints=HTTP_FQDN[,HTTP_FQDN]

    Example

    [ceph: root@host01 /]# radosgw-admin zone create --rgw-zonegroup=rdc1zg --rgw-zone=rdc1z --master --default --endpoints=http://rgw.example.com

18. Create a synchronization user and add the system user to the master zone for multi-site:

    Syntax

    radosgw-admin user create --uid="SYNCHRONIZATION_USER" --display-name="Synchronization User" --system
    radosgw-admin zone modify --rgw-zone=RGW_ZONE --access-key=ACCESS_KEY --secret=SECRET_KEY

    Example

    radosgw-admin user create --uid="synchronization-user" --display-name="Synchronization User" --system
    [ceph: root@host01 /]# radosgw-admin zone modify --rgw-zone=rdc1zg --access-key=3QV0D6ZMMCJZMSCXJ2QJ --secret=VpvQWcsfI9OPzUCpR4kynDLAbqa1OIKqRB6WEnH8

19. Commit the period:

    Example

    [ceph: root@host01 /]# radosgw-admin period update --commit

20. Optional: If you specified the realm and zone in the service specification during the deployment of the Ceph Object Gateway, update the spec section of the specification file:

    Syntax

    spec:
      rgw_realm: REALM_NAME
      rgw_zone: ZONE_NAME

21. You can either deploy the Ceph Object Gateway daemons with the appropriate realm and zone or update the configuration database:

    • Deploy the Ceph Object Gateway using placement specification:

      Syntax

      ceph orch apply rgw SERVICE_NAME --realm=REALM_NAME --zone=ZONE_NAME --placement="NUMBER_OF_DAEMONS HOST_NAME_1 HOST_NAME_2"

      Example

      [ceph: root@host01 /]# ceph orch apply rgw rgw --realm=rdc1 --zone=rdc1z --placement="1 host01"

    • Update the Ceph configuration database:

      Syntax

      ceph config set client.rgw.SERVICE_NAME  rgw_realm REALM_NAME
      ceph config set client.rgw.SERVICE_NAME  rgw_zonegroup ZONE_GROUP_NAME
      ceph config set client.rgw.SERVICE_NAME  rgw_zone ZONE_NAME

      Example

      [ceph: root@host01 /]# ceph config set client.rgw.rgwsvcid.mons-1.jwgwwp rgw_realm rdc1
      [ceph: root@host01 /]# ceph config set client.rgw.rgwsvcid.mons-1.jwgwwp rgw_zonegroup rdc1zg
      [ceph: root@host01 /]# ceph config set client.rgw.rgwsvcid.mons-1.jwgwwp rgw_zone rdc1z

22. Restart the Ceph Object Gateway.

    Note

    Use the output from the ceph orch ps command, under the NAME column, to get the SERVICE_TYPE.ID information.

    1. To restart the Ceph Object Gateway on individual node in the storage cluster:

       Syntax

       systemctl restart ceph-CLUSTER_ID@SERVICE_TYPE.ID.service

       Example

       [root@host01 ~]# systemctl restart ceph-c4b34c6f-8365-11ba-dc31-529020a7702d@rgw.realm.zone.host01.gwasto.service

    2. To restart the Ceph Object Gateways on all nodes in the storage cluster:

       Syntax

       ceph orch restart SERVICE_TYPE

       Example

       [ceph: root@host01 /]# ceph orch restart rgw

23. Pull the replicated realm on the second data center:

    Syntax

    radosgw-admin realm pull --url=https://tower-osd1.cephtips.com --access-key=ACCESS_KEY --secret-key=SECRET_KEY

    Example

    [ceph: root@host01 /]# radosgw-admin realm pull --url=https://tower-osd1.cephtips.com --access-key=3QV0D6ZMMCJZMSCXJ2QJ --secret-key=VpvQWcsfI9OPzUCpR4kynDLAbqa1OIKqRB6WEnH8

24. Pull the period from the first data center:

    Syntax

    radosgw-admin period pull --url=https://tower-osd1.cephtips.com --access-key=ACCESS_KEY --secret-key=SECRET_KEY

    Example

    [ceph: root@host01 /]# radosgw-admin period pull --url=https://tower-osd1.cephtips.com --access-key=3QV0D6ZMMCJZMSCXJ2QJ --secret-key=VpvQWcsfI9OPzUCpR4kynDLAbqa1OIKqRB6WEnH8

25. Create the secondary zone on the second data center:

    Syntax

    radosgw-admin zone create --rgw-zone=RGW_ZONE --rgw-zonegroup=RGW_ZONE_GROUP --endpoints=https://tower-osd4.cephtips.com --access-key=_ACCESS_KEY --secret-key=SECRET_KEY

    Example

    [ceph: root@host04 /]# radosgw-admin zone create --rgw-zone=rdc2z --rgw-zonegroup=rdc1zg --endpoints=https://tower-osd4.cephtips.com --access-key=3QV0D6ZMMCJZMSCXJ2QJ --secret-key=VpvQWcsfI9OPzUCpR4kynDLAbqa1OIKqRB6WEnH8

26. Commit the period:

    Example

    [ceph: root@host04 /]# radosgw-admin period update --commit

27. Optional: If you specified the realm and zone in the service specification during the deployment of the Ceph Object Gateway, update the spec section of the specification file:

    Syntax

    spec:
      rgw_realm: REALM_NAME
      rgw_zone: ZONE_NAME

28. You can either deploy the Ceph Object Gateway daemons with the appropriate realm and zone or update the configuration database:

    • Deploy the Ceph Object Gateway using placement specification:

      Syntax

      ceph orch apply rgw SERVICE_NAME --realm=REALM_NAME --zone=ZONE_NAME --placement="NUMBER_OF_DAEMONS HOST_NAME_1 HOST_NAME_2"

      Example

      [ceph: root@host04 /]# ceph orch apply rgw rgw --realm=rdc1 --zone=rdc2z --placement="1 host04"

    • Update the Ceph configuration database:

      Syntax

      ceph config set client.rgw.SERVICE_NAME rgw_realm REALM_NAME
      ceph config set client.rgw.SERVICE_NAME rgw_zonegroup ZONE_GROUP_NAME
      ceph config set client.rgw.SERVICE_NAME rgw_zone ZONE_NAME

      Example

      [ceph: root@host04 /]# ceph config set client.rgw.rgwsvcid.mons-1.jwgwwp rgw_realm rdc1
      [ceph: root@host04 /]# ceph config set client.rgw.rgwsvcid.mons-1.jwgwwp rgw_zonegroup rdc1zg
      [ceph: root@host04 /]# ceph config set client.rgw.rgwsvcid.mons-1.jwgwwp rgw_zone rdc2z

29. Restart the Ceph Object Gateway.

    Note

    Use the output from the ceph orch ps command, under the NAME column, to get the SERVICE_TYPE.ID information.

    1. To restart the Ceph Object Gateway on individual node in the storage cluster:

       Syntax

       systemctl restart ceph-CLUSTER_ID@SERVICE_TYPE.ID.service

       Example

       [root@host02 ~]# systemctl restart ceph-c4b34c6f-8365-11ba-dc31-529020a7702d@rgw.realm.zone.host01.gwasto.service

    2. To restart the Ceph Object Gateways on all nodes in the storage cluster:

       Syntax

       ceph orch restart SERVICE_TYPE

       Example

       [ceph: root@host04 /]# ceph orch restart rgw

30. Log in as root on the endpoint for the second data center.

31. Verify the synchronization status on the master realm:

    Syntax

    radosgw-admin sync status

    Example

    [ceph: root@host04 /]# radosgw-admin sync status
              realm 59762f08-470c-46de-b2b1-d92c50986e67 (ldc2)
          zonegroup 7cf8daf8-d279-4d5c-b73e-c7fd2af65197 (ldc2zg)
               zone 034ae8d3-ae0c-4e35-8760-134782cb4196 (ldc2z)
      metadata sync no sync (zone is master)
      current time 2023-08-17T05:49:56Z
    zonegroup features enabled: resharding
                       disabled: compress-encrypted

    Important

    In Red Hat Ceph Storage 5.3z5 release, compress-encrypted feature is displayed with radosgw-admin sync status command and it is disabled by default. Do not enable this feature as it is not supported until Red Hat Ceph Storage 6.1z2.

32. Log in as root on the endpoint for the first data center.

33. Verify the synchronization status for the replication-synchronization realm:

    Syntax

    radosgw-admin sync status --rgw-realm RGW_REALM_NAME

    Example

    [ceph: root@host01 /]# radosgw-admin sync status --rgw-realm rdc1
              realm 73c7b801-3736-4a89-aaf8-e23c96e6e29d (rdc1)
          zonegroup d67cc9c9-690a-4076-89b8-e8127d868398 (rdc1zg)
               zone 67584789-375b-4d61-8f12-d1cf71998b38 (rdc2z)
      metadata sync syncing
                    full sync: 0/64 shards
                    incremental sync: 64/64 shards
                    metadata is caught up with master
          data sync source: 705ff9b0-68d5-4475-9017-452107cec9a0 (rdc1z)
                            syncing
                            full sync: 0/128 shards
                            incremental sync: 128/128 shards
                            data is caught up with source
              realm 73c7b801-3736-4a89-aaf8-e23c96e6e29d (rdc1)
          zonegroup d67cc9c9-690a-4076-89b8-e8127d868398 (rdc1zg)
               zone 67584789-375b-4d61-8f12-d1cf71998b38 (rdc2z)
      metadata sync syncing
                    full sync: 0/64 shards
                    incremental sync: 64/64 shards
                    metadata is caught up with master
          data sync source: 705ff9b0-68d5-4475-9017-452107cec9a0 (rdc1z)
                            syncing
                            full sync: 0/128 shards
                            incremental sync: 128/128 shards
                            data is caught up with source

34. To store and access data in the local site, create the user for local realm:

    Syntax

    radosgw-admin user create --uid="LOCAL_USER" --display-name="Local user" --rgw-realm=_REALM_NAME --rgw-zonegroup=ZONE_GROUP_NAME --rgw-zone=ZONE_NAME

    Example

    [ceph: root@host04 /]# radosgw-admin user create --uid="local-user" --display-name="Local user" --rgw-realm=ldc1 --rgw-zonegroup=ldc1zg --rgw-zone=ldc1z

    Important

    By default, users are created under the default realm. For the users to access data in the local realm, the radosgw-admin command requires the --rgw-realm argument.

1. Extract/Download a PEM-formatted certificate for the Certificate Authority (CA) that signed the LDAP server’s SSL certificate.
2. Confirm that /etc/openldap/ldap.conf does not have TLS_REQCERT set.
3. Confirm that /etc/openldap/ldap.conf contains a TLS_CACERTDIR /etc/openldap/certs setting.

4. Use the certutil command to add the AD CA to the store at /etc/openldap/certs. For example, if the CA is "msad-frog-MSAD-FROG-CA", and the PEM-formatted CA file is ldap.pem, use the following command:

   Example

   # certutil -d /etc/openldap/certs -A -t "TC,," -n "msad-frog-MSAD-FROG-CA" -i /path/to/ldap.pem

5. Update SELinux on all remote LDAP sites:

   Example

   # setsebool -P httpd_can_network_connect on

   Note

   This still has to be set even if SELinux is in permissive mode.

6. Make the certs database world-readable:

   Example

   # chmod 644 /etc/openldap/certs/*

7. Connect to the server using the "ldapwhoami" command as a non-root user.

   Example

   $ ldapwhoami -H ldaps://rh-directory-server.example.com -d 9

   The -d 9 option will provide debugging information in case something went wrong with the SSL negotiation.

Procedure

1. Create an LDAP user for the Ceph Object Gateway, and make a note of the binddn. Since the Ceph object gateway uses the ceph user, consider using ceph as the username. The user needs to have permissions to search the directory. The Ceph Object Gateway binds to this user as specified in rgw_ldap_binddn.

2. Test to ensure that the user creation worked. Where ceph is the user ID under People and example.com is the domain, you can perform a search for the user.

   # ldapsearch -x -D "uid=ceph,ou=People,dc=example,dc=com" -W -H ldaps://example.com -b "ou=People,dc=example,dc=com" -s sub 'uid=ceph'

3. On each gateway node, create a file for the user’s secret. For example, the secret might get stored in a file entitled /etc/bindpass. For security, change the owner of this file to the ceph user and group to ensure it is not globally readable.

4. Add the rgw_ldap_secret option:

   Syntax

   ceph config set client.rgw OPTION VALUE

   Example

   [ceph: root@host01 /]# ceph config set client.rgw rgw_ldap_secret /etc/bindpass

5. Patch the bind password file to the Ceph Object Gateway container and reapply the Ceph Object Gateway specification:

   Example

   service_type: rgw
   service_id: rgw.1
   service_name: rgw.rgw.1
   placement:
     label: rgw
     extra_container_args:
     - -v
     - /etc/bindpass:/etc/bindpass

   Note

   /etc/bindpass is not shipped automatically with Red Hat Ceph Storage and you need to ensure that the content is available on all the possible Ceph Object Gateway instance nodes.

1. Specifying a partial filter:

   Example

   "objectclass=inetorgperson"

   The Ceph Object Gateway generates the search filter with the user name from the token and the value of rgw_ldap_dnattr. The constructed filter is then combined with the partial filter from the rgw_ldap_searchfilter value. For example, the user name and the settings generate the final search filter:

   Example

   "(&(uid=joe)(objectclass=inetorgperson))"

   User joe is only granted access if he is found in the LDAP directory, has an object class of inetorgperson, and specifies a valid password.

2. Specifying a complete filter:

   A complete filter must contain a USERNAME token which is substituted with the user name during the authentication attempt. The rgw_ldap_dnattr setting is not used in this case. For example, to limit valid users to a specific group, use the following filter:

   Example

   "(&(uid=@USERNAME@)(memberOf=cn=ceph-users,ou=groups,dc=mycompany,dc=com))"

1. Export the access key:

   Syntax

   export RGW_ACCESS_KEY_ID="USERNAME"

2. Export the secret key:

   Syntax

   export RGW_SECRET_ACCESS_KEY="PASSWORD"

3. Export the token. For LDAP, use ldap as the token type (ttype).

   Example

   radosgw-token --encode --ttype=ldap

   For Active Directory, use ad as the token type.

   Example

   radosgw-token --encode --ttype=ad

   The result is a base-64 encoded string, which is the access token. Provide this access token to S3 clients in lieu of the access key. The secret key is no longer required.

4. Optional: For added convenience, export the base-64 encoded string to the RGW_ACCESS_KEY_ID environment variable if the S3 client uses the environment variable.

   Example

   export RGW_ACCESS_KEY_ID="ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAibGRhcCIsCiAgICAgICAgImlkIjogImNlcGgiLAogICAgICAgICJrZXkiOiAiODAwI0dvcmlsbGEiCiAgICB9Cn0K"

Procedure

1. Use the RGW_ACCESS_KEY_ID environment variable to configure the Ceph Object Gateway client. Alternatively, you can copy the base-64 encoded string and specify it as the access key. Following is an example of the configured S3 client:

   Example

   cat .aws/credentials
   
   [default]
   aws_access_key_id = ewogICaGbnjlwe9UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAiYWQiLAogICAgICAgICJpZCI6ICJjZXBoIiwKICAgICAgICAia2V5IjogInBhc3M0Q2VwaCIKICAgIH0KfQo=
   aws_secret_access_key =

   Note

   The secret key is no longer required.

2. Run the aws s3 ls command to verify the user:

   Example

   [root@host01 ~]# aws s3 ls --endpoint http://host03
   
   2023-12-11 17:08:50 mybucket
   2023-12-24 14:55:44 mybucket2

3. Optional: You can also run the radosgw-admin user command to verify the user in the directory:

   Example

   [root@host01 ~]# radosgw-admin user info --uid dir1
   {
       "user_id": "dir1",
       "display_name": "dir1",
       "email": "",
       "suspended": 0,
       "max_buckets": 1000,
       "subusers": [],
       "keys": [],
       "swift_keys": [],
       "caps": [],
       "op_mask": "read, write, delete",
       "default_placement": "",
       "default_storage_class": "",
       "placement_tags": [],
       "bucket_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "user_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "temp_url_keys": [],
       "type": "ldap",
       "mfa_ids": []
   }

Procedure

1. Create an LDAP user for the Ceph Object Gateway, and make a note of the binddn. Since the Ceph object gateway uses the ceph user, consider using ceph as the username. The user needs to have permissions to search the directory. The Ceph Object Gateway binds to this user as specified in rgw_ldap_binddn.

2. Test to ensure that the user creation worked. Where ceph is the user ID under People and example.com is the domain, you can perform a search for the user.

   # ldapsearch -x -D "uid=ceph,ou=People,dc=example,dc=com" -W -H ldaps://example.com -b "ou=People,dc=example,dc=com" -s sub 'uid=ceph'

3. On each gateway node, create a file for the user’s secret. For example, the secret might get stored in a file entitled /etc/bindpass. For security, change the owner of this file to the ceph user and group to ensure it is not globally readable.

4. Add the rgw_ldap_secret option:

   Syntax

   ceph config set client.rgw OPTION VALUE

   Example

   [ceph: root@host01 /]# ceph config set client.rgw rgw_ldap_secret /etc/bindpass

5. Patch the bind password file to the Ceph Object Gateway container and reapply the Ceph Object Gateway specification:

   Example

   service_type: rgw
   service_id: rgw.1
   service_name: rgw.rgw.1
   placement:
     label: rgw
     extra_container_args:
     - -v
     - /etc/bindpass:/etc/bindpass

   Note

   /etc/bindpass is not shipped automatically with Red Hat Ceph Storage and you need to ensure that the content is available on all the possible Ceph Object Gateway instance nodes.

1. Export the access key:

   Syntax

   export RGW_ACCESS_KEY_ID="USERNAME"

2. Export the secret key:

   Syntax

   export RGW_SECRET_ACCESS_KEY="PASSWORD"

3. Export the token. For LDAP, use ldap as the token type (ttype).

   Example

   radosgw-token --encode --ttype=ldap

   For Active Directory, use ad as the token type.

   Example

   radosgw-token --encode --ttype=ad

   The result is a base-64 encoded string, which is the access token. Provide this access token to S3 clients in lieu of the access key. The secret key is no longer required.

4. Optional: For added convenience, export the base-64 encoded string to the RGW_ACCESS_KEY_ID environment variable if the S3 client uses the environment variable.

   Example

   export RGW_ACCESS_KEY_ID="ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAibGRhcCIsCiAgICAgICAgImlkIjogImNlcGgiLAogICAgICAgICJrZXkiOiAiODAwI0dvcmlsbGEiCiAgICB9Cn0K"

Procedure

1. Use the RGW_ACCESS_KEY_ID environment variable to configure the Ceph Object Gateway client. Alternatively, you can copy the base-64 encoded string and specify it as the access key. Following is an example of the configured S3 client:

   Example

   cat .aws/credentials
   
   [default]
   aws_access_key_id = ewogICaGbnjlwe9UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAiYWQiLAogICAgICAgICJpZCI6ICJjZXBoIiwKICAgICAgICAia2V5IjogInBhc3M0Q2VwaCIKICAgIH0KfQo=
   aws_secret_access_key =

   Note

   The secret key is no longer required.

2. Run the aws s3 ls command to verify the user:

   Example

   [root@host01 ~]# aws s3 ls --endpoint http://host03
   
   2023-12-11 17:08:50 mybucket
   2023-12-24 14:55:44 mybucket2

3. Optional: You can also run the radosgw-admin user command to verify the user in the directory:

   Example

   [root@host01 ~]# radosgw-admin user info --uid dir1
   {
       "user_id": "dir1",
       "display_name": "dir1",
       "email": "",
       "suspended": 0,
       "max_buckets": 1000,
       "subusers": [],
       "keys": [],
       "swift_keys": [],
       "caps": [],
       "op_mask": "read, write, delete",
       "default_placement": "",
       "default_storage_class": "",
       "placement_tags": [],
       "bucket_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "user_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "temp_url_keys": [],
       "type": "ldap",
       "mfa_ids": []
   }

Procedure

1. List the endpoints under the Swift service:

   [root@swift~]# openstack endpoint list --service=swift

2. Verify settings for the endpoints listed in the previous command:

   Syntax

   [root@swift~]# openstack endpoint show ENDPOINT_ID

   Showing the endpoints will echo the endpoints settings, and the service settings.

   Table 5.2. Example

   Field	Value
   adminurl	http://radosgw.example.com:8080/swift/v1
   enabled	True
   id	e4249d2b60e44743a67b5e5b38c18dd3
   internalurl	http://radosgw.example.com:8080/swift/v1
   publicurl	http://radosgw.example.com:8080/swift/v1
   region	us-west
   service_id	37c4c0e79571404cb4644201a4a6e5ee
   service_name	swift
   service_type	object-store

Procedure

1. Convert the OpenSSL certificate to the nss db format:

   Example

   [root@osp ~]# mkdir /var/ceph/nss
   
   [root@osp ~]# openssl x509 -in /etc/keystone/ssl/certs/ca.pem -pubkey | \
       certutil -d /var/ceph/nss -A -n ca -t "TCu,Cu,Tuw"
   
   [root@osp ~]# openssl x509 -in /etc/keystone/ssl/certs/signing_cert.pem -pubkey | \
       certutil -A -d /var/ceph/nss -n signing_cert -t "P,P,P"

2. Install Keystone’s SSL certificate in the node running the Ceph Object Gateway. Alternatively set the value of the configurable rgw_keystone_verify_ssl setting to false.

   Setting rgw_keystone_verify_ssl to false means that the gateway will not attempt to verify the certificate.

Procedure

1. Do the following for each gateway instance.

   1. Set the rgw_s3_auth_use_keystone option to true:

      Example

      [ceph: root@host01 /]# ceph config set client.rgw rgw_s3_auth_use_keystone true

   2. Set the nss_db_path setting to the path where the NSS database is stored:

      Example

      [ceph: root@host01 /]# ceph config set client.rgw nss_db_path "/var/lib/ceph/radosgw/ceph-rgw.rgw01/nss"

2. Provide authentication credentials:

   It is possible to configure a Keystone service tenant, user, and password for keystone for the OpenStack Identity API, similar to the way system administrators tend to configure OpenStack services. Providing a username and password avoids providing the shared secret to the rgw_keystone_admin_token setting.

   Important

   Red Hat recommends disabling authentication by admin token in production environments. The service tenant credentials should have admin privileges.

   The necessary configuration options are:

   Syntax

   ceph config set client.rgw rgw_keystone_url KEYSTONE_URL:ADMIN_PORT
   ceph config set client.rgw rgw_keystone_admin_user KEYSTONE_TENANT_USER_NAME
   ceph config set client.rgw rgw_keystone_admin_password KEYSTONE_TENANT_USER_PASSWORD
   ceph config set client.rgw rgw_keystone_admin_tenant KEYSTONE_TENANT_NAME
   ceph config set client.rgw rgw_keystone_accepted_roles KEYSTONE_ACCEPTED_USER_ROLES
   ceph config set client.rgw rgw_keystone_token_cache_size NUMBER_OF_TOKENS_TO_CACHE
   ceph config set client.rgw rgw_keystone_revocation_interval NUMBER_OF_SECONDS_BEFORE_CHECKING_REVOKED_TICKETS
   ceph config set client.rgw rgw_keystone_implicit_tenants TRUE_FOR_PRIVATE_TENANT_FOR_EACH_NEW_USER

   A Ceph Object Gateway user is mapped into a Keystone tenant. A Keystone user has different roles assigned to it on possibly more than a single tenant. When the Ceph Object Gateway gets the ticket, it looks at the tenant, and the user roles that are assigned to that ticket, and accepts or rejects the request according to the rgw_keystone_accepted_roles configurable.