For Red Hat Ceph Storage 1.3, Red Hat supports the Ceph Object Gateway running on Civetweb (embedded into the ceph-radosgw daemon) instead of Apache and FastCGI. Using Civetweb simplifies the Ceph Object Gateway installation and configuration.

Before you begin with the installation of the Ceph Object Gateway, perform the steps listed in the Pre-Installation section of the Installation Guide for Red Hat Enterprise Linux.

Red Hat packages the Ceph Object Gateway in the rhel-7-server-rhceph-1.3-tools-rpms repository. To ensure you are using the same version of Ceph as your storage cluster, execute the following on your Ceph Object Gateway node to enable the repository:

sudo subscription-manager repos --enable=rhel-7-server-rhceph-1.3-tools-rpms

From the working directory of your administration server, install the Ceph Object Gateway package on the Ceph Object Gateway node. For example:

ceph-deploy install --rgw <gateway-node1> [<gateway-node2> ...]

The ceph-common package is a dependency, so ceph-deploy will install this too. The ceph CLI tools are intended for administrators. To make your Ceph Object Gateway node an administrator node, execute the following from the working directory of your administration server.

ceph-deploy admin <node-name>

From the working directory of your administration server, execute the following to create an instance of the Ceph Object Gateway on the Ceph Object Gateway:

ceph-deploy rgw create <gateway-node1>

Once the gateway is running, you should be able to access it on port 7480 with an unauthenticated request in a web browser or a command line HTTP client (e.g., curl, wget etc.) like this:

http://gateway-node1:7480

If the gateway instance is working properly, your browser or CLI HTTP client should receive a response like this:

<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
	<Owner>
		<ID>anonymous</ID>
		<DisplayName></DisplayName>
	</Owner>
	<Buckets>
	</Buckets>
</ListAllMyBucketsResult>

If at any point you run into trouble and you want to start over, execute the following from the working directory of your administration server to purge the configuration:

ceph-deploy purge <gateway-node1> [<gateway-node2>]
ceph-deploy purgedata <gateway-node1> [<gateway-node2>]

If you execute purge, you must re-install Ceph.

Civetweb runs on port 7480 by default. To change the default port (e.g,. to port 80), modify your Ceph configuration file in /etc/ceph directory of your administration server. Add a section entitled [client.rgw.<gateway-node>], replacing <gateway-node> with the short node name of your Ceph Object Gateway node (i.e., hostname -s).

For example, if your node name is gateway-node1, add a section like this after the [global] section in /etc/ceph/ceph.conf file:

[client.rgw.gateway-node1]
rgw_frontends = "civetweb port=80"

Pull the updated configuration file from /etc/ceph directory to the working directory of your administration server (e.g. ceph-config directory). From the working directory of your administration server, execute:

ceph-deploy --overwrite-conf config pull <admin-node>

Push the updated configuration file to your Ceph Object Gateway node and other Ceph nodes. From the working directory of your administration server, execute:

ceph-deploy --overwrite-conf config push <gateway-node> [<other-nodes>]

To make the new port setting take effect, from your Ceph Object Gateway node, restart the Ceph Object Gateway.

sudo systemctl restart ceph-radosgw.service

Finally, check to ensure that the port you selected is open on the node’s firewall (e.g., port 80). If it is not open, add the port and reload the firewall configuration. For example, on your Ceph Object Gateway node, execute:

sudo firewall-cmd --list-all
sudo firewall-cmd --zone=public --add-port 80/tcp --permanent
sudo firewall-cmd --reload

If you’re running the Ceph Object Gateway on Apache and FastCGI with Red Hat Ceph Storage v1.2.x or above, you’re already running Civetweb—​it starts with the ceph-radosgw daemon and it’s running on port 7480 by default so that it doesn’t conflict with your Apache and FastCGI installation and other commonly used web service ports. Migrating to use Civetweb basically involves removing your Apache installation. Then, you must remove Apache and FastCGI settings from your Ceph configuration file and reset rgw_frontends to Civetweb.

Referring back to the description for installing a Ceph Object Gateway with ceph-deploy, notice that the configuration file only has one setting rgw_frontends (and that’s assuming you elected to change the default port). The ceph-deploy utility generates the data directory and the keyring for you—​placing the keyring in /var/lib/ceph/radosgw/{rgw-intance}. The daemon looks in default locations, whereas you may have specified different settings in your Ceph configuration file. Since you already have keys and a data directory, you will want to maintain those paths in your Ceph configuration file if you used something other than default paths.

A typical Ceph Object Gateway configuration file for an Apache-based deployment looks something like this:

[client.radosgw.gateway-node1]
host = {hostname}
keyring = /etc/ceph/ceph.client.radosgw.keyring
rgw socket path = ""
log file = /var/log/radosgw/client.radosgw.gateway-node1.log
rgw frontends = fastcgi socket_port=9000 socket_host=0.0.0.0
rgw print continue = false

To modify it for use with Civetweb, simply remove the Apache-specific settings such as rgw_socket_path and rgw_print_continue. Then, change the rgw_frontends setting to reflect Civetweb rather than the Apache FastCGI front end and specify the port number you intend to use. For example:

[client.radosgw.gateway-node1]
host = {hostname}
keyring = /etc/ceph/ceph.client.radosgw.keyring
log file = /var/log/radosgw/client.radosgw.gateway-node1.log
rgw_frontends = civetweb port=80

Finally, on your Ceph Object Gateway execute the following to restart the Ceph Object Gateway:

sudo systemctl restart ceph-radosgw.service

If you used a port number that is not open, you will also need to open that port on your firewall.

To use Ceph with S3-style subdomains (e.g., bucket-name.domain-name.com), you need to add a wildcard to the DNS record of the DNS server you use with the ceph-radosgw daemon.

The address of the DNS must also be specified in the Ceph configuration file with the rgw dns name = {hostname} setting.

For dnsmasq, add the following address setting with a dot (.) prepended to the host name:

address=/.{hostname-or-fqdn}/{host-ip-address}

For example:

address=/.gateway-node1/192.168.122.75

For bind, add a wildcard to the DNS record. For example:

$TTL    604800
@       IN      SOA     gateway-node1. root.gateway-node1. (
                              2         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@       IN      NS      gateway-node1.
@       IN      A       192.168.122.113
*       IN      CNAME   @

Restart your DNS server and ping your server with a subdomain to ensure that your ceph-radosgw daemon can process the subdomain requests:

ping mybucket.{hostname}

For example:

ping mybucket.gateway-node1

Once you finish the setup procedure, check your logging output to ensure it meets your needs. Log files are located in /var/log/radosgw by default. If you encounter issues with your configuration, you can increase logging and debugging messages in the [global] section of your Ceph configuration file and restart the gateway(s) to help troubleshoot any configuration issues. For example:

[global]
#append the following in the global section.
debug ms = 1
debug rgw = 20
debug civetweb = 20

You may also modify these settings at runtime. For example:

ceph tell osd.0 injectargs --debug_civetweb 10/20

For general details on logging and debugging, see Logging and Debugging. For Ceph Object Gateway-specific details on logging settings, see Logging Settings in this guide.

To use the REST interfaces, first create an initial Ceph Object Gateway user for the S3 interface. Then, create a subuser for the Swift interface. You then need to verify if the created users are able to access the gateway.

sudo radosgw-admin user create --uid="testuser" --display-name="First User"

{
	"user_id": "testuser",
	"display_name": "First User",
	"email": "",
	"suspended": 0,
	"max_buckets": 1000,
	"auid": 0,
	"subusers": [],
	"keys": [{
		"user": "testuser",
		"access_key": "I0PJDPCIYZ665MW88W9R",
		"secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA"
	}],
	"swift_keys": [],
	"caps": [],
	"op_mask": "read, write, delete",
	"default_placement": "",
	"placement_tags": [],
	"bucket_quota": {
		"enabled": false,
		"max_size_kb": -1,
		"max_objects": -1
	},
	"user_quota": {
		"enabled": false,
		"max_size_kb": -1,
		"max_objects": -1
	},
	"temp_url_keys": []
}

sudo radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --access=full

{
	"user_id": "testuser",
	"display_name": "First User",
	"email": "",
	"suspended": 0,
	"max_buckets": 1000,
	"auid": 0,
	"subusers": [{
		"id": "testuser:swift",
		"permissions": "full-control"
	}],
	"keys": [{
		"user": "testuser:swift",
		"access_key": "3Y1LNW4Q6X0Y53A52DET",
		"secret_key": ""
	}, {
		"user": "testuser",
		"access_key": "I0PJDPCIYZ665MW88W9R",
		"secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA"
	}],
	"swift_keys": [],
	"caps": [],
	"op_mask": "read, write, delete",
	"default_placement": "",
	"placement_tags": [],
	"bucket_quota": {
		"enabled": false,
		"max_size_kb": -1,
		"max_objects": -1
	},
	"user_quota": {
		"enabled": false,
		"max_size_kb": -1,
		"max_objects": -1
	},
	"temp_url_keys": []
}

sudo radosgw-admin key create --subuser=testuser:swift --key-type=swift --gen-secret

{
	"user_id": "testuser",
	"display_name": "First User",
	"email": "",
	"suspended": 0,
	"max_buckets": 1000,
	"auid": 0,
	"subusers": [{
		"id": "testuser:swift",
		"permissions": "full-control"
	}],
	"keys": [{
		"user": "testuser:swift",
		"access_key": "3Y1LNW4Q6X0Y53A52DET",
		"secret_key": ""
	}, {
		"user": "testuser",
		"access_key": "I0PJDPCIYZ665MW88W9R",
		"secret_key": "dxaXZ8U90SXydYzyS5ivamEP20hkLSUViiaR+ZDA"
	}],
	"swift_keys": [{
		"user": "testuser:swift",
		"secret_key": "244+fz2gSqoHwR3lYtSbIyomyPHf3i7rgSJrF\/IA"
	}],
	"caps": [],
	"op_mask": "read, write, delete",
	"default_placement": "",
	"placement_tags": [],
	"bucket_quota": {
		"enabled": false,
		"max_size_kb": -1,
		"max_objects": -1
	},
	"user_quota": {
		"enabled": false,
		"max_size_kb": -1,
		"max_objects": -1
	},
	"temp_url_keys": []
}

sudo yum install python-setuptools
sudo easy_install pip
sudo pip install --upgrade setuptools
sudo pip install --upgrade python-swiftclient

swift -A http://{IP ADDRESS}:{port}/auth/1.0 -U testuser:swift -K '{swift_secret_key}' list

swift -A http://10.19.143.116:7480/auth/1.0 -U testuser:swift -K '244+fz2gSqoHwR3lYtSbIyomyPHf3i7rgSJrF/IA' list

my-new-bucket

A Ceph Object Gateway stores administrative data in a series of pools defined in an instance’s zone configuration. For example, the buckets, users, user quotas and usage statistics discussed in the subsequent sections are stored in pools in the Ceph Storage Cluster. By default, Ceph Object Gateway will create the following pools and map them to the default zone.

You should consider creating these pools manually so that you can set the CRUSH ruleset and the number of placement groups. In a typical configuration, the pools that store the Ceph Object Gateway’s administrative data will often use the same CRUSH ruleset and use fewer placement groups, because there are 10 pools for the administrative data. See Pools and Storage Strategies for additional details.

Also see Ceph Placement Groups (PGs) per Pool Calculator for placement group calculation details. The mon_pg_warn_max_per_osd setting warns you if assign too many placement groups to a pool (i.e., 300 by default). You may adjust the value to suit your needs and the capabilities of your hardware where n is the maximum number of PGs per OSD.

mon_pg_warn_max_per_osd = n

Ceph Object Gateway stores the client bucket and object data by identifying placement targets, and storing buckets and objects in the pools associated with a placement target. If you don’t configure placement targets and map them to pools in the instance’s zone configuration, the Ceph Object Gateway will use default targets and pools (e.g., default_placement).

Storage policies give Ceph Object Gateway clients a way of accessing a storage strategy--i.e., the ability to target a particular type of storage (e.g., SSDs, SAS drives, SATA drives), a particular way of ensuring durability (replication, erasure coding), etc. To create a storage policy, use the following procedure:

Usage example:

curl -i http://10.0.0.1/swift/v1/TestContainer/file.txt -X PUT -H "X-Storage-Policy: special-placement" -H "X-Auth-Token: AUTH_rgwtxxxxxx"

The Ceph Object Gateway stores bucket index data in the index pool (index_pool), which defaults to .rgw.buckets.index. If you put many objects (hundreds of thousands to millions of objects) in a single bucket without having set quotas for the maximum number of objects per bucket, the index pool can suffer significant performance degradation.

Bucket index sharding helps prevent performance bottlenecks when allowing a high number of objects per bucket.

See Configuring Bucket Index Sharding for details on configuring bucket index sharding for new buckets.

See Bucket Index Resharding for details on changing the bucket index sharding on already existing buckets.

Configuring Bucket Index Sharding

To enable and configure bucket index sharding on all new buckets, use:

Set the settings to:

Use the following formula to calculate the recommended number of shards:

number of objects expected in a bucket / 100,000

Note that maximum number of shards is 7877.

Simple configurations

Federated configurations

In federated configurations, each zone can have a different index_pool setting to manage failover. To configure a consistent shard count for zones in one region, set the bucket_index_max_shards setting in the configuration for that region. To do so:

Bucket Index Resharding

If a bucket has grown larger than the initial configuration was optimized for, reshard the bucket index pool by using the radosgw-admin bucket reshard command. This command:

To reshard the bucket index pool:

Ceph Object Storage user management refers to users that are client applications of the Ceph Object Storage service (i.e., not the Ceph Object Gateway as a client application of the Ceph Storage Cluster). You must create a user, access key and secret to enable client applications to interact with the Ceph Object Gateway service.

There are two user types:

You can create, modify, view, suspend and remove users and subusers. In addition to user and subuser IDs, you may add a display name and an email address for a user. You can specify a key and secret, or generate a key and secret automatically. When generating or specifying keys, note that user IDs correspond to an S3 key type and subuser IDs correspond to a swift key type. Swift keys also have access levels of read, write, readwrite and full.

User management command-line syntax generally follows the pattern user <command> <user-id> where <user-id> is either the --uid= option followed by the user’s ID (S3) or the --subuser= option followed by the user name (Swift). For example:

radosgw-admin user <create|modify|info|rm|suspend|enable|check|stats> <--uid={id}|--subuser={name}> [other-options]

Additional options may be required depending on the command you execute.

radosgw-admin user create --uid=<id> \
[--key-type=<type>] [--gen-access-key|--access-key=<key>]\
[--gen-secret | --secret=<key>] \
[--email=<email>] --display-name=<name>

radosgw-admin user create --uid=janedoe --display-name="Jane Doe" --email=jane@example.com

{ "user_id": "janedoe",
  "display_name": "Jane Doe",
  "email": "jane@example.com",
  "suspended": 0,
  "max_buckets": 1000,
  "auid": 0,
  "subusers": [],
  "keys": [
        { "user": "janedoe",
          "access_key": "11BS02LGFB6AL6H1ADMW",
          "secret_key": "vzCEkuryfn060dfee4fgQPqFrncKEIkh3ZcdOANY"}],
  "swift_keys": [],
  "caps": [],
  "op_mask": "read, write, delete",
  "default_placement": "",
  "placement_tags": [],
  "bucket_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "user_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "temp_url_keys": []}

radosgw-admin subuser create --uid={uid} --subuser={uid} --access=[ read | write | readwrite | full ]

radosgw-admin subuser create --uid=janedoe --subuser=janedoe:swift --access=full

{ "user_id": "janedoe",
  "display_name": "Jane Doe",
  "email": "jane@example.com",
  "suspended": 0,
  "max_buckets": 1000,
  "auid": 0,
  "subusers": [
        { "id": "janedoe:swift",
          "permissions": "full-control"}],
  "keys": [
        { "user": "janedoe",
          "access_key": "11BS02LGFB6AL6H1ADMW",
          "secret_key": "vzCEkuryfn060dfee4fgQPqFrncKEIkh3ZcdOANY"}],
  "swift_keys": [],
  "caps": [],
  "op_mask": "read, write, delete",
  "default_placement": "",
  "placement_tags": [],
  "bucket_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "user_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "temp_url_keys": []}

radosgw-admin user info --uid=janedoe

radosgw-admin user modify --uid=janedoe --display-name="Jane E. Doe"

radosgw-admin subuser modify --uid=janedoe:swift --access=full

radosgw-admin user suspend --uid=johndoe

radosgw-admin user enable --uid=johndoe

radosgw-admin user rm --uid=<uid> [--purge-keys] [--purge-data]

radosgw-admin user rm --uid=johndoe --purge-data

radosgw-admin subuser rm --subuser=johndoe:swift --purge-keys

radosgw-admin subuser rm --uid=johndoe:swift

radosgw-admin key create --subuser=johndoe:swift --key-type=swift --gen-secret

{ "user_id": "johndoe",
  "rados_uid": 0,
  "display_name": "John Doe",
  "email": "john@example.com",
  "suspended": 0,
  "subusers": [
     { "id": "johndoe:swift",
       "permissions": "full-control"}],
  "keys": [
    { "user": "johndoe",
      "access_key": "QFAMEDSJP5DEKJO0DDXY",
      "secret_key": "iaSFLDVvDdQt6lkNzHyW4fPLZugBAI1g17LO0+87"}],
  "swift_keys": [
    { "user": "johndoe:swift",
      "secret_key": "E9T2rUZNu2gxUjcwUBO8n\/Ev4KX6\/GprEuH4qhu1"}]}

radosgw-admin key create --uid=johndoe --key-type=s3 --gen-access-key --gen-secret

radosgw-admin key rm --uid=johndoe

radosgw-admin caps add --uid={uid} --caps={caps}

--caps="[users|buckets|metadata|usage|zone]=[*|read|write|read, write]"

radosgw-admin caps add --uid=johndoe --caps="users=*"

radosgw-admin caps remove --uid=johndoe --caps={caps}

The Ceph Object Gateway enables you to set quotas on users and buckets owned by users. Quotas include the maximum number of objects in a bucket and the maximum storage size in megabytes.

radosgw-admin quota set --quota-scope=user --uid=<uid> [--max-objects=<num objects>] [--max-size=<max size>]

radosgw-admin quota set --quota-scope=user --uid=johndoe --max-objects=1024 --max-size=1024

radosgw-admin quota enable --quota-scope=user --uid=<uid>

radosgw-admin quota-disable --quota-scope=user --uid=<uid>

radosgw-admin quota set --uid=<uid> --quota-scope=bucket [--max-objects=<num objects>] [--max-size=<max size]

radosgw-admin quota enable --quota-scope=bucket --uid=<uid>

radosgw-admin quota-disable --quota-scope=bucket --uid=<uid>

radosgw-admin user info --uid=<uid>

radosgw-admin user stats --uid=<uid> --sync-stats

radosgw-admin user stats --uid=<uid>

radosgw-admin regionmap get > regionmap.json

radosgw-admin regionmap set < regionmap.json

The Ceph Object Gateway logs usage for each user. You can track user usage within date ranges too.

Options include:

radosgw-admin usage show --uid=johndoe --start-date=2012-03-01 --end-date=2012-04-01

radosgw-admin usage show --show-log-entries=false

radosgw-admin usage trim --start-date=2010-01-01 --end-date=2010-12-31
radosgw-admin usage trim --uid=johndoe
radosgw-admin usage trim --uid=johndoe --end-date=2013-12-31

The Ceph Object Gateway supports federated deployments and a global namespace via the notion of regions. A region defines the geographic location of one or more Ceph Object Gateway instances within one or more zones.

Configuring regions differs from typical configuration procedures, because not all of the settings end up in a Ceph configuration file. You can list regions, get a region configuration and set a region configuration.

sudo radosgw-admin regions list

{ "default_info": { "default_region": "default"},
  "regions": [
        "default"]}

sudo radosgw-admin region-map get

radosgw-admin region get [--rgw-region=<region>]

{"name": "default",
 "api_name": "",
 "is_master": "true",
 "endpoints": [],
 "hostnames": [],
 "master_zone": "",
 "zones": [
   {"name": "default",
    "endpoints": [],
    "log_meta": "false",
    "log_data": "false"}
  ],
 "placement_targets": [
   {"name": "default-placement",
    "tags": [] }],
 "default_placement": "default-placement"}

sudo radosgw-admin region set --infile region.json

sudo radosgw-admin region-map update

{
    "regions": [
        {
            "key": "default",
            "val": {
                "name": "default",
                "api_name": "",
                "is_master": "true",
                "endpoints": [],
                "hostnames": [],
                "master_zone": "",
                "zones": [
                    {
                        "name": "default",
                        "endpoints": [],
                        "log_meta": "false",
                        "log_data": "false",
                        "bucket_index_max_shards": 0
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": []
                    }
                ],
                "default_placement": "default-placement"
            }
        }
    ],
    "master_region": "default",
    "bucket_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    }
}

sudo radosgw-admin region-map set --infile regionmap.json

sudo radosgw-admin regionmap update

Ceph Object Gateway supports the notion of zones. A zone defines a logical group consisting of one or more Ceph Object Gateway instances.

Configuring zones differs from typical configuration procedures, because not all of the settings end up in a Ceph configuration file. You can list zones, get a zone configuration and set a zone configuration.

sudo radosgw-admin zone list

sudo radosgw-admin zone get [--rgw-zone=<zone>]

{ "domain_root": ".rgw",
  "control_pool": ".rgw.control",
  "gc_pool": ".rgw.gc",
  "log_pool": ".log",
  "intent_log_pool": ".intent-log",
  "usage_log_pool": ".usage",
  "user_keys_pool": ".users",
  "user_email_pool": ".users.email",
  "user_swift_pool": ".users.swift",
  "user_uid_pool": ".users.uid",
  "system_key": { "access_key": "", "secret_key": ""},
  "placement_pools": [
      {  "key": "default-placement",
         "val": { "index_pool": ".rgw.buckets.index",
                  "data_pool": ".rgw.buckets"}
      }
    ]
  }

sudo radosgw-admin zone set --rgw-zone={zone-name} --infile zone.json

When configuring a default region and zone, the pool name typically leaves off the region and zone names, but you may use any naming convention you prefer. The only requirement is to include a leading period to the region or zone name. For example:

To change the defaults, include the following settings in your Ceph configuration file under each [client.radosgw.{instance-name}] instance.

Ceph zones map to a series of Ceph Storage Cluster pools.

Manually Created Pools vs. Generated Pools

If you provide write capabilities to the user key for the Ceph Object Gateway, the gateway has the ability to create pools automatically. This is convenient, but the Ceph Object Storage Cluster uses the default values for the number of placement groups, which doesn’t have to be ideal, or the values you specified in the Ceph configuration file. If you allow the Ceph Object Gateway to create pools automatically, ensure that you have reasonable defaults for the number of placement groups.

The default pools for the Ceph Object Gateway’s default zone include:

You have significant discretion in determining how you want a zone to access pools. You can create pools on a per zone basis, or use the same pools for multiple zones. As a best practice, Red Hat recommends having a separate set of pools for the master zone and the secondary zones in each region. When creating pools for a specific zone, consider prepending the region name and zone name to the default pool names. For example:

Ceph Object Gateways store data for the bucket index (index_pool) and bucket data (data_pool) in placement pools. These might overlap; you can use the same pool for the index and the data. The index pool for default placement is .rgw.buckets.index and for the data pool for default placement is .rgw.buckets.

To use the Ceph Object Gateway Admin API, you must first:

Amazon’s S3 service uses the access key and a hash of the request header and the secret key to authenticate the request, which has the benefit of providing an authenticated request (especially large uploads) without SSL overhead.

Most use cases for the S3 API involve using open source S3 clients such as the AmazonS3Client in the Amazon SDK for Java or Python Boto. These libraries do not support the Ceph Object Gateway Admin API. You can subclass and extend these libraries to support the Ceph Admin API. Alternatively, you may create your own Gateway client.

The CephAdminAPI example class in this section illustrates how to create an execute() method that can take request parameters, authenticate the request, call the Ceph Admin API and receive a response. The CephAdminAPI class example is not supported or intended for commercial use. It is for illustrative purposes only. The client code contains five calls to the Ceph Object Gateway to demonstrate CRUD operations:

To use this example, you will have to get the Apache HTTP Components, unzip the tar file, navigate to its lib directory and copy the contents to the /jre/lib/ext directory of your JAVA_HOME directory (or a classpath of your choosing).

As you examine the CephAdminAPI class example, notice that the execute() method takes an HTTP method, a request path, an optional subresource (null if not specified) and a map of parameters. To execute with subresources (e.g., subuser, key, etc.), you will need to specify the subresource as an argument in the execute() method.

The example method:

Building the header string is the portion of the process that involves Amazon’s S3 authentication procedure. Specifically, the example method does the following:

The request type should be upper case with no leading or trailing white space. If you do not trim white space, authentication will fail. The date MUST be expressed in GMT, or authentication will fail.

The exemplary method does not have any other headers. The Amazon S3 authentication procedure sorts x-amz headers lexicographically. So if you are adding x-amz headers, be sure to add them lexicographically. See S3 Authentication in this guide for additional details. For a more extensive explanation of the Amazon S3 authentication procedure, consult the Signing and Authenticating REST Requests section of Amazon Simple Storage Service documentation.

Once you have built your header string, the next step is to instantiate an HTTP request and pass it the URI. The examplary method uses PUT for creating a user and subuser, GET for getting a user, POST for modifying a user and DELETE for deleting a user.

Once you instantiate a request, add the Date header followed by the Authorization header. Amazon’s S3 authentication uses the standard Authorization header, and has the following structure:

Authorization: AWS {access-key}:{hash-of-header-and-secret}

The CephAdminAPI example class has a base64Sha1Hmac() method, which takes the header string and the secret key for the admin user, and returns a SHA1 HMAC as a base-64 encoded string. Each execute() call will invoke the same line of code to build the Authorization header:

httpRequest.addHeader("Authorization", "AWS " + this.getAccessKey() + ":" + base64Sha1Hmac(headerString.toString(), this.getSecretKey()));

The following CephAdminAPI example class requires you to pass the access key, secret key and an endpoint to the constructor. The class provides accessor methods to change them at runtime.

import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.time.OffsetDateTime;
import java.time.format.DateTimeFormatter;
import java.time.ZoneId;

import org.apache.http.HttpEntity;
import org.apache.http.NameValuePair;
import org.apache.http.Header;
import org.apache.http.client.entity.UrlEncodedFormEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpRequestBase;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.methods.HttpPut;
import org.apache.http.client.methods.HttpDelete;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.message.BasicNameValuePair;
import org.apache.http.util.EntityUtils;
import org.apache.http.client.utils.URIBuilder;

import java.util.Base64;
import java.util.Base64.Encoder;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import javax.crypto.spec.SecretKeySpec;
import javax.crypto.Mac;

import java.util.Map;
import java.util.Iterator;
import java.util.Set;
import java.util.Map.Entry;

public class CephAdminAPI {

	/*
	 * Each call must specify an access key, secret key, endpoint and format.
	 */
	String accessKey;
	String secretKey;
	String endpoint;
	String scheme = "http"; //http only.
	int port = 80;

	/*
	 * A constructor that takes an access key, secret key, endpoint and format.
	 */
	public CephAdminAPI(String accessKey, String secretKey, String endpoint){
		this.accessKey = accessKey;
		this.secretKey = secretKey;
		this.endpoint = endpoint;
	}

	/*
	 * Accessor methods for access key, secret key, endpoint and format.
	 */
	public String getEndpoint(){
		return this.endpoint;
	}

	public void setEndpoint(String endpoint){
		this.endpoint = endpoint;
	}

	public String getAccessKey(){
		return this.accessKey;
	}

	public void setAccessKey(String accessKey){
		this.accessKey = accessKey;
	}

	public String getSecretKey(){
		return this.secretKey;
	}

	public void setSecretKey(String secretKey){
		this.secretKey = secretKey;
	}

	/*
	 * Takes an HTTP Method, a resource and a map of arguments and
	 * returns a CloseableHTTPResponse.
	 */
	public CloseableHttpResponse execute(String HTTPMethod, String resource,
                                        String subresource, Map arguments) {

		String httpMethod = HTTPMethod;
		String requestPath = resource;
		StringBuffer request = new StringBuffer();
		StringBuffer headerString = new StringBuffer();
		HttpRequestBase httpRequest;
		CloseableHttpClient httpclient;
		URI uri;
		CloseableHttpResponse httpResponse = null;

		try {

			uri = new URIBuilder()
				.setScheme(this.scheme)
				.setHost(this.getEndpoint())
				.setPath(requestPath)
				.setPort(this.port)
				.build();


			if (subresource != null){
				uri = new URIBuilder(uri)
					.setCustomQuery(subresource)
					.build();
			}


			for (Iterator iter = arguments.entrySet().iterator();
			iter.hasNext();) {
				Entry entry = (Entry)iter.next();
				uri = new URIBuilder(uri)
					.setParameter(entry.getKey().toString(),
                                 entry.getValue().toString())
					.build();

			}

			request.append(uri);

			headerString.append(HTTPMethod.toUpperCase().trim() + "\n\n\n");

			OffsetDateTime dateTime = OffsetDateTime.now(ZoneId.of("GMT"));
			DateTimeFormatter formatter = DateTimeFormatter.RFC_1123_DATE_TIME;
			String date = dateTime.format(formatter);

			headerString.append(date + "\n");
			headerString.append(requestPath);

			if (HTTPMethod.equalsIgnoreCase("PUT")){
				httpRequest = new HttpPut(uri);
			} else if (HTTPMethod.equalsIgnoreCase("POST")){
				httpRequest = new HttpPost(uri);
			} else if (HTTPMethod.equalsIgnoreCase("GET")){
				httpRequest = new HttpGet(uri);
			} else if (HTTPMethod.equalsIgnoreCase("DELETE")){
				httpRequest = new HttpDelete(uri);
			} else {
				System.err.println("The HTTP Method must be PUT,
				POST, GET or DELETE.");
				throw new IOException();
			}

			httpRequest.addHeader("Date", date);
			httpRequest.addHeader("Authorization", "AWS " + this.getAccessKey()
			+ ":" + base64Sha1Hmac(headerString.toString(),
			this.getSecretKey()));

			httpclient = HttpClients.createDefault();
			httpResponse = httpclient.execute(httpRequest);

		} 	catch  (URISyntaxException e){
			System.err.println("The URI is not formatted properly.");
			e.printStackTrace();
		}  catch (IOException e){
			System.err.println("There was an error making the request.");
			e.printStackTrace();
		}
			return httpResponse;
	}

	/*
	 * Takes a uri and a secret key and returns a base64-encoded
	 * SHA-1 HMAC.
	 */
	public String base64Sha1Hmac(String uri, String secretKey) {
		try {

			byte[] keyBytes = secretKey.getBytes("UTF-8");
			SecretKeySpec signingKey = new SecretKeySpec(keyBytes, "HmacSHA1");

			Mac mac = Mac.getInstance("HmacSHA1");
			mac.init(signingKey);

			byte[] rawHmac = mac.doFinal(uri.getBytes("UTF-8"));

			Encoder base64 = Base64.getEncoder();
			return base64.encodeToString(rawHmac);

		} catch (Exception e) {
			throw new RuntimeException(e);
		}
	}

}

The subsequent CephAdminAPIClient example illustrates how to instantiate the CephAdminAPI class, build a map of request parameters, and use the execute() method to create, get, update and delete a user.

import java.io.IOException;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.HttpEntity;
import org.apache.http.util.EntityUtils;
import java.util.*;


public class CephAdminAPIClient {

	public static void main (String[] args){

		CephAdminAPI adminApi = new CephAdminAPI ("FFC6ZQ6EMIF64194158N",
		                            "Xac39eCAhlTGcCAUreuwe1ZuH5oVQFa51lbEMVoT",
		                            "ceph-client");

		/*
		 * Create a user
		 */
		Map requestArgs = new HashMap();
		requestArgs.put("access", "usage=read, write; users=read, write");
		requestArgs.put("display-name", "New User");
		requestArgs.put("email", "new-user@email.com");
		requestArgs.put("format", "json");
		requestArgs.put("uid", "new-user");

		CloseableHttpResponse response =
			adminApi.execute("PUT", "/admin/user", null, requestArgs);

		System.out.println(response.getStatusLine());
		HttpEntity entity = response.getEntity();

		try {
			System.out.println("\nResponse Content is: "
				+ EntityUtils.toString(entity, "UTF-8") + "\n");
			response.close();
		} catch (IOException e){
			System.err.println ("Encountered an I/O exception.");
			e.printStackTrace();
		}

		/*
		 * Get a user
		 */
		requestArgs = new HashMap();
		requestArgs.put("format", "json");
		requestArgs.put("uid", "new-user");

		response = adminApi.execute("GET", "/admin/user", null, requestArgs);

		System.out.println(response.getStatusLine());
		entity = response.getEntity();

		try {
			System.out.println("\nResponse Content is: "
				+ EntityUtils.toString(entity, "UTF-8") + "\n");
			response.close();
		} catch (IOException e){
			System.err.println ("Encountered an I/O exception.");
			e.printStackTrace();
		}

		/*
		 * Modify a user
		 */
		requestArgs = new HashMap();
		requestArgs.put("display-name", "John Doe");
		requestArgs.put("email", "johndoe@email.com");
		requestArgs.put("format", "json");
		requestArgs.put("uid", "new-user");
		requestArgs.put("max-buckets", "100");

		response = adminApi.execute("POST", "/admin/user", null, requestArgs);

		System.out.println(response.getStatusLine());
		entity = response.getEntity();

		try {
			System.out.println("\nResponse Content is: "
				+ EntityUtils.toString(entity, "UTF-8") + "\n");
			response.close();
		} catch (IOException e){
			System.err.println ("Encountered an I/O exception.");
			e.printStackTrace();
		}


		/*
		 * Create a subuser
		 */
		requestArgs = new HashMap();
		requestArgs.put("format", "json");
		requestArgs.put("uid", "new-user");
		requestArgs.put("subuser", "foobar");

		response = adminApi.execute("PUT", "/admin/user", "subuser", requestArgs);
		System.out.println(response.getStatusLine());
		entity = response.getEntity();

		try {
			System.out.println("\nResponse Content is: "
				+ EntityUtils.toString(entity, "UTF-8") + "\n");
			response.close();
		} catch (IOException e){
			System.err.println ("Encountered an I/O exception.");
			e.printStackTrace();
		}


		/*
		 * Delete a user
		 */
		requestArgs = new HashMap();
		requestArgs.put("format", "json");
		requestArgs.put("uid", "new-user");

		response = adminApi.execute("DELETE", "/admin/user", null, requestArgs);
		System.out.println(response.getStatusLine());
		entity = response.getEntity();

		try {
			System.out.println("\nResponse Content is: "
				+ EntityUtils.toString(entity, "UTF-8") + "\n");
			response.close();
		} catch (IOException e){
			System.err.println ("Encountered an I/O exception.");
			e.printStackTrace();
		}
	}
}

An admin API request will be done on a URI that starts with the configurable 'admin' resource entry point. Authorization for the admin API duplicates the S3 authorization mechanism. Some operations require that the user holds special administrative capabilities. The response entity type (XML or JSON) may be specified as the 'format' option in the request and defaults to JSON if not specified.

GET /admin/usage?format=json HTTP/1.1
Host: {fqdn}

DELETE /admin/usage?format=json HTTP/1.1
Host: {fqdn}

GET /admin/user?format=json HTTP/1.1
Host: {fqdn}

PUT /admin/user?format=json HTTP/1.1
Host: {fqdn}

POST /admin/user?format=json HTTP/1.1
Host: {fqdn}

DELETE /admin/user?format=json HTTP/1.1
Host: {fqdn}

PUT /admin/user?subuser&format=json HTTP/1.1
Host {fqdn}

POST /admin/user?subuser&format=json HTTP/1.1
Host {fqdn}

DELETE /admin/user?subuser&format=json HTTP/1.1
Host {fqdn}

PUT /admin/user?key&format=json HTTP/1.1
Host {fqdn}

DELETE /admin/user?key&format=json HTTP/1.1
Host {fqdn}

GET /admin/bucket?format=json HTTP/1.1
Host {fqdn}

GET /admin/bucket?index&format=json HTTP/1.1
Host {fqdn}

DELETE /admin/bucket?format=json HTTP/1.1
Host {fqdn}

POST /admin/bucket?format=json HTTP/1.1
Host {fqdn}

PUT /admin/bucket?format=json HTTP/1.1
Host {fqdn}

DELETE /admin/bucket?object&format=json HTTP/1.1
Host {fqdn}

GET /admin/bucket?policy&format=json HTTP/1.1
Host {fqdn}

PUT /admin/user?caps&format=json HTTP/1.1
Host {fqdn}

PUT /admin/user?caps&format=json HTTP/1.1
Host: {fqdn}
Content-Type: text/plain
Authorization: {your-authorization-token}

usage=read

DELETE /admin/user?caps&format=json HTTP/1.1
Host {fqdn}

GET /admin/user?quota&uid=<uid>&quota-type=user

PUT /admin/user?quota&uid=<uid>&quota-type=user

GET /admin/user?quota&uid=<uid>&quota-type=bucket

PUT /admin/user?quota&uid=<uid>&quota-type=bucket

To set up an HA Proxy with the Ceph Object Gateway, you must have:

For a detailed discussion of HAProxy and keepalived, see Load Balancer Administration.

The following setup assumes two HAProxy nodes named haproxy and haproxy2, and two Ceph Object Gateway servers named rgw1 and rgw2. You may use any naming convention you prefer. Perform the following procedure on your at least two HAProxy nodes:

Perform the following procedure on your at least two HAProxy nodes:

Perform the following procedure on your at least two HAProxy nodes:

On your HAProxy nodes, check to ensure the virtual IP address from your keepalived configuration appears.

ip addr show

On your calamari node, see if you can reach the gateway nodes via the load balancer configuration. For example:

wget haproxy

This should return the same result as:

wget rgw1

If it returns an index.html file with the following contents:

<?xml version="1.0" encoding="UTF-8"?>
	<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
		<Owner>
			<ID>anonymous</ID>
			<DisplayName></DisplayName>
		</Owner>
		<Buckets>
		</Buckets>
	</ListAllMyBucketsResult>

Then, your configuration is working properly.

Requests to the Ceph Object Gateway can be either authenticated or unauthenticated. Ceph Object Gateway assumes unauthenticated requests are sent by an anonymous user. Ceph Object Gateway supports canned ACLs.

HTTP/1.1
PUT /buckets/bucket/object.mpeg
Host: cname.domain.com
Date: Mon, 2 Jan 2012 00:01:01 +0000
Content-Encoding: mpeg
Content-Length: 9999999

Authorization: AWS {access-key}:{hash-of-header-and-secret}