Object Gateway Guide

Red Hat Ceph Storage 5

Deploying, configuring, and administering a Ceph Object Gateway

Red Hat Ceph Storage Documentation Team

Abstract

This document provides guidance on deploying, configuring, and administering a Ceph Object Gateway environment. This guide uses a "Day Zero", "Day One", and "Day Two" organizational methodology, providing readers with a logical progression path. Day Zero is where research and planning are done before implementing a potential solution, see Chapters 1 and 2. Day One is where the actual deployment, and installation of the software happens, see Chapter 3. Day Two is where all the basic, and advanced configuration happens, see Chapters 4, 5, and 6.
Red Hat is committed to replacing problematic language in our code, documentation, and web properties. We are beginning with these four terms: master, slave, blacklist, and whitelist. Because of the enormity of this endeavor, these changes will be implemented gradually over several upcoming releases. For more details, see our CTO Chris Wright's message

Chapter 1. The Ceph Object Gateway

Ceph Object Gateway, also known as RADOS Gateway (RGW), is an object storage interface built on top of librados library to provide applications with a RESTful gateway to Ceph storage clusters. Ceph Object Gateway supports two interfaces:

S3-compatibility:

Provides object storage functionality with an interface that is compatible with a large subset of the Amazon S3 RESTful API.

Swift-compatibility:

Provides object storage functionality with an interface that is compatible with a large subset of the OpenStack Swift API.

The Ceph Object Gateway is a service interacting with a Ceph storage cluster. Since it provides interfaces compatible with OpenStack Swift and Amazon S3, the Ceph Object Gateway has its own user management system. Ceph Object Gateway can store data in the same Ceph storage cluster used to store data from Ceph block device clients; however, it would involve separate pools and likely a different CRUSH hierarchy. The S3 and Swift APIs share a common namespace, so you can write data with one API and retrieve it with the other.

Administrative API:

Provides an administrative interface for managing the Ceph Object Gateways.

Administrative API requests are done on a URI that starts with the admin resource end point. Authorization for the administrative API mimics the S3 authorization convention. Some operations require the user to have special administrative capabilities. The response type can be either XML or JSON by specifying the format option in the request, but defaults to the JSON format.

Basic Access Diagram

Additional Resources

Chapter 2. Considerations and recommendations

As a storage administrator, you can have a basic understanding about what things to consider before running a Ceph Object Gateway. Also, the things to consider before implementing a multi-site Ceph Object Gateway solution. Understanding such things better prepares you for knowing the hardware and network requirements, understanding what type of workloads work well with a Ceph Object Gateway, and Red Hat’s recommendations.

2.1. Prerequisites

  • Time to understand, consider, and plan a storage solution.

2.2. Basic Red Hat Ceph Storage considerations

The first consideration for using Red Hat Ceph Storage is developing a storage strategy for the data. A storage strategy is a method of storing data that serves a particular use case. If you need to store volumes and images for a cloud platform like OpenStack, you can choose to store data on faster Serial Attached SCSI (SAS) drives with Solid State Drives (SSD) for journals. By contrast, if you need to store object data for an S3- or Swift-compliant gateway, you can choose to use something more economical, like traditional Serial Advanced Technology Attachment (SATA) drives. Red Hat Ceph Storage can accommodate both scenarios in the same storage cluster, but you need a means of providing the fast storage strategy to the cloud platform, and a means of providing more traditional storage for your object store.

One of the most important steps in a successful Ceph deployment is identifying a price-to-performance profile suitable for the storage cluster’s use case and workload. It is important to choose the right hardware for the use case. For example, choosing IOPS-optimized hardware for a cold storage application increases hardware costs unnecessarily. Whereas, choosing capacity-optimized hardware for its more attractive price point in an IOPS-intensive workload will likely lead to unhappy users complaining about slow performance.

Red Hat Ceph Storage can support multiple storage strategies. Use cases, cost versus benefit performance tradeoffs, and data durability are the primary considerations that help develop a sound storage strategy.

Use Cases

Ceph provides massive storage capacity, and it supports numerous use cases, such as:

  • The Ceph Block Device client is a leading storage backend for cloud platforms that provides limitless storage for volumes and images with high performance features like copy-on-write cloning.
  • The Ceph Object Gateway client is a leading storage backend for cloud platforms that provides a RESTful S3-compliant and Swift-compliant object storage for objects like audio, bitmap, video and other data.
  • The Ceph File System for traditional file storage.

Cost vs. Benefit of Performance

Faster is better. Bigger is better. High durability is better. However, there is a price for each superlative quality, and a corresponding cost versus benefit trade off. Consider the following use cases from a performance perspective: SSDs can provide very fast storage for relatively small amounts of data and journaling. Storing a database or object index can benefit from a pool of very fast SSDs, but proves too expensive for other data. SAS drives with SSD journaling provide fast performance at an economical price for volumes and images. SATA drives without SSD journaling provide cheap storage with lower overall performance. When you create a CRUSH hierarchy of OSDs, you need to consider the use case and an acceptable cost versus performance trade off.

Data Durability

In large scale storage clusters, hardware failure is an expectation, not an exception. However, data loss and service interruption remain unacceptable. For this reason, data durability is very important. Ceph addresses data durability with multiple replica copies of an object or with erasure coding and multiple coding chunks. Multiple copies or multiple coding chunks present an additional cost versus benefit tradeoff: it is cheaper to store fewer copies or coding chunks, but it can lead to the inability to service write requests in a degraded state. Generally, one object with two additional copies, or two coding chunks can allow a storage cluster to service writes in a degraded state while the storage cluster recovers.

Replication stores one or more redundant copies of the data across failure domains in case of a hardware failure. However, redundant copies of data can become expensive at scale. For example, to store 1 petabyte of data with triple replication would require a cluster with at least 3 petabytes of storage capacity.

Erasure coding stores data as data chunks and coding chunks. In the event of a lost data chunk, erasure coding can recover the lost data chunk with the remaining data chunks and coding chunks. Erasure coding is substantially more economical than replication. For example, using erasure coding with 8 data chunks and 3 coding chunks provides the same redundancy as 3 copies of the data. However, such an encoding scheme uses approximately 1.5x of the initial data stored compared to 3x with replication.

The CRUSH algorithm aids this process by ensuring that Ceph stores additional copies or coding chunks in different locations within the storage cluster. This ensures that the failure of a single storage device or node does not lead to a loss of all of the copies or coding chunks necessary to preclude data loss. You can plan a storage strategy with cost versus benefit tradeoffs, and data durability in mind, then present it to a Ceph client as a storage pool.

Important

ONLY the data storage pool can use erasure coding. Pools storing service data and bucket indexes use replication.

Important

Ceph’s object copies or coding chunks make RAID solutions obsolete. Do not use RAID, because Ceph already handles data durability, a degraded RAID has a negative impact on performance, and recovering data using RAID is substantially slower than using deep copies or erasure coding chunks.

Additional Resources

2.3. Red Hat Ceph Storage workload considerations

One of the key benefits of a Ceph storage cluster is the ability to support different types of workloads within the same storage cluster using performance domains. Different hardware configurations can be associated with each performance domain. Storage administrators can deploy storage pools on the appropriate performance domain, providing applications with storage tailored to specific performance and cost profiles. Selecting appropriately sized and optimized servers for these performance domains is an essential aspect of designing a Red Hat Ceph Storage cluster.

To the Ceph client interface that reads and writes data, a Ceph storage cluster appears as a simple pool where the client stores data. However, the storage cluster performs many complex operations in a manner that is completely transparent to the client interface. Ceph clients and Ceph object storage daemons, referred to as Ceph OSDs, or simply OSDs, both use the Controlled Replication Under Scalable Hashing (CRUSH) algorithm for storage and retrieval of objects. Ceph OSDs can run on bare-metal servers, or in containers within the storage cluster.

A CRUSH map describes a topography of cluster resources, and the map exists both on client nodes as well as Ceph Monitor nodes within the cluster. Ceph clients and Ceph OSDs both use the CRUSH map and the CRUSH algorithm. Ceph clients communicate directly with OSDs, eliminating a centralized object lookup and a potential performance bottleneck. With awareness of the CRUSH map and communication with their peers, OSDs can handle replication, backfilling, and recovery—allowing for dynamic failure recovery.

Ceph uses the CRUSH map to implement failure domains. Ceph also uses the CRUSH map to implement performance domains, which simply take the performance profile of the underlying hardware into consideration. The CRUSH map describes how Ceph stores data, and it is implemented as a simple hierarchy, specifically a acyclic graph, and a ruleset. The CRUSH map can support multiple hierarchies to separate one type of hardware performance profile from another. Ceph implements performance domains with device "classes".

For example, you can have these performance domains coexisting in the same Red Hat Ceph Storage cluster:

  • Hard disk drives (HDDs) are typically appropriate for cost- and capacity-focused workloads.
  • Throughput-sensitive workloads typically use HDDs with Ceph write journals on solid state drives (SSDs).
  • IOPS-intensive workloads such as MySQL and MariaDB often use SSDs.

Figure 2.1. Performance and Failure Domains

perf and failure domains diagram

Workloads

Red Hat Ceph Storage is optimized for three primary workloads.

Important

Carefully consider the workload being ran by Red Hat Ceph Storage clusters BEFORE considering what hardware to purchase, because it can significantly impact the price and performance of the storage cluster. For example, if the workload is capacity-optimized and the hardware is better suited to a throughput-optimized workload, then hardware will be more expensive than necessary. Conversely, if the workload is throughput-optimized and the hardware is better suited to a capacity-optimized workload, then storage cluster can suffer from poor performance.

  • IOPS optimized: Input, output per second (IOPS) optimization deployments are suitable for cloud computing operations, such as running MYSQL or MariaDB instances as virtual machines on OpenStack. IOPS optimized deployments require higher performance storage such as 15k RPM SAS drives and separate SSD journals to handle frequent write operations. Some high IOPS scenarios use all flash storage to improve IOPS and total throughput.

    An IOPS-optimized storage cluster has the following properties:

    • Lowest cost per IOPS.
    • Highest IOPS per GB.
    • 99th percentile latency consistency.

    Uses for an IOPS-optimized storage cluster are:

    • Typically block storage.
    • 3x replication for hard disk drives (HDDs) or 2x replication for solid state drives (SSDs).
    • MySQL on OpenStack clouds.
  • Throughput optimized: Throughput-optimized deployments are suitable for serving up significant amounts of data, such as graphic, audio and video content. Throughput-optimized deployments require high bandwidth networking hardware, controllers and hard disk drives with fast sequential read and write characteristics. If fast data access is a requirement, then use a throughput-optimized storage strategy. Also, if fast write performance is a requirement, using Solid State Disks (SSD) for journals will substantially improve write performance.

    A throughput-optimized storage cluster has the following properties:

    • Lowest cost per MBps (throughput).
    • Highest MBps per TB.
    • Highest MBps per BTU.
    • Highest MBps per Watt.
    • 97th percentile latency consistency.

    Uses for an throughput-optimized storage cluster are:

    • Block or object storage.
    • 3x replication.
    • Active performance storage for video, audio, and images.
    • Streaming media, such as 4k video.
  • Capacity optimized: Capacity-optimized deployments are suitable for storing significant amounts of data as inexpensively as possible. Capacity-optimized deployments typically trade performance for a more attractive price point. For example, capacity-optimized deployments often use slower and less expensive SATA drives and co-locate journals rather than using SSDs for journaling.

    A cost- and capacity-optimized storage cluster has the following properties:

    • Lowest cost per TB.
    • Lowest BTU per TB.
    • Lowest Watts required per TB.

    Uses for an cost- and capacity-optimized storage cluster are:

    • Typically object storage.
    • Erasure coding for maximizing usable capacity
    • Object archive.
    • Video, audio, and image object repositories.

2.4. Ceph Object Gateway considerations

Another important aspect of designing a storage cluster is to determine if the storage cluster will be in one data center site or span multiple data center sites. Multi-site storage clusters benefit from geographically distributed failover and disaster recovery, such as long-term power outages, earthquakes, hurricanes, floods or other disasters. Additionally, multi-site storage clusters can have an active-active configuration, which can direct client applications to the closest available storage cluster. This is good storage strategy for content delivery networks. Consider placing data as close to the client as possible. This is important for throughput-intensive workloads, such as, streaming 4k video.

Important

Red Hat recommends identifying realm, zone group and zone names BEFORE creating Ceph’s storage pools. Prepend some pool names with the zone name as a standard naming convention.

Additional Resources

2.5. Ceph Object Gateway multi-site considerations

A Ceph Object Gateway multi-site configuration requires at least two Red Hat Ceph Storage clusters, and at least two Ceph object gateway instances, one for each Red Hat Ceph Storage cluster. Typically, the two Red Hat Ceph Storage clusters will be in geographically separate locations; however, this same multi-site configuration can work on two Red Hat Ceph Storage clusters located at the same physical site.

Multi-site configurations require a primary zone group and a primary zone. Additionally, each zone group requires a primary zone. Zone groups might have one or more secondary zones.

Important

The primary zone within the primary zone group of a realm is responsible for storing the primary copy of the realm’s metadata, including users, quotas, and buckets. This metadata gets synchronized to secondary zones and secondary zone groups automatically. Metadata operations executed with the radosgw-admin command-line interface (CLI) MUST be executed on a node within the primary zone of the primary zone group to ensure that they synchronize to the secondary zone groups and zones. Currently, it is possible to execute metadata operations on secondary zones and zone groups, but it is NOT recommended because they WILL NOT be synchronized, which can lead to fragmentation of the metadata.

The diagrams below illustrate the possible one, and two realm configurations in multi-site Ceph Object Gateway environments.

Figure 2.2. One Realm

mulitsite one realm

Figure 2.3. Two Realms

mulitsite two realm

Figure 2.4. Two Realms Variant

mulitsite two realm variant

2.6. Considerations for using NVMe with Object Gateway

Optionally, you can consider using NVMe for the Ceph Object Gateway.

If you plan to use the object gateway feature of Red Hat Ceph Storage and the OSD nodes are using NVMe-based SSDs, then consider following the procedures found in the Using NVMe with LVM optimally section of the Ceph Object Gateway for Production Guide. These procedures explain how to use specially designed Ansible playbooks which will place journals and bucket indexes together on SSDs, which can increase performance compared to having all journals on one device.

2.7. Network considerations for Red Hat Ceph Storage

An important aspect of a cloud storage solution is that storage clusters can run out of IOPS due to network latency, and other factors. Also, the storage cluster can run out of throughput due to bandwidth constraints long before the storage clusters run out of storage capacity. This means that the network hardware configuration must support the chosen workloads in order to meet price versus performance requirements.

Storage administrators prefer that a storage cluster recovers as quickly as possible. Carefully consider bandwidth requirements for the storage cluster network, be mindful of network link oversubscription, and segregate the intra-cluster traffic from the client-to-cluster traffic. Also consider that network performance is increasingly important when considering the use of Solid State Disks (SSD), flash, NVMe, and other high performing storage devices.

Ceph supports a public network and a storage cluster network. The public network handles client traffic and communication with Ceph Monitors. The storage cluster network handles Ceph OSD heartbeats, replication, backfilling and recovery traffic. At a minimum, a single 10 GB Ethernet link should be used for storage hardware, and you can add additional 10 GB Ethernet links for connectivity and throughput.

Important

Red Hat recommends allocating bandwidth to the storage cluster network, such that, it is a multiple of the public network using osd_pool_default_size as the basis for the multiple on replicated pools. Red Hat also recommends running the public and storage cluster networks on separate network cards.

Important

Red Hat recommends using 10 GB Ethernet for Red Hat Ceph Storage deployments in production. A 1 GB Ethernet network is not suitable for production storage clusters.

In the case of a drive failure, replicating 1 TB of data across a 1 GB Ethernet network takes 3 hours, and 3 TB takes 9 hours. Using 3 TB is the typical drive configuration. By contrast, with a 10 GB Ethernet network, the replication times would be 20 minutes and 1 hour respectively. Remember that when a Ceph OSD fails, the storage cluster will recover by replicating the data it contained to other Ceph OSDs within the pool.

The failure of a larger domain such as a rack means that the storage cluster will utilize considerably more bandwidth. When building a storage cluster consisting of multiple racks, which is common for large storage implementations, consider utilizing as much network bandwidth between switches in a "fat tree" design for optimal performance. A typical 10 GB Ethernet switch has 48 10 GB ports and four 40 GB ports. Use the 40 GB ports on the spine for maximum throughput. Alternatively, consider aggregating unused 10 GB ports with QSFP+ and SFP+ cables into more 40 GB ports to connect to other rack and spine routers. Also, consider using LACP mode 4 to bond network interfaces. Additionally, use jumbo frames, maximum transmission unit (MTU) of 9000, especially on the backend or cluster network.

Before installing and testing a Red Hat Ceph Storage cluster, verify the network throughput. Most performance-related problems in Ceph usually begin with a networking issue. Simple network issues like a kinked or bent Cat-6 cable could result in degraded bandwidth. Use a minimum of 10 GB ethernet for the front side network. For large clusters, consider using 40 GB ethernet for the backend or cluster network.

Important

For network optimization, Red Hat recommends using jumbo frames for a better CPU per bandwidth ratio, and a non-blocking network switch back-plane. Red Hat Ceph Storage requires the same MTU value throughout all networking devices in the communication path, end-to-end for both public and cluster networks. Verify that the MTU value is the same on all nodes and networking equipment in the environment before using a Red Hat Ceph Storage cluster in production.

Additional Resources

2.8. Tuning considerations for the Linux kernel when running Ceph

Production Red Hat Ceph Storage clusters generally benefit from tuning the operating system, specifically around limits and memory allocation. Ensure that adjustments are set for all nodes within the storage cluster. You can also open a case with Red Hat support asking for additional guidance.

Reserving Free Memory for Ceph OSDs

To help prevent insufficient memory-related errors during Ceph OSD memory allocation requests, set the specific amount of physical memory to keep in reserve. Red Hat recommends the following settings based on the amount of system RAM.

  • For 64 GB, reserve 1 GB:

    vm.min_free_kbytes = 1048576
  • For 128 GB, reserve 2 GB:

    vm.min_free_kbytes = 2097152
  • For 256 GB, reserve 3 GB:

    vm.min_free_kbytes = 3145728

Increase the File Descriptors

The Ceph Object Gateway can hang if it runs out of file descriptors. You can modify the /etc/security/limits.conf file on Ceph Object Gateway nodes to increase the file descriptors for the Ceph Object Gateway.

ceph       soft    nofile     unlimited

Adjusting the ulimit value for Large Storage Clusters

When running Ceph administrative commands on large storage clusters, ​for example, with 1024 Ceph OSDs or more, ​create an /etc/security/limits.d/50-ceph.conf file on each node that runs administrative commands with the following contents:

USER_NAME       soft    nproc     unlimited

Replace USER_NAME with the name of the non-root user account that runs the Ceph administrative commands.

Note

The root user’s ulimit value is already set to unlimited by default on Red Hat Enterprise Linux.

2.9. Considering Storage Sizing

One of the most important factors in designing a cluster is to determine the storage requirements (sizing). Ceph Storage is designed to scale into petabytes and beyond. The following examples are common sizes for Ceph storage clusters.

  • Small: 250 terabytes
  • Medium: 1 petabyte
  • Large: 2 petabytes or more.

Sizing includes current needs and the needs of the near future. Consider the rate at which the gateway client will add new data to the cluster. That can differ from use-case to use-case. For example, recording CCTV video, 4k video or medical imaging may add significant amounts of data far more quickly then less storage intensive information such as financial market data. Additionally, considering the data durability methods such as replication versus erasure coding can have a significant impact on the storage media required.

For additional information on sizing, see the Red Hat Ceph Storage Hardware Selection Guide and its associated links for selecting OSD hardware.

2.10. Considering Storage Density

Another important aspect of cluster design includes storage density. Generally, a cluster should store data across at least 10 nodes to ensure reasonable performance when replicating, backfilling and recovery. If a node fails, with at least 10 nodes in the cluster, only 10% of the data has to move to the surviving nodes. If the number of nodes is substantially less, a higher percentage of the data must move to the surviving nodes. Additionally, the full_ratio and near_full_ratio need to be set to accommodate a node failure to ensure that the cluster can write data. For this reason, it is is important to consider storage density. Higher storage density isn’t necessarily a good idea.

Another factor that favors more nodes over higher storage density is erasure coding. When writing an object using erasure coding and using node as the minimum CRUSH failure domain, the cluster will need as many nodes as data and coding chunks. For example, a cluster using k=8, m=3 should have at least 11 nodes so that each data or coding chunk is stored on a separate node.

Hot-swapping is also an important consideration. Most modern servers support drive hot-swapping. However, some hardware configurations require removing more than one drive to replace a drive. Red Hat recommends avoiding such configurations, because they can bring down more OSDs than required when swapping out failed disks.

2.11. Selecting Media for Indexes

When selecting OSD hardware for use with a Ceph Object Gateway--​irrespective of the use case--​it is required to have an OSD node that has at least one high performance drive for storing the index pool. This is particularly important when buckets contain a large number of objects.

The host should have at least one SSD or NVMe drive. In Red Hat lab testing, NVMe drives have shown enough performance to support both OSD journals and index pools on the same drive, but in different partitions of the NVMe drive.

Note

Red Hat does not support HDD devices for index pools. For more information on supported configurations, see the Red Hat Ceph Storage: Supported configurations article.

An index entry is approximately 200 bytes of data, stored as an object map (omap) in leveldb. While this is a trivial amount of data, some uses of Ceph Object Gateway can result in tens or hundreds of millions of objects in a single bucket. By mapping the index pool to a CRUSH hierarchy of high performance storage media, the reduced latency provides a dramatic performance improvement when buckets contain very large numbers of objects.

Important

In a production cluster, a typical OSD node will have at least one SSD or NVMe drive for storing the OSD journal and the index pool, which will use separate partitions or logical volumes when using the same physical drive.

2.12. Selecting Media for Monitor Nodes

Ceph monitors use leveldb, which is sensitive to synchronous write latency. Red Hat strongly recommends using SSDs to store monitor data. Ensure that the selected SSDs have sufficient sequential write and throughput characteristics.

2.13. Adjusting Backfill & Recovery Settings

I/O is negatively impacted by both backfilling and recovery operations, leading to poor performance and unhappy end users. To help accommodate I/O demand during a cluster expansion or recovery, set the following options and values in the Ceph Configuration file:

[osd]
osd_max_backfills = 1
osd_recovery_max_active = 1
osd_recovery_op_priority = 1

2.14. Adjusting the Cluster Map Size

For Red Hat Ceph Storage version 2 and earlier, when the cluster has thousands of OSDs, download the cluster map and check its file size. By default, the ceph-osd daemon caches 500 previous osdmaps. Even with deduplication, the map may consume a lot of memory per daemon. Tuning the cache size in the Ceph configuration file may help reduce memory consumption significantly. For example:

[global]
osd_map_message_max=10

[osd]
osd_map_cache_size=20
osd_map_max_advance=10
osd_map_share_max_epochs=10
osd_pg_epoch_persisted_max_stale=10

For Red Hat Ceph Storage version 3 and later, the ceph-manager daemon handles PG queries, so the cluster map should not impact performance.

2.15. Adjusting Scrubbing

By default, Ceph performs light scrubbing daily and deep scrubbing weekly. Light scrubbing checks object sizes and checksums to ensure that PGs are storing the same object data. Over time, disk sectors can go bad irrespective of object sizes and checksums. Deep scrubbing checks an object’s content with that of its replicas to ensure that the actual contents are the same. In this respect, deep scrubbing ensures data integrity in the manner of fsck, but the procedure imposes an I/O penalty on the cluster. Even light scrubbing can impact I/O.

The default settings may allow Ceph OSDs to initiate scrubbing at inopportune times such as peak operating times or periods with heavy loads. End users may experience latency and poor performance when scrubbing operations conflict with end user operations.

To prevent end users from experiencing poor performance, Ceph provides a number of scrubbing settings that can limit scrubbing to periods with lower loads or during off-peak hours. For details, see the Scrubbing section in the Configuration Guide for Red Hat Ceph Storage 5.

If the cluster experiences high loads during the day and low loads late at night, consider restricting scrubbing to night time hours. For example:

[osd]
osd_scrub_begin_hour = 23   #23:01H, or 10:01PM.
osd_scrub_end_hour = 6      #06:01H or 6:01AM.

If time constraints aren’t an effective method of determining a scrubbing schedule, consider using the osd_scrub_load_threshold. The default value is 0.5, but it could be modified for low load conditions. For example:

[osd]
osd_scrub_load_threshold = 0.25

2.16. Increase objecter_inflight_ops

In RHCS 3.0 and earlier releases, consider increasing objecter_inflight_ops to the default size for version 3.1 and later releases to improve scalability.

objecter_inflight_ops = 24576

2.17. Increase rgw_thread_pool_size

In RHCS 3.0 and earlier releases, consider increasing rgw_thread_pool_size to the default size for version 3.1 and later releases to improve scalability. For example:

rgw_thread_pool_size = 512

2.18. Additional Resources

Chapter 3. Deployment

As a storage administrator, you can deploy the Ceph Object Gateway using the Ceph Orchestrator by using the command line interface or by using the service specification. You can also configure multisite Ceph Object Gateways, and remove the Ceph Object Gateway using the Ceph Orchestrator.

The cephadm command deploys the Ceph Object Gateway as a collection of daemons that manages a single-cluster deployment or a particular realm and zone in a multisite deployment.

Note

With cephadm, the Ceph Object Gateway daemons are configured using the Ceph Monitor configuration database instead of the ceph.conf file or the command line options. If the configuration is not in the client.rgw section, then the Ceph Object Gateway daemons starts up with default settings and binds to the port 80.

This section covers the following administrative tasks:

3.1. Prerequisites

  • A running, and healthy Red Hat Ceph Storage cluster.
  • Root-level access to all the nodes.
  • Available nodes the storage cluster.
  • All the managers, monitors, and OSDs are deployed in the storage cluster.

3.2. Deploying the Ceph Object Gateway using the command-line interface

Using the Ceph Orchestrator, you can deploy the Ceph object gateway using the ceph orch command in the command line interface

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to all the nodes.
  • Hosts are added to the cluster.
  • All manager, monitor and OSD daemons are deployed.

Procedure

  1. Log into the Cephadm shell:

    Example

    [root@host01 ~]# cephadm shell

  2. You can deploy the Ceph object gateway daemons in three different ways:

Method 1

  • Create realm, zone group, zone and then use the placement specification with the host name:

    1. Create a realm:

      Syntax

      radosgw-admin realm create --rgw-realm=REALM_NAME --default

      Example

      [ceph: root@host01 /]# radosgw-admin realm create --rgw-realm=test_realm --default

    2. Create a zone group:

      Syntax

      radosgw-admin zonegroup create --rgw-zonegroup=ZONE_GROUP_NAME  --master --default

      Example

      [ceph: root@host01 /]# radosgw-admin zonegroup create --rgw-zonegroup=default  --master --default

    3. Create a zone:

      Syntax

      radosgw-admin zone create --rgw-zonegroup=ZONE_GROUP_NAME --rgw-zone=ZONE_NAME --master --default

      Example

      [ceph: root@host01 /]# radosgw-admin zone create --rgw-zonegroup=default --rgw-zone=test_zone --master --default

    4. Commit the changes:

      Syntax

      radosgw-admin period update --rgw-realm=REALM_NAME --commit

      Example

      [ceph: root@host01 /]# radosgw-admin period update --rgw-realm=test_realm --commit

    5. Run the ceph orch apply command:

      Syntax

      ceph orch apply rgw NAME [--realm=REALM_NAME] [--zone=ZONE_NAME] --placement="NUMBER_OF_DAEMONS [HOST_NAME_1 HOST_NAME_2]"

      Example

      [ceph: root@host01 /]# ceph orch apply rgw test --realm=test_realm --zone=test_zone --placement="2 host01 host02"

Method 2

  • Use an arbitrary service name to deploy two Ceph object gateway daemons for a single cluster deployment:

    Syntax

    ceph orch apply rgw SERVICE_NAME

    Example

    [ceph: root@host01 /]# ceph orch apply rgw foo

Method 3

  • Use an arbitrary service name on a labeled set of hosts:

    Syntax

    ceph orch host label add HOST_NAME_1 LABEL_NAME
    ceph orch host label add HOSTNAME_2 LABEL_NAME
    ceph orch apply rgw SERVICE_NAME '--placement=label:_LABEL_NAME_ count-per-host:_NUMBER_OF_DAEMONS_' --port=8000

    Example

    [ceph: root@host01 /]# ceph orch host label add host01 rgw  # the 'rgw' label can be anything
    [ceph: root@host01 /]# ceph orch host label add host02 rgw
    [ceph: root@host01 /]# ceph orch apply rgw foo '--placement=label:rgw count-per-host:2' --port=8000

Verification

  • List the service:

    Example

    [ceph: root@host01 /]# ceph orch ls

  • List the hosts, daemons, and processes:

    Syntax

    ceph orch ps --daemon_type=DAEMON_NAME

    Example

    [ceph: root@host01 /]# ceph orch ps --daemon_type=rgw

3.3. Deploying the Ceph Object Gateway using the service specification

Using the Ceph Orchestrator, you can deploy the Ceph object gateway using the service specification.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to all the nodes.
  • Hosts are added to the cluster.
  • All manager, monitor and OSD daemons are deployed.

Procedure

  1. Log into the Cephadm shell:

    Example

    [root@host01 ~]# cephadm shell

  2. Navigate to the following directory:

    Syntax

    cd /var/lib/ceph/DAEMON_PATH/

    Example

    [ceph: root@host01 nfs/]# cd /var/lib/ceph/rgw/

    If the rgw directory does not exist, create the directory in the path.

  3. Create the rgw.yml file:

    Example

    [ceph: root@host01 rgw]# touch rgw.yml

  4. Edit the rgw.yml file to include the following details:

    Syntax

    service_type: rgw
    service_id: REALM_NAME.ZONE_NAME
    placement:
      hosts:
      - HOST_NAME_1
      - HOST_NAME_2
    spec:
      rgw_realm: REALM_NAME
      rgw_zone: ZONE_NAME
    networks:
      -  NETWORK_IP_ADDRESS # RGW service binds to a specific network

    Example

    service_type: rgw
    service_id: test_realm.test_zone
    placement:
      hosts:
      - host01
      - host02
      - host03
    spec:
      rgw_realm: test_realm
      rgw_zone: test_zone
    networks:
      - 192.169.142.0/24

  5. Deploy Ceph object gateway using service specification:

    Syntax

    ceph orch apply -i FILE_NAME.yml

    Example

    [ceph: root@host01 rgw]# ceph orch apply -i rgw.yml

Verification

  • List the service:

    Example

    [ceph: root@host01 /]# ceph orch ls

  • List the hosts, daemons, and processes:

    Syntax

    ceph orch ps --daemon_type=DAEMON_NAME

    Example

    [ceph: root@host01 /]# ceph orch ps --daemon_type=rgw

3.4. Deploying a multi-site Ceph Object Gateway using the Ceph Orchestrator

Ceph Orchestrator supports multi-site configurations options for the Ceph Object Gateway.

You can configure each object gateway to work in an active-active zone configuration allowing for writes to a non-primary zone. The multi-site configuration is stored within a container called a realm.

The realm stores zone groups, zones and a time period. The rgw daemons handle the synchronization eliminating the need for a separate synchronization agent, thereby operating with an active-active configuration.

You can also deploy multi-site zones using the command line interface (CLI).

Note

The following configuration assumes at least two Red Hat Ceph Storage clusters are in geographically separate locations. However, the configuration also works on the same site.

Prerequisites

  • At least two running Red Hat Ceph Storage clusters
  • At least two Ceph Object Gateway instances, one for each Red Hat Ceph Storage cluster.
  • Root-level access to all the nodes.
  • Nodes or containers are added to the storage cluster.
  • All Ceph Manager, Monitor and OSD daemons are deployed.

Procedure

  1. In the cephadm shell, configure the primary zone:

    1. Create a realm:

      Syntax

      radosgw-admin realm create --rgw-realm=REALM_NAME --default

      Example

      [ceph: root@host01 /]# radosgw-admin realm create --rgw-realm=test_realm --default

      If the storage cluster has a single realm, then specify the --default flag.

    2. Create a primary zone group:

      Syntax

      radosgw-admin zonegroup create --rgw-zonegroup=ZONE_GROUP_NAME --endpoints=http://RGW_PRIMARY_HOSTNAME:_RGW_PRIMARY_PORT_NUMBER_1_ --master --default

      Example

      [ceph: root@host01 /]# radosgw-admin zonegroup create --rgw-zonegroup=us --endpoints=http://rgw1:80 --master --default

    3. Create a primary zone:

      Syntax

      radosgw-admin zone create --rgw-zonegroup=PRIMARY_ZONE_GROUP_NAME --rgw-zone=PRIMARY_ZONE_NAME --endpoints=http://RGW_PRIMARY_HOSTNAME:_RGW_PRIMARY_PORT_NUMBER_1_ --access-key=SYSTEM_ACCESS_KEY --secret=SYSTEM_SECRET_KEY

      Example

      [ceph: root@host01 /]# radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-1 --endpoints=http://rgw1:80

    4. Optional: Delete the default zone, zone group, and the associated pools:

      Important

      Do not delete the default zone and its pools if you are using the default zone and zone group to store data. Also, removing the default zone group, deletes the system user.

      Example

      [ceph: root@host01 /]# radosgw-admin zonegroup delete --rgw-zonegroup=default
      [ceph: root@host01 /]# ceph osd pool rm default.rgw.log default.rgw.log --yes-i-really-really-mean-it
      [ceph: root@host01 /]# ceph osd pool rm default.rgw.meta default.rgw.meta --yes-i-really-really-mean-it
      [ceph: root@host01 /]# ceph osd pool rm default.rgw.control default.rgw.control --yes-i-really-really-mean-it
      [ceph: root@host01 /]# ceph osd pool rm default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it
      [ceph: root@host01 /]# ceph osd pool rm default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it

    5. Create a system user:

      Syntax

      radosgw-admin user create --uid=USER_NAME --display-name="USER_NAME" --access-key=SYSTEM_ACCESS_KEY --secret=SYSTEM_SECRET_KEY --system

      Example

      [ceph: root@host01 /]# radosgw-admin user create --uid=zone.user --display-name="Zone user" --system

      Make a note of the access_key and secret_key.

    6. Add the access key and system key to the primary zone:

      Syntax

      radosgw-admin zone modify --rgw-zone=PRIMARY_ZONE_NAME --access-key=ACCESS_KEY --secret=SECRET_KEY

      Example

      [ceph: root@host01 /]# radosgw-admin zone modify --rgw-zone=us-east-1 --access-key=NE48APYCAODEPLKBCZVQ--secret=u24GHQWRE3yxxNBnFBzjM4jn14mFIckQ4EKL6LoW

    7. Commit the changes:

      Syntax

      radosgw-admin period update --commit

      Example

      [ceph: root@host01 /]# radosgw-admin period update --commit

    8. Outside the cephadm shell, fetch the FSID of the storage cluster and the processes:

      Example

      [root@host01 ~]#  systemctl list-units | grep ceph

    9. Start the Ceph Object Gateway daemon:

      Syntax

      systemctl start ceph-FSID@DAEMON_NAME
      systemctl enable ceph-FSID@DAEMON_NAME

      Example

      [root@host01 ~]# systemctl start ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-1.host01.ahdtsw.service
      [root@host01 ~]# systemctl enable ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-1.host01.ahdtsw.service

  2. In the Cephadm shell, configure the secondary zone.

    1. Use the URL path, access key, and secret key of the primary zone and primary zone group, and pull the realm configuration from the host:

      Syntax

      radosgw-admin period pull --url=URL_TO_PRIMARY_ZONE_GATEWAY --access-key=ACCESS_KEY --secret-key=SECRET_KEY

      Example

      [ceph: root@host04 /]# radosgw-admin period pull --url=http://10.74.249.26:80 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ

    2. Configure a secondary zone:

      Syntax

      radosgw-admin zone create --rgw-zonegroup=ZONE_GROUP_NAME \
                   --rgw-zone=SECONDARY_ZONE_NAME --endpoints=http://RGW_SECONDARY_HOSTNAME:_RGW_PRIMARY_PORT_NUMBER_1_ \
                   --access-key=SYSTEM_ACCESS_KEY --secret=SYSTEM_SECRET_KEY \
                   --endpoints=http://FQDN:80 \
                   [--read-only]

      Example

      [ceph: root@host04 /]# radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east-2 --endpoints=http://rgw2:80 --access-key=LIPEYZJLTWXRKXS9LPJC --secret-key=IsAje0AVDNXNw48LjMAimpCpI7VaxJYSnfD0FFKQ

    3. Optional: Delete the default zone:

      Important

      Do not delete the default zone and its pools if you are using the default zone and zone group to store data.

      Example

      [ceph: root@host04 /]# radosgw-admin zone rm --rgw-zone=default
      [ceph: root@host04 /]# ceph osd pool rm default.rgw.log default.rgw.log --yes-i-really-really-mean-it
      [ceph: root@host04 /]# ceph osd pool rm default.rgw.meta default.rgw.meta --yes-i-really-really-mean-it
      [ceph: root@host04 /]# ceph osd pool rm default.rgw.control default.rgw.control --yes-i-really-really-mean-it
      [ceph: root@host04 /]# ceph osd pool rm default.rgw.data.root default.rgw.data.root --yes-i-really-really-mean-it
      [ceph: root@host04 /]# ceph osd pool rm default.rgw.gc default.rgw.gc --yes-i-really-really-mean-it

    4. Update the Ceph configuration database:

      Syntax

      ceph config set _SERVICE_NAME_ rgw_zone _SECONDARY_ZONE_NAME_

      Example

      [ceph: root@host04 /]# ceph config set rgw rgw_zone us-east-2

    5. Commit the changes:

      Syntax

      radosgw-admin period update --commit

      Example

      [ceph: root@host04 /]# radosgw-admin period update --commit

    6. Outside the Cephadm shell, fetch the FSID of the storage cluster and the processes:

      Example

      [root@host04 ~]#  systemctl list-units | grep ceph

    7. Start the Ceph Object Gateway daemon:

      Syntax

      systemctl start ceph-FSID@DAEMON_NAME
      systemctl enable ceph-FSID@DAEMON_NAME

      Example

      [root@host04 ~]# systemctl start ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-2.host04.ahdtsw.service
      [root@host04 ~]# systemctl enable ceph-62a081a6-88aa-11eb-a367-001a4a000672@rgw.test_realm.us-east-2.host04.ahdtsw.service

  3. Optional: Deploy multi-site Ceph Object Gateways using the placement specification:

    Syntax

    ceph orch apply rgw _NAME_ --realm=_REALM_NAME_ --zone=_PRIMARY_ZONE_NAME_ --placement="_NUMBER_OF_DAEMONS_ _HOST_NAME_1_ _HOST_NAME_2_"

    Example

    [ceph: root@host04 /]# ceph orch apply rgw east --realm=test_realm --zone=us-east-1 --placement="2 host01 host02"

Verification

  • Check the synchronization status to verify the deployment:

    Example

    [ceph: root@host04 /]# radosgw-admin sync status

3.5. Removing the Ceph Object Gateway using the Ceph Orchestrator

You can remove the Ceph object gateway daemons using the ceph orch rm command.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to all the nodes.
  • Hosts are added to the cluster.
  • At least one Ceph object gateway daemon deployed on the hosts.

Procedure

  1. Log into the Cephadm shell:

    Example

    [root@host01 ~]# cephadm shell

  2. List the service:

    Example

    [ceph: root@host01 /]# ceph orch ls

    1. Remove the service

      Syntax

      ceph orch rm SERVICE_NAME

      Example

      [ceph: root@host01 /]# ceph orch rm rgw.test_realm.test_zone_bb

Verification

  • List the hosts, daemons, and processes:

    Syntax

    ceph orch ps

    Example

    [ceph: root@host01 /]# ceph orch ps

Additional Resources

Chapter 4. Basic configuration

As a storage administrator, you can know the basics of configuring the Ceph Object Gateway. You can learn about the default, and embedded web server called Beast. For troubleshooting issues with the Ceph Object Gateway, you can learn how to adjust the logging and debugging output generated by the Ceph Object Gateway. Also, learn how to provide a High-Availability proxy for storage cluster access using the Ceph Object Gateway.

4.1. Prerequisites

  • A running, and healthy Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway software package.

4.2. Add a Wildcard to the DNS

To use Ceph with S3-style subdomains, for example bucket-name.domain-name.com, add a wildcard to the DNS record of the DNS server the ceph-radosgw daemon uses to resolve domain names.

For dnsmasq, add the following address setting with a dot (.) prepended to the host name:

address=/.{hostname-or-fqdn}/{host-ip-address}

For example:

address=/.gateway-node1/192.168.122.75

For bind, add a wildcard to the DNS record. For example:

$TTL    604800
@       IN      SOA     gateway-node1. root.gateway-node1. (
                              2         ; Serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
;
@       IN      NS      gateway-node1.
@       IN      A       192.168.122.113
*       IN      CNAME   @

Restart the DNS server and ping the server with a subdomain to ensure that the ceph-radosgw daemon can process the subdomain requests:

ping mybucket.{hostname}

For example:

ping mybucket.gateway-node1

If the DNS server is on the local machine, you might need to modify /etc/resolv.conf by adding a nameserver entry for the local machine.

Finally, specify the host name or address of the DNS server:

Syntax

ceph config set client.rgw rgw_dns_name VALUE

Example

[root@mon ~]# ceph config set client.rgw rgw_dns_name client01

Finally, restart the Ceph Object Gateway so that DNS setting takes effect.

Additional Resources

4.3. The Beast front-end web server

The Ceph Object Gateway provides Beast, a C/C embedded front-end web server. Beast uses the `Boost.Beast` C library to parse HTTP, and Boost.Asio to do asynchronous network I/O.

Additional Resources

4.4. Configuring SSL for Beast

You can configure the Beast front-end web server to use the OpenSSL library to provide Transport Layer Security (TLS). To use Secure Socket Layer (SSL) with Beast, you need to obtain a certificate from a Certificate Authority (CA) that matches the hostname of the Ceph Object Gateway node. Beast also requires the secret key, server certificate, and any other CA in a single .pem file.

Important

Prevent unauthorized access to the .pem file, because it contains the secret key hash.

Important

Red Hat recommends obtaining a certificate from a CA with the Subject Alternative Name (SAN) field, and a wildcard for use with S3-style subdomains.

Important

Red Hat recommends only using SSL with the Beast front-end web server for small to medium sized test environments. For production environments you must use HAProxy and keepalived to terminate the SSL connection at the HAProxy.

Prerequisites

  • A running, and healthy Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway software package.
  • Installation of the OpenSSL software package.
  • Root-level access to the Ceph Object Gateway node.

Procedure

  1. Create a new file named rgw.yml in the current directory:

    Example

    [root@rgw ~]# touch rgw.yml

  2. Open the rgw.yml file for editing, and customize it for the environment:

    Syntax

    service_type: rgw
    service_id: SERVICE_ID
    service_name: SERVICE_NAME
    placement:
      hosts:
      - HOST_NAME
    spec:
      ssl: true
      rgw_frontend_ssl_certificate: CERT_HASH

    Example

    service_type: rgw
    service_id: foo
    service_name: rgw.foo
    placement:
      hosts:
      - host1
    spec:
      ssl: true
      rgw_frontend_ssl_certificate: |
        -----BEGIN RSA PRIVATE KEY-----
        MIIEpAIBAAKCAQEA+Cf4l9OagD6x67HhdCy4Asqw89Zz9ZuGbH50/7ltIMQpJJU0
        gu9ObNtIoC0zabJ7n1jujueYgIpOqGnhRSvsGJiEkgN81NLQ9rqAVaGpadjrNLcM
        bpgqJCZj0vzzmtFBCtenpb5l/EccMFcAydGtGeLP33SaWiZ4Rne56GBInk6SATI/
        JSKweGD1y5GiAWipBR4C74HiAW9q6hCOuSdp/2WQxWT3T1j2sjlqxkHdtInUtwOm
        j5Ism276IndeQ9hR3reFR8PJnKIPx73oTBQ7p9CMR1J4ucq9Ny0J12wQYT00fmJp
        -----END RSA PRIVATE KEY-----
        -----BEGIN CERTIFICATE-----
        MIIEBTCCAu2gAwIBAgIUGfYFsj8HyA9Zv2l600hxzT8+gG4wDQYJKoZIhvcNAQEL
        BQAwgYkxCzAJBgNVBAYTAklOMQwwCgYDVQQIDANLQVIxDDAKBgNVBAcMA0JMUjEM
        MAoGA1UECgwDUkhUMQswCQYDVQQLDAJCVTEkMCIGA1UEAwwbY2VwaC1zc2wtcmhj
        czUtOGRjeHY2LW5vZGU1MR0wGwYJKoZIhvcNAQkBFg5hYmNAcmVkaGF0LmNvbTAe
        -----END CERTIFICATE-----

  3. Deploy the Ceph Object Gateway using the service specification file:

    Example

    [root@rgw ~]# ceph orch apply -i rgw.yml

4.5. Adjusting logging and debugging output

Once you finish the setup procedure, check your logging output to ensure it meets your needs. By default, the Ceph daemons log to journald, and you can view the logs using the journalctl command. Alternatively, you can also have the Ceph daemons log to files, which are located under the /var/log/ceph/CEPH_CLUSTER_ID/ directory.

Important

Verbose logging can generate over 1 GB of data per hour. This type of logging can potentially fill up the operating system’s disk, causing the operating system to stop functioning.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway software.

Procedure

  1. Set the following parameter to increase the Ceph Object Gateway logging output:

    Syntax

    ceph config set client.rgw debug_rgw VALUE

    Example

    [root@rgw ~]# ceph config set client.rgw debug_rgw 20

    1. You can also modify these settings at runtime:

      Syntax

      ceph --admin-daemon /var/run/ceph/ceph-client.rgw.NAME.asok config set debug_rgw VALUE

      Example

      [root@rgw ~]# ceph --admin-daemon /var/run/ceph/ceph-client.rgw.rgw.asok config set debug_rgw 20

  2. Optionally, you can configure the Ceph daemons to log their output to files. Set the log_to_file, and mon_cluster_log_to_file options to true:

    Example

    [root@rgw ~]# ceph config set global log_to_file true
    [root@rgw ~]# ceph config set global mon_cluster_log_to_file true

Additional Resources

4.6. Static web hosting

As a storage administrator, you can configure the Ceph Object Gateway to host static websites in S3 buckets. Traditional website hosting involves configuring a web server for each website, which can use resources inefficiently when content does not change dynamically. For example, sites that do not use server-side services like PHP, servlets, databases, nodejs, and the like. This approach is substantially more economical than setting up virtual machines with web servers for each site.

4.6.1. Prerequisites

  • A healthy, running Red Hat Ceph Storage cluster.

4.6.2. Static web hosting assumptions

Static web hosting requires at least one running Red Hat Ceph Storage cluster, and at least two Ceph Object Gateway instances for the static web sites. Red Hat assumes that each zone will have multiple gateway instances using a load balancer, such as high-availability (HA) Proxy and keepalived.

Important

Red Hat DOES NOT support using a Ceph Object Gateway instance to deploy both standard S3/Swift APIs and static web hosting simultaneously.

Additional Resources

4.6.3. Static Web Hosting Requirements

Static web hosting functionality uses its own API, so configuring a gateway to use static web sites in S3 buckets requires the following:

  1. S3 static web hosting uses Ceph Object Gateway instances that are separate and distinct from instances used for standard S3/Swift API use cases.
  2. Gateway instances hosting S3 static web sites should have separate, non-overlapping domain names from the standard S3/Swift API gateway instances.
  3. Gateway instances hosting S3 static web sites should use separate public-facing IP addresses from the standard S3/Swift API gateway instances.
  4. Gateway instances hosting S3 static web sites load balance, and if necessary terminate SSL, using HAProxy/keepalived.

4.6.4. Static Web Hosting Gateway Setup

To enable a Ceph Object Gateway for static web hosting, set the following options:

Syntax

ceph config set client.rgw OPTION VALUE

Example

[root@mon ~]# ceph config set client.rgw rgw_enable_static_website true
[root@mon ~]# ceph config set client.rgw rgw_enable_apis s3,s3website
[root@mon ~]# ceph config set client.rgw rgw_dns_name objects-zonegroup.example.com
[root@mon ~]# ceph config set client.rgw rgw_dns_s3website_name objects-website-zonegroup.example.com
[root@mon ~]# ceph config set client.rgw rgw_resolve_cname true

The rgw_enable_static_website setting MUST be true. The rgw_enable_apis setting MUST enable the s3website API. The rgw_dns_name and rgw_dns_s3website_name settings must provide their fully qualified domains. If the site uses canonical name extensions, then set the rgw_resolve_cname option to true.

Important

The FQDNs of rgw_dns_name and rgw_dns_s3website_name MUST NOT overlap.

4.6.5. Static Web Hosting DNS Configuration

The following is an example of assumed DNS settings, where the first two lines specify the domains of the gateway instance using a standard S3 interface and point to the IPv4 and IPv6 addresses respectively. The third line provides a wildcard CNAME setting for S3 buckets using canonical name extensions. The fourth and fifth lines specify the domains for the gateway instance using the S3 website interface and point to their IPv4 and IPv6 addresses respectively.

objects-zonegroup.domain.com. IN    A 192.0.2.10
objects-zonegroup.domain.com. IN AAAA 2001:DB8::192:0:2:10
*.objects-zonegroup.domain.com. IN CNAME objects-zonegroup.domain.com.
objects-website-zonegroup.domain.com. IN    A 192.0.2.20
objects-website-zonegroup.domain.com. IN AAAA 2001:DB8::192:0:2:20
Note

The IP addresses in the first two lines differ from the IP addresses in the fourth and fifth lines.

If using Ceph Object Gateway in a multi-site configuration, consider using a routing solution to route traffic to the gateway closest to the client.

The Amazon Web Service (AWS) requires static web host buckets to match the host name. Ceph provides a few different ways to configure the DNS, and HTTPS will work if the proxy has a matching certificate.

Hostname to a Bucket on a Subdomain

To use AWS-style S3 subdomains, use a wildcard in the DNS entry and can redirect requests to any bucket. A DNS entry might look like the following:

*.objects-website-zonegroup.domain.com. IN CNAME objects-website-zonegroup.domain.com.

Access the bucket name in the following manner:

http://bucket1.objects-website-zonegroup.domain.com

Where the bucket name is bucket1.

Hostname to Non-Matching Bucket

Ceph supports mapping domain names to buckets without including the bucket name in the request, which is unique to Ceph Object Gateway. To use a domain name to access a bucket, map the domain name to the bucket name. A DNS entry might look like the following:

www.example.com. IN CNAME bucket2.objects-website-zonegroup.domain.com.

Where the bucket name is bucket2.

Access the bucket in the following manner:

http://www.example.com

Hostname to Long Bucket with CNAME

AWS typically requires the bucket name to match the domain name. To configure the DNS for static web hosting using CNAME, the DNS entry might look like the following:

www.example.com. IN CNAME www.example.com.objects-website-zonegroup.domain.com.

Access the bucket in the following manner:

http://www.example.com

Hostname to Long Bucket without CNAME

If the DNS name contains other non-CNAME records such as SOA, NS, MX or TXT, the DNS record must map the domain name directly to the IP address. For example:

www.example.com. IN A 192.0.2.20
www.example.com. IN AAAA 2001:DB8::192:0:2:20

Access the bucket in the following manner:

http://www.example.com

4.6.6. Creating a Static Web Hosting Site

To create a static website perform the following steps:

  1. Create an S3 bucket. The bucket name MAY be the same as the website’s domain name. For example, mysite.com may have a bucket name of mysite.com. This is required for AWS, but it is NOT required for Ceph. See DNS Settings for details.
  2. Upload the static website content to the bucket. Contents may include HTML, CSS, client-side JavaScript, images, audio/video content and other downloadable files. A website MUST have an index.html file and MAY have error.html file.
  3. Verify the website’s contents. At this point, only the creator of the bucket will have access to the contents.
  4. Set permissions on the files so that they are publicly readable.

4.7. HA Proxy and keepalived

As a storage administrator, you can assign many instances of the Ceph Object Gateway to a single zone. This allows you scale out as the load increases, that is, the same zone group and zone; however, you do not need a federated architecture to use HAProxy and keepalived. Since each object gateway instance has its own IP address, you can use HAProxy and keepalived to balance the load across Ceph Object Gateway servers.

Another use case for HAProxy and keepalived is to terminate HTTPS at the HAProxy server. You can use an HAProxy server to terminate HTTPS at the HAProxy server and use HTTP between the HAProxy server and the Beast web server instances.

4.7.1. HAProxy/keepalived Prerequisites

To set up an HA Proxy with the Ceph Object Gateway, you must have:

  • A running Ceph cluster
  • At least two Ceph Object Gateway servers within the same zone configured to run on port 80. If you follow the simple installation procedure, the gateway instances are in the same zone group and zone by default. If you are using a federated architecture, ensure that the instances are in the same zone group and zone.
  • At least two Red Hat Enterprise Linux 8 servers for HAProxy and keepalived.
Note

This section assumes that you have at least two Ceph Object Gateway servers running, and that you get a valid response from each of them when running test scripts over port 80.

4.7.2. Preparing HAProxy Nodes

The following setup assumes two HAProxy nodes named haproxy and haproxy2 and two Ceph Object Gateway servers named rgw1 and rgw2. You might use any naming convention you prefer. Perform the following procedure on at least two HAProxy nodes:

  1. Install Red Hat Enterprise Linux 8.
  2. Register the nodes.

    [root@haproxy]# subscription-manager register
  3. Enable the RHEL server repository.

    [root@haproxy]# subscription-manager repos --enable=rhel-8-server-rpms
  4. Update the server.

    [root@haproxy]# dnf update -y
  5. Install admin tools (e.g., wget, vim, etc.) as needed.
  6. Open port 80.

    [root@haproxy]# firewall-cmd --zone=public --add-port 80/tcp --permanent
    [root@haproxy]# firewall-cmd --reload
  7. For HTTPS, open port 443.

    [root@haproxy]# firewall-cmd --zone=public --add-port 443/tcp --permanent
    [root@haproxy]# firewall-cmd --reload

4.7.3. Installing and Configuring keepalived

Perform the following procedure on your at least two HAProxy nodes:

Prerequisites

  • A minimum of two HAProxy nodes.
  • A minimum of two Object Gateway nodes.

Procedure

  1. Install keepalived:

    [root@haproxy]# yum install -y keepalived
  2. Configure keepalived on both HAProxy nodes:

    [root@haproxy]# vim /etc/keepalived/keepalived.conf

    In the configuration file, there is a script to check the haproxy processes:

    vrrp_script chk_haproxy {
      script "killall -0 haproxy" # check the haproxy process
      interval 2 # every 2 seconds
      weight 2 # add 2 points if OK
    }

    Next, the instance on the primary and backup load balancers uses eno1 as the network interface. It also assigns a virtual IP address, that is, 192.168.1.20.

    Primary load balancer node

    vrrp_instance RGW {
        state MASTER # might not be necessary. This is on the primary LB node.
        @main interface eno1
        priority 100
        advert_int 1
        interface eno1
        virtual_router_id 50
        @main unicast_src_ip 10.8.128.43 80
        unicast_peer {
               10.8.128.53
               }
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
            192.168.1.20
        }
        track_script {
          chk_haproxy
        }
    }
    virtual_server 192.168.1.20 80 eno1 { #populate correct interface
        delay_loop 6
        lb_algo wlc
        lb_kind dr
        persistence_timeout 600
        protocol TCP
        real_server 10.8.128.43 80 { # ip address of rgw2 on physical interface, haproxy listens here, rgw listens to localhost:8080 or similar
            weight 100
            TCP_CHECK { # perhaps change these to a HTTP/SSL GET?
                connect_timeout 3
            }
        }
        real_server 10.8.128.53 80 { # ip address of rgw3 on physical interface, haproxy listens here, rgw listens to localhost:8080 or similar
            weight 100
            TCP_CHECK { # perhaps change these to a HTTP/SSL GET?
                connect_timeout 3
            }
        }
    }

    Backup load balancer node

    vrrp_instance RGW {
        state BACKUP # might not be necessary?
        priority 99
        advert_int 1
        interface eno1
        virtual_router_id 50
        unicast_src_ip 10.8.128.53 80
        unicast_peer {
               10.8.128.43
               }
        authentication {
            auth_type PASS
            auth_pass 1111
        }
        virtual_ipaddress {
            192.168.1.20
        }
        track_script {
          chk_haproxy
        }
    }
    virtual_server 192.168.1.20 80 eno1 { #populate correct interface
        delay_loop 6
        lb_algo wlc
        lb_kind dr
        persistence_timeout 600
        protocol TCP
        real_server 10.8.128.43 80 { # ip address of rgw2 on physical interface, haproxy listens here, rgw listens to localhost:8080 or similar
            weight 100
            TCP_CHECK { # perhaps change these to a HTTP/SSL GET?
                connect_timeout 3
            }
        }
        real_server 10.8.128.53 80 { # ip address of rgw3 on physical interface, haproxy listens here, rgw listens to localhost:8080 or similar
            weight 100
            TCP_CHECK { # perhaps change these to a HTTP/SSL GET?
                connect_timeout 3
            }
        }
    }

  3. Enable and start the keepalived service:

    [root@haproxy]# systemctl enable keepalived
    [root@haproxy]# systemctl start keepalived

Additional Resources

4.7.4. Installing and Configuring HAProxy

Perform the following procedure on your at least two HAProxy nodes:

  1. Install haproxy.

    [root@haproxy]# dnf install haproxy
  2. Configure haproxy for SELinux and HTTP.

    [root@haproxy]# vim /etc/firewalld/services/haproxy-http.xml

    Add the following lines:

    <?xml version="1.0" encoding="utf-8"?>
    <service>
    <short>HAProxy-HTTP</short>
    <description>HAProxy load-balancer</description>
    <port protocol="tcp" port="80"/>
    </service>

    As root, assign the correct SELinux context and file permissions to the haproxy-http.xml file.

    [root@haproxy]# cd /etc/firewalld/services
    [root@haproxy]# restorecon haproxy-http.xml
    [root@haproxy]# chmod 640 haproxy-http.xml
  3. If you intend to use HTTPS, configure haproxy for SELinux and HTTPS.

    [root@haproxy]# vim /etc/firewalld/services/haproxy-https.xml

    Add the following lines:

    <?xml version="1.0" encoding="utf-8"?>
    <service>
    <short>HAProxy-HTTPS</short>
    <description>HAProxy load-balancer</description>
    <port protocol="tcp" port="443"/>
    </service>

    As root, assign the correct SELinux context and file permissions to the haproxy-https.xml file.

    # cd /etc/firewalld/services
    # restorecon haproxy-https.xml
    # chmod 640 haproxy-https.xml
  4. If you intend to use HTTPS, generate keys for SSL. If you do not have a certificate, you may use a self-signed certificate. To generate a key, see to Generating a New Key and Certificate section in the System Administrator’s Guide for Red Hat Enterprise Linux 7.

    Finally, put the certificate and key into a PEM file.

    [root@haproxy]# cat example.com.crt example.com.key > example.com.pem
    [root@haproxy]# cp example.com.pem /etc/ssl/private/
  5. Configure haproxy.

    [root@haproxy]# vim /etc/haproxy/haproxy.cfg

    The global and defaults may remain unchanged. After the defaults section, you will need to configure frontend and backend sections. For example:

    frontend http_web
      bind *:80
      mode http
      default_backend rgw
    
    frontend rgw­-https
      bind *:443 ssl crt /etc/ssl/private/example.com.pem
      default_backend rgw
    
    backend rgw
        balance roundrobin
        mode http
        server  rgw1 10.0.0.71:80 check
        server  rgw2 10.0.0.80:80 check

    For a details on HAProxy configuration, refer to Add an HAProxy Load Balancer chapter in the Red Hat Update Infrastructure System Administrator’s Guide.

  6. Enable/start haproxy

    [root@haproxy]# systemctl enable haproxy
    [root@haproxy]# systemctl start haproxy

4.7.5. Testing the HAProxy Configuration

On your HAProxy nodes, check to ensure the virtual IP address from your keepalived configuration appears.

[root@haproxy]# ip addr show

On your calamari node, see if you can reach the gateway nodes via the load balancer configuration. For example:

[root@haproxy]# wget haproxy

This should return the same result as:

[root@haproxy]# wget rgw1

If it returns an index.html file with the following contents:

<?xml version="1.0" encoding="UTF-8"?>
	<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
		<Owner>
			<ID>anonymous</ID>
			<DisplayName></DisplayName>
		</Owner>
		<Buckets>
		</Buckets>
	</ListAllMyBucketsResult>

Then, your configuration is working properly.

4.8. Exporting the Namespace to NFS-Ganesha

To configure new NFS Ganesha exports for use with the Ceph Object Gateway, you have to use the Red Hat Ceph Storage Dashboard. See the Management of NFS Ganesha exports on the Ceph Dashboard section in the Red Hat Ceph Storage Dashboard Guide for more details.

Important

For existing NFS environments using the Ceph Object Gateway, upgrading from Red Hat Ceph Storage 4 to Red Hat Ceph Storage 5 is not supported at this time.

Important

Red Hat does not support NFS version 3 exports using the Ceph Object Gateway.

Chapter 5. Advanced configuration

As a storage administrator, you can configure some of the more advanced features of the Ceph Object Gateway. You can configure a multisite Ceph Object Gateway and integrate it with directory services, such as Microsoft Active Directory and OpenStack Keystone service.

5.1. Prerequisites

  • A healthy running Red Hat Ceph Storage cluster.

5.2. Multi-site configuration and administration

As a storage administrator, you can configure and administrate multiple Ceph Object Gateways for a variety of use cases. You can learn what to do during a disaster recovery, and failover events. Also, you can learn more about realms, zones, and syncing policies in a multi-site Ceph Object Gateway environments.

5.2.1. Prerequisites

  • A healthy, and running Red Hat Ceph Storage cluster.
  • Deployment of the Ceph Object Gateway software.

5.2.2. Pools

Red Hat recommends using the Ceph Placement Group’s per Pool Calculator to calculate a suitable number of placement groups for the pools the radosgw daemon will create. Set the calculated values as defaults in the Ceph configuration database.

Example

[root@mon ~]# ceph config set osd osd_pool_default_pg_num = 50
[root@mon ~]# ceph config set osd osd_pool_default_pgp_num = 50

Note

Making this change to the Ceph configuration will use those defaults when the Ceph Object Gateway instance creates the pools. Alternatively, you can create the pools manually.

Pool names particular to a zone follow the naming convention ZONE_NAME.POOL_NAME. For example, a zone named us-east will have the following pools:

  • .rgw.root
  • us-east.rgw.control
  • us-east.rgw.meta
  • us-east.rgw.log
  • us-east.rgw.buckets.index
  • us-east.rgw.buckets.data
  • us-east.rgw.buckets.non-ec
  • us-east.rgw.meta:users.keys
  • us-east.rgw.meta:users.email
  • us-east.rgw.meta:users.swift
  • us-east.rgw.meta:users.uid

Additional Resources

  • See the Pools chapter in the Red Hat Ceph Storage Storage Strategies Guide for details on creating pools.

5.2.3. Migrating a Single Site System to Multi-Site

To migrate from a single site system with a default zone group and zone to a multi site system, use the following steps:

  1. Create a realm. Replace <name> with the realm name.

    [root@primary-zone]# radosgw-admin realm create --rgw-realm=<name> --default
  2. Rename the default zone and zonegroup. Replace <name> with the zonegroup or zone name.

    [root@primary-zone]# radosgw-admin zonegroup rename --rgw-zonegroup default --zonegroup-new-name=<name>
    [root@primary-zone]# radosgw-admin zone rename --rgw-zone default --zone-new-name us-east-1 --rgw-zonegroup=<name>
  3. Configure the primary zonegroup. Replace <name> with the realm or zonegroup name. Replace <fqdn> with the fully qualified domain name(s) in the zonegroup.

    [root@primary-zone]# radosgw-admin zonegroup modify --rgw-realm=<name> --rgw-zonegroup=<name> --endpoints http://<fqdn>:80 --master --default
  4. Configure the primary zone. Replace <name> with the realm, zonegroup or zone name. Replace <fqdn> with the fully qualified domain name(s) in the zonegroup.

    [root@primary-zone]# radosgw-admin zone modify --rgw-realm=<name> --rgw-zonegroup=<name> \
                                --rgw-zone=<name> --endpoints http://<fqdn>:80 \
                                --access-key=<access-key> --secret=<secret-key> \
                                --master --default
  5. Create a system user. Replace <user-id> with the username. Replace <display-name> with a display name. It may contain spaces.

    [root@primary-zone]# radosgw-admin user create --uid=<user-id> \
                                --display-name="<display-name>" \
                                --access-key=<access-key> --secret=<secret-key> \ --system
  6. Commit the updated configuration.

    # radosgw-admin period update --commit
  7. Restart the Ceph Object Gateway.

    Note

    Use the output from the ceph orch ps command, under the NAME column to get the SERVICE_TYPE.ID information.

    1. To restart the Ceph Object Gateway on individual node in the storage cluster:

      Syntax

      systemctl restart ceph-CLUSTER_ID@SERVICE_TYPE.ID.service

      Example

      [root@primary-zone]# systemctl restart ceph-c4b34c6f-8365-11ba-dc31-529020a7702d@rgw.realm.zone.node1.gwasto.service

    2. To restart the Ceph Object Gateways on all nodes in the storage cluster:

      Syntax

      ceph orch restart SERVICE_TYPE

      Example

      [root@primary-zone]# ceph orch restart rgw

5.2.4. Configuring the archive sync module (Technology Preview)

The archive sync module leverages the versioning feature of S3 objects in Ceph object gateway to have an archive zone. The archive zone has a history of versions of S3 objects that can only be eliminated through the gateways associated with the archive zone. It captures all the data updates and metadata to consolidate them as versions of S3 objects.

Important

The archive sync module is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs), might not be functionally complete, and Red Hat does not recommend to use them for production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. See the support scope for Red Hat Technology Preview features for more details.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to a Ceph Monitor node.
  • Installation of the Ceph Object Gateway software.

Procedure

  1. Configure the archive sync module when creating a new zone by using the archive tier:

    Syntax

    radosgw-admin zone create --rgw-zonegroup={ZONE_GROUP_NAME} --rgw-zone={ZONE_NAME} --endpoints={http://FQDN: PORT}[,{http://FQDN: PORT}] --tier-type=archive

    Example

    [root@primary-zone]# radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east --endpoints={http://example.com:8080} --tier-type=archive

Additional resources

5.2.5. Failover and Disaster Recovery

If the primary zone would fail, failover to the secondary zone for disaster recovery.

  1. Make the secondary zone the primary and default zone. For example:

    # radosgw-admin zone modify --rgw-zone={zone-name} --master --default

    By default, Ceph Object Gateway runs in an active-active configuration. If the cluster was configured to run in an active-passive configuration, the secondary zone is a read-only zone. Remove the --read-only status to allow the zone to receive write operations. For example:

    # radosgw-admin zone modify --rgw-zone={zone-name} --master --default
  2. Update the period to make the changes take effect.

    # radosgw-admin period update --commit
  3. Restart the Ceph Object Gateway.

    Note

    Use the output from the ceph orch ps command, under the NAME column to get the SERVICE_TYPE.ID information.

    1. To restart the Ceph Object Gateway on individual node in the storage cluster:

      Syntax

      systemctl restart ceph-CLUSTER_ID@SERVICE_TYPE.ID.service

      Example

      [root@primary-zone]# systemctl restart ceph-c4b34c6f-8365-11ba-dc31-529020a7702d@rgw.realm.zone.node1.gwasto.service

    2. To restart the Ceph Object Gateways on all nodes in the storage cluster:

      Syntax

      ceph orch restart SERVICE_TYPE

      Example

      [root@primary-zone]# ceph orch restart rgw

If the former primary zone recovers, revert the operation.

  1. From the recovered zone, pull the realm from the current primary zone.

    # radosgw-admin realm pull --url={url-to-primary-zone-gateway} \
                                --access-key={access-key} --secret={secret}
  2. Make the recovered zone the primary and default zone.

    # radosgw-admin zone modify --rgw-zone={zone-name} --master --default
  3. Update the period to make the changes take effect.

    # radosgw-admin period update --commit
  4. Restart the Ceph Object Gateway in the recovered zone.

    # systemctl restart ceph-radosgw@rgw.`hostname -s`.rgw0
  5. If the secondary zone needs to be a read-only configuration, update the secondary zone.

    # radosgw-admin zone modify --rgw-zone={zone-name} --read-only
  6. Update the period to make the changes take effect.

    # radosgw-admin period update --commit
  7. Restart the Ceph Object Gateway in the secondary zone.

    # systemctl restart ceph-radosgw@rgw.`hostname -s`.rgw0

5.2.6. Manually Resharding Buckets with Multisite

Red Hat Ceph Storage DOES NOT support dynamic bucket resharding for multisite clusters. To manually reshard buckets in a multisite cluster, use the following procedure.

Warning

Manual resharding is a very expensive process, especially for huge buckets that warrant manual resharding. Every secondary zone deletes all of the objects, and then resynchronizes them from the primary zone.

Prerequisites

  • Stop all Ceph Object Gateway instances.

Procedure

  1. On a node within the master zone of the master zone group, execute the following command:

    Syntax

    # radosgw-admin bucket sync disable --bucket=BUCKET_NAME

    Wait for sync status on all zones to report that data synchronization is up to date.

  2. Stop ALL ceph-radosgw daemons in ALL zones.
  3. On a node within the master zone of the master zone group, reshard the bucket.

    Syntax

    # radosgw-admin bucket reshard --bucket=BUCKET_NAME --num-shards=NEW_SHARDS_NUMBER

  4. On EACH secondary zone, execute the following:

    Syntax

    # radosgw-admin bucket rm --purge-objects --bucket=BUCKET_NAME

  5. Restart ALL ceph-radosgw daemons in ALL zones.
  6. On a node within the master zone of the master zone group, execute the following command:

    Syntax

    # radosgw-admin bucket sync enable --bucket=BUCKET_NAME

    The metadata synchronization process will fetch the updated bucket entry point and bucket instance metadata. The data synchronization process will perform a full synchronization.

Additional resources

5.2.7. Configuring multiple zones without replication

You can configure multiple zones that will not replicate each other. For example, you can create a dedicated zone for each team in a company.

Prerequisites

  • Installation of the Ceph Object Gateway software.
  • Root-level access to the Ceph Object Gateway node.

Procedure

  1. Create a new realm:

    Syntax

    radosgw-admin realm create --rgw-realm=REALM_NAME [--default]

    Example

    [root@rgw-primary]# radosgw-admin realm create --rgw-realm=movies --default
    {
        "id": "0956b174-fe14-4f97-8b50-bb7ec5e1cf62",
        "name": "movies",
        "current_period": "1950b710-3e63-4c41-a19e-46a715000980",
        "epoch": 1
    }

  2. Create a new zone group:

    Syntax

    radosgw-admin zonegroup create --rgw-zonegroup=ZONE_GROUP_NAME --endpoints=FQDN : PORT [--rgw-realm=REALM_NAME|--realm-id=REALM_ID] --master --default

    Example

    [root@rgw-primary]# radosgw-admin zonegroup create --rgw-zonegroup=us --endpoints=http://rgw1:80 --rgw-realm=movies --master --default
    {
        "id": "f1a233f5-c354-4107-b36c-df66126475a6",
        "name": "us",
        "api_name": "us",
        "is_master": "true",
        "endpoints": [
            "http:\/\/rgw1:80"
        ],
        "hostnames": [],
        "hostnames_s3webzone": [],
        "master_zone": "",
        "zones": [],
        "placement_targets": [],
        "default_placement": "",
        "realm_id": "0956b174-fe14-4f97-8b50-bb7ec5e1cf62"
    }

  3. Create one or more zones depending on the use case:

    Syntax

    radosgw-admin zone create --rgw-zonegroup=ZONE_GROUP_NAME --rgw-zone=ZONE_NAME --master --default --endpoints=FQDN : PORT[,FQDN : PORT_]

    Example

    [root@rgw-primary]# radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-east --master --default --endpoints=http://rgw1:80

  4. Get the JSON file with the configuration of the zone group:

    Syntax

    radosgw-admin zonegroup get --rgw-zonegroup=ZONE_GROUP_NAME > JSON_FILE_NAME

    Example

    [root@rgw-primary]# radosgw-admin zonegroup get --rgw-zonegroup=us > zonegroup-us.json

    1. Open the file for editing, and set the log_meta, log_data, and sync_from_all fields to false:

      Example

          {
              "id": "72f3a886-4c70-420b-bc39-7687f072997d",
              "name": "default",
              "api_name": "",
              "is_master": "true",
              "endpoints": [],
              "hostnames": [],
              "hostnames_s3website": [],
              "master_zone": "a5e44ecd-7aae-4e39-b743-3a709acb60c5",
              "zones": [
                  {
                      "id": "975558e0-44d8-4866-a435-96d3e71041db",
                      "name": "testzone",
                      "endpoints": [],
                      "log_meta": "false",
                      "log_data": "false",
                      "bucket_index_max_shards": 0,
                      "read_only": "false",
                      "tier_type": "",
                      "sync_from_all": "false",
                      "sync_from": []
                  },
                  {
                      "id": "a5e44ecd-7aae-4e39-b743-3a709acb60c5",
                      "name": "default",
                      "endpoints": [],
                      "log_meta": "false",
                      "log_data": "false",
                      "bucket_index_max_shards": 0,
                      "read_only": "false",
                      "tier_type": "",
                      "sync_from_all": "false",
                      "sync_from": []
                  }
              ],
              "placement_targets": [
                  {
                      "name": "default-placement",
                      "tags": []
                  }
              ],
              "default_placement": "default-placement",
              "realm_id": "2d988e7d-917e-46e7-bb18-79350f6a5155"
          }

  5. Use the updated JSON file to set the zone group:

    Syntax

    radosgw-admin zonegroup set --rgw-zonegroup=ZONE_GROUP_NAME --infile=JSON_FILE_NAME

    Example

    [root@rgw-primary]# radosgw-admin zonegroup set --rgw-zonegroup=us --infile=zonegroup-us.json

  6. Update the period:

    Example

    [root@rgw-primary]# radosgw-admin period update --commit

5.2.8. Configuring multiple realms in the same storage cluster

This section discusses how to configure multiple realms in the same storage cluster. This is a more advanced use case for Multisite. Configuring multiple realms in the same storage cluster enables you to use a local realm to handle local RGW client traffic, as well as a replicated realm for data that will be replicated to a secondary site.

Note

Red Hat recommends that each realm has its own Ceph Object Gateway.

Prerequisites

  • The access key and secret key for each data center in the storage cluster.
  • Two running Red Hat Ceph Storage data centers in a storage cluster.
  • Each data center has its own local realm. They share a realm that replicates on both sites.
  • On the Ceph Object Gateway nodes, perform the tasks listed in the Requirements for Installing Red Hat Ceph Storage found in the Red Hat Ceph Storage Installation Guide.
  • For each Ceph Object Gateway node, perform steps 1-7 in the Installing the Ceph Object Gateway section of the Red Hat Ceph Storage Installation Guide.

Procedure

  1. Create the synchronization user:

    Syntax

    radosgw-admin user create --uid="SYNCHRONIZATION_USER" --display-name="Synchronization User" --system

  2. Create one local realm on the first data center in the storage cluster:

    Syntax

    radosgw-admin realm create --rgw-realm=REALM_NAME --default

    Example

    [user@rgw1]$ radosgw-admin realm create --rgw-realm=ldc1 --default

  3. Create one local master zonegroup on the first data center:

    Syntax

    radosgw-admin zonegroup create --rgw-zonegroup=ZONE_GROUP_NAME --endpoints=http://RGW_NODE_NAME:80 --rgw-realm=REALM_NAME --master --default

    Example

    [user@rgw1]$ radosgw-admin zonegroup create --rgw-zonegroup=ldc1zg --endpoints=http://rgw1:80 --rgw-realm=ldc1 --master --default

  4. Create one local zone on the first data center:

    Syntax

    radosgw-admin zone create --rgw-zonegroup=ZONE_GROUP_NAME --rgw-zone=ZONE_NAME --master --default --endpoints=HTTP_FQDN[,HTTP_FQDN]

    Example

    [user@rgw1]$ radosgw-admin zone create --rgw-zonegroup=ldc1zg --rgw-zone=ldc1z --master --default --endpoints=http://rgw.example.com

  5. Commit the period:

    Example

    [user@rgw1]$ radosgw-admin period update --commit

  6. Update ceph.conf with the rgw_realm, rgw_zonegroup and rgw_zone names:

    Syntax

    rgw_realm = REALM_NAME
    rgw_zonegroup = ZONE_GROUP_NAME
    rgw_zone = ZONE_NAME

    Example

    rgw_realm = ldc1
    rgw_zonegroup = ldc1zg
    rgw_zone = ldc1z

  7. Restart the Ceph Object Gateway.

    Note

    Use the output from the ceph orch ps command, under the NAME column to get the SERVICE_TYPE.ID information.

    1. To restart the Ceph Object Gateway on individual node in the storage cluster:

      Syntax

      systemctl restart ceph-CLUSTER_ID@SERVICE_TYPE.ID.service

      Example

      [root@rgw1]# systemctl restart ceph-c4b34c6f-8365-11ba-dc31-529020a7702d@rgw.realm.zone.node1.gwasto.service

    2. To restart the Ceph Object Gateways on all nodes in the storage cluster:

      Syntax

      ceph orch restart SERVICE_TYPE

      Example

      [root@rgw1]# ceph orch restart rgw

  8. Create one local realm on the second data center in the storage cluster:

    Syntax

    radosgw-admin realm create --rgw-realm=REALM_NAME --default

    Example

    [user@rgw2]$ radosgw-admin realm create --rgw-realm=ldc2 --default

  9. Create one local master zonegroup on the second data center:

    Syntax

    radosgw-admin zonegroup create --rgw-zonegroup=ZONE_GROUP_NAME --endpoints=http://RGW_NODE_NAME:80 --rgw-realm=REALM_NAME --master --default

    Example

    [user@rgw2]$ radosgw-admin zonegroup create --rgw-zonegroup=ldc2zg --endpoints=http://rgw2:80 --rgw-realm=ldc2 --master --default

  10. Create one local zone on the second data center:

    Syntax

    radosgw-admin zone create --rgw-zonegroup=ZONE_GROUP_NAME --rgw-zone=ZONE_NAME --master --default --endpoints=HTTP_FQDN[, HTTP_FQDN]

    Example

    [user@rgw2]$ radosgw-admin zone create --rgw-zonegroup=ldc2zg --rgw-zone=ldc2z --master --default --endpoints=http://rgw.example.com

  11. Commit the period:

    Example

    [user@rgw2]$ radosgw-admin period update --commit

  12. Update ceph.conf with the rgw_realm, rgw_zonegroup and rgw_zone names:

    Syntax

    rgw_realm = REALM_NAME
    rgw_zonegroup = ZONE_GROUP_NAME
    rgw_zone = ZONE_NAME

    Example

    rgw_realm = ldc2
    rgw_zonegroup = ldc2zg
    rgw_zone = ldc2z

  13. Restart the Ceph Object Gateway.

    Note

    Use the output from the ceph orch ps command, under the NAME column to get the SERVICE_TYPE.ID information.

    1. To restart the Ceph Object Gateway on individual node in the storage cluster:

      Syntax

      systemctl restart ceph-CLUSTER_ID@SERVICE_TYPE.ID.service

      Example

      [root@rgw2]# systemctl restart ceph-c4b34c6f-8365-11ba-dc31-529020a7702d@rgw.realm.zone.node1.gwasto.service

    2. To restart the Ceph Object Gateways on all nodes in the storage cluster:

      Syntax

      ceph orch restart SERVICE_TYPE

      Example

      [root@rgw2]# ceph orch restart rgw

  14. Create a replication/synchronization user:

    Syntax

    radosgw-admin user create --uid="REPLICATION_SYNCHRONIZATION_USER" --display-name="Replication-Synchronization User" --system

  15. Create a replicated realm on the first data center in the storage cluster:

    Syntax

    radosgw-admin realm create --rgw-realm=REPLICATED_REALM_1

    Example

    [user@rgw1] radosgw-admin realm create --rgw-realm=rdc1

  16. Create a master zonegroup for the first data center:

    Syntax

    radosgw-admin zonegroup create --rgw-zonegroup=RGW_ZONE_GROUP --endpoints=http://_RGW_NODE_NAME:80 --rgw-realm=_RGW_REALM_NAME --master --default

    Example

    [user@rgw1] radosgw-admin zonegroup create --rgw-zonegroup=rdc1zg --endpoints=http://rgw1:80 --rgw-realm=rdc1 --master --default

  17. Create a master zone on the first data center:

    Syntax

    radosgw-admin zone create --rgw-zonegroup=RGW_ZONE_GROUP --rgw-zone=_MASTER_RGW_NODE_NAME --master --default --endpoints=HTTP_FQDN[,HTTP_FQDN]

    Example

    [user@rgw1]$ radosgw-admin zone create --rgw-zonegroup=rdc1zg --rgw-zone=rdc1z --master --default --endpoints=http://rgw.example.com

  18. Commit the period:

    Example

    [user@rgw1]$ radosgw-admin period update --commit

  19. Update ceph.conf with the rgw_realm, rgw_zonegroup and rgw_zone names for the first data center:

    Syntax

    rgw_realm = REALM_NAME
    rgw_zonegroup = ZONE_GROUP_NAME
    rgw_zone = ZONE_NAME

    Example

    rgw_realm = rdc1
    rgw_zonegroup = rdc1zg
    rgw_zone = rdc1z

  20. Restart the Ceph Object Gateway.

    Note

    Use the output from the ceph orch ps command, under the NAME column to get the SERVICE_TYPE.ID information.

    1. To restart the Ceph Object Gateway on individual node in the storage cluster:

      Syntax

      systemctl restart ceph-CLUSTER_ID@SERVICE_TYPE.ID.service

      Example

      [root@rgw1]# systemctl restart ceph-c4b34c6f-8365-11ba-dc31-529020a7702d@rgw.realm.zone.node1.gwasto.service

    2. To restart the Ceph Object Gateways on all nodes in the storage cluster:

      Syntax

      ceph orch restart SERVICE_TYPE

      Example

      [root@rgw1]# ceph orch restart rgw

  21. Pull the replicated realm on the second data center:

    Syntax

    radosgw-admin realm pull --url=https://tower-osd1.cephtips.com --access-key=ACCESS_KEY --secret-key=SECRET_KEY

    Example

    [user@rgw1]$ radosgw-admin realm pull --url=https://tower-osd1.cephtips.com --access-key=3QV0D6ZMMCJZMSCXJ2QJ --secret-key=VpvQWcsfI9OPzUCpR4kynDLAbqa1OIKqRB6WEnH8

  22. Pull the period from the first data center:

    Syntax

    radosgw-admin period pull --url=https://tower-osd1.cephtips.com --access-key=ACCESS_KEY --secret-key=SECRET_KEY

    Example

    [user@rgw1]$radosgw-admin period pull --url=https://tower-osd1.cephtips.com --access-key=3QV0D6ZMMCJZMSCXJ2QJ --secret-key=VpvQWcsfI9OPzUCpR4kynDLAbqa1OIKqRB6WEnH8

  23. Create the secondary zone on the second data center:

    Syntax

    radosgw-admin zone create --rgw-zone=RGW_ZONE --rgw-zonegroup=RGW_ZONE_GROUP --endpoints=https://tower-osd4.cephtips.com --access-key=_ACCESS_KEY --secret-key=SECRET_KEY

    Example

    [user@rgw2]$ radosgw-admin zone create --rgw-zone=rdc2z --rgw-zonegroup=rdc1zg --endpoints=https://tower-osd4.cephtips.com --access-key=3QV0D6ZMMCJZMSCXJ2QJ --secret-key=VpvQWcsfI9OPzUCpR4kynDLAbqa1OIKqRB6WEnH8

  24. Commit the period:

    Example

    [user@rgw2]$ radosgw-admin period update --commit

  25. Update ceph.conf with the rgw_realm, rgw_zonegroup and rgw_zone names for the second data center:

    Syntax

    rgw_realm = REALM_NAME
    rgw_zonegroup = ZONE_GROUP_NAME
    rgw_zone = ZONE_NAME

    Example

    rgw realm = rdc1
    rgw zonegroup = rdc1zg
    rgw zone = rdc2z

  26. Restart the Ceph Object Gateway.

    Note

    Use the output from the ceph orch ps command, under the NAME column to get the SERVICE_TYPE.ID information.

    1. To restart the Ceph Object Gateway on individual node in the storage cluster:

      Syntax

      systemctl restart ceph-CLUSTER_ID@SERVICE_TYPE.ID.service

      Example

      [root@rgw2]# systemctl restart ceph-c4b34c6f-8365-11ba-dc31-529020a7702d@rgw.realm.zone.node1.gwasto.service

    2. To restart the Ceph Object Gateways on all nodes in the storage cluster:

      Syntax

      ceph orch restart SERVICE_TYPE

      Example

      [root@rgw2]# ceph orch restart rgw

  27. Log in as root on the endpoint for the second data center.
  28. Verify the synchronization status on the master realm:

    Syntax

    radosgw-admin sync status

    Example

    [root@rgw2]# radosgw-admin sync status
              realm 59762f08-470c-46de-b2b1-d92c50986e67 (ldc2)
          zonegroup 7cf8daf8-d279-4d5c-b73e-c7fd2af65197 (ldc2zg)
               zone 034ae8d3-ae0c-4e35-8760-134782cb4196 (ldc2z)
      metadata sync no sync (zone is master)

  29. Log in as root on the endpoint for the first data center.
  30. Verify the synchronization status for the replication-synchronization realm:

    Syntax

    radosgw-admin sync status --rgw-realm RGW_REALM_NAME

    Example

    [root@rgw1]# radosgw-admin sync status --rgw-realm rdc1
              realm 73c7b801-3736-4a89-aaf8-e23c96e6e29d (rdc1)
          zonegroup d67cc9c9-690a-4076-89b8-e8127d868398 (rdc1zg)
               zone 67584789-375b-4d61-8f12-d1cf71998b38 (rdc2z)
      metadata sync syncing
                    full sync: 0/64 shards
                    incremental sync: 64/64 shards
                    metadata is caught up with master
          data sync source: 705ff9b0-68d5-4475-9017-452107cec9a0 (rdc1z)
                            syncing
                            full sync: 0/128 shards
                            incremental sync: 128/128 shards
                            data is caught up with source
              realm 73c7b801-3736-4a89-aaf8-e23c96e6e29d (rdc1)
          zonegroup d67cc9c9-690a-4076-89b8-e8127d868398 (rdc1zg)
               zone 67584789-375b-4d61-8f12-d1cf71998b38 (rdc2z)
      metadata sync syncing
                    full sync: 0/64 shards
                    incremental sync: 64/64 shards
                    metadata is caught up with master
          data sync source: 705ff9b0-68d5-4475-9017-452107cec9a0 (rdc1z)
                            syncing
                            full sync: 0/128 shards
                            incremental sync: 128/128 shards
                            data is caught up with source

  31. To store and access data in the local site, create the user for local realm:

    Syntax

    radosgw-admin user create --uid="LOCAL_USER" --display-name="Local user" --rgw-realm=_REALM_NAME --rgw-zonegroup=ZONE_GROUP_NAME --rgw-zone=ZONE_NAME

    Example

    [user@rgw2] #radosgw-admin user create --uid="local-user" --display-name="Local user" --rgw-realm=ldc1 --rgw-zonegroup=ldc1zg --rgw-zone=ldc1z

    Important

    By default, users are added to the multi-site configuration. For the users to access data in the local zone, the radosgw-admin command requires the --rgw-realm arugument.

5.2.9. Using Multisite sync policies (Technology Preview)

Important

The Ceph Object Gateway multi-site sync policies is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs), might not be functionally complete, and Red Hat does not recommend to use them for production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. See the support scope for Red Hat Technology Preview features for more details.

As a storage administrator, you can use multisite sync policies at the bucket level to control data movement between buckets in different zones. These policies are called bucket-granularity sync policies. Previously, all buckets within zones were treated symmetrically. This means that each zone contained a mirror copy of a given bucket, and the copies of buckets were identical in all of the zones. The sync process assumed that the bucket sync source and the bucket sync destination referred to the same bucket.

Using bucket-granularity sync policies allows for the buckets in different zones to contain different data. This enables a bucket to pull data from other buckets in other zones, and those buckets do not have the same name or ID as the bucket pulling the data. In this case, the bucket sync source and the bucket sync destination refer to different buckets.

The sync policy supersedes the old zone group coarse configuration (sync_from*). The sync policy can be configured at the zone group level. If it is configured, it replaces the old-style configuration at the zone group level, but it can also be configured at the bucket level.

5.2.9.1. Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to a Ceph Monitor node.
  • Installation of the Ceph Object Gateway software.

5.2.9.2. Multisite sync policy data flow

In the sync policy, multiple groups that can contain lists of data-flow configurations can be defined, as well as lists of pipe configurations. The data-flow defines the flow of data between the different zones. It can define symmetrical data flow, in which multiple zones sync data from each other, and it can define directional data flow, in which the data moves in one way from one zone to another.

A pipe defines the actual buckets that can use these data flows, and the properties that are associated with it (for example: source object prefix).

A sync policy group can be in 3 states:

  • Enabled — sync is allowed and enabled.
  • Allowed — sync is allowed.
  • Forbidden — sync, as defined by this group, is not allowed. Sync state in this group can override other groups.

A policy can be defined at the bucket level. A bucket level sync policy inherits the data flow of the zonegroup policy, and can only define a subset of what the zonegroup allows.

A wildcard zone, and a wildcard bucket parameter in the policy defines all relevant zones, or all relevant buckets. In the context of a bucket policy it means the current bucket instance. A disaster recovery configuration where entire zones are mirrored doesn’t require configuring anything on the buckets. However, for a fine grained bucket sync it would be better to configure the pipes to be synced by allowing (status=allowed) them at the zonegroup level (e.g., using wildcards), but only enable the specific sync at the bucket level (status=enabled). If needed, the policy at the bucket level can limit the data movement to specific relevant zones.

Important

Any changes to the zonegroup policy needs to be applied on the zonegroup master zone, and require period update and commit. Changes to the bucket policy needs to be applied on the zonegroup master zone. The changes are dynamically handled by rgw.

S3 replication API

The S3 bucket replication api has also been implemented, and allows users to create replication rules between different buckets. Note though that while the AWS replication feature allows bucket replication within the same zone, rgw does not allow it at the moment. However, the rgw api also added a new ‘Zone’ array that allows users to select to what zones the specific bucket will be synced.

5.2.9.3. Retrieving the current policy

As a storage administrator, you can use the get command to retrieve the current zonegroup sync policy, or a specific bucket policy.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • root or sudo access.
  • The Ceph Object Gateway is installed.

Procedure

  1. Retrieve the current zonegroup sync policy or bucket policy. To retrieve a specific bucket policy, use the --bucket option.

    Syntax

    radosgw-admin sync policy get --bucket=BUCKET-NAME

    Example

    [root@master-zone]# radosgw-admin sync policy get --bucket=mybucket

5.2.9.4. Creating a sync policy group

As a storage administrator, you can create a sync policy group for the current zone group, or for a specific bucket.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • root or sudo access.
  • The Ceph Object Gateway is installed.

Procedure

  1. Create a sync policy group or a bucket policy. To create a bucket policy, use the --bucket option:

    Syntax

    radosgw-admin sync group create --bucket=BUCKET-NAME --group-id=GROUP-ID --status=enabled | allowed | forbidden

    Example

    [root@master-zone]# radosgw-admin sync group create --group-id=mygroup1 --status=enabled

5.2.9.5. Modifying a sync policy group

As a storage administrator, you can modify an existing sync policy group for the current zone group, or for a specific bucket.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • root or sudo access.
  • The Ceph Object Gateway is installed.

Procedure

  1. Modify the sync policy group or a bucket policy. To modify a bucket policy, use the --bucket option:

    Syntax

    radosgw-admin sync group modify --bucket=BUCKET-NAME --group-id=GROUP-ID --status=enabled | allowed | forbidden

    Example

    [root@master-zone]# radosgw-admin sync group modify --group-id=mygroup1 --status=forbidden

5.2.9.6. Showing a sync policy group

As a storage administrator, you can use the group get command to show the current sync policy group by group ID, or to show a specific bucket policy.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • root or sudo access.
  • The Ceph Object Gateway is installed.

Procedure

  1. Show the current sync policy group or bucket policy. To show a specific bucket policy, use the --bucket option.

    Syntax

    radosgw-admin sync group get --bucket=BUCKET-NAME --group-id=GROUP-ID

    Example

    [root@master-zone]# radosgw-admin sync group get --group-id=mygroup

5.2.9.7. Removing a sync policy group

As a storage administrator, you can use the group remove command to remove the current sync policy group by group ID, or to remove a specific bucket policy.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • root or sudo access.
  • The Ceph Object Gateway is installed.

Procedure

  1. Remove the current sync policy group or bucket policy. To remove a specific bucket policy, use the --bucket option.

    Syntax

    radosgw-admin sync group remove --bucket=BUCKET-NAME --group-id=GROUP-ID

    Example

    [root@master-zone]# radosgw-admin sync group remove --group-id=mygroup

5.2.9.8. Creating a sync flow

As a storage administrator, you can create two different types of flows for a sync policy group or for a specific bucket:

  • Directional sync flow
  • Symmetrical sync flow

The group flow create command creates a sync flow. If you issue the group flow create command for a sync policy group or bucket that already has a sync flow, the command overwrites the existing settings for the sync flow and applies the settings you specify.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • root or sudo access.
  • The Ceph Object Gateway is installed.

Procedure

  1. Create or update a directional sync flow. To create or update directional sync flow for a specific bucket, use the --bucket option.

    Syntax

    radosgw-admin sync group flow create --bucket=BUCKET-NAME --group-id=GROUP-ID --flow-id=FLOW-ID --flow-type=directional --source-zone=SOURCE-ZONE --dest-zone=DESTINATION-ZONE

    • Create or update a symmetrical sync flow. To specify multiple zones for a symmetrical flow type, use a comma-separated list for the --zones option.

      Syntax

      radosgw-admin sync group flow create --bucket=BUCKET-NAME --group-id=GROUP-ID --flow-id=FLOW-ID --flow-type=symmetrical --zones=ZONE-NAME

5.2.9.9. Removing sync flows and zones

The group flow remove command removes sync flows or zones from a sync policy group or bucket.

For sync policy groups or buckets using directional flows, group flow remove removes the flow. For sync policy groups or buckets using symmetrical flows, you can use the group flow remove command to remove specified zones from the flow, or to remove the flow.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • root or sudo access.
  • The Ceph Object Gateway is installed.

Procedure

  1. Remove a directional sync flow. To remove the directional sync flow for a specific bucket, use the --bucket option.

    Syntax

    radosgw-admin sync group flow remove --bucket=BUCKET-NAME --group-id=GROUP-ID --flow-id=FLOW-ID --flow-type=directional --source-zone=SOURCE-ZONE --dest-zone=DESTINATION-ZONE

    • Remove specific zones from a symmetrical sync flow. To remove multiple zones from a symmetrical flow, use a comma-separated list for the --zones option.

      Syntax

      radosgw-admin sync group flow remove --bucket=BUCKET-NAME --group-id=GROUP-ID --flow-id=FLOW-ID --flow-type=symmetrical --zones=ZONE-NAME

    • Remove a symmetrical sync flow. To remove the sync flow from a bucket, use the --bucket option.

      Syntax

      radosgw-admin sync group flow create --bucket=BUCKET-NAME --group-id=GROUP-ID --flow-id=FLOW-ID --flow-type=symmetrical --zones=ZONE-NAME

5.2.9.10. Creating or updating a sync group pipe

As a storage administrator, you can define pipes to specify which buckets can use your configured data flows and the properties associated with those data flows.

The sync group pipe create command enables you to create pipes, which are custom sync group data flows between specific buckets or groups of buckets, or between specific zones or groups of zones.

This command uses the following options:

OptionDescriptionRequired/Optional

--bucket

Name of the bucket containing the data flow. Use the wildcard * for all buckets.

Optional

--group-id

ID of the sync group

Required

--pipe-id

ID of the pipe

Required

--source-zones

Zones that send data to the sync group. Use commas to separate multiple zones. Use the wildcard * for all zones that match the data flow rules.

Required

--source-bucket

Bucket or buckets that send data to the sync group.

Optional

--source-bucket-id

ID of the source bucket.

Optional

--dest-zones

Zone or zones that receive the sync data. Use commas to separate multiple zones. Use the wildcard * for all zones that match the data flow rules.

Required

--dest-bucket

Bucket or buckets that receive the sync data.

Optional

--dest-bucket-id

ID of the destination bucket.

Optional

--prefix

Bucket prefix. Use the wildcard * to filter for source objects.

Optional

--prefix-rm

Do not use bucket prefix for filtering.

Optional

--tags-add

Comma-separated list of key=value pairs.

Optional

--tags-rm

Removes one or more key=value pairs of tags.

Optional

--dest-owner

Destination owner of the objects from source.

Optional

--storage-class

Destination stoarge class for the objects from source.

Optional

--mode

Use system for system mode or user for user mode.

Optional

--uid

Used for permissions validation in user mode. Specifies the user ID under which the sync operation will be executed.

Optional

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • root or sudo access.
  • The Ceph Object Gateway is installed.

Procedure

  1. Create the sync group pipe:

    Syntax

    radosgw-admin sync group pipe create --bucket=BUCKET-NAME --group-id=GROUP-ID --pipe-id=PIPE-ID --source-zones=ZONE-NAME, ZONE-NAME2... --source-bucket=SOURCE-BUCKET1, SOURCE-BUCKET2... --source-bucket-id=SOURCE-BUCKET-ID --dest-zones=ZONE-NAME, ZONE-NAME2... --dest-bucket=SOURCE-BUCKET1, SOURCE-BUCKET2... --dest-bucket-id=DESTINATION-BUCKET-ID --prefix=SOURCE-PREFIX --prefix-rm --tags-add=KEY1=VALUE1, KEY2=VALUE2, ... --tags-rm=KEY1=VALUE1, KEY2=VALUE2, ... --dest-owner=OWNER-ID --storage-class=STORAGE-CLASS --mode=USER --uid=USER-ID

5.2.9.11. Modifying or deleting a sync group pipe

As a storage administrator, you can use the sync group pipe remove command to modify the sync group pipe by removing certain options. You can also use this command to remove the sync group pipe completely.

This command uses the following options:

OptionDescriptionRequired/Optional

--bucket

Name of the bucket containing the data flow. Use the wildcard * for all buckets.

Optional

--group-id

ID of the sync group

Required

--pipe-id

ID of the pipe

Required

--source-zones

Zones that send data to the sync group. Use commas to separate multiple zones. Use the wildcard * for all zones that match the data flow rules.

Required

--source-bucket

Bucket or buckets that send data to the sync group.

Optional

--source-bucket-id

ID of the source bucket.

Optional

--dest-zones

Zone or zones that receive the sync data. Use commas to separate multiple zones. Use the wildcard * for all zones that match the data flow rules.

Required

--dest-bucket

Bucket or buckets that receive the sync data.

Optional

--dest-bucket-id

ID of the destination bucket.

Optional

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • root or sudo access.
  • The Ceph Object Gateway is installed.

Procedure

  1. Remove the sync group pipe options.

    Syntax

    radosgw-admin sync group pipe remove --bucket=BUCKET-NAME --group-id=GROUP-ID --pipe-id=PIPE-ID --source-zones=ZONE-NAME, ZONE-NAME2... --source-bucket=SOURCE-BUCKET1, SOURCE-BUCKET2... --source-bucket-id=SOURCE-BUCKET-ID --dest-zones=ZONE-NAME, ZONE-NAME2... --dest-bucket=SOURCE-BUCKET1, SOURCE-BUCKET2... --dest-bucket-id=DESTINATION-BUCKET-ID

5.2.9.12. Obtaining information about sync operations

The sync info command enables you to get information about the expected sync sources and targets, as defined by the sync policy.

When you create a sync policy for a bucket, that policy defines how data moves from that bucket toward a different bucket in a different zone. Creating the policy also creates a list of bucket dependencies that are used as hints whenever that bucket syncs with another bucket. Note that a bucket can refer to another bucket without actually syncing to it, since syncing depends on whether the data flow allows the sync to take place.

Both the --bucket and effective-zone-name parameters are optional. If you invoke the sync info command without specifying any options, the Object Gateway returns all of the sync operations defined by the sync policy in all zones.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • root or sudo access.
  • The Ceph Object Gateway is installed.
  • A group sync policy is defined.

Procedure

  1. Get information about sync operations:

    Syntax

    radosgw-admin sync info --bucket=BUCKET-NAME --effective-zone-name=ZONE-NAME

5.2.10. Multi-site Ceph Object Gateway command line usage

As a storage administrator, you can have a good understanding of how to use the Ceph Object Gateway in a multi-site environment. You can learn how to better manage the realms, zone groups, and zones in a multi-site environment.

5.2.10.1. Prerequisites

  • A running Red Hat Ceph Storage.
  • Deployment of the Ceph Object Gateway software.
  • Access to a Ceph Object Gateway node or container.

5.2.10.2. Realms

A realm represents a globally unique namespace consisting of one or more zonegroups containing one or more zones, and zones containing buckets, which in turn contain objects. A realm enables the Ceph Object Gateway to support multiple namespaces and their configuration on the same hardware.

A realm contains the notion of periods. Each period represents the state of the zone group and zone configuration in time. Each time you make a change to a zonegroup or zone, update the period and commit it.

By default, the Ceph Object Gateway version 2 does not create a realm for backward compatibility with version 1.3 and earlier releases. However, as a best practice, Red Hat recommends creating realms for new clusters.

5.2.10.2.1. Creating a Realm

To create a realm, execute realm create and specify the realm name. If the realm is the default, specify --default.

[root@master-zone]# radosgw-admin realm create --rgw-realm={realm-name} [--default]

For example:

[root@master-zone]# radosgw-admin realm create --rgw-realm=movies --default

By specifying --default, the realm will be called implicitly with each radosgw-admin call unless --rgw-realm and the realm name are explicitly provided.

5.2.10.2.2. Making a Realm the Default

One realm in the list of realms should be the default realm. There may be only one default realm. If there is only one realm and it wasn’t specified as the default realm when it was created, make it the default realm. Alternatively, to change which realm is the default, execute:

[root@master-zone]# radosgw-admin realm default --rgw-realm=movies
Note

When the realm is default, the command line assumes --rgw-realm=<realm-name> as an argument.

5.2.10.2.3. Deleting a Realm

To delete a realm, execute realm delete and specify the realm name.

[root@master-zone]# radosgw-admin realm delete --rgw-realm={realm-name}

For example:

[root@master-zone]# radosgw-admin realm delete --rgw-realm=movies
5.2.10.2.4. Getting a Realm

To get a realm, execute realm get and specify the realm name.

# radosgw-admin realm get --rgw-realm=<name>

For example:

# radosgw-admin realm get --rgw-realm=movies [> filename.json]

The CLI will echo a JSON object with the realm properties.

{
    "id": "0a68d52e-a19c-4e8e-b012-a8f831cb3ebc",
    "name": "movies",
    "current_period": "b0c5bbef-4337-4edd-8184-5aeab2ec413b",
    "epoch": 1
}

Use > and an output file name to output the JSON object to a file.

5.2.10.2.5. Setting a Realm

To set a realm, execute realm set, specify the realm name, and --infile= with an input file name.

[root@master-zone]# radosgw-admin realm set --rgw-realm=<name> --infile=<infilename>

For example:

[root@master-zone]# radosgw-admin realm set --rgw-realm=movies --infile=filename.json
5.2.10.2.6. Listing Realms

To list realms, execute realm list.

# radosgw-admin realm list
5.2.10.2.7. Listing Realm Periods

To list realm periods, execute realm list-periods.

# radosgw-admin realm list-periods
5.2.10.2.8. Pulling a Realm

To pull a realm from the node containing the master zone group and master zone to a node containing a secondary zone group or zone, execute realm pull on the node that will receive the realm configuration.

# radosgw-admin realm pull --url={url-to-master-zone-gateway} --access-key={access-key} --secret={secret}
5.2.10.2.9. Renaming a Realm

A realm is not part of the period. Consequently, renaming the realm is only applied locally, and will not get pulled with realm pull. When renaming a realm with multiple zones, run the command on each zone. To rename a realm, execute the following:

# radosgw-admin realm rename --rgw-realm=<current-name> --realm-new-name=<new-realm-name>
Note

Do NOT use realm set to change the name parameter. That changes the internal name only. Specifying --rgw-realm would still use the old realm name.

5.2.10.3. Zone Groups

The Ceph Object Gateway supports multi-site deployments and a global namespace by using the notion of zone groups. Formerly called a region, a zone group defines the geographic location of one or more Ceph Object Gateway instances within one or more zones.

Configuring zone groups differs from typical configuration procedures, because not all of the settings end up in a Ceph configuration file. You can list zone groups, get a zone group configuration, and set a zone group configuration.

Note

The radosgw-admin zonegroup operations can be performed on any node within the realm, because the step of updating the period propagates the changes throughout the cluster. However, radosgw-admin zone operations MUST be performed on a host within the zone.

5.2.10.3.1. Creating a Zone Group

Creating a zone group consists of specifying the zone group name. Creating a zone assumes it will live in the default realm unless --rgw-realm=<realm-name> is specified. If the zonegroup is the default zonegroup, specify the --default flag. If the zonegroup is the master zonegroup, specify the --master flag. For example:

# radosgw-admin zonegroup create --rgw-zonegroup=<name> [--rgw-realm=<name>][--master] [--default]
Note

Use zonegroup modify --rgw-zonegroup=<zonegroup-name> to modify an existing zone group’s settings.

5.2.10.3.2. Making a Zone Group the Default

One zonegroup in the list of zonegroups should be the default zonegroup. There may be only one default zonegroup. If there is only one zonegroup and it wasn’t specified as the default zonegroup when it was created, make it the default zonegroup. Alternatively, to change which zonegroup is the default, execute:

# radosgw-admin zonegroup default --rgw-zonegroup=comedy
Note

When the zonegroup is default, the command line assumes --rgw-zonegroup=<zonegroup-name> as an argument.

Then, update the period:

# radosgw-admin period update --commit
5.2.10.3.3. Adding a Zone to a Zone Group

To add a zone to a zonegroup, you MUST execute this step on a host that will be in the zone. To add a zone to a zonegroup, execute the following:

# radosgw-admin zonegroup add --rgw-zonegroup=<name> --rgw-zone=<name>

Then, update the period:

# radosgw-admin period update --commit
5.2.10.3.4. Removing a Zone from a Zone Group

To remove a zone from a zonegroup, execute the following:

# radosgw-admin zonegroup remove --rgw-zonegroup=<name> --rgw-zone=<name>

Then, update the period:

# radosgw-admin period update --commit
5.2.10.3.5. Renaming a Zone Group

To rename a zonegroup, execute the following:

# radosgw-admin zonegroup rename --rgw-zonegroup=<name> --zonegroup-new-name=<name>

Then, update the period:

# radosgw-admin period update --commit
5.2.10.3.6. Deleting a Zone Group

To delete a zonegroup, execute the following:

# radosgw-admin zonegroup delete --rgw-zonegroup=<name>

Then, update the period:

# radosgw-admin period update --commit
5.2.10.3.7. Listing Zone Groups

A Ceph cluster contains a list of zone groups. To list the zone groups, execute:

# radosgw-admin zonegroup list

The radosgw-admin returns a JSON formatted list of zone groups.

{
    "default_info": "90b28698-e7c3-462c-a42d-4aa780d24eda",
    "zonegroups": [
        "us"
    ]
}
5.2.10.3.8. Getting a Zone Group

To view the configuration of a zone group, execute:

# radosgw-admin zonegroup get [--rgw-zonegroup=<zonegroup>]

The zone group configuration looks like this:

{
    "id": "90b28698-e7c3-462c-a42d-4aa780d24eda",
    "name": "us",
    "api_name": "us",
    "is_master": "true",
    "endpoints": [
        "http:\/\/rgw1:80"
    ],
    "hostnames": [],
    "hostnames_s3website": [],
    "master_zone": "9248cab2-afe7-43d8-a661-a40bf316665e",
    "zones": [
        {
            "id": "9248cab2-afe7-43d8-a661-a40bf316665e",
            "name": "us-east",
            "endpoints": [
                "http:\/\/rgw1"
            ],
            "log_meta": "true",
            "log_data": "true",
            "bucket_index_max_shards": 0,
            "read_only": "false"
        },
        {
            "id": "d1024e59-7d28-49d1-8222-af101965a939",
            "name": "us-west",
            "endpoints": [
                "http:\/\/rgw2:80"
            ],
            "log_meta": "false",
            "log_data": "true",
            "bucket_index_max_shards": 0,
            "read_only": "false"
        }
    ],
    "placement_targets": [
        {
            "name": "default-placement",
            "tags": []
        }
    ],
    "default_placement": "default-placement",
    "realm_id": "ae031368-8715-4e27-9a99-0c9468852cfe"
}
5.2.10.3.9. Setting a Zone Group

Defining a zone group consists of creating a JSON object, specifying at least the required settings:

  1. name: The name of the zone group. Required.
  2. api_name: The API name for the zone group. Optional.
  3. is_master: Determines if the zone group is the master zone group. Required. note: You can only have one master zone group.
  4. endpoints: A list of all the endpoints in the zone group. For example, you may use multiple domain names to refer to the same zone group. Remember to escape the forward slashes (\/). You may also specify a port (fqdn:port) for each endpoint. Optional.
  5. hostnames: A list of all the hostnames in the zone group. For example, you may use multiple domain names to refer to the same zone group. Optional. The rgw dns name setting will automatically be included in this list. You should restart the gateway daemon(s) after changing this setting.
  6. master_zone: The master zone for the zone group. Optional. Uses the default zone if not specified. note: You can only have one master zone per zone group.
  7. zones: A list of all zones within the zone group. Each zone has a name (required), a list of endpoints (optional), and whether or not the gateway will log metadata and data operations (false by default).
  8. placement_targets: A list of placement targets (optional). Each placement target contains a name (required) for the placement target and a list of tags (optional) so that only users with the tag can use the placement target (i.e., the user’s placement_tags field in the user info).
  9. default_placement: The default placement target for the object index and object data. Set to default-placement by default. You may also set a per-user default placement in the user info for each user.

To set a zone group, create a JSON object consisting of the required fields, save the object to a file (e.g., zonegroup.json); then, execute the following command:

# radosgw-admin zonegroup set --infile zonegroup.json

Where zonegroup.json is the JSON file you created.

Important

The default zone group is_master setting is true by default. If you create a new zone group and want to make it the master zone group, you must either set the default zone group is_master setting to false, or delete the default zone group.

Finally, update the period:

# radosgw-admin period update --commit
5.2.10.3.10. Setting a Zone Group Map

Setting a zone group map consists of creating a JSON object consisting of one or more zone groups, and setting the master_zonegroup for the cluster. Each zone group in the zone group map consists of a key/value pair, where the key setting is equivalent to the name setting for an individual zone group configuration, and the val is a JSON object consisting of an individual zone group configuration.

You may only have one zone group with is_master equal to true, and it must be specified as the master_zonegroup at the end of the zone group map. The following JSON object is an example of a default zone group map.

{
    "zonegroups": [
        {
            "key": "90b28698-e7c3-462c-a42d-4aa780d24eda",
            "val": {
                "id": "90b28698-e7c3-462c-a42d-4aa780d24eda",
                "name": "us",
                "api_name": "us",
                "is_master": "true",
                "endpoints": [
                    "http:\/\/rgw1:80"
                ],
                "hostnames": [],
                "hostnames_s3website": [],
                "master_zone": "9248cab2-afe7-43d8-a661-a40bf316665e",
                "zones": [
                    {
                        "id": "9248cab2-afe7-43d8-a661-a40bf316665e",
                        "name": "us-east",
                        "endpoints": [
                            "http:\/\/rgw1"
                        ],
                        "log_meta": "true",
                        "log_data": "true",
                        "bucket_index_max_shards": 0,
                        "read_only": "false"
                    },
                    {
                        "id": "d1024e59-7d28-49d1-8222-af101965a939",
                        "name": "us-west",
                        "endpoints": [
                            "http:\/\/rgw2:80"
                        ],
                        "log_meta": "false",
                        "log_data": "true",
                        "bucket_index_max_shards": 0,
                        "read_only": "false"
                    }
                ],
                "placement_targets": [
                    {
                        "name": "default-placement",
                        "tags": []
                    }
                ],
                "default_placement": "default-placement",
                "realm_id": "ae031368-8715-4e27-9a99-0c9468852cfe"
            }
        }
    ],
    "master_zonegroup": "90b28698-e7c3-462c-a42d-4aa780d24eda",
    "bucket_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "max_size_kb": -1,
        "max_objects": -1
    }
}

To set a zone group map, execute the following:

# radosgw-admin zonegroup-map set --infile zonegroupmap.json

Where zonegroupmap.json is the JSON file you created. Ensure that you have zones created for the ones specified in the zone group map. Finally, update the period.

# radosgw-admin period update --commit

5.2.10.4. Zones

Ceph Object Gateway supports the notion of zones. A zone defines a logical group consisting of one or more Ceph Object Gateway instances.

Configuring zones differs from typical configuration procedures, because not all of the settings end up in a Ceph configuration file. You can list zones, get a zone configuration and set a zone configuration.

Important

All radosgw-admin zone operations MUST be executed on a host that operates or will operate within the zone.

5.2.10.4.1. Creating a Zone

To create a zone, specify a zone name. If it is a master zone, specify the --master option. Only one zone in a zone group may be a master zone. To add the zone to a zonegroup, specify the --rgw-zonegroup option with the zonegroup name.

Important

Zones must be created on a Ceph Object Gateway node that will be within the zone.

[root@zone] radosgw-admin zone create --rgw-zone=<name> \
                [--zonegroup=<zonegroup-name]\
                [--endpoints=<endpoint:port>[,<endpoint:port>] \
                [--master] [--default] \
                --access-key $SYSTEM_ACCESS_KEY --secret $SYSTEM_SECRET_KEY

Then, update the period:

# radosgw-admin period update --commit
5.2.10.4.2. Deleting a Zone

To delete zone, first remove it from the zonegroup.

# radosgw-admin zonegroup remove --rgw-zonegroup=<name>\
                                 --rgw-zone=<name>

Then, update the period:

# radosgw-admin period update --commit

Next, delete the zone.

Important

This procedure MUST be executed on a host within the zone.

Execute the following:

[root@zone]# radosgw-admin zone delete --rgw-zone<name>

Finally, update the period:

# radosgw-admin period update --commit
Important

Do not delete a zone without removing it from a zone group first. Otherwise, updating the period will fail.

If the pools for the deleted zone will not be used anywhere else, consider deleting the pools. Replace <del-zone> in the example below with the deleted zone’s name.

Important

Once Ceph deletes the zone pools, it deletes all of the data within them in an unrecoverable manner. Only delete the zone pools if Ceph clients no longer need the pool contents.

Important

In a multi-realm cluster, deleting the .rgw.root pool along with the zone pools will remove ALL the realm information for the cluster. Ensure that .rgw.root does not contain other active realms before deleting the .rgw.root pool.

# ceph osd pool delete <del-zone>.rgw.control <del-zone>.rgw.control --yes-i-really-really-mean-it
# ceph osd pool delete <del-zone>.rgw.data.root <del-zone>.rgw.data.root --yes-i-really-really-mean-it
# ceph osd pool delete <del-zone>.rgw.log <del-zone>.rgw.log --yes-i-really-really-mean-it
# ceph osd pool delete <del-zone>.rgw.users.uid <del-zone>.rgw.users.uid --yes-i-really-really-mean-it
Important

After deleting the pools, restart the RGW process.

5.2.10.4.3. Modifying a Zone

To modify a zone, specify the zone name and the parameters you wish to modify.

Important

Zones should be modified on a Ceph Object Gateway node that will be within the zone.

[root@zone]# radosgw-admin zone modify [options]

--access-key=<key>--secret/--secret-key=<key>--master--default--endpoints=<list>

Then, update the period:

# radosgw-admin period update --commit
5.2.10.4.4. Listing Zones

As root, to list the zones in a cluster, execute:

# radosgw-admin zone list
5.2.10.4.5. Getting a Zone

As root, to get the configuration of a zone, execute:

# radosgw-admin zone get [--rgw-zone=<zone>]

The default zone looks like this:

{ "domain_root": ".rgw",
  "control_pool": ".rgw.control",
  "gc_pool": ".rgw.gc",
  "log_pool": ".log",
  "intent_log_pool": ".intent-log",
  "usage_log_pool": ".usage",
  "user_keys_pool": ".users",
  "user_email_pool": ".users.email",
  "user_swift_pool": ".users.swift",
  "user_uid_pool": ".users.uid",
  "system_key": { "access_key": "", "secret_key": ""},
  "placement_pools": [
      {  "key": "default-placement",
         "val": { "index_pool": ".rgw.buckets.index",
                  "data_pool": ".rgw.buckets"}
      }
    ]
  }
5.2.10.4.6. Setting a Zone

Configuring a zone involves specifying a series of Ceph Object Gateway pools. For consistency, we recommend using a pool prefix that is the same as the zone name. See Pools_ for details of configuring pools.

Important

Zones should be set on a Ceph Object Gateway node that will be within the zone.

To set a zone, create a JSON object consisting of the pools, save the object to a file (e.g., zone.json); then, execute the following command, replacing {zone-name} with the name of the zone:

[root@zone]# radosgw-admin zone set --rgw-zone={zone-name} --infile zone.json

Where zone.json is the JSON file you created.

Then, as root, update the period:

# radosgw-admin period update --commit
5.2.10.4.7. Renaming a Zone

To rename a zone, specify the zone name and the new zone name. Execute the following on a host within the zone:

[root@zone]# radosgw-admin zone rename --rgw-zone=<name> --zone-new-name=<name>

Then, update the period:

# radosgw-admin period update --commit

5.3. Configure LDAP and Ceph Object Gateway

Perform the following steps to configure Red Hat Directory Server to authenticate Ceph Object Gateway users.

5.3.1. Installing a Red Hat Directory Server

Red Hat Directory Server should be installed on a Red Hat Enterprise Linux 8 with a graphical user interface (GUI) in order to use the Java Swing GUI Directory and Administration consoles. However, Red Hat Directory Server can still be serviced exclusively from the command line interface (CLI).

Prerequisites

  • Red Hat Enterprise Linux (RHEL) is installed on the server.
  • The Directory Server node’s FQDN is resolvable using DNS or the /etc/hosts file.
  • Register the Directory Server node to the Red Hat subscription management service.
  • A valid Red Hat Directory Server subscription is available in your Red Hat account.

Procedure

  1. Follow the instructions in Chapter 1 and in Chapter 2 of the Red Hat Directory Server Installation Guide.

Additional Resources

5.3.2. Configure the Directory Server Firewall

On the LDAP host, make sure that the firewall allows access to the Directory Server’s secure (636) port, so that LDAP clients can access the Directory Server. Leave the default unsecure port (389) closed.

# firewall-cmd --zone=public --add-port=636/tcp
# firewall-cmd --zone=public --add-port=636/tcp --permanent

5.3.3. Label Ports for SELinux

To ensure SELinux does not block requests, label the ports for SELinux. For details see the Changing Directory Server Port Numbers section in the Administration Guide for Red Hat Directory Server 10.

5.3.4. Configure LDAPS

The Ceph Object Gateway uses a simple ID and password to authenticate with the LDAP server, so the connection requires an SSL certificate for LDAP. To configure Directory Server for LDAP, see the Configuring Secure Connections chapter in the Administration Guide for Red Hat Directory Server 11.

Once the LDAP is working, configure the Ceph Object Gateway servers to trust the Directory Server’s certificate.

  1. Extract/Download a PEM-formatted certificate for the Certificate Authority (CA) that signed the LDAP server’s SSL certificate.
  2. Confirm that /etc/openldap/ldap.conf does not have TLS_REQCERT set.
  3. Confirm that /etc/openldap/ldap.conf contains a TLS_CACERTDIR /etc/openldap/certs setting.
  4. Use the certutil command to add the AD CA to the store at /etc/openldap/certs. For example, if the CA is "msad-frog-MSAD-FROG-CA", and the PEM-formatted CA file is ldap.pem, use the following command:

    # certutil -d /etc/openldap/certs -A -t "TC,," -n "msad-frog-MSAD-FROG-CA" -i /path/to/ldap.pem
  5. Update SELinux on all remote LDAP sites:

    # setsebool -P httpd_can_network_connect on
    Note

    This still has to be set even if SELinux is in permissive mode.

  6. Make the certs database world-readable.

    # chmod 644 /etc/openldap/certs/*

Connect to the server using "ldapwhoami" as a non-root user. For example:

$ ldapwhoami -H ldaps://rh-directory-server.example.com -d 9

The -d 9 option will provide debugging information in case something went wrong with the SSL negotiation.

5.3.5. Check if the Gateway User Exists

Before creating the gateway user, ensure that the Ceph Object Gateway doesn’t already have the user. For example:

# radosgw-admin metadata list user

The user name should NOT be in this list of users.

5.3.6. Add a Gateway User

Create an LDAP user for the Ceph Object Gateway, and make a note of the binddn. Since the Ceph object gateway uses the ceph user, consider using ceph as the username. The user needs to have permissions to search the directory.

Test to ensure that the user creation worked. Where ceph is the user ID under People and example.com is the domain, you can perform a search for the user.

The Ceph Object Gateway will bind to this user as specified in the rgw_ldap_binddn.

Test to ensure that the user creation worked. Where ceph is the user ID under People and example.com is the domain, you can perform a search for the user.

# ldapsearch -x -D "uid=ceph,ou=People,dc=example,dc=com" -W -H ldaps://example.com -b "ou=People,dc=example,dc=com" -s sub 'uid=ceph'

On each gateway node, create a file for the user’s secret. For example, the secret may get stored in a file entitled /etc/bindpass. For security, change the owner of this file to the ceph user and group to ensure it is not globally readable.

Add the rgw_ldap_secret option:

Syntax

ceph config set client.rgw OPTION VALUE

Example

[root@mon ~]# ceph config set client.rgw rgw_ldap_secret /etc/bindpass

Finally, copy the updated configuration file to each Ceph node.

# scp /etc/ceph/ceph.conf <node>:/etc/ceph

5.3.7. Configure the Gateway to use LDAP

  1. Add the following settings in the [global] section of the Ceph configuration file. For example:

    [global]
    rgw_ldap_uri = ldaps://<fqdn>:636
    rgw_ldap_binddn = "<binddn>"
    rgw_ldap_secret = "/etc/bindpass"
    rgw_ldap_searchdn = "<seachdn>"
    rgw_ldap_dnattr = "uid"
    rgw_s3_auth_use_ldap = true

    For the rgw_ldap_uri setting, substitute <fqdn> with the fully qualified domain name of the LDAP server. If there is more than one LDAP server, specify each domain.

    For the rgw_ldap_binddn setting, substitute <binddn> with the bind domain. With a domain of example.com and a ceph user under users and accounts, it should look something like this:

    rgw_ldap_binddn = "uid=ceph,cn=users,cn=accounts,dc=example,dc=com"

    For the rgw_ldap_searchdn setting, substitute <searchdn> with the search domain. With a domain of example.com and users under users and accounts, it should look something like this:

    rgw_ldap_searchdn = "cn=users,cn=accounts,dc=example,dc=com"
  2. Copy the updated configuration file to each Ceph node.

    scp /etc/ceph/ceph.conf <hostname>:/etc/ceph
  3. Restart the Ceph Object Gateway.

    Note

    Use the output from the ceph orch ps command, under the NAME column to get the SERVICE_TYPE.ID information.

    1. To restart the Ceph Object Gateway on individual node in the storage cluster:

      Syntax

      systemctl restart ceph-CLUSTER_ID@SERVICE_TYPE.ID.service

      Example

      [root@primary-zone]# systemctl restart ceph-c4b34c6f-8365-11ba-dc31-529020a7702d@rgw.realm.zone.node1.gwasto.service

    2. To restart the Ceph Object Gateways on all nodes in the storage cluster:

      Syntax

      ceph orch restart SERVICE_TYPE

      Example

      [root@primary-zone]# ceph orch restart rgw

5.3.8. Using a Custom Search Filter

You can create a custom search filter to limit user access by using the rgw_ldap_searchfilter setting. Specify this setting under the [global] section of the Ceph configuration file (/etc/ceph/ceph.conf). There are two ways to use the rgw_ldap_searchfilter setting:

  1. Specifying a Partial Filter

    Example

    "objectclass=inetorgperson"

    The Ceph Object Gateway will generate the search filter with the user name from the token and the value of rgw_ldap_dnattr. The constructed filter is then combined with the partial filter from the rgw_ldap_searchfilter value. For example, the user name and the settings generate the final search filter:

    Example

    "(&(uid=joe)(objectclass=inetorgperson))"

    User joe will only be granted access if he is found in the LDAP directory, he has an object class of inetorgperson, and he specifies a valid password.

  2. Specifying a Complete Filter

    A complete filter must contain a USERNAME token which will be substituted with the user name during the authentication attempt. The rgw_ldap_dnattr setting is not used in this case. For example, to limit valid users to a specific group, use the following filter:

    Example

    "(&(uid=@USERNAME@)(memberOf=cn=ceph-users,ou=groups,dc=mycompany,dc=com))"

5.3.9. Add an S3 User to the LDAP Server

In the administrative console on LDAP server, create at least one S3 user so that an S3 client can use the LDAP user credentials. Make a note of the user name and secret for use when passing the credentials to the S3 client.

5.3.10. Export an LDAP Token

When running Ceph Object Gateway with LDAP, the access token is all that is required. However, the access token is created from the access key and secret. Export the access key and secret key as an LDAP token.

  1. Export the access key.

    # export RGW_ACCESS_KEY_ID="<username>"
  2. Export the secret.

    # export RGW_SECRET_ACCESS_KEY="<password>"
  3. Export the token. For LDAP, use ldap as the token type (ttype).

    # radosgw-token --encode --ttype=ldap

    For Active Directory, use ad as the token type.

    # radosgw-token --encode --ttype=ad

    The result is a base-64 encoded string, which is the access token. Provide this access token to S3 clients in lieu of the access key. The secret is no longer required.

  4. (Optional) For added convenience, export the base-64 encoded string to the RGW_ACCESS_KEY_ID environment variable if the S3 client uses the environment variable.

    # export RGW_ACCESS_KEY_ID="ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAibGRhcCIsCiAgICAgICAgImlkIjogImNlcGgiLAogICAgICAgICJrZXkiOiAiODAwI0dvcmlsbGEiCiAgICB9Cn0K"

5.3.11. Test the Configuration with an S3 Client

Pick a Ceph Object Gateway client such as Python Boto. Configure it to use the RGW_ACCESS_KEY_ID environment variable. Alternatively, you may copy the base-64 encoded string and specify it as the access key. Then, run the Ceph client.

Note

The secret is no longer required.

5.4. Configure Active Directory and Ceph Object Gateway

Perform the following steps to configure an Active Directory server to authenticate Ceph Object Gateway users.

5.4.1. Using Microsoft Active Directory

Ceph Object Gateway LDAP authentication is compatible with any LDAP-compliant directory service that can be configured for simple bind, including Microsoft Active Directory. Using Active Directory is similar to using RH Directory server in that the Ceph object gateway binds as the user configured in the rgw_ldap_binddn setting, and using LDAPs to ensure security.

The process for configuring Active Directory is essentially identical with Configuring LDAP and Ceph Object Gateway, but may have some Windows-specific usage.

5.4.2. Configuring Active Directory for LDAPS

Active Directory LDAP servers are configured to use LDAPs by default. Windows Server 2012 and higher can use Active Directory Certificate Services. Instructions for generating and installing SSL certificates for use with Active Directory LDAP are available in the following MS TechNet article: LDAP over SSL (LDAPS) Certificate.

Note

Ensure that port 636 is open on the Active Directory host.

5.4.3. Check if the Gateway User Exists

Before creating the gateway user, ensure that the Ceph Object Gateway doesn’t already have the user. For example:

# radosgw-admin metadata list user

The user name should NOT be in this list of users.

5.4.4. Add a Gateway User

Create an LDAP user for the Ceph Object Gateway, and make a note of the binddn. Since the Ceph object gateway uses the ceph user, consider using ceph as the username. The user needs to have permissions to search the directory.

Test to ensure that the user creation worked. Where ceph is the user ID under People and example.com is the domain, you can perform a search for the user.

The Ceph Object Gateway will bind to this user as specified in the rgw_ldap_binddn.

Test to ensure that the user creation worked. Where ceph is the user ID under People and example.com is the domain, you can perform a search for the user.

# ldapsearch -x -D "uid=ceph,ou=People,dc=example,dc=com" -W -H ldaps://example.com -b "ou=People,dc=example,dc=com" -s sub 'uid=ceph'

On each gateway node, create a file for the user’s secret. For example, the secret may get stored in a file entitled /etc/bindpass. For security, change the owner of this file to the ceph user and group to ensure it is not globally readable.

Add the rgw_ldap_secret option:

Syntax

ceph config set client.rgw OPTION VALUE

Example

[root@mon ~]# ceph config set client.rgw rgw_ldap_secret /etc/bindpass

Finally, copy the updated configuration file to each Ceph node.

# scp /etc/ceph/ceph.conf <node>:/etc/ceph

5.4.5. Configuring the Gateway to use Active Directory

  1. Add the following options after setting the rgw_ldap_secret setting:

    Syntax

    ceph config set client.rgw OPTION VALUE

    Example

    [root@mon ~]# ceph config set client.rgw rgw_ldap_uri ldaps://_FQDN_:636
    [root@mon ~]# ceph config set client.rgw rgw_ldap_binddn "_BINDDN_"
    [root@mon ~]# ceph config set client.rgw rgw_ldap_searchdn "_SEARCHDN_"
    [root@mon ~]# ceph config set client.rgw rgw_ldap_dnattr "cn"
    [root@mon ~]# ceph config set client.rgw rgw_s3_auth_use_ldap true

    For the rgw_ldap_uri setting, substitute FQDN with the fully qualified domain name of the LDAP server. If there is more than one LDAP server, specify each domain.

    For the rgw_ldap_binddn setting, substitute BINDDN with the bind domain. With a domain of example.com and a ceph user under users and accounts, it should look something like this:

    Exaple

    rgw_ldap_binddn "uid=ceph,cn=users,cn=accounts,dc=example,dc=com"

    For the rgw_ldap_searchdn setting, substitute SEARCHDN with the search domain. With a domain of example.com and users under users and accounts, it should look something like this:

    rgw_ldap_searchdn "cn=users,cn=accounts,dc=example,dc=com"
  2. Copy the updated configuration file to each Ceph node.

    scp /etc/ceph/ceph.conf <hostname>:/etc/ceph
  3. Restart the Ceph Object Gateway.

    Note

    Use the output from the ceph orch ps command, under the NAME column to get the SERVICE_TYPE.ID information.

    1. To restart the Ceph Object Gateway on individual node in the storage cluster:

      Syntax

      systemctl restart ceph-CLUSTER_ID@SERVICE_TYPE.ID.service

      Example

      [root@primary-zone]# systemctl restart ceph-c4b34c6f-8365-11ba-dc31-529020a7702d@rgw.realm.zone.node1.gwasto.service

    2. To restart the Ceph Object Gateways on all nodes in the storage cluster:

      Syntax

      ceph orch restart SERVICE_TYPE

      Example

      [root@primary-zone]# ceph orch restart rgw

5.4.6. Add an S3 User to the LDAP Server

In the administrative console on LDAP server, create at least one S3 user so that an S3 client can use the LDAP user credentials. Make a note of the user name and secret for use when passing the credentials to the S3 client.

5.4.7. Export an LDAP Token

When running Ceph Object Gateway with LDAP, the access token is all that is required. However, the access token is created from the access key and secret. Export the access key and secret key as an LDAP token.

  1. Export the access key.

    # export RGW_ACCESS_KEY_ID="<username>"
  2. Export the secret.

    # export RGW_SECRET_ACCESS_KEY="<password>"
  3. Export the token. For LDAP, use ldap as the token type (ttype).

    # radosgw-token --encode --ttype=ldap

    For Active Directory, use ad as the token type.

    # radosgw-token --encode --ttype=ad

    The result is a base-64 encoded string, which is the access token. Provide this access token to S3 clients in lieu of the access key. The secret is no longer required.

  4. (Optional) For added convenience, export the base-64 encoded string to the RGW_ACCESS_KEY_ID environment variable if the S3 client uses the environment variable.

    # export RGW_ACCESS_KEY_ID="ewogICAgIlJHV19UT0tFTiI6IHsKICAgICAgICAidmVyc2lvbiI6IDEsCiAgICAgICAgInR5cGUiOiAibGRhcCIsCiAgICAgICAgImlkIjogImNlcGgiLAogICAgICAgICJrZXkiOiAiODAwI0dvcmlsbGEiCiAgICB9Cn0K"

5.4.8. Test the Configuration with an S3 Client

Pick a Ceph Object Gateway client such as Python Boto. Configure it to use the RGW_ACCESS_KEY_ID environment variable. Alternatively, you may copy the base-64 encoded string and specify it as the access key. Then, run the Ceph client.

Note

The secret is no longer required.

5.5. The Ceph Object Gateway and OpenStack Keystone

As a storage administrator, you can use OpenStack’s Keystone authentication service to authenticate users through the Ceph Object Gateway. Before you can configure the Ceph Object Gateway, you need to configure Keystone first. This enables the Swift service, and point the Keystone service to the Ceph Object Gateway. Next, you need to configure the Ceph Object Gateway to accept authentication requests from the Keystone service.

5.5.1. Prerequisites

  • A running Red Hat OpenStack Platform environment.
  • A running Red Hat Ceph Storage environment.
  • A running Ceph Object Gateway environment.

5.5.2. Keystone authentication and the Ceph Object Gateway

Organizations using OpenStack Keystone to authenticate users can integrate Keystone with the Ceph Object Gateway. The Ceph Object Gateway enables the gateway to accept a Keystone token, authenticate the user and create a corresponding Ceph Object Gateway user. When Keystone validates a token, the gateway considers the user authenticated.

Benefits

  • Managing users with Keystone
  • Automatic User Creation in the Ceph Object Gateway
  • The Ceph Object Gateway will query Keystone periodically for a list of revoked tokens.

5.5.3. Creating the Swift service

Before configuring the Ceph Object Gateway, configure Keystone so that the Swift service is enabled and pointing to the Ceph Object Gateway.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Access to the Ceph software repository.
  • Root-level access to OpenStack controller node.

Procedure

  1. Create the Swift service:

    [root@swift~]# openstack service create --name=swift --description="Swift Service" object-store

    Creating the service will echo the service settings.

    Table 5.1. Example

    FieldValue

    description

    Swift Service

    enabled

    True

    id

    37c4c0e79571404cb4644201a4a6e5ee

    name

    swift

    type

    object-store

5.5.4. Setting the Ceph Object Gateway endpoints

After creating the Swift service, point the service to a Ceph Object Gateway.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Access to the Ceph software repository.
  • A running Swift service on a Red Hat OpenStack Platform 13, 15, or 16 environment.

Procedure

  1. Create the OpenStack endpoints pointing to the Ceph Object Gateway:

    Syntax

    openstack endpoint create --region REGION_NAME swift admin "URL"
    openstack endpoint create --region REGION_NAME swift public "URL"
    openstack endpoint create --region REGION_NAME swift internal "URL"

    Replace REGION_NAME with the name of the gateway’s zone group name or region name. Replace URL with URLs appropriate for the Ceph Object Gateway.

    Example

    [root@osp ~]# openstack endpoint create --region us-west swift admin "http://radosgw.example.com:8080/swift/v1"
    [root@osp ~]# openstack endpoint create --region us-west swift public "http://radosgw.example.com:8080/swift/v1"
    [root@osp ~]# openstack endpoint create --region us-west swift internal "http://radosgw.example.com:8080/swift/v1"

    FieldValue

    adminurl

    http://radosgw.example.com:8080/swift/v1

    id

    e4249d2b60e44743a67b5e5b38c18dd3

    internalurl

    http://radosgw.example.com:8080/swift/v1

    publicurl

    http://radosgw.example.com:8080/swift/v1

    region

    us-west

    service_id

    37c4c0e79571404cb4644201a4a6e5ee

    service_name

    swift

    service_type

    object-store

    Setting the endpoints will output the service endpoint settings.

5.5.5. Verifying Openstack is using the Ceph Object Gateway endpoints

After creating the Swift service and setting the endpoints, show the endpoints to ensure that all settings are correct.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Access to the Ceph software repository.

Procedure

  1. Verify settings in the configuration file:
[root@swift~]# openstack endpoint show object-store

Showing the endpoints will echo the endpoints settings, and the service settings.

Table 5.2. Example

FieldValue

adminurl

http://radosgw.example.com:8080/swift/v1

enabled

True

id

e4249d2b60e44743a67b5e5b38c18dd3

internalurl

http://radosgw.example.com:8080/swift/v1

publicurl

http://radosgw.example.com:8080/swift/v1

region

us-west

service_id

37c4c0e79571404cb4644201a4a6e5ee

service_name

swift

service_type

object-store

5.5.6. Configuring the Ceph Object Gateway to use Keystone SSL

Converting the OpenSSL certificates that Keystone uses configures the Ceph Object Gateway to work with Keystone. When the Ceph Object Gateway interacts with OpenStack’s Keystone authentication, Keystone will terminate with a self-signed SSL certificate.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Access to the Ceph software repository.

Procedure

  1. Convert the OpenSSL certificate to the nss db format:

    Example

    [root@osp ~]# mkdir /var/ceph/nss
    
    [root@osp ~]# mkdir /var/ceph/nss openssl x509 -in /etc/keystone/ssl/certs/ca.pem -pubkey | \
        certutil -d /var/ceph/nss -A -n ca -t "TCu,Cu,Tuw"
    [root@osp ~]# mkdir /var/ceph/nss openssl x509 -in /etc/keystone/ssl/certs/signing_cert.pem -pubkey | \
        certutil -A -d /var/ceph/nss -n signing_cert -t "P,P,P"

  2. Install Keystone’s SSL certificate in the node running the Ceph Object Gateway. Alternatively set the value of the configurable rgw_keystone_verify_ssl setting to false.

    Setting rgw_keystone_verify_ssl to false means that the gateway won’t attempt to verify the certificate.

5.5.7. Configuring the Ceph Object Gateway to use Keystone authentication

Configure the Red Hat Ceph Storage to use OpenStack’s Keystone authentication.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Access to the Ceph software repository.
  • Have admin privileges to the production environment.

Procedure

  1. Do the following for each gateway instance.

    1. Set the rgw_s3_auth_use_keystone option to true:

      Example

      [root@mon ~]# ceph config set client.rgw rgw_s3_auth_use_keystone true

    2. Set the nss_db_path setting to the path where the NSS database is stored:

      Example

      [root@mon ~]# ceph config set client.rgw nss_db_path "/var/lib/ceph/radosgw/ceph-rgw.rgw01/nss"

  2. Provide authentication credentials:

    It is possible to configure a Keystone service tenant, user and password for keystone for the OpenStack Identity API, similar to the way system administrators tend to configure OpenStack services. Providing a username and password avoids providing the shared secret to the rgw_keystone_admin_token setting.

    Important

    Red Hat recommends disabling authentication by admin token in production environments. The service tenant credentials should have admin privileges.

    The necessary configuration options are:

    Syntax

    ceph config set client.rgw rgw_keystone_url KEYSTONE_URL:_ADMIN_PORT_
    ceph config set client.rgw rgw_keystone_admin_user KEYSTONE_TENANT_USER_NAME
    ceph config set client.rgw rgw_keystone_admin_password KEYSTONE_TENANT_USER_PASSWORD
    ceph config set client.rgw rgw_keystone_admin_tenant KEYSTONE_TENANT_NAME
    ceph config set client.rgw rgw_keystone_accepted_roles KEYSTONE_ACCEPTED_USER_ROLES
    ceph config set client.rgw rgw_keystone_token_cache_size NUMBER_OF_TOKENS_TO_CACHE
    ceph config set client.rgw rgw_keystone_revocation_interval NUMBER_OF_SECONDS_BEFORE_CHECKING_REVOKED_TICKETS
    ceph config set client.rgw rgw_keystone_make_new_tenants TRUE_FOR_PRIVATE_TENANT_FOR_EACH_NEW_USER

    A Ceph Object Gateway user is mapped into a Keystone tenant. A Keystone user has different roles assigned to it on possibly more than a single tenant. When the Ceph Object Gateway gets the ticket, it looks at the tenant, and the user roles that are assigned to that ticket, and accepts or rejects the request according to the rgw_keystone_accepted_roles configurable.

Additional Resources

5.5.8. Restarting the Ceph Object Gateway daemon

Restarting the Ceph Object Gateway must be done to active configuration changes.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Access to the Ceph software repository.
  • admin privileges to the production environment.

Procedure

  1. Once you have saved the Ceph configuration file and distributed it to each Ceph node, restart the Ceph Object Gateway instances:

    Note

    Use the output from the ceph orch ps command, under the NAME column to get the SERVICE_TYPE.ID information.

    1. To restart the Ceph Object Gateway on individual node in the storage cluster:

      Syntax

      systemctl restart ceph-CLUSTER_ID@SERVICE_TYPE.ID.service

      Example

      [root@primary-zone]# systemctl restart ceph-c4b34c6f-8365-11ba-dc31-529020a7702d@rgw.realm.zone.node1.gwasto.service

    2. To restart the Ceph Object Gateways on all nodes in the storage cluster:

      Syntax

      ceph orch restart SERVICE_TYPE

      Example

      [root@primary-zone]# ceph orch restart rgw

Chapter 6. Security

As a storage administrator, securing the storage cluster environment is important. Red Hat Ceph Storage provides encryption and key management to secure the Ceph Object Gateway access point.

6.1. Prerequisites

  • A healthy running Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway software.

6.2. S3 server-side encryption

The Ceph Object Gateway supports server-side encryption of uploaded objects for the S3 application programing interface (API). Server-side encryption means that the S3 client sends data over HTTP in its unencrypted form, and the Ceph Object Gateway stores that data in the Red Hat Ceph Storage cluster in encrypted form.

Note

Red Hat does NOT support S3 object encryption of Static Large Object (SLO) or Dynamic Large Object (DLO).

Important

To use encryption, client requests MUST send requests over an SSL connection. Red Hat does not support S3 encryption from a client unless the Ceph Object Gateway uses SSL. However, for testing purposes, administrators can disable SSL during testing by setting the rgw_crypt_require_ssl configuration setting to false at runtime, using the ceph config set client.rgw command, and then restarting the Ceph Object Gateway instance.

In a production environment, it might not be possible to send encrypted requests over SSL. In such a case, send requests using HTTP with server-side encryption.

For information about how to configure HTTP with server-side encryption, see the Additional Resources section below.

There are two options for the management of encryption keys:

Customer-provided Keys

When using customer-provided keys, the S3 client passes an encryption key along with each request to read or write encrypted data. It is the customer’s responsibility to manage those keys. Customers must remember which key the Ceph Object Gateway used to encrypt each object.

Ceph Object Gateway implements the customer-provided key behavior in the S3 API according to the Amazon SSE-C specification.

Since the customer handles the key management and the S3 client passes keys to the Ceph Object Gateway, the Ceph Object Gateway requires no special configuration to support this encryption mode.

Key Management Service

When using a key management service, the secure key management service stores the keys and the Ceph Object Gateway retrieves them on demand to serve requests to encrypt or decrypt data.

Ceph Object Gateway implements the key management service behavior in the S3 API according to the Amazon SSE-KMS specification.

Important

Currently, the only tested key management implementations are HashiCorp Vault, and OpenStack Barbican. However, OpenStack Barbican is a Technology Preview and is not supported for use in production systems.

6.3. Server-side encryption requests

In a production environment, clients often contact the Ceph Object Gateway through a proxy. This proxy is referred to as a load balancer because it connects to multiple Ceph Object Gateways. When the client sends requests to the Ceph Object Gateway, the load balancer routes those requests to the multiple Ceph Object Gateways, thus distributing the workload.

In this type of configuration, it is possible that SSL terminations occur both at a load balancer and between the load balancer and the multiple Ceph Object Gateways. Communication occurs using HTTP only. To set up the Ceph Object Gateways to accept the server-side encryption requests, see Configuring server-side encryption.

6.4. Configuring server-side encryption

As a storage administrator, you can set up server-side encryption to send requests to the Ceph Object Gateway using HTTP, in cases where it might not be possible to send encrypted requests over SSL.

This procedure uses HAProxy as proxy and load balancer.

Prerequisites

  • Root-level access to all nodes in the storage cluster.
  • A running Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway software.
  • Installation of the HAProxy software.

Procedure

  1. Edit the haproxy.cfg file:

    Example

    frontend http_web *:80
        mode http
        default_backend rgw
    
    frontend rgw­-https
      bind *:443 ssl crt /etc/ssl/private/example.com.pem
      default_backend rgw
    
    backend rgw
        balance roundrobin
        mode http
        server  rgw1 10.0.0.71:8080 check
        server  rgw2 10.0.0.80:8080 check

  2. Comment out the lines that allow access to the http front end and add instructions to direct HAProxy to use the https front end instead:

    Example

    #     frontend http_web *:80
    #     mode http
    #     default_backend rgw
    
    frontend rgw­-https
      bind *:443 ssl crt /etc/ssl/private/example.com.pem
      http-request set-header X-Forwarded-Proto https if { ssl_fc }
      http-request set-header X-Forwarded-Proto https
    # here we set the incoming HTTPS port on the load balancer (eg : 443)
      http-request set-header X-Forwarded-Port 443
      default_backend rgw
    
    backend rgw
        balance roundrobin
        mode http
        server  rgw1 10.0.0.71:8080 check
        server  rgw2 10.0.0.80:8080 check

  3. Set the rgw_trust_forwarded_https option to true:

    Example

    [root@mon ~]# ceph config set client.rgw rgw_trust_forwarded_https true

  4. Enable and start HAProxy:

    [root@haproxy]# systemctl enable haproxy
    [root@haproxy]# systemctl start haproxy

6.5. The HashiCorp Vault

As a storage administrator, you can securely store keys, passwords and certificates in the HashiCorp Vault for use with the Ceph Object Gateway. The HashiCorp Vault provides a secure key management service for server-side encryption used by the Ceph Object Gateway.

Ceph Vault Integration Diagram

The basic workflow:

  1. The client requests the creation of a secret key from the Vault based on an object’s key ID.
  2. The client uploads an object with the object’s key ID to the Ceph Object Gateway.
  3. The Ceph Object Gateway then requests the newly created secret key from the Vault.
  4. The Vault replies to the request by returning the secret key to the Ceph Object Gateway.
  5. Now the Ceph Object Gateway can encrypt the object using the new secret key.
  6. After encryption is done the object is then stored on the Ceph OSD.
Important

Red Hat works with our technology partners to provide this documentation as a service to our customers. However, Red Hat does not provide support for this product. If you need technical assistance for this product, then contact Hashicorp for support.

6.5.1. Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway software.
  • Installation of the HashiCorp Vault software.

6.5.2. Secret engines for Vault

The HashiCorp Vault provides several secret engines to generate, store, or encrypt data. The application programming interface (API) send data calls to the secret engine asking for action on that data, and the secret engine returns a result of that action request.

The Ceph Object Gateway supports two of the HashiCorp Vault secret engines:

  • Key/Value version 2
  • Transit

Key/Value version 2

The Key/Value secret engine stores random secrets within the Vault, on disk. With version 2 of the kv engine, a key can have a configurable number of versions. The default number of versions is 10. Deleting a version does not delete the underlying data, but marks the data as deleted, allowing deleted versions to be undeleted. You can use the API endpoint or the destroy command to permanently remove a version’s data. To delete all versions and metadata for a key, you can use the metadata command or the API endpoint. The key names must be strings, and the engine will convert non-string values into strings when using the command-line interface. To preserve non-string values, provide a JSON file or use the HTTP application programming interface (API).

Note

For access control list (ACL) policies, the Key/Value secret engine recognizes the distinctions between the create and update capabilities.

Transit

The Transit secret engine performs cryptographic functions on in-transit data. The Transit secret engine can generate hashes, can be a source of random bytes, and can also sign and verify data. The Vault does not store data when using the Transit secret engine. The Transit secret engine supports key derivation, by allowing the same key to be used for multiple purposes. Also, the transit secret engine supports key versioning. The Transit secret engine supports these key types:

aes128-gcm96
AES-GCM with a 128-bit AES key and a 96-bit nonce; supports encryption, decryption, key derivation, and convergent encryption
aes256-gcm96
AES-GCM with a 256-bit AES key and a 96-bit nonce; supports encryption, decryption, key derivation, and convergent encryption (default)
chacha20-poly1305
ChaCha20-Poly1305 with a 256-bit key; supports encryption, decryption, key derivation, and convergent encryption
ed25519
Ed25519; supports signing, signature verification, and key derivation
ecdsa-p256
ECDSA using curve P-256; supports signing and signature verification
ecdsa-p384
ECDSA using curve P-384; supports signing and signature verification
ecdsa-p521
ECDSA using curve P-521; supports signing and signature verification
rsa-2048
2048-bit RSA key; supports encryption, decryption, signing, and signature verification
rsa-3072
3072-bit RSA key; supports encryption, decryption, signing, and signature verification
rsa-4096
4096-bit RSA key; supports encryption, decryption, signing, and signature verification

Additional Resources

6.5.3. Authentication for Vault

The HashiCorp Vault supports several types of authentication mechanisms. The Ceph Object Gateway currently supports the Vault agent method. The Ceph Object Gateway uses the rgw_crypt_vault_auth, and rgw_crypt_vault_addr options to configure the use of the HashiCorp Vault.

Important

Red Hat supports the usage of Vault agent as the authentication method for containers and the usage of token authentication is not supported on containers.

Vault Agent

The Vault agent is a daemon that runs on a client node and provides client-side caching, along with token renewal. The Vault agent typically runs on the Ceph Object Gateway node. Run the Vault agent and refresh the token file. When the Vault agent is used in this mode, you can use file system permissions to restrict who has access to the usage of tokens. Also the Vault agent can act as a proxy server that is, Vault will add a token when required and add it to the requests passed to it before forwarding them to the actual server. The Vault agent can still handle token renewal just as it would when storing a token in the Filesystem. It is required to secure the network that Ceph Object Gateways uses to connect with the Vault agent for example the Vault agent listens to only the localhost.

Additional Resources

  • See the Vault Agent documentation on Vault’s project site for more information.

6.5.4. Namespaces for Vault

Using HashiCorp Vault as an enterprise service provides centralized management for isolated namespaces that teams within an organization can use. These isolated namespace environments are known as tenants, and teams within an organization can utilize these tenants to isolate their policies, secrets, and identities from other teams. The namespace features of Vault help support secure multi-tenancy from within a single infrastructure.

Additional Resources

6.5.5. Transit engine compatibility support

There is compatibility support for the previous versions of Ceph which used the Transit engine as a simple key store. You can use the compat option in the Transit engine to configure the compatibility support. You can disable previous support with the following command:

Example

[root@vault ~]# ceph config set client.rgw rgw_crypt_vault_secret_engine transit compat=0

Note

This is the default for future versions and you can use the current version for new installations.

The normal default with the current version is:

Example

[root@vault ~]# ceph config set client.rgw rgw_crypt_vault_secret_engine transit compat=1

This enables the new engine for newly created objects and still allows the old engine to be used for the old objects. To access old and new objects, the Vault token must have both the old and new transit policies.

You can force use only the old engine with the following command:

Example

[root@vault ~]# ceph config set client.rgw rgw_crypt_vault_secret_engine transit compat=2

This mode is selected by default if the Vault ends in export/encryption-key.

Important

After configuring the client.rgw options, you need to restart the Ceph Object Gateway daemons for the new values to take effect.

Additional Resources

  • See the Vault Agent documentation on Vault’s project site for more information.

6.5.6. Creating token policies for Vault

A token policy specifies the powers that all Vault tokens have. One token can have multiple policies. You should use the required policy for the configuration.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Installation of the HashiCorp Vault software.
  • Root-level access to the HashiCorp Vault node.

Procedure

  1. Create a token policy:

    1. For the Key/Value secret engine:

      Example

      [root@vault ~]# vault policy write rgw-kv-policy -<<EOF
       path "secret/data/*" {
         capabilities = ["read"]
       }
      EOF

    2. For the Transit engine:

      Example

      [root@vault ~]# vault policy write rgw-transit-policy -<<EOF
        path "transit/keys/*" {
          capabilities = [ "create", "update" ]
          denied_parameters = {"exportable" = [], "allow_plaintext_backup" = [] }
        }
      
        path "transit/keys/*" {
          capabilities = ["read", "delete"]
        }
      
        path "transit/keys/" {
          capabilities = ["list"]
        }
      
        path "transit/keys/+/rotate" {
          capabilities = [ "update" ]
        }
      
        path "transit/*" {
          capabilities = [ "update" ]
        }
      EOF

      Note

      If you have used the Transit secret engine on an older version of Ceph,the token policy is:

      Example

      [root@vault ~]# vault policy write old-rgw-transit-policy -<<EOF
        path "transit/export/encryption-key/*" {
          capabilities = ["read"]
        }
      EOF

6.5.7. Configuring the Ceph Object Gateway to use the Vault

To configure the Ceph Object Gateway to use the HashiCorp Vault it must be set as the encryption key store. Currently, the Ceph Object Gateway supports two different secret engines, and two different authentication methods.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway software.
  • Root-level access to a Ceph Object Gateway node.

Procedure

  1. Use the ceph config set client.rgw OPTION VALUE command to enable Vault as the encryption key store:

    Syntax

    ceph config set client.rgw rgw_crypt_s3_kms_backend vault

  2. Add the following options and values:

    Syntax

    ceph config set client.rgw rgw_crypt_vault_auth agent
    ceph config set client.rgw rgw_crypt_vault_addr http://VAULT_SERVER:8100

  3. Customize the policy as per the use case.
  4. Get the role-id:

    Example

    [root@vault ~]# vault read auth/approle/role/rgw-ap/role-id -format=json | \ jq -r .data.role_id > _PATH_TO_FILE_

  5. Get the secret-id:

    Example

    [root@vault ~]# vault read auth/approle/role/rgw-ap/role-id -format=json | \ jq -r .data.secret_id > _PATH_TO_FILE_

  6. Create the configuration for the Vault agent:

    Example

    pid_file = "/run/kv-vault-agent-pid"
    auto_auth {
      method "AppRole" {
        mount_path = "auth/approle"
        config = {
          role_id_file_path ="/root/vault_configs/kv-agent-role-id"
          secret_id_file_path ="/root/vault_configs/kv-agent-secret-id"
          remove_secret_id_file_after_reading ="false"
        }
      }
    }
    cache {
      use_auto_auth_token = true
    }
    listener "tcp" {
      address = "127.0.0.1:8100"
      tls_disable = true
    }
    vault {
      address = "http://10.8.128.9:8200"
    }

  7. Use systemctl to run the persistent daemon:

    Example

    [root@vault ~]# /usr/local/bin/vault agent -config=/usr/local/etc/vault/rgw-agent.hcl

  8. A token file is populated with a valid token when the Vault agent runs.
  9. Select a Vault secret engine, either Key/Value or Transit.

    1. If using Key/Value, then add the following line:

      Example

      [root@vault ~]# ceph config set client.rgw rgw_crypt_vault_secret_engine kv

    2. If using Transit, then add the following line:

      Example

      [root@vault ~]# ceph config set client.rgw rgw_crypt_vault_secret_engine transit

  10. Use the ceph config set client.rgw OPTION VALUE command to set the Vault namespace to retrieve the encryption keys:

    Example

    [root@vault ~]# ceph config set client.rgw rgw_crypt_vault_namespace testnamespace1

  11. Restrict where the Ceph Object Gateway retrieves the encryption keys from the Vault by setting a path prefix:

    Example

    [root@vault ~]# ceph config set client.rgw rgw_crypt_vault_prefix /v1/secret/data

    1. For exportable Transit keys, set the prefix path as follows:

      Example

      [root@vault ~]# ceph config set client.rgw rgw_crypt_vault_prefix /v1/transit/export/encryption-key

      Assuming the domain name of the Vault server is vault-server, the Ceph Object Gateway will fetch encrypted transit keys from the following URL:

      Example

      http://vault-server:8200/v1/transit/export/encryption-key

  12. Restart the Ceph Object Gateway daemons.

    1. To restart the Ceph Object Gateway on individual node in the storage cluster:

      Syntax

      systemctl restart ceph-CLUSTER_ID@SERVICE_TYPE.ID.service

      Example

      [root@vault]# systemctl restart ceph-c4b34c6f-8365-11ba-dc31-529020a7702d@rgw.realm.zone.node1.gwasto.service

    2. To restart the Ceph Object Gateways on all nodes in the storage cluster:

      Syntax

      ceph orch restart SERVICE_TYPE

      Example

      [root@vault]# ceph orch restart rgw

Additional Resources

  • See the Secret engines for Vault section of the Red Hat Ceph Storage Object Gateway Configuration and Administration Guide for more details.
  • See the Authentication for Vault section of the Red Hat Ceph Storage Object Gateway Configuration and Administration Guide for more details.

6.5.8. Creating a key using the kv engine

Configure the HashiCorp Vault Key/Value secret engine (kv) so you can create a key for use with the Ceph Object Gateway. Secrets are stored as key-value pairs in the kv secret engine.

Important

Keys for server-side encyption must be 256-bits long and encoded using base64.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Installation of the HashiCorp Vault software.
  • Root-level access to the HashiCorp Vault node.

Procedure

  1. Enable the Key/Value version 2 secret engine:

    Example

    vault secrets enable -path secret kv-v2

  2. Create a new key:

    Syntax

    vault kv put secret/PROJECT_NAME/BUCKET_NAME key=$(openssl rand -base64 32)

    Example

    [root@vault ~]# vault kv put secret/myproject/mybucketkey key=$(openssl rand -base64 32)
    
    ====== Metadata ======
    Key              Value
    ---              -----
    created_time     2020-02-21T17:01:09.095824999Z
    deletion_time    n/a
    destroyed        false
    version          1

6.5.9. Creating a key using the transit engine

Configure the HashiCorp Vault Transit secret engine (transit) so you can create a key for use with the Ceph Object Gateway. Creating keys with the Transit secret engine must be exportable in order to be used for server-side encryption with the Ceph Object Gateway.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Installation of the HashiCorp Vault software.
  • Root-level access to the HashiCorp Vault node.

Procedure

  1. Enable the Transit secret engine:

    [root@vault ~]# vault secrets enable transit
  2. Create a new exportable key:

    Syntax

    vault write -f transit/keys/BUCKET_NAME exportable=true

    Example

    [root@vault ~]# vault write -f transit/keys/mybucketkey exportable=true

    Note

    By default the above command creates a aes256-gcm96 type key.

  3. Verify the creation of the key:

    Syntax

    vault read transit/export/encryption-key/BUCKET_NAME/VERSION_NUMBER

    Example

    [root@vault ~]# vault read transit/export/encryption-key/mybucketkey/1
    
    Key     Value
    ---     -----
    keys    map[1:-gbTI9lNpqv/V/2lDcmH2Nq1xKn6FPDWarCmFM2aNsQ=]
    name    mybucketkey
    type    aes256-gcm96

    Note

    Providing the full key path, including the key version is required.

6.5.10. Uploading an object using AWS and the Vault

When uploading an object to the Ceph Object Gateway, the Ceph Object Gateway will fetch the key from the Vault, and then encrypt and store the object in a bucket. When a request is made to download the object, the Ceph Object Gateway will automatically retrieve the corresponding key from the Vault and decrypt the object. To upload an object, the Ceph object Gateway fetches the key from the Vault and then encrypts the object and stores it in the bucket. The Ceph Object Gateway retrieves the corresponding key from the Vault and decrypts the object when there is a request to download the object.

Note

The URL is constructed using the base address, set by the rgw_crypt_vault_addr option, and the path prefix, set by the rgw_crypt_vault_prefix option.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway software.
  • Installation of the HashiCorp Vault software.
  • Access to a Ceph Object Gateway client node.
  • Access to Amazon Web Services (AWS).

Procedure

  1. Upload an object using the AWS command-line client and provide the Secure Side Encryption(SSE) key ID in the request:

    1. For the Key/Value secret engine:

      Example

      [user@client ~]$ aws --endpoint=http://radosgw:8000 s3 cp plaintext.txt s3://mybucket/encrypted.txt --sse=aws:kms --sse-kms-key-id myproject/mybucketkey

      Note

      In the example the Ceph Object Gateway would fetch the secret from http://vault-server:8200/v1/secret/data/myproject/mybucketkey

    2. For the Transit engine:

      Example

      [user@client ~]$ aws --endpoint=http://radosgw:8000 s3 cp plaintext.txt s3://mybucket/encrypted.txt --sse=aws:kms --sse-kms-key-id mybucketkey

Note

In the example the Ceph Object Gateway would fetch the secret from http://vaultserver:8200/v1/transit/mybucketkey

6.5.11. Additional Resources

  • See the Install Vault documentation on Vault’s project site for more information.

6.6. The Ceph Object Gateway and multi-factor authentication

As a storage administrator, you can manage time-based one time password (TOTP) tokens for Ceph Object Gateway users.

6.6.1. Multi-factor authentication

When a bucket is configured for object versioning, a developer can optionally configure the bucket to require multi-factor authentication (MFA) for delete requests. Using MFA, a time-based one time password (TOTP) token is passed as a key to the x-amz-mfa header. The tokens are generated with virtual MFA devices like Google Authenticator, or a hardware MFA device like those provided by Gemalto.

Use radosgw-admin to assign time-based one time password tokens to a user. You must set a secret seed and a serial ID. You can also use radosgw-admin to list, remove, and resynchronize tokens.

Important

In a multisite environment it is advisable to use different tokens for different zones, because, while MFA IDs are set on the user’s metadata, the actual MFA one time password configuration resides on the local zone’s OSDs.

Table 6.1. Terminology

TermDescription

TOTP

Time-based One Time Password.

Token serial

A string that represents the ID of a TOTP token.

Token seed

The secret seed that is used to calculate the TOTP. It can be hexadecimal or base32.

TOTP seconds

The time resolution used for TOTP generation.

TOTP window

The number of TOTP tokens that are checked before and after the current token when validating tokens.

TOTP pin

The valid value of a TOTP token at a certain time.

6.6.2. Creating a seed for multi-factor authentication

To set up multi-factor authentication (MFA), you must create a secret seed for use by the one-time password generator and the back-end MFA system.

Prerequisites

  • A Linux system.
  • Access to the command line shell.

Procedure

  1. Generate a 30 character seed from the urandom Linux device file and store it in the shell variable SEED:

    Example

    [user@host ~]$ SEED=$(head -10 /dev/urandom | sha512sum | cut -b 1-30)

  2. Print the seed by running echo on the SEED variable:

    Example

    [user@host ~]$ echo $SEED
    492dedb20cf51d1405ef6a1316017e

    Configure the one-time password generator and the back-end MFA system to use the same seed.

Additional Resources

6.6.3. Creating a new multi-factor authentication TOTP token

Create a new multi-factor authentication (MFA) time-based one time password (TOTP) token.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Ceph Object Gateway is installed.
  • You have root access on a Ceph Monitor node.
  • A secret seed for the one-time password generator and Ceph Object Gateway MFA was generated.

Procedure

  1. Create a new MFA TOTP token:

    Syntax

    radosgw-admin mfa create --uid=USERID --totp-serial=SERIAL --totp-seed=SEED --totp-seed-type=SEED_TYPE --totp-seconds=TOTP_SECONDS --totp-window=TOTP_WINDOW

    Set USERID to the user name to set up MFA on, set SERIAL to a string that represents the ID for the TOTP token, and set SEED to a hexadecimal or base32 value that is used to calculate the TOTP. The following settings are optional: Set the SEED_TYPE to hex or base32, set TOTP_SECONDS to the timeout in seconds, or set TOTP_WINDOW to the number of TOTP tokens to check before and after the current token when validating tokens.

    Example

    [root@mon ~]# radosgw-admin mfa create --uid=johndoe --totp-serial=MFAtest --totp-seed=492dedb20cf51d1405ef6a1316017e

Additional Resources

6.6.4. Test a multi-factor authentication TOTP token

Test a multi-factor authentication (MFA) time-based one time password (TOTP) token.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Ceph Object Gateway is installed.
  • You have root access on a Ceph Monitor node.
  • An MFA TOTP token was created using radosgw-admin mfa create.

Procedure

  1. Test the TOTP token PIN to verify that TOTP functions correctly:

    Syntax

    radosgw-admin mfa check --uid=USERID --totp-serial=SERIAL --totp-pin=PIN

    Set USERID to the user name MFA is set up on, set SERIAL to the string that represents the ID for the TOTP token, and set PIN to the latest PIN from the one-time password generator.

    Example

    [root@mon ~] # radosgw-admin mfa check  --uid=johndoe --totp-serial=MFAtest --totp-pin=870305
    ok

    If this is the first time you have tested the PIN, it may fail. If it fails, resynchronize the token. See Resynchronizing a multi-factor authentication token in the Red Hat Ceph Storage Object Gateway Configuration and Administration Guide.

Additional Resources

6.6.5. Resynchronizing a multi-factor authentication TOTP token

Resynchronize a multi-factor authentication (MFA) time-based one time password token.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Ceph Object Gateway is installed.
  • You have root access on a Ceph Monitor node.
  • An MFA TOTP token was created using radosgw-admin mfa create.

Procedure

  1. Resynchronize a multi-factor authentication TOTP token in case of time skew or failed checks.

    This requires passing in two consecutive pins: the previous pin, and the current pin.

    Syntax

    radosgw-admin mfa resync --uid=USERID --totp-serial=SERIAL --totp-pin=PREVIOUS_PIN --totp=pin=CURRENT_PIN

    Set USERID to the user name MFA is set up on, set SERIAL to the string that represents the ID for the TOTP token, set PREVIOUS_PIN to the user’s previous PIN, and set CURRENT_PIN to the user’s current PIN.

    Example

    radosgw-admin mfa resync --uid=johndoe --totp-serial=MFAtest --totp-pin=802017 --totp-pin=439996

  2. Verify the token was successfully resynchronized by testing a new PIN:

    Syntax

    radosgw-admin mfa check --uid=USERID --totp-serial=SERIAL --totp-pin=PIN

    Set USERID to the user name MFA is set up on, set SERIAL to the string that represents the ID for the TOTP token, and set PIN to the user’s PIN.

    Example

    [root@mon ~]# radosgw-admin mfa check  --uid=johndoe --totp-serial=MFAtest --totp-pin=870305
    ok

Additional Resources

6.6.6. Listing multi-factor authentication TOTP tokens

List all multi-factor authentication (MFA) time-based one time password (TOTP) tokens that a particular user has.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Ceph Object Gateway is installed.
  • You have root access on a Ceph Monitor node.
  • An MFA TOTP token was created using radosgw-admin mfa create.

Procedure

  1. List MFA TOTP tokens:

    Syntax

    radosgw-admin mfa list --uid=USERID

    Set USERID to the user name MFA is set up on.

    Example

    [root@mon ~]# radosgw-admin mfa list --uid=johndoe
    {
        "entries": [
            {
                "type": 2,
                "id": "MFAtest",
                "seed": "492dedb20cf51d1405ef6a1316017e",
                "seed_type": "hex",
                "time_ofs": 0,
                "step_size": 30,
                "window": 2
            }
        ]
    }

Additional Resources

6.6.7. Display a multi-factor authentication TOTP token

Display a specific multi-factor authentication (MFA) time-based one time password (TOTP) token by specifying its serial.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Ceph Object Gateway is installed.
  • You have root access on a Ceph Monitor node.
  • An MFA TOTP token was created using radosgw-admin mfa create.

Procedure

  1. Show the MFA TOTP token:

    Syntax

    radosgw-admin mfa get --uid=USERID --totp-serial=SERIAL

    Set USERID to the user name MFA is set up on and set SERIAL to the string that represents the ID for the TOTP token.

Additional Resources

6.6.8. Deleting a multi-factor authentication TOTP token

Delete a multi-factor authentication (MFA) time-based one time password (TOTP) token.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Ceph Object Gateway is installed.
  • You have root access on a Ceph Monitor node.
  • An MFA TOTP token was created using radosgw-admin mfa create.

Procedure

  1. Delete an MFA TOTP token:

    Syntax

    radosgw-admin mfa remove --uid=USERID --totp-serial=SERIAL

    Set USERID to the user name MFA is set up on and set SERIAL to the string that represents the ID for the TOTP token.

    Example

    [root@mon ~]# radosgw-admin mfa remove --uid=johndoe --totp-serial=MFAtest

  2. Verify the MFA TOTP token was deleted:

    Syntax

    radosgw-admin mfa get --uid=_USERID_ --totp-serial=_SERIAL_

    Set USERID to the user name MFA is set up on and set SERIAL to the string that represents the ID for the TOTP token.

    Example

    [root@mon ~]# radosgw-admin mfa get --uid=johndoe --totp-serial=MFAtest
    MFA serial id not found

Additional Resources

Chapter 7. Administration

As a storage administrator, you can manage the Ceph Object Gateway using the radosgw-admin command-line interface (CLI) or using the Red Hat Ceph Storage Dashboard.

Note

Not all of Ceph Object Gateway features are available to the Red Hat Ceph Storage Dashboard.

7.1. Prerequisites

  • A healthy running Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway software.

7.2. Administrative Data Storage

A Ceph Object Gateway stores administrative data in a series of pools defined in an instance’s zone configuration. For example, the buckets, users, user quotas and usage statistics discussed in the subsequent sections are stored in pools in the Ceph Storage Cluster. By default, Ceph Object Gateway will create the following pools and map them to the default zone.

  • .rgw.root
  • .default.rgw.control
  • .default.rgw.meta
  • .default.rgw.log
  • .default.rgw.buckets.index
  • .default.rgw.buckets.data
  • .default.rgw.buckets.non-ec

You should consider creating these pools manually so that you can set the CRUSH ruleset and the number of placement groups. In a typical configuration, the pools that store the Ceph Object Gateway’s administrative data will often use the same CRUSH ruleset and use fewer placement groups, because there are 10 pools for the administrative data.

The mon_pg_warn_max_per_osd setting warns you if assign too many placement groups to a pool (i.e., 300 by default). You may adjust the value to suit your needs and the capabilities of your hardware where n is the maximum number of PGs per OSD.

mon_pg_warn_max_per_osd = n

Additional Resources

7.3. Creating Storage Policies

The Ceph Object Gateway stores the client bucket and object data by identifying placement targets, and storing buckets and objects in the pools associated with a placement target. If you don’t configure placement targets and map them to pools in the instance’s zone configuration, the Ceph Object Gateway will use default targets and pools, for example, default_placement.

Storage policies give Ceph Object Gateway clients a way of accessing a storage strategy, that is, the ability to target a particular type of storage, for example, SSDs, SAS drives, SATA drives. A particular way of ensuring durability, replication, erasure coding, and so on. For details, see the Storage Strategies guide for Red Hat Ceph Storage 5.

To create a storage policy, use the following procedure:

  1. Create a new pool .rgw.buckets.special with the desired storage strategy. For example, a pool customized with erasure-coding, a particular CRUSH ruleset, the number of replicas, and the pg_num and pgp_num count.
  2. Get the zone group configuration and store it in a file, for example, zonegroup.json:

    Syntax

    [root@master-zone]# radosgw-admin zonegroup --rgw-zonegroup=<zonegroup_name> get > zonegroup.json

    Example

    [root@master-zone]# radosgw-admin zonegroup --rgw-zonegroup=default get > zonegroup.json

  3. Add a special-placement entry under placement_target in the zonegroup.json file.

    {
    	"name": "default",
    	"api_name": "",
    	"is_master": "true",
    	"endpoints": [],
    	"hostnames": [],
    	"master_zone": "",
    	"zones": [{
    		"name": "default",
    		"endpoints": [],
    		"log_meta": "false",
    		"log_data": "false",
    		"bucket_index_max_shards": 5
    	}],
    	"placement_targets": [{
    		"name": "default-placement",
    		"tags": []
    	}, {
    		"name": "special-placement",
    		"tags": []
    	}],
    	"default_placement": "default-placement"
    }
  4. Set the zone group with the modified zonegroup.json file:

    [root@master-zone]# radosgw-admin zonegroup set < zonegroup.json
  5. Get the zone configuration and store it in a file, for example, zone.json:

    [root@master-zone]# radosgw-admin zone get > zone.json
  6. Edit the zone file and add the new placement policy key under placement_pool:

    {
    	"domain_root": ".rgw",
    	"control_pool": ".rgw.control",
    	"gc_pool": ".rgw.gc",
    	"log_pool": ".log",
    	"intent_log_pool": ".intent-log",
    	"usage_log_pool": ".usage",
    	"user_keys_pool": ".users",
    	"user_email_pool": ".users.email",
    	"user_swift_pool": ".users.swift",
    	"user_uid_pool": ".users.uid",
    	"system_key": {
    		"access_key": "",
    		"secret_key": ""
    	},
    	"placement_pools": [{
    		"key": "default-placement",
    		"val": {
    			"index_pool": ".rgw.buckets.index",
    			"data_pool": ".rgw.buckets",
    			"data_extra_pool": ".rgw.buckets.extra"
    		}
    	}, {
    		"key": "special-placement",
    		"val": {
    			"index_pool": ".rgw.buckets.index",
    			"data_pool": ".rgw.buckets.special",
    			"data_extra_pool": ".rgw.buckets.extra"
    		}
    	}]
    }
  7. Set the new zone configuration.

    [root@master-zone]# radosgw-admin zone set < zone.json
  8. Update the zone group map.

    [root@master-zone]# radosgw-admin period update --commit

    The special-placement entry is listed as a placement_target.

To specify the storage policy when making a request:

Example:

$ curl -i http://10.0.0.1/swift/v1/TestContainer/file.txt -X PUT -H "X-Storage-Policy: special-placement" -H "X-Auth-Token: AUTH_rgwtxxxxxx"

7.4. Creating Indexless Buckets

It is possible to configure a placement target where created buckets do not use the bucket index to store objects index; that is, indexless buckets. Placement targets that do not use data replication or listing may implement indexless buckets.

Indexless buckets provides a mechanism in which the placement target does not track objects in specific buckets. This removes a resource contention that happens whenever an object write happens and reduces the number of round trips that Ceph Object Gateway needs to make to the Ceph Storage cluster. This can have a positive effect on concurrent operations and small object write performance.

To specify a placement target as indexless, use the following procedure:

  1. Get the configuration for zone.json:

    $ radosgw-admin zone get --rgw-zone=<zone> > zone.json
  2. Modify zone.json by adding a new placement target or by modifying an existing one to have "index_type": 1, for example:

    "placement_pools": [
        {
          "key": "default-placement",
          "val": {
            "index_pool": "default.rgw.buckets.index",
            "data_pool": "default.rgw.buckets.data",
            "data_extra_pool": "default.rgw.buckets.non-ec",
            "index_type": 1,
            "compression": ""
          }
        },
        {
          "key": "indexless",
          "val": {
            "index_pool": "default.rgw.buckets.index",
            "data_pool": "default.rgw.buckets.data",
            "data_extra_pool": "default.rgw.buckets.non-ec",
            "index_type": 1
          }
        }
      ],
  3. Set the configuration for zone.json:

    $ radosgw-admin zone set --rgw-zone=<zone> --infile zone.json
  4. Make sure the zonegroup refers to the new placement target if you created a new placement target:

    $ radosgw-admin zonegroup get --rgw-zonegroup=<zonegroup> > zonegroup.json
  5. Set the zonegroup’s default_placement:

    $ radosgw-admin zonegroup placement default --placement-id indexless
  6. Modify the zonegroup.json as needed. For example:

      "placement_targets": [
        {
          "name": "default-placement",
          "tags": []
        },
        {    "name": "indexless",
    		     "tags": []
        }
      ],
      "default_placement": "default-placement",
    $ radosgw-admin zonegroup set --rgw-zonegroup=<zonegroup> < zonegroup.json
  7. Update and commit the period if the cluster is in a multi-site configuration:

    $ radosgw-admin period update --commit

In this example, the buckets created in the "indexless" target will be indexless buckets.

Important

The bucket index will not reflect the correct state of the bucket, and listing these buckets will not correctly return their list of objects. This affects multiple features. Specifically, these buckets will not be synced in a multi-zone environment because the bucket index is not used to store change information. It is not recommended to use S3 object versioning on indexless buckets because the bucket index is necessary for this feature.

Note

Using indexless buckets removes the limit of the max number of objects in a single bucket.

Note

Objects in indexless buckets cannot be listed from NFS

7.5. Configuring Bucket Sharding

The Ceph Object Gateway stores bucket index data in the index pool (index_pool), which defaults to .rgw.buckets.index. When the client puts many objects—​hundreds of thousands to millions of objects—​in a single bucket without having set quotas for the maximum number of objects per bucket, the index pool can suffer significant performance degradation.

Bucket index sharding helps prevent performance bottlenecks when allowing a high number of objects per bucket.

You can configure bucket index sharding for new buckets or change the bucket index on already existing ones.

Red Hat recommends to have the shard count as the nearest prime number to the calculated shard count. The bucket index shards that are prime numbers tend to work better in evenly distributing bucket index entries across the shards. For example, 7001 bucket index shards is better than 7000 since the former is prime.

To configure bucket index sharding:

To reshard a bucket:

7.5.1. Bucket Sharding Limitations

Important

Use the following limitations with caution. There are implications related to your hardware selections, so you should always discuss these requirements with your Red Hat account team.

  • Maximum number of objects in one bucket before it needs sharding: Red Hat Recommends a maximum of 102,400 objects per bucket index shard. To take full advantage of sharding, provide a sufficient number of OSDs in the Ceph Object Gateway bucket index pool to get maximum parallelism.
  • Maximum number of objects when using sharding: Based on prior testing, the number of bucket index shards currently supported is 65521. Red Hat quality assurance has NOT performed full scalability testing on bucket sharding.

7.5.2. Bucket lifecycle parallel thread processing

A new feature in Red Hat Ceph Storage 4.1 allows for parallel thread processing of bucket lifecycles. This parallelization scales with the number of Ceph Object Gateway instance, and replaces the in-order index shard enumeration with a number sequence. The default locking timeout has been extended from 60 seconds to 90 seconds. New tunable options have been added to tune lifecycle worker threads to run in parallel for each Ceph Object Gateway instance.

rgw_lc_max_worker

This option specifies the number of lifecycle worker thread to run in parallel, thereby processing bucket and index shards simultaneously. The default value for the rgw_lc_max_worker option is 3.

rgw_lc_max_wp_worker

This option specifies the number of threads in each lifecycle worker’s work pool. This option can help accelerate processing each bucket. The default value for the rgw_lc_max_wp_worker option is 3.

Additional Resources

7.5.3. Configuring Bucket Index Sharding in Simple Configurations

To enable and configure bucket index sharding on all new buckets, use the rgw_override_bucket_index_max_shards parameter. Set the parameter to:

  • 0 to disable bucket index sharding. This is the default value.
  • A value greater than 0 to enable bucket sharding and to set the maximum number of shards.

Prerequisites

Procedure

  1. Calculate the recommended number of shards. To do so, use the following formula:

    number of objects expected in a bucket / 100,000

    Note that maximum number of shards is 65521.

  2. Set the rgw_override_bucket_index_max_shards option accordingly:

    Syntax

    ceph config set client.rgw rgw_override_bucket_index_max_shards VALUE

    Replace VALUE with the recommended number of shards calculated in the previous step:

    Example

    [root@mon ~]# ceph config set client.rgw rgw_override_bucket_index_max_shards 10

    • To configure bucket index sharding for all instances of the Ceph Object Gateway, add rgw_override_bucket_index_max_shards under the [global] section.
    • To configure bucket index sharding only for a particular instance of the Ceph Object Gateway, add rgw_override_bucket_index_max_shards under the instance.
  3. Restart the Ceph Object Gateway:

    # systemctl restart ceph-radosgw.target

7.5.4. Configuring Bucket Index Sharding in Multi-site Configurations

In multi-site configurations, each zone can have a different index_pool setting to manage failover. To configure a consistent shard count for zones in one zone group, set the bucket_index_max_shards setting in the configuration for that zone group. Set the parameter to:

  • 0 to disable bucket index sharding. This is the default value.
  • A value greater than 0 to enable bucket sharding and to set the maximum number of shards.
Note

Mapping the index pool (for each zone, if applicable) to a CRUSH ruleset of SSD-based OSDs might also help with bucket index performance.

Prerequisites

Procedure

  1. Calculate the recommended number of shards. To do so, use the following formula:

    number of objects expected in a bucket / 100,000

    Note that maximum number of shards is 65521.

  2. Extract the zone group configuration to the zonegroup.json file:

    $ radosgw-admin zonegroup get > zonegroup.json
  3. In the zonegroup.json file, set the bucket_index_max_shards setting for each named zone:

    bucket_index_max_shards = VALUE

    Replace VALUE with the recommended number of shards calculated in the previous step, for example:

    bucket_index_max_shards = 10
  4. Reset the zone group:

    $ radosgw-admin zonegroup set < zonegroup.json
  5. Update the period:

    $ radosgw-admin period update --commit

7.5.5. Dynamic Bucket Index Resharding

The process for dynamic bucket resharding periodically checks all the Ceph Object Gateway buckets and detects buckets that require resharding. If a bucket has grown larger than the value specified in the rgw_max_objs_per_shard parameter, the Ceph Object Gateway reshards the bucket dynamically in the background. The default value for rgw_max_objs_per_shard is 100k objects per shard.

Important

Currently, Red Hat does not support dynamic bucket resharding in multi-site configurations. To reshard bucket index in such configuration, see Manually Resharding Buckets with Multi-site.

Prerequisites

Procedure

  1. Set the rgw_dynamic_resharding option to true, which is the default value.
  2. Optional settings to consider. Using the ceph config set client.rgw OPTION VALUE, change the following options:

    • rgw_reshard_num_logs: The number of shards for the resharding log. The default value is 16.
    • rgw_reshard_bucket_lock_duration: The duration of the lock on a bucket during resharding. The default value is 120 seconds.
    • rgw_dynamic_resharding: Enables or disables dynamic resharding. The default value is true.
    • rgw_max_objs_per_shard: The maximum number of objects per shard. The default value is 100000 objects per shard.
    • rgw_reshard_thread_interval: The maximum time between rounds of reshard thread processing. The default value is 600 seconds.
  3. To add a bucket to the resharding queue:

    radosgw-admin reshard add --bucket bucket --num-shards number

    Replace:

    • bucket with the name of the bucket to reshard
    • number with the new number of shards

    For example:

    $ radosgw-admin reshard add --bucket data --num-shards 10
  4. To list the resharding queue:

    $ radosgw-admin reshard list
  5. To check bucket resharding status:

    radosgw-admin reshard status --bucket bucket

    Replace:

    • bucket with the name of the bucket to reshard

    Example

    $ radosgw-admin reshard status --bucket data

    • To process entries on the resharding queue immediately:

      $ radosgw-admin reshard process
    • To cancel pending bucket resharding:

      radosgw-admin reshard cancel --bucket bucket

      Replace:

      • bucket with the name of the pending bucket

      For example:

      $ radosgw-admin reshard cancel --bucket data
      Important

      You can only cancel pending resharding operations. Do not cancel ongoing resharding operations.

    • If you use Red Hat Ceph Storage 3.1 and previous versions, remove stale bucket entries as described in the Cleaning stale instances after resharding section.

7.5.6. Manual Bucket Index Resharding

If a bucket has grown larger than the initial configuration was optimized for, reshard the bucket index pool by using the radosgw-admin bucket reshard command. This command:

  • Creates a new set of bucket index objects for the specified bucket.
  • Distributes object entries across these bucket index objects.
  • Creates a new bucket instance.
  • Links the new bucket instance with the bucket so that all new index operations go through the new bucket indexes.
  • Prints the old and the new bucket ID to the command output.
Important

Use this procedure only in simple configurations. To reshard buckets in multi-site configurations, see Manually Resharding Buckets with Multi-site.

Prerequisites

Procedure

  1. Back the original bucket index up:

    radosgw-admin bi list --bucket=bucket > bucket.list.backup

    Replace:

    • bucket with the name of the bucket to reshard

    For example, for a bucket named data, enter:

    $ radosgw-admin bi list --bucket=data > data.list.backup
  2. Reshard the bucket index:

    radosgw-admin bucket reshard --bucket=bucket
    --num-shards=number

    Replace:

    • bucket with the name of the bucket to reshard
    • number with the new number of shards

    For example, for a bucket named data and the required number of shards being 100, enter:

    $ radosgw-admin bucket reshard --bucket=data
    --num-shards=100
  3. If you use Red Hat Ceph Storage 3.1 and previous versions, remove stale bucket entries as described in the Cleaning stale instances after resharding section.

7.5.7. Cleaning stale instances after resharding

In Red Hat Ceph Storage 3.1 and previous versions, the resharding process does not clean stale instances of bucket entries automatically. These stale instances can impact performance of the cluster if they are not cleaned manually.

Important

Use this procedure only in simple configurations not in multi-site clusters.

Prerequisites

  • Ceph Object Gateway installed.

Procedure

  1. List stale instances:

    $ radosgw-admin reshard stale-instances list
  2. Clean the stale instances:

    $ radosgw-admin reshard stale-instances rm

7.6. Enabling Compression

The Ceph Object Gateway supports server-side compression of uploaded objects using any of Ceph’s compression plugins. These include:

  • zlib: Supported.
  • snappy: Technology Preview.
  • zstd: Technology Preview.
Note

The snappy and zstd compression plugins are Technology Preview features and as such they are not fully supported, as Red Hat has not completed quality assurance testing on them yet.

Configuration

To enable compression on a zone’s placement target, provide the --compression=<type> option to the radosgw-admin zone placement modify command. The compression type refers to the name of the compression plugin to use when writing new object data.

Each compressed object stores the compression type. Changing the setting does not hinder the ability to decompress existing compressed objects, nor does it force the Ceph Object Gateway to recompress existing objects.

This compression setting applies to all new objects uploaded to buckets using this placement target.

To disable compression on a zone’s placement target, provide the --compression=<type> option to the radosgw-admin zone placement modify command and specify an empty string or none.

For example:

$ radosgw-admin zone placement modify --rgw-zone=default --placement-id=default-placement --compression=zlib
{
...
    "placement_pools": [
        {
            "key": "default-placement",
            "val": {
                "index_pool": "default.rgw.buckets.index",
                "data_pool": "default.rgw.buckets.data",
                "data_extra_pool": "default.rgw.buckets.non-ec",
                "index_type": 0,
                "compression": "zlib"
            }
        }
    ],
...
}

After enabling or disabling compression, restart the Ceph Object Gateway instance so the change will take effect.

Note

Ceph Object Gateway creates a default zone and a set of pools. For production deployments, see the Creating a Realm section first.

Statistics

While all existing commands and APIs continue to report object and bucket sizes based on their uncompressed data, the radosgw-admin bucket stats command includes compression statistics for all buckets.

$ radosgw-admin bucket stats --bucket=<name>
{
...
    "usage": {
        "rgw.main": {
            "size": 1075028,
            "size_actual": 1331200,
            "size_utilized": 592035,
            "size_kb": 1050,
            "size_kb_actual": 1300,
            "size_kb_utilized": 579,
            "num_objects": 104
        }
    },
...
}

The size_utilized and size_kb_utilized fields represent the total size of compressed data in bytes and kilobytes, respectively.

7.7. User Management

Ceph Object Storage user management refers to users that are client applications of the Ceph Object Storage service; not the Ceph Object Gateway as a client application of the Ceph Storage Cluster. You must create a user, access key and secret to enable client applications to interact with the Ceph Object Gateway service.

There are two user types:

  • User: The term 'user' reflects a user of the S3 interface.
  • Subuser: The term 'subuser' reflects a user of the Swift interface. A subuser is associated to a user .

You can create, modify, view, suspend and remove users and subusers.

Important

When managing users in a multi-site deployment, ALWAYS execute the radosgw-admin command on a Ceph Object Gateway node within the master zone of the master zone group to ensure that users synchronize throughout the multi-site cluster. DO NOT create, modify or delete users on a multi-site cluster from a secondary zone or a secondary zone group. This document uses [root@master-zone]# as a command line convention for a host in the master zone of the master zone group.

In addition to creating user and subuser IDs, you may add a display name and an email address for a user. You can specify a key and secret, or generate a key and secret automatically. When generating or specifying keys, note that user IDs correspond to an S3 key type and subuser IDs correspond to a swift key type. Swift keys also have access levels of read, write, readwrite and full.

User management command-line syntax generally follows the pattern user <command> <user-id> where <user-id> is either the --uid= option followed by the user’s ID (S3) or the --subuser= option followed by the user name (Swift). For example:

[root@master-zone]# radosgw-admin user <create|modify|info|rm|suspend|enable|check|stats> <--uid={id}|--subuser={name}> [other-options]

Additional options may be required depending on the command you execute.

7.7.1. Multi Tenancy

In Red Hat Ceph Storage 2 and later, the Ceph Object Gateway supports multi-tenancy for both the S3 and Swift APIs, where each user and bucket lies under a "tenant." Multi tenancy prevents namespace clashing when multiple tenants are using common bucket names, such as "test", "main" and so forth.

Each user and bucket lies under a tenant. For backward compatibility, a "legacy" tenant with an empty name is added. Whenever referring to a bucket without specifically specifying a tenant, the Swift API will assume the "legacy" tenant. Existing users are also stored under the legacy tenant, so they will access buckets and objects the same way as earlier releases.

Tenants as such do not have any operations on them. They appear and and disappear as needed, when users are administered. In order to create, modify, and remove users with explicit tenants, either an additional option --tenant is supplied, or a syntax "<tenant>$<user>" is used in the parameters of the radosgw-admin command.

To create a user testx$tester for S3, execute the following:

[root@master-zone]# radosgw-admin --tenant testx --uid tester \
                    --display-name "Test User" --access_key TESTER \
                    --secret test123 user create

To create a user testx$tester for Swift, execute one of the following:

[root@master-zone]# radosgw-admin --tenant testx --uid tester \
                    --display-name "Test User" --subuser tester:swift \
                    --key-type swift --access full subuser create

[root@master-zone]# radosgw-admin key create --subuser 'testx$tester:swift' \
                    --key-type swift --secret test123
Note

The subuser with explicit tenant had to be quoted in the shell.

7.7.2. Create a User

Use the user create command to create an S3-interface user. You MUST specify a user ID and a display name. You may also specify an email address. If you DO NOT specify a key or secret, radosgw-admin will generate them for you automatically. However, you may specify a key and/or a secret if you prefer not to use generated key/secret pairs.

[root@master-zone]# radosgw-admin user create --uid=<id> \
[--key-type=<type>] [--gen-access-key|--access-key=<key>]\
[--gen-secret | --secret=<key>] \
[--email=<email>] --display-name=<name>

For example:

[root@master-zone]# radosgw-admin user create --uid=janedoe --display-name="Jane Doe" --email=jane@example.com
{ "user_id": "janedoe",
  "display_name": "Jane Doe",
  "email": "jane@example.com",
  "suspended": 0,
  "max_buckets": 1000,
  "auid": 0,
  "subusers": [],
  "keys": [
        { "user": "janedoe",
          "access_key": "11BS02LGFB6AL6H1ADMW",
          "secret_key": "vzCEkuryfn060dfee4fgQPqFrncKEIkh3ZcdOANY"}],
  "swift_keys": [],
  "caps": [],
  "op_mask": "read, write, delete",
  "default_placement": "",
  "placement_tags": [],
  "bucket_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "user_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "temp_url_keys": []}
Important

Check the key output. Sometimes radosgw-admin generates a JSON escape (\) character, and some clients do not know how to handle JSON escape characters. Remedies include removing the JSON escape character (\), encapsulating the string in quotes, regenerating the key and ensuring that it does not have a JSON escape character or specify the key and secret manually.

7.7.3. Create a Subuser

To create a subuser (Swift interface), you must specify the user ID (--uid={username}), a subuser ID and the access level for the subuser. If you DO NOT specify a key or secret, radosgw-admin will generate them for you automatically. However, you may specify a key and/or a secret if you prefer not to use generated key/secret pairs.

Note

full is not readwrite, as it also includes the access control policy.

[root@master-zone]# radosgw-admin subuser create --uid={uid} --subuser={uid} --access=[ read | write | readwrite | full ]

For example:

[root@master-zone]# radosgw-admin subuser create --uid=janedoe --subuser=janedoe:swift --access=full
{ "user_id": "janedoe",
  "display_name": "Jane Doe",
  "email": "jane@example.com",
  "suspended": 0,
  "max_buckets": 1000,
  "auid": 0,
  "subusers": [
        { "id": "janedoe:swift",
          "permissions": "full-control"}],
  "keys": [
        { "user": "janedoe",
          "access_key": "11BS02LGFB6AL6H1ADMW",
          "secret_key": "vzCEkuryfn060dfee4fgQPqFrncKEIkh3ZcdOANY"}],
  "swift_keys": [],
  "caps": [],
  "op_mask": "read, write, delete",
  "default_placement": "",
  "placement_tags": [],
  "bucket_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "user_quota": { "enabled": false,
      "max_size_kb": -1,
      "max_objects": -1},
  "temp_url_keys": []}

7.7.4. Get User Information

To get information about a user, specify user info and the user ID (--uid={username}).

[root@master-zone]# radosgw-admin user info --uid=janedoe

To get information about a tenanted user, specify both the user ID and the name of the tenant.

[root@master-zone]# radosgw-admin user info --uid=janedoe --tenant=test

7.7.5. Modify User Information

To modify information about a user, you must specify the user ID (--uid={username}) and the attributes you want to modify. Typical modifications are to keys and secrets, email addresses, display names and access levels. For example:

[root@master-zone]# radosgw-admin user modify --uid=janedoe / --display-name="Jane E. Doe"

To modify subuser values, specify subuser modify and the subuser ID. For example:

[root@master-zone]# radosgw-admin subuser modify --subuser=janedoe:swift / --access=full

7.7.6. Enable and Suspend Users

When you create a user, the user is enabled by default. However, you may suspend user privileges and re-enable them at a later time. To suspend a user, specify user suspend and the user ID.

[root@master-zone]# radosgw-admin user suspend --uid=johndoe

To re-enable a suspended user, specify user enable and the user ID. :

[root@master-zone]# radosgw-admin user enable --uid=johndoe
Note

Disabling the user disables the subuser.

7.7.7. Remove a User

When you remove a user, the user and subuser are removed from the system. However, you may remove just the subuser if you wish. To remove a user (and subuser), specify user rm and the user ID.

[root@master-zone]# radosgw-admin user rm --uid=<uid> [--purge-keys] [--purge-data]

For example:

[root@master-zone]# radosgw-admin user rm --uid=johndoe --purge-data

To remove the subuser only, specify subuser rm and the subuser name.

[root@master-zone]# radosgw-admin subuser rm --subuser=johndoe:swift --purge-keys

Options include:

  • Purge Data: The --purge-data option purges all data associated to the UID.
  • Purge Keys: The --purge-keys option purges all keys associated to the UID.

7.7.8. Remove a Subuser

When you remove a sub user, you are removing access to the Swift interface. The user will remain in the system. The Ceph Object Gateway To remove the subuser, specify subuser rm and the subuser ID.

[root@master-zone]# radosgw-admin subuser rm --subuser=johndoe:test

Options include:

  • Purge Keys: The --purge-keys option purges all keys associated to the UID.

7.7.9. Rename a User

To change a name of a user, use the radosgw-admin user rename command. The time that this command takes depends on the number of buckets and objects that the user has. If the number is large, Red Hat recommends to use the command in the Screen utility provided by the screen package.

Prerequisites

  • A working Ceph cluster
  • root or sudo access
  • Installed Ceph Object Gateway

Procedure

  1. Rename a user:

    radosgw-admin user rename --uid=current-user-name --new-uid=new-user-name

    For example, to rename user1 to user2:

    # radosgw-admin user rename --uid=user1 --new-uid=user2
    
    {
        "user_id": "user2",
        "display_name": "user 2",
        "email": "",
        "suspended": 0,
        "max_buckets": 1000,
        "auid": 0,
        "subusers": [],
        "keys": [
            {
                "user": "user2",
                "access_key": "59EKHI6AI9F8WOW8JQZJ",
                "secret_key": "XH0uY3rKCUcuL73X0ftjXbZqUbk0cavD11rD8MsA"
            }
        ],
        "swift_keys": [],
        "caps": [],
        "op_mask": "read, write, delete",
        "default_placement": "",
        "placement_tags": [],
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "temp_url_keys": [],
        "type": "rgw"
    }

    If a user is inside a tenant, specify both the user name and the tenant:

    Syntax

    radosgw-admin user rename --uid user-name --new-uid new-user-name --tenant tenant

    For example, to rename user1 to user2 inside a test tenant:

    Example

    # radosgw-admin user rename --uid=test$user1 --new-uid=test$user2 --tenant test
    
    1000 objects processed in tvtester1. Next marker 80_tVtester1_99
    2000 objects processed in tvtester1. Next marker 64_tVtester1_44
    3000 objects processed in tvtester1. Next marker 48_tVtester1_28
    4000 objects processed in tvtester1. Next marker 2_tVtester1_74
    5000 objects processed in tvtester1. Next marker 14_tVtester1_53
    6000 objects processed in tvtester1. Next marker 87_tVtester1_61
    7000 objects processed in tvtester1. Next marker 6_tVtester1_57
    8000 objects processed in tvtester1. Next marker 52_tVtester1_91
    9000 objects processed in tvtester1. Next marker 34_tVtester1_74
    9900 objects processed in tvtester1. Next marker 9_tVtester1_95
    1000 objects processed in tvtester2. Next marker 82_tVtester2_93
    2000 objects processed in tvtester2. Next marker 64_tVtester2_9
    3000 objects processed in tvtester2. Next marker 48_tVtester2_22
    4000 objects processed in tvtester2. Next marker 32_tVtester2_42
    5000 objects processed in tvtester2. Next marker 16_tVtester2_36
    6000 objects processed in tvtester2. Next marker 89_tVtester2_46
    7000 objects processed in tvtester2. Next marker 70_tVtester2_78
    8000 objects processed in tvtester2. Next marker 51_tVtester2_41
    9000 objects processed in tvtester2. Next marker 33_tVtester2_32
    9900 objects processed in tvtester2. Next marker 9_tVtester2_83
    {
        "user_id": "test$user2",
        "display_name": "User 2",
        "email": "",
        "suspended": 0,
        "max_buckets": 1000,
        "auid": 0,
        "subusers": [],
        "keys": [
            {
                "user": "test$user2",
                "access_key": "user2",
                "secret_key": "123456789"
            }
        ],
        "swift_keys": [],
        "caps": [],
        "op_mask": "read, write, delete",
        "default_placement": "",
        "placement_tags": [],
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "temp_url_keys": [],
        "type": "rgw"
    }

  2. Verify that the user has been renamed successfully:

    Syntax

    radosgw-admin user info --uid=new-user-name

    For example:

    Example

    # radosgw-admin user info --uid=user2

    If a user is inside a tenant, use the tenant$user-name format:

    radosgw-admin user info --uid=tenant$new-user-name
    # radosgw-admin user info --uid=test$user2

Additional Resources

  • The screen(1) manual page

7.7.10. Create a Key

To create a key for a user, you must specify key create. For a user, specify the user ID and the s3 key type. To create a key for subuser, you must specify the subuser ID and the swift keytype. For example:

[root@master-zone]# radosgw-admin key create --subuser=johndoe:swift --key-type=swift --gen-secret
{ "user_id": "johndoe",
  "rados_uid": 0,
  "display_name": "John Doe",
  "email": "john@example.com",
  "suspended": 0,
  "subusers": [
     { "id": "johndoe:swift",
       "permissions": "full-control"}],
  "keys": [
    { "user": "johndoe",
      "access_key": "QFAMEDSJP5DEKJO0DDXY",
      "secret_key": "iaSFLDVvDdQt6lkNzHyW4fPLZugBAI1g17LO0+87"}],
  "swift_keys": [
    { "user": "johndoe:swift",
      "secret_key": "E9T2rUZNu2gxUjcwUBO8n\/Ev4KX6\/GprEuH4qhu1"}]}

7.7.11. Add and Remove Access Keys

Users and subusers must have access keys to use the S3 and Swift interfaces. When you create a user or subuser and you do not specify an access key and secret, the key and secret get generated automatically. You may create a key and either specify or generate the access key and/or secret. You may also remove an access key and secret. Options include:

  • --secret=<key> specifies a secret key (e.g,. manually generated).
  • --gen-access-key generates random access key (for S3 user by default).
  • --gen-secret generates a random secret key.
  • --key-type=<type> specifies a key type. The options are: swift, s3

To add a key, specify the user:

[root@master-zone]# radosgw-admin key create --uid=johndoe --key-type=s3 --gen-access-key --gen-secret

You may also specify a key and a secret.

To remove an access key, you need to specify the user and the key:

  1. Find the access key for the specific user:

    [root@master-zone]# radosgw-admin user info --uid=<testid>

    The access key is the "access_key" value in the output, for example:

    $ radosgw-admin user info --uid=johndoe
    {
        "user_id": "johndoe",
        ...
        "keys": [
            {
                "user": "johndoe",
                "access_key": "0555b35654ad1656d804",
                "secret_key": "h7GhxuBLTrlhVUyxSPUKUV8r/2EI4ngqJxD7iBdBYLhwluN30JaT3Q=="
            }
        ],
        ...
    }
  2. Specify the user ID and the access key from the previous step to remove the access key:

    [root@master-zone]# radosgw-admin key rm --uid=<user_id> --access-key <access_key>

    For example:

    [root@master-zone]# radosgw-admin key rm --uid=johndoe --access-key 0555b35654ad1656d804

7.7.12. Add and Remove Admin Capabilities

The Ceph Storage Cluster provides an administrative API that enables users to execute administrative functions via the REST API. By default, users DO NOT have access to this API. To enable a user to exercise administrative functionality, provide the user with administrative capabilities.

To add administrative capabilities to a user, execute the following:

[root@master-zone]# radosgw-admin caps add --uid={uid} --caps={caps}

You can add read, write or all capabilities to users, buckets, metadata and usage (utilization). For example:

--caps="[users|buckets|metadata|usage|zone]=[*|read|write|read, write]"

For example:

[root@master-zone]# radosgw-admin caps add --uid=johndoe --caps="users=*"

To remove administrative capabilities from a user, execute the following:

[root@master-zone]# radosgw-admin caps remove --uid=johndoe --caps={caps}

7.8. Quota Management

The Ceph Object Gateway enables you to set quotas on users and buckets owned by users. Quotas include the maximum number of objects in a bucket and the maximum storage size in megabytes.

  • Bucket: The --bucket option allows you to specify a quota for buckets the user owns.
  • Maximum Objects: The --max-objects setting allows you to specify the maximum number of objects. A negative value disables this setting.
  • Maximum Size: The --max-size option allows you to specify a quota for the maximum number of bytes. A negative value disables this setting.
  • Quota Scope: The --quota-scope option sets the scope for the quota. The options are bucket and user. Bucket quotas apply to buckets a user owns. User quotas apply to a user.
Important

Buckets with a large number of objects can cause serious performance issues. The recommended maximum number of objects in a one bucket is 100,000. To increase this number, configure bucket index sharding. See Section 7.5, “Configuring Bucket Sharding” for details.

7.8.1. Set User Quotas

Before you enable a quota, you must first set the quota parameters. For example:

[root@master-zone]# radosgw-admin quota set --quota-scope=user --uid=<uid> [--max-objects=<num objects>] [--max-size=<max size>]

For example:

radosgw-admin quota set --quota-scope=user --uid=johndoe --max-objects=1024 --max-size=1024

A negative value for num objects and / or max size means that the specific quota attribute check is disabled.

7.8.2. Enable and Disable User Quotas

Once you set a user quota, you may enable it. For example:

[root@master-zone]# radosgw-admin quota enable --quota-scope=user --uid=<uid>

You may disable an enabled user quota. For example:

[root@master-zone]# radosgw-admin quota disable --quota-scope=user --uid=<uid>

7.8.3. Set Bucket Quotas

Bucket quotas apply to the buckets owned by the specified uid. They are independent of the user.

[root@master-zone]# radosgw-admin quota set --uid=<uid> --quota-scope=bucket [--max-objects=<num objects>] [--max-size=<max size in bytes>]

A negative value for num objects and / or max size means that the specific quota attribute check is disabled.

7.8.4. Enable and Disable Bucket Quotas

Once you set a bucket quota, you may enable it. For example:

[root@master-zone]# radosgw-admin quota enable --quota-scope=bucket --uid=<uid>

You may disable an enabled bucket quota. For example:

[root@master-zone]# radosgw-admin quota-disable --quota-scope=bucket --uid=<uid>

7.8.5. Get Quota Settings

You may access each user’s quota settings via the user information API. To read user quota setting information with the CLI interface, execute the following:

# radosgw-admin user info --uid=<uid>

To get quota settings for a tenanted user, specify the user ID and the name of the tenant:

+ radosgw-admin user info --uid=_user-id_ --tenant=_tenant_

7.8.6. Update Quota Stats

Quota stats get updated asynchronously. You can update quota statistics for all users and all buckets manually to retrieve the latest quota stats.

[root@master-zone]# radosgw-admin user stats --uid=<uid> --sync-stats

7.8.7. Get User Quota Usage Stats

To see how much of the quota a user has consumed, execute the following:

# radosgw-admin user stats --uid=<uid>
Note

You should execute radosgw-admin user stats with the --sync-stats option to receive the latest data.

7.8.8. Quota Cache

Quota statistics are cached for each Ceph Gateway instance. If there are multiple instances, then the cache can keep quotas from being perfectly enforced, as each instance will have a different view of the quotas. The options that control this are rgw bucket quota ttl, rgw user quota bucket sync interval and rgw user quota sync interval. The higher these values are, the more efficient quota operations are, but the more out-of-sync multiple instances will be. The lower these values are, the closer to perfect enforcement multiple instances will achieve. If all three are 0, then quota caching is effectively disabled, and multiple instances will have perfect quota enforcement. See Appendix A, Configuration Reference for more details on these options.

7.8.9. Reading and Writing Global Quotas

You can read and write quota settings in a zonegroup map. To get a zonegroup map:

[root@master-zone]# radosgw-admin global quota get

The global quota settings can be manipulated with the global quota counterparts of the quota set, quota enable, and quota disable commands, for example:

[root@master-zone]# radosgw-admin global quota set --quota-scope bucket --max-objects 1024
[root@master-zone]# radosgw-admin global quota enable --quota-scope bucket
Note

In a multi-site configuration, where there is a realm and period present, changes to the global quotas must be committed using period update --commit. If there is no period present, the Ceph Object Gateways must be restarted for the changes to take effect.

7.9. Bucket management

As a storage administrator, when using the Ceph Object Gateway you can manage buckets by moving them between users and renaming them. You can create bucket notifications to trigger on specific events. Also, you can find orphan or leaky objects within the Ceph Object Gateway that can occur over the lifetime of a storage cluster.

7.9.1. Renaming buckets

You can rename buckets.

Note

If you want to allow underscores in bucket names, then set the rgw_s3_relaxed_bucket_names option to true.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway software.
  • An existing bucket.

Procedure

  1. List the buckets:

    radosgw-admin bucket list

    For example, note a bucket from the output:

    # radosgw-admin bucket list
    [
        "34150b2e9174475db8e191c188e920f6/swcontainer",
        "s3bucket1",
        "34150b2e9174475db8e191c188e920f6/swimpfalse",
        "c278edd68cfb4705bb3e07837c7ad1a8/ec2container",
        "c278edd68cfb4705bb3e07837c7ad1a8/demoten1",
        "c278edd68cfb4705bb3e07837c7ad1a8/demo-ct",
        "c278edd68cfb4705bb3e07837c7ad1a8/demopostup",
        "34150b2e9174475db8e191c188e920f6/postimpfalse",
        "c278edd68cfb4705bb3e07837c7ad1a8/demoten2",
        "c278edd68cfb4705bb3e07837c7ad1a8/postupsw"
    ]
  2. Rename the bucket:

    radosgw-admin bucket link --bucket=original-name --bucket-new-name=new-name --uid=user-ID

    For example, to rename the s3bucket1 bucket to s3newb:

    # radosgw-admin bucket link --bucket=s3bucket1 --bucket-new-name=s3newb --uid=testuser

    If the bucket is inside a tenant, specify the tenant as well:

    radosgw-admin bucket link --bucket=tenant/original-name --bucket-new-name=new-name --uid=tenant$user-ID

    For example:

    # radosgw-admin bucket link --bucket=test/s3bucket1 --bucket-new-name=s3newb --uid=test$testuser
  3. Verify the bucket was renamed:

    radosgw-admin bucket list

    For example, a bucket named s3newb exists now:

    # radosgw-admin bucket list
    [
        "34150b2e9174475db8e191c188e920f6/swcontainer",
        "34150b2e9174475db8e191c188e920f6/swimpfalse",
        "c278edd68cfb4705bb3e07837c7ad1a8/ec2container",
        "s3newb",
        "c278edd68cfb4705bb3e07837c7ad1a8/demoten1",
        "c278edd68cfb4705bb3e07837c7ad1a8/demo-ct",
        "c278edd68cfb4705bb3e07837c7ad1a8/demopostup",
        "34150b2e9174475db8e191c188e920f6/postimpfalse",
        "c278edd68cfb4705bb3e07837c7ad1a8/demoten2",
        "c278edd68cfb4705bb3e07837c7ad1a8/postupsw"
    ]

7.9.2. Moving buckets

The radosgw-admin bucket utility provides the ability to move buckets between users. To do so, link the bucket to a new user and change the ownership of the bucket to the new user.

You can move buckets:

7.9.2.1. Prerequisites

  • A running Red Hat Ceph Storage cluster
  • Ceph Object Gateway is installed
  • A bucket
  • Various tenanted and non-tenanted users

7.9.2.2. Moving buckets between non-tenanted users

The radosgw-admin bucket chown command provides the ability to change the ownership of buckets and all objects they contain from one user to another. To do so, unlink a bucket from the current user, link it to a new user, and change the ownership of the bucket to the new user.

Procedure

  1. Link the bucket to a new user:

    radosgw-admin bucket link --uid=user --bucket=bucket

    Replace:

    • user with the user name of the user to link the bucket to
    • bucket with the name of the bucket

    For example, to link the data bucket to the user named user2:

    # radosgw-admin bucket link --uid=user2 --bucket=data
  2. Verify that the bucket has been linked to user2 successfully:

    # radosgw-admin bucket list --uid=user2
    [
        "data"
    ]
  3. Change the ownership of the bucket to the new user:

    radosgw-admin bucket chown --uid=user --bucket=bucket

    Replace:

    • user with the user name of the user to change the bucket ownership to
    • bucket with the name of the bucket

    For example, to change the ownership of the data bucket to user2:

    # radosgw-admin bucket chown --uid=user2 --bucket=data
  4. Verify that the ownership of the data bucket has been successfully changed by checking the owner line in the output of the following command:

    # radosgw-admin bucket list --bucket=data

7.9.2.3. Moving buckets between tenanted users

You can move buckets between one tenanted user to another.

Procedure

  1. Link the bucket to a new user:

    radosgw-admin bucket link --bucket=current-tenant/bucket --uid=new-tenant$user

    Replace:

    • current-tenant with the name of the tenant the bucket is
    • bucket with the name of the bucket to link
    • new-tenant with the name of the tenant where the new user is
    • user with the user name of the new user

    For example, to link the data bucket from the test tenant to the user named user2 in the test2 tenant:

    # radosgw-admin bucket link --bucket=test/data --uid=test2$user2
  2. Verify that the bucket has been linked to user2 successfully:

    # radosgw-admin bucket list --uid=test$user2
    [
        "data"
    ]
  3. Change the ownership of the bucket to the new user:

    radosgw-admin bucket chown --bucket=new-tenant/bucket --uid=new-tenant$user

    Replace:

    • bucket with the name of the bucket to link
    • new-tenant with the name of the tenant where the new user is
    • user with the user name of the new user

    For example, to change the ownership of the data bucket to the user2 inside the test2 tenant:

    # radosgw-admin bucket chown --bucket='test2/data' --uid='test$tuser2'
  4. Verify that the ownership of the data bucket has been successfully changed by checking the owner line in the output of the following command:

    # radosgw-admin bucket list --bucket=test2/data

7.9.2.4. Moving buckets from non-tenanted users to tenanted users

You can move buckets from a non-tenanted user to a tenanted user.

Procedure

  1. Optional. If you do not already have multiple tenants, you can create them by enabling rgw_keystone_implicit_tenants and accessing the Ceph Object Gateway from an external tenant:

    Enable the rgw_keystone_implicit_tenants option:

    Example

    [root@mon ~]# ceph config set client.rgw rgw_keystone_implicit_tenants true

    Access the Ceph Object Gateway from an eternal tenant using either the s3cmd or swift command:

    # swift list

    Or use s3cmd:

    # s3cmd ls

    The first access from an external tenant creates an equivalent Ceph Object Gateway user.

  2. Move a bucket to a tenanted user:

    radosgw-admin bucket link --bucket=/bucket --uid='tenant$user'

    Replace:

    • bucket with the name of the bucket
    • tenant with the name of the tenant where the new user is
    • user with the user name of the new user

    For example, to move the data bucket to the tenanted-user inside the test tenant:

    # radosgw-admin bucket link --bucket=/data --uid='test$tenanted-user'
  3. Verify that the data bucket has been linked to tenanted-user successfully:

    # radosgw-admin bucket list --uid='test$tenanted-user'
    [
        "data"
    ]
  4. Change the ownership of the bucket to the new user:

    radosgw-admin bucket chown --bucket='tenant/bucket name' --uid='tenant$user'

    Replace:

    • bucket with the name of the bucket
    • tenant with the name of the tenant where the new user is
    • user with the user name of the new user

    For example, to change the ownership of the data bucket to tenanted-user that is inside the test tenant:

    # radosgw-admin bucket chown --bucket='test/data' --uid='test$tenanted-user'
  5. Verify that the ownership of the data bucket has been successfully changed by checking the owner line in the output of the following command:

    # radosgw-admin bucket list --bucket=test/data

7.9.3. Finding orphan and leaky objects

A healthy storage cluster does not have any orphan or leaky objects, but in some cases orphan or leaky objects can occur. For example, if the Ceph Object Gateway goes down in the middle of an operation, this can cause some objects to become orphans. Also, an undiscovered bug can cause orphan objects to occur.

Starting with Red Hat Ceph Storage 4.1, storage administrators can see how the Ceph Object Gateway objects map to the RADOS objects. The radosgw-admin command provides you a new tool to search for and produce a list of these potential orphan or leaky objects. Using the radoslist subcommand will display objects stored within buckets, or all buckets in the storage cluster. The rgw-orphan-list script will display orphan objects within a pool.

Important

The radoslist subcommand is replacing the deprecated orphans find and orphans finish subcommands.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • A running Ceph Object Gateway.

Procedure

  1. To generate a list of objects that hold data within a bucket:

    Syntax

    radosgw-admin bucket radoslist --bucket BUCKET_NAME

    Example

    [root@rgw ~]# radosgw-admin bucket radoslist --bucket mybucket

    Note

    If the BUCKET_NAME is omitted, then all objects in all buckets are displayed.

  2. To generate a list of orphans for a pool:

    [root@rgw ~]# rgw-orphan-list

    Example

    Available pools:
        .rgw.root
        default.rgw.control
        default.rgw.meta
        default.rgw.log
        default.rgw.buckets.index
        default.rgw.buckets.data
        rbd
        default.rgw.buckets.non-ec
        ma.rgw.control
        ma.rgw.meta
        ma.rgw.log
        ma.rgw.buckets.index
        ma.rgw.buckets.data
        ma.rgw.buckets.non-ec
    Which pool do you want to search for orphans?

    Enter the pool name to search for orphans.

    Important

    A data pool must be specified when using the rgw-orphan-list command, and not a metadata pool.

  3. Review the orphan objects in the list.
  4. To remove orphan objects:

    Syntax

    rados -p POOL_NAME rm OBJECT_NAME

    Example

    [root@rgw ~]# rados -p default.rgw.buckets.data rm myobject

    Warning

    Verify you are removing the correct objects. Executing the rados rm command will remove data from the storage cluster.

Additional Resources

  • See the Finding Orphan Objects section in the Red Hat Ceph Storage 3 Object Gateway Administration Guide for more details on the legacy radosgw-admin orphans find subcommand.

7.9.4. Bucket notifications

Bucket notifications provide a way to send information out of the Ceph Object Gateway when certain events happen in the bucket. Bucket notifications can be sent to HTTP, AMQP0.9.1 and Kafka endpoints. A notification entry must be created to send bucket notifications for events on a specific bucket and to a specific topic. A bucket notification can be created on a subset of event types or by default for all event types. The bucket notification can filter out events based on key prefix or suffix, regular expression matching the keys, and on the metadata attributes attached to the object, or the object tags. Bucket notifications have a REST API to provide configuration and control interfaces for the bucket notification mechanism.

Note

The bucket notifications API are enabled by default. If rgw_enable_apis configuration parameter is explicitly set, ensure that s3, and notifications are included. To verify this, run ceph --admin-daemon /var/run/ceph/ceph-client.rgw.NAME.asok config get rgw_enable_apis command. Replace NAME with the Ceph Object Gateway instance name.

Topic management using CLI

You can manage list, get, and remove topics for the Ceph Object Gateway buckets:

  • List topics: Run the following command to list the configuration of all topics:

    Example

    [ceph: host01 /]# radosgw-admin topic list

  • Get topics: Run the following command to get the configuration of a specific topic:

    Example

    [ceph: host01 /]# radosgw-admin topic get --topic=topic1

  • Remove topics: Run the following command to remove the configuration of a specific topic:

    Example

    [ceph: host01 /]# radosgw-admin topic rm --topic=topic1

    Note

    The topic is removed even if the Ceph Object Gateway bucket is configured to that topic.

7.9.5. Creating bucket notifications

Create bucket notifications at the bucket level. The notification configuration has the Red Hat Ceph Storage Object Gateway S3 events, ObjectCreated and ObjectRemoved. These need to be published and the destination to send the bucket notifications. Bucket notifications are S3 operations.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • A running HTTP server, RabbitMQ server, or a Kafka server.
  • Root-level access.
  • Installation of the Red Hat Ceph Storage Object Gateway.
  • User access key and secret key.
  • Endpoint parameters.
Important

Red Hat supports ObjectCreate events, such as, put, post, multipartUpload, and copy. Red Hat also supports ObjectRemove events, such as, object_delete and s3_multi_object_delete.

Procedure

  1. Create a s3 bucket.
  2. Create a SNS topic for http,amqp or kafka protocol.
  3. Create a s3 bucket notification for s3:objectCreate and s3:objectRemove events:

    Example

    client.put_bucket_notification_configuration(
       Bucket=bucket_name,
       NotificationConfiguration={
           'TopicConfigurations': [
               {
                   'Id': notification_name,
                   'TopicArn': topic_arn,
                   'Events': ['s3:ObjectCreated:*', 's3:ObjectRemoved:*']
               }]})

  4. Create s3 objects in the bucket.
  5. Verify the object creation events at the http or rabbitmq or kafka receiver.
  6. Delete the objects.
  7. Verify the object deletion events at the http or rabbitmq or kafka receiver.

7.9.6. Additional Resources

7.10. Bucket lifecycle

As a storage administrator, you can use a bucket lifecycle configuration to manage your objects so they are stored effectively throughout their lifetime. For example you can transition objects to less expensive storage classes, archive or even delete them based on your use case.

7.10.1. S3 bucket lifecycle transition within a storage cluster

You can use a bucket lifecycle configuration to manage objects so objects are stored effectively throughout the object’s lifetime. The object lifecycle transition rule allows you to manage,and effectively store the objects throughout the object’s lifetime. You can transition objects to less expensive storage classes,archive or even delete them.

You can create storage classes for:

  • Fast media, such as, SSD or NVMe for I/O sensitive workloads
  • Slow magnetic media, such as, SAS or SATA for archiving.

You can create a schedule for data movement between a hot storage class and a cold storage class. You can schedule this movement after a specified time so that the object expires and is deleted permanently for example you can transition objects to a storage class 30 days after you have created or even archived the objects to a storage class one year after creating them. You can do this through a transition rule. This rule applies to an object transitioning from one storage class to another. The lifecycle configuration contains one or more rules using the <Rule> element.

Additional Resources

  • See the Red Hat Ceph Storage Developer Guide for details on bucket lifecycle.

7.10.2. Transitioning an object from one storage class to another

The object lifecycle transition rule allows you to transition an object from one storage class to another class.

Prerequisites

  • Installation of the Ceph Object Gateway software.
  • Root-level access to the Ceph Object Gateway node.
  • An S3 user created with user access.

Procedure

  1. Create a new data pool:

    Syntax

    ceph osd pool create POOL_NAME

    Example

    [root@rgw /]# ceph osd pool create test.hot.data

  2. Add a new storage class:

    Syntax

    radosgw-admin zonegroup placement add  --rgw-zonegroup default --placement-id PLACEMENT_TARGET --storage-class STORAGE_CLASS

    Example

    [root@rgw /]# radosgw-admin zonegroup placement add  --rgw-zonegroup default --placement-id default-placement --storage-class hot.test
    {
            "key": "default-placement",
            "val": {
                "name": "default-placement",
                "tags": [],
                "storage_classes": [
                    "STANDARD",
                    "hot.test"
                ]
            }
        }

  3. Provide the zone placement information for the new storage class:

    Syntax

    radosgw-admin zone placement add --rgw-zone default --placement-id PLACEMENT_TARGET --storage-class STORAGE_CLASS --data-pool DATA_POOL

    Example

    [root@rgw /]# radosgw-admin zone placement add --rgw-zone default --placement-id default-placement --storage-class hot.test --data-pool test.hot.data
    {
               "key": "default-placement",
               "val": {
                   "index_pool": "test_zone.rgw.buckets.index",
                   "storage_classes": {
                       "STANDARD": {
                           "data_pool": "test.hot.data"
                       },
                       "hot.test": {
                           "data_pool": "test.hot.data",
                      }
                   },
                   "data_extra_pool": "",
                   "index_type": 0
               }

    Note

    Consider setting the compression_type when creating cold or archival data storage pools with write once.

  4. Enable rgw application on the data pool:

    Syntax

    ceph osd pool application enable POOL_NAME rgw

    Example

    [root@rgw /] ceph osd pool application enable test.hot.data rgw
    enabled application 'rgw' on pool 'test.hot.data'

  5. Restart all the rgw daemons.
  6. Create a bucket:

    Example

    [root@rgw /]# aws s3api create-bucket --bucket testbucket10 --create-bucket-configuration LocationConstraint=default:default-placement --endpoint-url http://1x.7x.2xx.1xx:80

  7. Add the object:

    Example

    [root@rgw /]#aws --endpoint=http://1x.7x.2xx.1xx:80 s3api put-object --bucket testbucket10  --key compliance-upload --body /root/test2.txt

  8. Create a second data pool:

    Syntax

    ceph osd pool create POOL_NAME

    Example

    [root@rgw /]# ceph osd pool create test.cold.data

  9. Add a new storage class:

    Syntax

    radosgw-admin zonegroup placement add  --rgw-zonegroup default --placement-id PLACEMENT_TARGET --storage-class STORAGE_CLASS

    Example

    [root@rgw /]# radosgw-admin zonegroup placement add  --rgw-zonegroup default --placement-id default-placement --storage-class cold.test
    {
            "key": "default-placement",
            "val": {
                "name": "default-placement",
                "tags": [],
                "storage_classes": [
                    "STANDARD",
                    "cold.test"
                ]
            }
        }

  10. Provide the zone placement information for the new storage class:

    Syntax

    radosgw-admin zone placement add --rgw-zone default --placement-id PLACEMENT_TARGET --storage-class STORAGE_CLASS --data-pool DATA_POOL

    Example

    [root@rgw /]# radosgw-admin zone placement add --rgw-zone default --placement-id default-placement --storage-class cold.test --data-pool test.cold.data

  11. Enable rgw application on the data pool:

    Syntax

    ceph osd pool application enable POOL_NAME rgw

    Example

    [root@rgw /] ceph osd pool application enable test.cold.data rgw
    enabled application 'rgw' on pool 'test.cold.data'

  12. Restart all the rgw daemons.
  13. To view zone group configuration, execute:

    Syntax

    radosgw-admin zonegroup get
    {
        "id": "3019de59-ddde-4c5c-b532-7cdd29de09a1",
        "name": "default",
        "api_name": "default",
        "is_master": "true",
        "endpoints": [],
        "hostnames": [],
        "hostnames_s3website": [],
        "master_zone": "adacbe1b-02b4-41b8-b11d-0d505b442ed4",
        "zones": [
            {
                "id": "adacbe1b-02b4-41b8-b11d-0d505b442ed4",
                "name": "default",
                "endpoints": [],
                "log_meta": "false",
                "log_data": "false",
                "bucket_index_max_shards": 11,
                "read_only": "false",
                "tier_type": "",
                "sync_from_all": "true",
                "sync_from": [],
                "redirect_zone": ""
            }
        ],
        "placement_targets": [
            {
                "name": "default-placement",
                "tags": [],
                "storage_classes": [
                    "hot.test",
                    "cold.test",
                    "STANDARD"
                ]
            }
        ],
        "default_placement": "default-placement",
        "realm_id": "",
        "sync_policy": {
            "groups": []
        }
    }

  14. To view the zone configuration,execute:

    Syntax

    radosgw-admin zone get
    {
        "id": "adacbe1b-02b4-41b8-b11d-0d505b442ed4",
        "name": "default",
        "domain_root": "default.rgw.meta:root",
        "control_pool": "default.rgw.control",
        "gc_pool": "default.rgw.log:gc",
        "lc_pool": "default.rgw.log:lc",
        "log_pool": "default.rgw.log",
        "intent_log_pool": "default.rgw.log:intent",
        "usage_log_pool": "default.rgw.log:usage",
        "roles_pool": "default.rgw.meta:roles",
        "reshard_pool": "default.rgw.log:reshard",
        "user_keys_pool": "default.rgw.meta:users.keys",
        "user_email_pool": "default.rgw.meta:users.email",
        "user_swift_pool": "default.rgw.meta:users.swift",
        "user_uid_pool": "default.rgw.meta:users.uid",
        "otp_pool": "default.rgw.otp",
        "system_key": {
            "access_key": "",
            "secret_key": ""
        },
        "placement_pools": [
            {
                "key": "default-placement",
                "val": {
                    "index_pool": "default.rgw.buckets.index",
                    "storage_classes": {
                        "cold.test": {
                            "data_pool": "test.cold.data"
                        },
                        "hot.test": {
                            "data_pool": "test.hot.data"
                        },
                        "STANDARD": {
                            "data_pool": "default.rgw.buckets.data"
                        }
                    },
                    "data_extra_pool": "default.rgw.buckets.non-ec",
                    "index_type": 0
                }
            }
        ],
        "realm_id": "",
        "notif_pool": "default.rgw.log:notif"
    }

  15. Create a bucket:

    Example

    [root@rgw /]# aws s3api create-bucket --bucket testbucket10 --create-bucket-configuration LocationConstraint=default:default-placement --endpoint-url http://1x.7x.2xx.1xx:80

  16. Create a JSON file for lifecycle configuration:

    Example

    [root@rgw ~]# vi lifecycle.json

  17. Add the specific lifecyle configuration rule in the file:

    Example

    {
        "Rules": [
            {
                "Filter": {
                    "Prefix": ""
                },
                "Status": "Enabled",
                "Transitions": [
                    {
                        "Days": 5,
                        "StorageClass": "hot.test"
                    },
     {
                        "Days": 20,
                        "StorageClass": "cold.test"
                    }
                ],
                "Expiration": {
                    "Days": 365
                },
                "ID": "double transition and expiration"
            }
        ]
    }

    The lifecycle configuration example shows an object that will transition from the default STANDARD storage class to the hot.test storage class after 5 days, again transitions after 20 days to the cold.test storage class, and finally expires after 365 days in the cold.test storage class.

  18. Set the lifecycle configuration on the bucket:

    Example

    [root@rgw-2 ~] aws s3api put-bucket-lifecycle-configuration --bucket testbucket20 --lifecycle-configuration file://lifecycle.json

  19. Retrieve the lifecycle configuration on the bucket:

    Example

    [root@rgw ~]aws s3api get-bucket-lifecycle-configuration --bucket testbucket20
    {
        "Rules": [
            {
                "Expiration": {
                    "Days": 365
                },
                "ID": "double transition and expiration",
                "Prefix": "",
                "Status": "Enabled",
                "Transitions": [
                    {
                        "Days": 20,
                        "StorageClass": "cold.test"
                    },
                    {
                        "Days": 5,
                        "StorageClass": "hot.test"
                    }
                ]
            }
        ]
    }

Additional Resources

  • See the Red Hat Ceph Storage Developer Guide for details on bucket lifecycle.

7.11. Usage

The Ceph Object Gateway logs usage for each user. You can track user usage within date ranges too.

Options include:

  • Start Date: The --start-date option allows you to filter usage stats from a particular start date (format: yyyy-mm-dd[HH:MM:SS]).
  • End Date: The --end-date option allows you to filter usage up to a particular date (format: yyyy-mm-dd[HH:MM:SS]).
  • Log Entries: The --show-log-entries option allows you to specify whether or not to include log entries with the usage stats (options: true | false).
Note

You can specify time with minutes and seconds, but it is stored with 1 hour resolution.

7.11.1. Show Usage

To show usage statistics, specify the usage show. To show usage for a particular user, you must specify a user ID. You may also specify a start date, end date, and whether or not to show log entries.

# radosgw-admin usage show \
                --uid=johndoe --start-date=2012-03-01 \
                --end-date=2012-04-01

You may also show a summary of usage information for all users by omitting a user ID.

# radosgw-admin usage show --show-log-entries=false

7.11.2. Trim Usage

With heavy use, usage logs can begin to take up storage space. You can trim usage logs for all users and for specific users. You may also specify date ranges for trim operations.

[root@master-zone]# radosgw-admin usage trim --start-date=2010-01-01 \
                    --end-date=2010-12-31

[root@master-zone]# radosgw-admin usage trim --uid=johndoe
[root@master-zone]# radosgw-admin usage trim --uid=johndoe --end-date=2013-12-31

7.12. Optimize the Ceph Object Gateway’s garbage collection

When new data objects are written into the storage cluster, the Ceph Object Gateway immediately allocates the storage for these new objects. After you delete or overwrite data objects in the storage cluster, the Ceph Object Gateway deletes those objects from the bucket index. Some time afterward, the Ceph Object Gateway then purges the space that was used to store the objects in the storage cluster. The process of purging the deleted object data from the storage cluster is known as Garbage Collection, or GC.

Garbage collection operations typically run in the background. You can configure these operations to either execute continuously, or to run only during intervals of low activity and light workloads. By default, the Ceph Object Gateway conducts GC operations continuously. Because GC operations are a normal part of Ceph Object Gateway operations, deleted objects that are eligible for garbage collection exist most of the time.

7.12.1. Viewing the garbage collection queue

Before you purge deleted and overwritten objects from the storage cluster, use radosgw-admin to view the objects awaiting garbage collection.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to the Ceph Object Gateway.

Procedure

  1. To view the queue of objects awaiting garbage collection:

    Example

    [root@rgw ~] radosgw-admin gc list

Note

To list all entries in the queue, including unexpired entries, use the --include-all option.

7.12.2. Adjusting Garbage Collection Settings

The Ceph Object Gateway allocates storage for new and overwritten objects immediately. Additionally, the parts of a multi-part upload also consume some storage.

The Ceph Object Gateway purges the storage space used for deleted objects after deleting the objects from the bucket index. Similarly, the Ceph Object Gateway will delete data associated with a multi-part upload after the multi-part upload completes or when the upload has gone inactive or failed to complete for a configurable amount of time. The process of purging the deleted object data from the Red Hat Ceph Storage cluster is known as garbage collection (GC).

Viewing the objects awaiting garbage collection can be done with the following command:

radosgw-admin gc list

Garbage collection is a background activity that executes continuously or during times of low loads, depending upon how the storage administrator configures the Ceph Object Gateway. By default, the Ceph Object Gateway conducts garbage collection operations continuously. Since garbage collection operations are a normal function of the Ceph Object Gateway, especially with object delete operations, objects eligible for garbage collection exist most of the time.

Some workloads can temporarily or permanently outpace the rate of garbage collection activity. This is especially true of delete-heavy workloads, where many objects get stored for a short period of time and then deleted. For these types of workloads, storage administrators can increase the priority of garbage collection operations relative to other operations with the following configuration parameters:

  • The rgw_gc_obj_min_wait configuration option waits a minimum length of time, in seconds, before purging a deleted object’s data. The default value is two hours, or 7200 seconds. The object is not purged immediately, because a client might be reading the object. Under delete heavy workloads, this setting can consume too much storage or have a large number of deleted objects to purge. Red Hat recommends not setting this value below 30 minutes, or 1800 seconds.
  • The rgw_gc_processor_period configuration option is the garbage collection cycle run time. That is, the amount of time between the start of consecutive runs of garbage collection threads. If garbage collection runs longer than this period, the Ceph Object Gateway will not wait before running a garbage collection cycle again.
  • The rgw_gc_max_concurrent_io configuration option specifies the maximum number of concurrent IO operations that the gateway garbage collection thread will use when purging deleted data. Under delete heavy workloads, consider increasing this setting to a larger number of concurrent IO operations.
  • The rgw_gc_max_trim_chunk configuration option specifies the maximum number of keys to remove from the garbage collector log in a single operation. Under delete heavy operations, consider increasing the maximum number of keys so that more objects are purged during each garbage collection operation.

Starting with Red Hat Ceph Storage 4.1, offloading the index object’s OMAP from the garbage collection log helps lessen the performance impact of garbage collection activities on the storage cluster. Some new configuration parameters have been added to Ceph Object Gateway to tune the garbage collection queue, as follows:

  • The rgw_gc_max_deferred_entries_size configuration option sets the maximum size of deferred entries in the garbage collection queue.
  • The rgw_gc_max_queue_size configuration option sets the maximum queue size used for garbage collection. This value should not be greater than osd_max_object_size minus rgw_gc_max_deferred_entries_size minus 1 KB.
  • The rgw_gc_max_deferred configuration option sets the maximum number of deferred entries stored in the garbage collection queue.
Note

These garbage collection configuration parameters are for Red Hat Ceph Storage 5 and higher.

Note

In testing, with an evenly balanced delete-write workload, such as 50% delete and 50% write operations, the storage cluster fills completely in 11 hours. This is because Ceph Object Gateway garbage collection fails to keep pace with the delete operations. The cluster status switches to the HEALTH_ERR state if this happens. Aggressive settings for parallel garbage collection tunables significantly delayed the onset of storage cluster fill in testing and can be helpful for many workloads. Typical real-world storage cluster workloads are not likely to cause a storage cluster fill primarily due to garbage collection.

7.12.3. Adjusting garbage collection for delete-heavy workloads

Some workloads may temporarily or permanently outpace the rate of garbage collection activity. This is especially true of delete-heavy workloads, where many objects get stored for a short period of time and are then deleted. For these types of workloads, consider increasing the priority of garbage collection operations relative to other operations. Contact Red Hat Support with any additional questions about Ceph Object Gateway Garbage Collection.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to all nodes in the storage cluster.

Procedure

  1. Set the value of rgw_gc_max_concurrent_io to 20, and the value of rgw_gc_max_trim_chunk to 64:

    Example

    [root@mon ~]# ceph config set client.rgw rgw_gc_max_concurrent_io 20
    [root@mon ~]# ceph config set client.rgw rgw_gc_max_trim_chunk 64

  2. Restart the Ceph Object Gateway to allow the changed settings to take effect.
  3. Monitor the storage cluster during GC activity to verify that the increased values do not adversely affect performance.
Important

Never modify the value for the rgw_gc_max_objs option in a running cluster. You should only change this value before deploying the RGW nodes.

7.13. Optimize the Ceph Object Gateway’s data object storage

Bucket life cycle configuration optimizes data object storage to increase its efficiency and to provide effective storage throughout the lifetime of the data.

The S3 API in the Ceph Object Gateway currently supports a subset of the AWS bucket life cycle configuration actions:

  • Expiration
  • NoncurrentVersionExpiration
  • AbortIncompleteMultipartUpload

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to all of the nodes in the storage cluster.

7.13.1. Parallel thread processing for bucket life cycles

The Ceph Object Gateway now allows for parallel thread processing of bucket life cycles across multiple Ceph Object Gateway instances. Increasing the number of threads that run in parallel enables the Ceph Object Gateway to process large workloads more efficiently. In addition, the Ceph Object Gateway now uses a numbered sequence for index shard enumeration instead of using in-order numbering.

7.13.2. Optimizing the bucket life cycle

Two options in the Ceph configuration file affect the efficiency of bucket life cycle processing:

  • rgw_lc_max_worker specifies the number of life cycle worker threads to run in parallel. This enables the simultaneous processing of both bucket and index shards. The default value for this option is 3.
  • rgw_lc_max_wp_worker specifies the number of threads in each life cycle worker thread’s work pool. This option helps to accelerate processing for each bucket. The default value for this option is 3.

For a workload with a large number of buckets — for example, a workload with thousands of buckets — consider increasing the value of the rgw_lc_max_worker option.

For a workload with a smaller number of buckets but with a higher number of objects in each bucket — such as in the hundreds of thousands — consider increasing the value of the rgw_lc_max_wp_worker option.

Note

Before increasing the value of either of these options, please validate current storage cluster performance and Ceph Object Gateway utilization. Red Hat does not recommend that you assign a value of 10 or above for either of these options.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to all of the nodes in the storage cluster.

Procedure

  1. To increase the number of threads to run in parallel, set the value of rgw_lc_max_worker to a value between 3 and 9:

    Example

    [root@mon ~]# ceph config set client.rgw rgw_lc_max_worker 7

  2. To increase the number of threads in each thread’s work pool, set the value of rgw_lc_max_wp_worker to a value between 3 and 9:

    Example

    [root@mon ~]# ceph config set client.rgw rgw_lc_max_wp_worker 7

  3. Restart the Ceph Object Gateway to allow the changed settings to take effect.
  4. Monitor the storage cluster to verify that the increased values do not adversely affect performance.

Additional Resources

7.13.3. Additional Resources

Chapter 8. Testing

As a storage administrator, you can do basic functionality testing to verify that the Ceph Object Gateway environment is working as expected. You can use the REST interfaces by creating an initial Ceph Object Gateway user for the S3 interface, and then create a subuser for the Swift interface.

8.1. Prerequisites

  • A healthy running Red Hat Ceph Storage cluster.
  • Installation of the Ceph Object Gateway software.

8.2. Create an S3 User

To test the gateway, create an S3 user and grant the user access. The man radosgw-admin command provides information on additional command options.

Note

In a multi-site deployment, always create a user on a host in the master zone of the master zone group.

Prerequisites

  • root or sudo access
  • Ceph Object Gateway installed

Procedure

  1. Create an S3 user:

    radosgw-admin user create --uid=name --display-name="First User"

    Replace name with the name of the S3 user, for example:

    [root@master-zone]# radosgw-admin user create --uid="testuser" --display-name="First User"
    {
        "user_id": "testuser",
        "display_name": "First User",
        "email": "",
        "suspended": 0,
        "max_buckets": 1000,
        "auid": 0,
        "subusers": [],
        "keys": [
            {
                "user": "testuser",
                "access_key": "CEP28KDIQXBKU4M15PDC",
                "secret_key": "MARoio8HFc8JxhEilES3dKFVj8tV3NOOYymihTLO"
            }
        ],
        "swift_keys": [],
        "caps": [],
        "op_mask": "read, write, delete",
        "default_placement": "",
        "placement_tags": [],
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "temp_url_keys": [],
        "type": "rgw"
    }
  2. Verify the output to ensure that the values of access_key and secret_key do not include a JSON escape character (\). These values are needed for access validation, but certain clients cannot handle if the values include JSON escape character. To fix this problem, perform one of the following actions:

    • Remove the JSON escape character.
    • Encapsulate the string in quotes.
    • Regenerate the key and ensure that is does not include a JSON escape character.
    • Specify the key and secret manually.

    Do not remove the forward slash / because it is a valid character.

8.3. Create a Swift user

To test the Swift interface, create a Swift subuser. Creating a Swift user is a two step process. The first step is to create the user. The second step is to create the secret key.

Note

In a multi-site deployment, always create a user on a host in the master zone of the master zone group.

Prerequisites

  • Installation of the Ceph Object Gateway.
  • Root-level access to the Ceph Object Gateway node.

Procedure

  1. Create the Swift user:

    Syntax

    radosgw-admin subuser create --uid=NAME --subuser=NAME:swift --access=full

    Replace NAME with the Swift user name, for example:

    Example

    [root@rgw]# radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --access=full
    {
        "user_id": "testuser",
        "display_name": "First User",
        "email": "",
        "suspended": 0,
        "max_buckets": 1000,
        "auid": 0,
        "subusers": [
            {
                "id": "testuser:swift",
                "permissions": "full-control"
            }
        ],
        "keys": [
            {
                "user": "testuser",
                "access_key": "O8JDE41XMI74O185EHKD",
                "secret_key": "i4Au2yxG5wtr1JK01mI8kjJPM93HNAoVWOSTdJd6"
            }
        ],
        "swift_keys": [
            {
                "user": "testuser:swift",
                "secret_key": "13TLtdEW7bCqgttQgPzxFxziu0AgabtOc6vM8DLA"
            }
        ],
        "caps": [],
        "op_mask": "read, write, delete",
        "default_placement": "",
        "placement_tags": [],
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "temp_url_keys": [],
        "type": "rgw"
    }

  2. Create the secret key:

    Syntax

    radosgw-admin key create --subuser=NAME:swift --key-type=swift --gen-secret

    Replace NAME with the Swift user name, for example:

    Example

    [root@rgw]# radosgw-admin key create --subuser=testuser:swift --key-type=swift --gen-secret
    {
        "user_id": "testuser",
        "display_name": "First User",
        "email": "",
        "suspended": 0,
        "max_buckets": 1000,
        "auid": 0,
        "subusers": [
            {
                "id": "testuser:swift",
                "permissions": "full-control"
            }
        ],
        "keys": [
            {
                "user": "testuser",
                "access_key": "O8JDE41XMI74O185EHKD",
                "secret_key": "i4Au2yxG5wtr1JK01mI8kjJPM93HNAoVWOSTdJd6"
            }
        ],
        "swift_keys": [
            {
                "user": "testuser:swift",
                "secret_key": "a4ioT4jEP653CDcdU8p4OuhruwABBRZmyNUbnSSt"
            }
        ],
        "caps": [],
        "op_mask": "read, write, delete",
        "default_placement": "",
        "placement_tags": [],
        "bucket_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "user_quota": {
            "enabled": false,
            "check_on_raw": false,
            "max_size": -1,
            "max_size_kb": 0,
            "max_objects": -1
        },
        "temp_url_keys": [],
        "type": "rgw"
    }

8.4. Test S3 Access

You need to write and run a Python test script for verifying S3 access. The S3 access test script will connect to the radosgw, create a new bucket and list all buckets. The values for aws_access_key_id and aws_secret_access_key are taken from the values of access_key and secret_key returned by the radosgw_admin command.

Prerequisites

  • A running Red Hat Ceph Storage cluster.
  • Root-level access to the nodes.

Procedure

  1. Enable the High Availability repository for Red Hat Enterprise Linux 8:

    subscription-manager repos --enable=rhel-8-for-x86_64-highavailability-rpms
  2. Install the python3-boto3 package:

    dnf install python3-boto3
  3. Create the Python script:

    vi s3test.py
  4. Add the following contents to the file:

    Syntax

    import boto3
    
    endpoint = "" # enter the endpoint URL along with the port "http://URL:_PORT_"
    
    access_key = 'ACCESS'
    secret_key = 'SECRET'
    
    s3 = boto3.client(
            's3',
            endpoint_url=endpoint,
            aws_access_key_id=access_key,
            aws_secret_access_key=secret_key
            )
    
    s3.create_bucket(Bucket='my-new-bucket')
    
    response = s3.list_buckets()
    for bucket in response['Buckets']:
        print("{name}\t{created}".format(
    		name = bucket['Name'],
    		created = bucket['CreationDate']
    ))

    1. Replace ZONE with the zone name of the host where you have configured the gateway service. That is, the gateway host. Ensure that the host`setting resolves with DNS. Replace `PORT with the port number of the gateway.
    2. Replace ACCESS and SECRET with the access_key and secret_key values from the Create an S3 User section in the Red Hat Ceph Storage Object Gateway Configuration and Administration Guide.
  5. Run the script:

    python3 s3test.py

    The output will be something like the following:

    my-new-bucket 2021-08-16T17:09:10.000Z

8.5. Test Swift Access

Swift access can be verified via the swift command line client. The command man swift will provide more information on available command line options.

To install swift client, execute the following:

sudo yum install python-setuptools
sudo easy_install pip
sudo pip install --upgrade setuptools
sudo pip install --upgrade python-swiftclient

To test swift access, execute the following:

swift -A http://{IP ADDRESS}:{port}/auth/1.0 -U testuser:swift -K '{swift_secret_key}' list

Replace {IP ADDRESS} with the public IP address of the gateway server and {swift_secret_key} with its value from the output of radosgw-admin key create command executed for the swift user. Replace {port} with the port number you are using with Beast. If you don’t replace the port, it will default to port 80.

For example:

swift -A http://10.19.143.116:80/auth/1.0 -U testuser:swift -K '244+fz2gSqoHwR3lYtSbIyomyPHf3i7rgSJrF/IA' list

The output should be:

my-new-bucket

Appendix A. Configuration Reference

As a storage administrator, you can set various options for the Ceph Object Gateway. These options contain default values. If you do not specify each option, then the default value is set automatically.

To set specific values for these options, update the configuration database by using the ceph config set client.rgw OPTION VALUE command.

A.1. General Settings

NameDescriptionTypeDefault

rgw_data

Sets the location of the data files for Ceph Object Gateway.

String

/var/lib/ceph/radosgw/$cluster-$id

rgw_enable_apis

Enables the specified APIs.

String

s3, s3website, swift, swift_auth, admin, sts, iam, notifications

rgw_cache_enabled

Whether the Ceph Object Gateway cache is enabled.

Boolean

true

rgw_cache_lru_size

The number of entries in the Ceph Object Gateway cache.

Integer

10000

rgw_socket_path

The socket path for the domain socket. FastCgiExternalServer uses this socket. If you do not specify a socket path, Ceph Object Gateway will not run as an external server. The path you specify here must be the same as the path specified in the rgw.conf file.

String

N/A

rgw_host

The host for the Ceph Object Gateway instance. Can be an IP address or a hostname.

String

0.0.0.0

rgw_port

Port the instance listens for requests. If not specified, Ceph Object Gateway runs external FastCGI.

String

None

rgw_dns_name

The DNS name of the served domain. See also the hostnames setting within zone groups.

String

None

rgw_script_uri

The alternative value for the SCRIPT_URI if not set in the request.

String

None

rgw_request_uri

The alternative value for the REQUEST_URI if not set in the request.

String

None

rgw_print_continue

Enable 100-continue if it is operational.

Boolean

true

rgw_remote_addr_param

The remote address parameter. For example, the HTTP field containing the remote address, or the X-Forwarded-For address if a reverse proxy is operational.

String

REMOTE_ADDR

rgw_op_thread_timeout

The timeout in seconds for open threads.

Integer

600

rgw_op_thread_suicide_timeout

The time timeout in seconds before a Ceph Object Gateway process dies. Disabled if set to 0.

Integer

0

rgw_thread_pool_size

The size of the thread pool.

Integer

512 threads.

rgw_num_control_oids

The number of notification objects used for cache synchronization between different rgw instances.

Integer

8

rgw_init_timeout

The number of seconds before Ceph Object Gateway gives up on initialization.

Integer

30

rgw_mime_types_file

The path and location of the MIME types. Used for Swift auto-detection of object types.

String

/etc/mime.types

rgw_gc_max_objs

The maximum number of objects that may be handled by garbage collection in one garbage collection processing cycle.

Integer

32

rgw_gc_obj_min_wait

The minimum wait time before the object may be removed and handled by garbage collection processing.

Integer

2 * 3600

rgw_gc_processor_max_time

The maximum time between the beginning of two consecutive garbage collection processing cycles.

Integer

3600

rgw_gc_processor_period

The cycle time for garbage collection processing.

Integer

3600

rgw_s3 success_create_obj_status

The alternate success status response for create-obj.

Integer

0

rgw_resolve_cname

Whether rgw should use DNS CNAME record of the request hostname field (if hostname is not equal to rgw_dns name).

Boolean

false

rgw_object_stripe_size

The size of an object stripe for Ceph Object Gateway objects.

Integer

4 << 20

rgw_extended_http_attrs

Add new set of attributes that could be set on an object. These extra attributes can be set through HTTP header fields when putting the objects. If set, these attributes will return as HTTP fields when doing GET/HEAD on the object.

String

None. For example: "content_foo, content_bar"

rgw_exit_timeout_secs

Number of seconds to wait for a process before exiting unconditionally.

Integer

120

rgw_get_obj_window_size

The window size in bytes for a single object request.

Integer

16 << 20

rgw_get_obj_max_req_size

The maximum request size of a single get operation sent to the Ceph Storage Cluster.

Integer

4 << 20

rgw_relaxed_s3_bucket_names

Enables relaxed S3 bucket names rules for zone group buckets.

Boolean

false

rgw_list buckets_max_chunk

The maximum number of buckets to retrieve in a single operation when listing user buckets.

Integer

1000

rgw_override_bucket_index_max_shards

The number of shards for the bucket index object. A value of 0 indicates there is no sharding. Red Hat does not recommend to set a value too large (for example, 1000) as it increases the cost for bucket listing.

This variable should be set in the [client] or the [global] section so it is automatically applied to radosgw-admin commands.

Integer

0

rgw_curl_wait_timeout_ms

The timeout in milliseconds for certain curl calls.

Integer

1000

rgw_copy_obj_progress

Enables output of object progress during long copy operations.

Boolean

true

rgw_copy_obj_progress_every_bytes

The minimum bytes between copy progress output.

Integer

1024 * 1024

rgw_admin_entry

The entry point for an admin request URL.

String

admin

rgw_content_length_compat

Enable compatability handling of FCGI requests with both CONTENT_LENGTH AND HTTP_CONTENT_LENGTH set.

Boolean

false

rgw_bucket_default_quota_max_objects

The default maximum number of objects per bucket. This value is set on new users if no other quota is specified. It has no effect on existing users.

This variable should be set in the [client] or the [global] section so it is automatically applied to radosgw-admin commands.

Integer

-1

rgw_bucket_quota_ttl

The amount of time in seconds cached quota information is trusted. After this timeout, the quota information will be re-fetched from the cluster.

Integer

600

rgw_user_quota_bucket_sync_interval

The amount of time in seconds bucket quota information is accumulated before syncing to the cluster. During this time, other RGW instances will not see the changes in bucket quota stats from operations on this instance.

Integer

180

rgw_user_quota_sync_interval

The amount of time in seconds user quota information is accumulated before syncing to the cluster. During this time, other RGW instances will not see the changes in user quota stats from operations on this instance.

Integer

3600 * 24

A.2. About pools

Ceph zones map to a series of Ceph Storage Cluster pools.

Manually Created Pools vs. Generated Pools

If the user key for the Ceph Object Gateway contains write capabilities, the gateway has the ability to create pools automatically. This is convenient for getting started. However, the Ceph Object Storage Cluster uses the placement group default values unless they were set in the Ceph configuration file. Additionally, Ceph will use the default CRUSH hierarchy. These settings are NOT ideal for production systems.

The default pools for the Ceph Object Gateway’s default zone include:

  • .rgw.root
  • .default.rgw.control
  • .default.rgw.meta
  • .default.rgw.log
  • .default.rgw.buckets.index
  • .default.rgw.buckets.data
  • .default.rgw.buckets.non-ec

The Ceph Object Gateway creates pools on a per zone basis. If you create the pools manually, prepend the zone name. The system pools store objects related to system control, logging, user information, etc. By convention, these pool names have the zone name prepended to the pool name.

  • .<zone-name>.rgw.control: The control pool.
  • .<zone-name>.log: The log pool contains logs of all bucket/container and object actions such as create, read, update and delete.
  • .<zone-name>.rgw.buckets.index: This pool stores index of the buckes.
  • .<zone-name>.rgw.buckets.data: This pool stores data of the buckets.
  • .<zone-name>.rgw.meta: The metadata pool stores user_keys and other critical metadata.
  • .<zone-name>.meta:users.uid: The user ID pool contains a map of unique user IDs.
  • .<zone-name>.meta:users.keys: The keys pool contains access keys and secret keys for each user ID.
  • .<zone-name>.meta:users.email: The email pool contains email addresses associated to a user ID.
  • .<zone-name>.meta:users.swift: The Swift pool contains the Swift subuser information for a user ID.

Ceph Object Gateways store data for the bucket index (index_pool) and bucket data (data_pool) in placement pools. These may overlap; that is, you may use the same pool for the index and the data. The index pool for default placement is {zone-name}.rgw.buckets.index and for the data pool for default placement is {zone-name}.rgw.buckets.

NameDescriptionTypeDefault

rgw_zonegroup_root_pool

The pool for storing all zone group-specific information.

String

.rgw.root

rgw_zone_root_pool

The pool for storing zone-specific information.

String

.rgw.root

A.3. Swift Settings

NameDescriptionTypeDefault

rgw_enforce_swift_acls

Enforces the Swift Access Control List (ACL) settings.

Boolean

true

rgw_swift_token_expiration

The time in seconds for expiring a Swift token.

Integer

24 * 3600

rgw_swift_url

The URL for the Ceph Object Gateway Swift API.

String

None

rgw_swift_url_prefix

The URL prefix for the Swift API (e.g., http://fqdn.com/swift).

swift

N/A

rgw_swift_auth_url

Default URL for verifying v1 auth tokens (if not using internal Swift auth).

String

None

rgw_swift_auth_entry

The entry point for a Swift auth URL.

String

auth

A.4. Logging Settings

NameDescriptionTypeDefault

rgw_log_nonexistent_bucket

Enables Ceph Object Gateway to log a request for a non-existent bucket.

Boolean

false

rgw_log_object_name

The logging format for an object name. See manpage date for details about format specifiers.

Date

%Y-%m-%d-%H-%i-%n

rgw_log_object_name_utc

Whether a logged object name includes a UTC time. If false, it uses the local time.

Boolean

false

rgw_usage_max_shards

The maximum number of shards for usage logging.

Integer

32

rgw_usage_max_user_shards

The maximum number of shards used for a single user’s usage logging.

Integer

1

rgw_enable_ops_log

Enable logging for each successful Ceph Object Gateway operation.

Boolean

false

rgw_enable_usage_log

Enable the usage log.

Boolean

false

rgw_ops_log_rados

Whether the operations log should be written to the Ceph Storage Cluster backend.

Boolean

true

rgw_ops_log_socket_path

The Unix domain socket for writing operations logs.

String

None

rgw_ops_log_data-backlog

The maximum data backlog data size for operations logs written to a Unix domain socket.

Integer

5 << 20

rgw_usage_log_flush_threshold

The number of dirty merged entries in the usage log before flushing synchronously.

Integer

1024

rgw_usage_log_tick_interval

Flush pending usage log data every n seconds.

Integer

30

rgw_intent_log_object_name

The logging format for the intent log object name. See manpage date for details about format specifiers.

Date

%Y-%m-%d-%i-%n

rgw_intent_log_object_name_utc

Whether the intent log object name includes a UTC time. If false, it uses the local time.

Boolean

false

rgw_data_log_window

The data log entries window in seconds.

Integer

30

rgw_data_log_changes_size

The number of in-memory entries to hold for the data changes log.

Integer

1000

rgw_data_log_num_shards

The number of shards (objects) on which to keep the data changes log.

Integer

128

rgw_data_log_obj_prefix

The object name prefix for the data log.

String

data_log

rgw_replica_log_obj_prefix

The object name prefix for the replica log.

String

replica log

rgw_md_log_max_shards

The maximum number of shards for the metadata log.

Integer

64

A.5. Keystone Settings

NameDescriptionTypeDefault

rgw_keystone_url

The URL for the Keystone server.

String

None

rgw_keystone_admin_token

The Keystone admin token (shared secret).

String

None

rgw_keystone_accepted_roles

The roles requires to serve requests.

String

Member, admin

rgw_keystone_token_cache_size

The maximum number of entries in each Keystone token cache.

Integer

10000

rgw_keystone_revocation_interval

The number of seconds between token revocation checks.

Integer

15 * 60

A.6. LDAP Settings

NameDescriptionTypeExample

rgw_ldap_uri

A space-separated list of LDAP servers in URI format.

String

ldaps://<ldap.your.domain>

rgw_ldap_searchdn

The LDAP search domain name, also known as base domain.

String

cn=users,cn=accounts,dc=example,dc=com

rgw_ldap_binddn

The gateway will bind with this LDAP entry (user match).

String

uid=admin,cn=users,dc=example,dc=com

rgw_ldap_secret

A file containing credentials for rgw_ldap_binddn

String

/etc/openldap/secret

rgw_ldap_dnattr

LDAP attribute containing Ceph object gateway user names (to form binddns).

String

uid

Legal Notice

Copyright © 2021 Red Hat, Inc.
The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.