Chapter 2. Configuration

Procedure

1. Open for editing the Ceph configuration file, by default /etc/ceph/ceph.conf, and enable the Vault as the encryption key store:

   rgw_crypt_s3_kms_backend = vault

2. Under the [client.radosgw.INSTANCE_NAME] section, choose a Vault authentication method, either Token or the Vault agent.

   1. If using Token, then add the following lines:

      rgw_crypt_vault_auth = token
      rgw_crypt_vault_token_file = /etc/ceph/vault.token
      rgw_crypt_vault_addr = http://VAULT_SERVER:8200

   2. If using the Vault agent, then add the following lines:

      rgw_crypt_vault_auth = agent
      rgw_crypt_vault_addr = http://VAULT_SERVER:8100

3. Under the [client.radosgw.INSTANCE_NAME] section, choose a Vault secret engine, either Key/Value or Transit.

   1. If using Key/Value, then add the following line:

      rgw_crypt_vault_secret_engine = kv

   2. If using Transit, then add the following line:

      rgw_crypt_vault_secret_engine = transit

4. Optionally, Under the [client.radosgw.INSTANCE_NAME] section, you can set the Vault namespace where the encryption keys will be retrieved:

   rgw_crypt_vault_namespace = NAME_OF_THE_NAMESPACE

5. Restrict where the Ceph Object Gateway retrieves the encryption keys from the Vault by setting a path prefix:

   Example

   rgw_crypt_vault_prefix = /v1/secret/data

   1. For exportable Transit keys, set the prefix path as follows:

      rgw_crypt_vault_prefix = /v1/transit/export/encryption-key

      Assuming the domain name of the Vault server is vault-server, the Ceph Object Gateway will fetch encrypted transit keys from the following URL:

      Example

      http://vault-server:8200/v1/transit/export/encryption-key

6. Save the changes to the Ceph configuration file.

Procedure

1. Enable the Key/Value version 2 secret engine:

   [root@vault ~]# vault secrets enable kv-v2

2. Create a new key:

   Syntax

   vault kv put secret/PROJECT_NAME/BUCKET_NAME key=$(openssl rand -base64 32)

   Example

   [root@vault ~]# vault kv put secret/myproject/mybucketkey key=$(openssl rand -base64 32)
   
   ====== Metadata ======
   Key              Value
   ---              -----
   created_time     2020-02-21T17:01:09.095824999Z
   deletion_time    n/a
   destroyed        false
   version          1

Procedure

1. Enable the Transit secret engine:

   [root@vault ~]# vault secrets enable transit

2. Create a new exportable key:

   Syntax

   vault write -f transit/keys/BUCKET_NAME exportable=true

   Example

   [root@vault ~]# vault write -f transit/keys/mybucketkey exportable=true

   Note

   By default the above command creates a aes256-gcm96 type key.

3. Verify the creation of the key:

   Syntax

   vault read transit/export/encryption-key/BUCKET_NAME/VERSION_NUMBER

   Example

   [root@vault ~]# vault read transit/export/encryption-key/mybucketkey/1
   
   Key     Value
   ---     -----
   keys    map[1:-gbTI9lNpqv/V/2lDcmH2Nq1xKn6FPDWarCmFM2aNsQ=]
   name    mybucketkey
   type    aes256-gcm96

   Note

   Providing the full key path, including the key version is required.

Procedure

1. Upload an object using the AWS command-line client:

   Example

   [user@client ~]$ aws --endpoint=http://radosgw:8000 s3 cp plaintext.txt s3://mybucket/encrypted.txt --sse=aws:kms --sse-kms-key-id myproject/mybucketkey

   Note

   The key fetching URL used in the example is: http://vault-server:8200/v1/secret/data/myproject/mybucketkey

Procedure

1. Create an S3 user:

   radosgw-admin user create --uid=name --display-name="First User"

   Replace name with the name of the S3 user, for example:

   [root@master-zone]# radosgw-admin user create --uid="testuser" --display-name="First User"
   {
       "user_id": "testuser",
       "display_name": "First User",
       "email": "",
       "suspended": 0,
       "max_buckets": 1000,
       "auid": 0,
       "subusers": [],
       "keys": [
           {
               "user": "testuser",
               "access_key": "CEP28KDIQXBKU4M15PDC",
               "secret_key": "MARoio8HFc8JxhEilES3dKFVj8tV3NOOYymihTLO"
           }
       ],
       "swift_keys": [],
       "caps": [],
       "op_mask": "read, write, delete",
       "default_placement": "",
       "placement_tags": [],
       "bucket_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "user_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "temp_url_keys": [],
       "type": "rgw"
   }

2. Verify the output to ensure that the values of access_key and secret_key do not include a JSON escape character (\). These values are needed for access validation, but certain clients cannot handle if the values include JSON escape character. To fix this problem, perform one of the following actions:

   • Remove the JSON escape character.
   • Encapsulate the string in quotes.
   • Regenerate the key and ensure that is does not include a JSON escape character.
   • Specify the key and secret manually.

   Do not remove the forward slash / because it is a valid character.

Procedure

1. Create the Swift user:

   Syntax

   radosgw-admin subuser create --uid=NAME --subuser=NAME:swift --access=full

   Replace NAME with the Swift user name, for example:

   Example

   [root@rgw]# radosgw-admin subuser create --uid=testuser --subuser=testuser:swift --access=full
   {
       "user_id": "testuser",
       "display_name": "First User",
       "email": "",
       "suspended": 0,
       "max_buckets": 1000,
       "auid": 0,
       "subusers": [
           {
               "id": "testuser:swift",
               "permissions": "full-control"
           }
       ],
       "keys": [
           {
               "user": "testuser",
               "access_key": "O8JDE41XMI74O185EHKD",
               "secret_key": "i4Au2yxG5wtr1JK01mI8kjJPM93HNAoVWOSTdJd6"
           }
       ],
       "swift_keys": [
           {
               "user": "testuser:swift",
               "secret_key": "13TLtdEW7bCqgttQgPzxFxziu0AgabtOc6vM8DLA"
           }
       ],
       "caps": [],
       "op_mask": "read, write, delete",
       "default_placement": "",
       "placement_tags": [],
       "bucket_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "user_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "temp_url_keys": [],
       "type": "rgw"
   }

2. Create the secret key:

   Syntax

   radosgw-admin key create --subuser=NAME:swift --key-type=swift --gen-secret

   Replace NAME with the Swift user name, for example:

   Example

   [root@rgw]# radosgw-admin key create --subuser=testuser:swift --key-type=swift --gen-secret
   {
       "user_id": "testuser",
       "display_name": "First User",
       "email": "",
       "suspended": 0,
       "max_buckets": 1000,
       "auid": 0,
       "subusers": [
           {
               "id": "testuser:swift",
               "permissions": "full-control"
           }
       ],
       "keys": [
           {
               "user": "testuser",
               "access_key": "O8JDE41XMI74O185EHKD",
               "secret_key": "i4Au2yxG5wtr1JK01mI8kjJPM93HNAoVWOSTdJd6"
           }
       ],
       "swift_keys": [
           {
               "user": "testuser:swift",
               "secret_key": "a4ioT4jEP653CDcdU8p4OuhruwABBRZmyNUbnSSt"
           }
       ],
       "caps": [],
       "op_mask": "read, write, delete",
       "default_placement": "",
       "placement_tags": [],
       "bucket_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "user_quota": {
           "enabled": false,
           "check_on_raw": false,
           "max_size": -1,
           "max_size_kb": 0,
           "max_objects": -1
       },
       "temp_url_keys": [],
       "type": "rgw"
   }

Procedure

1. Enable the common repository for Red Hat Enterprise Linux 7 and the High Availability repository for Red Hat Enterprise Linux 8:

   Red Hat Enterprise Linux 7

   # subscription-manager repos --enable=rhel-7-server-rh-common-rpms

   Red Hat Enterprise Linux 8

   # subscription-manager repos --enable=rhel-8-for-x86_64-highavailability-rpms

2. Install the python-boto package.

   Red Hat Enterprise Linux 7

   # yum install python-boto

   Red Hat Enterprise Linux 8

   # dnf install python3-boto3

3. Create the Python script:

   vi s3test.py

4. Add the following contents to the file:

   Red Hat Enterprise Linux 7

   import boto
   import boto.s3.connection
   
   access_key = 'ACCESS'
   secret_key = 'SECRET'
   
   boto.config.add_section('s3')
   
   conn = boto.connect_s3(
           aws_access_key_id = access_key,
           aws_secret_access_key = secret_key,
           host = 's3.ZONE.hostname',
           port = PORT,
           is_secure=False,
           calling_format = boto.s3.connection.OrdinaryCallingFormat(),
           )
   
   bucket = conn.create_bucket('my-new-bucket')
   for bucket in conn.get_all_buckets():
   	print "{name}\t{created}".format(
   		name = bucket.name,
   		created = bucket.creation_date,
   )

   Red Hat Enterprise Linux 8

   import boto3
   
   endpoint = "" # enter the endpoint URL along with the port "http://URL:_PORT_"
   
   access_key = 'ACCESS'
   secret_key = 'SECRET'
   
   s3 = boto3.client(
           's3',
           endpoint_url=endpoint,
           aws_access_key_id=access_key,
           aws_secret_access_key=secret_key
           )
   
   s3.create_bucket(Bucket='my-new-bucket')
   
   response = s3.list_buckets()
   for bucket in response['Buckets']:
       print("{name}\t{created}".format(
   		name = bucket['Name'],
   		created = bucket['CreationDate']
   ))

   1. Replace ZONE with the zone name of the host where you have configured the gateway service. That is, the gateway host. Ensure that the host`setting resolves with DNS. Replace `PORT with the port number of the gateway.
   2. Replace ACCESS and SECRET with the access_key and secret_key values from the Create an S3 User section in the Red Hat Ceph Storage Object Gateway Configuration and Administration Guide.

5. Run the script:

   Red Hat Enterprise Linux 7

   python s3test.py

   Red Hat Enterprise Linux 8

   python3 s3test.py

   Example output:

   my-new-bucket 2021-08-16T17:09:10.000Z

1. Install Red Hat Enterprise Linux 7.

2. Register the nodes.

   [root@haproxy]# subscription-manager register

3. Enable the RHEL server repository.

   [root@haproxy]# subscription-manager repos --enable=rhel-7-server-rpms

4. Update the server.

   [root@haproxy]# yum update -y

5. Install admin tools (e.g., wget, vim, etc.) as needed.

6. Open port 80.

   [root@haproxy]# firewall-cmd --zone=public --add-port 80/tcp --permanent
   [root@haproxy]# firewall-cmd --reload

7. For HTTPS, open port 443.

   [root@haproxy]# firewall-cmd --zone=public --add-port 443/tcp --permanent
   [root@haproxy]# firewall-cmd --reload

8. Connect to the required port.

   [root@haproxy]# semanage port -m -t http_cache_port_t -p tcp 8081

Procedure

1. Install keepalived:

   [root@haproxy]# yum install -y keepalived

2. Configure keepalived on both HAProxy nodes:

   [root@haproxy]# vim /etc/keepalived/keepalived.conf

   In the configuration file, there is a script to check the haproxy processes:

   vrrp_script chk_haproxy {
     script "killall -0 haproxy" # check the haproxy process
     interval 2 # every 2 seconds
     weight 2 # add 2 points if OK
   }

   Next, the instance on the master and backup load balancers uses eno1 as the network interface. It also assigns a virtual IP address, that is, 192.168.1.20.

   Master load balancer node

   vrrp_instance RGW {
       state MASTER # might not be necessary. This is on the Master LB node.
       @main interface eno1
       priority 100
       advert_int 1
       interface eno1
       virtual_router_id 50
       @main unicast_src_ip 10.8.128.43 80
       unicast_peer {
              10.8.128.53
              }
       authentication {
           auth_type PASS
           auth_pass 1111
       }
       virtual_ipaddress {
           192.168.1.20
       }
       track_script {
         chk_haproxy
       }
   }
   virtual_server 192.168.1.20 80 eno1 { #populate correct interface
       delay_loop 6
       lb_algo wlc
       lb_kind dr
       persistence_timeout 600
       protocol TCP
       real_server 10.8.128.43 80 { # ip address of rgw2 on physical interface, haproxy listens here, rgw listens to localhost:8080 or similar
           weight 100
           TCP_CHECK { # perhaps change these to a HTTP/SSL GET?
               connect_timeout 3
           }
       }
       real_server 10.8.128.53 80 { # ip address of rgw3 on physical interface, haproxy listens here, rgw listens to localhost:8080 or similar
           weight 100
           TCP_CHECK { # perhaps change these to a HTTP/SSL GET?
               connect_timeout 3
           }
       }
   }

   Backup load balancer node

   vrrp_instance RGW {
       state BACKUP # might not be necessary?
       priority 99
       advert_int 1
       interface eno1
       virtual_router_id 50
       unicast_src_ip 10.8.128.53 80
       unicast_peer {
              10.8.128.43
              }
       authentication {
           auth_type PASS
           auth_pass 1111
       }
       virtual_ipaddress {
           192.168.1.20
       }
       track_script {
         chk_haproxy
       }
   }
   virtual_server 192.168.1.20 80 eno1 { #populate correct interface
       delay_loop 6
       lb_algo wlc
       lb_kind dr
       persistence_timeout 600
       protocol TCP
       real_server 10.8.128.43 80 { # ip address of rgw2 on physical interface, haproxy listens here, rgw listens to localhost:8080 or similar
           weight 100
           TCP_CHECK { # perhaps change these to a HTTP/SSL GET?
               connect_timeout 3
           }
       }
       real_server 10.8.128.53 80 { # ip address of rgw3 on physical interface, haproxy listens here, rgw listens to localhost:8080 or similar
           weight 100
           TCP_CHECK { # perhaps change these to a HTTP/SSL GET?
               connect_timeout 3
           }
       }
   }

3. Enable and start the keepalived service:

   [root@haproxy]# systemctl enable keepalived
   [root@haproxy]# systemctl start keepalived

1. Install haproxy.

   [root@haproxy]# yum install haproxy

2. Configure haproxy for SELinux and HTTP.

   [root@haproxy]# vim /etc/firewalld/services/haproxy-http.xml

   Add the following lines:

   <?xml version="1.0" encoding="utf-8"?>
   <service>
   <short>HAProxy-HTTP</short>
   <description>HAProxy load-balancer</description>
   <port protocol="tcp" port="80"/>
   </service>

   As root, assign the correct SELinux context and file permissions to the haproxy-http.xml file.

   [root@haproxy]# cd /etc/firewalld/services
   [root@haproxy]# restorecon haproxy-http.xml
   [root@haproxy]# chmod 640 haproxy-http.xml

3. If you intend to use HTTPS, configure haproxy for SELinux and HTTPS.

   [root@haproxy]# vim /etc/firewalld/services/haproxy-https.xml

   Add the following lines:

   <?xml version="1.0" encoding="utf-8"?>
   <service>
   <short>HAProxy-HTTPS</short>
   <description>HAProxy load-balancer</description>
   <port protocol="tcp" port="443"/>
   </service>

   As root, assign the correct SELinux context and file permissions to the haproxy-https.xml file.

   # cd /etc/firewalld/services
   # restorecon haproxy-https.xml
   # chmod 640 haproxy-https.xml

4. If you intend to use HTTPS, generate keys for SSL. If you do not have a certificate, you may use a self-signed certificate. To generate a key, see to Generating a New Key and Certificate section in the System Administrator’s Guide for Red Hat Enterprise Linux 7.

   Finally, put the certificate and key into a PEM file.

   [root@haproxy]# cat example.com.crt example.com.key > example.com.pem
   [root@haproxy]# cp example.com.pem /etc/ssl/private/

5. Configure haproxy.

   [root@haproxy]# vim /etc/haproxy/haproxy.cfg

   The global and defaults may remain unchanged. After the defaults section, you will need to configure frontend and backend sections. For example:

   frontend http_web
       bind *:80
       mode http
       default_backend rgw
   
   frontend rgw­-https
     bind *:443 ssl crt /etc/ssl/private/example.com.pem
     default_backend rgw
   
   backend rgw
       balance roundrobin
       mode http
       server  rgw1 10.0.0.71:80 check
       server  rgw2 10.0.0.80:80 check

   For a detailed discussion of HAProxy configuration, refer to HAProxy Configuration.

6. Enable/start haproxy

   [root@haproxy]# systemctl enable haproxy
   [root@haproxy]# systemctl start haproxy

1. S3 static web hosting uses Ceph Object Gateway instances that are separate and distinct from instances used for standard S3/Swift API use cases.
2. Gateway instances hosting S3 static web sites should have separate, non-overlapping domain names from the standard S3/Swift API gateway instances.
3. Gateway instances hosting S3 static web sites should use separate public-facing IP addresses from the standard S3/Swift API gateway instances.
4. Gateway instances hosting S3 static web sites load balance, and if necessary terminate SSL, using HAProxy/keepalived.

1. Create an S3 bucket. The bucket name MAY be the same as the website’s domain name. For example, mysite.com may have a bucket name of mysite.com. This is required for AWS, but it is NOT required for Ceph. See DNS Settings for details.
2. Upload the static website content to the bucket. Contents may include HTML, CSS, client-side JavaScript, images, audio/video content and other downloadable files. A website MUST have an index.html file and MAY have error.html file.
3. Verify the website’s contents. At this point, only the creator of the bucket will have access to the contents.
4. Set permissions on the files so that they are publicly readable.