Red Hat Training

A Red Hat training course is available for Red Hat OpenStack Platform

Chapter 3. Object Storage

OpenStack Object Storage (openstack-swift) stores its objects (data) in containers, which are similar to directories in a file system although they cannot be nested. Containers provide an easy way for users to store any kind of unstructured data; for example, objects might include photos, text files, or images. Stored objects are not compressed.

3.1. Object Storage Service Administration

The following procedures explain how to further customize the Object Storage service to suit your needs.

3.1.1. Configure Erasure Coding

Erasure coding (EC) is a method of data protection in which the data is broken into fragments, expanded and encoded with redundant data pieces and stored across a set of different locations or storage media. It uses a smaller volume of storage to attain the required durability than traditional replication. When compared to replication factor of 3, savings of 50% may be attained with careful deployment. However, depending on the workload, erasure coding may incur a performance penalty.

Erasure coding is supported for Object Storage service as a Storage Policy. A Storage Policy allows segmenting the cluster for various purposes through the creation of multiple object rings. Red Hat recommends you split off devices used by erasure coding and replication Storage Policies. This way behavior of the cluster is easier to analyze.

The direction you choose depends on why the erasure coding policy is being deployed. Some of the main considerations are:

  • Layout of existing infrastructure.
  • Cost of adding dedicated erasure coding nodes (or just dedicated erasure coding devices).
  • Intended usage model(s).

3.1.1.1. Configure an Erasure Coding Policy

To use erasure coding, first define an additional policy for it in /etc/swift/swift.conf. The Sample Erasure Coding Policy below shows a typical example:

Sample Erasure Coding Policy

[storage-policy:1]
default = no
name = ec104
alias = myec,erasure_coding
policy_type = erasure_coding
ec_type = jerasure_rs_vand
ec_num_data_fragments = 10
ec_num_parity_fragments = 4
ec_object_segment_size = 1048576

Note

Object Storage policy headers (for example, [storage-policy:1]) include an index number — in this case, 1. The Object Storage index count begins at 0, and therefore the Sample Erasure Coding Policy above assumes that another policy with the policy index 0 already exists. For example: 

[storage-policy:0]
name = default
default = yes

After defining the erasure coding policy, you will need to create and configure its associated object storage ring. See Section 3.1.1.2, “Configure an Object Storage Ring” for instructions.

The following table explains the different parameters used in Sample Erasure Coding Policy:

Table 3.1. Storage Policy Parameters

TermDescription

default

Sets whether this policy is the default one or not (yes/no). This is used if there are multiple policies defined in /etc/swift/swift.conf.

name

This is a standard storage policy parameter.

alias

Comma-delimited list of other names that the policy is known as.

policy_type

Set this to erasure_coding to indicate that this is an erasure coding policy.

ec_type

This specifies the erasure coding scheme that is to be used. See Table 3.2, “Supported Erasure Coding Schemes” for a list of supported values.

ec_num_data_fragments

The total number of fragments that will be comprised of data.

ec_num_parity_fragments

The total number of fragments that will be comprised of parity.

ec_object_segment_size

The amount of data that will be buffered up before feeding a segment into the encoder/decoder. The default value is 1048576.

Table 3.2. Supported Erasure Coding Schemes

SchemeDescription/Reference

liberasurecode_rs_vand

Vandermonde Reed-Solomon encoding, software-only backend implemented by liberasurecode

jerasure_rs_vand

Vandermonde Reed-Solomon encoding, based on Jerasure

jerasure_rs_cauchy

Cauchy Reed-Solomon encoding (Jerasure variant), based on Jerasure

flat_xor_hd_3, flat_xor_hd_4

Flat-XOR based HD combination codes, liberasurecode

isa_l_rs_vand

Intel Storage Acceleration Library (ISA-L) - SIMD accelerated Erasure Coding backends

When the Object Storage service encodes an object, it breaks it into fragments. It is important during configuration to know how many of those fragments are data and how many are parity. So in the Sample Erasure Coding Policy, an object will be broken into 14 different fragments, 10 of them will be made up of actual object data and 4 of them will be made of parity data (calculations depending on ec_type). With such a configuration, the system can sustain 4 disk failures before the data is lost. Other commonly used configurations are 4+2 (with 4 data fragments and 2 parity fragments) or 8+3 (with 8 data fragments and 3 parity fragments).

Note

This procedure involves configuring a service outside of the director. As such, you may need to repeat it the next time you re-deploy or update the overcloud.

Important

Once you have deployed a policy and have created objects with that policy, these configurations options cannot be changed. In case a change in the configuration is desired, you must create a new policy and migrate the data to a new container. However, once defined, policy indices cannot be discarded. If policies are to be retired, they may be retired, but not be removed. There is essentially no performance penalty for having old policies around, but a minor administrative overhead.

3.1.1.2. Configure an Object Storage Ring

Object Storage uses a data structure called the Ring to distribute a partition space across the cluster. This partition space is core to the data durability engine in Object Storage service. It allows the Object Storage service to quickly and easily synchronize each partition across the cluster. When any component in Swift needs to interact with data, a quick lookup is done locally in the Ring to determine the possible partitions for each object.

The Object Storage service already has three rings to store different types of data. There is one for account information, another for containers (so that it’s convenient to organize objects under an account) and another for the object replicas. To support erasure codes, there will be an additional ring that is created to store erasure code objects.

To create a typical replication ring, for example, you can use the following command: 

# swift-ring-builder object-1.builder create 10 3 1

Where 3 is the number of replicas.

In order to create an erasure coding object ring, you need to use the number of fragments in place of the number of replicas, for example: 

# swift-ring-builder object-1.builder create 10 14 1

Where 14 is for a 10+4 configuration with 10 data fragments and 4 parity fragments.

Consider the performance impacts when deciding which devices to use in the erasure coding policy’s object ring. We recommend that you run some performance benchmarking in a test environment for the configuration before deployment. After you have configured your erasure coding policy in the /etc/swift/swift.conf and created your object ring, your application is ready to start using erasure coding by creating a container with the specified policy name and interacting as usual.

3.1.1.3. Using Erasure Coding

After defining a new Erasure Coding policy and configuring its object storage ring, you can use it when creating a new container for the first time. If you have multiple policies defined, each container will be created with the default storage policy assigned.

To use a non-default storage policy with a new container, you need to submit a special metadata header when creating the container. For example, to use the Sample Erasure Coding Policy policy defined in Section 3.1.1.1, “Configure an Erasure Coding Policy” on a new container (CONTAINERNAME), run: 

# swift post -H "X-Storage-Policy:ec104" CONTAINERNAME

3.1.2. Configure Fast-POST

By default, the Object Storage service will copy an object whole whenever any part of its metadata changes. You can prevent this using the fast-post feature. This is useful if you want to save time in changing the content types of multiple large objects.

To enable the fast-post feature, disable the object_post_as_copy option on the Object Storage proxy service. To do this:

  1. Log in to the nodes hosting the Object Storage proxy service.
  2. Open /etc/swift/proxy-server.conf.

  3. Under the [app:proxy-server] section, set object_post_as_copy to false: 

    [app:proxy-server]
use = egg:swift#proxy
set log_name = proxy-server
…
object_post_as_copy = false
…

  4. Restart the Object Storage proxy services on the node: 

    # systemctl restart openstack-swift-proxy.service
# systemctl restart openstack-swift-object-expirer.service

    If any storage policies other than policy-0 are listed in /etc/swift/swift.conf, run the following as well: 

    # systemctl restart openstack-swift-container-reconciler.service
Note

This procedure involves configuring a service outside of the director. As such, you may need to repeat it the next time you re-deploy or update the overcloud.

Note

On a typical overcloud deployment, the Object Storage service is installed on the Controller nodes.

3.1.3. Set Object Storage as a Back End for the Image Service

The OpenStack Image service, by default, saves images and instance snapshots to the local filesystem in /var/lib/glance/images/. Alternatively, you can configure the Image service to save images and snapshots to the Object Storage service (when available).

To do so, perform the following procedure:

  1. Log into the node running the Image service (the controller node also running Identity) as root and source your OpenStack credentials (this is typically a file named openrc). 

    # source ~/openrc

  2. Verify that the Image service is part of the tenant services with role admin. 

    # openstack user role list --project services glance

    One of the roles returned should be admin.

  3. Open the /etc/glance/glance.conf file and comment out the following lines: 

    ##### DEFAULT OPTIONS #####
#default_store = file
#filesystem_store_datadir = /var/lib/glance/images/

  4. In the same file, add the following lines to the DEFAULT OPTIONS section. 

    default_store = swift
swift_store_auth_address = http://KEYSTONEIP:35357/v2.0/
swift_store_user = service:glance
swift_store_key = ADMINPW
swift_store_create_container_on_put = True

    Where:

    • KEYSTONEIP is the IP address of the Identity service, and
    • ADMINPW is the value of admin password attribute in the /etc/glance/glance-api.conf file.

  5. Apply the changes by restarting the Image service: 

    # systemctl restart openstack-glance-api
# systemctl restart openstack-glance-registry

From this point onwards, images uploaded to the Image service (whether through the Dashboard or glance) should now be saved to an Object Storage container named glance. This container exists in the service account.

To verify whether newly-created images are saved to the Image service, run: 

# ls /var/lib/glance/images

Once the Dashboard or the glance image-list reports the image is active, you can verify whether it is in Object Storage by running the following command: 

# swift --os-auth-url http://KEYSTONEIP:5000/v2.0 --os-tenant-name service --os-username glance --os-password ADMINPW list glance
Note

This procedure involves configuring a service outside of the director. As such, you may need to repeat it the next time you re-deploy or update the overcloud.

3.1.4. Enable At-Rest Encryption

Note

At-rest encryption is marked as Technology Preview for this release. For more information on the support scope for features marked as technology previews, see https://access.redhat.com/support/offerings/techpreview/

By default, objects uploaded to Object Storage are kept unencrypted. Because of this, it is possible to access objects directly from the file system. This can present a security risk if disks are not properly erased before they are discarded.

Even with encryption, it is important to use a different method to dispose of the root disk of the proxy, as access to both this and the storage node disks may allow an attacker to decrypt the data.

The following changes need to be made on the overcloud.

To enable encryption of swift object stores:

  1. In /etc/swift/proxy-server.conf, add these lines, replacing the value of encryption_root_secret with your encryption key: 

    [pipeline:main]
pipeline = catch_errors healthcheck proxy-logging cache ratelimit bulk tempurl formpost authtoken keystone staticweb keymaster encryption proxy-logging proxy-server

[filter:keymaster]
use = egg:swift#keymaster
encryption_root_secret =

[filter:encryption]
use = egg:swift#encryption
# disable_encryption = False

  2. Restart the proxy service for the changes to come into effect: 

    # systemctl restart openstack-swift-proxy.service

To verify that objects are now being encrypted:

  1. Add a new object to test: 

    # swift upload container1 testobj

  2. Verify the data is encrypted by viewing it with cat. For example: 

    # cat /srv/node/sdb/objects/603/424/35c4ea21cc96663792465462570b2424/1477399821.58002.data
Note

Changing the encryption key and then attempting to download previously uploaded files will cause an error and may result in data loss.

3.2. Basic Container Management

To help with organization, pseudo-folders are logical devices that can contain objects (and can be nested). For example, you might create an Images folder in which to store pictures and a Media folder in which to store videos.

You can create one or more containers in each project, and one or more objects or pseudo-folders in each container.

3.2.1. Create a Container

  1. In the dashboard, select Project > Object Store > Containers.
  2. Click Create Container.

  3. Specify the Container Name, and select one of the following in the Container Access field.

    TypeDescription

    Private

    Limits access to a user in the current project.

    Public

    Permits API access to anyone with the public URL. However, in the dashboard, project users cannot see public containers and data from other projects.

  4. Click Create Container.

New containers use the default storage policy. If you have multiple storage policies defined (for example, a default one and another that enables erasure coding), you can configure a container to use a non-default storage policy through the command line. To do so, run: 

# swift post -H "X-Storage-Policy:POLICY" CONTAINERNAME

Where:

3.2.2. Create a Pseudo Folder for a Container

  1. In the dashboard, select Project > Object Store > Containers.
  2. Click the name of the container to which you want to add the pseudo-folder.
  3. Click Create Pseudo-folder.
  4. Specify the name in the Pseudo-folder Name field, and click Create.

3.2.3. Delete a Container

  1. In the dashboard, select Project > Object Store > Containers.
  2. Browse for the container in the Containers section, and ensure all objects have been deleted (see Section 3.2.6, “Delete an Object”).
  3. Select Delete Container in the container’s arrow menu.
  4. Click Delete Container to confirm the container’s removal.

3.2.4. Upload an Object

If you do not upload an actual file, the object is still created (as placeholder) and can later be used to upload the file.

  1. In the dashboard, select Project > Object Store > Containers.
  2. Click the name of the container in which the uploaded object will be placed; if a pseudo-folder already exists in the container, you can click its name.
  3. Browse for your file, and click Upload Object.

  4. Specify a name in the Object Name field:

    • Pseudo-folders can be specified in the name using a / character (for example, Images/myImage.jpg). If the specified folder does not already exist, it is created when the object is uploaded.
    • A name that is not unique to the location (that is, the object already exists) overwrites the object’s contents.
  5. Click Upload Object.

3.2.5. Copy an Object

  1. In the dashboard, select Project > Object Store > Containers.
  2. Click the name of the object’s container or folder (to display the object).
  3. Click Upload Object.
  4. Browse for the file to be copied, and select Copy in its arrow menu.

  5. Specify the following:

    FieldDescription

    Destination container

    Target container for the new object.

    Path

    Pseudo-folder in the destination container; if the folder does not already exist, it is created.

    Destination object name

    New object’s name. If you use a name that is not unique to the location (that is, the object already exists), it overwrites the object’s previous contents.

  6. Click Copy Object.

3.2.6. Delete an Object

  1. In the dashboard, select Project > Object Store > Containers.
  2. Browse for the object, and select Delete Object in its arrow menu.
  3. Click Delete Object to confirm the object’s removal.