Chapter 3. Object Storage service

OpenStack Object Storage (swift) stores its objects (data) in containers, which are similar to directories in a file system although they cannot be nested. Containers provide an easy way for users to store any kind of unstructured data. For example, objects might include photos, text files, or images. Stored objects are not compressed.

3.1. Object Storage rings

Object Storage uses a data structure called the Ring to distribute partition space across the cluster. This partition space is core to the data durability engine in the Object Storage service. It allows the Object Storage service to quickly and easily synchronize each partition across the cluster.

Rings contain information about Object Storage partitions and how partitions are distributed among the different nodes and disks. When any Object Storage component interacts with data, a quick lookup is performed locally in the ring to determine the possible partitions for each object.

The Object Storage service has three rings to store different types of data: one for account information, another for containers (to facilitate organizing objects under an account), and another for object replicas.

3.1.1. Rebalancing rings

When you change the Object Storage environment by adding or removing storage capacity, nodes, or disks, you must rebalance the rings. You can run openstack overcloud deploy to rebalance the rings, but this method redeploys the entire overcloud. This can be cumbersome, especially if you have a large overcloud. Alternatively, run the following command on the undercloud to rebalance the rings:

source ~/stackrc
ansible-playbook -i /usr/bin/tripleo-ansible-inventory
/usr/share/openstack-tripleo-common/playbooks/swift_ring_rebalance.yaml

3.1.2. Checking cluster health

The Object Storage service runs many processes in the background to ensure long-term data availability, durability, and persistence. For example:

  • Auditors constantly re-read database and object files and compare them using checksums to make sure there is no silent bit-rot. Any database or object file that no longer matches its checksum is quarantined and becomes unreadable on that node. The replicators then copy one of the other replicas to make the local copy available again.
  • Objects and files can disappear when you replace disks or nodes or when objects are quarantined. When this happens, replicators copy missing objects or database files to one of the other nodes.

The Object Storage service includes a tool called swift-recon that collects data from all nodes and checks the overall cluster health.

To use swift-recon, log in to one of the controller nodes and run the following command:

[root@overcloud-controller-2 ~]# sudo podman exec -it -u swift swift_object_server /usr/bin/swift-recon -arqlT --md5

======================================================================--> Starting reconnaissance on 3 hosts (object)
======================================================================[2018-12-14 14:55:47] Checking async pendings
[async_pending] - No hosts returned valid data.
======================================================================[2018-12-14 14:55:47] Checking on replication
[replication_failure] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 3
[replication_success] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 3
[replication_time] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 3
[replication_attempted] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 3
Oldest completion was 2018-12-14 14:55:39 (7 seconds ago) by 172.16.3.186:6000.
Most recent completion was 2018-12-14 14:55:42 (4 seconds ago) by 172.16.3.174:6000.
======================================================================[2018-12-14 14:55:47] Checking load averages
[5m_load_avg] low: 1, high: 2, avg: 2.1, total: 6, Failed: 0.0%, no_result: 0, reported: 3
[15m_load_avg] low: 2, high: 2, avg: 2.6, total: 7, Failed: 0.0%, no_result: 0, reported: 3
[1m_load_avg] low: 0, high: 0, avg: 0.8, total: 2, Failed: 0.0%, no_result: 0, reported: 3
======================================================================[2018-12-14 14:55:47] Checking ring md5sums
3/3 hosts matched, 0 error[s] while checking hosts.
======================================================================[2018-12-14 14:55:47] Checking swift.conf md5sum
3/3 hosts matched, 0 error[s] while checking hosts.
======================================================================[2018-12-14 14:55:47] Checking quarantine
[quarantined_objects] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 3
[quarantined_accounts] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 3
[quarantined_containers] low: 0, high: 0, avg: 0.0, total: 0, Failed: 0.0%, no_result: 0, reported: 3
======================================================================[2018-12-14 14:55:47] Checking time-sync
3/3 hosts matched, 0 error[s] while checking hosts.
======================================================================
Note

As an alternative, use the --all option to return additional output.

This command queries all servers on the ring for the following data:

  • Async pendings: If the cluster load is too high and processes can’t update database files fast enough, some updates will occur asynchronously. These numbers should decrease over time.
  • Replication metrics: Notice the replication timestamps; full replication passes should happen frequently and there should be few errors. An old entry, (for example, an entry with a timestamp from six months ago) indicates that replication on the node has not completed in the last six months.
  • Ring md5sums: This ensures that all ring files are consistent on all nodes.
  • swift.conf md5sums: This ensures that all ring files are consistent on all nodes.
  • Quarantined files: There should be no (or very few) quarantined files for all nodes.
  • Time-sync: All nodes must be synchronized.

3.1.3. Increasing ring partition power

The ring power determines the partition to which a resource (account, container, or object) is mapped. The partition is included in the path under which the resource is stored in a back end file system. Therefore, changing the partition power requires relocating resources to new paths in the back end file systems.

In a heavily populated cluster, a relocation process is time-consuming. To avoid downtime, relocate resources while the cluster is still operating. You must do this without temporarily losing access to data or compromising the performance of processes, such as replication and auditing. For assistance with increasing ring partition power, contact Red Hat support.

3.1.4. Creating custom rings

As technology advances and demands for storage capacity increase, creating custom rings is a way to update existing Object Storage clusters.

When you add new nodes to a cluster, their characteristics may differ from those of the original nodes. Without custom adjustments, the larger capacity of the new nodes may be underutilized. Or, if weights change in the rings, data dispersion can become uneven, which reduces safety.

Automation may not keep pace with future technology trends. For example, some older Object Storage clusters in use today originated before SSDs were available.

The ring builder helps manage Object Storage as clusters grow and technologies evolve. For assistance with creating custom rings, contact Red Hat support.

3.2. Object Storage service administration

The following procedures explain how to customize the Object Storage service.

3.2.1. Configuring fast-post

By default, the Object Storage service copies an object whole whenever any part of its metadata changes. You can prevent this by using the fast-post feature. The fast-post feature saves time when you change the content types of multiple large objects.

To enable the fast-post feature, disable the object_post_as_copy option on the Object Storage proxy service by doing the following:

  1. Edit swift_params.yaml:

    cat > swift_params.yaml << EOF
    parameter_defaults:
        ExtraConfig:
          swift::proxy::copy::object_post_as_copy: False
    EOF
  2. Include the parameter file when you deploy or update the overcloud:

    openstack overcloud deploy [... previous args ...] -e swift_params.yaml

3.2.2. Enabling at-rest encryption

By default, objects uploaded to Object Storage are kept unencrypted. Because of this, it is possible to access objects directly from the file system. This can present a security risk if disks are not properly erased before they are discarded.

You can use OpenStack Key Manager (barbican) to encrypt at-rest swift objects. For more information, see Encrypt at-rest swift objects.

3.2.3. Deploying a standalone Object Storage cluster

You can use the composable role concept to deploy a standalone Object Storage (openstack-swift) cluster with the bare minimum of additional services (for example Keystone, HAProxy). The Creating a roles_data File section has information on roles.

3.2.3.1. Creating the roles_data.yaml File

  1. Copy the roles_data.yaml from /usr/share/openstack-tripleo-heat-templates.
  2. Edit the new file.
  3. Remove unneeded controller roles, for example Aodh*, Ceilometer*, Ceph*, Cinder*, Glance*, Heat*, Ironic*, Manila*, Mistral*, Nova*, Octavia*, Swift*.
  4. Locate the ObjectStorage role within roles_data.yaml.
  5. Copy this role to a new role within that same file and name it ObjectProxy.
  6. Replace SwiftStorage with SwiftProxy in this role.

The example roles_data.yaml file below shows sample roles.

- name: Controller
  description: |
	Controller role that has all the controller services loaded and handles
	Database, Messaging and Network functions.
  CountDefault: 1
  tags:
	- primary
	- controller
  networks:
	- External
	- InternalApi
	- Storage
	- StorageMgmt
	- Tenant
  HostnameFormatDefault: '%stackname%-controller-%index%'
  ServicesDefault:
	- OS::TripleO::Services::AuditD
	- OS::TripleO::Services::CACerts
	- OS::TripleO::Services::CertmongerUser
	- OS::TripleO::Services::Clustercheck
	- OS::TripleO::Services::Docker
	- OS::TripleO::Services::Ec2Api
	- OS::TripleO::Services::Etcd
	- OS::TripleO::Services::HAproxy
	- OS::TripleO::Services::Keepalived
	- OS::TripleO::Services::Kernel
	- OS::TripleO::Services::Keystone
	- OS::TripleO::Services::Memcached
	- OS::TripleO::Services::MySQL
	- OS::TripleO::Services::MySQLClient
	- OS::TripleO::Services::Ntp
	- OS::TripleO::Services::Pacemaker
	- OS::TripleO::Services::RabbitMQ
	- OS::TripleO::Services::Securetty
	- OS::TripleO::Services::Snmp
	- OS::TripleO::Services::Sshd
	- OS::TripleO::Services::Timezone
	- OS::TripleO::Services::TripleoFirewall
	- OS::TripleO::Services::TripleoPackages
	- OS::TripleO::Services::Vpp

- name: ObjectStorage
  CountDefault: 1
  description: |
	Swift Object Storage node role
  networks:
	- InternalApi
	- Storage
	- StorageMgmt
  disable_upgrade_deployment: True
  ServicesDefault:
	- OS::TripleO::Services::AuditD
	- OS::TripleO::Services::CACerts
	- OS::TripleO::Services::CertmongerUser
	- OS::TripleO::Services::Collectd
	- OS::TripleO::Services::Docker
	- OS::TripleO::Services::FluentdClient
	- OS::TripleO::Services::Kernel
	- OS::TripleO::Services::MySQLClient
	- OS::TripleO::Services::Ntp
	- OS::TripleO::Services::Securetty
	- OS::TripleO::Services::SensuClient
	- OS::TripleO::Services::Snmp
	- OS::TripleO::Services::Sshd
	- OS::TripleO::Services::SwiftRingBuilder
	- OS::TripleO::Services::SwiftStorage
	- OS::TripleO::Services::Timezone
	- OS::TripleO::Services::TripleoFirewall
	- OS::TripleO::Services::TripleoPackages

- name: ObjectProxy
  CountDefault: 1
  description: |
	Swift Object proxy node role
  networks:
	- InternalApi
	- Storage
	- StorageMgmt
  disable_upgrade_deployment: True
  ServicesDefault:
	- OS::TripleO::Services::AuditD
	- OS::TripleO::Services::CACerts
	- OS::TripleO::Services::CertmongerUser
	- OS::TripleO::Services::Collectd
	- OS::TripleO::Services::Docker
	- OS::TripleO::Services::FluentdClient
	- OS::TripleO::Services::Kernel
	- OS::TripleO::Services::MySQLClient
	- OS::TripleO::Services::Ntp
	- OS::TripleO::Services::Securetty
	- OS::TripleO::Services::SensuClient
	- OS::TripleO::Services::Snmp
	- OS::TripleO::Services::Sshd
	- OS::TripleO::Services::SwiftRingBuilder
	- OS::TripleO::Services::SwiftProxy
	- OS::TripleO::Services::Timezone
	- OS::TripleO::Services::TripleoFirewall
	- OS::TripleO::Services::TripleoPackages

3.2.3.2. Deploying the New Roles

Deploy the overcloud with your regular openstack deploy command, including the new roles.

openstack overcloud deploy --templates -r roles_data.yaml -e [...]

3.2.4. Using external SAN disks

By default, when the Red Hat OpenStack Platform director deploys the Object Storage service (swift), Object Storage is configured and optimized to use independent local disks. This configuration ensures that the workload is distributed across all disks, which helps minimize performance impacts during node failure or other system issues.

In similar performance-impacting events, an environment that uses a single SAN can experience decreased performance across all LUNs. The Object Storage service cannot mitigate performance issues in an environment that uses SAN disks.

Therefore, Red Hat strongly recommends that you use additional local disks for Object Storage instead to meet performance and disk space requirements. For more information, see Object Storage in the Deployment Recommendations for Specific Red Hat OpenStack Platform Services guide.

Using an external SAN for Object Storage requires evaluation on a per-case basis. For more information, contact Red Hat Support.

Important

If you choose to use an external SAN for Object Storage, be aware of the following conditions:

  • The Object Storage service stores telemetry data and Image service (glance) images by default. Glance images require more disk space, but from a performance perspective, the impact of storing glance images impacts performance less than storing telemetry data. Storing and processing telemetry data requires increased performance. Red Hat does not provide support for issues related to performance that result from using an external SAN for Object Storage.
  • Red Hat does not provide support for issues that arise outside of the core Object Storage service offering. For support with high availability and performance, contact your storage vendor.
  • Red Hat does not test SAN solutions with the Object Storage service. For more information about compatibility, guidance, and support for third-party products, contact your storage vendor.
  • Red Hat recommends that you evaluate and test performance demands with your deployment. To confirm that your SAN deployment is tested, supported, and meets your performance requirements, contact your storage vendor.

3.2.4.1. SAN disk deployment configuration

This template is an example of how to use two devices (/dev/mapper/vdb and /dev/mapper/vdc) for Object Storage storage:

parameter_defaults:
  SwiftMountCheck: true
  SwiftUseLocalDir: false
  SwiftRawDisks: {"vdb": {"base_dir":"/dev/mapper/"}, "vdc": {"base_dir":"/dev/mapper/"}}

3.3. Install and configure storage nodes for Red Hat Enterprise Linux

To use Red Hat OpenStack Platform Object Storage service (swift) on external storage nodes, you must install and configure the storage nodes that operate the account, container, and object services processes. This configuration references two storage nodes, each of which contain two empty local block storage devices.

Note

The internal network for the Object Storage service is not authenticated. For security purposes, Red Hat recommends that you keep the storage nodes on a dedicated network or VLAN.

Note

The instructions use /dev/sdb and /dev/sdc as device names, but you can substitute the values for specific nodes in your environment.

3.3.1. Preparing storage devices

Before you install and configure the Object Storage service on the storage nodes, you must prepare the storage devices.

Note

Perform all of these steps on each storage node.

Procedure

  1. Install the supporting utility packages:

    # yum install xfsprogs rsync
  2. Format the /dev/sdb and /dev/sdc devices as XFS:

    # mkfs.xfs /dev/sdb
    # mkfs.xfs /dev/sdc
  3. Create the mount point directory structure:

    # mkdir -p /srv/node/sdb
    # mkdir -p /srv/node/sdc
  4. Edit the /etc/fstab file and add the following data to it:

    /dev/sdb /srv/node/sdb xfs defaults 0 2
    /dev/sdc /srv/node/sdc xfs defaults 0 2
  5. Mount the devices:

    # mount /srv/node/sdb
    # mount /srv/node/sdc
  6. Create or edit the /etc/rsyncd.conf file to contain the following data:

    uid = swift
    gid = swift
    log file = /var/log/rsyncd.log
    pid file = /var/run/rsyncd.pid
    address = MANAGEMENT_INTERFACE_IP_ADDRESS
    
    [account]
    max connections = 2
    path = /srv/node/
    read only = False
    lock file = /var/lock/account.lock
    
    [container]
    max connections = 2
    path = /srv/node/
    read only = False
    lock file = /var/lock/container.lock
    
    [object]
    max connections = 2
    path = /srv/node/
    read only = False
    lock file = /var/lock/object.lock

    Replace MANAGEMENT_INTERFACE_IP_ADDRESS with the IP address of the management network on the storage node.

  7. Start the rsyncd service and configure it to start when the system boots:

    # systemctl enable rsyncd.service
    # systemctl start rsyncd.service

3.3.2. Configuring components

Configure the account, container, and object storage servers.

Procedure

  1. Install the packages:

    # yum install openstack-swift-account openstack-swift-container \
      openstack-swift-object
  2. Edit the /etc/swift/account-server.conf file and complete the following actions:

    1. In the [DEFAULT] section, configure the bind IP address, bind port, user, configuration directory, and mount point directory:

      [DEFAULT]
      ...
      bind_ip = MANAGEMENT_INTERFACE_IP_ADDRESS
      bind_port = 6202
      user = swift
      swift_dir = /etc/swift
      devices = /srv/node
      mount_check = True

      Replace MANAGEMENT_INTERFACE_IP_ADDRESS with the IP address of the management network on the storage node.

    2. In the [pipeline:main] section, enable the healthcheck and recon modules:

      [pipeline:main]
      pipeline = healthcheck recon account-server
    3. In the [filter:recon] section, configure the recon cache directory:

      [filter:recon]
      use = egg:swift#recon
      ...
      recon_cache_path = /var/cache/swift
  3. Open the default firewall port for the account service:

    # firewall-cmd --permanent --add-port=6202/tcp
  4. Edit the /etc/swift/container-server.conf file and complete the following actions:

    1. In the [DEFAULT] section, configure the bind IP address, bind port, user, configuration directory, and mount point directory:

      [DEFAULT]
      ...
      bind_ip = MANAGEMENT_INTERFACE_IP_ADDRESS
      bind_port = 6201
      user = swift
      swift_dir = /etc/swift
      devices = /srv/node
      mount_check = True

      Replace MANAGEMENT_INTERFACE_IP_ADDRESS with the IP address of the management network on the storage node.

    2. In the [pipeline:main] section, enable the healthcheck and recon modules:

      [pipeline:main]
      pipeline = healthcheck recon container-server
    3. In the [filter:recon] section, configure the recon cache directory:

      [filter:recon]
      use = egg:swift#recon
      ...
      recon_cache_path = /var/cache/swift
  5. Open the default firewall port for the container service:

    # firewall-cmd --permanent --add-port=6201/tcp
  6. Edit the /etc/swift/object-server.conf file and complete the following actions:

    1. In the [DEFAULT] section, configure the bind IP address, bind port, user, configuration directory, and mount point directory:

      [DEFAULT]
      ...
      bind_ip = MANAGEMENT_INTERFACE_IP_ADDRESS
      bind_port = 6200
      user = swift
      swift_dir = /etc/swift
      devices = /srv/node
      mount_check = True

      Replace MANAGEMENT_INTERFACE_IP_ADDRESS with the IP address of the management network on the storage node.

    2. In the [pipeline:main] section, enable the healthcheck and recon modules:

      [pipeline:main]
      pipeline = healthcheck recon object-server
    3. In the [filter:recon] section, configure the recon_cache_path and the recon_lock_path directories:

      [filter:recon]
      use = egg:swift#recon
      ...
      recon_cache_path = /var/cache/swift
      recon_lock_path = /var/lock
  7. Open the default firewall port for the object service:

    # firewall-cmd --permanent --add-port=6200/tcp
  8. Ensure that the ownership of the mount point directory structure is correct:

    # chown -R swift:swift /srv/node
  9. Create the recon directory and ensure proper ownership of it:

    # mkdir -p /var/cache/swift
    # chown -R root:swift /var/cache/swift
    # chmod -R 775 /var/cache/swift

3.4. Basic container management

To help with organization, pseudo-folders are logical devices that can contain objects (and can be nested). For example, you might create an Images folder in which to store pictures and a Media folder in which to store videos.

You can create one or more containers in each project, and one or more objects or pseudo-folders in each container.

3.4.1. Creating a container

  1. In the dashboard, select Project > Object Store > Containers.
  2. Click Create Container.
  3. Specify the Container Name, and select one of the following in the Container Access field.

    TypeDescription

    Private

    Limits access to a user in the current project.

    Public

    Permits API access to anyone with the public URL. However, in the dashboard, project users cannot see public containers and data from other projects.

  4. Click Create Container.

New containers use the default storage policy. If you have multiple storage policies defined (for example, a default and another that enables erasure coding), you can configure a container to use a non-default storage policy through the command line. To do so, run:

# swift post -H "X-Storage-Policy:POLICY" CONTAINERNAME

Where:

  • POLICY is the name or alias of the policy you want the container to use.
  • CONTAINERNAME is the name of the container.

3.4.2. Creating a pseudo folder for a container

  1. In the dashboard, select Project > Object Store > Containers.
  2. Click the name of the container to which you want to add the pseudo-folder.
  3. Click Create Pseudo-folder.
  4. Specify the name in the Pseudo-folder Name field, and click Create.

3.4.3. Deleting a container

  1. In the dashboard, select Project > Object Store > Containers.
  2. Browse for the container in the Containers section, and ensure all objects have been deleted (see Section 3.4.6, “Deleting an object”).
  3. Select Delete Container in the container’s arrow menu.
  4. Click Delete Container to confirm the container’s removal.

3.4.4. Uploading an object

If you do not upload an actual file, the object is still created (as placeholder) and can later be used to upload the file.

  1. In the dashboard, select Project > Object Store > Containers.
  2. Click the name of the container in which the uploaded object will be placed; if a pseudo-folder already exists in the container, you can click its name.
  3. Browse for your file, and click Upload Object.
  4. Specify a name in the Object Name field:

    • Pseudo-folders can be specified in the name using a / character (for example, Images/myImage.jpg). If the specified folder does not already exist, it is created when the object is uploaded.
    • A name that is not unique to the location (that is, the object already exists) overwrites the object’s contents.
  5. Click Upload Object.

3.4.5. Copying an object

  1. In the dashboard, select Project > Object Store > Containers.
  2. Click the name of the object’s container or folder (to display the object).
  3. Click Upload Object.
  4. Browse for the file to be copied, and select Copy in its arrow menu.
  5. Specify the following:

    FieldDescription

    Destination container

    Target container for the new object.

    Path

    Pseudo-folder in the destination container; if the folder does not already exist, it is created.

    Destination object name

    New object’s name. If you use a name that is not unique to the location (that is, the object already exists), it overwrites the object’s previous contents.

  6. Click Copy Object.

3.4.6. Deleting an object

  1. In the dashboard, select Project > Object Store > Containers.
  2. Browse for the object, and select Delete Object in its arrow menu.
  3. Click Delete Object to confirm the object’s removal.