Red Hat Storage 2.0

2.0 Release Notes

Release Notes for Red Hat Storage 2.0

Divya Muntimadugu

Red Hat Engineering Content Services

Anjana Suparna Sriram

Red Hat Engineering Content Services

Legal Notice

Copyright © 2012 Red Hat Inc.
This document is licensed by Red Hat under the Creative Commons Attribution-ShareAlike 3.0 Unported License. If you distribute this document, or a modified version of it, you must provide attribution to Red Hat, Inc. and provide a link to the original. If the document is modified, all Red Hat trademarks must be removed.
Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.
Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, MetaMatrix, Fedora, the Infinity Logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.
Linux® is the registered trademark of Linus Torvalds in the United States and other countries.
Java® is a registered trademark of Oracle and/or its affiliates.
XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.
MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.
Node.js® is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
The OpenStack® Word Mark and OpenStack Logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.
All other trademarks are the property of their respective owners.


This Release Notes introduces Red Hat Storage and provides information including key features, installing/launching and managing the software.

1. Introducing Red Hat Storage

Red Hat Storage is software only, scale-out storage that provides flexible and affordable unstructured data storage for the enterprise. Red Hat Storage 2.0 provides new opportunities to unify data storage and infrastructure, increase performance, and improve availability and manageability in order to meet a broader set of an organization’s storage challenges and needs.
GlusterFS, a key building block of Red Hat Storage, is based on a stackable user space design and can deliver exceptional performance for diverse workloads. GlusterFS aggregates various storage servers over network interconnects into one large parallel network file system. The POSIX compatible GlusterFS servers, which use XFS file system format to store data on disks, can be accessed using industry standard access protocols including NFS and CIFS.
Red Hat Storage can be deployed in the private cloud or datacenter using Red Hat Storage Server for On-premise. Red Hat Storage can be installed on commodity servers and storage hardware resulting in a powerful, massively scalable, and highly available NAS environment. Additionally, Red Hat Storage can be deployed in the public cloud using Red Hat Storage Server for Public Cloud, for example, within the Amazon Web Services (AWS) cloud. It delivers all the features and functionality possible in a private cloud or datacenter to the public cloud by providing massively scalable and high available NAS in the cloud.
Red Hat Storage Server for On-Premise
Red Hat Storage Server for On-Premise enables enterprises to treat physical storage as a virtualized, scalable, and centrally managed pool of storage by using commodity server and storage hardware.
Red Hat Storage Server for Public Cloud
Red Hat Storage Server for Public Cloud packages GlusterFS as an Amazon Machine Image (AMI) for deploying scalable NAS in the AWS public cloud. This powerful storage server provides a highly available, scalable, virtualized, and centrally managed pool of storage for Amazon users.

2. What is New in this Release?

This section describes the key features available in Red Hat Storage. The following is a list of feature highlights of this new version of the Red Hat Storage software:
  • Unified File and Object Storage
    Unified File and Object Storage (UFO) unifies NAS and object storage technology. It provides a system for data storage that enables users to access the same data, both as an object and as a file, thus simplifying management and controlling storage costs.
  • Replicate Improvements (Pro-active Self-heal)
    In replicate module, previously you had to manually trigger a self-heal when a brick goes offline and comes back online, to bring all the replicas in sync. Now the pro-active self-heal daemon runs in the background, diagnoses issues and automatically initiates self-healing when the brick comes on-line. You can view the list of files that need healing, the list of files which are recently healed, list of files which are in split-brain state, and you can manually trigger self-heal on the entire volume or only on the files which need healing.
  • Network Lock Manager
    Red Hat Storage includes network lock manager (NLM) v4. NLM is a standard and an extension to NFSv3 which allows NFSv3 clients to lock on files across the network. NLM is required to make applications running on top of NFSv3 mount points to use the standard fcntl() (POSIX) and flock() (BSD) lock system calls to synchronize access across clients.
  • Volume Statedump
    Statedump is a mechanism through which you can get details of all internal variables and state of the glusterfs process at the time of issuing the command.You can perform statedumps of the brick processes and nfs server process of a volume using the statedump command. The statedump information is useful while debugging.
  • Volume Status and Brick Information
    You can display the status information about a specific volume, brick or all volumes, as needed. Volume status information includes memory usage, memory pool details of the bricks, inode tables of the volume, pending calls of the volume and other statistics. This information can be used to understand the current status of the brick, nfs processes, and overall file system. Status information can also be used to monitor and debug the volume information.
  • Automated NFS and CIFS IP Failover
    In replicated volume environment, you can configure Cluster Trivial Database (CTDB) to provide high availability for NFS and CIFS exports. CTDB adds virtual IP addresses (VIPs) and a heartbeat service to each Red Hat Storage Server.
    When a node in a cluster fails, CTDB enables a different node to take over the IP address of the failed node. This ensures the IP addresses for the services provided are always available.
  • Geo-replication Enhancements
    • Configuring Secure Geo-replication Slave
      Now you can configure a secure slave using SSH so that master is granted a restricted access. You need not specify configuration parameters regarding the slave on the master-side configuration. You can also rotate the log file of a particular master-slave session, all sessions of a master volume, and all geo-replication sessions, as needed. You can also set ignore-deletes option to 1 so that the file deleted on the master will not trigger a delete operation on the slave. Hence, the slave will remain as a superset of the master and can be used to recover the master in case of crash and/or accidental delete.
    • Geo-replication Failback and Failover
      Red Hat Storage 2.0 supports Geo-Replication failover and failback. If the master goes down, you can trigger a failover procedure so that the slave can be replaced as the master. During this time, all I/O operations including writes and reads are done on the slave. When the master is back online, you can trigger a failback procedure so that the slave syncs the delta back to the master.
    • Geo-replication Checkpointing
      Red Hat Storage 2.0 introduces a new introspection feature, Geo-replication Checkpointing. Using Checkpointing, you can get information on the progress of replication. By setting a checkpoint, the actual time is recorded as a reference timepoint, and from then on, enhanced synchronization information is available on whether the data on master as of the reference timepoint has been replicated on slave.
  • Mount Server Fail-over
    Now there is an option to add backup volfile server while mounting fuse client. When the first volfile server fails, then the server specified in backup volfile-server option is used as volfile server to mount the client. You can also specify the number of attempts to fetch while mounting glusterFS server. This option is useful when you mount a server with multiple IPs.
  • Debugging Locks
    You can use statedump command to list the locks held on files. The statedump output also provides information on each lock with its range, basename, PID of the application holding the lock, and so on. You can analyze and know which locks are valid and relevant at a point of time. After ensuring that no application is using the file, you can clear the lock using the clear lock command.
  • Change in Working Directory
    The working directory of glusterd has changed to /var/lib/glusterd from /etc/glusterd.
  • Gluster Volume Life-Cycle Extensions
    Red Hat Storage allows you to define custom actions for volume events such as volume start, stop, create, set, delete, and add brick. You can define both pre and post event actions. The actions can be present as executables or scripts in the defined directory structure.
  • Agile Provisioning
    • Remove Brick Enhancements
      Previously, remove-bick command was used to remove a brick that is inaccessible due to hardware or network failure and as a clean-up operation to remove dead server details from the volume configuration. Now remove-brick command can migrate data to existing bricks before deleting given brick.
    • Rebalance Enhancements
      Red Hat Storage 2.0 supports open file rebalance and files that have hardlinks. Rebalance is now enchanced to be more efficient with respect to network usage, completion time, and amount of data movement. It starts migration of data immediately without waiting for directory layout to be fixed.
    • Dynamic Alteration of Volume Type
      You can now change the type of the volume from Distributed volume to Distributed Replicated Volume when performing add-brick and remove-brick operation. You must specify the replica count parameter to increase the number of replicas to change it to distributed replicated volume.


      Currently, changing of stripe count while changing volume configurations is not supported.
  • Hadoop Compatible Storage (Technology Preview)
    Red Hat Storage provides compatibility for Apache Hadoop and it uses the standard file system APIs available in Hadoop to provide a new storage option for Hadoop deployments. Existing MapReduce based applications can use Red Hat Storage seamlessly. This new functionality opens up data within Hadoop deployments to any file-based or object-based application.


    Technology Preview features are not fully supported under Red Hat subscription level agreements (SLAs), may not be functionally complete, and are not intended for production use. However, these features provide early access to upcoming product innovations, enabling customers to test functionality and provide feedback during the development process. As Red Hat considers making future iterations of Technology Preview features generally available, we will provide commercially reasonable efforts to resolve any reported issues that customers experience when using these features.
  • Red Hat Storage Console (Technology Preview)
    Red Hat Storage Console is a powerful and simple web based Graphical User Interface for managing a Red Hat Storage 2.0 environment. It helps Storage Administrators to easily create and manage multiple storage pools. This includes features like elastically expanding / shrinking a cluster, creating and managing volumes.
  • Granual Locking for Large Files (Technology Preview)
    Enables using Red Hat Storage as a backing store for preserving large files like virtual machine images. Granualar locking enables internal file operations (like self-heal) without blocking user level file operations. The latency for user I/O is reduced during self-heal operation.
  • Read-only Volume (Technology Preview)
    Red Hat Storage enables you to mount volumes as read-only. While mounting the client, you can mount a volume as read-only and you can also make the entire volume as read-only, which applies for all the clients (including NFS clients) using volume set option.
  • RDMA (Remote Direct Memory Access) Support (Technology Preview)
    You can optionally configure Red Hat Storage to work over RDMA.

3. Installing Red Hat Storage

For step-by-step instructions to install Red Hat Storage, see Red Hat Storage Installation Guide.

4. Known Issues in Red Hat Storage v2.0

This chapter provides a list of known issues at the time of release:
  • yum update command may fail when it is executed for the first time in Red Hat Virtual Storage Appliance 3.2 with the following error "GPG key retrieval failed: [Errno 14] Could not open/read file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-auxiliary " .
    Work Around: Run yum update command again and it works fine on subsequent runs.
  • Issues related to Rebalancing Volumes:
    • Rebalance does not happen if bricks are down.
      Currently while running rebalance, make sure all the bricks are in operating or connected state.
    • After rebalancing a volume, if you run rm -rf command at the mount point to remove all contents of the current working directory recursively without prompting, you may get "Directory not Empty" error message.
    • Rebalance operation fails to migrate data in distributed striped volume.
    • Rebalance operation on geo-replicated volumes in the slave can lead to data inconsistency on the slave till files are updated on master.
  • glusterfsd - Error return code is always 0 after daemonizing the process.
    Due to this, scripts that mount glusterfs or start glusterfs process must not depend on its return value.
  • After # gluster volume replace-brick VOLNAME Brick New-Brick commit command is issued, the file system operations on that particular volume, which are in transit will fail.
  • RDMA transport is not supported in Red Hat Storage 2.0 release.
  • Issues related to Directory Quota:
    • Some writes can appear to pass even though the quota limit is exceeded (write returns success). This is because they could be cached in write-behind. However disk-space would not exceed the quota limit, since when writes to backend happen, quota does not allow them. Hence it is advised that applications should check for return value of close call.
    • If a user has done cd into a directory on which the administrator is setting the limit, even though the command succeeds and the new limit value will be applicable to all the users except for those users’ who has done cd in to that particular directory. The old limit value will be applicable until the user has cd out of that directory.
    • Rename operation (that is, removing oldpath and creating newpath) requires additional disk space equal to file size. This is because, during rename, it subtracts the size on oldpath after rename operation is performed, but it checks whether quota limit is exceeded on parents of newfile before rename operation.
    • With striped volumes, Quota feature is not available.
    • When renaming a file, if the available free size is less than the size of the file, quota displays "Disk limit exceeded" error without renaming the file.
  • Issues related to POSIX ACLs:
    • Even though POSIX ACLs are set on the file or directory, the + (plus) sign in the file permissions will not be displayed. This is for performance optimization and will be fixed in a future release.
    • When glusterfs is mounted with -o acl, directory read performance can be bad. Commands like recursive directory listing can be slower than normal.
    • When POSIX ACLs are set and multiple NFS clients are used, there could be inconsistency in the way ACLs are applied due to attribute caching in NFS. For a consistent view of POSIX ACLs in a multiple client setup, use -o noac option on NFS mount to switch off attribute caching. This could have a performance impact on operations involving attributes.
  • If you have enabled Gluster NLM, you cannot mount kernel NFS client on your storage nodes.
  • Due to enhancements in Graphs, you may experience excessive memory usage with this release.
  • After you restart the NFS server, the unlock within the grace-period may fail and previously held locks may not be reclaimed.
  • fcntl locking (NLM) does not work over IPv6.
  • You cannot perform NFS mount on a machine on which glusterfs-NFS process is already running unless you use the NFS mount -o nolock option. This is because glusterfs-NFS has already registered NLM port with portmapper.
  • If the NFS client is behind a firewall such as NAT (Network Address Translation) router, the locking behavior is unpredictable. The current implementation of NLM assumes there are no NAT happening to client's IP.
  • nfs.mount-udp option is disabled by default. You must enable it if you want to use posix-locks on solaris when you NFS mount the gluster volume. If you enable nfs.mount-udp option, while mounting a subdirectory (exported using nfs.export-dir option) on linux, you must mount using -o proto=tcp option.
  • For NLM to function properly, you must ensure that all the servers and clients have resolvable hostnames. That is, servers must be able to resolve client names and clients must be able to resolve server hostnames.
  • After replace-brick operation, the stat information is different on NFS mount and FUSE mount. This happens due to internal time stamp changes when the replace-brick operation is performed.
  • GET and PUT commands fails on the large files while using Unified File and Object Storage.
    Work Around: You must ensure to add node_timeout=60 variable in proxy, container, and object server configuration files.
  • In Red Hat Storage 2.0, Object Expiration feature of Swift-1.4.8 version is not supported.
  • Excessive logging while deleting files when Quota or gsync-indexing options are enabled.
  • Geo-replication uses rsync to sync files from master to slave, but rsync does not sync mknod and pipe files.
  • The following is a known missing (minor) feature:
    • locks - mandatory locking is not supported.

5. Product Support

You can reach support at

6. Product Documentation

Product documentation of Red Hat Storage is available at .

A. Revision History

Revision History
Revision 1-15Tue Dec 31 2013Pavithra Srinivasan
Updated Known Issues chapter.
Revision 1-13Tue Mar 19 2013Divya Muntimadugu
Updated Known Issues chapter.
Revision 1-12Tue Mar 12 2013Divya Muntimadugu
Changed product support URL.
Revision 1-11Tue Sep 18 2012Divya Muntimadugu
Changed references from to
Revision 1-10Wed Jul 18 2012Anthony Towns
Rebuild for Publican 3.0
Revision 1-8Tue Jun 26 2012Divya Muntimadugu
Version for 2.0 GA release
Revision 1-7Tue May 22 2012Divya Muntimadugu
Bug fixes
Revision 1-6Thu May 10 2012Divya Muntimadugu
Bug fixes
Revision 1-5Thu May 03 2012Divya Muntimadugu
Beta 2 updates and bug fixes
Revision 1-4Fri Mar 30 2012Divya Muntimadugu
Updated new features list in chapter 2
Revision 1-3Thu Mar 22 2012Divya Muntimadugu
Technical review comments incorporation
Revision 1-2Tue Mar 20 2012Divya Muntimadugu
Red Hat Storage Beta 1 release
Revision 1-1Thu Mar 1 2012Divya Muntimadugu
Red Hat Storage Alpha release
Revision 1-0Mon Feb 27 2012Divya Muntimadugu