21.2. Brick Configuration

Format bricks using the following configurations to enhance performance:

Procedure 21.1. Brick Configuration

  1. LVM layer

    The steps for creating a brick from a physical device is listed below. An outline of steps for creating multiple bricks on a physical device is listed as Example - Creating multiple bricks on a physical device below.
    • Creating the Physical Volume
      The pvcreate command is used to create the physical volume. The Logical Volume Manager can use a portion of the physical volume for storing its metadata while the rest is used as the data portion.Align the I/O at the Logical Volume Manager (LVM) layer using --dataalignment option while creating the physical volume.
      The command is used in the following format:
      # pvcreate --dataalignment alignment_value disk
      For JBOD, use an alignment value of 256K.
      In case of hardware RAID, the alignment_value should be obtained by multiplying the RAID stripe unit size with the number of data disks. If 12 disks are used in a RAID 6 configuration, the number of data disks is 10; on the other hand, if 12 disks are used in a RAID 10 configuration, the number of data disks is 6.
      For example, the following command is appropriate for 12 disks in a RAID 6 configuration with a stripe unit size of 128 KiB:
      # pvcreate --dataalignment 1280k disk
      The following command is appropriate for 12 disks in a RAID 10 configuration with a stripe unit size of 256 KiB:
      # pvcreate --dataalignment 1536k disk
      To view the previously configured physical volume settings for --dataalignment, run the following command:
      # pvs -o +pe_start disk
        PV         VG   Fmt  Attr PSize PFree 1st PE
        /dev/sdb        lvm2 a--  9.09t 9.09t   1.25m
    • Creating the Volume Group
      The volume group is created using the vgcreate command.
      For hardware RAID, in order to ensure that logical volumes created in the volume group are aligned with the underlying RAID geometry, it is important to use the -- physicalextentsize option. Execute the vgcreate command in the following format:
      # vgcreate --physicalextentsize extent_size VOLGROUP physical_volume
      The extent_size should be obtained by multiplying the RAID stripe unit size with the number of data disks. If 12 disks are used in a RAID 6 configuration, the number of data disks is 10; on the other hand, if 12 disks are used in a RAID 10 configuration, the number of data disks is 6.
      For example, run the following command for RAID-6 storage with a stripe unit size of 128 KB, and 12 disks (10 data disks):
      # vgcreate --physicalextentsize 1280k VOLGROUP physical_volume
      In the case of JBOD, use the vgcreate command in the following format:
      # vgcreate VOLGROUP physical_volume
    • Creating the Thin Pool
      A thin pool provides a common pool of storage for thin logical volumes (LVs) and their snapshot volumes, if any.
      Execute the following commands to create a thin pool of a specific size:
      # lvcreate --thin VOLGROUP/POOLNAME --size POOLSIZE --chunksize CHUNKSIZE --poolmetadatasize METASIZE --zero n
      You can also create a thin pool of the maximum possible size for your device by executing the following command:
      # lvcreate --thin VOLGROUP/POOLNAME --extents 100%FREE --chunksize CHUNKSIZE --poolmetadatasize METASIZE --zero n

      Recommended parameter values for thin pool creation

      poolmetadatasize
      Internally, a thin pool contains a separate metadata device that is used to track the (dynamically) allocated regions of the thin LVs and snapshots. The poolmetadatasize option in the above command refers to the size of the pool metadata device.
      The maximum possible size for a metadata LV is 16 GiB. Red Hat Gluster Storage recommends creating the metadata device of the maximum supported size. You can allocate less than the maximum if space is a concern, but in this case you should allocate a minimum of 0.5% of the pool size.

      Warning

      If your metadata pool runs out of space, you cannot create data. This includes the data required to increase the size of the metadata pool or to migrate data away from a volume that has run out of metadata space. Monitor your metadata pool using the lvs -o+metadata_percent command and ensure that it does not run out of space.
      chunksize
      An important parameter to be specified while creating a thin pool is the chunk size,which is the unit of allocation. For good performance, the chunk size for the thin pool and the parameters of the underlying hardware RAID storage should be chosen so that they work well together.
      For JBOD, use a thin pool chunk size of 256 KiB.
      For RAID 6 storage, the striping parameters should be chosen so that the full stripe size (stripe_unit size * number of data disks) is between 1 MiB and 2 MiB, preferably in the low end of the range. The thin pool chunk size should be chosen to match the RAID 6 full stripe size. Matching the chunk size to the full stripe size aligns thin pool allocations with RAID 6 stripes, which can lead to better performance. Limiting the chunk size to below 2 MiB helps reduce performance problems due to excessive copy-on-write when snapshots are used.
      For example, for RAID 6 with 12 disks (10 data disks), stripe unit size should be chosen as 128 KiB. This leads to a full stripe size of 1280 KiB (1.25 MiB). The thin pool should then be created with the chunk size of 1280 KiB.
      For RAID 10 storage, the preferred stripe unit size is 256 KiB. This can also serve as the thin pool chunk size. Note that RAID 10 is recommended when the workload has a large proportion of small file writes or random writes. In this case, a small thin pool chunk size is more appropriate, as it reduces copy-on-write overhead with snapshots.
      If the addressable storage on the device is smaller than the device itself, you need to adjust the recommended chunk size. Calculate the adjustment factor using the following formula:
      adjustment_factor = device_size_in_tb / (preferred_chunk_size_in_kb * 4 / 64 )
      Round the adjustment factor up. Then calculate the new chunk size using the following:
      chunk_size = preferred_chunk_size * rounded_adjustment_factor
      block zeroing
      By default, the newly provisioned chunks in a thin pool are zeroed to prevent data leaking between different block devices. In the case of Red Hat Gluster Storage, where data is accessed via a file system, this option can be turned off for better performance with the --zero n option. Note that n does not need to be replaced.
      The following example shows how to create the thin pool:
      # lvcreate --thin VOLGROUP/thin_pool --size 2T --chunksize 1280k --poolmetadatasize 16G --zero n
      You can also use --extents 100%FREE to ensure the thin pool takes up all available space once the metadata pool is created.
      # lvcreate --thin VOLGROUP/thin_pool --extents 100%FREE --chunksize 1280k --poolmetadatasize 16G --zero n
      The following example shows how to create a 2 TB thin pool:
      # lvcreate --thin VOLGROUP/thin_pool --size 2T --chunksize 1280k --poolmetadatasize 16G --zero n
      The following example creates a thin pool that takes up all remaining space once the metadata pool has been created.
      # lvcreate --thin VOLGROUP/thin_pool --extents 100%FREE --chunksize 1280k --poolmetadatasize 16G --zero n
    • Creating a Thin Logical Volume
      After the thin pool has been created as mentioned above, a thinly provisioned logical volume can be created in the thin pool to serve as storage for a brick of a Red Hat Gluster Storage volume.
      # lvcreate --thin --name LV_name --virtualsize LV_size VOLGROUP/thin_pool
    • Example - Creating multiple bricks on a physical device
      The steps above (LVM Layer) cover the case where a single brick is being created on a physical device. This example shows how to adapt these steps when multiple bricks need to be created on a physical device.

      Note

      In this following steps, we are assuming the following:
      • Two bricks must be created on the same physical device
      • One brick must be of size 4 TiB and the other is 2 TiB
      • The device is /dev/sdb, and is a RAID-6 device with 12 disks
      • The 12-disk RAID-6 device has been created according to the recommendations in this chapter, that is, with a stripe unit size of 128 KiB
      1. Create a single physical volume using pvcreate
        # pvcreate --dataalignment 1280k /dev/sdb
      2. Create a single volume group on the device
        # vgcreate --physicalextentsize 1280k vg1 /dev/sdb
      3. Create a separate thin pool for each brick using the following commands:
        # lvcreate --thin vg1/thin_pool_1 --size 4T --chunksize 1280K --poolmetadatasize 16G --zero n
        # lvcreate --thin vg1/thin_pool_2 --size 2T --chunksize 1280K --poolmetadatasize 16G --zero n
        In the examples above, the size of each thin pool is chosen to be the same as the size of the brick that will be created in it. With thin provisioning, there are many possible ways of managing space, and these options are not discussed in this chapter.
      4. Create a thin logical volume for each brick
        # lvcreate --thin --name lv1 --virtualsize 4T vg1/thin_pool_1
        # lvcreate --thin --name lv2 --virtualsize 2T vg1/thin_pool_2
      5. Follow the XFS Recommendations (next step) in this chapter for creating and mounting filesystems for each of the thin logical volumes
        # mkfs.xfs options /dev/vg1/lv1
        # mkfs.xfs options /dev/vg1/lv2
        # mount options /dev/vg1/lv1 mount_point_1
        # mount options /dev/vg1/lv2 mount_point_2
  2. XFS Recommendataions

    • XFS Inode Size
      As Red Hat Gluster Storage makes extensive use of extended attributes, an XFS inode size of 512 bytes works better with Red Hat Gluster Storage than the default XFS inode size of 256 bytes. So, inode size for XFS must be set to 512 bytes while formatting the Red Hat Gluster Storage bricks. To set the inode size, you have to use -i size option with the mkfs.xfs command as shown in the following Logical Block Size for the Directory section.
    • XFS RAID Alignment
      When creating an XFS file system, you can explicitly specify the striping parameters of the underlying storage in the following format:
      # mkfs.xfs other_options -d su=stripe_unit_size,sw=stripe_width_in_number_of_disks device
      For RAID 6, ensure that I/O is aligned at the file system layer by providing the striping parameters. For RAID 6 storage with 12 disks, if the recommendations above have been followed, the values must be as following:
      # mkfs.xfs other_options -d su=128k,sw=10 device
      For RAID 10 and JBOD, the -d su=<>,sw=<> option can be omitted. By default, XFS will use the thin-p chunk size and other parameters to make layout decisions.
    • Logical Block Size for the Directory
      An XFS file system allows to select a logical block size for the file system directory that is greater than the logical block size of the file system. Increasing the logical block size for the directories from the default 4 K, decreases the directory I/O, which in turn improves the performance of directory operations. To set the block size, you need to use -n size option with the mkfs.xfs command as shown in the following example output.
      Following is the example output of RAID 6 configuration along with inode and block size options:
      # mkfs.xfs -f -i size=512 -n size=8192 -d su=128k,sw=10 logical volume
      meta-data=/dev/mapper/gluster-brick1 isize=512    agcount=32, agsize=37748736 blks
               =    sectsz=512   attr=2, projid32bit=0
      data     =     bsize=4096   blocks=1207959552, imaxpct=5
               =    sunit=32     swidth=320 blks
      naming   = version 2   bsize=8192   ascii-ci=0
      log      =internal log   bsize=4096   blocks=521728, version=2
               =    sectsz=512   sunit=32 blks, lazy-count=1
      realtime =none    extsz=4096   blocks=0, rtextents=0
    • Allocation Strategy
      inode32 and inode64 are two most common allocation strategies for XFS. With inode32 allocation strategy, XFS places all the inodes in the first 1 TiB of disk. With larger disk, all the inodes would be stuck in first 1 TiB. inode32 allocation strategy is used by default.
      With inode64 mount option inodes would be replaced near to the data which would be minimize the disk seeks.
      To set the allocation strategy to inode64 when file system is being mounted, you need to use -o inode64 option with the mount command as shown in the following Access Time section.
    • Access Time
      If the application does not require to update the access time on files, than file system must always be mounted with noatime mount option. For example:
      # mount -t xfs -o inode64,noatime <logical volume> <mount point>
      This optimization improves performance of small-file reads by avoiding updates to the XFS inodes when files are read.
      /etc/fstab entry for option E + F
       <logical volume> <mount point>xfs     inode64,noatime   0 0
    • Allocation groups
      Each XFS file system is partitioned into regions called allocation groups. Allocation groups are similar to the block groups in ext3, but allocation groups are much larger than block groups and are used for scalability and parallelism rather than disk locality. The default allocation for an allocation group is 1 TiB.
      Allocation group count must be large enough to sustain the concurrent allocation workload. In most of the cases allocation group count chosen by mkfs.xfs command would give the optimal performance. Do not change the allocation group count chosen by mkfs.xfs, while formatting the file system.
    • Percentage of space allocation to inodes
      If the workload is very small files (average file size is less than 10 KB ), then it is recommended to set maxpct value to 10, while formatting the file system.
  3. Performance tuning option in Red Hat Gluster Storage

    A tuned profile is designed to improve performance for a specific use case by tuning system parameters appropriately. Red Hat Gluster Storage includes tuned profiles tailored for its workloads. These profiles are available in both Red Hat Enterprise Linux 6 and Red Hat Enterprise Linux 7.

    Table 21.1. Recommended Profiles for Different Workloads

    WorkloadProfile Name
    Large-file, sequential I/O workloads rhgs-sequential-io
    Small-file workloads rhgs-random-io
    Random I/O workloads rhgs-random-io
    Earlier versions of Red Hat Gluster Storage on Red Hat Enterprise Linux 6 recommended tuned profiles rhs-high-throughput and rhs-virtualization. These profiles are still available on Red Hat Enterprise Linux 6. However, switching to the new profiles is recommended.
    To apply tunings contained in the tuned profile, run the following command after creating a Red Hat Gluster Storage volume.
    # tuned-adm profile profile-name
    For example:
    # tuned-adm profile rhgs-sequential-io
  4. Writeback Caching

    For small-file and random write performance, we strongly recommend writeback cache, that is, non-volatile random-access memory (NVRAM) in your storage controller. For example, normal Dell and HP storage controllers have it. Ensure that NVRAM is enabled, that is, the battery is working. Refer your hardware documentation for details on enabling NVRAM.
    Do not enable writeback caching in the disk drives, this is a policy where the disk drive considers the write is complete before the write actually made it to the magnetic media (platter). As a result, the disk write cache might lose its data during a power failure or even loss of metadata leading to file system corruption.

21.2.1. Many Bricks per Node

By default, for every brick configured on a Red Hat Gluster Storage server node, one process is created and one port is consumed. If you have a large number of bricks configured on a single server, enabling brick multiplexing reduces port and memory consumption by allowing compatible bricks to use the same process and port. Red Hat recommends restarting all volumes after enabling or disabling brick multiplexing.
As of Red Hat Gluster Storage 3.4, brick multiplexing is supported only for Container-Native Storage (CNS) and Container-Ready Storage(CRS) use cases.

Configuring Brick Multiplexing

  1. Set cluster.brick-multiplex to on. This option affects all volumes.
    # gluster volume set all cluster.brick-multiplex on
  2. Increase the limits for socket traffic.
    # gluster volume set all transport.listen-backlog 128

    Note

    The transport.listen-backlog parameter is not the only parameter that controls queue length. You may also need to change system settings like net.ipv4.tcp_max_syn_backlog and net.core.somaxconn.
  3. Restart all volumes for brick multiplexing to take effect.
    # gluster volume stop VOLNAME
    # gluster volume start VOLNAME

Important

Brick compatibility is determined when the volume starts, and depends on volume options shared between bricks. When brick multiplexing is enabled, Red Hat recommends restarting the volume whenever any volume configuration details are changed in order to maintain the compatibility of the bricks grouped under a single process.

21.2.2. Port Range Configuration

By default, for every brick configured on a Red Hat Gluster Storage server node, one process is created and one port is consumed. If you have a large number of bricks configured on a single server, configuring port range lets you control the range of ports allocated by glusterd to newly created or existing bricks and volumes.
This can be achieved with the help of the glusterd.vol file. The base-port and max-port options can be used to set the port range. By default, base-port is set to 49152, and max-port is set to 65535.

Important

If glusterd runs out of free ports to allocate within the specified range of base-port and max-port, newer bricks and volumes fail to start.

Configuring Port Range

  1. Edit the glusterd.vol file on all the nodes.
    # vi /etc/glusterfs/glusterd.vol
  2. Remove the comment marker # corresponding to the base-port and max-port options.
    volume management
        type mgmt/glusterd
        option working-directory /var/lib/glusterd
        option transport-type socket,rdma
        option transport.socket.keepalive-time 10
        option transport.socket.keepalive-interval 2
        option transport.socket.read-fail-log off
        option ping-timeout 0
        option event-threads 1
    #   option lock-timer 180
    #   option transport.address-family inet6
        option base-port 49152
        option max-port  65535
    end-volume
    
  3. Define the port number in the base-port, and max-port options.
       option base-port 49152
       option max-port  65535
    
  4. Save the glusterd.vol file and restart the glusterd service on each Red Hat Gluster Storage node.