What are the advantages and disadvantages to using partitioning on LUNs, either directly or with LVM in between?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 5, 6, 7 and 8

Issue

  • What are the advantages and disadvantages to using partitioning on LUNs, either directly or with LVM in between? What should be taken into consideration in terms of administration, performance and resilience in this context?
  • Is it supported if we create Physical Volumes directly on disks, instead of creating single partition on the disk.
    For example, is the below configuration supported
# pvcreate /dev/sdb  OR
# pvcreate /dev/mapper/mpatha

Resolution

  • Creating physical volumes on the entire disk instead of partition on it when you want to use the entire disk is completely supported. Following are the advantages/disadvantages of creating the physical volume on entire disk or a partition on the disk.

Administrative Considerations

Firstly, partitioning reduces the likelihood of administrative oversights. Having no partition, no /proc/partitions entity, etc., increases the possibility that a systems administrator may overlook a PV while searching for unused drives. Partitioned drives are recognised by the system's partitioning interfaces and tools (for example. /proc/partitions, parted, etc.) and thus are less prone to administrative errors (where the drive is seen as unused). This obviously is not the case when whole drives are used as PVs.

Performance Considerations

Alignment

If partitioning is used in order to reduce the administrative risk, then partitions need to be created with a full understanding as to how to most effectively set out partition alignment. If not, introducing partitioning might cause alignment (and thus performance) issues which may not be seen in alternative configurations where no partition is created before 'pvcreate' is executed, for example

This is especially true of kernel versions that lack kernel support for auto-alignment. Alignment may be difficult to get right depending on which version of the Linux kernel is running. Storage I/O alignment support has been added to recent kernels and tools in order to try to help with this. The aim is to try to avoid the need to hand-craft alignment.

The following chapter of the RHEL 6 Storage Administration Guide provides more details on auto-alignment:

⁠Chapter 23. Storage I/O Alignment and Size

"Recent enhancements to the SCSI and ATA standards allow storage devices to indicate their preferred (and in some cases, required) I/O alignment and I/O size. This information is particularly useful with newer disk drives that increase the physical sector size from 512 bytes to 4k bytes. This information may also be beneficial for RAID devices, where the chunk size and stripe size may impact performance.

The Linux I/O stack has been enhanced to process vendor-provided I/O alignment and I/O size information, allowing storage management tools (parted, lvm, mkfs.*, and the like) to optimize data placement and access. If a legacy device does not export I/O alignment and size data, then storage management tools in Red Hat Enterprise Linux 6 will conservatively align I/O on a 4k (or larger power of 2) boundary. This will ensure that 4k-sector devices operate correctly even if they do not indicate any required/preferred I/O alignment and size."

The following section of the RHEL 6.0 Release Notes also provides some detail on that topic: RHEL 6.0 Release Notes - 4.1. Storage Input/Output Alignment and Size

Depending on the properties of the drive (SSDs, high end storage using cache upfront, etc.) it is necessary to know about the particular first level storage alignment properties being exposed to the operating system. As such, it is generally easier to align correctly when partitioning is not used, in cases where alignment support in the kernel is not available.

Influence from frequent open/close operations

Under certain circumstances, when applications repeatedly open(RDWR) and then close disk devices without partitions, invalidations of the whole blocks device caches have been seen, leading to performance degradation.

In reproducers for this scenario, a udev event drops the device's cache when it rescans the partition table. This causes the reproducer to be slower, because the cache is getting dropped.

A real world scenario where this occurs requires:

  • an application which uses the whole block device to read and write the data. The application sometimes experiences
    the I/O performance degradation.
  • an other application which repeatedly opens and closes the whole block device in read-write mode without
    updating the contents.

This is not a normal situation. The more common scenario would be some application like certain DBs directly opening and closing devices. But here it's usually not the cache flush which is the problem since such applications typically use direct I/O but the extra udev processing.

To solve the performance degradation in this situation, partitions can be used. Alternatively, udev's "nowatch" option could be used.

Overall Considerations:

Using (one) partition on any single drive, with proper alignment, reduces administrative oversight and helps maintain acceptable performance levels.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

4 Comments

Is the disk alignment concern relevant to virtual environments(VMWare, RHEV, AWS, OpenStack, etc.)?

I can not immediately think of a situation where this would not apply to virtual environments..

One problem I see is that resizing a partition on a running system is not supported although it's possible (found that in some Red Hat documentation). Therefore I switched to using the complete disk with LVM. Now resizing the disk, i.e. the VMDK, is followed by a pvresize and everything seems to be fine.

I remember bz's about the kernel detection of resized partitions on rhel6 not capturing as many cases as it did on rhel5, that was due to changes due to stability considerations. I think that online resizing is the reason for not supporting resizing with partitions. If one wants to resize, there should be other, supported solutions for the specific setup/problem to solve. When you extend the underlying device, adding new partitions, and creating LVM PV's and VG's ontop could also be used instead of the raw device.