What is the suggested I/O scheduler to improve disk performance when using Red Hat Enterprise Linux with virtualization?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 4, 5, 6 or 7
  • Virtualization, e.g. KVM, Xen, VMware or Microsoft Hyper-V
  • Virtualization guest or virtualization host
  • Virtual disk

Issue

  • What is the recommended I/O scheduler for Red Hat Enterprise Linux as a virtualization host?

Resolution

Red Hat Enterprise Linux as a virtualization host (Xen, KVM or VMware)

  • When using RHEL as a host for virtualized guests, the default cfq scheduler is usually ideal. This scheduler performs well on nearly all workloads.
  • If, however, minimizing I/O latency is more important than maximizing I/O throughput on the guest workloads, it may be beneficial to use the deadline scheduler . The deadline is also the scheduler used by the tuned profile virtual-host.

Red Hat Enterprise Linux as a virtualization guest (any hypervisor technology)

  • RHEL guests often benefit greatly from the noop I/O scheduler, which allows the host/hypervisor to optimize the I/O requests and prioritize based on incoming guest load. The noop scheduler can still combine small requests from the guest OS into larger requests before handing the I/O to the hypervisor, however noop follows the idea to spend as few CPU cycles as possible in the guest for I/O scheduling. The host/hypervisor will have an overview of the requests of all guests and have a separate strategy for handling I/O.
  • Depending on the disk presentation (Virtual Disk vs RDM) and I/O workload, schedulers like deadline can be more advantageous. Performance testing is required to verify which scheduler is the most advantageous.
  • Guests using storage accessed by iSCSI, SR-IOV or physical device pass-through should not use the noop scheduler, since these methods do not allow the host to optimize I/O requests to the underlying physical device.

Root Cause

  • In virtualized environments, it is not advantageous to schedule I/O at both the guest and hypervisor layers. If multiple guests use storage on a filesystem or block device managed by the host operating system, the host will likely schedule I/O more efficiently because it is aware of requests from all guests and knows the physical layout of storage, which may not map linearly to the guests' virtual storage. On the other hand, depending on the workload, it can also be beneficial to use a scheduler like deadline in the guest.

  • All scheduler tuning should be tested under normal operating conditions, as synthetic benchmarks typically do not accurately compare performance of systems using shared resources in virtual environments.

Configuring the I/O scheduler on Red Hat Enterprise Linux 4, 5 and 6

  • The I/O scheduler can be selected at boot time using the elevator kernel parameter. In the following example grub.conf stanza, the system has been configured to use the noop scheduler:

    title Red Hat Enterprise Linux Server (2.6.18-8.el5)
    root (hd0,0)
    kernel /vmlinuz-2.6.18-8.el5 ro root=/dev/vg0/lv0 elevator=noop
    initrd /initrd-2.6.18-8.el5.img
    
  • The default scheduler in Red Hat Enterprise Linux 4, 5 and 6 is CFQ. The available tuned profiles use the deadline elevator. A custom tuned profile can also be used to specify the elevator. More information on creating a custom tuned profile can be found in solution 1305833

Configuring the I/O scheduler on Red Hat Enterprise Linux 7

  • The default scheduler in Red Hat Enterprise Linux 7 is now deadline.
  • To make the changes persistent through boot you have to add elevator=noop toGRUB_CMDLINE_LINUX in /etc/default/grub as shown below.

    # cat /etc/default/grub
    [...]
    GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=vg00/lvroot rhgb quiet elevator=noop"
    [...]
    
    • After the entry has been created/updated, rebuild the /boot/grub2/grub.cfg file to include the new configuration with the added parameter:
      • On BIOS-based machines: ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
      • On UEFI-based machines: ~]# grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
  • Another way to change the default I/O scheduler is to use tuned.

  • More information on creating a custom tuned profile can be found in solution 1305833

Online configuring the I/O scheduler on Red Hat Enterprise Linux

  • In Red Hat Enterprise Linux 5, 6 or 7 it is also possible to change the I/O scheduler for a particular disk after the system has been booted. This makes it possible to use different I/O schedulers for different disks.

    # cat /sys/block/hda/queue/scheduler
    noop anticipatory deadline [cfq]
    
    # echo 'noop' > /sys/block/hda/queue/scheduler
    # cat /sys/block/hda/queue/scheduler
    [noop] anticipatory deadline cfq
    

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

10 Comments

We've done lots of testing on this with RHEL 5 and found cfq to be best for the guest in an ESXi env.

If I'm not mistaken, there is a race condition when running RHEL 4 with elevator=NOOP in VMWare.  Use deadline or cfq.

Typo, one does not edit /etc/grub2.cfg See /etc/sysconfig/grub

etc/grub2.cfg should not be edited, the correct way is to edit /etc/default/grub and then run grub2-mkconfig

On UEFI- based machines, one must execute grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg instead of grub2-mkconfig -o /boot/grub2/grub.cfg.

For more info, see: Customizing the GRUB 2 Configuration File.

Another option would be to specify the scheduler depending on the disk being in use.
Create this new rule -> sudo nano /etc/udev/rules.d/60-schedulers.rules

Put the following content into the empty file (save it - reboot the system afterwards) :

# set cfq scheduler for rotating disks
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="cfq"

# set deadline scheduler for non-rotating disks
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="deadline"

The rule does not differentiate between the virtual disks and pass-through (for example enterprise san storage). So my assumption is the above is for virtual disks only.

Hi Dwight, this method is working for physical disks as well - I have tested it on various Linux distributions.
When you are running a RHEL host with KVM virtualization on a SSD, the deadline scheduler will be in use.

Pass thru san based disks are typically marked rotational, but because they are nominally backed by multiple physical disks, deadline is a better scheduler than cfq for those disks. The above rules will set cfq for rotational based san disks, which is often sub-optimal.

Hi Dwight, thanks for the information - I only wanted to provide the additional option to tweak the scheduler.
If you want to use the deadline scheduler for rotational disks, you of course can do this by changing the rule :
ACTION=="add|change", KERNEL=="sd[a-z]", ATTR{queue/rotational}=="1", ATTR{queue/scheduler}="deadline"