What is the suggested I/O scheduler to improve disk performance when using Red Hat Enterprise Linux with virtualization?

"If the schedulers noop and deadline disables the OS I/O-scheduling completely ... the automatic I/O-prioritization in the OS does not work anymore"

Clarification is needed for above. noop and deadline do not disable io scheduling completely. They just don't allow significant tuning changes on how the scheduling (queueing) is performed. For example deadline has two queues: read and write, but the two queues are each accessed via two methods, location sorted and time sorted: so two inputs and 4 outputs. That's all its got and it cannot be tuned further in terms of what is in the queue. It can be turned to bias the draining of io out of the queues more for reads or writes depending on how the tunables are set -- that is how many io are stripped from each and interleaved/sent to storage or how long a "deadline" period is used before switching between location ordered stripping of io from queues to disk vs time ordered draining of io from queues.

The cfq has some additional automatic sorting of io (not really "scheduling") but allows ionice and cGroup manipulation of processes io into different queues and these queues are in sets of idle, best effort, real-time... and then further broken down into queues of different priorities within those queue types.

"there must definitely be times when disabling OS scheduling is a bad idea."

Yes there can be, hence the wiggle room with the descriptions of each io scheduler:

• "...cfq scheduler is usually ideal" which conversely means there will be circumstances where it is not ideal but in general is a reasonable starting point for tuning.
• "...guests often benefit greatly from the noop I/O scheduler", which conversely measn there will be circumstances where noop in the application environment isn't a benefit.
• "...schedulers like deadline can be more advantageous", not shall be but can be for certain storage environments and application io loads, especially if those loads are read latency sensitive and less sensitive to and have a major component of buffered writes associated with the application.

This article is about the majority of cases and a reasonable starting point for further tuning. It doesn't cover cases for things like RT guests or where there is significant need to tuning/assigning priorities of individual processes... that is just way too specific to niche configurations. Also see the caveats in above, specifically about not treating paravirt or pass through devices to the same scheduling tuning as virtual disks.

Tuning outside of the general recommendations tend to be corner cases where the physical resources of the hypervisor being spread out over a number of guest is well separated and/or well bounded in terms of potential collective io load from all guests. But if using virtual disk io, then a single guest's scheduling of io is done in a vacuum and when it all trickles downhill into the hypervisor to be mixed in with other vm guests io going to the same underlying disks - setting up one guest to use RT queue scheduling means nothing to the hypervisor when it blends that io in with other guests io. Trying to tune one vm guest cannot know the definitive steady state impact to performance since it cannot know what io load being presented from other vm guests is at any given moment. Basically its trial and error with generally noop or deadline being nominally best over a wide range of vm guests and application loads. If you know the underlying physical storage is only being used by one guest's virtual disks, then you can almost treat it as physical pass-thru storage within the guest in terms of tuning... but most the time you may not know you have dedicated physical disks behind virtual disks just for the current vm guest. And if you did, it would be best to present them as pass-thru resources to the guest in the first place.

"for example have a low priority backup-process and a real-time scheduled production application/process."

You have to be very careful doing things like that as you can end up in a hung or really poorly performing system due to priority inversion. For example the backup process locks a directory on disk as it processes files so things can't change while grabbing files. While it has that filesystem lock, higher RT io to same filesystem is going on which blocks the backup io from being processed (while backup is holding that fs lock) to the point that the RT process trips across the filesystem lock the low priority backup process is holding. In truth I've see tuned applications using RT to the point that the filesystem's metadata transactions can't be flushed resulting in some parts of the RT application environment locking up (while other parts continue to read/write data to other parts of the filesystem or to other partitions on the disk) and it ends up locking out the filesystem's metadata transactions to the detriment of all other processes... eventually.

Tuning isn't for the faint of heart.

I think our internal discussions did not end with a clear winner. The mentions currently both noop and deadline. Having a view over the workloads/envirionments our customers run would be required, and then an investigation whether noop or deadline would be better.

If you have hints on relevant statistics which help to point out that noop would be the better default, then that could be a base to rethink the defaults. That would work best in requesting an RFE via a customer center case.

Tim,

I'm seeing the same schedulers available in CentOS8 as well.

Christian,

I've also tested later builds of CentOS7 (ex: v7.6) and the noop scheduler is missing:

$ cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core)
$cat /sys/block/vda/queue/scheduler
[mq-deadline] kyber none

Poking around on this, it appears that some RHEL/CentOS images are built with certain schedulers, ex: Red Hat hosted QCOW2 images, OpenStack Glance Cloudinit images, etc.; so it would help if there were a list of which build includes which schedulers.

The approaches apply also to rhel8/9, but the modification of kernel options might need modifications - I suspect because of this the scope has not yet been extended to rhel8/9 so far. Do you mind opening a case for this?