ext4 mounted with default delalloc option can cause slow IO performance when batching writes

Solution Verified - Updated -

Issue

  • Performing a large amount of batched writes using ext4 file system can result in slow write to disk performance:

    Device:  rrqm/s   wrqm/s     r/s       w/s    rkB/s      wkB/s avgrq-sz avgqu-sz   await   svctm  %util
    sda        0.00     0.00    0.07  10331.83     0.27   41327.34     8.00    41.27    2.11   0.00   4.02
    sda        0.00     0.00    1.00 124663.00     4.00  498652.00     8.00 73829.03  880.79   0.01  99.70
    sda        0.00     0.00    0.00 299222.00     0.00 1196888.00     8.00 36525.34  101.35   0.00 100.00
    sda        0.00     0.00    0.00 114954.00     0.00  459816.00     8.00 37551.29  383.98   0.01  71.90
    sda        0.00     0.00    1.00  40990.00     4.00  163960.00     8.00   154.46    3.56   0.00  13.90 << %util drop
    sda        0.00     0.00    0.00  24594.00     0.00   98376.00     8.00   112.33    4.83   0.00   9.60 
    sda        0.00     0.00    0.00  24594.00     0.00   98376.00     8.00    98.75    4.02   0.00   8.70
    sda        0.00     0.00    0.00  24594.00     0.00   98376.00     8.00   130.04    5.29   0.00  10.10
    sda        0.00     0.00    0.00  16233.66     0.00   64934.65     8.00    65.16    4.01   0.00   5.84
    sda        0.00     0.00    0.00  16396.00     0.00   65584.00     8.00    68.55    4.18   0.00   6.10
    sda        0.00     0.00    0.00   8198.00     0.00   32792.00     8.00    29.24    3.57   0.00   3.10
    sda        0.00     0.00    0.00  16396.00     0.00   65584.00     8.00    74.37    4.54   0.00   6.10
    sda        0.00     0.00    0.00   8198.00     0.00   32792.00     8.00    36.70    4.48   0.00   3.10
    sda        0.00     0.00    0.00   8198.00     0.00   32792.00     8.00    37.12    4.53   0.00   3.20
    
  • large amount of dirty write buffers exists in the system as seen in /proc/meminfo, but its not being flushed very fast to ext4 filesystem (low %util in iostat)

  • during large file writes, iostat shows a steady flow of near 100% util and good throughput and then suddenly flushing seems to almost stop and %util drops to 5% or less (while there still remains a large amount of dirty pages to be flushed in the backlog) resulting in low io throughput.
  • default delalloc mount command parameter causes slow performance in opening a file on ext4 file system when there is a large amount of dirty buffers already present
  • default delalloc mount option causes slow performance in writing large amounts of data to file on ext4 filesystem

Environment

  • Red Hat Enterprise Linux 6, 7
  • Ext4 file system

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In