ext4 mounted with default delalloc option can cause slow IO performance when batching writes

Issue

• Performing a large amount of batched writes using ext4 file system can result in slow write to disk performance:

  Device:  rrqm/s   wrqm/s     r/s       w/s    rkB/s      wkB/s avgrq-sz avgqu-sz   await   svctm  %util
  sda        0.00     0.00    0.07  10331.83     0.27   41327.34     8.00    41.27    2.11   0.00   4.02
  sda        0.00     0.00    1.00 124663.00     4.00  498652.00     8.00 73829.03  880.79   0.01  99.70
  sda        0.00     0.00    0.00 299222.00     0.00 1196888.00     8.00 36525.34  101.35   0.00 100.00
  sda        0.00     0.00    0.00 114954.00     0.00  459816.00     8.00 37551.29  383.98   0.01  71.90
  sda        0.00     0.00    1.00  40990.00     4.00  163960.00     8.00   154.46    3.56   0.00  13.90 << %util drop
  sda        0.00     0.00    0.00  24594.00     0.00   98376.00     8.00   112.33    4.83   0.00   9.60 
  sda        0.00     0.00    0.00  24594.00     0.00   98376.00     8.00    98.75    4.02   0.00   8.70
  sda        0.00     0.00    0.00  24594.00     0.00   98376.00     8.00   130.04    5.29   0.00  10.10
  sda        0.00     0.00    0.00  16233.66     0.00   64934.65     8.00    65.16    4.01   0.00   5.84
  sda        0.00     0.00    0.00  16396.00     0.00   65584.00     8.00    68.55    4.18   0.00   6.10
  sda        0.00     0.00    0.00   8198.00     0.00   32792.00     8.00    29.24    3.57   0.00   3.10
  sda        0.00     0.00    0.00  16396.00     0.00   65584.00     8.00    74.37    4.54   0.00   6.10
  sda        0.00     0.00    0.00   8198.00     0.00   32792.00     8.00    36.70    4.48   0.00   3.10
  sda        0.00     0.00    0.00   8198.00     0.00   32792.00     8.00    37.12    4.53   0.00   3.20
  

• large amount of dirty write buffers exists in the system as seen in /proc/meminfo, but its not being flushed very fast to ext4 filesystem (low %util in iostat)

• during large file writes, iostat shows a steady flow of near 100% util and good throughput and then suddenly flushing seems to almost stop and %util drops to 5% or less (while there still remains a large amount of dirty pages to be flushed in the backlog) resulting in low io throughput.
• default delalloc mount command parameter causes slow performance in opening a file on ext4 file system when there is a large amount of dirty buffers already present
• default delalloc mount option causes slow performance in writing large amounts of data to file on ext4 filesystem