Issue

The dirty page count is not being reduced by background writeout quickly enough.

# free
             total       used       free     shared    buffers     cached
Mem:        894972     851184      43788          0      34064     635024
-/+ buffers/cache:     182096     712876
Swap:      1052248          8    1052240

# grep present /proc/zoneinfo 
        present  2419
        present  254518

# cat /proc/sys/vm/dirty_ratio
40
# cat /proc/sys/vm/dirty_background_ratio 
10

Dirty page threshold is ~400MB (100000 pages), background threshold is ~100MB (25000 pages).

Write enough data to exceed the dirty ratio:

# dd if=/dev/zero of=file bs=1M count=400
400+0 records in
400+0 records out
419430400 bytes (419 MB) copied, 0.397827 seconds, 1.1 GB/s

Watch nr_dirty:

# export set SECONDS=0; while true; do COUNT=`grep nr_dirty /proc/vmstat | cut -d ' ' -f 2`; echo $SECONDS $COUNT; sleep 1; done | tee output
0 5
1 6
2 6
3 6
4 6
5 79306   <--- above dd started here
6 98824
7 98822
8 98822
9 98822
10 98822
11 98822
12 98822
13 97872
14 97872
15 97865
16 97865
18 97890
19 97890
20 97904
21 97904
22 97904
23 97904
24 97904
25 97904
26 97904
27 97904
28 97904
29 97781
30 97781
...
42 97781
43 97781
44 96756
45 96747
46 96747
...
76 96747
77 96747
78 95722
...
111 95724
112 95724
113 94674
115 94673
116 94673
...
147 94673
148 94673
149 93637
150 93637
...
181 93639
182 93639
183 92614
184 92614
...
216 92601
217 92601
219 91565
220 91565
...
252 91567
253 91567
254 90542
255 90531
256 90531
...
286 90531
287 90531
288 89506
289 89495
290 89495
...
357 88459
358 88459
359 87423
360 87423
...
391 87421
392 87421
393 86396
394 86396
...

Every 30 seconds ~1024 pages are written out. Background writeout should take the dirty page count down to the background ratio ~25000 pages much more quickly than this.