Does FS-Cache performance depends on file size ?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 6
  • Red Hat Enterprise Linux (RHEL) 7

Issue

  • Cachefilesd is activated for NFS filesystem, but no files are being cached.
    # du -sh /var/cache/fscache does not show any increase in size.
  • It seems that small files do not benefit from FS-cache.
  • Is there any other alternative for FS-cache ?

Resolution

  • To know about FS-Cache and its implementation, refer FS-Cache
  • To check the accurate size(in Bytes) of FS-Cache, use # du -sb /var/cache/fscache command.
  • It is found (from the tcpdump analysis) that the 'READ' calls are not made for the NFS share mounted with 'fsc' option, this will reduce the network traffic and improve the performance. However, 'READ' calls can be made if the file is created or modified and not yet cached.
  • The performance is improved as the file size increases. Nevertheless, there is some performance improvement for the files of smaller size. Refer the diagnostic steps.
  • There is no alternative for FS-cache.

Note : FS-Cache in Red Hat Enterprise Linux 6 enables networked file systems (for example, NFS) to have a persistent cache of data on the client machine. cachefilesd package is a Technology Preview in Red Hat Enterprise Linux 6 Update 5.

Root Cause

  • The file size was very small to be reflected in # du -sh output.

Diagnostic Steps

Test for larger files

  • On the NFS server, created a test file 'test.out' of large size, for example about 200MB:
# dd if=/dev/urandom of=/nfsshare/test.out bs=1M count=200
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 31.7595 s, 6.6 MB/s
  • From the NFS client(mounted with fsc option), read the file for the first time. The file was read in about 18 seconds at speed of 11.7 MB/s :
# time dd if=/nfsmount/test.out of=/root/localfile.out bs=1M count=200
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 17.9864 s, 11.7 MB/s

real    0m18.011s
user    0m0.004s
sys     0m3.905s
  • Check if data is cached :
# du -sh /var/cache/fscache
200M    /var/cache/fscache

# du -sb /var/cache/fscache
209715200     /var/cache/fscache
  • Again, reading the same file took about 3.5 seconds at the speed of 64.5 MB/s :
# time dd if=/nfsmount/test.out of=/root/localfile.out bs=1M count=200
200+0 records in
200+0 records out
209715200 bytes (210 MB) copied, 3.25342 s, 64.5 MB/s

real    0m3.449sDU
user    0m0.003s
sys     0m2.276s

Test for smaller files

  • For performance check, related to smaller files, create some sample files with dd command at the server. Here files from 20KB to 250 KB are created. A file of 20KB can be created as :
# dd if=/dev/urandom of=/nfsshare/file8 bs=1K count=20
20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.0167591 s, 1.2 MB/s
  • Check the size of created files :
# du -sh /nfsshare
252K file1
60K file10
200K file2
152K file3
100K file4
52K file5
32K file6
20K file7
20K file8
32K file9
  • On the NFS client, check the fs-cache size :
# du -sh /var/cache/fscache
16K /var/cache/fscache
  • Then read the files and analyse the speed and time required from NFS client. Read the files for first time :
# for i in {1..10}; do time dd if=/nfsmount/file$i of=/root/file$i bs=1K count=20; done

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.0298465 s, 686 kB/s

real    0m0.095s
user    0m0.005s
sys 0m0.024s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.018726 s, 1.1 MB/s

real    0m0.037s
user    0m0.002s
sys 0m0.023s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.016645 s, 1.2 MB/s

real    0m0.031s
user    0m0.001s
sys 0m0.017s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.0153253 s, 1.3 MB/s

real    0m0.030s
user    0m0.004s
sys 0m0.016s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.0090961 s, 2.3 MB/s

real    0m0.025s
user    0m0.003s
sys 0m0.013s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00804983 s, 2.5 MB/s

real    0m0.023s
user    0m0.003s
sys 0m0.010s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00684779 s, 3.0 MB/s

real    0m0.021s
user    0m0.003s
sys 0m0.010s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00715758 s, 2.9 MB/s

real    0m0.023s
user    0m0.000s
sys 0m0.014s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.0090423 s, 2.3 MB/s

real    0m0.022s
user    0m0.003s
sys 0m0.011s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00901518 s, 2.3 MB/s

real    0m0.025s
user    0m0.003s
sys 0m0.013s
  • Check if data is cached :
# du -sh /var/cache/fscache
1.1M    /var/cache/fscache

# du -sb /var/cache/fscache
1005568 /var/cache/fscache
  • Read the files for second time :
# for i in {1..10}; do time dd if=/nfsmount/file$i of=/root/file$i bs=1K count=20; done

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00449734 s, 4.6 MB/s

real    0m0.021s
user    0m0.003s
sys 0m0.012s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00415585 s, 4.9 MB/s

real    0m0.018s
user    0m0.001s
sys 0m0.011s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00424477 s, 4.8 MB/s

real    0m0.019s
user    0m0.002s
sys 0m0.011s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.0040832 s, 5.0 MB/s

real    0m0.017s
user    0m0.004s
sys 0m0.009s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00404666 s, 5.1 MB/s

real    0m0.017s
user    0m0.002s
sys 0m0.011s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00390854 s, 5.2 MB/s

real    0m0.018s
user    0m0.001s
sys 0m0.012s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00437571 s, 4.7 MB/s

real    0m0.018s
user    0m0.001s
sys 0m0.011s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.0041066 s, 5.0 MB/s

real    0m0.016s
user    0m0.004s
sys 0m0.008s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00605065 s, 3.4 MB/s

real    0m0.022s
user    0m0.002s
sys 0m0.014s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00448082 s, 4.6 MB/s

real    0m0.019s
user    0m0.002s
sys 0m0.011s
  • Read the files for third time :
# for i in {1..10}; do time dd if=/nfsmount/file$i of=/root/file$i bs=1K count=20; done

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.0041669 s, 4.9 MB/s

real    0m0.019s
user    0m0.003s
sys 0m0.009s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00443218 s, 4.6 MB/s

real    0m0.019s
user    0m0.002s
sys 0m0.011s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00404093 s, 5.1 MB/s

real    0m0.021s
user    0m0.003s
sys 0m0.011s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00406511 s, 5.0 MB/s

real    0m0.017s
user    0m0.001s
sys 0m0.012s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00689074 s, 3.0 MB/s

real    0m0.019s
user    0m0.004s
sys 0m0.009s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00440333 s, 4.7 MB/s

real    0m0.018s
user    0m0.004s
sys 0m0.009s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00393502 s, 5.2 MB/s

real    0m0.018s
user    0m0.002s
sys 0m0.011s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00400276 s, 5.1 MB/s

real    0m0.018s
user    0m0.002s
sys 0m0.011s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00576982 s, 3.5 MB/s

real    0m0.018s
user    0m0.002s
sys 0m0.010s

20+0 records in
20+0 records out
20480 bytes (20 kB) copied, 0.00393811 s, 5.2 MB/s

real    0m0.018s
user    0m0.003s
sys 0m0.009s
  • Results :
    Average speed for first Read operation = 1.9586 MB/s
    Average speed on second Read operation = 4.73 MB/s
    Average speed on third Read operation = 4.63 MB/s
  • The time required to read the file is subsequently reduced from around 0.095 sec to 0.021 sec for a file of 250 KB size.
    For a 20 KB file, it drops from 0.022 to 0.016 sec.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments