dmcache and writeback performance

Issue

• We are trying to use the dmcache feature in RHEL 7.1 with write back.
• The SSD is capable of 20k IOPS (Intel DC S3700)
• The command used to create the cache device :

# /usr/sbin/lvconvert --yes --type cache-pool --cachemode writeback /dev/${volume_group}/cachedisk --poolmetadata /dev/${volume_group}/metadisk
# /usr/sbin/lvconvert --force --type cache /dev/${volume_group}/backingdisk --cachepool /dev/${volume_group}/cachedisk

• We getting really bad performance:

# fio --name myfile --rw=randwrite --size 256M --direct=1 

# cat 017856f0-bacf-4e55-a0b7-55bc1554b373.write.txt
8d3ab868-19a3-4aa3-9f79-a0e5dd300718/017856f0-bacf-4e55-a0b7-55bc1554b373/017856f0-bacf-4e55-a0b7-55bc1554b373: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.1.11
Starting 1 process
8d3ab868-19a3-4aa3-9f79-a0e5dd300718/017856f0-bacf-4e55-a0b7-55bc1554b373/017856f0-bacf-4e55-a0b7-55bc1554b373: Laying out IO file(s) (1 file(s) / 256MB)

8d3ab868-19a3-4aa3-9f79-a0e5dd300718/017856f0-bacf-4e55-a0b7-55bc1554b373/017856f0-bacf-4e55-a0b7-55bc1554b373: (groupid=0, jobs=1): err= 0: pid=23136: Mon Jun  8 09:56:13 2015
  write: io=262144KB, bw=1001.8KB/s, iops=250, runt=261682msec
    clat (usec): min=54, max=396738, avg=3989.71, stdev=13373.73
     lat (usec): min=54, max=396739, avg=3989.97, stdev=13373.73
    clat percentiles (usec):
     |  1.00th=[   57],  5.00th=[   63], 10.00th=[   77], 20.00th=[  175],
     | 30.00th=[  253], 40.00th=[  884], 50.00th=[ 1592], 60.00th=[ 1896],
     | 70.00th=[ 2128], 80.00th=[ 2352], 90.00th=[ 5024], 95.00th=[15424],
     | 99.00th=[54016], 99.50th=[71168], 99.90th=[203776], 99.95th=[240640],
     | 99.99th=[309248]
    bw (KB  /s): min=   92, max= 2616, per=100.00%, avg=1008.24, stdev=435.20
    lat (usec) : 100=13.98%, 250=15.79%, 500=3.47%, 750=3.38%, 1000=6.50%
    lat (msec) : 2=21.28%, 4=24.72%, 10=4.06%, 20=2.29%, 50=2.80%
    lat (msec) : 100=1.48%, 250=0.21%, 500=0.04%
  cpu          : usr=0.11%, sys=1.09%, ctx=65767, majf=0, minf=45
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=0/w=65536/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: io=262144KB, aggrb=1001KB/s, minb=1001KB/s, maxb=1001KB/s, mint=261682msec, maxt=261682msec

Disk stats (read/write):
    dm-3: ios=0/65794, merge=0/0, ticks=0/322100, in_queue=322100, util=98.78%, aggrios=9923/42678, aggrmerge=0/0, aggrticks=43153/519545, aggrin_queue=562698, aggrutil=95.71%
    dm-0: ios=25745/31745, merge=0/0, ticks=43632/20756, in_queue=64388, util=12.71%, aggrios=25703/59860, aggrmerge=50/4021, aggrticks=43508/22855, aggrin_queue=66333, aggrutil=13.22%
  sdc: ios=25703/59860, merge=50/4021, ticks=43508/22855, in_queue=66333, util=13.22%
  dm-1: ios=8/32136, merge=0/0, ticks=1/2586, in_queue=2587, util=0.54%
    dm-2: ios=4017/64155, merge=0/0, ticks=85826/1535294, in_queue=1621120, util=95.71%, aggrios=4017/64155, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
    md5: ios=4017/64155, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=2008/64772, aggrmerge=0/159, aggrticks=42908/958600, aggrin_queue=1001482, aggrutil=80.26%
  sda: ios=2815/64743, merge=0/184, ticks=62761/1252718, in_queue=1315444, util=80.26%
  sdb: ios=1202/64801, merge=0/134, ticks=23056/664482, in_queue=687521, util=50.85%


# smartctl -a /dev/sdc
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.10.0-229.4.2.el7.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, 



=== START OF INFORMATION SECTION ===
Model Family:     Intel DC S3700 Series SSDs
Device Model:     INTEL SSDSC2BA200G3
Serial Number:    BTTV429202KX200GGN
LU WWN Device Id: 5 5cd2e4 04b695a46
Firmware Version: 5DV10270
User Capacity:    200,049,647,616 bytes [200 GB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    Solid State Device
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 T13/2015-D revision 3
SATA Version is:  SATA 2.6, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Mon Jun  8 15:35:43 2015 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

• I was able to reproduce this on my desktop with completely different SSD and HDD.
• Could you try to reproduce with any SSD if you see the same issue?
• It maybe that writeback is not taken in account and is always doing write-through mode ?
• Do you have other reports for such behavior?