dmcache and writeback performance
Issue
- We are trying to use the dmcache feature in RHEL 7.1 with write back.
- The SSD is capable of 20k IOPS (Intel DC S3700)
- The command used to create the cache device :
# /usr/sbin/lvconvert --yes --type cache-pool --cachemode writeback /dev/${volume_group}/cachedisk --poolmetadata /dev/${volume_group}/metadisk
# /usr/sbin/lvconvert --force --type cache /dev/${volume_group}/backingdisk --cachepool /dev/${volume_group}/cachedisk
- We getting really bad performance:
# fio --name myfile --rw=randwrite --size 256M --direct=1
# cat 017856f0-bacf-4e55-a0b7-55bc1554b373.write.txt
8d3ab868-19a3-4aa3-9f79-a0e5dd300718/017856f0-bacf-4e55-a0b7-55bc1554b373/017856f0-bacf-4e55-a0b7-55bc1554b373: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=1
fio-2.1.11
Starting 1 process
8d3ab868-19a3-4aa3-9f79-a0e5dd300718/017856f0-bacf-4e55-a0b7-55bc1554b373/017856f0-bacf-4e55-a0b7-55bc1554b373: Laying out IO file(s) (1 file(s) / 256MB)
8d3ab868-19a3-4aa3-9f79-a0e5dd300718/017856f0-bacf-4e55-a0b7-55bc1554b373/017856f0-bacf-4e55-a0b7-55bc1554b373: (groupid=0, jobs=1): err= 0: pid=23136: Mon Jun 8 09:56:13 2015
write: io=262144KB, bw=1001.8KB/s, iops=250, runt=261682msec
clat (usec): min=54, max=396738, avg=3989.71, stdev=13373.73
lat (usec): min=54, max=396739, avg=3989.97, stdev=13373.73
clat percentiles (usec):
| 1.00th=[ 57], 5.00th=[ 63], 10.00th=[ 77], 20.00th=[ 175],
| 30.00th=[ 253], 40.00th=[ 884], 50.00th=[ 1592], 60.00th=[ 1896],
| 70.00th=[ 2128], 80.00th=[ 2352], 90.00th=[ 5024], 95.00th=[15424],
| 99.00th=[54016], 99.50th=[71168], 99.90th=[203776], 99.95th=[240640],
| 99.99th=[309248]
bw (KB /s): min= 92, max= 2616, per=100.00%, avg=1008.24, stdev=435.20
lat (usec) : 100=13.98%, 250=15.79%, 500=3.47%, 750=3.38%, 1000=6.50%
lat (msec) : 2=21.28%, 4=24.72%, 10=4.06%, 20=2.29%, 50=2.80%
lat (msec) : 100=1.48%, 250=0.21%, 500=0.04%
cpu : usr=0.11%, sys=1.09%, ctx=65767, majf=0, minf=45
IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued : total=r=0/w=65536/d=0, short=r=0/w=0/d=0
latency : target=0, window=0, percentile=100.00%, depth=1
Run status group 0 (all jobs):
WRITE: io=262144KB, aggrb=1001KB/s, minb=1001KB/s, maxb=1001KB/s, mint=261682msec, maxt=261682msec
Disk stats (read/write):
dm-3: ios=0/65794, merge=0/0, ticks=0/322100, in_queue=322100, util=98.78%, aggrios=9923/42678, aggrmerge=0/0, aggrticks=43153/519545, aggrin_queue=562698, aggrutil=95.71%
dm-0: ios=25745/31745, merge=0/0, ticks=43632/20756, in_queue=64388, util=12.71%, aggrios=25703/59860, aggrmerge=50/4021, aggrticks=43508/22855, aggrin_queue=66333, aggrutil=13.22%
sdc: ios=25703/59860, merge=50/4021, ticks=43508/22855, in_queue=66333, util=13.22%
dm-1: ios=8/32136, merge=0/0, ticks=1/2586, in_queue=2587, util=0.54%
dm-2: ios=4017/64155, merge=0/0, ticks=85826/1535294, in_queue=1621120, util=95.71%, aggrios=4017/64155, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00%
md5: ios=4017/64155, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=2008/64772, aggrmerge=0/159, aggrticks=42908/958600, aggrin_queue=1001482, aggrutil=80.26%
sda: ios=2815/64743, merge=0/184, ticks=62761/1252718, in_queue=1315444, util=80.26%
sdb: ios=1202/64801, merge=0/134, ticks=23056/664482, in_queue=687521, util=50.85%
# smartctl -a /dev/sdc
smartctl 6.2 2013-07-26 r3841 [x86_64-linux-3.10.0-229.4.2.el7.x86_64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke,
=== START OF INFORMATION SECTION ===
Model Family: Intel DC S3700 Series SSDs
Device Model: INTEL SSDSC2BA200G3
Serial Number: BTTV429202KX200GGN
LU WWN Device Id: 5 5cd2e4 04b695a46
Firmware Version: 5DV10270
User Capacity: 200,049,647,616 bytes [200 GB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: Solid State Device
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 T13/2015-D revision 3
SATA Version is: SATA 2.6, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Mon Jun 8 15:35:43 2015 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
- I was able to reproduce this on my desktop with completely different SSD and HDD.
- Could you try to reproduce with any SSD if you see the same issue?
- It maybe that writeback is not taken in account and is always doing write-through mode ?
- Do you have other reports for such behavior?
Environment
- Red Hat Enterprise Linux 7.1
- LVM dm-cache
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
