Impact of Disk Speed on Satellite 6 Operations

I got pretty slow results when using the defaults (4K block size), but when bumping up the block size, things improved drastically.

[ root@rhsat01aplpd.rjf.com : Fri Aug 10, 08:18 AM : /root ] $ fio --name=job1 --rw=read --size=1g --directory=/var/lib/pulp --direct=1 --blocksize=4k READ: bw=15.3MiB/s (16.0MB/s), 15.3MiB/s-15.3MiB/s (16.0MB/s-16.0MB/s), io=150MiB (157MB), run=9803-9803msec

block size 16K:

[ root@rhsat01aplpd.rjf.com : Fri Aug 10, 08:35 AM : /root ] $ fio --name=job1 --rw=read --size=1g --directory=/var/lib/pulp --direct=1 --blocksize=16k Run status group 0 (all jobs): READ: bw=52.0MiB/s (55.6MB/s), 52.0MiB/s-52.0MiB/s (55.6MB/s-55.6MB/s), io=1024MiB (1074MB), run=19321-19321msec

block size = 1MB:

[ root@rhsat01aplpd.rjf.com : Fri Aug 10, 08:17 AM : /root ] $ fio --name=job1 --rw=read --size=1g --directory=/var/lib/pulp --direct=1 --blocksize=1024k Run status group 0 (all jobs): READ: bw=489MiB/s (513MB/s), 489MiB/s-489MiB/s (513MB/s-513MB/s), io=1024MiB (1074MB), run=2095-2095 msec

blocksize = 4MB:

[ root@rhsat01aplpd.rjf.com : Fri Aug 10, 08:18 AM : /root ] $ fio --name=job1 --rw=read --size=1g --directory=/var/lib/pulp --direct=1 --blocksize=4096k Run status group 0 (all jobs): READ: bw=828MiB/s (869MB/s), 828MiB/s-828MiB/s (869MB/s-869MB/s), io=1024MiB (1074MB), run=1236-1236msec

Can someone comment on how this will relate to Sat performance?

Thanks

Mike,

I am a little confused by all of this myself as I am seeing some very poor results. I tested using fio without "--blocksize=" set and then with it set for 4K and I saw wildly different results. When I looked at "man fio", though, it specifically says that the default blocksize value is 4096:

"   Block size
       blocksize=int[,int][,int], bs=int[,int][,int]
              The block size in bytes used for I/O units. Default: 4096"

Results with no "--blocksize=" value set:

# fio --name=job1 --rw=read --size=1g  --directory=/var/lib/pulp --direct=1
job1: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
READ: bw=8211KiB/s (8408kB/s), 8211KiB/s-8211KiB/s (8408kB/s-8408kB/s), io=1024MiB (1074MB), run=127701-127701msec

Results with "--blocksize=4K" value set:

# fio --name=job1 --rw=read --size=1g  --directory=/var/lib/pulp --direct=1 --blocksize=4k
job1: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
READ: bw=14.3MiB/s (15.0MB/s), 14.3MiB/s-14.3MiB/s (15.0MB/s-15.0MB/s), io=1024MiB (1074MB), run=71464-71464msec

Granted, both results fall far below the the required 60-80 Megabytes per second but I am still left wondering why there is such a large and repeatable discrepancy in results using no "--blocksize=" value (accepting the default of 4096 bytes) and setting a "--blocksize=4K" which equates to the default.

Is this the same behavior you have seen?

This article seems to effectively eliminate using a VM as a Satellite Server or capsule.

Not sure this will help, but here's some additional data points. Our satellite is a 16 vCPU/64GB virtual machine on VMWare. We're using pure storage ssd-only arrays.

You definitely need to set the block size to the blocksize of your device. You can make fio return some really nice results if you set blocksize much higher than what the device is configured for. If you did everything using defaults, you probably have a 4k block device.

Pure Storage recommends using the --ioengine=libaio flag for linux, but that's the standard async engine. Since Red Hat's original recommendations are for synchronous reads, change the ioengine to sync.

 fio --name TEST --directory /var/lib --filename=fio-tempfile.dat --rw=read --size=1g  --blocksize=4k --ioengine=sync --fsync=10000 --iodepth=32 --direct=1 --numjobs=8  --group_reporting

Run status group 0 (all jobs):
   READ: bw=98.8MiB/s (104MB/s), 98.8MiB/s-98.8MiB/s (104MB/s-104MB/s), io=8192MiB (8590MB), run=82925-82925msec

Based on the docs, since I'm using the sync engine, iodepth doesn't matter and since I'm only doing reads, the fsync is also not needed.

However, dropping the numjobs to 1 drops my results to a consistent ~10MB/s, so instead of what I thought were 8 parallel jobs, maybe it's 8 separate jobs and then the results are combined. Not sure how useful that is.

Anyway, doing all of this just showed to me that i/o in a virtual machine seems all over the place. I can make this tool give me good results, but I'm not entirely sure how to make it give me accurate results.

We've been using this satellite for over a year now with over 3000 content hosts and multiple capsules, so personally I think the concern in the docs about I/O is a little overblown.

VMDK?

I think this information is incorrect or out of date. The foreman-maintain scripts disk check says our disk is too slow, but uses dio. When I use it or do manually, my host fails.

When I turn dio off, my results are excellent.

The performance tuning guide for Satellite addresses this: https://access.redhat.com/node/5001161/40/0/17386571