When fio is used for performance testing, the average I/O latency is normal and the maximum I/O latency exceeds 2 seconds

Solution In Progress - Updated -

Issue

I/O request is slow. So using fio to test the performance of Ceph cluster.
Here are the fiO test results:

Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=36.7MiB/s][r=0,w=9384 IOPS][eta 00m:00s]
4randwrite: (groupid=0, jobs=1): err= 0: pid=1116460: Wed Sep 25 18:22:31 2019
  write: IOPS=5428, BW=21.2MiB/s (22.2MB/s)(3823MiB/180302msec)
    slat (nsec): min=968, max=517429, avg=2960.57, stdev=4274.86
    clat (usec): min=850, max=1569.3k, avg=5840.28, stdev=25725.92
     lat (usec): min=862, max=1569.3k, avg=5843.24, stdev=25725.94

The average latency for IO requests is 5843 usec. But the maximum latency for IO requests is 1569.3k usec. So the latency of IO requests is burry. Performance is unstable in the production environment.

Environment

Red Hat Ceph Storage 3.3
Red Hat Enterprise Linux 7.6

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content