When fio is used for performance testing, the average I/O latency is normal and the maximum I/O latency exceeds 2 seconds

Issue

I/O request is slow. So using fio to test the performance of Ceph cluster.
Here are the fiO test results:

Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=36.7MiB/s][r=0,w=9384 IOPS][eta 00m:00s]
4randwrite: (groupid=0, jobs=1): err= 0: pid=1116460: Wed Sep 25 18:22:31 2019
  write: IOPS=5428, BW=21.2MiB/s (22.2MB/s)(3823MiB/180302msec)
    slat (nsec): min=968, max=517429, avg=2960.57, stdev=4274.86
    clat (usec): min=850, max=1569.3k, avg=5840.28, stdev=25725.92
     lat (usec): min=862, max=1569.3k, avg=5843.24, stdev=25725.94

The average latency for IO requests is 5843 usec. But the maximum latency for IO requests is 1569.3k usec. So the latency of IO requests is burry. Performance is unstable in the production environment.