Ceph cluster status shows slow request when scrubing and deep-scrubing

Issue

• Ceph -s shows slow request
• IO commit to kv latency

2021-04-19 04:32:40.431 7f3d87c82700  0 bluestore(/var/lib/ceph/osd/ceph-9) log_latency slow operation observed for kv_final, latency = 54.4763s
2021-04-19 04:32:40.431 7f3d87c82700  0 bluestore(/var/lib/ceph/osd/ceph-9) log_latency_fn slow operation observed for _txc_committed_kv, latency = 54.4745s, txc = 0x55aced9bec00
2021-04-19 04:32:40.431 7f3d87c82700  0 bluestore(/var/lib/ceph/osd/ceph-9) log_latency_fn slow operation observed for _txc_committed_kv, latency = 54.2247s, txc = 0x55acdb6d3800
2021-04-19 04:32:40.431 7f3d87c82700  0 bluestore(/var/lib/ceph/osd/ceph-9) log_latency_fn slow operation observed for _txc_committed_kv, latency = 42.8612s, txc = 0x55acdc8eb500

• slow request in osd log

2021-04-19 04:32:16.642 -1 osd.9 74415 get_health_metrics reporting 2 slow ops, oldest is osd_repop
2021-04-19 04:32:17.688 -1 osd.9 74415 get_health_metrics reporting 2 slow ops, oldest is osd_repop
2021-04-19 04:32:18.643 -1 osd.9 74415 get_health_metrics reporting 2 slow ops, oldest is osd_repop
2021-04-19 04:32:19.662 -1 osd.9 74415 get_health_metrics reporting 2 slow ops, oldest is osd_repop
2021-04-19 04:32:20.655 -1 osd.9 74415 get_health_metrics reporting 2 slow ops, oldest is osd_repop
2021-04-19 04:32:21.643 -1 osd.9 74415 get_health_metrics reporting 2 slow ops, oldest is osd_repop

• heartbeat failed

2021-04-19 03:49:17.390 7f3d8f491700 -1 osd.9 74415 heartbeat_check: no reply from 192.168.0.101:6828 osd.85 since back 2021-04-19 03:48:17.741472 front 2021-04-19 03:48:17.741328 (oldest deadline 2021-04-19 03:48:41.838336)
2021-04-19 03:49:18.378 7f3d8f491700 -1 osd.9 74415 heartbeat_check: no reply from 192.168.0.101:6828 osd.85 since back 2021-04-19 03:48:17.741472 front 2021-04-19 03:48:17.741328 (oldest deadline 2021-04-19 03:48:41.838336)
2021-04-19 03:49:19.370 7f3d8f491700 -1 osd.9 74415 heartbeat_check: no reply from 192.168.0.101:6828 osd.85 since back 2021-04-19 03:48:17.741472 front 2021-04-19 03:48:17.741328 (oldest deadline 2021-04-19 03:48:41.838336)