Ceph - Bluestore OSD - bad performance, slow requests, logs reporting "_txc_state_proc slow aio_wait"

Issue

We are observing slow requests and poor performance, the ceph-osd logs or journalctl logs are showing messages like:

...
Jun 15 20:01:11 <node_name> conmon[249878]: 2021-06-15 20:01:11.013 7f381231d700  0 bluestore(/var/lib/ceph/osd/ceph-<ID>) _txc_state_proc slow aio_wait, txc = 0x558900498600, latency = 35.602411
Jun 15 20:01:14 <node_name> conmon[249878]: 2021-06-15 20:01:14.196 7f381231d700  0 bluestore(/var/lib/ceph/osd/ceph-<ID>) _txc_state_proc slow aio_wait, txc = 0x5588ff3bef00, latency = 38.855250
...