Why we get the error "failed to flush rbd_data.XXXX: (16) Device or resource busy" when we try to evict data from the cache tier in Ceph?

Issue

When trying to evict data from the cache tier using the following command:

$ rados -p volumes_ssd_tier cache-flush-evict-all

It fails with the following error:

failed to flush rbd_data.XXXX: (16) Device or resource busy

NOTE:

Using cache-tiering in production is not supported by Red Hat and it will be deprecated in Red Hat Ceph Storage 2.0.