Red Hat Ceph Storage 1.3 Long term slow requests not clearing on otherwise healthy cluster

Issue

We see the following slow requests stuck for over 37 hours when the cluster health is OK.
Why are these slow requests stuck?

Mon Nov 23 21:38:29 EST 2015
HEALTH_WARN 41 requests are blocked > 32 sec; 2 osds have slow requests
41 ops are blocked > 134218 sec
1 ops are blocked > 134218 sec on osd.120
40 ops are blocked > 134218 sec on osd.143
2 osds have slow requests

health HEALTH_WARN
41 requests are blocked > 32 sec
noscrub,nodeep-scrub flag(s) set
monmap e1: 3 mons at {boxen091=XXX.XXX.XXX.31:6789/0,boxen092=XXX.XXX.XXX.31:6789/0,boxen093=XXX.XXX.XXX.31:6789/0}
election epoch 52, quorum 0,1,2 boxen093,boxen092,boxen091
osdmap e95069: 478 osds: 478 up, 478 in
flags noscrub,nodeep-scrub
pgmap v6090904: 33248 pgs, 27 pools, 173 TB data, 56732 kobjects
520 TB used, 2087 TB / 2608 TB avail
33248 active+clean
client io 368 kB/s rd, 4141 kB/s wr, 785 op/s