System hang due to I/O stuck in cfq scheduler
Issue
System hung and was forced crashed. Various processes stuck waiting for I/O but no requests inflight to the disks:
crash> dev -d
MAJOR GENDISK NAME REQUEST_QUEUE TOTAL ASYNC SYNC DRV
11 ffff880532a73c00 sr0 ffff880532390e68 0 0 0 0
8 ffff880532168400 sda ffff880532390338 166 159 7 0
8 ffff880532167000 sdb ffff8805319c8ea8 0 0 0 0
8 ffff880532167c00 sdc ffff8805319c8378 17 5 12 0
8 ffff880532066400 sdd ffff8805319ccee8 0 0 0 0
8 ffff880532066c00 sde ffff8805319cc3b8 0 0 0 0
8 ffff880532065400 sdf ffff8805319d0f28 0 0 0 0
8 ffff880532065c00 sdg ffff8805319d03f8 0 0 0 0
8 ffff880532063400 sdh ffff880531afefa8 0 0 0 0
8 ffff880532063c00 sdi ffff880531afe478 0 0 0 0
8 ffff880532062400 sdj ffff880531b00fe8 0 0 0 0
253 ffff880532000000 dm-0 ffff880531497028 0 0 0 0
253 ffff88053212a400 dm-1 ffff880531b004b8 0 0 0 0
Environment
- Redhat Enterprise Linux 6.4
- CFQ scheduler
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.