System hang due to I/O stuck in cfq scheduler

Solution In Progress - Updated -

Issue

System hung and was forced crashed. Various processes stuck waiting for I/O but no requests inflight to the disks:

crash> dev -d
MAJOR GENDISK            NAME       REQUEST_QUEUE      TOTAL ASYNC  SYNC   DRV
   11 ffff880532a73c00   sr0        ffff880532390e68       0     0     0     0
    8 ffff880532168400   sda        ffff880532390338     166   159     7     0
    8 ffff880532167000   sdb        ffff8805319c8ea8       0     0     0     0
    8 ffff880532167c00   sdc        ffff8805319c8378      17     5    12     0
    8 ffff880532066400   sdd        ffff8805319ccee8       0     0     0     0
    8 ffff880532066c00   sde        ffff8805319cc3b8       0     0     0     0
    8 ffff880532065400   sdf        ffff8805319d0f28       0     0     0     0
    8 ffff880532065c00   sdg        ffff8805319d03f8       0     0     0     0
    8 ffff880532063400   sdh        ffff880531afefa8       0     0     0     0
    8 ffff880532063c00   sdi        ffff880531afe478       0     0     0     0
    8 ffff880532062400   sdj        ffff880531b00fe8       0     0     0     0
  253 ffff880532000000   dm-0       ffff880531497028       0     0     0     0
  253 ffff88053212a400   dm-1       ffff880531b004b8       0     0     0     0

Environment

  • Redhat Enterprise Linux 6.4
  • CFQ scheduler

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.