RHEL6.2: Dell Poweredge R310 hung, a lot of processes in wait_for_completion, waiting on generic 'events/*' thread to continue executing

Issue

We noticed one of our RHEL Servers having a higher than usual load. The load was actually growing, however the CPU was fairly idle and applications were responsive.

Upon further investigating, I noticed some processes stuck in a D state (e.g. [bash] and sshd: root@pts/1). There were others as well.

After a few minutes new SSH/Bash sessions locked up and could not be killed.

Eventually we had to reboot the system, but not before I had a vmcore generated (which I will attach).