RHEL6: System hang occurred in congestion_wait() and shrink_inactive_list()

Issue

• What is legacy_scan_congestion_wait sysctl parameter ?

• Why is legacy_scan_congestion_wait sysctl parameter added in Red Hat Enterprise Linux 6 ?

• Server experiences system hang while some tasks are waiting in congestion_wait() after being called from shrink_inactive_list().

 Example 1: 
 PID: 8056   TASK: ffff880272a2c0c0  CPU: 1   COMMAND: "ntpd"
  #0 [ffff880257f87228] schedule at ffffffff814ed202
  #1 [ffff880257f872f0] schedule_timeout at ffffffff814ee042
  #2 [ffff880257f873a0] io_schedule_timeout at ffffffff814ecccf
  #3 [ffff880257f873d0] congestion_wait at ffffffff81134230
  #4 [ffff880257f87430] shrink_inactive_list at ffffffff8112ac47
  #5 [ffff880257f875e0] shrink_zone at ffffffff8112b5cf
  #6 [ffff880257f87690] do_try_to_free_pages at ffffffff8112b85e
  #7 [ffff880257f87720] try_to_free_pages at ffffffff8112be6d
  #8 [ffff880257f877d0] __alloc_pages_nodemask at ffffffff81123ded
  #9 [ffff880257f878f0] alloc_pages_current at ffffffff811585ca
 #10 [ffff880257f87920] __get_free_pages at ffffffff81121f8e
 #11 [ffff880257f87930] __pollwait at ffffffff8118bf14
 #12 [ffff880257f87970] datagram_poll at ffffffff81425f6a
 #13 [ffff880257f87990] udp_poll at ffffffff8148ba34
 #14 [ffff880257f879c0] sock_poll at ffffffff8141690a
 #15 [ffff880257f879d0] do_select at ffffffff8118bb42
 #16 [ffff880257f87d70] core_sys_select at ffffffff8118c94a
 #17 [ffff880257f87f10] sys_select at ffffffff8118ccd7
 #18 [ffff880257f87f80] system_call_fastpath at ffffffff8100b0f2

 Example 2:
 PID: 32675  TASK: ffff880102ce6100  CPU: 0   COMMAND: "sh"
  #0 [ffff8801cb01b528] schedule at ffffffff814ed202
  #1 [ffff8801cb01b5f0] schedule_timeout at ffffffff814ee042
  #2 [ffff8801cb01b6a0] io_schedule_timeout at ffffffff814ecccf
  #3 [ffff8801cb01b6d0] congestion_wait at ffffffff81134230
  #4 [ffff8801cb01b730] shrink_inactive_list at ffffffff8112ac47
  #5 [ffff8801cb01b8e0] shrink_zone at ffffffff8112b5cf
  #6 [ffff8801cb01b990] do_try_to_free_pages at ffffffff8112b85e
  #7 [ffff8801cb01ba20] try_to_free_pages at ffffffff8112be6d
  #8 [ffff8801cb01bad0] __alloc_pages_nodemask at ffffffff81123ded
  #9 [ffff8801cb01bbf0] alloc_pages_vma at ffffffff811586ca
 #10 [ffff8801cb01bc40] __do_fault at ffffffff8113b81b
 #11 [ffff8801cb01bcd0] handle_pte_fault at ffffffff8113bae7
 #12 [ffff8801cb01bdb0] handle_mm_fault at ffffffff8113c724
 #13 [ffff8801cb01be00] __do_page_fault at ffffffff81042c29
 #14 [ffff8801cb01bf20] do_page_fault at ffffffff814f2c8e
 #15 [ffff8801cb01bf50] page_fault at ffffffff814f0045