system hang with negative NR_WRITEBACK counter

Solution Verified - Updated -

Issue

  • System hung and numerous tasks in 'D' state with /proc/zoneinfo showing the NR_WRITEBACK counter less than zero and high NR_DIRTY:
crash> kmem -z
NODE: 0  ZONE: 2  ADDR: ffff810000053600  NAME: "Normal"
  SIZE: 15892480  PRESENT: 15675200  MIN/LOW/HIGH: 7707/9633/11560
  NR_ACTIVE: 2405635  NR_INACTIVE: 894899  FREE: 1574933
  VM_STAT:
      NR_ANON_PAGES: 611835
     NR_FILE_MAPPED: 29430
      NR_FILE_PAGES: 2690400
            NR_SLAB: 108743
       NR_PAGETABLE: 14111
      NR_FILE_DIRTY: 252
       NR_WRITEBACK: -9 <<----------- negative
    NR_UNSTABLE_NFS: 0
          NR_BOUNCE: 0
           NUMA_HIT: 1184945616
          NUMA_MISS: 0
       NUMA_FOREIGN: 0
NUMA_INTERLEAVE_HIT: 139442
         NUMA_LOCAL: 1184945616
         NUMA_OTHER: 0
  ALL_UNRECLAIMABLE: no  PAGES_SCANNED: 0  
  • Several hung tasks waiting on I/O, e.g.
 [<ffffffff8006f1f5>] do_gettimeofday+0x40/0x90
 [<ffffffff80028adc>] sync_page+0x0/0x43
 [<ffffffff800647ea>] io_schedule+0x3f/0x67
 [<ffffffff80028b1a>] sync_page+0x3e/0x43
 [<ffffffff8006492e>] __wait_on_bit_lock+0x36/0x66
 [<ffffffff8003ff92>] __lock_page+0x5e/0x64
 [<ffffffff800a1bd2>] wake_bit_function+0x0/0x23
 [<ffffffff8000c2e7>] do_generic_mapping_read+0x1df/0x354
 [<ffffffff8000d0fb>] file_read_actor+0x0/0x159
 [<ffffffff8000c5a8>] __generic_file_aio_read+0x14c/0x198
 [<ffffffff80016dd4>] generic_file_aio_read+0x34/0x39
 [<ffffffff8000ce24>] do_sync_read+0xc7/0x104
 [<ffffffff80067b88>] do_page_fault+0x4fe/0x874
 [<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8000e17e>] do_mmap_pgoff+0x66c/0x7d7
 [<ffffffff8000b6b0>] vfs_read+0xcb/0x171
 [<ffffffff80011c01>] sys_read+0x45/0x6e
 [<ffffffff8005e28d>] tracesys+0xd5/0xe0

Environment

  • RHEL 5.5GA kernel 2.6.18-194.el5

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In