system hang with negative NR_WRITEBACK counter

Solution Verified - Updated -

Issue

  • System hung and numerous tasks in 'D' state with /proc/zoneinfo showing the NR_WRITEBACK counter less than zero and high NR_DIRTY:
crash> kmem -z
NODE: 0  ZONE: 2  ADDR: ffff810000053600  NAME: "Normal"
  SIZE: 15892480  PRESENT: 15675200  MIN/LOW/HIGH: 7707/9633/11560
  NR_ACTIVE: 2405635  NR_INACTIVE: 894899  FREE: 1574933
  VM_STAT:
      NR_ANON_PAGES: 611835
     NR_FILE_MAPPED: 29430
      NR_FILE_PAGES: 2690400
            NR_SLAB: 108743
       NR_PAGETABLE: 14111
      NR_FILE_DIRTY: 252
       NR_WRITEBACK: -9 <<----------- negative
    NR_UNSTABLE_NFS: 0
          NR_BOUNCE: 0
           NUMA_HIT: 1184945616
          NUMA_MISS: 0
       NUMA_FOREIGN: 0
NUMA_INTERLEAVE_HIT: 139442
         NUMA_LOCAL: 1184945616
         NUMA_OTHER: 0
  ALL_UNRECLAIMABLE: no  PAGES_SCANNED: 0  
  • Several hung tasks waiting on I/O, e.g.
 [<ffffffff8006f1f5>] do_gettimeofday+0x40/0x90
 [<ffffffff80028adc>] sync_page+0x0/0x43
 [<ffffffff800647ea>] io_schedule+0x3f/0x67
 [<ffffffff80028b1a>] sync_page+0x3e/0x43
 [<ffffffff8006492e>] __wait_on_bit_lock+0x36/0x66
 [<ffffffff8003ff92>] __lock_page+0x5e/0x64
 [<ffffffff800a1bd2>] wake_bit_function+0x0/0x23
 [<ffffffff8000c2e7>] do_generic_mapping_read+0x1df/0x354
 [<ffffffff8000d0fb>] file_read_actor+0x0/0x159
 [<ffffffff8000c5a8>] __generic_file_aio_read+0x14c/0x198
 [<ffffffff80016dd4>] generic_file_aio_read+0x34/0x39
 [<ffffffff8000ce24>] do_sync_read+0xc7/0x104
 [<ffffffff80067b88>] do_page_fault+0x4fe/0x874
 [<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8000e17e>] do_mmap_pgoff+0x66c/0x7d7
 [<ffffffff8000b6b0>] vfs_read+0xcb/0x171
 [<ffffffff80011c01>] sys_read+0x45/0x6e
 [<ffffffff8005e28d>] tracesys+0xd5/0xe0

Environment

  • RHEL 5.5GA kernel 2.6.18-194.el5

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content