system hang with negative NR_WRITEBACK counter
Issue
- System hung and numerous tasks in 'D' state with /proc/zoneinfo showing the NR_WRITEBACK counter less than zero and high NR_DIRTY:
crash> kmem -z
NODE: 0 ZONE: 2 ADDR: ffff810000053600 NAME: "Normal"
SIZE: 15892480 PRESENT: 15675200 MIN/LOW/HIGH: 7707/9633/11560
NR_ACTIVE: 2405635 NR_INACTIVE: 894899 FREE: 1574933
VM_STAT:
NR_ANON_PAGES: 611835
NR_FILE_MAPPED: 29430
NR_FILE_PAGES: 2690400
NR_SLAB: 108743
NR_PAGETABLE: 14111
NR_FILE_DIRTY: 252
NR_WRITEBACK: -9 <<----------- negative
NR_UNSTABLE_NFS: 0
NR_BOUNCE: 0
NUMA_HIT: 1184945616
NUMA_MISS: 0
NUMA_FOREIGN: 0
NUMA_INTERLEAVE_HIT: 139442
NUMA_LOCAL: 1184945616
NUMA_OTHER: 0
ALL_UNRECLAIMABLE: no PAGES_SCANNED: 0
- Several hung tasks waiting on I/O, e.g.
[<ffffffff8006f1f5>] do_gettimeofday+0x40/0x90
[<ffffffff80028adc>] sync_page+0x0/0x43
[<ffffffff800647ea>] io_schedule+0x3f/0x67
[<ffffffff80028b1a>] sync_page+0x3e/0x43
[<ffffffff8006492e>] __wait_on_bit_lock+0x36/0x66
[<ffffffff8003ff92>] __lock_page+0x5e/0x64
[<ffffffff800a1bd2>] wake_bit_function+0x0/0x23
[<ffffffff8000c2e7>] do_generic_mapping_read+0x1df/0x354
[<ffffffff8000d0fb>] file_read_actor+0x0/0x159
[<ffffffff8000c5a8>] __generic_file_aio_read+0x14c/0x198
[<ffffffff80016dd4>] generic_file_aio_read+0x34/0x39
[<ffffffff8000ce24>] do_sync_read+0xc7/0x104
[<ffffffff80067b88>] do_page_fault+0x4fe/0x874
[<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
[<ffffffff8000e17e>] do_mmap_pgoff+0x66c/0x7d7
[<ffffffff8000b6b0>] vfs_read+0xcb/0x171
[<ffffffff80011c01>] sys_read+0x45/0x6e
[<ffffffff8005e28d>] tracesys+0xd5/0xe0
Environment
- RHEL 5.5GA kernel 2.6.18-194.el5
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
