system hang with negative NR_WRITEBACK counter
Issue
- System hung and numerous tasks in 'D' state with /proc/zoneinfo showing the NR_WRITEBACK counter less than zero and high NR_DIRTY:
crash> kmem -z
NODE: 0 ZONE: 2 ADDR: ffff810000053600 NAME: "Normal"
SIZE: 15892480 PRESENT: 15675200 MIN/LOW/HIGH: 7707/9633/11560
NR_ACTIVE: 2405635 NR_INACTIVE: 894899 FREE: 1574933
VM_STAT:
NR_ANON_PAGES: 611835
NR_FILE_MAPPED: 29430
NR_FILE_PAGES: 2690400
NR_SLAB: 108743
NR_PAGETABLE: 14111
NR_FILE_DIRTY: 252
NR_WRITEBACK: -9 <<----------- negative
NR_UNSTABLE_NFS: 0
NR_BOUNCE: 0
NUMA_HIT: 1184945616
NUMA_MISS: 0
NUMA_FOREIGN: 0
NUMA_INTERLEAVE_HIT: 139442
NUMA_LOCAL: 1184945616
NUMA_OTHER: 0
ALL_UNRECLAIMABLE: no PAGES_SCANNED: 0
- Several hung tasks waiting on I/O, e.g.
[<ffffffff8006f1f5>] do_gettimeofday+0x40/0x90
[<ffffffff80028adc>] sync_page+0x0/0x43
[<ffffffff800647ea>] io_schedule+0x3f/0x67
[<ffffffff80028b1a>] sync_page+0x3e/0x43
[<ffffffff8006492e>] __wait_on_bit_lock+0x36/0x66
[<ffffffff8003ff92>] __lock_page+0x5e/0x64
[<ffffffff800a1bd2>] wake_bit_function+0x0/0x23
[<ffffffff8000c2e7>] do_generic_mapping_read+0x1df/0x354
[<ffffffff8000d0fb>] file_read_actor+0x0/0x159
[<ffffffff8000c5a8>] __generic_file_aio_read+0x14c/0x198
[<ffffffff80016dd4>] generic_file_aio_read+0x34/0x39
[<ffffffff8000ce24>] do_sync_read+0xc7/0x104
[<ffffffff80067b88>] do_page_fault+0x4fe/0x874
[<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
[<ffffffff8000e17e>] do_mmap_pgoff+0x66c/0x7d7
[<ffffffff8000b6b0>] vfs_read+0xcb/0x171
[<ffffffff80011c01>] sys_read+0x45/0x6e
[<ffffffff8005e28d>] tracesys+0xd5/0xe0
Environment
- RHEL 5.5GA kernel 2.6.18-194.el5
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.