System locks up due to 'vmtoolsd' process freezing the root filesystem and then getting blocked reading from '/etc/mtab'
Issue
- VMWare-virtualized system appears to lockup after VM backup is performed and the related VM snapshot is made. The system log shows that
vmtoolsdprocess is blocked, trying to read from a file:
INFO: task vmtoolsd:2621 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
vmtoolsd D ffff810001036420 0 2621 1 2708 647 (NOTLB)
Call Trace:
[<ffffffff88031e5d>] :jbd:start_this_handle+0x140/0x36c
[<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
[<ffffffff88032152>] :jbd:journal_start+0xc9/0x100
[<ffffffff88050c72>] :ext3:ext3_dirty_inode+0x28/0x7b
[<ffffffff80013c93>] __mark_inode_dirty+0x29/0x16e
[<ffffffff8000c44a>] do_generic_mapping_read+0x342/0x354
[<ffffffff8000d0fb>] file_read_actor+0x0/0x159
[<ffffffff8000c5a8>] __generic_file_aio_read+0x14c/0x198
[<ffffffff80016dd4>] generic_file_aio_read+0x34/0x39
[<ffffffff8000ce24>] do_sync_read+0xc7/0x104
[<ffffffff8002a6d0>] __vma_link+0x42/0x4b
[<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
[<ffffffff8000e17e>] do_mmap_pgoff+0x66c/0x7d7
[<ffffffff8000b6b0>] vfs_read+0xcb/0x171
[<ffffffff80011c01>] sys_read+0x45/0x6e
[<ffffffff8005e116>] system_call+0x7e/0x83
- There are number of other processes reported as blocked trying to write to a frozen filesystem:
INFO: task rsyslogd:3460 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rsyslogd D ffff810001036420 0 3460 1 3461 3458 (NOTLB)
Call Trace:
[<ffffffff800163f7>] __generic_file_aio_write_nolock+0x11f/0x3b6
[<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
[<ffffffff80021763>] generic_file_aio_write+0x65/0xc1
[<ffffffff8804c1b6>] :ext3:ext3_file_write+0x16/0x91
[<ffffffff80018266>] do_sync_write+0xc7/0x104
[<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
[<ffffffff800634b6>] __sched_text_start+0xf6/0xbd6
[<ffffffff80063ff8>] thread_return+0x62/0xfe
[<ffffffff80016a49>] vfs_write+0xce/0x174
[<ffffffff80017316>] sys_write+0x45/0x6e
[<ffffffff8005e28d>] tracesys+0xd5/0xe0
INFO: task dd:6316 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
dd D ffff81000103eaa0 0 6316 6259 (NOTLB)
Call Trace:
[<ffffffff800163f7>] __generic_file_aio_write_nolock+0x11f/0x3b6
[<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
[<ffffffff8008f9cb>] __cond_resched+0x1c/0x44
[<ffffffff80021763>] generic_file_aio_write+0x65/0xc1
[<ffffffff8804c1b6>] :ext3:ext3_file_write+0x16/0x91
[<ffffffff80018266>] do_sync_write+0xc7/0x104
[<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
[<ffffffff8002f074>] __clear_user+0x12/0x50
[<ffffffff801a6af2>] read_zero+0x1cc/0x225
[<ffffffff80016a49>] vfs_write+0xce/0x174
[<ffffffff80017316>] sys_write+0x45/0x6e
[<ffffffff8005e28d>] tracesys+0xd5/0xe0
Environment
- Red Hat Enterprise Linux 5.10 and earlier
- Kernel version prior to
kernel-2.6.18-398.el5 vmtoolsddaemon running
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
