System locks up due to 'vmtoolsd' process freezing the root filesystem and then getting blocked reading from '/etc/mtab'
Issue
- VMWare-virtualized system appears to lockup after VM backup is performed and the related VM snapshot is made. The system log shows that
vmtoolsd
process is blocked, trying to read from a file:
INFO: task vmtoolsd:2621 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
vmtoolsd D ffff810001036420 0 2621 1 2708 647 (NOTLB)
Call Trace:
[<ffffffff88031e5d>] :jbd:start_this_handle+0x140/0x36c
[<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
[<ffffffff88032152>] :jbd:journal_start+0xc9/0x100
[<ffffffff88050c72>] :ext3:ext3_dirty_inode+0x28/0x7b
[<ffffffff80013c93>] __mark_inode_dirty+0x29/0x16e
[<ffffffff8000c44a>] do_generic_mapping_read+0x342/0x354
[<ffffffff8000d0fb>] file_read_actor+0x0/0x159
[<ffffffff8000c5a8>] __generic_file_aio_read+0x14c/0x198
[<ffffffff80016dd4>] generic_file_aio_read+0x34/0x39
[<ffffffff8000ce24>] do_sync_read+0xc7/0x104
[<ffffffff8002a6d0>] __vma_link+0x42/0x4b
[<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
[<ffffffff8000e17e>] do_mmap_pgoff+0x66c/0x7d7
[<ffffffff8000b6b0>] vfs_read+0xcb/0x171
[<ffffffff80011c01>] sys_read+0x45/0x6e
[<ffffffff8005e116>] system_call+0x7e/0x83
- There are number of other processes reported as blocked trying to write to a frozen filesystem:
INFO: task rsyslogd:3460 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rsyslogd D ffff810001036420 0 3460 1 3461 3458 (NOTLB)
Call Trace:
[<ffffffff800163f7>] __generic_file_aio_write_nolock+0x11f/0x3b6
[<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
[<ffffffff80021763>] generic_file_aio_write+0x65/0xc1
[<ffffffff8804c1b6>] :ext3:ext3_file_write+0x16/0x91
[<ffffffff80018266>] do_sync_write+0xc7/0x104
[<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
[<ffffffff800634b6>] __sched_text_start+0xf6/0xbd6
[<ffffffff80063ff8>] thread_return+0x62/0xfe
[<ffffffff80016a49>] vfs_write+0xce/0x174
[<ffffffff80017316>] sys_write+0x45/0x6e
[<ffffffff8005e28d>] tracesys+0xd5/0xe0
INFO: task dd:6316 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
dd D ffff81000103eaa0 0 6316 6259 (NOTLB)
Call Trace:
[<ffffffff800163f7>] __generic_file_aio_write_nolock+0x11f/0x3b6
[<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
[<ffffffff8008f9cb>] __cond_resched+0x1c/0x44
[<ffffffff80021763>] generic_file_aio_write+0x65/0xc1
[<ffffffff8804c1b6>] :ext3:ext3_file_write+0x16/0x91
[<ffffffff80018266>] do_sync_write+0xc7/0x104
[<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
[<ffffffff8002f074>] __clear_user+0x12/0x50
[<ffffffff801a6af2>] read_zero+0x1cc/0x225
[<ffffffff80016a49>] vfs_write+0xce/0x174
[<ffffffff80017316>] sys_write+0x45/0x6e
[<ffffffff8005e28d>] tracesys+0xd5/0xe0
Environment
- Red Hat Enterprise Linux 5.10 and earlier
- Kernel version prior to
kernel-2.6.18-398.el5
vmtoolsd
daemon running
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.