System locks up due to 'vmtoolsd' process freezing the root filesystem and then getting blocked reading from '/etc/mtab'

Solution Verified - Updated -

Issue

  • VMWare-virtualized system appears to lockup after VM backup is performed and the related VM snapshot is made. The system log shows that vmtoolsd process is blocked, trying to read from a file:
INFO: task vmtoolsd:2621 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
vmtoolsd      D ffff810001036420     0  2621      1          2708   647 (NOTLB)
Call Trace:
 [<ffffffff88031e5d>] :jbd:start_this_handle+0x140/0x36c
 [<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
 [<ffffffff88032152>] :jbd:journal_start+0xc9/0x100
 [<ffffffff88050c72>] :ext3:ext3_dirty_inode+0x28/0x7b
 [<ffffffff80013c93>] __mark_inode_dirty+0x29/0x16e
 [<ffffffff8000c44a>] do_generic_mapping_read+0x342/0x354
 [<ffffffff8000d0fb>] file_read_actor+0x0/0x159
 [<ffffffff8000c5a8>] __generic_file_aio_read+0x14c/0x198
 [<ffffffff80016dd4>] generic_file_aio_read+0x34/0x39
 [<ffffffff8000ce24>] do_sync_read+0xc7/0x104
 [<ffffffff8002a6d0>] __vma_link+0x42/0x4b
 [<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8000e17e>] do_mmap_pgoff+0x66c/0x7d7
 [<ffffffff8000b6b0>] vfs_read+0xcb/0x171
 [<ffffffff80011c01>] sys_read+0x45/0x6e
 [<ffffffff8005e116>] system_call+0x7e/0x83
  • There are number of other processes reported as blocked trying to write to a frozen filesystem:
INFO: task rsyslogd:3460 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rsyslogd      D ffff810001036420     0  3460      1          3461  3458 (NOTLB)
Call Trace:
 [<ffffffff800163f7>] __generic_file_aio_write_nolock+0x11f/0x3b6
 [<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
 [<ffffffff80021763>] generic_file_aio_write+0x65/0xc1
 [<ffffffff8804c1b6>] :ext3:ext3_file_write+0x16/0x91
 [<ffffffff80018266>] do_sync_write+0xc7/0x104
 [<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
 [<ffffffff800634b6>] __sched_text_start+0xf6/0xbd6
 [<ffffffff80063ff8>] thread_return+0x62/0xfe
 [<ffffffff80016a49>] vfs_write+0xce/0x174
 [<ffffffff80017316>] sys_write+0x45/0x6e
 [<ffffffff8005e28d>] tracesys+0xd5/0xe0

INFO: task dd:6316 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
dd            D ffff81000103eaa0     0  6316   6259                     (NOTLB)
Call Trace:
 [<ffffffff800163f7>] __generic_file_aio_write_nolock+0x11f/0x3b6
 [<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8008f9cb>] __cond_resched+0x1c/0x44
 [<ffffffff80021763>] generic_file_aio_write+0x65/0xc1
 [<ffffffff8804c1b6>] :ext3:ext3_file_write+0x16/0x91
 [<ffffffff80018266>] do_sync_write+0xc7/0x104
 [<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8002f074>] __clear_user+0x12/0x50
 [<ffffffff801a6af2>] read_zero+0x1cc/0x225
 [<ffffffff80016a49>] vfs_write+0xce/0x174
 [<ffffffff80017316>] sys_write+0x45/0x6e
 [<ffffffff8005e28d>] tracesys+0xd5/0xe0

Environment

  • Red Hat Enterprise Linux 5.10 and earlier
  • Kernel version prior to kernel-2.6.18-398.el5
  • vmtoolsd daemon running

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In