System locks up due to 'vmtoolsd' process freezing the root filesystem and then getting blocked reading from '/etc/mtab'

Solution Verified - Updated -

Issue

  • VMWare-virtualized system appears to lockup after VM backup is performed and the related VM snapshot is made. The system log shows that vmtoolsd process is blocked, trying to read from a file:
INFO: task vmtoolsd:2621 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
vmtoolsd      D ffff810001036420     0  2621      1          2708   647 (NOTLB)
Call Trace:
 [<ffffffff88031e5d>] :jbd:start_this_handle+0x140/0x36c
 [<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
 [<ffffffff88032152>] :jbd:journal_start+0xc9/0x100
 [<ffffffff88050c72>] :ext3:ext3_dirty_inode+0x28/0x7b
 [<ffffffff80013c93>] __mark_inode_dirty+0x29/0x16e
 [<ffffffff8000c44a>] do_generic_mapping_read+0x342/0x354
 [<ffffffff8000d0fb>] file_read_actor+0x0/0x159
 [<ffffffff8000c5a8>] __generic_file_aio_read+0x14c/0x198
 [<ffffffff80016dd4>] generic_file_aio_read+0x34/0x39
 [<ffffffff8000ce24>] do_sync_read+0xc7/0x104
 [<ffffffff8002a6d0>] __vma_link+0x42/0x4b
 [<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8000e17e>] do_mmap_pgoff+0x66c/0x7d7
 [<ffffffff8000b6b0>] vfs_read+0xcb/0x171
 [<ffffffff80011c01>] sys_read+0x45/0x6e
 [<ffffffff8005e116>] system_call+0x7e/0x83
  • There are number of other processes reported as blocked trying to write to a frozen filesystem:
INFO: task rsyslogd:3460 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
rsyslogd      D ffff810001036420     0  3460      1          3461  3458 (NOTLB)
Call Trace:
 [<ffffffff800163f7>] __generic_file_aio_write_nolock+0x11f/0x3b6
 [<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
 [<ffffffff80021763>] generic_file_aio_write+0x65/0xc1
 [<ffffffff8804c1b6>] :ext3:ext3_file_write+0x16/0x91
 [<ffffffff80018266>] do_sync_write+0xc7/0x104
 [<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
 [<ffffffff800634b6>] __sched_text_start+0xf6/0xbd6
 [<ffffffff80063ff8>] thread_return+0x62/0xfe
 [<ffffffff80016a49>] vfs_write+0xce/0x174
 [<ffffffff80017316>] sys_write+0x45/0x6e
 [<ffffffff8005e28d>] tracesys+0xd5/0xe0

INFO: task dd:6316 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
dd            D ffff81000103eaa0     0  6316   6259                     (NOTLB)
Call Trace:
 [<ffffffff800163f7>] __generic_file_aio_write_nolock+0x11f/0x3b6
 [<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8008f9cb>] __cond_resched+0x1c/0x44
 [<ffffffff80021763>] generic_file_aio_write+0x65/0xc1
 [<ffffffff8804c1b6>] :ext3:ext3_file_write+0x16/0x91
 [<ffffffff80018266>] do_sync_write+0xc7/0x104
 [<ffffffff800a1ba4>] autoremove_wake_function+0x0/0x2e
 [<ffffffff8002f074>] __clear_user+0x12/0x50
 [<ffffffff801a6af2>] read_zero+0x1cc/0x225
 [<ffffffff80016a49>] vfs_write+0xce/0x174
 [<ffffffff80017316>] sys_write+0x45/0x6e
 [<ffffffff8005e28d>] tracesys+0xd5/0xe0

Environment

  • Red Hat Enterprise Linux 5.10 and earlier
  • Kernel version prior to kernel-2.6.18-398.el5
  • vmtoolsd daemon running

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content