On Red Hat Enterprise Linux 5.9, automount leaves stale entries in /net during umount's

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 5.9
  • kernel 2.6.18-348.1.1.el5
  • autofs 5.0.1-0.rc2.177.el5

Issue

  • autofs fails to unmount entries in /net the directory will appear to exist but will be completely empty.
  • Changing to a recently unmounted /net directory and running an ls produces the error:
 # ls 
 ls: .: No such file or directory

Resolution

Workaround

  • It is possible to recover from this behavior by issuing a forcerestart against autofs: 
 # service autofs forcerestart

Note: The forcerestart option typically is not recommended except as a last resort.

  • If you find that you have to restart autofs too often downgrading it also works :
# yum downgrade autofs

Root Cause

  • This issue is due to a problem with how autofs is handling a tree of mounts if an intermediate share isn't accessible by the client. autofs can incorrectly think a mountpoint and the mounts underneath it are not in use, causing autofs to prematurely try and release a set of mounts. However, part of the umount process will fail due to a mount actually being busy, leaving autofs in an errant, unexpected state.

Diagnostic Steps

  • Enable autofs debugging by editing /etc/sysconfig/autofs and modifiying the LOGGING= line to debug : 
LOGGING="debug"
  • Restart autofs to introduce the changes:
 # service autofs restart
  • Reproduce the issue and tail /var/log/messages the following types of messages should be present. Note the umount_autofs_offset: offset /net/server/export1/home1/dir1/subdir1 not mounted and rmdir_path: lstat of server/export1/users failed messages specifically:
Mar 11 17:12:39 client automount[6446]: st_expire: state 1 path /net
Mar 11 17:12:39 client automount[6446]: expire_proc: exp_proc = 47573664651584 path /net
Mar 11 17:12:39 client automount[6446]: handle_packet: type = 4
Mar 11 17:12:39 client automount[6446]: handle_packet_expire_indirect: token 24, name server
Mar 11 17:12:39 client automount[6446]: expiring path /net/server
Mar 11 17:12:39 client automount[6446]: umount_multi: path /net/server incl 1
Mar 11 17:12:39 client automount[6446]: umount_multi_triggers: umount offset /net/server/export0
Mar 11 17:12:39 client automount[6446]: umount_autofs_offset: offset /net/server/export0 not mounted
Mar 11 17:12:39 client automount[6446]: rmdir_path: lstat of server/export0 failed
Mar 11 17:12:39 client automount[6446]: umount_multi_triggers: umount offset /net/server/export/opt/foo
Mar 11 17:12:39 client automount[6446]: umount_autofs_offset: offset /net/server/export/opt/foo not mounted
Mar 11 17:12:39 client automount[6446]: rmdir_path: lstat of server/export/opt/foo failed
Mar 11 17:12:39 client automount[6446]: umount_multi_triggers: umount offset /net/server/export1/home1/dir1/subdir2
Mar 11 17:12:39 client automount[6446]: umount_autofs_offset: offset /net/server/export1/home1/dir1/subdir2 not mounted
Mar 11 17:12:39 client automount[6446]: umount_multi_triggers: umount offset /net/server/export1/home1/dir1/subdir3
Mar 11 17:12:39 client automount[6446]: umount_autofs_offset: offset /net/server/export1/home1/dir1/subdir3 not mounted
Mar 11 17:12:39 client automount[6446]: umount_multi_triggers: umount offset /net/server/export1/home1/dir1/subdir4
Mar 11 17:12:39 client automount[6446]: umount_autofs_offset: offset /net/server/export1/home1/dir1/subdir4 not mounted
Mar 11 17:12:39 client automount[6446]: umount_multi_triggers: umount offset /net/server/export1/home1/dir1/subdir5
Mar 11 17:12:39 client automount[6446]: umounted offset mount /net/server/export1/home1/dir1/subdir5
Mar 11 17:12:39 client automount[6446]: umount_multi_triggers: umount offset /net/server/export1/home1/dir1/subdir1
Mar 11 17:12:39 client automount[6446]: umount_autofs_offset: offset /net/server/export1/home1/dir1/subdir1 not mounted
Mar 11 17:12:39 client automount[6446]: umount_multi_triggers: umount offset /net/server/export1/home1/dir1/subdir6
Mar 11 17:12:39 client automount[6446]: umount_autofs_offset: offset /net/server/export1/home1/dir1/subdir6 not mounted
Mar 11 17:12:39 client automount[6446]: umount_multi_triggers: umount offset /net/server/export1/home1/dir1/incoming
Mar 11 17:12:39 client automount[6446]: umount_autofs_offset: offset /net/server/export1/home1/dir1/incoming not mounted
Mar 11 17:12:39 client automount[6446]: umount_multi_triggers: umount offset /net/server/export1/home1/dir1/outbound
Mar 11 17:12:39 client automount[6446]: umount_autofs_offset: offset /net/server/export1/home1/dir1/outbound not mounted
Mar 11 17:12:39 client automount[6446]: umount_multi_triggers: umount offset /net/server/export1/home1/dir1
Mar 11 17:12:39 client automount[6446]: failed to umount offset
Mar 11 17:12:39 client automount[6446]: umount_multi_triggers: umount offset /net/server/export1/users
Mar 11 17:12:39 client automount[6446]: umount_autofs_offset: offset /net/server/export1/users not mounted
Mar 11 17:12:39 client automount[6446]: rmdir_path: lstat of server/export1/users failed
Mar 11 17:12:39 client automount[6446]: umount_multi_triggers: umount offset /net/server/export2/home2
Mar 11 17:12:39 client automount[6446]: umount_autofs_offset: offset /net/server/export2/home2 not mounted
Mar 11 17:12:39 client automount[6446]: rmdir_path: lstat of server/export2/home2 failed
Mar 11 17:12:39 client automount[6446]: some offset mounts still present under /net/server
Mar 11 17:12:39 client automount[6446]: couldn't complete expire of /net/server
Mar 11 17:12:39 client automount[6446]: dev_ioctl_send_fail: token = 24
Mar 11 17:12:39 client automount[6446]: expire_cleanup: got thid 47573664651584 path /net stat 2
Mar 11 17:12:39 client automount[6446]: expire_cleanup: sigchld: exp 47573664651584 finished, dir1itching from 2 to 1
Mar 11 17:12:39 client automount[6446]: st_ready: st_ready(): state = 2 path /net

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.