Sudo hangs when IO logging is enabled
Environment
- Red Hat Enterprise Linux (RHEL) 7.4
sudo-1.8.19p2
with IO logging enabled
Issue
When sudo
is configured with IO logging enabled and a command that produces output to terminal is run, it does sometimes hang in a poll()
system call indefinitely. Only sending a SIGKILL
to the main sudo
process terminates it.
Sometimes the child process (the actual command being run in sudo
) is left in a zombie (Z
) state.
Resolution
Update to sudo-1.8.19p2-14.el7_5
shipped with Advisory RHBA-2018:1992 or newer.
Before the fix was available temporarily disabling IO logging in /etc/sudoers
using visudo
served as a workaround.
Root Cause
The issue is a blocking poll()
system call for input from master PTY /dev/ptmx
(the PTY multiplexer
) that will not time out. It occurs once the child process terminates, and the input and output buffers are being flushed.
The root cause is the open master PTY
file descriptor at the time of the buffer flushing and a subsequent blocking attempt to read output from the already terminated child process.
Diagnostic Steps
-
Verify in
/etc/sudoers
, i.e. usingvisudo
, that you have IO logging enabled. Look forlog_input
andlog_output
keywords. Configuration of IO logging may look similar to the following:Defaults iolog_dir=/var/log/sudo-io/%{user}, log_input, log_output
-
If you have
gdb
and the appropriatedebuginfo
packages installed, a backtrace of the hangedsudo
process can be analyzed. You can get a backtrace of a running process using the following command, where$PID
is the PID of the hangedsudo
process:# gdb `which sudo` $PID
A backtrace of the hanged process will look similar to the following:
# gdb `which sudo` 2516 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7 Copyright (C) 2013 Free Software Foundation, Inc. ... lines omitted ... 0x00007fd72683ca20 in __poll_nocancel () at ../sysdeps/unix/syscall-template.S:81 81 T_PSEUDO (SYSCALL_SYMBOL, SYSCALL_NAME, SYSCALL_NARGS) (gdb) bt #0 0x00007fd72683ca20 in __poll_nocancel () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007fd726d21675 in poll (__timeout=<optimized out>, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/bits/poll2.h:46 #2 sudo_ev_scan_impl (base=base@entry=0x55ce7db9a9e0, flags=flags@entry=0) at ./event_poll.c:155 #3 0x00007fd726d1ac9e in sudo_ev_loop_v1 (base=base@entry=0x55ce7db9a9e0, flags=flags@entry=0) at ./event.c:302 #4 0x000055ce7c04a5ba in del_io_events (nonblocking=nonblocking@entry=false) at ./exec_pty.c:1045 #5 0x000055ce7c04e142 in pty_close (cstat=cstat@entry=0x7ffe00867180) at ./exec_pty.c:895 #6 0x000055ce7c049917 in sudo_execute (details=details@entry=0x55ce7c262fa0 <command_details>, cstat=cstat@entry=0x7ffe00867180) at ./exec.c:537 #7 0x000055ce7c054222 in run_command (details=0x55ce7c262fa0 <command_details>) at ./sudo.c:1161 #8 0x000055ce7c04689c in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at ./sudo.c:311
-
Verify what file descriptors we are polling() for:
(gdb) up #1 0x00007fd726d21675 in poll (__timeout=<optimized out>, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/bits/poll2.h:46 46 /usr/include/bits/poll2.h: No such file or directory. (gdb) up #2 sudo_ev_scan_impl (base=base@entry=0x55ce7db9a9e0, flags=flags@entry=0) at ./event_poll.c:155 155 nready = poll(base->pfds, base->pfd_high + 1, timeout); (gdb) print base->pfd_high $15 = 0 (gdb) print base->pfds[0] $18 = {fd = 14, events = 1, revents = 0}
-
Check what file the file descriptor belongs to. You can do that by issuing
info os files
ingdb
and looking forsudo
process, or you can checklsof
. For example by running the follwing command (ourPID
was2516
, andfd
=14
):# lsof | grep 2516 | grep '14' | grep 'CHR' | awk '{ print $1,$2,$4,$9 }' sudo 2516 14u /dev/ptmx
-
If the file descriptor belongs to
/dev/ptmx
, then you are likely hitting this issue.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments