How can I gather straces from ssh and sshd?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 7, 8, 9, 10

Issue

How can I gather straces from ssh and sshd to attach to a support case, without disrupting existing ssh traffic to my server?

Resolution

We want to run a one-off sshd on a custom port so that we can capture maximally verbose output and gather an strace from start to stop.

We will arbitrarily choose port 2222 to listen on, although this can be another port if needed. We can also just use the default port 22 if we're able to temporarily disable the normal sshd and ensure that there won't be any unexpected connections to it.

We strongly recommend that you obtain packet captures concurrent with this procedure. Please see the relevant KCS article:

If firewalld is running, we'll need to allow port 2222 this way:

# firewall-cmd --add-port=2222/tcp
# firewall-cmd --add-port=2222/tcp --permanent

If you're using another firewall, you'll need to find the equivalent.

When run as root, the following will run sshd using all of the existing configuration options, but with maximally verbose output, and in the foreground, capturing both the verbose output and the strace we'll gather to files:

# strace -ftttTvyyo sshd-$(date -Ins).strace -s 4096 \
    env -i /usr/sbin/sshd -ddd -p 2222 2>&1 \
    | while read line; do printf '[%s] %s\n' "$(date -Ins)" "$line"
    done > sshd-$(date -Ins).debug

NOTE: This standalone ssh server will not fork, and will only process one connection, and then exit. Repeat the steps, above, to collect data for more than one connection.

On the ssh client, we connect to the server noted earlier using a command that will log output:

# DESTINATION="user@server"

# strace -ftttTvyyo ssh-$(date -Ins).strace -s 4096 \
    env -i ssh -vvv -p 2222 ${DESTINATION} date 2>&1 \
    | while read line; do printf '[%s] %s\n' "$(date -Ins)" "$line"
    done > ssh-$(date -Ins).debug

When the test is complete, please attach each command-date.strace, command-date.debug and dump_*.cap files, from the /tmp directories of both the server and client, to your support case.

Please note that it's conceivable you'll capture sensitive (e.g., authentication) data in your strace. If this is a concern, reasonable options might include using a test account created for the purpose or temporarily changing the password.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments