Logging sftp commands

Updated -

Using sftp to store data on a file server became a popular and secure way. In the past, there were problems with logging user activity in chrooted environment because of missing files to do so. This was finally solved in RHEL 6 using file descriptor passing and in RHEL 7 this feature is achieved using a privileged monitor. On the next lines, I would like to elaborate on the possibilities.

Logging without chroot

Single file

Basically, if we don't use chroot, we can rely on the default configuration and the only thing needed is to allow logging from sftp-server by adding command-line arguments to the Subsystem sftp line in /etc/ssh/sshd_config:

Subsystem   sftp    /usr/libexec/openssh/sftp-server -l VERBOSE

After restarting sshd and performing sftp session, these lines will appear in /var/log/messages:

Mar  9 09:39:07 localhost sftp-server[1829]: received client version 3
Mar  9 09:39:07 localhost sftp-server[1829]: realpath "."
Mar  9 09:39:09 localhost sftp-server[1829]: lstat name "/root"

Specific file

If you want to achieve logging into a different file, you have to configure rsyslog to direct messages into the other file, for example using the log_facility option, in /etc/ssh/sshd_config:

Subsystem   sftp    /usr/libexec/openssh/sftp-server -l VERBOSE -f LOCAL3

And in /etc/rsyslog.conf:

local3.*                                                /var/log/sftp.log

After restarting sshd and rsyslog, you will find the following logs in /var/log/sftp.log:

Mar  9 09:49:02 localhost sftp-server[1947]: received client version 3
Mar  9 09:49:02 localhost sftp-server[1947]: realpath "."
Mar  9 09:49:04 localhost sftp-server[1947]: lstat name "/root"
  • If you'd like to log the sftp transactions in one specific only then edit the /etc/rsyslog.conf file to have the following lines :-
..snip..
local3.*                        /var/log/sftp.log
&~
*.info;mail.none;authpriv.none;cron.none                /var/log/messages
..snip..

and then restart rsyslog service.

The messages are now logged to /var/log/sftp.log and owing to the presence of '&~' they would be limited to /var/log/sftp.log only.

Logging in chroot

When using chroot, there are basically two possibilities. One is chroot without any support files, which requires logging through a privileged monitor. This is useful if there are many users and no way to have a separate /dev/log socket in every chroot. The other case is to have a /dev/log socket configured, which should be honoured by sshd and has higher priority over logging through the monitor. However, both options require to use the internal-sftp subsystem instead of the executable path like above, because there is no such file in chroot.

RHEL 6 (since RHEL 6.7)

Via monitor

The base release of openssh doesn't have the ability to log from a chrooted environment, if there is no available and configured socket located in /dev/log. In order to enable chroot, we need to modify /etc/ssh/sshd_config in this way:

Subsystem   sftp    internal-sftp -l VERBOSE
Match Group sftponly
    ChrootDirectory /chroots/%u

After restarting sshd and performing sftp session with user from group sftponly, we will get these lines in /var/log/secure (note the process name sshd instead of sftp-server, because the file descriptor is owned by the privileged monitor, not the sftp-server) :

Mar  9 10:04:35 localhost sshd[2159]: received client version 3
Mar  9 10:04:35 localhost sshd[2159]: realpath "."
Mar  9 10:04:36 localhost sshd[2159]: lstat name "/"

This use case doesn't allow us to log into different files, because the log_facility option is honoured only for newly open log descriptors, but this is skipped in this case by the sftp-server.

Via socket in chroot

If we want to log through the /dev/log socket in chroot, we set up /etc/ssh/sshd_config in the same way like in previous point, but we specify rsyslog configuration in /etc/rsyslog.d/sftp.conf, which creates the required socket automatically. The following example is for rsyslog v7 (rsyslog7 package):

input(type="imuxsock" HostName="user" Socket="/chroots/user/dev/log" CreatePath="on")
if $fromhost == 'user' then /var/log/sftp.log
& stop

After restarting sshd and rsyslog, we can get these log entries in /var/log/sftp.log:

Mar  9 10:53:39 user internal-sftp[2475]: received client version 3
Mar  9 10:53:39 user internal-sftp[2475]: realpath "."
Mar  9 10:53:40 user internal-sftp[2475]: lstat name "/"

This setup requires more configuration but gives clean results. Important is to set up SELinux rules for rsyslog, so that it has access to this socket, SELinux rules for sshd (internal-sftp), so that it has access to the /chroots/user/ directory, and SELinux and Linux ACL for the socket directory /chroots/user/dev/ (default is 0700 root owned).

Via socket in chroot, filtering by log facility

Or we can again use log_facility in /etc/ssh/sshd_config to achieve the same results:

Subsystem   sftp    internal-sftp -l VERBOSE -f LOCAL3
Match Group sftponly
    ChrootDirectory /chroots/%u

And in /etc/rsyslog.conf:

input(type="imuxsock" Socket="/chroots/user/dev/log" CreatePath="on")
local3.*                                                /var/log/sftp.log

The logs will end up in /var/log/sftp.log:

Mar  9 12:28:46 rhel6 internal-sftp[4008]: received client version 3
Mar  9 12:28:46 rhel6 internal-sftp[4008]: realpath "."
Mar  9 12:28:47 rhel6 internal-sftp[4008]: lstat name "/"

RHEL 7 (since RHEL 7.1)

Via monitor

Base openssh release in this version is more recent and provides ability to log via a privileged monitor. Simply enough if you just set up internal-sftp and chroot like in the previous example:

Subsystem   sftp    internal-sftp -l VERBOSE
Match Group sftponly
    ChrootDirectory /chroots/%u

And the results will show up in /var/log/secure again:

Mar  9 11:24:04 rhel7 sshd[20327]: received client version 3 [postauth]
Mar  9 11:24:04 rhel7 sshd[20327]: realpath "." [postauth]
Mar  9 11:24:05 rhel7 sshd[20327]: opendir "/" [postauth]

Via socket in chroot

If we want to log into a different file, we can use the same sshd configuration, but we need to adjust /etc/rsyslog.conf:

input(type="imuxsock" HostName="user" Socket="/chroots/user/dev/log" CreatePath="on")
if $fromhost == 'user' then /var/log/sftp.log
& stop

After restarting sshd and rsyslog, sftp messages are logged into /var/log/sftp.log (currently /var/log/messages because of bug #1184402 in rsyslog)

Mar  9 13:04:49 rhel7 internal-sftp[12699]: received client version 3
Mar  9 13:04:49 rhel7 internal-sftp[12699]: realpath "."
Mar  9 13:04:50 rhel7 internal-sftp[12699]: opendir "/"

Via socket in chroot, filtering using log facility

We can also use log_facility in /etc/ssh/sshd_config like in the previous cases:

Subsystem   sftp    internal-sftp -l VERBOSE -f LOCAL3
Match Group sftponly
    ChrootDirectory /chroots/%u

Also we need a socket in chroot, otherwise we can't change the facility for the sftp-server. This is done in /etc/rsyslog.conf:

input(type="imuxsock" Socket="/chroots/user/dev/log" CreatePath="on")
local3.*                                                /var/log/sftp.log

Provided that we take care of the SELinux configuration and rsyslog is able to create a socket and log using it, we will get the logs in /var/log/sftp.log:

Mar 12 09:40:06 rhel7 internal-sftp[2472]: received client version 3
Mar 12 09:40:06 rhel7 internal-sftp[2472]: realpath "."
Mar 12 09:40:07 rhel7 internal-sftp[2472]: opendir "/"

Chroot comments

  • Minimal configuration of access rights for the chroot directory structure can be defined in the following way (where user user is a member of group sftponly):
drwxr-xr-x. 3 root root 17 Jan 20 16:17 /chroots/
drwxr-xr-x. 4 root sftponly 27 Mar 11 17:55 /chroots/user/
drwxr-xr-x. 2 user sftponly  6 Mar  9 11:05 /chroots/user/home
drwxr-xr-x. 2 root root     16 Mar 12 09:39 /chroots/user/dev
srw-rw-rw-. 1 root root 0 Mar 12 09:39 /chroots/user/dev/log
  • To make sure that the user has only sftp access, it is recommended to use ForceCommand internal-sftp with the same arguments as Subsystem sftp option above.
  • RHEL 7 has a bug in rsyslog that prevents socket creation with missing parent directories (you need to create the dev directory in the user's chroot). (bug #1184410)
  • RHEL 7 has a bug in rsyslog that prevents the change of hostname parameter that is used for filtering into a different directory in one example. (bug #1184402)
  • Without SELinux configuration on RHEL 6, the user is not able to list directories in the sftp chroot, because it is forbidden by SELinux.
  • Without SELinux configuration on RHEL 7, rsyslog is unable to create a socket in chroot.

12 Comments

Unless I am not comprehending the document correctly, it does not seem to work. I have a RHEL 7.2 system that I have set up a chroot jail and need to get some verbose logging but have not been able to get it to work based on this article.

There are explained two ways how to do that on RHEL7 and it should be tested by our QA. Which one did you try? What problems are you facing? Do you see some errors?

This worked on my RHEL6 system... I had to update rsyslogd to v7 from the rsyslogd official repository:

/etc/ssh/sshd_config:

Subsystem       sftp    internal-sftp -l VERBOSE -f AUTHPRIV

Match Group sftponly
        ChrootDirectory /home/%u/chroot
        AllowTcpForwarding no
        X11Forwarding no
        ForceCommand internal-sftp -l VERBOSE -f AUTHPRIV

####
add sftp only user:
adduser -g sftponly -s /bin/false username1
passwd username1
chown root.root /home/username1
chmod 755 /home/username1
mkdir -p /home/username1/chroot/Upload
chmod 755 /home/username1/chroot
chmod 700 /home/username1/chroot/Upload
chown username1.sftponly /home/username1/chroot/Upload

####
/etc/rsyslog.conf (rsyslogd v7 added from rsyslogd's official rhel6 repository)

input(type="imuxsock" Socket="/home/username1/chroot/dev/log" CreatePath="on")
authpriv.*                                                /var/log/auth.log

####
service sshd restart ; service rsyslog restart
chmod go+rx /home/username1/chroot/dev
service rsyslog restart (just in case)
tail -F /var/log/auth.log
now test by sftping in with username1 and tailing auth.log

I've redhat 7 and there is only something written about a bug - bad for me. I need to know how I can log the access and work of sftp users and can't do it till now

Using the monitor solution all syslog entries originating from the monitored chrooted process will get the '[postauth]' suffix attached.

I assume that with the a simple rsyslog filter for both 'sshd' and ' [postauth]' the sftp commands can be written to a dedicted sftp.log file:

if $programname == "sshd" and $msg contains " [postauth]" then { action(type="omfile" FileCreateMode="0600" fileOwner="root" fileGroup="root" File="/var/log/sftp.log") stop

}

This solution does not offer a proper chrooted sshd logging solution. It simply relies upon sshd's internal chroot functionality. ---For example: 1) Create the base chroot dir (eg: /chroot) export CHROOT=/chroot export RELEASEVER=8 mkdir -p $CHROOT

2) Install the chroot with packages you will need (eg: openssh-server) dnf --assumeyes --setopt=reposdir=/etc/yum.repos.d --installroot=$CHROOT --releasever=$RELEASEVER install openssh-server

3) Set up the required system mounts to be available in the chroot mount -t proc proc $CHROOT/proc/ mount -t sysfs sys $CHROOT/sys/ mount -o bind /dev $CHROOT/dev/ mount -o bind /dev/pts /$CHROOT/dev/pts

4) Configure and Verify that your chroot environment is operational NB: Ensure that the base OS sshd is disabled or does not use the same port as the chroot sshd

Set/change sshd port

vi $CHROOT/etc/ssh/sshd_config (insert or change the existing 'Port' line to - Port 22221

Generate host keys

/usr/sbin/chroot $CHROOT /bin/bash -c "ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa" /usr/sbin/chroot $CHROOT /bin/bash -c "ssh-keygen -f /etc/ssh/ssh_host_dsa_key -N '' -t dsa" /usr/sbin/chroot $CHROOT /bin/bash -c "ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N '' -t ed25519"

Manually launch chrooted sshd

/usr/sbin/chroot $CHROOT /bin/bash -c "/usr/sbin/sshd -f /etc/ssh/sshd_config"

5) Create real chrooted ssh user BASEUSERDIR=/home USERSHELL=/bin/bash BASEUSER=ch_user1 USERACC=ch_user1 localUID=4001 chroot $CHROOT /sbin/groupadd -g ${localUID} ${USERACC} chroot $CHROOT /sbin/useradd -r -u ${localUID} -g ${localUID} -s ${USERSHELL} -m -b ${BASEUSERDIR} ${USERACC}

chroot $CHROOT ls -l /home

chroot $CHROOT ls -l /home

total 0 drwx------. 2 ch_user1 ch_user1 62 Aug 22 04:05 ch_user1

6) Test connectivity to chrooted ssh user from base OS ssh -p 22221 ch_user@localhost

ssh -p 22221 ch_user@localhost

The authenticity of host '[localhost]:22221 ([::1]:22221)' can't be established. ECDSA key fingerprint is SHA256:0KImYyWPJRCFLydtEbdPxhuY4d/gxg29GOtssv3GhiY. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '[localhost]:22221' (ECDSA) to the list of known hosts. ch_user@localhost's password:

But where are the logs ? ls -l /$CHROOT/var/log #Does NOT contain any sshd logs!, neither does the base OS

Right. But this article is not about chrooting SSHD and collecting logs there. This is about chrooting the SFTP user and collecting logs in the OS. If you need to achieve something else, please open a new support ticket.

I did raise an issue, and they pointed me to this page.

The sftp session/file transfer logs are getting merged with /var/log/secure for some duration instead being redirected to /var/log/sftp.log and it auto-recovers as well. We have configured chrooted env with /etc/ssh/sshd_config ForceCommand internal-sftp -m770 -l VERBOSE -f local3 Also created /etc/rsyslog.d/sftp.log file for socket and dev/log file creation Specifically mentioned ocal3.* /var/log/sftp.log in rsysog.conf What could be reason? RHEL 7.9 rsyslog version: rsyslog-8.24.0-55.el7.x86_64

Do you have some examples of the logs that go to the system log file? I think you will get there logs from before the ssh does chroot and before the internal-sftp is executed, but afterward, all the messages should land in the dedicated log file. Not sure if there can be much done about that though.

/*********/ Mar 13 11:30:28 internal-sftp[13860]: session closed for local user x from [10.......109] Mar 14 03:57:54 internal-sftp[30529]: session opened for local user y from [1.....3] /***********/ ( Above Log file username, ip is modified) There are no logs in SFTP.log during this period except for 1st and last line, and all further logs are available in secure log file. Then automatically it's recovered

my log file can't show filename with Chinese.

Aug 15 16:31:47 rh84 sftp-server[48511]: realpath "/root/."

Aug 15 16:31:47 rh84 sftp-server[48511]: opendir "/root"

Aug 15 16:31:47 rh84 sftp-server[48511]: closedir "/root"

Aug 15 16:31:56 rh84 sftp-server[48511]: realpath "/root"

Aug 15 16:31:56 rh84 sftp-server[48511]: remove name "/root/\344\270\255\346\226\207\346\252\224"

I modify the rsyslog.conf

Save boot messages also to boot.log

local7.* /var/log/boot.log

Save sftp log

module(load="mmutf8fix")

action(type="mmutf8fix" mode="controlcharacters")

local3.* /var/log/sftp.log

It's not a crrent setting..

How should I fix it?