Logging sftp commands
Using sftp to store data on a file server became a popular and secure way. In the past, there were problems with logging user activity in chrooted environment because of missing files to do so. This was finally solved in RHEL 6 using file descriptor passing and in RHEL 7 this feature is achieved using a privileged monitor. On the next lines, I would like to elaborate on the possibilities.
Logging without chroot
Single file
Basically, if we don't use chroot, we can rely on the default configuration and the only thing needed is to allow logging from sftp-server
by adding command-line arguments to the Subsystem sftp
line in /etc/ssh/sshd_config
:
Subsystem sftp /usr/libexec/openssh/sftp-server -l VERBOSE
After restarting sshd
and performing sftp session, these lines will appear in /var/log/messages
:
Mar 9 09:39:07 localhost sftp-server[1829]: received client version 3
Mar 9 09:39:07 localhost sftp-server[1829]: realpath "."
Mar 9 09:39:09 localhost sftp-server[1829]: lstat name "/root"
Specific file
If you want to achieve logging into a different file, you have to configure rsyslog
to direct messages into the other file, for example using the log_facility
option, in /etc/ssh/sshd_config
:
Subsystem sftp /usr/libexec/openssh/sftp-server -l VERBOSE -f LOCAL3
And in /etc/rsyslog.conf
:
local3.* /var/log/sftp.log
After restarting sshd
and rsyslog
, you will find the following logs in /var/log/sftp.log
:
Mar 9 09:49:02 localhost sftp-server[1947]: received client version 3
Mar 9 09:49:02 localhost sftp-server[1947]: realpath "."
Mar 9 09:49:04 localhost sftp-server[1947]: lstat name "/root"
- If you'd like to log the sftp transactions in one specific only then edit the
/etc/rsyslog.conf
file to have the following lines :-
..snip..
local3.* /var/log/sftp.log
&~
*.info;mail.none;authpriv.none;cron.none /var/log/messages
..snip..
and then restart rsyslog service.
The messages are now logged to /var/log/sftp.log
and owing to the presence of '&~' they would be limited to /var/log/sftp.log
only.
Logging in chroot
When using chroot, there are basically two possibilities. One is chroot without any support files, which requires logging through a privileged monitor. This is useful if there are many users and no way to have a separate /dev/log
socket in every chroot. The other case is to have a /dev/log
socket configured, which should be honoured by sshd
and has higher priority over logging through the monitor. However, both options require to use the internal-sftp
subsystem instead of the executable path like above, because there is no such file in chroot.
RHEL 6 (since RHEL 6.7)
Via monitor
The base release of openssh
doesn't have the ability to log from a chrooted environment, if there is no available and configured socket located in /dev/log
. In order to enable chroot, we need to modify /etc/ssh/sshd_config
in this way:
Subsystem sftp internal-sftp -l VERBOSE
Match Group sftponly
ChrootDirectory /chroots/%u
After restarting sshd
and performing sftp session with user from group sftponly
, we will get these lines in /var/log/secure
(note the process name sshd
instead of sftp-server
, because the file descriptor is owned by the privileged monitor, not the sftp-server
) :
Mar 9 10:04:35 localhost sshd[2159]: received client version 3
Mar 9 10:04:35 localhost sshd[2159]: realpath "."
Mar 9 10:04:36 localhost sshd[2159]: lstat name "/"
This use case doesn't allow us to log into different files, because the log_facility
option is honoured only for newly open log descriptors, but this is skipped in this case by the sftp-server
.
Via socket in chroot
If we want to log through the /dev/log
socket in chroot, we set up /etc/ssh/sshd_config
in the same way like in previous point, but we specify rsyslog
configuration in /etc/rsyslog.d/sftp.conf
, which creates the required socket automatically. The following example is for rsyslog v7 (rsyslog7
package):
input(type="imuxsock" HostName="user" Socket="/chroots/user/dev/log" CreatePath="on")
if $fromhost == 'user' then /var/log/sftp.log
& stop
After restarting sshd
and rsyslog
, we can get these log entries in /var/log/sftp.log
:
Mar 9 10:53:39 user internal-sftp[2475]: received client version 3
Mar 9 10:53:39 user internal-sftp[2475]: realpath "."
Mar 9 10:53:40 user internal-sftp[2475]: lstat name "/"
This setup requires more configuration but gives clean results. Important is to set up SELinux rules for rsyslog
, so that it has access to this socket, SELinux rules for sshd
(internal-sftp
), so that it has access to the /chroots/user/
directory, and SELinux and Linux ACL for the socket directory /chroots/user/dev/
(default is 0700 root owned).
Via socket in chroot, filtering by log facility
Or we can again use log_facility
in /etc/ssh/sshd_config
to achieve the same results:
Subsystem sftp internal-sftp -l VERBOSE -f LOCAL3
Match Group sftponly
ChrootDirectory /chroots/%u
And in /etc/rsyslog.conf
:
input(type="imuxsock" Socket="/chroots/user/dev/log" CreatePath="on")
local3.* /var/log/sftp.log
The logs will end up in /var/log/sftp.log
:
Mar 9 12:28:46 rhel6 internal-sftp[4008]: received client version 3
Mar 9 12:28:46 rhel6 internal-sftp[4008]: realpath "."
Mar 9 12:28:47 rhel6 internal-sftp[4008]: lstat name "/"
RHEL 7 (since RHEL 7.1)
Via monitor
Base openssh
release in this version is more recent and provides ability to log via a privileged monitor. Simply enough if you just set up internal-sftp
and chroot like in the previous example:
Subsystem sftp internal-sftp -l VERBOSE
Match Group sftponly
ChrootDirectory /chroots/%u
And the results will show up in /var/log/secure
again:
Mar 9 11:24:04 rhel7 sshd[20327]: received client version 3 [postauth]
Mar 9 11:24:04 rhel7 sshd[20327]: realpath "." [postauth]
Mar 9 11:24:05 rhel7 sshd[20327]: opendir "/" [postauth]
Via socket in chroot
If we want to log into a different file, we can use the same sshd
configuration, but we need to adjust /etc/rsyslog.conf
:
input(type="imuxsock" HostName="user" Socket="/chroots/user/dev/log" CreatePath="on")
if $fromhost == 'user' then /var/log/sftp.log
& stop
After restarting sshd
and rsyslog
, sftp messages are logged into /var/log/sftp.log
(currently /var/log/messages
because of bug #1184402 in rsyslog
)
Mar 9 13:04:49 rhel7 internal-sftp[12699]: received client version 3
Mar 9 13:04:49 rhel7 internal-sftp[12699]: realpath "."
Mar 9 13:04:50 rhel7 internal-sftp[12699]: opendir "/"
Via socket in chroot, filtering using log facility
We can also use log_facility
in /etc/ssh/sshd_config
like in the previous cases:
Subsystem sftp internal-sftp -l VERBOSE -f LOCAL3
Match Group sftponly
ChrootDirectory /chroots/%u
Also we need a socket in chroot, otherwise we can't change the facility for the sftp-server
. This is done in /etc/rsyslog.conf
:
input(type="imuxsock" Socket="/chroots/user/dev/log" CreatePath="on")
local3.* /var/log/sftp.log
Provided that we take care of the SELinux configuration and rsyslog
is able to create a socket and log using it, we will get the logs in /var/log/sftp.log
:
Mar 12 09:40:06 rhel7 internal-sftp[2472]: received client version 3
Mar 12 09:40:06 rhel7 internal-sftp[2472]: realpath "."
Mar 12 09:40:07 rhel7 internal-sftp[2472]: opendir "/"
Chroot comments
- Minimal configuration of access rights for the chroot directory structure can be defined in the following way (where user
user
is a member of groupsftponly
):
drwxr-xr-x. 3 root root 17 Jan 20 16:17 /chroots/
drwxr-xr-x. 4 root sftponly 27 Mar 11 17:55 /chroots/user/
drwxr-xr-x. 2 user sftponly 6 Mar 9 11:05 /chroots/user/home
drwxr-xr-x. 2 root root 16 Mar 12 09:39 /chroots/user/dev
srw-rw-rw-. 1 root root 0 Mar 12 09:39 /chroots/user/dev/log
- To make sure that the user has only sftp access, it is recommended to use
ForceCommand internal-sftp
with the same arguments asSubsystem sftp
option above. - RHEL 7 has a bug in
rsyslog
that prevents socket creation with missing parent directories (you need to create thedev
directory in the user's chroot). (bug #1184410) - RHEL 7 has a bug in
rsyslog
that prevents the change of hostname parameter that is used for filtering into a different directory in one example. (bug #1184402) - Without SELinux configuration on RHEL 6, the user is not able to list directories in the sftp chroot, because it is forbidden by SELinux.
- Without SELinux configuration on RHEL 7,
rsyslog
is unable to create a socket in chroot.
12 Comments
Unless I am not comprehending the document correctly, it does not seem to work. I have a RHEL 7.2 system that I have set up a chroot jail and need to get some verbose logging but have not been able to get it to work based on this article.
There are explained two ways how to do that on RHEL7 and it should be tested by our QA. Which one did you try? What problems are you facing? Do you see some errors?
I've redhat 7 and there is only something written about a bug - bad for me. I need to know how I can log the access and work of sftp users and can't do it till now
Using the monitor solution all syslog entries originating from the monitored chrooted process will get the '[postauth]' suffix attached.
I assume that with the a simple rsyslog filter for both 'sshd' and ' [postauth]' the sftp commands can be written to a dedicted sftp.log file:if $programname == "sshd" and $msg contains " [postauth]" then { action(type="omfile" FileCreateMode="0600" fileOwner="root" fileGroup="root" File="/var/log/sftp.log") stop
}This solution does not offer a proper chrooted sshd logging solution. It simply relies upon sshd's internal chroot functionality. ---For example: 1) Create the base chroot dir (eg: /chroot) export CHROOT=/chroot export RELEASEVER=8 mkdir -p $CHROOT
2) Install the chroot with packages you will need (eg: openssh-server) dnf --assumeyes --setopt=reposdir=/etc/yum.repos.d --installroot=$CHROOT --releasever=$RELEASEVER install openssh-server
3) Set up the required system mounts to be available in the chroot mount -t proc proc $CHROOT/proc/ mount -t sysfs sys $CHROOT/sys/ mount -o bind /dev $CHROOT/dev/ mount -o bind /dev/pts /$CHROOT/dev/pts
4) Configure and Verify that your chroot environment is operational NB: Ensure that the base OS sshd is disabled or does not use the same port as the chroot sshd
Set/change sshd portvi $CHROOT/etc/ssh/sshd_config (insert or change the existing 'Port' line to - Port 22221
Generate host keys/usr/sbin/chroot $CHROOT /bin/bash -c "ssh-keygen -f /etc/ssh/ssh_host_rsa_key -N '' -t rsa" /usr/sbin/chroot $CHROOT /bin/bash -c "ssh-keygen -f /etc/ssh/ssh_host_dsa_key -N '' -t dsa" /usr/sbin/chroot $CHROOT /bin/bash -c "ssh-keygen -f /etc/ssh/ssh_host_ed25519_key -N '' -t ed25519"
Manually launch chrooted sshd/usr/sbin/chroot $CHROOT /bin/bash -c "/usr/sbin/sshd -f /etc/ssh/sshd_config"
5) Create real chrooted ssh user BASEUSERDIR=/home USERSHELL=/bin/bash BASEUSER=ch_user1 USERACC=ch_user1 localUID=4001 chroot $CHROOT /sbin/groupadd -g ${localUID} ${USERACC} chroot $CHROOT /sbin/useradd -r -u ${localUID} -g ${localUID} -s ${USERSHELL} -m -b ${BASEUSERDIR} ${USERACC}
chroot $CHROOT ls -l /home
chroot $CHROOT ls -l /hometotal 0 drwx------. 2 ch_user1 ch_user1 62 Aug 22 04:05 ch_user1
6) Test connectivity to chrooted ssh user from base OS ssh -p 22221 ch_user@localhost
ssh -p 22221 ch_user@localhostThe authenticity of host '[localhost]:22221 ([::1]:22221)' can't be established. ECDSA key fingerprint is SHA256:0KImYyWPJRCFLydtEbdPxhuY4d/gxg29GOtssv3GhiY. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '[localhost]:22221' (ECDSA) to the list of known hosts. ch_user@localhost's password:
But where are the logs ? ls -l /$CHROOT/var/log #Does NOT contain any sshd logs!, neither does the base OSRight. But this article is not about chrooting SSHD and collecting logs there. This is about chrooting the SFTP user and collecting logs in the OS. If you need to achieve something else, please open a new support ticket.
I did raise an issue, and they pointed me to this page.
The sftp session/file transfer logs are getting merged with /var/log/secure for some duration instead being redirected to /var/log/sftp.log and it auto-recovers as well. We have configured chrooted env with /etc/ssh/sshd_config ForceCommand internal-sftp -m770 -l VERBOSE -f local3 Also created /etc/rsyslog.d/sftp.log file for socket and dev/log file creation Specifically mentioned ocal3.* /var/log/sftp.log in rsysog.conf What could be reason? RHEL 7.9 rsyslog version: rsyslog-8.24.0-55.el7.x86_64
Do you have some examples of the logs that go to the system log file? I think you will get there logs from before the ssh does chroot and before the internal-sftp is executed, but afterward, all the messages should land in the dedicated log file. Not sure if there can be much done about that though.
/*********/ Mar 13 11:30:28 internal-sftp[13860]: session closed for local user x from [10.......109] Mar 14 03:57:54 internal-sftp[30529]: session opened for local user y from [1.....3] /***********/ ( Above Log file username, ip is modified) There are no logs in SFTP.log during this period except for 1st and last line, and all further logs are available in secure log file. Then automatically it's recovered
my log file can't show filename with Chinese.
Aug 15 16:31:47 rh84 sftp-server[48511]: realpath "/root/."
Aug 15 16:31:47 rh84 sftp-server[48511]: opendir "/root"
Aug 15 16:31:47 rh84 sftp-server[48511]: closedir "/root"
Aug 15 16:31:56 rh84 sftp-server[48511]: realpath "/root"
Aug 15 16:31:56 rh84 sftp-server[48511]: remove name "/root/\344\270\255\346\226\207\346\252\224"
I modify the rsyslog.conf
Save boot messages also to boot.loglocal7.* /var/log/boot.log
Save sftp logmodule(load="mmutf8fix")
action(type="mmutf8fix" mode="controlcharacters")
local3.* /var/log/sftp.log
It's not a crrent setting..
How should I fix it?