'ls' fails in sftp session using wildcards

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 7
  • openssh earlier version than openssh-6.6.1p1-31.el7

Issue

  • sftp is failing using wild-cards and many files

  • When running ls with wild-cards in a sftp session against a folder containing more than 128 files following error is shown although at least one file matching search criteria is present:

    sftp>
sftp> ls /var/tmp/test/*
Can't ls: "/var/tmp/test/*" not found
sftp> ls /var/tmp/test/*.txt
Can't ls: "/var/tmp/test/*.txt" not found
sftp> 
sftp> ls /var/tmp/test/
/var/tmp/test/test-1.txt                                                                                             
/var/tmp/test/test-10.txt                                                                                                      
/var/tmp/test/test-100.txt                                                                                   
/var/tmp/test/test-1000.txt                                   
[...]
sftp>

Resolution

  • Update openssh package to openssh-6.6.1p1-31.el7 or later. The fixed package will accept 50K+ files to glob operation.

Root Cause

This is hitting the hard limit GLOB_LIMIT_STAT which is set to 128 or 8192 by default.
If a wild-card is used, ls needs to go and stat all the resulting files because star expansion is done before calling ls on remote shell, which is less efficient than the another mentioned method (ls /var/tmp/987../ ):
The related patches are below:

Until RHEL7.1

diff -up openssh-5.8p1/sftp-glob.c.glob openssh-5.8p1/sftp-glob.c
--- openssh-5.8p1/sftp-glob.c.glob  2011-03-07 20:17:34.000000000 +0100
+++ openssh-5.8p1/sftp-glob.c   2011-03-07 20:18:47.000000000 +0100
@@ -145,5 +145,5 @@ remote_glob(struct sftp_conn *conn, cons
    memset(&cur, 0, sizeof(cur));
    cur.conn = conn;

-   return(glob(pattern, flags | GLOB_ALTDIRFUNC, errfunc, pglob));
+   return(glob(pattern, flags | GLOB_LIMIT | GLOB_ALTDIRFUNC, errfunc, pglob));
 }

Until RHEL7.2

diff -up openssh-5.8p1/sftp-glob.c.glob openssh-5.8p1/sftp-glob.c
--- openssh-5.8p1/sftp-glob.c.glob  2011-03-07 20:17:34.000000000 +0100
+++ openssh-5.8p1/sftp-glob.c   2011-03-07 20:18:47.000000000 +0100
@@ -145,5 +145,5 @@ remote_glob(struct sftp_conn *conn, cons
    memset(&cur, 0, sizeof(cur));
    cur.conn = conn;

-   return(glob(pattern, flags | GLOB_ALTDIRFUNC, errfunc, pglob));
+   return(glob(pattern, flags | GLOB_LIMIT | GLOB_ALTDIRFUNC, errfunc, pglob));
 }
diff --git a/openbsd-compat/glob.c b/openbsd-compat/glob.c
index 742b4b9..acae399 100644
--- a/openbsd-compat/glob.c
+++ b/openbsd-compat/glob.c
@@ -130,8 +130,8 @@ typedef char Char;
 #define    M_CLASS     META(':')
 #define    ismeta(c)   (((c)&M_QUOTE) != 0)

-#define    GLOB_LIMIT_MALLOC   65536
-#define    GLOB_LIMIT_STAT     128
+#define    GLOB_LIMIT_MALLOC   65536*64
+#define    GLOB_LIMIT_STAT     128*64
 #define    GLOB_LIMIT_READDIR  16384

 /* Limit of recursion during matching attempts. */

After RHEL7.3, the patch was removed from the package.

Diagnostic Steps

[root@rhel7ga ~]# x="test"
[root@rhel7ga ~]# dir=/var/tmp/$x
[root@rhel7ga ~]# mkdir -p $dir
[root@rhel7ga ~]# for n in `seq 2000`
> do
> touch $dir/$x-$n.txt
> done
[root@rhel7ga ~]# ls -la $dir/* |wc -l 
2000
[root@rhel7ga ~]# sftp localhost
root@localhost's password: 
Connected to localhost.
sftp> 
sftp> ls /var/tmp/test/*
Can't ls: "/var/tmp/test/*" not found
sftp> ls /var/tmp/test/*.txt
Can't ls: "/var/tmp/test/*.txt" not found
sftp> 
sftp> ls /var/tmp/test/
/var/tmp/test/test-1.txt                                                                                                                         
/var/tmp/test/test-10.txt                                                                                                                        
/var/tmp/test/test-100.txt                                                                                                                       
/var/tmp/test/test-1000.txt                                                                                                                      
[...]
sftp>

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments