RHEL5 client machine faster than RHEL6 client machine while listing files on a CIFS NetApp share, why?
Issue
-
RHEL5 and RHEL6 machines act as client nodes for a NetApp CIFS share.
-
Red Hat Enterprise Linux 5 machine with kernel
2.6.18-194.el5 -
Red Hat Enterprise Linux 6 machine with kernel
2.6.32-279.5.2.el6. -
RHEL5 has faster access times compared to RHEL6, while listing the same files from the NetApp share.
-
The differences are drastic:
-
On a RHEL5
# time ls -al </mount-point>
real 0m1.616s
user 0m0.476s
sys 0m0.538s
- On the RHEL6 node:
# time ls -al </mount-point>
real 0m23.213s
user 0m0.818s
sys 0m1.919s
-
There are around ~32K files in the directory.
-
The RHEL5 machine has lower specs compared to the RHEL6 machine, yet has better speeds in listing the data. On the RHEL6 machine, the listing hangs on for a few seconds and then outputs properly, while this is not seen on the RHEL5 machine.
-
An strace and tcpdump shows that on the RHEL6 (slower listing), a listing has both 'lstat' and 'lgetxattr' executing twice, while the RHEL5 (faster listing) machine has it only once.
-
On RHEL5 :
# strace ls -l
...
lstat("20140930_0114713337_F_173_28017_207821587.007161.pdf", {st_mode=S_IFREG|S_ISGID|0767, st_size=150201, ...}) = 0
getxattr("20140930_0114713337_F_173_28017_207821587.007161.pdf", "system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
- On RHEL6:
lstat("20140930_0114713165_F_173_28007_227604840.001003.pdf", {st_mode=S_IFREG|0750, st_size=149546, ...}) = 0
lgetxattr("20140930_0114713165_F_173_28007_227604840.001003.pdf", "security.selinux", 0x1034830, 255) = -1 EOPNOTSUPP (Operation not supported)
lstat("20140930_0114713165_F_173_28007_227604840.001003.pdf", {st_mode=S_IFREG|0750, st_size=149546, ...}) = 0
lgetxattr("20140930_0114713165_F_173_28007_227604840.001003.pdf", "system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported)
- The slowness is evident on RHEL7 clients as well.
RHEL5 (Server without problems)
# time ls -al /opt/efactura/wrk/lote-25916/pdf | grep real
real 0m1.237s
user 0m0.572s
sys 0m0.358s
RHEL6 (Server with the problem)
# time ls -al /mnt/masivo-greco-efactura/lote-25916/pdf | grep real
real 0m15.254s
user 0m0.789s
sys 0m1.628s
RHEL7 (Server with the problem)
# time ls -al /mnt/masivo-greco-efactura/lote-25916/pdf | grep real
real 0m15.189s
user 0m0.798s
sys 0m1.084s
Environment
-
Red Hat Enterprise Linux 5
-
Red Hat Enterprise Linux 6
-
Red Hat Enterprise Linux 7
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
