RHEL5 client machine faster than RHEL6 client machine while listing files on a CIFS NetApp share, why?

Issue

• RHEL5 and RHEL6 machines act as client nodes for a NetApp CIFS share.

• Red Hat Enterprise Linux 5 machine with kernel 2.6.18-194.el5

• Red Hat Enterprise Linux 6 machine with kernel 2.6.32-279.5.2.el6.

• RHEL5 has faster access times compared to RHEL6, while listing the same files from the NetApp share.

• The differences are drastic:

• On a RHEL5

# time ls -al </mount-point>

    real 0m1.616s
    user 0m0.476s
    sys 0m0.538s

• On the RHEL6 node:

# time ls -al </mount-point>

    real 0m23.213s
    user 0m0.818s
    sys 0m1.919s

• There are around ~32K files in the directory.

• The RHEL5 machine has lower specs compared to the RHEL6 machine, yet has better speeds in listing the data. On the RHEL6 machine, the listing hangs on for a few seconds and then outputs properly, while this is not seen on the RHEL5 machine.

• An strace and tcpdump shows that on the RHEL6 (slower listing), a listing has both 'lstat' and 'lgetxattr' executing twice, while the RHEL5 (faster listing) machine has it only once.

• On RHEL5 :

# strace ls -l
...
lstat("20140930_0114713337_F_173_28017_207821587.007161.pdf", {st_mode=S_IFREG|S_ISGID|0767, st_size=150201, ...}) = 0
getxattr("20140930_0114713337_F_173_28017_207821587.007161.pdf", "system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported)

• On RHEL6:

lstat("20140930_0114713165_F_173_28007_227604840.001003.pdf", {st_mode=S_IFREG|0750, st_size=149546, ...}) = 0
lgetxattr("20140930_0114713165_F_173_28007_227604840.001003.pdf", "security.selinux", 0x1034830, 255) = -1 EOPNOTSUPP (Operation not supported)
lstat("20140930_0114713165_F_173_28007_227604840.001003.pdf", {st_mode=S_IFREG|0750, st_size=149546, ...}) = 0
lgetxattr("20140930_0114713165_F_173_28007_227604840.001003.pdf", "system.posix_acl_access", 0x0, 0) = -1 EOPNOTSUPP (Operation not supported)

• The slowness is evident on RHEL7 clients as well.

RHEL5 (Server without problems)
# time ls -al /opt/efactura/wrk/lote-25916/pdf | grep real
real    0m1.237s
user    0m0.572s
sys     0m0.358s

RHEL6 (Server with the problem)
# time ls -al /mnt/masivo-greco-efactura/lote-25916/pdf | grep real
real    0m15.254s
user    0m0.789s
sys     0m1.628s

RHEL7 (Server with the problem)
# time ls -al /mnt/masivo-greco-efactura/lote-25916/pdf | grep real
real    0m15.189s
user    0m0.798s
sys     0m1.084s