Why does RHEL5 copy files with a buffer of only 512 bytes when the filesystem can handle 8MB chunks?
Issue
It was found out that the process of copying a file from one location to another on the same shared file system GPFS (set up using RDMA) does take a significantly long time. This issue looks to be related to our cluster components running under Red Hat Enterprise Linux 5. The following example consists of a 2 GB file, copied within the same folder:
ls -lh /gpfs3/trash/largefile
-rw-r--r-- 1 root root 2.0G Mar 14 12:08 /gpfs3/trash/largefile
Red Hat Enterprise Linux 5:
[root@hpc-rhel5 ~]# time cp /gpfs3/trash/largefile /gpfs3/trash/largefile2
real 0m25.323s
user 0m0.633s
sys 0m24.325s
Red Hat Enterprise Linux 6
[root@hpc-rhel6 ~]# time cp /gpfs3/trash/largefile /gpfs3/trash/largefile2
real 0m0.666s
user 0m0.001s
sys 0m0.632s
When coping data on attached local drives using cp, the copy duration is quite identical on both systems. Additionally, e.g. using dd to "copy" data does not seem to be affected.
Environment
- Red Hat Enterprise Linux 5.7
- Mellanox Infiniband hardware (OFED-1.5.3-4.0.22 drivers)
- IBM GPFS shared filesystem with support for 8MB blocks
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.