It was found out that the process of copying a file from one location to another on the same shared file system GPFS (set up using RDMA) does take a significantly long time. This issue looks to be related to our cluster components running under Red Hat Enterprise Linux 5. The following example consists of a 2 GB file, copied within the same folder:
ls -lh /gpfs3/trash/largefile -rw-r--r-- 1 root root 2.0G Mar 14 12:08 /gpfs3/trash/largefile
Red Hat Enterprise Linux 5:
[root@hpc-rhel5 ~]# time cp /gpfs3/trash/largefile /gpfs3/trash/largefile2 real 0m25.323s user 0m0.633s sys 0m24.325s
Red Hat Enterprise Linux 6
[root@hpc-rhel6 ~]# time cp /gpfs3/trash/largefile /gpfs3/trash/largefile2 real 0m0.666s user 0m0.001s sys 0m0.632s
When coping data on attached local drives using
cp, the copy duration is quite identical on both systems. Additionally, e.g. using
dd to "copy" data does not seem to be affected.
- Red Hat Enterprise Linux 5.7
- Mellanox Infiniband hardware (OFED-1.5.3-4.0.22 drivers)
- IBM GPFS shared filesystem with support for 8MB blocks
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.