Poor GFS2 write performance for large single files on Red Hat Enterprise Linux 5

Issue

• Copying a big file to GFS2 file system needs more time than ext3, and these device are backed by the same SAN disk.
• We were trying to create a 400 Gb file in a GFS2 share. It was running at around 50Mb/s approx. Suddenly, after copying 80%, the rate dropped to around 30Kb/s and stayed like that. We tried to kill the process but couldn't (which is normal) and after a few minutes it was killed. Then tried again after a few minutes and it was successfully created at approximately 50Mb/s.
• The performance on GFS2 volumes decrease constantly and we have to unmount and remount them every day to keep our application running.
• GFS2 copying to backup hangs or drops to very slow write speeds after the kernel was updated to 2.6.18-238.1.1.el5.
• First we create the backup files locally, 7 files 16Gb and 1 file 3Gb, the last step is copy the local files from ext3 (not from GFS2 filesystem) dir to the GFS2.
• GFS2 with NFS becomes very sluggish when transferring large files to the SAN via NFS v3 or v4 (files larger than 20Gb) the whole system becomes very sluggish.
• GFS2 write performance degrades after ~500Gb copy.