Poor GFS2 write performance for large single files on Red Hat Enterprise Linux 5
Issue
- Copying a big file to GFS2 file system needs more time than ext3, and these device are backed by the same SAN disk.
- We were trying to create a 400 Gb file in a GFS2 share. It was running at around 50Mb/s approx. Suddenly, after copying 80%, the rate dropped to around 30Kb/s and stayed like that. We tried to kill the process but couldn't (which is normal) and after a few minutes it was killed. Then tried again after a few minutes and it was successfully created at approximately 50Mb/s.
- The performance on GFS2 volumes decrease constantly and we have to unmount and remount them every day to keep our application running.
- GFS2 copying to backup hangs or drops to very slow write speeds after the kernel was updated to
2.6.18-238.1.1.el5
. - First we create the backup files locally, 7 files 16Gb and 1 file 3Gb, the last step is copy the local files from ext3 (not from GFS2 filesystem) dir to the GFS2.
- GFS2 with NFS becomes very sluggish when transferring large files to the SAN via NFS v3 or v4 (files larger than 20Gb) the whole system becomes very sluggish.
- GFS2 write performance degrades after ~500Gb copy.
Environment
- Red Hat Enterprise Linux 5
- Kernels prior to
2.6.18-274.el5
(most common on2.6.18-238*.el5
versions, but possible on all earlier versions) - Red Hat Cluster Suite
- Red Hat Resilient Storage
- Global File System 2 (GFS2)
- Writing a large file (usually several gigabytes) to the GFS2 filesystem
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.