Why does NFS perform poorly when using XFS and how can I correct this ?

Issue

• With a RHEL 6.4 NFS server under load (writing ~600MB/sec average) performance in unexpectedly slow

• Using perf

  • perf top shows mutex_spin_on_owner is the busiest function

    --- mutex_spin_on_owner
        |
        |--100.00%-- __mutex_lock_slowpath
        |          mutex_lock
        |          |
        |          |--100.00%-- xfs_file_buffered_aio_write
        |          |          xfs_file_aio_write
        |          |          do_sync_readv_writev
        |          |          do_readv_writev
        |          |          vfs_writev
        |          |          nfsd_vfs_write
        |          |          nfsd_write
        |          |          nfsd3_proc_write
        |          |          nfsd_dispatch
        |          |          svc_process_common
        |          |          svc_process
        |          |          nfsd
        |          |          kthread
        |          |          child_rip
        |           --0.00%-- [...]
         --0.00%-- [...]
    

  • perf report shows xfs_file_buffered_aio_write() as the source of the mutex requests.

• Can anything be done to improve the write performance?