FTP service slow and timing out when writing to GFS2 filesystem in Active/Active mode Red Hat High Availability cluster

Solution Unverified - Updated -

Issue

  • We need help to limit memory for filesystem cache. The symptom is that application is intermittent slowness to accessing files in loader, or FTP.

    • Many tasks are showing "blocked for more than 120 seconds" in /var/log/messages:
    kernel: INFO: task intransporter:31104 blocked for more than 120 seconds.
kernel: INFO: task intransporter:31104 blocked for more than 120 seconds.
kernel: INFO: task pdflush:3628 blocked for more than 120 seconds.
kernel: INFO: task intransporter:31104 blocked for more than 120 seconds.
kernel: INFO: task disposer:31102 blocked for more than 120 seconds.
kernel: INFO: task intransporter:31104 blocked for more than 120 seconds.
kernel: INFO: task intransporter:31104 blocked for more than 120 seconds.
kernel: INFO: task intransporter:31104 blocked for more than 120 seconds.
kernel: INFO: task intransporter:17053 blocked for more than 120 seconds.
kernel: INFO: task pdflush:3628 blocked for more than 120 seconds.

Environment

  • Red Hat Enterprise Linux 5.8 (RHEL5.8)

    • Red Hat Enterprise Linux Server 5 (with the High Availability Add on)
    • Red Hat Enterprise Linux Server 6 (with the High Availability Add on)
  • Red Hat High Availability cluster with two or more nodes

  • GFS2 filesystem

    • Filesystem mounted on 2 or more nodes simultaneously
    • GFS2 filesystem performance is slow/lower than expected.
    • Plock rate limit is set to unlimited in /etc/cluster/cluster.conf:
        <gfs_controld plock_rate_limit="0"/>
    <dlm plock_ownership="1" plock_rate_limit="0"/>
    • Majority of file access is to 2 specific directories (files created and removed simultaneously by multiple cluster nodes).

  • Workload is described as "Active/Active write access" from multiple cluster nodes.

    • Very I/O intensive workload
    • Many clients connect to all and create files on the same GFS2 filesystem in the same folder.
    • An application moves files from ftp incoming directory to processing directory on the same GFS2 filesystem.
    • A second application processes the file, but does not change its contents.
    • A third application determines when a file has finished processing and cleans it up.
    • Until the file has been processed and cleaned up, clients poll both shared directories every 10 seconds to determine if processing of that file has finished yet.
    • FTP filesystem access (if FTP is used) is sometimes slow - (200mb/200s) ftp performance both reading and writing to gfs2
    • There are no backups configured for this filesystem.
  • Sar data doesn't show excessive load when the issue occurs. Constant moderate load throughout the day.
  • Issue began after increased load was applied to GFS2 filesystem and clustered application.

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content