4.9. Direct I/O

Direct I/O is a feature of the file system whereby file reads and writes go directly from the applications to the storage device, bypassing the operating system read and write caches. Direct I/O is used only by applications (such as databases) that manage their own caches.
An application invokes direct I/O by opening a file with the O_DIRECT flag. Alternatively, GFS can attach a direct I/O attribute to a file, in which case direct I/O is used regardless of how the file is opened.
When a file is opened with O_DIRECT, or when a GFS direct I/O attribute is attached to a file, all I/O operations must be done in block-size multiples of 512 bytes. The memory being read from or written to must also be 512-byte aligned.

Note

Performing I/O through a memory mapping and also via direct I/O to the same file at the same time may result in the direct I/O being failed with an I/O error. This occurs because the page invalidation required for the direct I/O can race with a page fault generated through the mapping. This is a problem only when the memory mapped I/O and the direct I/O are both performed on the same node as each other, and to the same file at the same point in time. A workaround is to use file locking to ensure that memory mapped (i.e., page faults) and direct I/O do not occur simultaneously on the same file.
The Oracle database, which is one of the main direct I/O using applications, does not memory map the files to which it uses direct I/O and thus is unaffected. In addition, writing to a file that is memory mapped will succeed, as expected, unless there are page faults in flight at that point in time. The mmap system call on its own is safe when direct I/O is in use.
One of the following methods can be used to enable direct I/O on a file:
  • O_DIRECT
  • GFS file attribute
  • GFS directory attribute

4.9.1. O_DIRECT

If an application uses the O_DIRECT flag on an open() system call, direct I/O is used for the opened file.
To cause the O_DIRECT flag to be defined with recent glibc libraries, define _GNU_SOURCE at the beginning of a source file before any includes, or define it on the cc line when compiling.

4.9.2. GFS File Attribute

The gfs_tool command can be used to assign (set) a direct I/O attribute flag, directio, to a GFS file. The directio flag can also be cleared.
You can use the gfs_tool stat filename to check what flags have been set for a GFS file. The output for this command includes a Flags: at the end of the display followed by a listing of the flags that are set for the indicated file.

Usage

Setting the directio Flag
gfs_tool setflag directio File
Clearing the directio Flag
gfs_tool clearflag directio File
File
Specifies the file where the directio flag is assigned.

Example

In this example, the command sets the directio flag on the file named datafile in directory /mnt/gfs.
gfs_tool setflag directio /mnt/gfs/datafile
The following command checks whether the directio flag is set for /mnt/gfs/datafile. The output has been elided to show only the relevant information.
[root@tng3-1 gfs]# gfs_tool stat /mnt/gfs/datafile
  mh_magic = 0x01161970
...

Flags:
  directio

4.9.3. GFS Directory Attribute

The gfs_tool command can be used to assign (set) a direct I/O attribute flag, inherit_directio, to a GFS directory. Enabling the inherit_directio flag on a directory causes all newly created regular files in that directory to automatically inherit the directio flag. Also, the inherit_directio flag is inherited by any new subdirectories created in the directory. The inherit_directio flag can also be cleared.

Usage

Setting the inherit_directio flag
gfs_tool setflag inherit_directio Directory
Clearing the inherit_directio flag
gfs_tool clearflag inherit_directio Directory
Directory
Specifies the directory where the inherit_directio flag is set.

Example

In this example, the command sets the inherit_directio flag on the directory named /mnt/gfs/data.
gfs_tool setflag inherit_directio /mnt/gfs/data
This command displays the flags that have been set for the /mnt/gfs/data directory. The full output has been truncated.
[root@tng3-1 gfs]# gfs_tool stat /mnt/gfs/data
...
Flags:
  inherit_directio