Chapter 3. Mounting NFS shares
As a system administrator, you can mount remote NFS shares on your system to access shared data.
3.1. Supported NFS versions
Currently, Red Hat Enterprise Linux 8 supports the following major versions of NFS:
- NFS version 3 (NFSv3) supports safe asynchronous writes and is more robust at error handling than the previous NFSv2; it also supports 64-bit file sizes and offsets, allowing clients to access more than 2 GB of file data.
NFS version 4 (NFSv4) works through firewalls and on the Internet, no longer requires an
rpcbindservice, supports Access Control Lists (ACLs), and utilizes stateful operations.
NFS version 2 (NFSv2) is no longer supported by Red Hat.
Default NFS version
The default NFS version in Red Hat Enterprise Linux 8 is 4.2. NFS clients attempt to mount using NFSv4.2 by default, and fall back to NFSv4.1 when the server does not support NFSv4.2. The mount later falls back to NFSv4.0 and then to NFSv3.
Features of minor NFS versions
Following are the features of NFSv4.2 in Red Hat Enterprise Linux 8:
- Server-side copy
Enables the NFS client to efficiently copy data without wasting network resources using the
Note that only intra (within the same server) copies are supported. There is no support for inter (copying between different servers) copies.
- Sparse files
Enables files to have one or more holes, which are unallocated or uninitialized data blocks consisting only of zeros. The
lseek()operation in NFSv4.2 supports
seek_data(), which enables applications to map out the location of holes in the sparse file.
- Space reservation
Permits storage servers to reserve free space, which prohibits servers to run out of space. NFSv4.2 supports the
allocate()operation to reserve space, the
deallocate()operation to unreserve space, and the
fallocate()operation to preallocate or deallocate space in a file.
- Labeled NFS
- Enforces data access rights and enables SELinux labels between a client and a server for individual files on an NFS file system.
- Layout enhancements
layoutstats()operation, which enables some Parallel NFS (pNFS) servers to collect better performance statistics.
Following are the features of NFSv4.1:
- Enhances performance and security of network, and also includes client-side support for pNFS.
- No longer requires a separate TCP connection for callbacks, which allows an NFS server to grant delegations even when it cannot contact the client: for example, when NAT or a firewall interferes.
- Provides exactly once semantics (except for reboot operations), preventing a previous issue whereby certain operations sometimes returned an inaccurate result if a reply was lost and the operation was sent twice.
3.2. Services required by NFS
This section lists system services that are required for running an NFS server or mounting NFS shares. Red Hat Enterprise Linux starts these services automatically.
Red Hat Enterprise Linux uses a combination of kernel-level support and service processes to provide NFS file sharing. All NFS versions rely on Remote Procedure Calls (RPC) between clients and servers. To share or mount NFS file systems, the following services work together depending on which version of NFS is implemented:
- The NFS server kernel module that services requests for shared NFS file systems.
Accepts port reservations from local RPC services. These ports are then made available (or advertised) so the corresponding remote RPC services can access them. The
rpcbindservice responds to requests for RPC services and sets up connections to the requested RPC service. This is not used with NFSv4.
This process is used by an NFS server to process
MOUNTrequests from NFSv3 clients. It checks that the requested NFS share is currently exported by the NFS server, and that the client is allowed to access it. If the mount request is allowed, the
nfs-mountdservice replies with a Success status and provides the File-Handle for this NFS share back to the NFS client.
This process enables explicit NFS versions and protocols the server advertises to be defined. It works with the Linux kernel to meet the dynamic demands of NFS clients, such as providing server threads each time an NFS client connects. This process corresponds to the
- This is a kernel thread that runs on both clients and servers. It implements the Network Lock Manager (NLM) protocol, which enables NFSv3 clients to lock files on the server. It is started automatically whenever the NFS server is run and whenever an NFS file system is mounted.
This process implements the Network Status Monitor (NSM) RPC protocol, which notifies NFS clients when an NFS server is restarted without being gracefully brought down. The
rpc-statdservice is started automatically by the
nfs-serverservice, and does not require user configuration. This is not used with NFSv4.
This process provides user quota information for remote users. The
rpc-rquotadservice, which is provided by the
quota-rpcpackage, has to be started by user when the
This process provides NFSv4 client and server upcalls, which map between on-the-wire NFSv4 names (strings in the form of
user@domain) and local UIDs and GIDs. For
idmapdto function with NFSv4, the
/etc/idmapd.conffile must be configured. At a minimum, the
Domainparameter should be specified, which defines the NFSv4 mapping domain. If the NFSv4 mapping domain is the same as the DNS domain name, this parameter can be skipped. The client and server must agree on the NFSv4 mapping domain for ID mapping to function properly.
Only the NFSv4 server uses
rpc.idmapd, which is started by the
nfs-idmapdservice. The NFSv4 client uses the keyring-based
nfsidmaputility, which is called by the kernel on-demand to perform ID mapping. If there is a problem with
nfsidmap, the client falls back to using
The RPC services with NFSv4
The mounting and locking protocols have been incorporated into the NFSv4 protocol. The server also listens on the well-known TCP port 2049. As such, NFSv4 does not need to interact with
rpc-statd services. The
nfs-mountd service is still required on the NFS server to set up the exports, but is not involved in any over-the-wire operations.
3.3. NFS host name formats
This section describes different formats that you can use to specify a host when mounting or exporting an NFS share.
You can specify the host in the following formats:
- Single machine
Either of the following:
- A fully-qualified domain name (that can be resolved by the server)
- Host name (that can be resolved by the server)
- An IP address.
- IP networks
Either of the following formats is valid:
a.b.c.dis the network and
zis the number of bits in the netmask; for example
a.b.c.dis the network and
netmaskis the netmask; for example,
@group-nameformat , where
group-nameis the NIS netgroup name.
3.4. Configuring an NFSv3 client to run behind a firewall
The procedure to configure an NFSv3 client to run behind a firewall is similar to the procedure to configure an NFSv3 server to run behind a firewall.
If the machine you are configuring is both an NFS client and an NFS server, follow the procedure described in Configuring the NFSv3-enabled server to run behind a firewall.
The following procedure describes how to configure a machine that is an NFS client only to run behind a firewall.
To allow the NFS server to perform callbacks to the NFS client when the client is behind a firewall, add the
rpc-bindservice to the firewall by running the following command on the NFS client:
firewall-cmd --permanent --add-service rpc-bind
Specify the ports to be used by the RPC service
/etc/nfs.conffile as follows:
[lockd] port=port-number udp-port=upd-port-number
Alternatively, you can specify
Open the specified ports in the firewall by running the following commands on the NFS client:
firewall-cmd --permanent --add-port=<lockd-tcp-port>/tcp firewall-cmd --permanent --add-port=<lockd-udp-port>/udp
Add static ports for
rpc.statdby editing the
[statd]section of the
/etc/nfs.conffile as follows:
Open the added ports in the firewall by running the following commands on the NFS client:
firewall-cmd --permanent --add-port=<statd-tcp-port>/tcp firewall-cmd --permanent --add-port=<statd-udp-port>/udp
Reload the firewall configuration:
# systemctl restart rpc-statd.service
Alternatively, if you specified the
lockdports in the
Update the current values of
# sysctl -w fs.nfs.nlm_tcpport=<tcp-port> # sysctl -w fs.nfs.nlm_udpport=<udp-port>
# systemctl restart rpc-statd.service
3.5. Configuring an NFSv4 client to run behind a firewall
Perform this procedure only if the client is using NFSv4.0. In that case, it is necessary to open a port for NFSv4.0 callbacks.
This procedure is not needed for NFSv4.1 or higher because in the later protocol versions the server performs callbacks on the same connection that was initiated by the client.
To allow NFSv4.0 callbacks to pass through firewalls, set
/proc/sys/fs/nfs/nfs_callback_tcpportand allow the server to connect to that port on the client as follows:
# echo "fs.nfs.nfs_callback_tcpport = <callback-port>" >/etc/sysctl.d/90-nfs-callback-port.conf # sysctl -p /etc/sysctl.d/90-nfs-callback-port.conf
Open the specified port in the firewall by running the following command on the NFS client:
firewall-cmd --permanent --add-port=<callback-port>/tcp
Reload the firewall configuration:
3.6. Installing NFS
This procedure installs all packages necessary to mount or export NFS shares.
# yum install nfs-utils
3.7. Discovering NFS exports
This procedure discovers which file systems a given NFSv3 or NFSv4 server exports.
With any server that supports NFSv3, use the
$ showmount --exports my-server Export list for my-server /exports/foo /exports/bar
With any server that supports NFSv4, mount the root directory and look around:
# mount my-server:/ /mnt/ # ls /mnt/ exports # ls /mnt/exports/ foo bar
On servers that support both NFSv4 and NFSv3, both methods work and give the same results.
3.8. Mounting an NFS share with mount
Mount an NFS share exported from a server by using the
You can experience conflicts in your NFSv4
clientid and their sudden expiration if your NFS clients have the same short hostname. To avoid any possible sudden expiration of your NFSv4
clientid, you must use either unique hostnames for NFS clients or configure identifier on each container, depending on what system you are using. For more information, see the NFSv4 clientid was expired suddenly due to use same hostname on several NFS clients Knowledgebase article.
To mount an NFS share, use the following command:
# mount -t nfs -o options host:/remote/export /local/directory
This command uses the following variables:
- A comma-delimited list of mount options.
- The host name, IP address, or fully qualified domain name of the server exporting the file system you want to mount.
- The file system or directory being exported from the server, that is, the directory you want to mount.
- The client location where /remote/export is mounted.
3.9. Setting up pNFS SCSI on the client
This procedure configures an NFS client to mount a pNFS SCSI layout.
- The NFS server is configured to export an XFS file system over pNFS SCSI.
On the client, mount the exported XFS file system using NFS version 4.1 or higher:
# mount -t nfs -o nfsvers=4.1 host:/remote/export /local/directory
Do not mount the XFS file system directly without NFS.
3.10. Checking pNFS SCSI operations from the client using mountstats
This procedure uses the
/proc/self/mountstats file to monitor pNFS SCSI operations from the client.
List the per-mount operation counters:
# cat /proc/self/mountstats \ | awk /scsi_lun_0/,/^$/ \ | egrep device\|READ\|WRITE\|LAYOUT device 192.168.122.73:/exports/scsi_lun_0 mounted on /mnt/rhel7/scsi_lun_0 with fstype nfs4 statvers=1.1 nfsv4: bm0=0xfdffbfff,bm1=0x40f9be3e,bm2=0x803,acl=0x3,sessions,pnfs=LAYOUT_SCSI READ: 0 0 0 0 0 0 0 0 WRITE: 0 0 0 0 0 0 0 0 READLINK: 0 0 0 0 0 0 0 0 READDIR: 0 0 0 0 0 0 0 0 LAYOUTGET: 49 49 0 11172 9604 2 19448 19454 LAYOUTCOMMIT: 28 28 0 7776 4808 0 24719 24722 LAYOUTRETURN: 0 0 0 0 0 0 0 0 LAYOUTSTATS: 0 0 0 0 0 0 0 0
In the results:
LAYOUTstatistics indicate requests where the client and server use pNFS SCSI operations.
WRITEstatistics indicate requests where the client and server fall back to NFS operations.
3.11. Common NFS mount options
The following are the commonly used options when mounting NFS shares. You can use these options wth manual
mount commands, the
/etc/fstab settings, and
Specifies how the kernel should manage its cache of directory entries for a given mount point. Valid arguments for mode are
Specifies which version of the NFS protocol to use, where version is
4.2. This is useful for hosts that run multiple NFS servers, or to disable retrying a mount with lower versions. If no version is specified, NFS uses the highest version supported by the kernel and the
versis identical to
nfsvers, and is included in this release for compatibility reasons.
- Turns off all ACL processing. This may be needed when interfacing with older versions of Red Hat Enterprise Linux, Red Hat Linux, or Solaris, because the most recent ACL technology is not compatible with older systems.
- Disables file locking. This setting is sometimes required when connecting to very old NFS servers.
- Prevents execution of binaries on mounted file systems. This is useful if the system is mounting a non-Linux file system containing incompatible binaries.
set-group-identifierbits. This prevents remote users from gaining higher privileges by running a
Specifies the numeric value of the NFS server port. If num is
0(the default value), then
rpcbindservice on the remote host for the port number to use. If the NFS service on the remote host is not registered with its
rpcbindservice, the standard NFS port number of TCP 2049 is used instead.
These options set the maximum number of bytes to be transferred in a single NFS read or write operation.
There is no fixed default value for
wsize. By default, NFS uses the largest possible value that both the server and the client support. In Red Hat Enterprise Linux 8, the client and server maximum is 1,048,576 bytes. For more details, see the What are the default and maximum values for rsize and wsize with NFS mounts? KBase article.
Security flavors to use for accessing files on the mounted export. The flavors value is a colon-separated list of one or more security flavors.
By default, the client attempts to find a security flavor that both the client and the server support. If the server does not support any of the selected flavors, the mount operation fails.
sec=sysuses local UNIX UIDs and GIDs. These use
AUTH_SYSto authenticate NFS operations.
sec=krb5uses Kerberos V5 instead of local UNIX UIDs and GIDs to authenticate users.
sec=krb5iuses Kerberos V5 for user authentication and performs integrity checking of NFS operations using secure checksums to prevent data tampering.
sec=krb5puses Kerberos V5 for user authentication, integrity checking, and encrypts NFS traffic to prevent traffic sniffing. This is the most secure setting, but it also involves the most performance overhead.
- Instructs the NFS mount to use the TCP protocol.
3.12. Storing user settings over NFS
If you use GNOME on a system with NFS home directories, you must set the
keyfile back end for the
dconf database. Otherwise,
dconf might not work correctly. With this configuration,
dconf stores settings in the
Ensure that the
glib2-fampackage is installed on the system:
# yum install glib2-fam
Without this package, notifications on configuration changes made on remote machines are not displayed properly.
Create or edit the
/etc/dconf/profile/userfile on every client.
At the very beginning of the
/etc/dconf/profile/userfile, add the following line:
Users must log out and log back in.
keyfileback end to determine whether updates have been made, so settings might not be updated immediately.
3.13. Getting started with FS-Cache
FS-Cache is a persistent local cache that file systems can use to take data retrieved from over the network and cache it on local disk. This helps minimize network traffic for users accessing data from a file system mounted over the network (for example, NFS).
3.13.1. Overview of the FS-Cache
The following diagram is a high-level illustration of how FS-Cache works:
Figure 3.1. FS-Cache Overview
FS-Cache is designed to be as transparent as possible to the users and administrators of a system. Unlike
cachefs on Solaris, FS-Cache allows a file system on a server to interact directly with a client’s local cache without creating an overmounted file system. With NFS, a mount option instructs the client to mount the NFS share with FS-cache enabled. The mount point will cause automatic upload for two kernel modules:
cachefilesd daemon communicates with the kernel modules to implement the cache.
FS-Cache does not alter the basic operation of a file system that works over the network - it merely provides that file system with a persistent place in which it can cache data. For example, a client can still mount an NFS share whether or not FS-Cache is enabled. In addition, cached NFS can handle files that will not fit into the cache (whether individually or collectively) as files can be partially cached and do not have to be read completely up front. FS-Cache also hides all I/O errors that occur in the cache from the client file system driver.
To provide caching services, FS-Cache needs a cache back end. A cache back end is a storage driver configured to provide caching services, which is
cachefiles. In this case, FS-Cache requires a mounted block-based file system, such as
ext3, that supports
bmap and extended attributes as its cache back end.
File systems that support functionalities required by FS-Cache cache back end include the Red Hat Enterprise Linux 8 implementations of the following file systems:
- ext3 (with extended attributes enabled)
FS-Cache cannot arbitrarily cache any file system, whether through the network or otherwise: the shared file system’s driver must be altered to allow interaction with FS-Cache, data storage/retrieval, and metadata setup and validation. FS-Cache needs indexing keys and coherency data from the cached file system to support persistence: indexing keys to match file system objects to cache objects, and coherency data to determine whether the cache objects are still valid.
In Red Hat Enterprise Linux 8, the cachefilesd package is not installed by default and needs to be installed manually.
3.13.2. Performance guarantee
FS-Cache does not guarantee increased performance. Using a cache incurs a performance penalty: for example, cached NFS shares add disk accesses to cross-network lookups. While FS-Cache tries to be as asynchronous as possible, there are synchronous paths, such as
read operations, where this is not possible.
For example, using FS-Cache to cache an NFS share between two computers over an otherwise unladen GigE network likely will not demonstrate any performance improvements on file access. Rather, NFS requests would be satisfied faster from server memory rather than from local disk.
The use of FS-Cache, therefore, is a compromise between various factors. If FS-Cache is being used to cache NFS traffic, for example, it may slow the client down a little, but massively reduce the network and server loading by satisfying read requests locally without consuming network bandwidth.
3.13.3. Using the cache with NFS
NFS will not use the cache unless explicitly instructed. This paragraph shows how to configure an NFS mount by using FS-Cache.
NFS indexes cache contents using NFS file handle, not the file name, which means hard-linked files share the cache correctly.
NFS versions 3, 4.0, 4.1 and 4.2 support caching. However, each version uses different branches for caching.
The cachefilesd package is installed and running. To ensure it is running, use the following command:
# systemctl start cachefilesd # systemctl status cachefilesd
The status must be active (running).
Mount NFS shares with the following option:
# mount nfs-share:/ /mount/point -o fsc
All access to files under
/mount/pointwill go through the cache, unless the file is opened for direct I/O or writing.
3.13.4. Setting up a cache
Currently, Red Hat Enterprise Linux 8 only provides the
cachefiles caching back end. The
cachefilesd daemon initiates and manages
/etc/cachefilesd.conf file controls how
cachefiles provides caching services.
The cache back end works by maintaining a certain amount of free space on the partition hosting the cache. It grows and shrinks the cache in response to other elements of the system using up free space, making it safe to use on the root file system (for example, on a laptop). FS-Cache sets defaults on this behavior, which can be configured via cache cull limits. For more information about configuring cache cull limits, see Cache cull limits configuration.
This procedure shows how to set up a cache.
The cachefilesd package is installed and service has started successfully. To be sure the service is running, use the following command:
# systemctl start cachefilesd # systemctl status cachefilesd
The status must be active (running).
Configure in a cache back end which directory to use as a cache, use the following parameter:
$ dir /path/to/cache
Typically, the cache back end directory is set in
/var/cache/fscache, as in:
$ dir /var/cache/fscache
If you want to change the cache back end directory, the selinux context must be same as
# semanage fcontext -a -e /var/cache/fscache /path/to/cache # restorecon -Rv /path/to/cache
- Replace /path/to/cache with the directory name while setting up cache.
If the given commands for setting selinux context did not work, use the following commands:
# semanage permissive -a cachefilesd_t # semanage permissive -a cachefiles_kernel_t
FS-Cache will store the cache in the file system that hosts
/path/to/cache. On a laptop, it is advisable to use the root file system (
/) as the host file system, but for a desktop machine it would be more prudent to mount a disk partition specifically for the cache.
The host file system must support user-defined extended attributes. FS-Cache uses these attributes to store coherency maintenance information. To enable user-defined extended attributes on a device with ext3 file systems, enter:
# tune2fs -o user_xattr /dev/device
To enable extended attributes for a file system at the mount time, as an alternative, use the following command:
# mount /dev/device /path/to/cache -o user_xattr
Once the configuration file is in place, start up the
# systemctl start cachefilesd
cachefilesdto start at boot time, execute the following command as root:
# systemctl enable cachefilesd
3.13.5. Configuring NFS cache sharing
There are several potential issues to do with NFS cache sharing. Because the cache is persistent, blocks of data in the cache are indexed on a sequence of four keys:
- Level 1: Server details
- Level 2: Some mount options; security type; FSID; uniquifier
- Level 3: File Handle
- Level 4: Page number in file
To avoid coherency management problems between superblocks, all NFS superblocks that require to cache the data have unique Level 2 keys. Normally, two NFS mounts with same source volume and options share a superblock, and therefore share the caching, even if they mount different directories within that volume.
This is an example how to configure cache sharing with different options.
Mount NFS shares with the following commands:
mount home0:/disk0/fred /home/fred -o fsc mount home0:/disk0/jim /home/jim -o fsc
/home/jimlikely share the superblock as they have the same options, especially if they come from the same volume/partition on the NFS server (
To not share the superblock, use the
mountcommand with the following options:
mount home0:/disk0/fred /home/fred -o fsc,rsize=8192 mount home0:/disk0/jim /home/jim -o fsc,rsize=65536
In this case,
/home/jimwill not share the superblock as they have different network access parameters, which are part of the Level 2 key.
To cache the contents of the two subtrees (
/home/fred2) twice with not sharing the superblock, use the following command:
mount home0:/disk0/fred /home/fred1 -o fsc,rsize=8192 mount home0:/disk0/fred /home/fred2 -o fsc,rsize=65536
Another way to avoid superblock sharing is to suppress it explicitly with the
nosharecacheparameter. Using the same example:
mount home0:/disk0/fred /home/fred -o nosharecache,fsc mount home0:/disk0/jim /home/jim -o nosharecache,fsc
However, in this case only one of the superblocks is permitted to use cache since there is nothing to distinguish the Level 2 keys of
To specify the addressing to the superblock, use the
fsc=unique-identifiermount option to set a unique identifier on at least one of the mounts, for example:
mount home0:/disk0/fred /home/fred -o nosharecache,fsc mount home0:/disk0/jim /home/jim -o nosharecache,fsc=jim
Here, the unique identifier
jimis added to the Level 2 key used in the cache for
The user can not share caches between superblocks that have different communications or protocol parameters. For example, it is not possible to share between NFSv4.0 and NFSv3 or between NFSv4.1 and NFSv4.2 because they force different superblocks. Also setting parameters, such as the read size (rsize), prevents cache sharing because, again, it forces a different superblock.
3.13.6. Cache limitations with NFS
There are some cache limitations with NFS:
- Opening a file from a shared file system for direct I/O automatically bypasses the cache. This is because this type of access must be direct to the server.
- Opening a file from a shared file system for either direct I/O or writing flushes the cached copy of the file. FS-Cache will not cache the file again until it is no longer opened for direct I/O or writing.
- Furthermore, this release of FS-Cache only caches regular NFS files. FS-Cache will not cache directories, symlinks, device files, FIFOs and sockets.
3.13.7. Cache cull limits configuration
cachefilesd daemon works by caching remote data from shared file systems to free space on the disk. This could potentially consume all available free space, which could be bad if the disk also contains the root partition. To control this,
cachefilesd tries to maintain a certain amount of free space by discarding old objects, such as less-recently accessed objects, from the cache. This behavior is known as cache culling.
Cache culling is done on the basis of the percentage of blocks and the percentage of files available in the underlying file system. There are settings in
/etc/cachefilesd.conf which control six limits:
- brun N% (percentage of blocks), frun N% (percentage of files)
- If the amount of free space and the number of available files in the cache rises above both these limits, then culling is turned off.
- bcull N% (percentage of blocks), fcull N% (percentage of files)
- If the amount of available space or the number of files in the cache falls below either of these limits, then culling is started.
- bstop N% (percentage of blocks), fstop N% (percentage of files)
- If the amount of available space or the number of available files in the cache falls below either of these limits, then no further allocation of disk space or files is permitted until culling has raised things above these limits again.
The default value of
N for each setting is as follows:
When configuring these settings, the following must hold true:
These are the percentages of available space and available files and do not appear as 100 minus the percentage displayed by the
Culling depends on both bxxx and fxxx pairs simultaneously; the user can not treat them separately.
3.13.8. Retrieving statistical information from the fscache kernel module
FS-Cache also keeps track of general statistical information. This procedure shows how to get this information.
To view the statistical information about FS-Cache, use the following command:
# cat /proc/fs/fscache/stats
FS-Cache statistics includes information about decision points and object counters. For more information, see the following kernel document:
3.13.9. FS-Cache references
This section provides reference information for FS-Cache.
For more information about
cachefilesdand how to configure it, see
man cachefilesd.conf. The following kernel documents also provide additional information:
For general information about FS-Cache, including details on its design constraints, available statistics, and capabilities, see the following kernel document: