The Ceph File System introduces the following features and enhancements:

The Ceph File System is provided as a Technical Preview and as such, there are several limitations:

The Ceph File System aims to adhere to POSIX semantics wherever possible. For example, in contrast to many other common network file systems like NFS, CephFS maintains strong cache coherency across clients. The goal is for processes using the file system to behave the same when they are on different hosts as when they are on the same host.

However, there are a few places where CephFS diverges from strict POSIX semantics for various reasons:

The following procedure assumes that:

Use the Ansible automation application to install a Ceph Metadata Server. Perform the following steps on the Ansible administration server:

The Ceph Metadata Servers (MDS) have two modes:

The first MDS that you started becomes active. The rest of the MDS are in standby mode.

When the active MDS becomes unresponsive, the monitor will wait the number of seconds specified by the mds_beacon_grace option. Then the monitor marks the MDS as laggy. When this happens, one of the standby servers becomes active depending on your configuration. See Section 2.3.2, “Configuring Standby Daemons” for details.

To change the value of mds_beacon_grace, add this option to the Ceph configuration file and specify the new value.

ceph mds fail 5446     # GID
ceph mds fail myhost   # Daemon name
ceph mds fail 0        # Unqualified rank
ceph mds fail 3:0      # FSCID and rank
ceph mds fail myfs:0   # File system name and rank

  [mds.a]
  mds_standby_replay = true
  mds_standby_for_rank = 0

  [mds.b]
  mds_standby_replay = true
  mds_standby_for_rank = 0

  [mds.a]
  mds_standby_for_fscid = 1

  [mds.b]
  mds_standby_for_fscid = 1

  [mds.c]
  mds_standby_for_fscid = 2

  [mds.d]
  mds_standby_for_fscid = 2

To use the Ceph File System, you must have:

Before creating the Ceph File System, ensure that you have the ceph-common package installed and if not, install it.

To create a Ceph File System:

ceph fs new <file_system_name> <metadata> <pool>

Specify the name of the new Ceph File System and the metadata and data pools, for example:

$ ceph fs new cephfs cephfs-metadata cephfs_data

Once the file system is created, the Ceph Metadata Server (MDS) enters to the active state:

$ ceph mds stat
e5: 1/1/1 up {0=a=up:active}

After creating the Ceph File System, mount it. See Mounting Ceph File Systems for details.

To restrict the Ceph File System clients to the lowest possible level of authority needed, use Ceph authentication capabilities.

CephFS supports the following restrictions:

To restrict clients to only mount and work within a certain directory, use path-based MDS authentication capabilities. For example, to restrict the MDS daemon to write metadata only to a particular directory, specify that directory while creating the client capabilities:

ceph auth get-or-create client.<client-name/id> mon 'allow r' mds 'allow r, allow rw path=<directory>' osd 'allow rw pool=data'

Example

The following example command restricts the MDS to write metadata only to the /home/cephfs/ directory. Also, it restricts the CephFS client to perform read and write operations only within the data pool:

$ ceph auth get-or-create client.1 mon 'allow r' mds 'allow r, allow rw path=/home/cephfs' osd 'allow rw pool=data'
[client.1]
    key = AQACNoZXhrzqIRAABPKHTach4x03JeNadeQ9Uw==

To view the created key:

$ ceph auth get client.1
exported keyring for client.1
[client.1]
    key = AQACNoZXhrzqIRAABPKHTach4x03JeNadeQ9Uw==
    caps mds = "allow r, allow rw path=/home/cephfs"
    caps mon = "allow r"
    caps osd = "allow rw pool=data"

Path restriction using the authentication capabilities is the most common way to restrict clients. See the User Management chapter in the Administration Guide for details on authentication capabilities.

When a client has capabilities that restrict it to a path, use the -r option with the ceph-fuse command so that the client will treat that path as its root:

ceph-fuse -n client.<client-name/id> --keyring=<path_to_keyring> <mount-point> -r <directory>

Example

To instruct the client with ID 1 to treat the /home/cephfs/ directory as its root:

# ceph-fuse -n client.1 --keyring=/etc/ceph/client.1.keyring /mnt/cephfs -r /home/cephf

client.<client-name/id>
    key: <key>
    caps: [mds] allow rw
    caps: [mon] allow r
    caps: [osd] allow rw pool=<pool-name>

To restrict clients from writing data, use r instead of rw:

client.<client-name/id>
    key: <key>
    caps: [mds] allow rw
    caps: [mon] allow r
    caps: [osd] allow r pool=<pool-name>

This does not affect the ability of the clients to update file system metadata for files it has read access to, but it prevents them from persistently writing data in a way that would be visible to other clients.

Example:

To restrict client with id 1 to have read and write access to pool data and read access to pool stack:

client.1
    key: AQAz7EVWygILFRAAdIcuJ12opU/JKyfFmxhuaw==
    caps: [mds] allow rw
    caps: [mon] allow r
    caps: [osd] allow rw pool=data, allow r pool=stack

See the User Management chapter in the Administration Guide for details.

Example

In the following snippet client.0 can modify the pool used for files, but client.1 cannot:

client.0
    key: AQAz7EVWygILFRAAdIcuJ12opU/JKyfFmxhuaw==
    caps: [mds] allow rwp
    caps: [mon] allow r
    caps: [osd] allow rw pool=data

client.1
    key: AQAz7EVWygILFRAAdIcuJ12opU/JKyfFmxhuaw==
    caps: [mds] allow rw
    caps: [mon] allow r
    caps: [osd] allow rw pool=data

To mount a Ceph File System as a kernel client, use the mount utility.

For details on cephx, see the User Management chapter in the Administration Guide.

For more information about mount, see the mount(8) manual page.

To mount a Ceph File System as a FUSE client:

For more information about ceph-fuse see the ceph-fuse(8) manual page.

To automatically mount Ceph File Systems on startup, add them to the /etc/fstab file. The form of the entry depends on how the Ceph File System is mounted.

In all cases, use the _netdev option. This option ensures that the file system is mounted after the networking subsystem to prevent networking issues.

#DEVICE               	   PATH           TYPE     OPTIONS
<mon1-hostanme>:<port>:/,  <mountpoint>   ceph     [name=username
<mon1-hostanme>:<port>:/,        	           ,secret=secretkey|
<mon1-hostanme>:<port>:/		           secretfile=
						   path_to_secretfile],
						   [<mount.options>]

Example

mon1:6789:/,    	 /mnt/cephfs   ceph    	name=admin,
mon2:6789:/,					secretfile=
mon3:6789:/					/etc/ceph/secret.key,
					    	_netdev,
					   	noatime 0 0

#DEVICE                                  PATH          TYPE      OPTIONS
id=<user-ID>[,conf=<configuration_file>] <mount-point> fuse.ceph _netdev,
                                                                 defaults
								 0 0

Examples

id=client1  				/mnt/ceph  fuse.ceph  	_netdev,
								defaults
								0 0

id=myuser,conf=/etc/ceph/ceph.conf  	/mnt/ceph2  fuse.ceph 	_netdev,
								defaults
								0 0

The DEVICE field is a comma-delimited list of options to pass to the command line. Ensure to use the ID (for example, admin, not client.admin). You can pass any valid ceph-fuse option to the command line this way.

umount <mount-point>

Example

# umount /mnt/cephfs

See the umount(8) manual page for details.

fusermount -u <mount-point>

Example

# fusermount -u /mnt/cephfs

See the ceph-fuse(8) manual page for details.