Chapter 2. Installing and Configuring Ceph Metadata Servers (MDS)
The Ceph Metadata Server (MDS) node runs the MDS daemon (
ceph-mds), which manages metadata related to files stored on the Ceph File System. The MDS daemon also coordinates access to the shared Ceph Storage Cluster.
The following procedure assumes that:
- You have a working Ceph Storage Cluster (see the Storage Cluster Installation chapter in the Installation Guide for Red Hat Enterprise Linux or Installation Guide for Ubuntu).
- You have an administration node with Ansible installed (see the Installing Ceph Ansible section in the Installation Guide for Red Hat Enterprise Linux or Installation Guide for Ubuntu).
- On the MDS node, you have performed the tasks in listed the Prerequisites chapter of the Installation Guide for Red Hat Enterprise Linux or Installation Guide for Ubuntu. Especially, ensure to enable the Red Hat Ceph Storage 2 Tools repository. See Enabling Ceph Repositories on Red Hat Enterprise Linux or Enabling Ceph Repositories on Ubuntu for details.
2.2. Installing a Ceph Metadata Server
Use the Ansible automation application to install a Ceph Metadata Server. Perform the following steps on the Ansible administration server:
Add a new section
<mdss-hostname>with the host name of the node where you want to install the Ceph Metadata Server.
Navigate to the Ansible configuration directory,
$ cd /usr/share/ceph-ansible
Run the Ansible playbook:
$ ansible-playbook site.yml
2.3. Configuring a Ceph Metadata Server
The Ceph Metadata Servers (MDS) have two modes:
The first MDS that you started becomes active. The rest of the MDS are in standby mode.
When the active MDS becomes unresponsive, the monitor will wait the number of seconds specified by the
mds_beacon_grace option. Then the monitor marks the MDS as
laggy. When this happens, one of the standby servers becomes active depending on your configuration. See Section 2.3.2, “Configuring Standby Daemons” for details.
To change the value of
mds_beacon_grace, add this option to the Ceph configuration file and specify the new value.
A Ceph cluster can have zero or more Ceph File Systems. Ceph File Systems have a human readable name (set by the
fs new command) and an integer ID. The ID is called the File System Cluster ID, or FSCID.
Each Ceph File System has a number of ranks, one by default, which start at zero.
Ranks are how the metadata workload is shared between multiple MDS (
ceph-mds) daemons. The number of ranks is the maximum number of MDS daemons that may be active at one time. Each MDS handles the subset of the file system metadata that is assigned to that rank.
Each MDS daemon initially starts without a rank. The monitor cluster assigns a rank to the daemon. An MDS daemon can only hold one rank at a time. Daemons only lose ranks when they are stopped.
Ranks can be:
Up - A rank is
uponce it is assigned to a daemon.
Failed - A rank is
failedif it is not associated with an instance of the MDS daemon.
Damaged - A rank is damaged when its metadata is corrupted or missing. Damaged ranks will not be assigned to any MDS daemon until you fix the problem and use the
ceph mds repairedcommand on the damaged rank.
max_mds setting controls how many ranks will be created.
The actual number of ranks in the file system is only increased if a spare daemon is available to accept the new rank.
Each daemon has a static name that is set by the administrator when configuring the daemon for the first time. Usually, the host name of the host where the daemon runs is used as the daemon name.
Each time a daemon starts, it is also assigned a GID, which is unique to the process lifetime of the daemon.
Referring to MDS daemons
Most of the administrative commands that refer to MDS daemons accept a flexible argument format.
A rank can be optionally qualified with a leading file system name or ID. If a daemon is in standby mode (meaning that it does not currently have a rank assigned), it can only be referred to by GID or name.
For example, an MDS daemon is called
myhost and has GID 5446. It was assigned rank 0 in the file system
myfs, which has FSCID 3. The following examples show possible forms of the
ceph mds fail 5446 # GID ceph mds fail myhost # Daemon name ceph mds fail 0 # Unqualified rank ceph mds fail 3:0 # FSCID and rank ceph mds fail myfs:0 # File system name and rank
2.3.2. Configuring Standby Daemons
There are four configuration settings that control how daemons behave in standby mode:
These settings can be set in the Ceph configuration file (
ceph.conf by default) on the host where the MDS daemon runs as opposed to the one on the monitor node. The MDS daemon loads these settings when it starts and sends them to the monitor node.
By default, if none of these settings are used, all MDS daemons that do not hold a rank will be used as standby daemons for any rank.
mds_standby_replay option is set to
true for a daemon, this daemon will continuously read the metadata journal of a rank associated with another MDS daemon (the
up rank). This behavior gives the standby replay daemon a more recent metadata cache and makes the failover process faster if the daemon serving the rank fails.
up rank can only have one standby replay daemon assigned to it. If two daemons are both set to be standby replay then one of them becomes a normal non-replay standby daemon.
mon_force_standby_active option is set to
false, then a standby replay daemon is only used as a standby for the rank that it is following. If another rank fails, the standby replay daemon will not be used as a replacement, even if no other standby daemons are available. By default,
mon_force_standby_active is set to
Standby for Name
When setting the
mds_standby_for_name option, the standby daemon only takes over a failed rank if the name of the daemon that previously held the rank matches the given name.
Standby for Rank
mds_standby_for_rank option to configure the standby daemon to only take over the specified rank. If another rank fails, this daemon will not replace it.
If you have multiple file systems, use this option in conjunction with the
mds_standby_for_fscid option to specify which file system rank you target.
Standby for FSCID
mds_standby_for_fscid option is used in conjunction with
mds_standby_for_rank it only specifies which file system rank is referred to.
mds_standby_for_rank is not set, then setting
mds_standby_for_fscid causes the standby daemon to target any rank in the specified FSCID.
mds_standby_for_fscid if you want to use the standby daemon for any rank, but only within a particular file system.
For more information about MSD configuration options, see Configuration Reference.
The following examples of parts of the Ceph configuration file can be:
- in the main Ceph configuration file present on all servers
- in different configuration files on each server that contain just configuration related to that server
Two MDS daemons 'a' and 'b' acting as a pair, where whichever one has not currently assigned a rank will be the standby replay follower of the other:
[mds.a] mds_standby_replay = true mds_standby_for_rank = 0 [mds.b] mds_standby_replay = true mds_standby_for_rank = 0
Two MDS clusters
There are two file systems and four MDS daemons, each file has a pair of daemons:
[mds.a] mds_standby_for_fscid = 1 [mds.b] mds_standby_for_fscid = 1 [mds.c] mds_standby_for_fscid = 2 [mds.d] mds_standby_for_fscid = 2