Chapter 4. Ceph on-wire encryption

You can enable encryption for all Ceph traffic over the network with the messenger version 2 protocol. The secure mode setting for messenger v2 encrypts communication between Ceph daemons and Ceph clients, giving you end-to-end encryption.

The second version of Ceph’s on-wire protocol, msgr2, includes several new features:

  • A secure mode encrypting all data moving through the network.
  • Encapsulation improvement of authentication payloads.
  • Improvements to feature advertisement and negotiation.

The Ceph daemons bind to multiple ports allowing both the legacy, v1-compatible, and the new, v2-compatible, Ceph clients to connect to the same storage cluster. Ceph clients or other Ceph daemons connecting to the Ceph Monitor daemon will try to use the v2 protocol first, if possible, but if not, then the legacy v1 protocol will be used. By default, both messenger protocols, v1 and v2, are enabled. The new v2 port is 3300, and the legacy v1 port is 6789, by default.

The messenger v2 protocol has two configuration options that control whether the v1 or the v2 protocol is used:

  • ms_bind_msgr1 - This option controls whether a daemon binds to a port speaking the v1 protocol; it is true by default.
  • ms_bind_msgr2 - This option controls whether a daemon binds to a port speaking the v2 protocol; it is true by default.

Similarly, two options control based on IPv4 and IPv6 addresses used:

  • ms_bind_ipv4 - This option controls whether a daemon binds to an IPv4 address; it is true by default.
  • ms_bind_ipv6 - This option controls whether a daemon binds to an IPv6 address; it is true by default.
Note

Ceph daemons or clients using messenger protocol v1 or v2 can implement a throttle, that is, a mechanism for limiting message queue growth. In rare cases, a daemon or client can exceed its throttle, which causes possible delays in message processing. Once throttle limit is hit, you get the following low-level warning message: 

Throttler Limit has been hit.
Some message processing may be significantly delayed.

The msgr2 protocol supports two connection modes:

  • crc

    • Provides strong initial authentication when a connection is established with cephx.
    • Provides a crc32c integrity check to protect against bit flips.
    • Does not provide protection against a malicious man-in-the-middle attack.
    • Does not prevent an eavesdropper from seeing all post-authentication traffic.

  • secure

    • Provides strong initial authentication when a connection is established with cephx.
    • Provides full encryption of all post-authentication traffic.
    • Provides a cryptographic integrity check.

The default mode is crc.

Ensure that you consider cluster CPU requirements when you plan the Red Hat Ceph Storage cluster, to include encryption overhead.

Important

Using secure mode is currently supported by Ceph kernel clients, such as CephFS and krbd on Red Hat Enterprise Linux. Using secure mode is supported by Ceph clients using librbd, such as OpenStack Nova, Glance, and Cinder.

Address Changes

For both versions of the messenger protocol to coexist in the same storage cluster, the address formatting has changed:

  • Old address format was: IP_ADDR:PORT/CLIENT_ID, for example, 1.2.3.4:5678/91011.
  • New address format is, PROTOCOL_VERSION:IP_ADDR:PORT/CLIENT_ID, for example, v2:1.2.3.4:5678/91011, where PROTOCOL_VERSION can be either v1 or v2.

Because the Ceph daemons now bind to multiple ports, the daemons display multiple addresses instead of a single address. Here is an example from a dump of the monitor map: 

epoch 1
fsid 50fcf227-be32-4bcb-8b41-34ca8370bd17
last_changed 2021-12-12 11:10:46.700821
created 2021-12-12 11:10:46.700821
min_mon_release 14 (nautilus)
0: [v2:10.0.0.10:3300/0,v1:10.0.0.10:6789/0] mon.a
1: [v2:10.0.0.11:3300/0,v1:10.0.0.11:6789/0] mon.b
2: [v2:10.0.0.12:3300/0,v1:10.0.0.12:6789/0] mon.c

Also, the mon_host configuration option and specifying addresses on the command line, using -m, supports the new address format.

Connection Phases

There are four phases for making an encrypted connection:

Banner
On connection, both the client and the server send a banner. Currently, the Ceph banner is ceph 0 0n.
Authentication Exchange
All data, sent or received, is contained in a frame for the duration of the connection. The server decides if authentication has completed, and what the connection mode will be. The frame format is fixed, and can be in three different forms depending on the authentication flags being used.
Message Flow Handshake Exchange
The peers identify each other and establish a session. The client sends the first message, and the server will reply with the same message. The server can close connections if the client talks to the wrong daemon. For new sessions, the client and server proceed to exchanging messages. Client cookies are used to identify a session, and can reconnect to an existing session.
Message Exchange
The client and server start exchanging messages, until the connection is closed.

Additional Resources