Chapter 5. Ceph on-wire encryption

Starting with Red Hat Ceph Storage 4 and later, you can enable encryption for all Ceph traffic over the network with the introduction of the messenger version 2 protocol. The secure mode setting for messenger v2 encrypts communication between Ceph daemons and Ceph clients, giving you end-to-end encryption.

The second version of Ceph’s on-wire protocol, msgr2, includes several new features:

  • A secure mode encrypting all data moving through the network.
  • Encapsulation improvement of authentication payloads.
  • Improvements to feature advertisement and negotiation.

The Ceph daemons bind to multiple ports allowing both the legacy, v1-compatible, and the new, v2-compatible, Ceph clients to connect to the same storage cluster. Ceph clients or other Ceph daemons connecting to the Ceph Monitor daemon will try to use the v2 protocol first, if possible, but if not, then the legacy v1 protocol will be used. By default, both messenger protocols, v1 and v2, are enabled. The new v2 port is 3300, and the legacy v1 port is 6789, by default.

The msgr2 protocol supports two connection modes:

  • crc

    • Provides strong initial authentication when a connection is established with cephx.
    • Provides a crc32c integrity check to protect against bit flips.
    • Does not provide protection against a malicious man-in-the-middle attack.
    • Does not prevent an eavesdropper from seeing all post-authentication traffic.

  • secure

    • Provides strong initial authentication when a connection is established with cephx.
    • Provides full encryption of all post-authentication traffic.
    • Provides a cryptographic integrity check.

The default mode is crc.

Ensure that you consider cluster CPU requirements when you plan the Red Hat Ceph Storage cluster, to include encryption overhead.

Important

Using secure mode is currently supported by Ceph kernel clients, such as CephFS and krbd on Red Hat Enterprise Linux 8.2. Using secure mode is supported by Ceph clients using librbd, such as OpenStack Nova, Glance, and Cinder.

Address Changes

For both versions of the messenger protocol to coexist in the same storage cluster, the address formatting has changed:

  • Old address format was, IP_ADDR : PORT / CLIENT_ID, for example, 1.2.3.4:5678/91011.
  • New address format is, PROTOCOL_VERSION : IP_ADDR : PORT / CLIENT_ID, for example, v2:1.2.3.4:5678/91011, where PROTOCOL_VERSION can be either v1 or v2.

Because the Ceph daemons now bind to multiple ports, the daemons display multiple addresses instead of a single address. Here is an example from a dump of the monitor map: 

epoch 1
fsid 50fcf227-be32-4bcb-8b41-34ca8370bd17
last_changed 2019-12-12 11:10:46.700821
created 2019-12-12 11:10:46.700821
min_mon_release 14 (nautilus)
0: [v2:10.0.0.10:3300/0,v1:10.0.0.10:6789/0] mon.a
1: [v2:10.0.0.11:3300/0,v1:10.0.0.11:6789/0] mon.b
2: [v2:10.0.0.12:3300/0,v1:10.0.0.12:6789/0] mon.c

Also, the mon_host configuration option and specifying addresses on the command line, using -m, supports the new address format.

Connection Phases

There are four phases for making an encrypted connection:

Banner
On connection, both the client and the server send a banner. Currently, the Ceph banner is ceph 0 0n.
Authentication Exchange
All data, sent or received, is contained in a frame for the duration of the connection. The server decides if authentication has completed, and what the connection mode will be. The frame format is fixed, and can be in three different forms depending on the authentication flags being used.
Message Flow Handshake Exchange
The peers identify each other and establish a session. The client sends the first message, and the server will reply with the same message. The server can close connections if the client talks to the wrong daemon. For new sessions, the client and server proceed to exchanging messages. Client cookies are used to identify a session, and can reconnect to an existing session.
Message Exchange
The client and server start exchanging messages, until the connection is closed.

Additional Resources