What does the error message "feature set mismatch missing 400000000000000" mean, in Red Hat Ceph Storage?

Issue

• What does the error message "feature set mismatch missing 400000000000000" mean, in Red Hat Ceph Storage?

• A client node connecting to an RHCS cluster faces the following messages:

2017-05-22 13:26:23.985496 7fd73c6a0700  0 -- 10.65.8.226:0/1003360 >> 10.65.8.216:6789/0 pipe(0x7fd73805f050 sd=3 :34430 s=1 pgs=0 cs=0 l=1 c=0x7fd73805bcb0).connect protocol feature mismatch, my 83ffffffffffff < peer 483ffffefdfbfff missing 400000000000000

• This can either be for any RHCS client nodes such as RBD clients, RHCS Console nodes or any other nodes. For example, an RBD client may see something like following while trying to map an RBD image from an RHCS cluster.

# rbd map rbd/image1

# journalctl -f
14:01:34 node1 kernel: libceph: mon0 192.168.22.11:6789 feature set mismatch, my 102b84a842a42 < server's 40102b84a842a42, missing 400000000000000
14:01:34 node1 kernel: libceph: mon0 192.168.22.11:6789 missing required protocol features

• The cluster has been moved from legacy tunables to optimal.

# ceph osd crush tunables optimal

# ceph osd crush show-tunables
{
    "choose_local_tries": 0,
    "choose_local_fallback_tries": 0,
    "choose_total_tries": 50,
    "chooseleaf_descend_once": 1,
    "chooseleaf_vary_r": 1,
    "chooseleaf_stable": 1,
    "straw_calc_version": 1,
    "allowed_bucket_algs": 54,
    "profile": "jewel",
    "optimal_tunables": 1,
    "legacy_tunables": 0,
    "minimum_required_version": "jewel",
    "require_feature_tunables": 1,
    "require_feature_tunables2": 1,
    "has_v2_rules": 0,
    "require_feature_tunables3": 1,
    "has_v3_rules": 0,
    "has_v4_buckets": 0,
    "require_feature_tunables5": 1,
    "has_v5_rules": 0
}

• The OS and kernel version on the client machines are:

[root@node1 ~]# uname -a
Linux node1 3.10.0-327.22.2.el7.x86_64 #1 SMP Thu Jun 9 10:09:10 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

[root@node1  ~]# lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 7.2 (Maipo)
Release:        7.2
Codename:       Maipo