Why rbd creation fail with protocol feature mismatch during pv creation in k8s cluster ?

Solution Verified - Updated -

Issue

  • PV creation fail with below error

     Failed to provision volume with StorageClass "rhat-ceph-storage": failed to create rbd image: exit status 110, command output: 2020-02-02 02:22:50.795455 7f792708d100 -1 did not load config file, using default settings. 2020-02-02 02:22:50.807589 7f792708d100 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2020-02-02 02:22:50.814859 7f792707e700 0 -- 10.0.0.1:0/1339522814 >> 10.0.0.2:6789/0 pipe(0x5653cf4ad960 sd=3 :52990 s=1 pgs=0 cs=0 l=1 c=0x5653cf4ab7e0).connect protocol feature mismatch, my aaaabbbbccccddddffff < peer aaaabbbbccc11112222 missing 200000 2020-02-02 02:22:50.814937 7f792707e700 0 -- 10.0.0.1:0/1339522814 >> 10.0.0.2:6789/0 pipe(0x5653cf4ad960 sd=3 :52990 s=1 pgs=0 cs=0 l=1 c=0x5653cf4ab7e0).fault 2020-02-02 02:22:50.816058 7f792707e700 0 -- 10.0.0.1:0/1339522814 >> 10.0.0.2:6789/0 pipe(0x5653cf4ad960 sd=3 :52992 s=1 pgs=0 cs=0 l=1 c=0x5653cf4ab7e0).connect protocol feature mismatch, my aaaabbbbccccddddffff < peer aaaabbbbccc11112222 missing 200000 2020-02-02 02:22:51.016957 7f792707e700 0 -- 10.0.0.1:0/1339522814 >> 10.0.0.2:6789/0 pipe(0x5653cf4ad960 sd=3 :52998 s=1 pgs=0 cs=0 l=1 c=0x5653cf4ab7e0).connect protocol feature mismatch, my aaaabbbbccccddddffff < peer aaaabbbbccc11112222 missing 200000 2020-02-02 02:22:51.417914 7f792707e700 0 -- 10.0.0.1:0/1339522814 >> 10.0.0.2:6789/0 pipe(0x5653cf4ad960 sd=3 :53002 s=1 pgs=0 cs=0 l=1 c=0x5653cf4ab7e0).connect protocol feature mismatch, my aaaabbbbccccddddffff < peer aaaabbbbccc11112222 missing 200000 2020-02-02 02:22:52.218949 7f792707e700 0 -- 10.0.0.1:0/1339522814 >> 10.0.0.2:6789/0 pipe(0x5653cf4ad960 sd=3 :53006 s=1 pgs=0 cs=0 l=1 c=0x5653cf4ab7e0).connect protocol feature mismatch
    
  • If ceph mgr balancer mode set to upmap, the pv creation succeeds but when we disable upmap subsequently pv creation fails.

Environment

  • Red Hat Ceph Storage 3.x
  • Docker 1.13.1

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In