Why rbd creation fail with protocol feature mismatch during pv creation in k8s cluster ?

Solution Verified - Updated -

Issue

  • PV creation fail with below error

     Failed to provision volume with StorageClass "rhat-ceph-storage": failed to create rbd image: exit status 110, command output: 2020-02-02 02:22:50.795455 7f792708d100 -1 did not load config file, using default settings. 2020-02-02 02:22:50.807589 7f792708d100 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2020-02-02 02:22:50.814859 7f792707e700 0 -- 10.0.0.1:0/1339522814 >> 10.0.0.2:6789/0 pipe(0x5653cf4ad960 sd=3 :52990 s=1 pgs=0 cs=0 l=1 c=0x5653cf4ab7e0).connect protocol feature mismatch, my aaaabbbbccccddddffff < peer aaaabbbbccc11112222 missing 200000 2020-02-02 02:22:50.814937 7f792707e700 0 -- 10.0.0.1:0/1339522814 >> 10.0.0.2:6789/0 pipe(0x5653cf4ad960 sd=3 :52990 s=1 pgs=0 cs=0 l=1 c=0x5653cf4ab7e0).fault 2020-02-02 02:22:50.816058 7f792707e700 0 -- 10.0.0.1:0/1339522814 >> 10.0.0.2:6789/0 pipe(0x5653cf4ad960 sd=3 :52992 s=1 pgs=0 cs=0 l=1 c=0x5653cf4ab7e0).connect protocol feature mismatch, my aaaabbbbccccddddffff < peer aaaabbbbccc11112222 missing 200000 2020-02-02 02:22:51.016957 7f792707e700 0 -- 10.0.0.1:0/1339522814 >> 10.0.0.2:6789/0 pipe(0x5653cf4ad960 sd=3 :52998 s=1 pgs=0 cs=0 l=1 c=0x5653cf4ab7e0).connect protocol feature mismatch, my aaaabbbbccccddddffff < peer aaaabbbbccc11112222 missing 200000 2020-02-02 02:22:51.417914 7f792707e700 0 -- 10.0.0.1:0/1339522814 >> 10.0.0.2:6789/0 pipe(0x5653cf4ad960 sd=3 :53002 s=1 pgs=0 cs=0 l=1 c=0x5653cf4ab7e0).connect protocol feature mismatch, my aaaabbbbccccddddffff < peer aaaabbbbccc11112222 missing 200000 2020-02-02 02:22:52.218949 7f792707e700 0 -- 10.0.0.1:0/1339522814 >> 10.0.0.2:6789/0 pipe(0x5653cf4ad960 sd=3 :53006 s=1 pgs=0 cs=0 l=1 c=0x5653cf4ab7e0).connect protocol feature mismatch

  • If ceph mgr balancer mode set to upmap, the pv creation succeeds but when we disable upmap subsequently pv creation fails.

Environment

  • Red Hat Ceph Storage 3.x
  • Docker 1.13.1

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content