Issue

• If we bond the two Mellanox physical ports ("ib0" & "ib1") then we cannot bond any additional virtual IPoIB PKEY interfaces. Red Hat Enterprise Linux reboots result in a hang in this case. If we do NOT bond the physical ports, then there is no problem bonding any number of additional IPoIB PKEY interfaces.
• Two ConnextX3 interfaces being used as IPoIB:
  • ib0
  • ib1
    P_Key child interfaces are created on those physical parents:
  • ib0
    • ib0.8a06
    • ib0.8106
    • ib0.8406
    • ib0.8706
  • ib1
    • ib1.8a06
    • ib1.8106
    • ib1.8406
    • ib1.8706
      Bonds are then configured on top of the P_Key child interfaces:
  • ib0
    • ib0.8a06 (bond0.8a06)
    • ib0.8106 (bond0.8106)
    • ib0.8406 (bond0.8406)
    • ib0.8706 (bond0.8706)
  • ib1
    • ib1.8a06 (bond0.8a06)
    • ib1.8106 (bond0.8106)
    • ib1.8406 (bond0.8406)
    • ib1.8706 (bond0.8706)
      Everything is working fine up to this point. The system boots, traffic flows as expected. Bonding is then configured on the physical parent interfaces as well:
  • ib0 (bond0)
    • ib0.8a06 (bond0.8a06)
    • ib0.8106 (bond0.8106)
    • ib0.8406 (bond0.8406)
    • ib0.8706 (bond0.8706)
  • ib1 (bond0)
    • ib1.8a06 (bond0.8a06)
    • ib1.8106 (bond0.8106)
    • ib1.8406 (bond0.8406)
    • ib1.8706 (bond0.8706)
      This does not work, boot hangs.