Setting up Infiniband and IPoIB with partitions

Latest response

Hello group, curious if anyone has got this working.
I have 2 hosts on an infiniband fabric that I would like to communicate using a specific partition pkey. I'm using Mellanox ConnectX-3 HCAs and Mellanox SX6036 switches with RHEL 6.4. Subnet Manager is running on the Mellanox switch cluster.

Everything works fine over the default partition. I am able to get the ib0 up on the hosts, slap on an IP and ping between them.

I then created a partition and added the port-guids to it. 'smpquery pkey
' on both hosts lists my newly created pkey. For example:

# smpquery pkey 18 0: 0xffff 0x83e7 0x0000 0x0000 0x0000 0x0000 0x0000 0x0000 ...

I can successfully add a child device (echo 0x83e7 > /sys/class/net/ib0/create_child) and ifup the resulting ib0.83e7 interface on both servers. However, when I configure an IP address on both servers, I cannot ping.

Tcpdump shows 'ARP who has' traffic outbound on the ib0.83e7, so my routing table is good. However the other server never sees this traffic.

ibtracert, ibping, etc work between hosts, and again, IPoIB works just fine with only the default interface up. But introducing the child interface seems to cause a problem.

Am I missing any steps here?

Thanks

  • Melvin
me

Responses