9.4. 配置子网管理器

9.4.1. 确定必要性

Most InfiniBand switches come with an embedded subnet manager. However, if a more up to date subnet manager is required than the one in the switch firmware, or if more complete control than the switch manager allows is required, Red Hat Enterprise Linux 7 includes the opensm subnet manager. All InfiniBand networks must have a subnet manager running for the network to function. This is true even when doing a simple network of two machines with no switch and the cards are plugged in back to back, a subnet manager is required for the link on the cards to come up. It is possible to have more than one, in which case one will act as master, and any other subnet managers will act as slaves that will take over should the master subnet manager fail.

9.4.2. 配置 opensm 主配置文件

The opensm program keeps its master configuration file in /etc/rdma/opensm.conf. Users may edit this file at any time and edits will be kept on upgrade. There is extensive documentation of the options in the file itself. However, for the two most common edits needed, setting the GUID to bind to and the PRIORITY to run with, it is highly recommended that the opensm.conf file is not edited but instead edit /etc/sysconfig/opensm. If there are no edits to the base /etc/rdma/opensm.conf file, it will get upgraded whenever the opensm package is upgraded. As new options are added to this file regularly, this makes it easier to keep the current configuration up to date. If the opensm.conf file has been changed, then on upgrade, it might be necessary to merge new options into the edited file.

9.4.3. 配置 opensm 启动选项

/etc/sysconfig/opensm 文件中的选项控制子网管理器的实际启动方式,以及启动的子网管理器副本数量。例如:在双端口 InfiniBand 卡中,会将每个端口与独立的物理网络相连,就是说需要在每个端口中都有一个运行的子网管理器副本。opensm 子网管理器只会在管理应用程序实例的一个子网,且必须为每个需要管理的子网启动一次。另外,如果有一个以上的 opensm 服务器,则需要为每台服务器设置优先顺序,以决定哪些是从属服务器,哪些是主服务器。
/etc/sysconfig/opensm 文件是用来设定子网管理器优先顺序并控制子网管理器所绑定 GUID 的简单工具。/etc/sysconfig/opensm 文件本身包括其选项的详尽说明。用户只需要阅读该文件并按照其说明操作即可启用 opensm 的故障转移及多结构操作功能。

9.4.4. 创建 P_Key 定义

默认情况下,opensm.conf 会寻找文件 /etc/rdma/partitions.conf 以获取要在其中创建结构的分区列表。所有结构必须包含 0x7fff 子网,且所有交换机及所有主机都必须属于那个结构。可在该结构外另行创建其他分区,且所有主机及交换机不一定是那些附加分区的成员。这样就可以让管理员在 InfiniBand 结构中创建类似以太网 VLAN 的子网。如果使用给定速率(比如 40Gbps)定义某个分区,而在该网络中有一台主机无法达到 40Gbps,则那台主机即使有授权也无法加入该分区,因为它无法满足速度要求,因此建议将分区速率设定为所有有权限加入该分区主机中的最慢速度主机。如果某些主机子网需要较快的速度,则可使用较高的速度创建不同分区。
以下分区文件会得到默认 0x7fff 分区(速度降低 10 Gbps)以及速度为 40 Gbps 的分区 0x0002
~]$ more /etc/rdma/partitions.conf
# For reference:
# IPv4 IANA reserved multicast addresses:
#   http://www.iana.org/assignments/multicast-addresses/multicast-addresses.txt
# IPv6 IANA reserved multicast addresses:
#   http://www.iana.org/assignments/ipv6-multicast-addresses/ipv6-multicast-addresses.xml
#
# mtu =
#   1 = 256
#   2 = 512
#   3 = 1024
#   4 = 2048
#   5 = 4096
#
# rate =
#   2  =   2.5 GBit/s
#   3  =  10   GBit/s
#   4  =  30   GBit/s
#   5  =   5   GBit/s
#   6  =  20   GBit/s
#   7  =  40   GBit/s
#   8  =  60   GBit/s
#   9  =  80   GBit/s
#   10 = 120   GBit/s

Default=0x7fff, rate=3, mtu=4, scope=2, defmember=full:
        ALL, ALL_SWITCHES=full;
Default=0x7fff, ipoib, rate=3, mtu=4, scope=2:
        mgid=ff12:401b::ffff:ffff       # IPv4 Broadcast address
        mgid=ff12:401b::1               # IPv4 All Hosts group
        mgid=ff12:401b::2               # IPv4 All Routers group
        mgid=ff12:401b::16              # IPv4 IGMP group
        mgid=ff12:401b::fb              # IPv4 mDNS group
        mgid=ff12:401b::fc              # IPv4 Multicast Link Local Name Resolution group
        mgid=ff12:401b::101             # IPv4 NTP group
        mgid=ff12:401b::202             # IPv4 Sun RPC
        mgid=ff12:601b::1               # IPv6 All Hosts group
        mgid=ff12:601b::2               # IPv6 All Routers group
        mgid=ff12:601b::16              # IPv6 MLDv2-capable Routers group
        mgid=ff12:601b::fb              # IPv6 mDNS group
        mgid=ff12:601b::101             # IPv6 NTP group
        mgid=ff12:601b::202             # IPv6 Sun RPC group
        mgid=ff12:601b::1:3             # IPv6 Multicast Link Local Name Resolution group
        ALL=full, ALL_SWITCHES=full;

ib0_2=0x0002, rate=7, mtu=4, scope=2, defmember=full:
        ALL, ALL_SWITCHES=full;
ib0_2=0x0002, ipoib, rate=7, mtu=4, scope=2:
        mgid=ff12:401b::ffff:ffff       # IPv4 Broadcast address
        mgid=ff12:401b::1               # IPv4 All Hosts group
        mgid=ff12:401b::2               # IPv4 All Routers group
        mgid=ff12:401b::16              # IPv4 IGMP group
        mgid=ff12:401b::fb              # IPv4 mDNS group
        mgid=ff12:401b::fc              # IPv4 Multicast Link Local Name Resolution group
        mgid=ff12:401b::101             # IPv4 NTP group
        mgid=ff12:401b::202             # IPv4 Sun RPC
        mgid=ff12:601b::1               # IPv6 All Hosts group
        mgid=ff12:601b::2               # IPv6 All Routers group
        mgid=ff12:601b::16              # IPv6 MLDv2-capable Routers group
        mgid=ff12:601b::fb              # IPv6 mDNS group
        mgid=ff12:601b::101             # IPv6 NTP group
        mgid=ff12:601b::202             # IPv6 Sun RPC group
        mgid=ff12:601b::1:3             # IPv6 Multicast Link Local Name Resolution group
        ALL=full, ALL_SWITCHES=full;

9.4.5. 启用 opensm

opensm 服务安装时默认不启动,用户需要自行启动该服务。请作为 root 运行以下命令:
~]# systemctl enable opensm