Chapter 4. Configuring an InfiniBand subnet manager

All InfiniBand networks must have a subnet manager running for the network to function. This is true even if two machines are connected directly with no switch involved.

It is possible to have more than one subnet manager. In that case one acts as a master and another subnet manager acts as a slave that will take over in case the master subnet manager fails.

Most InfiniBand switches contain an embedded subnet manager. However, if you need a more up-to-date subnet manager or if you require more control, use the OpenSM subnet manager provided by Red Hat Enterprise Linux.

4.1. Installing the OpenSM subnet manager

This section describes how to install the OpenSM subnet manager.

Procedure

  1. Install the opensm package:

    # yum install opensm
  2. Configure OpenSM if the default installation does not match your environment.

    If only one InfiniBand port is installed, the host should act as the master subnet manager, and no custom changes are needed. The default configuration works without any modification.

  3. Enable and start the opensm service:

    # systemctl enable --now opensm

Additional resources

  • For a list of command line options for the opensm service, as well as additional descriptions of the partitions configurations, Quality of Service (QoS), and other advanced topics, see the opensm(8) man page.

4.2. Configuring OpenSM using the simple method

This section describes how to configure OpenSM if you do not need any customized settings.

Prerequisites

  • One or more InfiniBand ports are installed in the server.

Procedure

  1. Obtain the GUIDs for the ports using the ibstat utility:

    # ibstat -d device_name
    CA 'mlx4_0'
       CA type: MT4099
       Number of ports: 2
       Firmware version: 2.42.5000
       Hardware version: 1
       Node GUID: 0xf4521403007be130
       System image GUID: 0xf4521403007be133
       Port 1:
          State: Active
          Physical state: LinkUp
          Rate: 56
          Base lid: 3
          LMC: 0
          SM lid: 1
          Capability mask: 0x02594868
          Port GUID: 0xf4521403007be131
          Link layer: InfiniBand
       Port 2:
          State: Down
          Physical state: Disabled
          Rate: 10
          Base lid: 0
          LMC: 0
          SM lid: 0
          Capability mask: 0x04010000
          Port GUID: 0xf65214fffe7be132
          Link layer: Ethernet
    Note

    Some InfiniBand adapters use the same GUID for the node, system, and port.

  2. Edit the /etc/sysconfig/opensm file and set the GUIDs in the GUIDS parameter:

    GUIDS="GUID_1 GUID_2"
  3. Optionally, set the PRIORITY parameter if multiple subnet managers are available in your subnet. For example:

    PRIORITY=15

Additional resources

  • For additional information on the parameters you can set in /etc/sysconfig/opensm, see the documentation in that file.

4.3. Configuring OpenSM by editing the opensm.conf file

This section describes how to configure OpenSM by editing the /etc/rdma/opensm.conf file. Use this method to customize the OpenSM configuration if only one InfiniBand port is available.

Prerequisites

  • Only one InfiniBand port is installed in the server.

Procedure

  1. Edit the /etc/rdma/opensm.conf file and customize the settings to match your environment.
  2. Restart the opensm service:

    # systemctl restart opensm

Additional resources

  • When you install an updated opensm package, the yum utility stores the new OpenSM configuration file as /etc/rdma/opensm.conf.rpmnew. Compare this file with your customized /etc/rdma/opensm.conf file and manually incorporate the changes.

4.4. Configuring multiple OpenSM instances

This section describes how to set up multiple instances of OpenSM.

Prerequisites

  • One or more InfiniBand ports are installed in the server.

Procedure

  1. Optionally, copy the /etc/rdma/opensm.conf file to /etc/rdma/opensm.conf.orig file:

    # cp /etc/rdma/opensm.conf /etc/rdma/opensm.conf.orig

    When you install an updated opensm package, the yum utility overrides the /etc/rdma/opensm.conf. With the copy created in this step, you can compare the previous and new file to identify changes and incorporate them manually in the instance-specific opensm.conf files.

  2. Create a copy of the /etc/rdma/opensm.conf file:

    # cp /etc/rdma/opensm.conf /etc/rdma/opensm.conf.1

    For each instance you create, append a unique and continuous number to the copy of the configuration file.

  3. Edit the copy you created in the previous step, and customize the settings for the instance to match your environment. For example, set the guid, subnet_prefix, and logdir parameters.
  4. Optionally, create a partitions.conf file with a unique name specifically for this subnet and reference that file in the partition_config_file parameter in the corresponding copy of the opensm.conf file.
  5. Repeat the previous steps for each instance you want to create.
  6. Start the opensm service:

    # systemctl start opensm

    The opensm service automatically starts a unique instance for each opensm.conf.* file in the /etc/rdma/ directory. If multiple opensm.conf.* files exist, the service ignores settings in the /etc/sysconfig/opensm file as well as in the base /etc/rdma/opensm.conf file.

Additional resources

  • When you install an updated opensm package, the yum utility stores the new OpenSM configuration file as /etc/rdma/opensm.conf.rpmnew. Compare this file with your customized /etc/rdma/opensm.conf.\* files and manually incorporate the changes.

4.5. Creating a partition configuration

This section describes how to create InfiniBand partition configurations for OpenSM. Partitions enable administrators to create subnets on InfiniBand similar to Ethernet VLANs.

Important

If you define a partition with a specific speed, such as 40 Gbps, all hosts within this partition must support at least this speed. If a host does not meet the speed requirements, it cannot join the partition. Therefore, set the speed of a partition to the lowest speed supported by any host with permission to join the partition.

Prerequisites

  • One or more InfiniBand ports are installed in the server.

Procedure

  1. Edit the /etc/rdma/partitions.conf file and configure the partitions.

    Note

    All fabrics must contain the 0x7fff partition, and all switches and all hosts must belong to that fabric.

    For example, add the following content to the file to create the 0x7fff default partition at a reduced speed of 10 Gbps, and a partition 0x0002 with a speed of 40 Gbps:

    # For reference:
    # IPv4 IANA reserved multicast addresses:
    #   http://www.iana.org/assignments/multicast-addresses/multicast-addresses.txt
    # IPv6 IANA reserved multicast addresses:
    #   http://www.iana.org/assignments/ipv6-multicast-addresses/ipv6-multicast-addresses.xml
    #
    # mtu =
    #   1 = 256
    #   2 = 512
    #   3 = 1024
    #   4 = 2048
    #   5 = 4096
    #
    # rate =
    #   2  =   2.5 GBit/s
    #   3  =  10   GBit/s
    #   4  =  30   GBit/s
    #   5  =   5   GBit/s
    #   6  =  20   GBit/s
    #   7  =  40   GBit/s
    #   8  =  60   GBit/s
    #   9  =  80   GBit/s
    #   10 = 120   GBit/s
    
    Default=0x7fff, rate=3, mtu=4, scope=2, defmember=full:
        ALL, ALL_SWITCHES=full;
    Default=0x7fff, ipoib, rate=3, mtu=4, scope=2:
        mgid=ff12:401b::ffff:ffff   # IPv4 Broadcast address
        mgid=ff12:401b::1           # IPv4 All Hosts group
        mgid=ff12:401b::2           # IPv4 All Routers group
        mgid=ff12:401b::16          # IPv4 IGMP group
        mgid=ff12:401b::fb          # IPv4 mDNS group
        mgid=ff12:401b::fc          # IPv4 Multicast Link Local Name Resolution group
        mgid=ff12:401b::101         # IPv4 NTP group
        mgid=ff12:401b::202         # IPv4 Sun RPC
        mgid=ff12:601b::1           # IPv6 All Hosts group
        mgid=ff12:601b::2           # IPv6 All Routers group
        mgid=ff12:601b::16          # IPv6 MLDv2-capable Routers group
        mgid=ff12:601b::fb          # IPv6 mDNS group
        mgid=ff12:601b::101         # IPv6 NTP group
        mgid=ff12:601b::202         # IPv6 Sun RPC group
        mgid=ff12:601b::1:3         # IPv6 Multicast Link Local Name Resolution group
        ALL=full, ALL_SWITCHES=full;
    
    ib0_2=0x0002, rate=7, mtu=4, scope=2, defmember=full:
            ALL, ALL_SWITCHES=full;
    ib0_2=0x0002, ipoib, rate=7, mtu=4, scope=2:
        mgid=ff12:401b::ffff:ffff   # IPv4 Broadcast address
        mgid=ff12:401b::1           # IPv4 All Hosts group
        mgid=ff12:401b::2           # IPv4 All Routers group
        mgid=ff12:401b::16          # IPv4 IGMP group
        mgid=ff12:401b::fb          # IPv4 mDNS group
        mgid=ff12:401b::fc          # IPv4 Multicast Link Local Name Resolution group
        mgid=ff12:401b::101         # IPv4 NTP group
        mgid=ff12:401b::202         # IPv4 Sun RPC
        mgid=ff12:601b::1           # IPv6 All Hosts group
        mgid=ff12:601b::2           # IPv6 All Routers group
        mgid=ff12:601b::16          # IPv6 MLDv2-capable Routers group
        mgid=ff12:601b::fb          # IPv6 mDNS group
        mgid=ff12:601b::101         # IPv6 NTP group
        mgid=ff12:601b::202         # IPv6 Sun RPC group
        mgid=ff12:601b::1:3         # IPv6 Multicast Link Local Name Resolution group
        ALL=full, ALL_SWITCHES=full;