Chapter 28. Getting started with Multipath TCP

Multipath TCP (MPTCP) is an extension to the Transmission Control Protocol (TCP). Using Internet Protocol (IP), a host can send packets to a destination. TCP ensures reliable delivery of the data through the Internet and automatically adjusts its bandwidth in response to network load.

This section describes how to:

  • Create a new MPTCP connection
  • Enable the server to use MPTCP
  • Disable MPTCP in the kernel

It also includes the advantages of using MPTCP.

28.1. MPTCP benefits

The Multipath TCP (MPTCP) design improves connection stability. Note, that in MPTCP terminology, links are considered as paths.

The following are the advantages of MPTCP:

  • It allows a connection to simultaneously use multiple network interfaces.
  • In case a connection is bound to a link speed, the usage of multiple links can increase the connection throughput. Note, that in case of the connection is bound to a CPU, the usage of multiple links causes the connection slowdown.
  • It increases the resilience to link failures.

28.2. Preparing RHEL to enable MPTCP support

By default the MPTCP support is disabled in RHEL. Enable MPTCP so that applications that support this feature can use it. Additionally, you have to configure user space applications to force use MPTCP sockets if those applications have TCP sockets by default.

This procedure describes how to use the sysctl tool to enable MPTCP support and prepare RHEL for enabling MPTCP for applications system-wide using a SystemTap script.

Prerequisites

The following packages are installed:

  • kernel-debuginfo
  • kernel-debuginfo-common
  • systemtap
  • systemtap-devel
  • kernel-devel
  • iperf3

Procedure

  1. Enable MPTCP sockets in the kernel:

    # echo "net.mptcp.enabled=1" > /etc/sysctl.d/90-enable-MPTCP.conf
    # sysctl -p /etc/sysctl.d/90-enable-MPTCP.conf
  2. Verify that MPTCP is enabled in the kernel:

    # sysctl -a | grep mptcp.enabled
    net.mptcp.enabled = 1
  3. Create a mptcp-app.stap file with the following content:

    #!/usr/bin/env stap
    
    %{
    #include <linux/in.h>
    #include <linux/ip.h>
    %}
    
    /* according to [1], RSI contains 'type' and RDX
     * contains 'protocol'.
     * [1] https://github.com/torvalds/linux/blob/master/arch/x86/entry/entry_64.S#L79
     */
    
    function mptcpify () %{
        if (CONTEXT->kregs->si == SOCK_STREAM &&
            (CONTEXT->kregs->dx == IPPROTO_TCP ||
             CONTEXT->kregs->dx == 0)) {
                    CONTEXT->kregs->dx = IPPROTO_MPTCP;
                    STAP_RETVALUE = 1;
        } else {
               STAP_RETVALUE = 0;
        }
    %}
    
    probe kernel.function("__sys_socket") {
            if (mptcpify() == 1) {
                    printf("command %16s mptcpified\n", execname());
            }
    }
  4. Force user space applications to create MPTCP sockets instead of TCP ones:

    # stap -vg mptcp-app.stap

    Note: This operation affects all TCP sockets which are started after the command. The applications will continue using TCP sockets after you interrupt the command above with Ctrl+C.

  5. Alternatively, to allow MPTCP usage to only specific application, you can modify the mptcp-app.stap file with the following content:

    #!/usr/bin/env stap
    
    %{
    #include <linux/in.h>
    #include <linux/ip.h>
    %}
    
    /* according to [1], RSI contains 'type' and RDX
     * contains 'protocol'.
     * [1] https://github.com/torvalds/linux/blob/master/arch/x86/entry/entry_64.S#L79
     */
    
    function mptcpify () %{
    	if (CONTEXT->kregs->si == SOCK_STREAM &&
    	    (CONTEXT->kregs->dx == IPPROTO_TCP ||
    	     CONTEXT->kregs->dx == 0)) {
    		CONTEXT->kregs->dx = IPPROTO_MPTCP;
    		STAP_RETVALUE = 1;
    	} else {
    		STAP_RETVALUE = 0;
    	}
    %}
    
    probe kernel.function("__sys_socket") {
    	cur_proc = execname()
    	if ((cur_proc == @1) && (mptcpify() == 1)) {
    		printf("command %16s mptcpified\n", cur_proc);
    	}
    }
  6. In case of alternative choice, assuming, you want to force the iperf3 tool to use MPTCP instead of TCP. To do so, enter the following command:

    # stap -vg mptcp-app.stap iperf3

  7. After the mptcp-app.stap script installs the kernel probe, the following warnings appear in the kernel dmesg output

    # dmesg
    ...
    [ 1752.694072] Kprobes globally unoptimized
    [ 1752.730147] stap_1ade3b3356f3e68765322e26dec00c3d_1476: module_layout: kernel tainted.
    [ 1752.732162] Disabling lock debugging due to kernel taint
    [ 1752.733468] stap_1ade3b3356f3e68765322e26dec00c3d_1476: loading out-of-tree module taints kernel.
    [ 1752.737219] stap_1ade3b3356f3e68765322e26dec00c3d_1476: module verification failed: signature and/or required key missing - tainting kernel
    [ 1752.737219] stap_1ade3b3356f3e68765322e26dec00c3d_1476 (mptcp-app.stap): systemtap: 4.5/0.185, base: ffffffffc0550000, memory: 224data/32text/57ctx/65638net/367alloc kb, probes: 1
  8. Start the iperf3 server:

    # iperf3 -s
    
    Server listening on 5201
  9. Connect the client to the server:

    # iperf3 -c 127.0.0.1 -t 3

  10. After the connection is established, verify the ss output to see the subflow-specific status:

    # ss -nti '( dport :5201 )'
    
    State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
    ESTAB 0      0      127.0.0.1:41842    127.0.0.1:5201
    cubic wscale:7,7 rto:205 rtt:4.455/8.878 ato:40 mss:21888 pmtu:65535 rcvmss:536 advmss:65483 cwnd:10 bytes_sent:141 bytes_acked:142 bytes_received:4 segs_out:8 segs_in:7 data_segs_out:3 data_segs_in:3 send 393050505bps lastsnd:2813 lastrcv:2772 lastack:2772 pacing_rate 785946640bps delivery_rate 10944000000bps delivered:4 busy:41ms rcv_space:43690 rcv_ssthresh:43690 minrtt:0.008 tcp-ulp-mptcp flags:Mmec token:0000(id:0)/2ff053ec(id:0) seq:3e2cbea12d7673d4 sfseq:3 ssnoff:ad3d00f4 maplen:2
  11. Verify MPTCP counters by using nstat MPTcp* command:

    # nstat MPTcp*
    
    #kernel
    MPTcpExtMPCapableSYNRX          2                  0.0
    MPTcpExtMPCapableSYNTX          2                  0.0
    MPTcpExtMPCapableSYNACKRX       2                  0.0
    MPTcpExtMPCapableACKRX          2                  0.0

28.3. Using iproute2 to configure and enable multiple paths for MPTCP applications

Each MPTCP connection uses a single subflow similar to plain TCP. To leverage the MPTCP benefits specify a higher limit for maximum number of subflows for each MPTCP connection and configure additional endpoints to create those subflows.

Note that MPTCP does not yet support mixed IPv6 and IPv4 endpoints for the same socket. Use endpoints belonging to the same address family.

Prerequisites

  • Application must support MPTCP
  • Server network interface settings:

    • enp4s0: 192.0.2.1/24
    • enp1s0: 198.51.100.1/24
  • Client network interface settings:

    • enp4s0f0: 192.0.2.2/24
    • enp4s0f1: 198.51.100.2/24

Procedure

  1. Set the per connection additional subflow limits to 1 on the server:

    # ip mptcp limits set subflow 1

    Note, that sets a maximum number of additional subflows which each connection can have, excluding the initial one.

  2. Set the per connection and additional subflow limits to 1 on the client:

    # ip mptcp limits set subflow 1 add_addr_accepted 1

  3. Add IP address 198.51.100.1 as a new MPTCP endpoint on the server:

    # ip mptcp endpoint add 198.51.100.1 dev enp1s0 signal

    Important

    You can set the following values for flags to subflow, backup, signal. Setting the flag to;

    • signal, sends an ADD_ADDR packet after the three-way-handshake is completed
    • subflow, sends an MP_JOIN SYN by the client
    • backup, sets the endpoint as a backup address
  4. Start the iperf3 server:

    # iperf3 -s
    
    Server listening on 5201
  5. Connect the client to the server:

    # iperf3 -c 192.0.2.1 -t 3

Verification steps

  1. Verify the connection is established:

    # ss -nti '( sport :5201 )'

  2. Verify the connection and IP address limit:

    # ip mptcp limit show

  3. Verify the newly added endpoint:

    # ip mptcp endpoint show

  4. Verify MPTCP counters by using the nstat MPTcp* command on a server:

    # nstat MPTcp*
    
    #kernel
    MPTcpExtMPCapableSYNRX          2                  0.0
    MPTcpExtMPCapableACKRX          2                  0.0
    MPTcpExtMPJoinSynRx             2                  0.0
    MPTcpExtMPJoinAckRx             2                  0.0
    MPTcpExtEchoAdd                 2                  0.0

Additional resources

  • ip-mptcp(8) man page

28.4. Disabling Multipath TCP in the kernel

This procedure describes how to disable the MPTCP option in the kernel.

Procedure

  • Disable the mptcp.enabled option.

    # echo "net.mptcp.enabled=0" > /etc/sysctl.d/90-enable-MPTCP.conf
    # sysctl -p /etc/sysctl.d/90-enable-MPTCP.conf

Verification steps

  • Verify whether the mptcp.enabled is disabled in the kernel.

    # sysctl -a | grep mptcp.enabled
    net.mptcp.enabled = 0