Chapter 25. Getting started with Multipath TCP

Important

The Multipath TCP is provided as a Technology Preview only. Technology Preview features are not supported with Red Hat production Service Level Agreements (SLAs), might not be functionally complete, and Red Hat does not recommend using them for production. These previews provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

See Technology Preview Features Support Scope on the Red Hat Customer Portal for information about the support scope for Technology Preview features.

Multipath TCP (MPTCP) is an extension to the Transmission Control Protocol (TCP). Using Internet Protocol (IP), a host can send packets to a destination. TCP ensures reliable delivery of the data through the Internet and automatically adjusts its bandwidth in response to network load.

The following are the advantages of MPTCP:

  • It enables TCP for use on devices equipped with two or more network interfaces.
  • It allows users to simultaneously use different network interfaces or switch seamlessly from one connection to another.
  • It improves resource usage within the network and resilience to network failure.

This section describes how to:

  • create a new MPTCP connection,
  • use iproute2 to add new subflows and IP addresses to the MPTCP connections, and
  • disable MPTCP in the kernel to avoid applications using MPTCP connections.

25.1. Preparing RHEL to enable MPTCP support

Few applications natively support MPTCP. Mostly, the connection and stream-oriented sockets request TCP protocol in the socket() call to the operating system. You can enable MPTCP support in RHEL using the sysctl tool for natively MPTCP-supported programs. The MPTCP implementation is also designed to allow usage of MPTCP protocol for applications requesting IPPROTO_TCP call to the kernel.

This procedure describes how to enable MPTCP support and prepare RHEL for enabling MPTCP system-wide using a SystemTap script.

Prerequisites

The following packages are installed:

  • kernel-debuginfo
  • kernel-debuginfo-common
  • systemtap
  • systemtap-devel
  • kernel-devel
  • nmap-ncat

Procedure

  1. Enable MPTCP sockets in the kernel:

    # echo "net.mptcp.enabled=1" > /etc/sysctl.d/90-enable-MPTCP.conf
    # sysctl -p /etc/sysctl.d/90-enable-MPTCP.conf
  2. Create a mptcp.stap file with the following content:

    #! /usr/bin/env stap
    
    %{
    #include <linux/in.h>
    #include <linux/ip.h>
    %}
    
    /* according to [1], RSI contains 'type' and RDX
     * contains 'protocol'.
     * [1] https://github.com/torvalds/linux/blob/master/arch/x86/entry/entry_64.S#L79
     */
    
    function mptcpify () %{
        if (CONTEXT->kregs->si == SOCK_STREAM &&
            (CONTEXT->kregs->dx == IPPROTO_TCP ||
             CONTEXT->kregs->dx == 0)) {
                    CONTEXT->kregs->dx = IPPROTO_MPTCP;
                    STAP_RETVALUE = 1;
        } else {
               STAP_RETVALUE = 0;
        }
    %}
    
    probe kernel.function("__sys_socket") {
            if (mptcpify() == 1) {
                    printf("command %16s mptcpified\n", execname());
            }
    }
  3. Replace the TCP socket with MPTCP:

    # stap -vg mptcp.stap

    Note: Use Ctrl+C to convert the connection back to TCP from MPTCP.

  4. Start a server that listens to TCP port 4321:

    # ncat -4 -l 4321

  5. Connect to the server and exchange traffic. For example, the client here writes “Hello world” to the server 5 times, then it terminates the connection.

    # ncat -4 192.0.2.1 4321
    Hello world 1
    Hello world 2
    Hello world 3
    Hello world 4
    Hello world 5

    Press Ctrl+D to quit.

Verification steps

  1. Verify that MPTCP is enabled in the kernel:

    # sysctl -a | grep mptcp.enabled
    net.mptcp.enabled = 1
  2. After the mptcp.stap script installs the kernel probe, the following warnings appear in the kernel dmesg output

    # dmesg
    ...
    [ 1752.694072] Kprobes globally unoptimized
    [ 1752.730147] stap_1ade3b3356f3e68765322e26dec00c3d_1476: module_layout: kernel tainted.
    [ 1752.732162] Disabling lock debugging due to kernel taint
    [ 1752.733468] stap_1ade3b3356f3e68765322e26dec00c3d_1476: loading out-of-tree module taints kernel.
    [ 1752.737219] stap_1ade3b3356f3e68765322e26dec00c3d_1476: module verification failed: signature and/or required key missing - tainting kernel
  3. After the connection is established, verify the ss output to see the subflow-specific status:

    # ss -nti '( dport :4321 )' dst 192.0.2.1
    State Recv-Q Send-Q Local Address:Port    Peer Address:Port Process
    
    ESTAB 0      0          192.0.2.2:60874      192.0.2.1:4321
    cubic wscale:7,7 rto:201 rtt:0.042/0.017 mss:1448 pmtu:1500 rcvmss:536 advmss:1448 cwnd:10 bytes_sent:64 bytes_$cked:65 segs_out:6 segs_in:5 data_segs_out:4 send 2758095238bps lastsnd:57 lastrcv:3054 lastack:57 pacing_rate 540361516$bps delivery_rate 413714280bps delivered:5 rcv_space:29200 rcv_ssthresh:29200 minrtt:0.009 tcp-ulp-mptcp flags:Mmec token:0000(id:0)/4bffe73d(id:0) seq:c11f40d6c5337463 sfseq:1 ssnoff:f7455705 maplen:0
  4. Capture traffic using tcpdump and check for MPTCP sub-option usage:

    # tcpdump -tnni interface tcp port 4321
    client Out IP 192.0.2.2.60802 > 192.0.2.1.4321: Flags [S], seq 3420255622, win 29200, options [mss 1460,sackOK,TS val 411 4539945 ecr 0,nop,wscale 7,mptcp capable v1], length 0
    client In  IP 192.0.2.1.4321 > 192.0.2.2.60802: Flags [S.], seq 2619315374, ack 3420255623, win 28960, options [mss 1460 sackOK,TS val 3241564233 ecr 4114539945,nop,wscale 7,mptcp capable v1 {0xb6f8dc721aee7f64}], length 0
    client Out IP 192.0.2.2.60802 > 192.0.2.1.4321: Flags [.], ack 1, win 229, options [nop,nop,TS val 4114539945 ecr 3241564 233,mptcp capable v1 {0xcc58d5d632a32d13,0xb6f8dc721aee7f64}], length 0
    client Out IP 192.0.2.2.60802 > 192.0.2.1.4321: Flags [P.], seq 1:17, ack 1, win 229, options [nop,nop,TS val 4114539945 ecr 3241564233,mptcp capable v1 {0xcc58d5d632a32d13,0xb6f8dc721aee7f64},nop,nop], length 16
    client In  IP 192.0.2.1.4321 > 192.0.2.2.60802: Flags [.], ack 17, win 227, options [nop,nop,TS val 3241564233 ecr 411459945,mptcp dss ack 1105509586894558345], length 0
    client Out IP 192.0.2.2.60802 > 192.0.2.1.4321: Flags [P.], seq 17:33, ack 1, win 229, options [nop,nop,TS val 4114540939 ecr 3241564233,mptcp dss ack 13265586846326199424 seq 105509586894558345 subseq 17 len 16,nop,nop], length 16

    The tcpdump package is required to run this command.

Additional resources

25.2. Using iproute2 to notify applications about multiple available paths

By default, the MPTCP socket starts with a single subflow but you can add new subflows and IP addresses to the connection once you create it for the first time. This procedure describes how to update per connection limits for subflows and IP addresses, and add new IP addresses (endpoints) to the MPTCP connection.

Note that MPTCP does not yet support mixed IPv6 and IPv4 endpoints for the same socket. Use endpoints belonging to the same address family.

Procedure

  1. Set the per connection and IP address limits to 1 on the server:

    # ip mptcp limits set subflow 1

  2. Set the per connection and IP address limits to 1 on the client:

    # ip mptcp limits set subflow 1 add_addr_accepted 1

  3. Add IP address 198.51.100.1 as a new MPTCP endpoint on the server:

    # ip mptcp endpoint add 198.51.100.1 dev enp1s0 signal

    Important

    You can set the following values for flags to subflow, backup, signal. Setting the flag to;

    • signal, sends an ADD_ADDR packet after the three-way-handshake is completed
    • subflow, sends an MP_JOIN SYN by the client
    • backup, sets the endpoint as a backup address
  4. Start the server binding to 0.0.0.0 with the -k argument to prevent [systemitem]‘ncat’ from closing the listening socket after accepting the first connection and making the server reject MP_JOIN SYN done by the client.

    # ncat -4 0.0.0.0 -k -l 4321

  5. Start the client and connect to the server to exchange traffic. For example, the client here writes “Hello world” to the server 5 times, then it terminates the connection.

    # ncat -4 192.0.2.1 4321
    Hello world 1
    Hello world 2
    Hello world 3
    Hello world 4
    Hello world 5

    Press Ctrl+D to quit.

Verification steps

  1. Verify the connection and IP address limit:

    # ip mptcp limit show

  2. Verify the newly added endpoint:

    # ip mptcp endpoint show

  3. Capture traffic using tcpdump and check for MPTCP sub-option usage:

    # tcpdump -tnni interface tcp port 4321
    client Out IP 192.0.2.2.56868 > 192.0.2.1.4321: Flags [S], seq 3107783947, win 29200, options [mss 1460,sackOK,TS val 2568752336 ecr 0,nop,wscale 7,mptcp capable v1], length 0
    client In  IP 192.0.2.1.4321 > 192.0.2.2.56868: Flags [S.], seq 4222339923, ack 3107783948, win 28960, options [mss 1460,sackOK,TS val 1713130246 ecr 2568752336,nop,wscale 7,mptcp capable v1 {0xf51c07a47cc2ba75}], length 0
    client Out IP 192.0.2.2.56868 > 192.0.2.1.4321: Flags [.], ack 1, win 229, options [nop,nop,TS val 2568752336 ecr 1713130246,mptcp capable v1 {0xb243376cc5af60bd,0xf51c07a47cc2ba75}], length 0
    client Out IP 192.0.2.2.56868 > 192.0.2.1.4321: Flags [P.], seq 1:17, ack 1, win 229, options [nop,nop,TS val 2568752336 ecr 1713130246,mptcp capable v1 {0xb243376cc5af60bd,0xf51c07a47cc2ba75},nop,nop], length 16
    client In  IP 192.0.2.1.4321 > 192.0.2.2.56868: Flags [.], ack 17, win 227, options [nop,nop,TS val 1713130246 ecr 2568752336,mptcp add-addr id 1 198.51.100.1 hmac 0xe445335073818837,mptcp dss ack 5562689076006296132], length 0
    client Out IP 198.51.100.2.42403 > 198.51.100.1.4321: Flags [S], seq 3356992178, win 29200, options [mss 1460,sackOK,TS val 4038525523 ecr 0,nop,wscale 7,mptcp join backup id 0 token 0xad58df1 nonce 0x74a8137f], length 0
    client In  IP 198.51.100.1.4321 > 198.51.100.2.42403: Flags [S.], seq 1680863152, ack 3356992179, win 28960, options [mss 1460,sackOK,TS val 4213669942 ecr 4038525523,nop,wscale 7,mptcp join backup id 0 hmac 0x9eff7a1bf4e65937 nonce 0x77303fd8], length 0
    client Out IP 198.51.100.2.42403 > 198.51.100.1.4321: Flags [.], ack 1, win 229, options [nop,nop,TS val 4038525523 ecr 4213669942,mptcp join hmac 0xdfdc0129424f627ea774c094461328ce49d195bc], length 0
    client In  IP 198.51.100.1.4321 > 198.51.100.2.42403: Flags [.], ack 1, win 227, options [nop,nop,TS val 4213669942 ecr 4038525523,mptcp dss ack 5562689076006296132], length 0

    The tcpdump package is required to run this command.

Additional resources

  • For more information on available endpoint flags, refer to the ip-mptcp(8) man page.

25.3. Disabling Multipath TCP in the kernel

This procedure describes how to disable the MPTCP option in the kernel.

Procedure

  • Disable the mptcp.enabled option.

    # echo "net.mptcp.enabled=0" > /etc/sysctl.d/90-enable-MPTCP.conf
    # sysctl -p /etc/sysctl.d/90-enable-MPTCP.conf

Verification steps

  • Verify whether the mptcp.enabled is disabled in the kernel.

    # sysctl -a | grep mptcp.enabled
    net.mptcp.enabled = 0