SegmentSmack and FragmentSmack: IP fragments and TCP segments with random offsets may cause a remote denial of service [CVE-2018-5390, CVE-2018-5391]

Overview

Two security flaws named SegmentSmack and FragmentSmack were recently found in the Linux kernel.

A flaw named SegmentSmack was found in the way the Linux kernel handled specially crafted TCP packets. A remote attacker could use this flaw to trigger time and calculation expensive calls to tcp_collapse_ofo_queue() and tcp_prune_ofo_queue() functions by sending specially modified packets within ongoing TCP sessions which could lead to a CPU saturation and hence a denial of service on the system with relatively small bandwidth of the incoming network traffic. In a worst case scenario, an attacker can stall an affected host or device with less than 2 kpps of an attack traffic. Maintaining the denial of service condition requires continuous two-way TCP sessions to a reachable open port, thus the attacks cannot be performed using spoofed IP addresses.

A result of the attack with 4 streams can look like a complete saturation of 4 CPU cores and delays in a network packets processing:

$ top
%Cpu25 :  0.0 us,  0.0 sy,  0.0 ni,  1.4 id,  0.0 wa,  0.0 hi, 98.5 si,  0.0 st
%Cpu26 :  0.0 us,  0.0 sy,  0.0 ni,  1.4 id,  0.0 wa,  0.0 hi, 98.6 si,  0.0 st
%Cpu28 :  0.0 us,  0.3 sy,  0.0 ni,  0.7 id,  0.0 wa,  0.0 hi, 99.0 si,  0.0 st
%Cpu30 :  0.0 us,  0.0 sy,  0.0 ni,  1.4 id,  0.0 wa,  0.0 hi, 98.6 si,  0.0 st
   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
   141 root      20   0       0      0      0 R  97.3  0.0   1:16.33 ksoftirqd/26
   151 root      20   0       0      0      0 R  97.3  0.0   1:16.68 ksoftirqd/28
   136 root      20   0       0      0      0 R  97.0  0.0   0:39.09 ksoftirqd/25
   161 root      20   0       0      0      0 R  97.0  0.0   1:16.48 ksoftirqd/30

A flaw named FragmentSmack was found in the way the Linux kernel handled reassembly of fragmented IPv4 and IPv6 packets. A remote attacker could use this flaw to trigger time and calculation expensive fragment reassembly algorithm by sending specially crafted packets which could lead to a CPU saturation and hence a denial of service on the system.

A result of the 30 kpps attack on the physical system with Intel(R) Xeon(R) D-1587@1.70GHz CPUs and 32 cores in total may look like a complete saturation of a core:

top - 08:59:45 up  1:34,  2 users,  load average: 0.39, 0.15, 0.08
%Cpu9  :  0.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,100.0 si,  0.0 st
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
   54 root      20   0       0      0      0 R  99.7  0.0   0:47.53 ksoftirqd/9

An attack from a single IP host may saturate more than 1 CPU core by forging packets to be sent from different IP addresses. The Linux kernel uses complex algorithm to schedule such IP fragment reassembly among the CPU cores. So such reassembly could be distributed to the different CPU cores, but it is quite harder to achieve this compared to the SegmentSmack flaw. Such an attack from 2 forged IP addresses may look like a complete saturation of 2 cores, but it is harder for an attacker to achieve this:

top - 10:10:36 up 34 min,  2 users,  load average: 0.51, 0.29, 0.15
%Cpu3  :  0.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,100.0 si,  0.0 st
%Cpu7  :  0.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,100.0 si,  0.0 st
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
   24 root      20   0       0      0      0 R 100.0  0.0   1:50.69 ksoftirqd/3
   44 root      20   0       0      0      0 R 100.0  0.0   1:07.11 ksoftirqd/7

Affected Products

SegmentSmack and FragmentSmack attacks are possible due to the algorithms used in the Linux kernel network stack; all the Red Hat products with moderately new Linux kernel versions are affected.

• Red Hat Enterprise Linux 6
• Red Hat Enterprise Linux 7
• Red Hat Enterprise Linux 7 for Real Time
• Red Hat Enterprise Linux 7 for ARM64
• Red Hat Enterprise Linux 7 for IBM Power9
• Red Hat Enterprise Linux 7 for IBM SystemZ
• Red Hat Enterprise Linux Atomic Host
• Red Hat Enterprise MRG 2
• Red Hat Virtualization 4

RHEL-5 is affected by these flaws by a significantly lesser degree. Namely, in our tests only a high-speed attack of 1Mpps (packets, not bytes or bits) was able to barely saturate 1 CPU core. As such, the flaws severity for RHEL5 is considered Moderate.

Resolution

SegmentSmack

No effective workaround or mitigation besides a fixed kernel is known at this time. Red Hat is tracking fixes via Bugzilla ticket 1601704. Red Hat Enterprise Linux kernel updates will be released as they become available.

Updates for Affected Products

*An active EUS subscription is required for access to this patch. Please contact Red Hat sales or your specific sales representative for more information if your account does not have an active EUS subscription. Also see What is the Red Hat Enterprise Linux Extended Update Support Subscription?.

**An active AUS subscription is required for access to this patch. Please contact Red Hat sales or your specific sales representative for more information if your account does not have an active AUS subscription. Also see What is Advanced mission critical Update Support (AUS)?.

***An active TUS subscription is required for access to this patch. Please contact Red Hat sales or your specific sales representative for more information if your account does not have an active TUS subscription.

FragmentSmack

Red Hat is tracking fixes via Bugzilla ticket 1609664. Red Hat Enterprise Linux kernel updates will be released as they become available.

Updates for Affected Products

*An active EUS subscription is required for access to this patch. Please contact Red Hat sales or your specific sales representative for more information if your account does not have an active EUS subscription. Also see What is the Red Hat Enterprise Linux Extended Update Support Subscription?.

**An active AUS subscription is required for access to this patch. Please contact Red Hat sales or your specific sales representative for more information if your account does not have an active AUS subscription. Also see What is Advanced mission critical Update Support (AUS)?.

***An active TUS subscription is required for access to this patch. Please contact Red Hat sales or your specific sales representative for more information if your account does not have an active TUS subscription.

Mitigation

Except installing a fixed kernel, one may try to change the default 4MB and 3MB values of net.ipv4.ipfrag_high_thresh and net.ipv4.ipfrag_low_thresh (and their IPv6 counterparts net.ipv6.ipfrag_high_thresh and net.ipv6.ipfrag_low_thresh) sysctl parameters to 256 kB and 192 kB (respectively) or below. The result is from some to significant CPU saturation drop during an attack, depending on a hardware and environment. For example, this mitigation applied to the 32-cores system mentioned above made a high-speed attack (~500 kpps) not noticeable.

There can be some impact on performance though, due to ipfrag_high_thresh being set to 262144 bytes, as this way only two 64K fragments can fit in the reassembly queue at the same time. For example, there is a risk of breaking applications that rely on large UDP packets.

The following simple script can be used to quickly change to/from the default and lower settings:

#!/bin/sh
if [ "x$1" == "xlow" ]; then
    echo Settinig limits low:
    sysctl -w net.ipv4.ipfrag_low_thresh=196608
    sysctl -w net.ipv4.ipfrag_high_thresh=262144
    sysctl -w net.ipv6.ip6frag_low_thresh=196608
    sysctl -w net.ipv6.ip6frag_high_thresh=262144
    echo
elif [ "x$1" == "xdef" ]; then
    echo Settinig limits default:
    sysctl -w net.ipv4.ipfrag_high_thresh=4194304
    sysctl -w net.ipv4.ipfrag_low_thresh=3145728
    sysctl -w net.ipv6.ip6frag_high_thresh=4194304
    sysctl -w net.ipv6.ip6frag_low_thresh=3145728
    echo
fi
echo Current values:
sysctl net.ipv4.ipfrag_low_thresh
sysctl net.ipv4.ipfrag_high_thresh
sysctl net.ipv6.ip6frag_low_thresh
sysctl net.ipv6.ip6frag_high_thresh

Acknowledgments

Red Hat thanks Juha-Matti Tilli from Aalto University, Department of Communications and Networking and Nokia Bell Labs for reporting this vulnerability.