kernel: Possible SYN flooding on port #. Sending cookies.

Resolution

If required, refer to the below Root Cause section to obtain an understanding of TCP SYN, TCP handshake, listening sockets, SYN flood, and SYN cookies. An understanding of these terms is recommended before investigating or implementing any changes.

Table of Contents

• Determine whether the traffic is valid or malicious
• If the traffic is malicious
• If the traffic is valid
  • Confirm the application is accepting new connections
  • If the application is accepting new connections
    • Increase kernel socket backlog limit
    • Increase application socket listen backlog
    • Confirm the change in application behaviour

Determine whether the traffic is valid or malicious

Use application debugging, network monitoring tools, or work with your network team or service provider.

This requires an understanding of:

• the application's expected workload
• expected client IP addresses
• and expected client behaviour

Use the netstat or ss commands to inspect TCP socket states as follows, where X is the port number reported in the Possible SYN flooding on port X message:

netstat -nta | egrep "State|X"
ss -nta '( dport = :X )'

Having many sockets in the SYN-RECV state could mean a malicious "SYN flood" attack, though this is not the only type of malicious attack. You may also wish to inspect the source IP addresses of traffic to the port in question to confirm if client IPs are expected or unexpected.

The SystemTap script at Where are TCP SYNs coming from? can be used to monitor valid incoming SYNs to sockets in LISTEN state, even SYNs which are later rejected as SYN Flood or with SYN Cookies.

Use the tcpdump command to capture network traffic. Use the Packet Capture Syntax Generator to generate meaningful command options. Also refer to How to capture network packets with tcpdump? or the manual page man tcpdump.

A packet capture which displays many SYNs, which the server responds to with SYN+ACK, but the client never replies with the final ACK could mean a traditional "SYN flood" attack, though this is not the only type of malicious attack.

It is up to you to determine whether incoming traffic is valid or not. Red Hat have no knowledge of your application traffic or environment or expected client addresses. Red Hat can help to use these commands to extract meaningful results, but the decision whether traffic is valid or malicious is up to you and your business to make.

If the traffic is malicious

Work with your network team or service provider to block the traffic before it reaches the listening system or your network.

You may also use the iptables firewall to block traffic using the limit or hashlimit or connlimit match extensions. For full syntax and examples see:

• RHEL 7: man iptables-extensions
• RHEL 5 and 6: man iptables

Note that Red Hat are able to assist with usage of the iptables commands but are not able to write firewall rules to resolve malicious attacks against customers. Such an action would be development and implementation of security policy which is outside the Production Support Scope of Coverage.

If the traffic is valid

Confirm the application is accepting new connections

Confirm the application is actually making the accept() system call to move new connections out of the socket backlog.

Use the strace command as described at How do I use strace to trace system calls made by a command? or use application-specific debugging.

If the application is not calling accept() at all, or is calling it slower than expected, then debug the application to determine why it is not accepting new connections fast enough.

If the application is accepting new connections

If you confirm that application is accepting new connections and the rate of valid traffic is too high for the application, then two changes must be made to allow this listening application to cope with the workload.

These changes are:

• the kernel's socket backlog limit
• the application's socket listen backlog

Both of these must be changed. There is no point changing one but not the other.

Increase kernel socket backlog limit

The kernel's socket backlog limit is controlled by the net.core.somaxconn kernel tunable.

View the current value of the tunable with the command:

# sysctl net.core.somaxconn
net.core.somaxconn = 128

Increase the value with a command such as:

# sysctl -w net.core.somaxconn=2048
net.core.somaxconn = 2048

Confirm the change by viewing again:

# sysctl net.core.somaxconn
net.core.somaxconn = 2048

Persist this change across reboots by entering the corresponding line into /etc/sysctl.conf:

# echo "net.core.somaxconn = 2048" >> /etc/sysctl.conf

Note that your value may not be 2048, it may be smaller or larger.

After changing this tunable, restart the application for the changes to take effect at the next listen() call.

Note: On some systems, it may also be necessary to change the limit of currently-handshaking (SYN-RECV and waiting for ACK) connections on a socket with the kernel tunable:

# sysctl net.ipv4.tcp_max_syn_backlog
net.ipv4.tcp_max_syn_backlog = 512

Increase with sysctl -w and persist across reboots with /etc/sysctl.conf as per the examples above.

After changing this tunable, restart the application for the changes to take effect at the next listen() call.

Increase application socket listen backlog

The application's socket listen backlog is applied when the application makes the listen() system call against its socket.

This example in the C language shows the change from a small listen backlog to a larger backlog:

-   rc = listen(sockfd, 128);    /* old line */
+   rc = listen(sockfd, 2048);   /* new line */
    if (rc < 0)
    {
        perror("listen() failed");
        close(sockfd);
        exit(-1);
    }

The manual page for man listen shows the syntax in C:

int listen(int sockfd, int backlog);

Other programming languages may implement the listen backlog with a different syntax.

An application may even make the listen backlog a configurable value.

After changing the application listen backlog, recompile (if written in a compiled language) or restart (if written in an interpreted language or if configuration is changed) the application for the change to apply.

In the event an application has a hard-coded listen backlog which cannot be applied, an unsupported method to override the listen() function is described at How can I increase the TCP listen backlog value of a socket when the application has a hardcoded value?.

Some known programming language constructs and configuration options are listed below.

Java

In the Java ServerSocket object the syntax is:

ServerSocket(int port, int backlog)

Python

In Python 2 socket library and Python 3 socket library implement the .listen() method on a socket object like:

Socket.listen(backlog)

Apache

In the Apache Web Server, the listen backlog can be configured in the httpd.conf configuration file:

ListenBacklog 512

nginx

In nginx, the listen backlog can be configured as part of the listen directive:

listen 80 backlog=512;

named

In named, the listen backlog can be configured using the tcp-listen-queue directive, which is 10 by default:

tcp-listen-queue 512;

Squid

In the Squid proxy, the listen backlog can be configured in squid.conf with:

max_filedescriptors 512

If squid is compiled with USE_SELECT, the maximum value for this option is 1024. If the value is not compatible, Squid will log the error WARNING: 'max_filedescriptors X' does not work with select() when the service is started.

OpenSSH (sshd)

The OpenSSH server listen backlog is hard-coded to 128 and cannot be changed:

• Is it possible to increase TCP listen backlog for sshd?

If you believe you have a SSH accept performance issue, please open a support case with Red Hat for investigation.

Samba (smbd)

The Samba listen backlog is hard-coded to 50 and cannot be changed:

• Is it possible to increase TCP listen backlog for smbd?

If you believe you have a Samba accept performance issue, please open a support case with Red Hat for investigation.

Confirm the change in application behaviour

This can be done multiple ways.

1) Run the application as normal. Once the application has started, view the backlog value under the Send-Q in ss -ntlp output.

The following example shows the listen backlog in use is 10:

# ss -ntlp | more
State     Recv-Q  Send-Q  Local Address:Port  Peer Address:Port              
LISTEN    0       10                  *:9001             *:*       users:(("nc",pid=1234,fd=3))
                  ^^
                  value is 10

2) Run the application under strace system call tracer and observe the values passed to the listen() system call.

The following example shows the listen backlog in use is 10:

# strace -fvttTyyx -s 4096 -e socket,bind,listen nc -n4l 9001
socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 3<TCP:[502295]>
bind(3<TCP:[502295]>, {sa_family=AF_INET, sin_port=htons(9001), sin_addr=inet_addr("0.0.0.0")}, 16) = 0
listen(3<TCP:[502295]>, 10) = 0
                        ^^
                        value is 10

The strace method is normally only useful when you can capture the application startup. Applications usually open their listening socket when the application starts, so it is often not useful to attach strace to an already-running process.