When RHEL has multiple IPs configured, only one is reachable from a remote network. Or why does RHEL ignore packets when the route for outbound traffic differs from the route of incoming traffic?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 6, 7, 8

Issue

  • Why does Red Hat Enterprise Linux 6 invalidate / discard packets when the route for outbound traffic differs from the route of incoming traffic?
  • Why does Red Hat Enterprise Linux 6 differ from Red Hat Enterprise Linux 5 in handling asymmetrically routed packets?
  • Why does Red Hat Enterprise Linux not respond to connection attempts to a second NIC?

Resolution

Before making this change please see the Root Cause section of this article to understand what it does and review alternative solutions.

  • Set the sysctl net.ipv4.conf.all.rp_filter parameter value to 2:

    # sysctl net.ipv4.conf.all.rp_filter=2
    
  • To make this change persistent across reboots add it to the /etc/sysctl.conf file.

Root Cause

  • RHEL 6 and above are configured by default to apply Strict Reverse Path Forwarding filtering recommended in RFC 3704 - Ingress Filtering for Multihomed Networks.

  • Strict filtering means that when a packet arrives on the system, the kernel takes the source IP of the packet and makes a lookup of its routing table to see if the interface the packet arrived on is the same interface the kernel would use to send a packet to that IP. If the interfaces are the same then the packet has passed the strict filtering test and it is processed normally. If the interfaces are not the same then the packet is discarded without any further processing and in RHEL 7+, the IPReversePathFilter counter is incremented.

  • The main effect of strict filtering is that for a given remote IP, the system will only communicate with it via a specific interface. Set up static routes to control which interface responds to a given remote IP or network.

  • The filtering method is controlled globally by the sysctl net.ipv4.conf.all.rp_filter described in the kernel documentation: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt. RHEL 6+ override the kernel default value of 0 (disabled) for this parameter and set it to 1 (strict).

    rp_filter - INTEGER
        0 - No source validation.
        1 - Strict mode as defined in RFC3704 Strict Reverse Path
            Each incoming packet is tested against the FIB and if the interface
            is not the best reverse path the packet check will fail.
            By default failed packets are discarded.
        2 - Loose mode as defined in RFC3704 Loose Reverse Path
            Each incoming packet's source address is also tested against the FIB
            and if the source address is not reachable via any interface
            the packet check will fail.
    
        Current recommended practice in RFC3704 is to enable strict mode
        to prevent IP spoofing from DDos attacks. If using asymmetric routing
        or other complicated routing, then loose mode is recommended.
    
        The max value from conf/{all,interface}/rp_filter is used
        when doing source validation on the {interface}.
    
        Default value is 0. Note that some distributions enable it
        in startup scripts.
    
  • The most simple way to disable the strict check is to set the sysctl net.ipv4.conf.all.rp_filter to 2 (loose) as this will override the interface-specific settings. Setting net.ipv4.conf.all.rp_filter to 0 (disabled) does not override interface-specific settings so is not recommended.

  • Note that without the strict check, the system may respond to a packet via a different interface than it arrived on. Whether this will result in the expected connectivity depends on many factors external to the system such as physical network topology and firewall policies.

  • An alternate way to configure a system to function with rp_filter in strict mode is to set up policy based routing: How can I route network traffic such that the packets go out via the same interface they came in?

  • Note that in RHEL 5 and earlier, the kernel did not support strict filtering. So in these older releases, the net.ipv4.conf.all.rp_filter parameter only has two possible values, 0 (disabled) and 1 (loose). The default is 1 (loose).

Diagnostic Steps

  • Check the value of the rp_filter sysctls:

    $ sysctl -a 2>/dev/null | grep "\.rp_filter"
    net.ipv4.conf.all.rp_filter = 1
    net.ipv4.conf.default.rp_filter = 0
    net.ipv4.conf.enp0s31f6.rp_filter = 0
    net.ipv4.conf.lo.rp_filter = 0
    net.ipv4.conf.tun0.rp_filter = 0
    net.ipv4.conf.virbr0.rp_filter = 0
    net.ipv4.conf.virbr0-nic.rp_filter = 0
    net.ipv4.conf.virbr1.rp_filter = 0
    net.ipv4.conf.virbr1-nic.rp_filter = 0
    net.ipv4.conf.wlp58s0.rp_filter = 0
    
  • For RHEL 7 and above, check the IPReversePathFilter SNMP counter. If packets are being ignored due to strict filtering then this counter will increment each time it happens:

    $ nstat -rsz | grep IPReversePathFilter
    TcpExtIPReversePathFilter       52537                  0.0
    
    $ netstat -s | grep IPReversePathFilter
        IPReversePathFilter: 52537
    
  • For a given remote IP whose packets appear to be ignored by the system, perform a route lookup to see which interface the system will use to reach that remote IP. If this is not the same interface the remote IP's packets are arriving on then strict enforcing will discard them:

    $ ip route get <remote IP>
    

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

10 Comments

It would be nice to get some additional tags added.  The problem I had was specific to iscsi and was difficult to find the correlation.

Thank you for your feedback. If you can describe your problem and how you found the resolution useful, we could try and incorporate that into this article or in another article for your future reference. If you have access to Red Hat support, you can even open a ticket for this and have a support technician actively work on this with you.

Regards,

Siddhesh

Global Support Services, Red Hat

Thanks for publishing this. I have a host with a leg in both a DMZ and a LAN, and packets coming in through the firewall (NAT) to the DMZ address would get dropped on the floor (since the leg in the LAN space had the same address). It was necessary to set rp_filter=0 in this case (2 did not work). Also, after setting conf/default/rp_filter and restarting networking, the existing devices' setting was not changed. Setting them manually worked, as did conf/default via sysctl.conf on reboot.

For temporary change, the two commands

    # echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter  
    # service network restart

are not sufficient.

You need

    # echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter  
    # echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter  

or

    # sysctl -w "net.ipv4.conf.default.rp_filter=2" 
    # sysctl -w "net.ipv4.conf.all.rp_filter=2"

network restart is not needed.

Thanks for (correct) feedback, solution already updated.

The following command lines mentioned above:

# echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter
# service network restart

hardly could be recognizes as "good practice" or even acceptable good style of system/network administration.

First, it is bad idea to use echo for changing values in /proc. The following command line:

# sysctl -w "net.ipv4.conf.default.rp_filter=2"

is more safety and effective replacement of

# echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter

Please note that most of the network init scripts in RHEL6 (see /etc/sysconfig/network-scripts) are adapted to use sysctl tool and echos are removed.

Secondly, there is no need to restart the service network in this particular case. Yes, if the only command line executed so far is:

# sysctl -w "net.ipv4.conf.default.rp_filter=2"

all the interfaces must be re-initialized in order to read the newly set default option. But there is very simple woraround to do this on-the-fly, without losing any conectivity. The execution of the command line:

# sysctl -w "net.ipv4.conf.all.rp_filter=2"

(change all to a particular interface named, if needed) will solve the problem with re-configuring the interfaces and it will flush all the current source tracking.

Pfff, took me half a day to work this out after upgrading a firewall with IPIP tunneling from RHEL 5 to RHEL 6!
A-symmetric routed packets just disappear in the dark hole of Linux networking .. no logging to turn to.

Thank you for this. This all worked out perfectly for me.

After a long time I came to know about this...
Very very thankful information,,,

Tip: to see if rp_filter is affecting traffic for you, enable martian logging:

sysctl -w net.ipv4.conf.all.log_martians=1