When RHEL has multiple IPs configured, only one is reachable from a remote network. Or why does RHEL ignore packets when the route for outbound traffic differs from the route of incoming traffic?

Root Cause

• RHEL 6 and above are configured by default to apply Strict Reverse Path Forwarding filtering recommended in RFC 3704 - Ingress Filtering for Multihomed Networks.

• Strict filtering means that when a packet arrives on the system, the kernel takes the source IP of the packet and makes a lookup of its routing table to see if the interface the packet arrived on is the same interface the kernel would use to send a packet to that IP. If the interfaces are the same then the packet has passed the strict filtering test and it is processed normally. If the interfaces are not the same then the packet is discarded without any further processing and in RHEL 7+, the IPReversePathFilter counter is incremented.

• The main effect of strict filtering is that for a given remote IP, the system will only communicate with it via a specific interface. Set up static routes to control which interface responds to a given remote IP or network.

• The filtering method is controlled globally by the sysctl net.ipv4.conf.all.rp_filter described in the kernel documentation: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt. RHEL 6+ override the kernel default value of 0 (disabled) for this parameter and set it to 1 (strict).

  rp_filter - INTEGER
      0 - No source validation.
      1 - Strict mode as defined in RFC3704 Strict Reverse Path
          Each incoming packet is tested against the FIB and if the interface
          is not the best reverse path the packet check will fail.
          By default failed packets are discarded.
      2 - Loose mode as defined in RFC3704 Loose Reverse Path
          Each incoming packet's source address is also tested against the FIB
          and if the source address is not reachable via any interface
          the packet check will fail.
  
      Current recommended practice in RFC3704 is to enable strict mode
      to prevent IP spoofing from DDos attacks. If using asymmetric routing
      or other complicated routing, then loose mode is recommended.
  
      The max value from conf/{all,interface}/rp_filter is used
      when doing source validation on the {interface}.
  
      Default value is 0. Note that some distributions enable it
      in startup scripts.
  

• The most simple way to disable the strict check is to set the sysctl net.ipv4.conf.all.rp_filter to 2 (loose) as this will override the interface-specific settings. Setting net.ipv4.conf.all.rp_filter to 0 (disabled) does not override interface-specific settings so is not recommended.

• Note that without the strict check, the system may respond to a packet via a different interface than it arrived on. Whether this will result in the expected connectivity depends on many factors external to the system such as physical network topology and firewall policies.

• An alternate way to configure a system to function with rp_filter in strict mode is to set up policy based routing: How can I route network traffic such that the packets go out via the same interface they came in?

• Note that in RHEL 5 and earlier, the kernel did not support strict filtering. So in these older releases, the net.ipv4.conf.all.rp_filter parameter only has two possible values, 0 (disabled) and 1 (loose). The default is 1 (loose).