ICMP handling not as expected with TPROXY causes PMTUd failure on Red Hat Enterprise Linux 6

Issue

• The ICMP destination unreachable errors corresponding to transparent TCP connections are not being processed by the TCP stack. They are simply routed through. This causes TCP Path MTU Discovery (PMTUd) to fail because it relies on processing of "ICMP Destination Unreachable Fragmentation Needed/Do-Not-Fragment Set" error. This ultimately causes application data transfer failure.

  Server> iptables -t mangle -nL
  Chain PREROUTING (policy ACCEPT)
  target    prot opt source              destination       
  DIVERT    tcp  --  0.0.0.0/0            0.0.0.0/0          socket
  ACCEPT    all  --  0.0.0.0/0            10.0.220.1         
  TPROXY    tcp  --  0.0.0.0/0            0.0.0.0/0          TPROXY redirect 172.16.1.215:4102 mark 0x9/0xffffffff
  TPROXY    tcp  --  0.0.0.0/0            0.0.0.0/0          tcp dpt:80 TPROXY redirect 172.16.1.215:4101 mark 0x9/0xffffffff
  
  Chain INPUT (policy ACCEPT)
  target    prot opt source              destination
  
  Chain FORWARD (policy ACCEPT)
  target    prot opt source              destination
  
  Chain OUTPUT (policy ACCEPT)
  target    prot opt source              destination
  
  Chain POSTROUTING (policy ACCEPT)
  target    prot opt source              destination
  
  Chain DIVERT (1 references)
  target    prot opt source              destination       
  MARK      all  --  0.0.0.0/0            0.0.0.0/0          MARK set 0x9
  ACCEPT    all  --  0.0.0.0/0            0.0.0.0/0
  
  
  
  
  
  Server> ip rule show
  0:     from all lookup local 
  32764:     from all fwmark 0x9 iif eth1 lookup 9 
  32765:     from all fwmark 0x9 iif eth0 lookup 9 
  32766:     from all lookup main 
  32767:     from all lookup default
  
  
  
  Server> ip route show table 9
  local default dev lo  scope host
  
  
  
  Router> tcpdump -nneti eth1 icmp or tcp port 3032
  large packet sent by the server:
  00:1a:a0:36:43:d4 > 00:e0:81:73:1e:0f, ethertype IPv4 (0x0800), length 1466:
  172.16.22.215.3032 > 10.0.4.211.48667: Flags [P.], seq 5:1405, ack 1405, win 17,
  options [nop,nop,TS val 2980158344 ecr 3051678126], length 1400
  
  gateway replies with ICMP unreachable - need to frag:
  00:e0:81:73:1e:0f > 00:1a:a0:36:43:d4, ethertype IPv4 (0x0800), length 590:
  172.16.2.16 > 172.16.22.215: ICMP 10.0.4.211 unreachable - need to frag (mtu 1000),
  length 556
  
  server ignores it and send the same large packet again:
  00:1a:a0:36:43:d4 > 00:e0:81:73:1e:0f, ethertype IPv4 (0x0800), length 1466:
  172.16.22.215.3032 > 10.0.4.211.48667: Flags [P.], seq 5:1405, ack 1405, win 17,
  options [nop,nop,TS val 2980158545 ecr 3051678126], length 1400
  
  again the gateway replies with ICMP unreachable - need to frag:
  00:e0:81:73:1e:0f > 00:1a:a0:36:43:d4, ethertype IPv4 (0x0800), length 590:
  172.16.2.16 > 172.16.22.215: ICMP 10.0.4.211 unreachable - need to frag (mtu 1000),
  length 556