TCP connection timeout received when reject method for TCP connections is set to ICMP

Issue

When configuring iptables to reject TCP connections with an ICMP message, a client application trying to connect should detect that it's being rejected, however it claims that the connection times out.

For example when rejecting TCP connections with a icmp-host-prohibited message:

iptables -p tcp -A INPUT -j REJECT --reject-with icmp-host-prohibited

The first few connections are rejected properly from the client perspective, however at some point, subsequent connections are not rejected, but timed out.