SCTP source interface selection when no response from destination
I'm trying to understand why the multi-homed SCTP association I have setup is not using the second interface IP address to send/respond to HEARTBEAT message to the standby path.
SCTP only ever attempts to use the same source interface to both destination SCTP end-points.
I have the following SCTP multi-homed association with two paths:
Client Server
192.200.2.1 <--> 192.168.130.21 (eth2)
192.201.2.1 <--> 192.168.150.21 (eth3)
The above interfaces are connected to separate networks to provide the redundant paths for the multi-homed association.
I have IP rules at the server end to attempt to send traffic as per following:
- from 192.168.130.21 lookup opnet4-routes
- from 192.168.150.21 lookup opnet5-routes
The above route tables have route entries per:
- opnet4-routes: default via 192.168.130.254 dev eth2
- opnet5-routes: default via 192.168.150.254 dev eth3
I do not have an static routes on the server.
The packet sequence initiated from the client was good for the primary path but not so good for the secondary path:
Client Server
INIT 192.200.2.1 --> 192.168.130.21
192.200.2.1 <-- 192.168.130.21 INIT-ACK
COOKIE_ECHO 192.200.2.1 --> 192.168.130.21
192.200.2.1 <-- 192.168.130.21 COOKIE_ACK
HEARTBEAT 192.200.2.1 --> 192.168.130.21
192.200.2.1 <-- 192.168.130.21 HEARTBEAT_ACK
The heartbeat attempt from the server fails to the second interface of the client since eth3 is being used.
192.201.2.1 <-- 192.168.130.21 HEARTBEAT
The client also attempts to heartbeat the secondary path but the server responds on the wrong interface:
HEARTBEAT 192.201.2.1 --> 192.168.150.21
192.200.2.1 <-- 192.168.130.21 HEARTBEAT_ACK
The server never attempts to send the HEARTBEAT messages using the secondary path interface (192.168.150.21 eth3). Neither will it respond to the HEARTBEAT from the secondary path interface.
I have read that SCTP routing is not source based and that it is based on destination only. I understand this can be solved if I use destination static routes.
However my question is more why is SCTP not even attempting to send on the second interface when it know's it not getting responses from the primary path?
Is there a config item I'm missing that would allow SCTP to attempt the second interface to validate the secondary path after a certain number of attempts?
FYI, the problem I'm trying to solve (as you may have guessed) is to avoid having potentially hundreds of static routes to manage hundreds if SCTP client trying to connect to the server. Hence the use of the IP rules functionality.
Thanks,
Chas.
Responses
CentOS and RHEL should behave the same, they are built from the same sources.
Policy routing with from x.x.x.x rules do not seem to work with SCTP on RHEL 6, the default table is used instead.
We couldn't reproduce the same error on RHEL 7. There's been a lot of changes to IP routing between those releases so we were unable to quantify the change.
Hmm, I don't think you should be policy-routing to try and encourage heartbeats out a certain interface.
I don't expect that a kernel-generated HEARTBEAT chunk would have a source address at route lookup time, not in the same way that a process calling bind() or sctp_bindx() would set a source address on data written into its socket. If I'm correct, then the heartbeat would skip the specific from rules during routing (as the SCTP header does not have a source address set yet) and use the main routing table which would explain why you see the server sending heartbeats out the wrong interface.
You really shouldn't need to setup this complex a network to achieve multi-homing. If I were you, I would not use policy routing at all for this setup. Instead, your main routing tables should look something like this:
client
192.168.130.0/24 via 192.200.2.254 dev eth2
192.168.150.0/24 via 192.201.2.254 dev eth3
server
192.200.2.0/24 via 192.168.130.254 dev eth2
192.201.2.0/24 via 192.168.150.254 dev eth3
To answer your last question, the SCTP Socket API Extensions do not have any option to set a next hop, that is done by IP routing.
I don't believe so, not for heartbeats anyway. The socket owner has only indirect control over heartbeat timing via setsockopt(SCTP_PEER_ADDR_PARAMS).
Let me have a play with this and discuss with our SCTP maintainers.
Do you mind if I open a support case for you so that I can dedicate some proper time to it and ensure a reply for you?
That's expected: SCTP behaviour when primary path is down during INIT
I'll continue further via a support case, which I'll open shortly.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
