SCTP source interface selection when no response from destination

Latest response

I'm trying to understand why the multi-homed SCTP association I have setup is not using the second interface IP address to send/respond to HEARTBEAT message to the standby path.

SCTP only ever attempts to use the same source interface to both destination SCTP end-points.

I have the following SCTP multi-homed association with two paths:

Client           Server
192.200.2.1 <--> 192.168.130.21 (eth2)
192.201.2.1 <--> 192.168.150.21 (eth3)

The above interfaces are connected to separate networks to provide the redundant paths for the multi-homed association.

I have IP rules at the server end to attempt to send traffic as per following:

- from 192.168.130.21 lookup opnet4-routes
- from 192.168.150.21 lookup opnet5-routes

The above route tables have route entries per:

- opnet4-routes: default via 192.168.130.254 dev eth2
- opnet5-routes: default via 192.168.150.254 dev eth3

I do not have an static routes on the server.

The packet sequence initiated from the client was good for the primary path but not so good for the secondary path:

Client                       Server
INIT 192.200.2.1         --> 192.168.130.21
192.200.2.1              <-- 192.168.130.21 INIT-ACK
COOKIE_ECHO 192.200.2.1  --> 192.168.130.21 
192.200.2.1              <-- 192.168.130.21 COOKIE_ACK
HEARTBEAT 192.200.2.1    --> 192.168.130.21 
192.200.2.1              <-- 192.168.130.21 HEARTBEAT_ACK

The heartbeat attempt from the server fails to the second interface of the client since eth3 is being used.

192.201.2.1              <-- 192.168.130.21 HEARTBEAT

The client also attempts to heartbeat the secondary path but the server responds on the wrong interface:

HEARTBEAT 192.201.2.1    --> 192.168.150.21 
192.200.2.1              <-- 192.168.130.21 HEARTBEAT_ACK

The server never attempts to send the HEARTBEAT messages using the secondary path interface (192.168.150.21 eth3). Neither will it respond to the HEARTBEAT from the secondary path interface.

I have read that SCTP routing is not source based and that it is based on destination only. I understand this can be solved if I use destination static routes.

However my question is more why is SCTP not even attempting to send on the second interface when it know's it not getting responses from the primary path?
Is there a config item I'm missing that would allow SCTP to attempt the second interface to validate the secondary path after a certain number of attempts?

FYI, the problem I'm trying to solve (as you may have guessed) is to avoid having potentially hundreds of static routes to manage hundreds if SCTP client trying to connect to the server. Hence the use of the IP rules functionality.

Thanks,
Chas.

Responses

What RHEL version are you using? We've seen some situations where policy routing doesn't work on RHEL 6, though this is resolved on RHEL 7.

Apologies for the delayed response - I'm in Australia so there is some timezone delay.

We were predominately testing this on CentOS 7.3. I am working to validate that the same behaviour exists on RHEL 7.3. Initially I was assuming (may be incorrectly) that the SCTP & networking layers were the same between RHEL & CentOS versions such that the behaviour should be the same.

I will post back the results shortly.

In the meantime you mentioned that there was an issue with policy routing but was resolved. Was this in relation to SCTP and could you point me to what was done to resolve it please?

CentOS and RHEL should behave the same, they are built from the same sources.

Policy routing with from x.x.x.x rules do not seem to work with SCTP on RHEL 6, the default table is used instead.

We couldn't reproduce the same error on RHEL 7. There's been a lot of changes to IP routing between those releases so we were unable to quantify the change.

Thanks Jamie. I have just tested this with RHEL7.3 and I get the same behaviour as CentOS testing which is that SCTP packets from server only use the one interface.

When I add destination static routes, to the main table or the specific policy routing table, the issue is resolved where both source addresses are used.

So policy routing does seem to work but in my case only if I specify a specific destination route for the endpoint.

It seems that SCTP has to see a destination route before it will to attempt to use an interface in the SCTP bind list.

This is somewhat supported by the response here: - https://access.redhat.com/solutions/193863

Basically if it can't find a route then it defaults to the first (or primary path) interface in the bind list. Even though heartbeats to 192.201.2.1 from 192.168.130.21 are failing.

So is this a SCTP coding issue where the alternative interface is not attempted because of no specific destination route? Even if heartbeats are failing?

Note that crux of the issue is to avoid populating a large number of static routes. I want to only specify a default route in the policy route. The assumption here is that SCTP will actually attempt to use both source interfaces.

Is SCTP is not considering that a default route in the policy routing tables is a valid route option?

BTW, I used the following resource to be sure that I hadn't missed anything in the routing setup: - https://access.redhat.com/solutions/19596

This is specifically meant to show how to return packets to the interface that they came in on. Note I did not implement step 5.

As mentioned it looks like SCTP has decided there is no route and defaulting to using one interface.

SSH (and maybe TCP in general) does adhere to the resource mentioned above.

Does anyone know of a mechanism in SCTP multi-homing configuration/binding that tells SCTP how to use the interfaces? Essentially tell SCTP to do the following:

  • LOCAL_ADDR1 -> REMOTE_ADDR1

  • LOCAL_ADDR2 -> REMOTE_ADDR2

Therefore policy routing would only need to look at the source address and route based on the correct table?

Basically I find it strange that SCTP has decided to send to both destinations via one network which goes against the concept of a multi-homed (redundant) configuration.

Hmm, I don't think you should be policy-routing to try and encourage heartbeats out a certain interface.

I don't expect that a kernel-generated HEARTBEAT chunk would have a source address at route lookup time, not in the same way that a process calling bind() or sctp_bindx() would set a source address on data written into its socket. If I'm correct, then the heartbeat would skip the specific from rules during routing (as the SCTP header does not have a source address set yet) and use the main routing table which would explain why you see the server sending heartbeats out the wrong interface.

You really shouldn't need to setup this complex a network to achieve multi-homing. If I were you, I would not use policy routing at all for this setup. Instead, your main routing tables should look something like this:

client

192.168.130.0/24 via 192.200.2.254 dev eth2
192.168.150.0/24 via 192.201.2.254 dev eth3
server

192.200.2.0/24 via 192.168.130.254 dev eth2
192.201.2.0/24 via 192.168.150.254 dev eth3

To answer your last question, the SCTP Socket API Extensions do not have any option to set a next hop, that is done by IP routing.

Thanks Jamie. Yep aware that static routes will work and this is my fallback plan. I'm trying to explore methods for not having to create a large number of static routes.

The intention is allows 200+ SCTP multi-homed clients to connect to the server. This would require 400+ static routes to be managed which is not pretty.

I'm not asking for the next hop to be set, instead I'd actually want SCTP to set the source IP (along with the destination address) and then pass the packet for IP routing.

The RFC seems to indicate that only the source transport address is available in the recvmsg() call. I guess this doesn't bode well. I can't see much in the way of setting the source transport when sending the message. I'm not an expert at this level though.

Is setting the source address permitted through API calls?

I don't believe so, not for heartbeats anyway. The socket owner has only indirect control over heartbeat timing via setsockopt(SCTP_PEER_ADDR_PARAMS).

Let me have a play with this and discuss with our SCTP maintainers.

Do you mind if I open a support case for you so that I can dedicate some proper time to it and ensure a reply for you?

Sure that would be great! Thanks for your help Jamie!

From what I have seen since doing some subsequent testing, without specifying a static the issue can occur for INIT messages as well. Where the INIT_ACK is sent using the other interface in the BIND list as per the HEARTBEAT_ACK scenario.

That's expected: SCTP behaviour when primary path is down during INIT

I'll continue further via a support case, which I'll open shortly.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.