Connections on remoting port (4447) never closed on server when client crashes or network is interrupted

Latest response

We run a rich client using ejb-client/remoting to access EJB and JMS on JBoss EAP 6.3.3

When a client starts, 2 tcp/ip connections are opened. netstat (on windows) listens the connections:

  TCP    172.21.15.24:4447      172.21.15.163:49421    HERGESTELLT
  TCP    172.21.15.24:4447      172.21.15.163:49886    HERGESTELLT
  TCP    172.21.15.24:4447      172.21.15.163:49887    HERGESTELLT
  TCP    172.21.15.24:4447      172.21.15.163:49939    HERGESTELLT
  TCP    172.21.15.24:4447      172.21.15.163:49941    HERGESTELLT

When the client crashes (or the network is interrupted e.g. unplug RJ45 ), the ports on the server are not closed. Even if you run netstat after 2 days, the connections are
still 'established'. If the user now restarts the client, new connections are opened. The old ones are still in the list.

If the client stops in normal case (no crash, not network interruption), you see the following lines in the server log:

14:24:21,001 TRACE [org.jboss.remoting.remote] (Remoting "zugnb-als8-01" read-1) Closed inbound channel on Connection handler for Remoting connection 27b9186e to /172.21.15.163:56099
14:24:21,001 TRACE [org.jboss.remoting.remote.connection] (Remoting "zugnb-als8-01" read-1) Shut down writes on channel

You don't see anything like this if the client terminates abnormaly. That means the channel is still open. This corresponds to what I see in netstat.

HornetQ detects such 'dead' connections and closes them, but remoting not.

If this happens too often, JBoss server becomes slow, instabel and at the end it does not accept any new connections and clients can not login anymore. The only way to recover is restarting JBoss server.

We also tried to set KEEP_ALIVE property in the remoting subsystem, but that doesn't help.

            <remote connector-ref="remoting-connector" thread-pool-name="remoting-connector-thread-pool">
                <channel-creation-options>
                    <option name="KEEP_ALIVE" value="true" type="xnio"/>
                    <option name="READ_TIMEOUT" value="60000" type="xnio"/>
                    <option name="WRITE_TIMEOUT" value="60000" type="xnio"/>
                    <option name="HEARTBEAT_INTERVAL" value="60000" type="remoting"/>
                </channel-creation-options>
            </remote>

Any known solution or idea? Thanks in advance.

Responses

I'm going to take a guess and say you need to set up the timeouts on the server to force close the connections that are stagnent. See [1] "How to set web connector timeout (connectionTimeout) in EAP 6"

There is no point in setting a keep-alive on the remoting subsystem as that will be outgoing connections. See [2]. Have a play around with these connection settings and let us know if this helps.

[1] https://access.redhat.com/solutions/456733
[2] https://access.redhat.com/solutions/35791

Hi Mustafa

Thanks for your answer.
But unfortunatley these links are all about http connections.
We have an EJB Client application that uses remoting for communication to the server. Guess the implementation of remoting is based on XNIO

I can't help but think that the underlying TCP connections are not in JBoss's control and should be recognized as closed by the underlying OS layer which should close them. However, please feel free to raise a support case with us so our engineers who specialize this this area can advice you some more Jürg.

Hi Jurg, I have similiar problem like You, many ESTABLISHED connections between two servers in cluster. Did you find solution of Your problem? I see in netstat thousands connections and i have to restart servers every day. tcp 0 0 3.193.134.47:8447 3.193.134.47:34260 ESTABLISHED tcp 0 0 3.193.134.47:8447 3.193.134.48:55737 ESTABLISHED tcp 0 0 3.193.134.47:8447 3.193.134.47:54630 ESTABLISHED tcp 0 0 3.193.134.47:8447 3.193.134.48:33981 ESTABLISHED tcp 0 0 3.193.134.47:8447 3.193.134.48:60963 ESTABLISHED tcp 0 0 3.193.134.47:8447 3.193.134.47:52812 ESTABLISHED tcp 0 0 3.193.134.47:8447 3.193.134.47:36629 ESTABLISHED tcp 0 0 3.193.134.47:8447 3.193.134.48:33236 ESTABLISHED

Hello, we are having exactly the same problem. We have different instances of Jboss 6.4 EAP (31 in total) that are sharing information between them using remote EJBs. There are no problems between the majority of them but I see that one instance is not able to take the remote EJB from the other, although doing netstat I see a lot of opened connections in the machine from which the EJB needs to be retrieved.

We have similar problem using jboss6.4 EAP. any solution provided for this issue..?? My remote client is in tomcat.