How do I configure or setup TCP-based clustering in EAP 7 / 8 ?

Solution Verified - Updated -

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 7.x
    • 8.x
  • Red Hat Single Sign-On (RH-SSO)
    • 7.x
  • TCP-based protocol
  • Clustering / High Availability with JGroups

Issue

  • How do I configure or setup TCP-based clustering?
  • Please provide instructions for configuration without multicast.
  • Control and explicitly indicate which server is a member of the cluster.

Resolution

  1. Change the default stack from udp to tcp in the jgroups subsystem:

    <channel name="ee" stack="tcp" cluster="ejb"/>

  2. Set node name

    • In standalone (HA-based) configuration (top of the page):

      <server xmlns="urn:jboss:domain:8.0" name="node_1">

      or specify a unique name to the system property jboss.node.name when starting each instance.

    • In domain mode, the servers are listed in the host-*.xml in the server's tag. The default configuration specifies names.

      <servers>
    <server name="server-one" group="main-server-group"/>
    <server name="server-two" group="other-server-group">
        <socket-bindings port-offset="150"/>
    </server>
</servers>

  3. Decide on a discovery protocol to use

    • TCPGOSSIP - see documentation

      This method uses an external gossip router service to discover the members of a cluster. This requires configuration and management of the additional process (or multiple for high availability) but allows the individual EAP instances to not need to list the cluster members. This can be advantageous if the cluster members change frequently. All the info can be found in the documentation.

    • TCPPING - see documentation

      This method requires each node to list all the other potential cluster members. This is the simplest method when cluster member addresses are known and do not change frequently. The documentation link in the title has the CLI commands, and these are preferable so that the configuration can be automated, and minor version upgrades will be easier.

TCPPING example configurations

  • Standalone mode / Basics

    1. Change the default stack from udp to tcp in the jgroups subsystem:

    2. Setup the TCP stack. In most cases, this involves switching out the default MPING protocol for the TCPPING protocol. For more information in regards to the available JGroups protocols, refer to JGroups documentation.
      Note: the port number ([7600] in the below example) in the initial_hosts setting must be matched to the port number defined as the jgroups-tcp socket-binding definition. If you use the port-offset feature for the socket-binding, you need to specify the matched value after offset to the initial_hosts.

          <stack name="tcp">
        <transport type="TCP" socket-binding="jgroups-tcp"/>
        <protocol type="TCPPING">
            <property name="initial_hosts">192.168.1.5[7600],192.168.1.9[7600]</property>
            <property name="port_range">0</property>
        </protocol>
        <protocol type="MERGE3"/>
        <protocol type="FD_SOCK" socket-binding="jgroups-tcp-fd"/>
        <protocol type="FD_ALL"/>
        <protocol type="VERIFY_SUSPECT"/>
        <protocol type="pbcast.NAKACK2"/>
        <protocol type="UNICAST3"/>
        <protocol type="pbcast.STABLE"/>
        <protocol type="pbcast.GMS"/>
        <protocol type="MFC"/>
        <protocol type="FRAG3"/>
    </stack>

    3. Set the IP address of the private interface (which is used by the JGroups component by default). The IP address should correlate to one of the IP addresses specified in the initial_hosts configuration:

          <interface name="private">
        <inet-address value="${jboss.bind.address.private:192.168.1.5}"/>
    </interface>

    4. Repeat the above steps for the other nodes within the cluster. Once complete, start up each node and deploy a clustered application. Nodes should form a cluster. Logs would look something like such:

          INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-2,ee,node_1) ISPN000094: Received new cluster view for channel server: [node_1|1] (2) [node_1, node_2]
    INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-2,ee,node_1) ISPN000094: Received new cluster view for channel web: [node_1|1] (2) [node_1, node_2]
    INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-2,ee,node_1) ISPN000094: Received new cluster view for channel hibernate: [node_1|1] (2) [node_1, node_2]
    INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (thread-2,ee,node_1) ISPN000094: Received new cluster view for channel ejb: [node_1|1] (2) [node_1, node_2]

  • Domain mode

    1. Apply the aforementioned changes to the JGroups subsystem of the profile being used.

    2. If the same profile is used for multiple clusters, set the initial_hosts property as a system property:

          <protocol type="TCPPING">
        <property name="initial_hosts">${jboss.cluster.tcp.initial_hosts}</property>

Set the system property at the `server-group` level:

    <server-groups>
        <server-group name="a-server-group" profile="ha">
            <socket-binding-group ref="ha-sockets"/>
            <system-properties>
                <property name="jboss.cluster.tcp.initial_hosts" value="192.168.1.5[7600],192.168.1.9[7600]" />
            </system-properties>

    3. Users will need to define a private interface within the host XML configuration of the host controller. The IP address of the private interface should correlate to one of the IP addresses listed in initial_hosts.

          <interfaces>
    ....
        <interface name="private">
            <inet-address value="${jboss.bind.address.private:192.168.1.5}"/>
        </interface>
    </interfaces>

Notes:

  • initial_hosts correlates to a listing of all members within the cluster.
  • 7600 correlates to the default jgroups-tcp port. Depending on one's configuration, port offsets, etc., this port may be different in some environments.
  • One can specify the system property jboss.server.name instead of setting name within the standalone configuration.
  • The property num_initial_members (from previous JBoss EAP versions) has been deprecated.

  • The timeout property for the TCPPING protocol has been deprecated. If desired, join_timeout can be set in the pbcast.GMS protocol instead:

            <protocol type="pbcast.GMS">
            <property name="join_timeout">
                2000
            </property>
        </protocol>

Root Cause

TCP mode can be used when multicast is not preferable or allowed by security policies.

Protocols Valid config?
FD_SOCK+FD valid
FD_SOCK+FD_ALL valid
FD_ALL+FD redundant given they do the same

See solution Setting FD_ALL/FD timeout in JDG 7 for explanations on the protocols.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments