26.2. Configure JGroups (Library Mode)

Red Hat JBoss Data Grid must have an appropriate JGroups configuration in order to operate in clustered mode.

Example 26.5. JGroups Programmatic Configuration

GlobalConfiguration gc = new GlobalConfigurationBuilder()
  .transport()
  .defaultTransport()
  .addProperty("configurationFile","jgroups.xml")
  .build();

Example 26.6. JGroups XML Configuration

<infinispan>
  <global>
    <transport>
      <properties>
        <property name="configurationFile" value="jgroups.xml" />
      </properties>
    </transport>
  </global>
  <!-- Additional configuration elements here -->
</infinispan>
In either programmatic or XML configuration methods, JBoss Data Grid searches for jgroups.xml in the classpath before searching for an absolute path name if it is not found in the classpath.

26.2.1. JGroups Transport Protocols

A transport protocol is the protocol at the bottom of a protocol stack. Transport Protocols are responsible for sending and receiving messages from the network.
Red Hat JBoss Data Grid ships with both UDP and TCP transport protocols.

26.2.1.1. The UDP Transport Protocol

UDP is a transport protocol that uses:
  • IP multicasting to send messages to all members of a cluster.
  • UDP datagrams for unicast messages, which are sent to a single member.
When the UDP transport is started, it opens a unicast socket and a multicast socket. The unicast socket is used to send and receive unicast messages, the multicast socket sends and receives multicast sockets. The physical address of the channel will be the same as the address and port number of the unicast socket.

26.2.1.2. The TCP Transport Protocol

TCP/IP is a replacement transport for UDP in situations where IP multicast cannot be used, such as operations over a WAN where routers may discard IP multicast packets.
TCP is a transport protocol used to send unicast and multicast messages.
  • When sending multicast messages, TCP sends multiple unicast messages.
  • When using TCP, each message to all cluster members is sent as multiple unicast messages, or one to each member.
As IP multicasting cannot be used to discover initial members, another mechanism must be used to find initial membership.
Red Hat JBoss Data Grid's Hot Rod is a custom TCP client/server protocol.

26.2.1.3. Using the TCPPing Protocol

Some networks only allow TCP to be used. The pre-configured default-configs/default-jgroups-tcp.xml includes the MPING protocol, which uses UDP multicast for discovery. When UDP multicast is not available, the MPING protocol, has to be replaced by a different mechanism. The recommended alternative is the TCPPING protocol. The TCPPING configuration contains a static list of IP addresses which are contacted for node discovery.

Example 26.7. Configure the JGroups Subsystem to Use TCPPING

<TCP bind_port="7800" />
<TCPPING initial_hosts="${jgroups.tcpping.initial_hosts:HostA[7800],HostB[7801]}"
         port_range="1" />

26.2.2. Pre-Configured JGroups Files

Red Hat JBoss Data Grid ships with a number of pre-configured JGroups files packaged in infinispan-embedded.jar, and are available on the classpath by default. In order to use one of these files, specify one of these file names instead of using jgroups.xml.
The JGroups configuration files shipped with JBoss Data Grid are intended to be used as a starting point for a working project. JGroups will usually require fine-tuning for optimal network performance.
The available configurations are:
  • default-configs/default-jgroups-udp.xml
  • default-configs/default-jgroups-tcp.xml
  • default-configs/default-jgroups-ec2.xml

26.2.2.1. default-jgroups-udp.xml

The default-configs/default-jgroups-udp.xml file is a pre-configured JGroups configuration in Red Hat JBoss Data Grid. The default-jgroups-udp.xml configuration
  • uses UDP as a transport and UDP multicast for discovery.
  • is suitable for large clusters (over 8 nodes).
  • is suitable if using Invalidation or Replication modes.
The behavior of some of these settings can be altered by adding certain system properties to the JVM at startup. The settings that can be configured are shown in the following table.

Table 26.1. default-jgroups-udp.xml System Properties

System Property Description Default Required?
jgroups.udp.mcast_addr IP address to use for multicast (both for communications and discovery). Must be a valid Class D IP address, suitable for IP multicast. 228.6.7.8 No
jgroups.udp.mcast_port Port to use for multicast socket 46655 No
jgroups.udp.ip_ttl Specifies the time-to-live (TTL) for IP multicast packets. The value here refers to the number of network hops a packet is allowed to make before it is dropped 2 No

26.2.2.2. default-jgroups-tcp.xml

The default-configs/default-jgroups-tcp.xml file is a pre-configured JGroups configuration in Red Hat JBoss Data Grid. The default-jgroups-tcp.xml configuration
  • uses TCP as a transport and UDP multicast for discovery.
  • is generally only used where multicast UDP is not an option.
  • TCP does not perform as well as UDP for clusters of eight or more nodes. Clusters of four nodes or fewer result in roughly the same level of performance for both UDP and TCP.
As with other pre-configured JGroups files, the behavior of some of these settings can be altered by adding certain system properties to the JVM at startup. The settings that can be configured are shown in the following table.

Table 26.2. default-jgroups-tcp.xml System Properties

System Property Description Default Required?
jgroups.tcp.address IP address to use for the TCP transport. 127.0.0.1 No
jgroups.tcp.port Port to use for TCP socket 7800 No
jgroups.udp.mcast_addr IP address to use for multicast (for discovery). Must be a valid Class D IP address, suitable for IP multicast. 228.6.7.8 No
jgroups.udp.mcast_port Port to use for multicast socket 46655 No
jgroups.udp.ip_ttl Specifies the time-to-live (TTL) for IP multicast packets. The value here refers to the number of network hops a packet is allowed to make before it is dropped 2 No

26.2.2.3. default-jgroups-ec2.xml

The default-configs/default-jgroups-ec2.xml file is a pre-configured JGroups configuration in Red Hat JBoss Data Grid. The default-jgroups-ec2.xml configuration
  • uses TCP as a transport and S3_PING for discovery.
  • is suitable on Amazon EC2 nodes where UDP multicast isn't available.
As with other pre-configured JGroups files, the behavior of some of these settings can be altered by adding certain system properties to the JVM at startup. The settings that can be configured are shown in the following table.

Table 26.3. default-jgroups-ec2.xml System Properties

System Property Description Default Required?
jgroups.tcp.address IP address to use for the TCP transport. 127.0.0.1 No
jgroups.tcp.port Port to use for TCP socket 7800 No
jgroups.s3.access_key The Amazon S3 access key used to access an S3 bucket Yes
jgroups.s3.secret_access_key The Amazon S3 secret key used to access an S3 bucket Yes
jgroups.s3.bucket Name of the Amazon S3 bucket to use. Must be unique and must already exist Yes
jgroups.s3.pre_signed_delete_url The pre-signed URL to be used for the DELETE operation. Yes
jgroups.s3.pre_signed_put_url The pre-signed URL to be used for the PUT operation. Yes
jgroups.s3.prefix If set, S3_PING searches for a bucket with a name that starts with the prefix value. No