Chapter 8. Clustered Jakarta Enterprise Beans

Jakarta Enterprise Beans components can be clustered for high-availability scenarios. They use different protocols than HTTP components, so they are clustered in different ways. Enterprise Beans 2 and 3 stateful and stateless beans can be clustered.

For information on singletons, see HA Singleton Service in the JBoss EAP Development Guide

You can simplify the Jakarta Enterprise Beans client code when invoking the Jakarta Enterprise Beans server-side clustered components. The following procedures outline the multiple ways to simplify the Jakarta Enterprise Beans client code:

Clustering support is available in the HA profiles of JBoss EAP 7.4. Starting the standalone server with HA capabilities enabled involves starting it with the standalone-ha.xml or standalone-full-ha.xml file:

$ EAP_HOME/bin/standalone.sh --server-config=standalone-ha.xml

This will start a single instance of the server with HA capabilities.

To be able to see the benefits of clustering, you will need more than one instance of the server. So let us start another server with HA capabilities. That another instance of the server can either be on the same machine or on some other machine. If it is on the same machine, you will need to take care of two things:

You can do that by passing the following two system properties to the startup command:

$ EAP_HOME/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.socket.binding.port-offset=PORT_OFFSET -Djboss.node.name=UNIQUE_NODE_NAME

Follow whichever approach you feel comfortable with for deploying the Jakarta Enterprise Beans deployment to this instance too.

Now that you have deployed an application with clustered Jakarta Enterprise Beans on both the instances, the Jakarta Enterprise Beans are now capable of making use of the clustering features.

@Stateful(passivationCapable=false)

/subsystem=ejb3:write-attribute(name=default-sfsb-cache,value=simple)

Clustered Jakarta Enterprise Beans have failover capability. The state of the @Stateful Jakarta Enterprise Beans is replicated across the cluster nodes so that if one of the nodes in the cluster goes down, some other node will be able to take over the invocations.

Under some circumstances in a clustered environment, such as when a server in the cluster crashes, the Jakarta Enterprise Beans client might receive an exception instead of a response. The Jakarta Enterprise Beans client library will automatically retry the invocation when it is safe to do so, depending on the type of the failure that occurs. However, if a request fails and it cannot be determined conclusively to be safe to retry, then you can handle the exception as appropriate for your environment. You can, however, use custom interceptors to add additional retry behavior.

A standalone remote client can use either the Java Naming and Directory Interface approach or native JBoss Jakarta Enterprise Beans client APIs to communicate with the servers. The important thing to note is that when you are invoking clustered Jakarta Enterprise Beans deployments, you do not have to list all the servers within the cluster. This would not have been feasible due the dynamic nature of cluster node additions within a cluster.

The remote client has to list only one of the servers with the clustering capability. This server will act as the starting point for cluster topology communication between the client and the clustered nodes.

Note that you have to configure the ejb cluster in the jboss-ejb-client.properties configuration file:

remote.clusters=ejb
remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.cluster.ejb.connect.options.org.xnio.Options.SSL_ENABLED=false

When a client connects to a server, the JBoss Jakarta Enterprise Beans client implementation communicates internally with the server for the cluster topology information, if the server has clustering capability. For example, assuming that server X is listed as the initial server to connect to, when the client connects to server X, the server will send back an asynchronous cluster topology message to the client. This topology message consists of the cluster name and the information of the nodes that belong to the cluster. The node information includes the node address and port number to connect to, when required. So in this example, server X will send back the cluster topology consisting of the other server Y that belongs to the cluster.

In case of stateful clustered Jakarta Enterprise Beans, the invocation flow happens in two steps.

The lookup for the stateful bean, internally, triggers a synchronous session creation request from the client to the server. In this case, the session creation request goes to server X because it was configured in the jboss-ejb-client.properties file. Since server X is clustered, it will return a session id and send back an affinity of that session. In case of clustered servers, the affinity is equal to the name of the cluster to which the stateful bean belongs on the server side. For non-clustered beans, the affinity is the node name on which the session was created. This affinity will help the Jakarta Enterprise Beans client to route the invocations on the proxy, as appropriate, to either a node within a cluster for clustered beans, or to a specific node for non-clustered beans. While this session creation request is going on, server X will also send back an asynchronous message that contains the cluster topology. The JBoss Jakarta Enterprise Beans client implementation will record this topology information and use it later for connection creation to nodes within the cluster and routing invocations to those nodes, when required.

To understand how failover works, consider the same example of server X being the starting point and a client application looking up a stateful bean and invoking it. During these invocations, the client side collects the cluster topology information from the server. Assuming that for some reason server X goes down and the client application subsequently invokes on the proxy. The JBoss Jakarta Enterprise Beans client implementation at this stage must be aware of the affinity, and in this case it is the cluster affinity. From the cluster topology information that the client has, it knows that the cluster has two nodes, server X and server Y. When the invocation arrives, the client notices that server X is down, so it uses a selector to fetch a suitable node from the cluster nodes. When the selector returns a node from the cluster nodes, the JBoss Jakarta Enterprise Beans client implementation creates a connection to that node, if the connection was not already created earlier, and creates a Jakarta Enterprise Beans receiver out of it. Since in this example, the only other node in the cluster is server Y, the selector will return server Y as the node and the JBoss Jakarta Enterprise Beans client implementation will use it to create an Jakarta Enterprise Beans receiver out of it and use this receiver to pass on the invocation on the proxy. Effectively, the invocation has now failed over to a different node within the cluster.

A transaction object, which is looked up from the same context as the Jakarta Enterprise Beans proxy, targets the same host. Having an active transaction pins the invocation context to the same node, if the context is multi-host or clustered.

This behavior depends on whether you have outflowed your transaction or you are using a remote user transaction.

For an outflowed transaction, when an application is looked up on a specific node, all the invocations to that application under the same transaction attempt to target this node. The nodes that have already received the outflowed transaction will be preferred over nodes that have not received it yet.

For a remote user transaction, the first successful invocation will lock the transaction to the given node, and subsequent invocations under this transaction must go to the same node, otherwise an exception is thrown.

This section explains how a client application deployed on a JBoss EAP instance invokes a clustered stateful bean that is deployed on another JBoss EAP instance.

In the following example, there are three servers involved. Servers X and Y both belong to a cluster and have clustered Jakarta Enterprise Beans deployed on them. There is another server instance server C, which may or may not have clustering capability. Server C acts as a client on which there is a deployment that wants to invoke the clustered beans deployed on servers X and Y and achieve failover.

The configurations are done in the jboss-ejb-client.xml file, which points to a remote outbound connection to the other server. The configuration in the jboss-ejb-client.xml file is in the deployment of server C because server C is the client. The client configuration need not point to all the clustered nodes, but just to one of them. This will act as a starting point for the communication.

In this case, a remote outbound connection is created from server C to server X and then server X is used as the starting point for the communication. Similar to the case of remote standalone clients, when the application on server C looks up a stateful bean, a session creation request is sent to server X that returns a session id and the cluster affinity for it. Server X also sends back an asynchronous message to server C containing the cluster topology. This topology information includes the node information of server Y, because server Y belongs to the cluster along with server X. Subsequent invocations on the proxy will be routed appropriately to the nodes in the cluster. If server X goes down, as explained earlier, a different node from the cluster will be selected and the invocation will be forwarded to that node.

Both remote standalone clients as well as remote clients on another JBoss EAP instance act similarly in terms of failover.

To connect an Jakarta Enterprise Beans client to a clustered Jakarta Enterprise Beans application, you need to expand the existing configuration in standalone Jakarta Enterprise Beans client or in-server Jakarta Enterprise Beans client to include cluster connection configuration. The jboss-ejb-client.properties for standalone Jakarta Enterprise Beans client, or even jboss-ejb-client.xml file for a server-side application must be expanded to include a cluster configuration.

This example shows the additional cluster configuration required for a standalone Jakarta Enterprise Beans client.

remote.clusters=ejb
remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.cluster.ejb.connect.options.org.xnio.Options.SSL_ENABLED=false
remote.cluster.ejb.username=test
remote.cluster.ejb.password=password

If an application uses the remote-outbound-connection, you need to configure the jboss-ejb-client.xml file and add cluster configuration as shown in the following example:

<jboss-ejb-client xmlns:xsi="urn:jboss:ejb-client:1.2" xsi:noNamespaceSchemaLocation="jboss-ejb-client_1_2.xsd">
  <client-context>
    <ejb-receivers>
      <!-- this is the connection to access the app-one -->
      <remoting-ejb-receiver outbound-connection-ref="remote-ejb-connection-1" />
      <!-- this is the connection to access the app-two -->
      <remoting-ejb-receiver outbound-connection-ref="remote-ejb-connection-2" />
    </ejb-receivers>

    <!-- If an outbound connection connects to a cluster,
             a list of members is provided after successful connection.
         To connect to this node this cluster element must be defined. -->

    <clusters>
      <!-- cluster of remote-ejb-connection-1 -->
      <cluster name="ejb" security-realm="ejb-security-realm-1" username="quickuser1">
        <connection-creation-options>
        <property name="org.xnio.Options.SSL_ENABLED" value="false" />
        <property name="org.xnio.Options.SASL_POLICY_NOANONYMOUS" value="false" />
        </connection-creation-options>
      </cluster>
    </clusters>
  </client-context>
</jboss-ejb-client>

For more information about remote-outbound-connection, see About the Remoting Subsystem in the JBoss EAP Configuration Guide.

It is possible to implement an alternate or customized load balancing policy in order to balance an application’s Jakarta Enterprise Beans calls across servers.

You can implement AllClusterNodeSelector for Jakarta Enterprise Beans calls. The node selection behavior of AllClusterNodeSelector is similar to a default selector except that AllClusterNodeSelector uses all available cluster nodes even in case of a large cluster (number of nodes > 20). If an unconnected cluster node is returned, it is opened automatically. The following example shows AllClusterNodeSelector implementation:

package org.jboss.as.quickstarts.ejb.clients.selector;

import java.util.Arrays;
import java.util.Random;
import java.util.logging.Level;
import java.util.logging.Logger;

import org.jboss.ejb.client.ClusterNodeSelector;
public class AllClusterNodeSelector implements ClusterNodeSelector {
  private static final Logger LOGGER = Logger.getLogger(AllClusterNodeSelector.class.getName());

  @Override
  public String selectNode(final String clusterName, final String[] connectedNodes, final String[] availableNodes) {
    if(LOGGER.isLoggable(Level.FINER)) {
      LOGGER.finer("INSTANCE "+this+ " : cluster:"+clusterName+" connected:"+Arrays.deepToString(connectedNodes)+" available:"+Arrays.deepToString(availableNodes));
    }

    if (availableNodes.length == 1) {
        return availableNodes[0];
    }
    final Random random = new Random();
    final int randomSelection = random.nextInt(availableNodes.length);
    return availableNodes[randomSelection];
  }

}

You can also implement the SimpleLoadFactorNodeSelector for Jakarta Enterprise Beans calls. Load balancing in SimpleLoadFactorNodeSelector happens based on a load factor. The load factor (2/3/4) is calculated based on the names of nodes (A/B/C) irrespective of the load on each node. The following example shows SimpleLoadFactorNodeSelector implementation:

package org.jboss.as.quickstarts.ejb.clients.selector;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.logging.Level;
import java.util.logging.Logger;

import org.jboss.ejb.client.DeploymentNodeSelector;
public class SimpleLoadFactorNodeSelector implements DeploymentNodeSelector {
  private static final Logger LOGGER = Logger.getLogger(SimpleLoadFactorNodeSelector.class.getName());
  private final Map<String, List<String>[]> nodes = new HashMap<String, List<String>[]>();
  private final Map<String, Integer> cursor = new HashMap<String, Integer>();

  private ArrayList<String> calculateNodes(Collection<String> eligibleNodes) {
    ArrayList<String> nodeList = new ArrayList<String>();

    for (String string : eligibleNodes) {
      if(string.contains("A") || string.contains("2")) {
        nodeList.add(string);
        nodeList.add(string);
      } else if(string.contains("B") || string.contains("3")) {
        nodeList.add(string);
        nodeList.add(string);
        nodeList.add(string);
      } else if(string.contains("C") || string.contains("4")) {
        nodeList.add(string);
        nodeList.add(string);
        nodeList.add(string);
        nodeList.add(string);
      }
    }
    return nodeList;
  }

  @SuppressWarnings("unchecked")
  private void checkNodeNames(String[] eligibleNodes, String key) {
    if(!nodes.containsKey(key) || nodes.get(key)[0].size() != eligibleNodes.length || !nodes.get(key)[0].containsAll(Arrays.asList(eligibleNodes))) {
      // must be synchronized as the client might call it concurrent
      synchronized (nodes) {
        if(!nodes.containsKey(key) || nodes.get(key)[0].size() != eligibleNodes.length || !nodes.get(key)[0].containsAll(Arrays.asList(eligibleNodes))) {
          ArrayList<String> nodeList = new ArrayList<String>();
          nodeList.addAll(Arrays.asList(eligibleNodes));

          nodes.put(key, new List[] { nodeList, calculateNodes(nodeList) });
        }
      }
    }
  }
   private synchronized String nextNode(String key) {
    Integer c = cursor.get(key);
    List<String> nodeList = nodes.get(key)[1];

    if(c == null || c >= nodeList.size()) {
      c = Integer.valueOf(0);
    }

    String node = nodeList.get(c);
    cursor.put(key, Integer.valueOf(c + 1));

    return node;
  }

  @Override
  public String selectNode(String[] eligibleNodes, String appName, String moduleName, String distinctName) {
    if (LOGGER.isLoggable(Level.FINER)) {
      LOGGER.finer("INSTANCE " + this + " : nodes:" + Arrays.deepToString(eligibleNodes) + " appName:" + appName + " moduleName:" + moduleName
          + " distinctName:" + distinctName);
    }

    // if there is only one there is no sense to choice
    if (eligibleNodes.length == 1) {
      return eligibleNodes[0];
    }
    final String key = appName + "|" + moduleName + "|" + distinctName;

    checkNodeNames(eligibleNodes, key);
    return nextNode(key);
  }
}

Configuring the jboss-ejb-client.properties File

You need to add the property remote.cluster.ejb.clusternode.selector with the name of your implementation class (AllClusterNodeSelector or SimpleLoadFactorNodeSelector). The selector will see all configured servers that are available at the invocation time. The following example uses AllClusterNodeSelector as the cluster node selector:

remote.clusters=ejb
remote.cluster.ejb.clusternode.selector=org.jboss.as.quickstarts.ejb.clients.selector.AllClusterNodeSelector
remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.cluster.ejb.connect.options.org.xnio.Options.SSL_ENABLED=false
remote.cluster.ejb.username=test
remote.cluster.ejb.password=password

remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=one,two
remote.connection.one.host=localhost
remote.connection.one.port = 8080
remote.connection.one.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.one.username=user
remote.connection.one.password=user123
remote.connection.two.host=localhost
remote.connection.two.port = 8180
remote.connection.two.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false

Using Jakarta Enterprise Beans Client API

You need to add the property remote.cluster.ejb.clusternode.selector to the list for the PropertiesBasedEJBClientConfiguration constructor. The following example uses AllClusterNodeSelector as the cluster node selector:

Properties p = new Properties();
p.put("remote.clusters", "ejb");
p.put("remote.cluster.ejb.clusternode.selector", "org.jboss.as.quickstarts.ejb.clients.selector.AllClusterNodeSelector");
p.put("remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS", "false");
p.put("remote.cluster.ejb.connect.options.org.xnio.Options.SSL_ENABLED", "false");
p.put("remote.cluster.ejb.username", "test");
p.put("remote.cluster.ejb.password", "password");

p.put("remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED", "false");
p.put("remote.connections", "one,two");
p.put("remote.connection.one.port", "8080");
p.put("remote.connection.one.host", "localhost");
p.put("remote.connection.two.port", "8180");
p.put("remote.connection.two.host", "localhost");

EJBClientConfiguration cc = new PropertiesBasedEJBClientConfiguration(p);
ContextSelector<EJBClientContext> selector = new ConfigBasedEJBClientContextSelector(cc);
EJBClientContext.setSelector(selector);

p = new Properties();
p.put(Context.URL_PKG_PREFIXES, "org.jboss.ejb.client.naming");
InitialContext context = new InitialContext(p);

Configuring the jboss-ejb-client.xml File

To use the load balancing policy for server to server communication, package the class together with the application and configure it within the jboss-ejb-client.xml settings located in META-INF folder. The following example uses AllClusterNodeSelector as the cluster node selector:

<jboss-ejb-client xmlns:xsi="urn:jboss:ejb-client:1.2" xsi:noNamespaceSchemaLocation="jboss-ejb-client_1_2.xsd">
  <client-context deployment-node-selector="org.jboss.ejb.client.DeploymentNodeSelector">
    <ejb-receivers>
      <!-- This is the connection to access the application. -->
      <remoting-ejb-receiver outbound-connection-ref="remote-ejb-connection-1" />
    </ejb-receivers>
    <!-- Specify the cluster configurations applicable for this client context -->
    <clusters>
      <!-- Configure the cluster of remote-ejb-connection-1. -->
      <cluster name="ejb" security-realm="ejb-security-realm-1" username="test" cluster-node-selector="org.jboss.as.quickstarts.ejb.clients.selector.AllClusterNodeSelector">
        <connection-creation-options>
          <property name="org.xnio.Options.SSL_ENABLED" value="false" />
          <property name="org.xnio.Options.SASL_POLICY_NOANONYMOUS" value="false" />
        </connection-creation-options>
      </cluster>
    </clusters>
  </client-context>
</jboss-ejb-client>

To use the above configuration with security, you will need to add ejb-security-realm-1 to client-server configuration. The following example shows the CLI commands for adding security realm (ejb-security-realm-1) the value is the base64 encoded password for the user "test":

/core-service=management/security-realm=ejb-security-realm-1:add()
/core-service=management/security-realm=ejb-security-realm-1/server-identity=secret:add(value=cXVpY2sxMjMr)

If the load balancing policy should be used for server to server communication, the class can be packaged together with the application or as a module. This class is configured in the jboss-ejb-client settings file located in the META-INF directory of the top-level EAR archive. The following example uses RoundRobinNodeSelector as the deployment node selector.

<jboss-ejb-client xmlns="urn:jboss:ejb-client:1.2">
    <client-context deployment-node-selector="org.jboss.example.RoundRobinNodeSelector">
        <ejb-receivers>
            <remoting-ejb-receiver outbound-connection-ref="..."/>
        </ejb-receivers>
        ...
    </client-context>
</jboss-ejb-client>

If the client code invokes a clustered Jakarta Enterprise Beans, then the cluster affinity is set automatically. If you manage transactions on the client side, you can choose to target a specific node in the cluster or you can allow the client to lazily select the cluster node to handle transactions. This section describes both options.

Jakarta Enterprise Beans Transactions Target a Specific Node

You can target a specific node in the cluster to handle a transaction using the following procedure.

Jakarta Enterprise Beans Transactions Lazily Select a Node

You can allow the client to lazily select the cluster node to handle transactions during the first invocation pertaining to a transaction. This allows for load balancing of transactions across the cluster. To use this option, follow the procedure below.

JBoss EAP supports clustered database-backed timers for persisting Jakarta Enterprise Beans timers in a clustered environment. Because the clustering is provided through a database, if the number of timers going off within a short interval of time increases, the performance decreases. You can optimize performance by using the refresh-interval and allow-execution attributes of the ejb3/service=timer-service/database-data-store component.

You can also use database timers in non-clustered mode as follows:

Jakarta Enterprise Beans-clustered database timers work as following:

Due to the use of a transaction and READ_COMMITTED or SERIALIZABLE isolation mode, only one node succeeds in updating the row, and this is the node that the timer executes on.