Chapter 8. Clustered Enterprise JavaBeans (EJB)

8.1. About Clustered EJBs

EJB components can be clustered for high-availability scenarios. They use different protocols than HTTP components, so they are clustered in different ways. EJB 2 and 3 stateful and stateless beans can be clustered.

For information on singletons, see HA Singleton Service in the JBoss EAP Development Guide

8.2. EJB Client Code Simplification

You can simplify the EJB client code when invoking the EJB server-side clustered components. The following procedures outline the multiple ways to simplify the EJB client code:

Note

The use of the jboss-ejb-client.properties file is deprecated in favor of the wildfly-config.xml file.

8.3. Deploying Clustered EJB

Clustering support is available in the HA profiles of JBoss EAP 7.3. Starting the standalone server with HA capabilities enabled involves starting it with the standalone-ha.xml or standalone-full-ha.xml file:

$ EAP_HOME/bin/standalone.sh --server-config=standalone-ha.xml

This will start a single instance of the server with HA capabilities.

To be able to see the benefits of clustering, you will need more than one instance of the server. So let us start another server with HA capabilities. That another instance of the server can either be on the same machine or on some other machine. If it is on the same machine, you will need to take care of two things:

  • Pass the port offset for the second instance
  • Make sure that each of the server instances have a unique jboss.node.name system property.

You can do that by passing the following two system properties to the startup command:

$ EAP_HOME/bin/standalone.sh --server-config=standalone-ha.xml -Djboss.socket.binding.port-offset=PORT_OFFSET -Djboss.node.name=UNIQUE_NODE_NAME

Follow whichever approach you feel comfortable with for deploying the EJB deployment to this instance too.

Warning

Deploying the application on just one node of a standalone instance of a clustered server does not mean that it will be automatically deployed to the other clustered instance. You will have to do deploy it explicitly on the other standalone clustered instance too. Or you can start the servers in domain mode so that the deployment can be deployed to all the servers within a server group.

Now that you have deployed an application with clustered EJB on both the instances, the EJB are now capable of making use of the clustering features.

Note

Starting with JBoss EAP 7, if JBoss EAP is started using an HA profile, the state of your stateful session bean will be replicated. You no longer need to use the @Clustered annotation to enable clustering behavior.

You can disable replication for a stateful session bean by setting passivationCapable to false in the @Stateful annotation:

@Stateful(passivationCapable=false)

This instructs the server to use the ejb cache defined by passivation-disabled-cache-ref instead of cache-ref.

To globally disable the replication of stateful session beans, use the following management CLI command:

/subsystem=ejb3:write-attribute(name=default-sfsb-cache,value=simple)

8.4. Failover for Clustered EJB

Clustered EJB have failover capability. The state of the @Stateful EJB is replicated across the cluster nodes so that if one of the nodes in the cluster goes down, some other node will be able to take over the invocations.

Under some circumstances in a clustered environment, such as when a server in the cluster crashes, the EJB client might receive an exception instead of a response. The EJB client library will automatically retry the invocation when it is safe to do so, depending on the type of the failure that occurs. However, if a request fails and it cannot be determined conclusively to be safe to retry, then you can handle the exception as appropriate for your environment. You can, however, use custom interceptors to add additional retry behavior.

8.5. Remote Standalone Clients

Note

The use of the jboss-ejb-client.properties file is deprecated in favor of the wildfly-config.xml file.

A standalone remote client can use either the Java Naming and Directory Interface approach or native JBoss EJB client APIs to communicate with the servers. The important thing to note is that when you are invoking clustered EJB deployments, you do not have to list all the servers within the cluster. This would not have been feasible due the dynamic nature of cluster node additions within a cluster.

The remote client has to list only one of the servers with the clustering capability. This server will act as the starting point for cluster topology communication between the client and the clustered nodes.

Note that you have to configure the ejb cluster in the jboss-ejb-client.properties configuration file:

remote.clusters=ejb
remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.cluster.ejb.connect.options.org.xnio.Options.SSL_ENABLED=false

8.6. Cluster Topology Communication

Note

The use of the jboss-ejb-client.properties file is deprecated in favor of the wildfly-config.xml file.

When a client connects to a server, the JBoss EJB client implementation communicates internally with the server for the cluster topology information, if the server has clustering capability. For example, assuming that server X is listed as the initial server to connect to, when the client connects to server X, the server will send back an asynchronous cluster topology message to the client. This topology message consists of the cluster name and the information of the nodes that belong to the cluster. The node information includes the node address and port number to connect to, when required. So in this example, server X will send back the cluster topology consisting of the other server Y that belongs to the cluster.

In case of stateful clustered EJB, the invocation flow happens in two steps.

  1. Creation of a session for the stateful bean, which happens when you do a Java Naming and Directory Interface lookup for that bean.
  2. Invocation of the returned proxy.

The lookup for the stateful bean, internally, triggers a synchronous session creation request from the client to the server. In this case, the session creation request goes to server X because it was configured in the jboss-ejb-client.properties file. Since server X is clustered, it will return a session id and send back an affinity of that session. In case of clustered servers, the affinity is equal to the name of the cluster to which the stateful bean belongs on the server side. For non-clustered beans, the affinity is the node name on which the session was created. This affinity will help the EJB client to route the invocations on the proxy, as appropriate, to either a node within a cluster for clustered beans, or to a specific node for non-clustered beans. While this session creation request is going on, server X will also send back an asynchronous message that contains the cluster topology. The JBoss EJB client implementation will record this topology information and use it later for connection creation to nodes within the cluster and routing invocations to those nodes, when required.

To understand how failover works, consider the same example of server X being the starting point and a client application looking up a stateful bean and invoking it. During these invocations, the client side collects the cluster topology information from the server. Assuming that for some reason server X goes down and the client application subsequently invokes on the proxy. The JBoss EJB client implementation at this stage must be aware of the affinity, and in this case it is the cluster affinity. From the cluster topology information that the client has, it knows that the cluster has two nodes, server X and server Y. When the invocation arrives, the client notices that server X is down, so it uses a selector to fetch a suitable node from the cluster nodes. When the selector returns a node from the cluster nodes, the JBoss EJB client implementation creates a connection to that node, if the connection was not already created earlier, and creates an EJB receiver out of it. Since in this example, the only other node in the cluster is server Y, the selector will return server Y as the node and the JBoss EJB client implementation will use it to create an EJB receiver out of it and use this receiver to pass on the invocation on the proxy. Effectively, the invocation has now failed over to a different node within the cluster.

8.7. Automatic Transaction Stickiness for EJB

A transaction object, which is looked up from the same context as the EJB proxy, targets the same host. Having an active transaction pins the invocation context to the same node, if the context is multi-host or clustered.

This behavior depends on whether you have outflowed your transaction or you are using a remote user transaction.

For an outflowed transaction, when an application is looked up on a specific node, all the invocations to that application under the same transaction attempt to target this node. The nodes that have already received the outflowed transaction will be preferred over nodes that have not received it yet.

For a remote user transaction, the first successful invocation will lock the transaction to the given node, and subsequent invocations under this transaction must go to the same node, otherwise an exception is thrown.

8.8. Remote Clients on Another Instance

This section explains how a client application deployed on a JBoss EAP instance invokes a clustered stateful bean that is deployed on another JBoss EAP instance.

In the following example, there are three servers involved. Servers X and Y both belong to a cluster and have clustered EJB deployed on them. There is another server instance server C, which may or may not have clustering capability. Server C acts as a client on which there is a deployment that wants to invoke the clustered beans deployed on servers X and Y and achieve failover.

The configurations are done in the jboss-ejb-client.xml file, which points to a remote outbound connection to the other server. The configuration in the jboss-ejb-client.xml file is in the deployment of server C because server C is the client. The client configuration need not point to all the clustered nodes, but just to one of them. This will act as a starting point for the communication.

In this case, a remote outbound connection is created from server C to server X and then server X is used as the starting point for the communication. Similar to the case of remote standalone clients, when the application on server C looks up a stateful bean, a session creation request is sent to server X that returns a session id and the cluster affinity for it. Server X also sends back an asynchronous message to server C containing the cluster topology. This topology information includes the node information of server Y, because server Y belongs to the cluster along with server X. Subsequent invocations on the proxy will be routed appropriately to the nodes in the cluster. If server X goes down, as explained earlier, a different node from the cluster will be selected and the invocation will be forwarded to that node.

Both remote standalone clients as well as remote clients on another JBoss EAP instance act similarly in terms of failover.

8.9. Standalone and In-server Client Configuration

Note

The use of the jboss-ejb-client.properties file is deprecated in favor of the wildfly-config.xml file.

To connect an EJB client to a clustered EJB application, you need to expand the existing configuration in standalone EJB client or in-server EJB client to include cluster connection configuration. The jboss-ejb-client.properties for standalone EJB client, or even jboss-ejb-client.xml file for a server-side application must be expanded to include a cluster configuration.

Note

An EJB client is any program that uses an EJB on a remote server. A client is in-server when the EJB client calling the remote server is itself running inside of a server. In other words, a JBoss EAP instance calling out to another JBoss EAP instance would be considered an in-server client.

This example shows the additional cluster configuration required for a standalone EJB client.

remote.clusters=ejb
remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.cluster.ejb.connect.options.org.xnio.Options.SSL_ENABLED=false
remote.cluster.ejb.username=test
remote.cluster.ejb.password=password

If an application uses the remote-outbound-connection, you need to configure the jboss-ejb-client.xml file and add cluster configuration as shown in the following example:

<jboss-ejb-client xmlns:xsi="urn:jboss:ejb-client:1.2" xsi:noNamespaceSchemaLocation="jboss-ejb-client_1_2.xsd">
  <client-context>
    <ejb-receivers>
      <!-- this is the connection to access the app-one -->
      <remoting-ejb-receiver outbound-connection-ref="remote-ejb-connection-1" />
      <!-- this is the connection to access the app-two -->
      <remoting-ejb-receiver outbound-connection-ref="remote-ejb-connection-2" />
    </ejb-receivers>

    <!-- If an outbound connection connects to a cluster,
             a list of members is provided after successful connection.
         To connect to this node this cluster element must be defined. -->

    <clusters>
      <!-- cluster of remote-ejb-connection-1 -->
      <cluster name="ejb" security-realm="ejb-security-realm-1" username="quickuser1">
        <connection-creation-options>
        <property name="org.xnio.Options.SSL_ENABLED" value="false" />
        <property name="org.xnio.Options.SASL_POLICY_NOANONYMOUS" value="false" />
        </connection-creation-options>
      </cluster>
    </clusters>
  </client-context>
</jboss-ejb-client>

For more information about remote-outbound-connection, see About the Remoting Subsystem in the JBoss EAP Configuration Guide.

Note

For a secure connection you need to add the credentials to cluster configuration in order to avoid an authentication exception.

8.10. Implementing a Custom Load Balancing Policy for EJB Calls

Note

The use of the jboss-ejb-client.properties file is deprecated in favor of the wildfly-config.xml file.

It is possible to implement an alternate or customized load balancing policy in order to balance an application’s EJB calls across servers.

You can implement AllClusterNodeSelector for EJB calls. The node selection behavior of AllClusterNodeSelector is similar to a default selector except that AllClusterNodeSelector uses all available cluster nodes even in case of a large cluster (number of nodes > 20). If an unconnected cluster node is returned, it is opened automatically. The following example shows AllClusterNodeSelector implementation:

package org.jboss.as.quickstarts.ejb.clients.selector;

import java.util.Arrays;
import java.util.Random;
import java.util.logging.Level;
import java.util.logging.Logger;

import org.jboss.ejb.client.ClusterNodeSelector;
public class AllClusterNodeSelector implements ClusterNodeSelector {
  private static final Logger LOGGER = Logger.getLogger(AllClusterNodeSelector.class.getName());

  @Override
  public String selectNode(final String clusterName, final String[] connectedNodes, final String[] availableNodes) {
    if(LOGGER.isLoggable(Level.FINER)) {
      LOGGER.finer("INSTANCE "+this+ " : cluster:"+clusterName+" connected:"+Arrays.deepToString(connectedNodes)+" available:"+Arrays.deepToString(availableNodes));
    }

    if (availableNodes.length == 1) {
        return availableNodes[0];
    }
    final Random random = new Random();
    final int randomSelection = random.nextInt(availableNodes.length);
    return availableNodes[randomSelection];
  }

}

You can also implement the SimpleLoadFactorNodeSelector for EJB calls. Load balancing in SimpleLoadFactorNodeSelector happens based on a load factor. The load factor (2/3/4) is calculated based on the names of nodes (A/B/C) irrespective of the load on each node. The following example shows SimpleLoadFactorNodeSelector implementation:

package org.jboss.as.quickstarts.ejb.clients.selector;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.logging.Level;
import java.util.logging.Logger;

import org.jboss.ejb.client.DeploymentNodeSelector;
public class SimpleLoadFactorNodeSelector implements DeploymentNodeSelector {
  private static final Logger LOGGER = Logger.getLogger(SimpleLoadFactorNodeSelector.class.getName());
  private final Map<String, List<String>[]> nodes = new HashMap<String, List<String>[]>();
  private final Map<String, Integer> cursor = new HashMap<String, Integer>();

  private ArrayList<String> calculateNodes(Collection<String> eligibleNodes) {
    ArrayList<String> nodeList = new ArrayList<String>();

    for (String string : eligibleNodes) {
      if(string.contains("A") || string.contains("2")) {
        nodeList.add(string);
        nodeList.add(string);
      } else if(string.contains("B") || string.contains("3")) {
        nodeList.add(string);
        nodeList.add(string);
        nodeList.add(string);
      } else if(string.contains("C") || string.contains("4")) {
        nodeList.add(string);
        nodeList.add(string);
        nodeList.add(string);
        nodeList.add(string);
      }
    }
    return nodeList;
  }

  @SuppressWarnings("unchecked")
  private void checkNodeNames(String[] eligibleNodes, String key) {
    if(!nodes.containsKey(key) || nodes.get(key)[0].size() != eligibleNodes.length || !nodes.get(key)[0].containsAll(Arrays.asList(eligibleNodes))) {
      // must be synchronized as the client might call it concurrent
      synchronized (nodes) {
        if(!nodes.containsKey(key) || nodes.get(key)[0].size() != eligibleNodes.length || !nodes.get(key)[0].containsAll(Arrays.asList(eligibleNodes))) {
          ArrayList<String> nodeList = new ArrayList<String>();
          nodeList.addAll(Arrays.asList(eligibleNodes));

          nodes.put(key, new List[] { nodeList, calculateNodes(nodeList) });
        }
      }
    }
  }
   private synchronized String nextNode(String key) {
    Integer c = cursor.get(key);
    List<String> nodeList = nodes.get(key)[1];

    if(c == null || c >= nodeList.size()) {
      c = Integer.valueOf(0);
    }

    String node = nodeList.get(c);
    cursor.put(key, Integer.valueOf(c + 1));

    return node;
  }

  @Override
  public String selectNode(String[] eligibleNodes, String appName, String moduleName, String distinctName) {
    if (LOGGER.isLoggable(Level.FINER)) {
      LOGGER.finer("INSTANCE " + this + " : nodes:" + Arrays.deepToString(eligibleNodes) + " appName:" + appName + " moduleName:" + moduleName
          + " distinctName:" + distinctName);
    }

    // if there is only one there is no sense to choice
    if (eligibleNodes.length == 1) {
      return eligibleNodes[0];
    }
    final String key = appName + "|" + moduleName + "|" + distinctName;

    checkNodeNames(eligibleNodes, key);
    return nextNode(key);
  }
}

Configuring the jboss-ejb-client.properties File

You need to add the property remote.cluster.ejb.clusternode.selector with the name of your implementation class (AllClusterNodeSelector or SimpleLoadFactorNodeSelector). The selector will see all configured servers that are available at the invocation time. The following example uses AllClusterNodeSelector as the cluster node selector:

remote.clusters=ejb
remote.cluster.ejb.clusternode.selector=org.jboss.as.quickstarts.ejb.clients.selector.AllClusterNodeSelector
remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.cluster.ejb.connect.options.org.xnio.Options.SSL_ENABLED=false
remote.cluster.ejb.username=test
remote.cluster.ejb.password=password

remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=one,two
remote.connection.one.host=localhost
remote.connection.one.port = 8080
remote.connection.one.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.one.username=user
remote.connection.one.password=user123
remote.connection.two.host=localhost
remote.connection.two.port = 8180
remote.connection.two.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false

Using EJB Client API

You need to add the property remote.cluster.ejb.clusternode.selector to the list for the PropertiesBasedEJBClientConfiguration constructor. The following example uses AllClusterNodeSelector as the cluster node selector:

Properties p = new Properties();
p.put("remote.clusters", "ejb");
p.put("remote.cluster.ejb.clusternode.selector", "org.jboss.as.quickstarts.ejb.clients.selector.AllClusterNodeSelector");
p.put("remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS", "false");
p.put("remote.cluster.ejb.connect.options.org.xnio.Options.SSL_ENABLED", "false");
p.put("remote.cluster.ejb.username", "test");
p.put("remote.cluster.ejb.password", "password");

p.put("remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED", "false");
p.put("remote.connections", "one,two");
p.put("remote.connection.one.port", "8080");
p.put("remote.connection.one.host", "localhost");
p.put("remote.connection.two.port", "8180");
p.put("remote.connection.two.host", "localhost");

EJBClientConfiguration cc = new PropertiesBasedEJBClientConfiguration(p);
ContextSelector<EJBClientContext> selector = new ConfigBasedEJBClientContextSelector(cc);
EJBClientContext.setSelector(selector);

p = new Properties();
p.put(Context.URL_PKG_PREFIXES, "org.jboss.ejb.client.naming");
InitialContext context = new InitialContext(p);

Configuring the jboss-ejb-client.xml File

To use the load balancing policy for server to server communication, package the class together with the application and configure it within the jboss-ejb-client.xml settings located in META-INF folder. The following example uses AllClusterNodeSelector as the cluster node selector:

<jboss-ejb-client xmlns:xsi="urn:jboss:ejb-client:1.2" xsi:noNamespaceSchemaLocation="jboss-ejb-client_1_2.xsd">
  <client-context deployment-node-selector="org.jboss.ejb.client.DeploymentNodeSelector">
    <ejb-receivers>
      <!-- This is the connection to access the application. -->
      <remoting-ejb-receiver outbound-connection-ref="remote-ejb-connection-1" />
    </ejb-receivers>
    <!-- Specify the cluster configurations applicable for this client context -->
    <clusters>
      <!-- Configure the cluster of remote-ejb-connection-1. -->
      <cluster name="ejb" security-realm="ejb-security-realm-1" username="test" cluster-node-selector="org.jboss.as.quickstarts.ejb.clients.selector.AllClusterNodeSelector">
        <connection-creation-options>
          <property name="org.xnio.Options.SSL_ENABLED" value="false" />
          <property name="org.xnio.Options.SASL_POLICY_NOANONYMOUS" value="false" />
        </connection-creation-options>
      </cluster>
    </clusters>
  </client-context>
</jboss-ejb-client>

To use the above configuration with security, you will need to add ejb-security-realm-1 to client-server configuration. The following example shows the CLI commands for adding security realm (ejb-security-realm-1) the value is the base64 encoded password for the user "test":

/core-service=management/security-realm=ejb-security-realm-1:add()
/core-service=management/security-realm=ejb-security-realm-1/server-identity=secret:add(value=cXVpY2sxMjMr)

If the load balancing policy should be used for server to server communication, the class can be packaged together with the application or as a module. This class is configured in the jboss-ejb-client settings file located in the META-INF directory of the top-level EAR archive. The following example uses RoundRobinNodeSelector as the deployment node selector.

<jboss-ejb-client xmlns="urn:jboss:ejb-client:1.2">
    <client-context deployment-node-selector="org.jboss.example.RoundRobinNodeSelector">
        <ejb-receivers>
            <remoting-ejb-receiver outbound-connection-ref="..."/>
        </ejb-receivers>
        ...
    </client-context>
</jboss-ejb-client>
Note

If you are running a standalone server, use the start option -Djboss.node.name= or the server configuration file standalone.xml to configure the server name. Ensure that the server name is unique. If you are running a managed domain, the host controller automatically validates that the names are unique.

8.11. EJB Transactions in a Clustered Environment

If the client code invokes a clustered EJB, then the cluster affinity is set automatically. If you manage transactions on the client side, you can choose to target a specific node in the cluster or you can allow the client to lazily select the cluster node to handle transactions. This section describes both options.

EJB Transactions Target a Specific Node

You can target a specific node in the cluster to handle a transaction using the following procedure.

  1. Specify the target cluster node address using the PROVIDER_URL property when creating the InitialContext.

    props.put(Context.PROVIDER_URL, "remote+http://127.0.0.1:8080");
    ...
    InitialContext ctx = new InitialContext(props);
  2. In the client, look up the txn:RemoteUserTransaction from the InitialContext.

    UserTransaction ut = (UserTransaction)ctx.lookup("txn:RemoteUserTransaction");

    You can do a Java Naming and Directory Interface lookup for a UserTransaction by setting the PROVIDER_URL property to the URL of the server and then look up txn:UserTransaction, as shown in the code example below:

    final Hashtable<String, String> jndiProperties = new Hashtable<>();
    jndiProperties.put(Context.INITIAL_CONTEXT_FACTORY, "org.wildfly.naming.client.WildFlyInitialContextFactory");
    jndiProperties.put(Context.PROVIDER_URL, "remote+http://localhost:8080");
    final Context context = new InitialContext(jndiProperties);
    
    SecuredEJBRemote reference = (SecuredEJBRemote) context.lookup("txn:UserTransaction");

    UserTransaction is not bound to any particular destination until an actual invocation takes place. Upon invocation, this UserTransaction is bound to the respective destination for the entire lifetime of the transaction.

    You do not need to know the node name or the destination before beginning a UserTransaction. The org.jboss.ejb.client.EJBClient.getUserTransaction() method gives you a remote UserTransaction that automatically selects its destination based on the first invocation. Looking up a remote UserTransaction from Java Naming and Directory Interface also works the same way.

    Note

    The org.jboss.ejb.client.EJBClient.getUserTransaction() method is deprecated.

  3. When the transaction begins, all EJB invocations are then bound to that specific node for duration of the transaction, establishing server affinity.
  4. When the transaction ends, the server affinity is released, and the EJB proxies return to a general cluster affinity.

EJB Transactions Lazily Select a Node

You can allow the client to lazily select the cluster node to handle transactions during the first invocation pertaining to a transaction. This allows for load balancing of transactions across the cluster. To use this option, follow the procedure below.

  1. Do not specify the PROVIDER_URL property in the InitialContext used to invoke the EJB.
  2. In the client, look up the txn:RemoteUserTransaction from the InitialContext.

    UserTransaction ut = (UserTransaction)ctx.lookup("txn:RemoteUserTransaction");
  3. When the transaction begins, one cluster node is selected automatically, establishing server affinity, and all EJB invocations are then bound to that specific node for duration of the transaction.
  4. When the transaction ends, the server affinity is released, and the EJB proxies return to a general cluster affinity.