Chapter 6. Clustering in Web Applications

6.1. Session Replication

6.1.1. About HTTP Session Replication

Session replication ensures that client sessions of distributable applications are not disrupted by failovers of nodes in a cluster. Each node in the cluster shares information about ongoing sessions, and can take over sessions if a node disappears.

Session replication is the mechanism by which mod_cluster, mod_jk, mod_proxy, ISAPI, and NSAPI clusters provide high availability.

6.1.2. Enable Session Replication in Your Application

To take advantage of JBoss EAP High Availability (HA) features and enable clustering of your web application, you must configure your application to be distributable.

Make your Application Distributable
  1. Indicate that your application is distributable. If your application is not marked as distributable, its sessions will never be distributed. Add the <distributable/> element inside the <web-app> tag of your application’s web.xml descriptor file:

    Example: Minimum Configuration for a Distributable Application

    <?xml version="1.0"?>
    <web-app  xmlns=""

  2. Next, if desired, modify the default replication behavior. If you want to change any of the values affecting session replication, you can override them inside a <replication-config> element inside <jboss-web> in an application’s WEB-INF/jboss-web.xml file. For a given element, only include it if you want to override the defaults.

    Example: <replication-config> Values

    <jboss-web xmlns=""

The <replication-granularity> parameter determines the granularity of data that is replicated. It defaults to SESSION, but can be set to ATTRIBUTE to increase performance on sessions where most attributes remain unchanged.

Valid values for <replication-granularity> can be :

  • SESSION: The default value. The entire session object is replicated if any attribute is dirty. This policy is required if an object reference is shared by multiple session attributes. The shared object references are maintained on remote nodes since the entire session is serialized in one unit.
  • ATTRIBUTE: This is only for dirty attributes in the session and for some session data, such as the last-accessed timestamp.
Immutable Session Attributes

For JBoss EAP7, session replication is triggered when the session is mutated or when any mutable attribute of the session is accessed. Session attributes are assumed to be mutable unless one of the following is true:

  • The value is a known immutable value:

    • null
    • java.util.Collections.EMPTY_LIST, EMPTY_MAP, EMPTY_SET
  • The value type is or implements a known immutable type:

    • java.lang.Boolean, Character, Byte, Short, Integer, Long, Float, Double
    • java.lang.Class, Enum, StackTraceElement, String
    •, java.nio.file.Path
    • java.math.BigDecimal, BigInteger, MathContext
    •, Inet6Address, InetSocketAddress, URI, URL
    • java.util.Currency, Locale, TimeZone, UUID
    • java.time.Clock, Duration, Instant, LocalDate, LocalDateTime, LocalTime, MonthDay, Period, Year, YearMonth, ZoneId, ZoneOffset, ZonedDateTime
    • java.time.chrono.ChronoLocalDate, Chronology, Era
    • java.time.format.DateTimeFormatter, DecimalStyle
    • java.time.temporal.TemporalField, TemporalUnit, ValueRange, WeekFields
    •, ZoneOffsetTransitionRule, ZoneRules
  • The value type is annotated with:

    • @org.wildfly.clustering.web.annotation.Immutable
    • @net.jcip.annotations.Immutable

6.2. HTTP Session Passivation and Activation

6.2.1. About HTTP Session Passivation and Activation

Passivation is the process of controlling memory usage by removing relatively unused sessions from memory while storing them in persistent storage.

Activation is when passivated data is retrieved from persisted storage and put back into memory.

Passivation occurs at different times in an HTTP session’s lifetime:

  • When the container requests the creation of a new session, if the number of currently active sessions exceeds a configurable limit, the server attempts to passivate some sessions to make room for the new one.
  • When a web application is deployed and a backup copy of sessions active on other servers is acquired by the newly deploying web application’s session manager, sessions may be passivated.

A session is passivated if the number of active sessions exceeds a configurable maximum.

Sessions are always passivated using a Least Recently Used (LRU) algorithm.

6.2.2. Configure HTTP Session Passivation in Your Application

HTTP session passivation is configured in your application’s WEB-INF/jboss-web.xml and META-INF/jboss-web.xml file.

Example: jboss-web.xml File

<jboss-web xmlns=""


The <max-active-sessions> element dictates the maximum number of active sessions allowed, and is used to enable session passivation. If session creation would cause the number of active sessions to exceed <max-active-sessions/>, then the oldest session known to the session manager will passivate to make room for the new session.


The total number of sessions in memory includes sessions replicated from other cluster nodes that are not being accessed on this node. Take this into account when setting <max-active-sessions>. The number of sessions replicated from other nodes also depends on whether REPL or DIST cache mode is enabled. In REPL cache mode, each session is replicated to each node. In DIST cache mode, each session is replicated only to the number of nodes specified by the owners parameter. See Configure the Cache Mode in the JBoss EAP Config Guide for information on configuring session cache modes. For example, consider an eight node cluster, where each node handles requests from 100 users. With REPL cache mode, each node would store 800 sessions in memory. With DIST cache mode enabled, and the default owners setting of 2, each node stores 200 sessions in memory.

6.3. Public API for Clustering Services

JBoss EAP 7 introduces a refined public clustering API for use by applications. The new services are designed to be lightweight, easily injectable, with no external dependencies.

The group service provides a mechanism to view the cluster topology for a JGroups channel, and to be notified when the topology changes.

@Resource(lookup = "java:jboss/clustering/group/channel-name")
private Group channelGroup;

The CommandDispatcherFactory service provides a mechanism to create a dispatcher for executing commands on nodes in the cluster. The resulting CommandDispatcher is a command-pattern analog to the reflection-based GroupRpcDispatcher from previous JBoss EAP releases.

@Resource(lookup = "java:jboss/clustering/dispatcher/channel-name")
private CommandDispatcherFactory factory;

public void foo() {
    String context = "Hello world!";
    try (CommandDispatcher<String> dispatcher = this.factory.createCommandDispatcher(context)) {
        dispatcher.executeOnCluster(new StdOutCommand());

public static class StdOutCommand implements Command<Void, String> {
    public Void execute(String context) {
        return null;

6.4. HA Singleton Service

A clustered singleton service, also known as a high-availability (HA) singleton, is a service deployed on multiple nodes in a cluster. The service is provided on only one of the nodes. The node running the singleton service is usually called the master node.

When the master node either fails or shuts down, another master is selected from the remaining nodes and the service is restarted on the new master. Other than a brief interval when one master has stopped and another has yet to take over, the service is provided by one, but only one, node.

HA Singleton ServiceBuilder API

JBoss EAP 7 introduces a new public API for building singleton services that simplifies the process significantly.

The SingletonServiceBuilder implementation installs its services so they will start asynchronously, preventing deadlocking of the Modular Service Container (MSC).

HA Singleton Service Election Policies

If there is a preference for which node should start the ha-singleton, you can set the election policy in the ServiceActivator class.

JBoss EAP provides two election policies:

  1. Simple Election Policy

    The simple election policy selects a master node based on the relative age. The required age is configured in the position property, which is the index in the list of available nodes where,

    • position = 0 – refers to the oldest node (the default)
    • position = 1 – refers to the 2nd oldest etc.

      Position can also be negative to indicate the youngest nodes.

    • position = -1 – refers to the youngest node
    • position = -2 – refers to the 2nd youngest node etc.
  2. Random Election Policy

    The random election policy elects a random member to be the provider of a singleton service.

Create an HA Singleton Service Application

The following is an abbreviated example of the steps required to create and deploy an application as a cluster-wide singleton service. This example service activates a scheduled timer that is started only once in the cluster.

  1. Create an HATimerService service that implements the org.jboss.msc.service.Service interface and contains the getValue(), start(), and stop() methods.

    Service Class Code Example

    public class HATimerService implements Service<String> {
        private static final Logger LOGGER = Logger.getLogger(HATimerService.class.toString());
        public static final ServiceName SINGLETON_SERVICE_NAME = ServiceName.JBOSS.append("quickstart", "ha", "singleton", "timer");
         * A flag whether the service is started.
        private final AtomicBoolean started = new AtomicBoolean(false);
         * @return the name of the server node
        public String getValue() throws IllegalStateException, IllegalArgumentException {
  "%s is %s at %s", HATimerService.class.getSimpleName(), (started.get() ? "started" : "not started"), System.getProperty("")));
            return System.getProperty("");
        public void start(StartContext arg0) throws StartException {
            if (!started.compareAndSet(false, true)) {
                throw new StartException("The service is still started!");
  "Start HASingleton timer service '" + this.getClass().getName() + "'");
            final String node = System.getProperty("");
            try {
                InitialContext ic = new InitialContext();
                ((Scheduler) ic.lookup("global/jboss-cluster-ha-singleton-service/SchedulerBean!"))
                    .initialize("HASingleton timer @" + node + " " + new Date());
            } catch (NamingException e) {
                throw new StartException("Could not initialize timer", e);
        public void stop(StopContext arg0) {
            if (!started.compareAndSet(true, false)) {
                LOGGER.warning("The service '" + this.getClass().getName() + "' is not active!");
            } else {
      "Stop HASingleton timer service '" + this.getClass().getName() + "'");
                try {
                    InitialContext ic = new InitialContext();
                    ((Scheduler) ic.lookup("global/jboss-cluster-ha-singleton-service/SchedulerBean!")).stop();
                } catch (NamingException e) {
          "Could not stop timer:" + e.getMessage());

  2. Create a service activator that implements the org.jboss.msc.service.ServiceActivator interface and installs the HATimerService as a clustered singleton in the activate() method. This example specifies that node1 should start the singleton service.

    Service Activator Code Example

    public class HATimerServiceActivator implements ServiceActivator {
        private final Logger log = Logger.getLogger(this.getClass().toString());
        public void activate(ServiceActivatorContext context) {
  "HATimerService will be installed!");
            HATimerService service = new HATimerService();
            ServiceName factoryServiceName = SingletonServiceName.BUILDER.getServiceName("server", "default");
            ServiceController<?> factoryService = context.getServiceRegistry().getRequiredService(factoryServiceName);
            SingletonServiceBuilderFactory factory = (SingletonServiceBuilderFactory) factoryService.getValue();
            ServiceName ejbComponentService = ServiceName.of("jboss", "deployment", "unit", "jboss-cluster-ha-singleton-service.jar", "component", "SchedulerBean", "START");
            factory.createSingletonServiceBuilder(HATimerService.SINGLETON_SERVICE_NAME, service)
                .electionPolicy(new PreferredSingletonElectionPolicy(new SimpleSingletonElectionPolicy(), new NamePreference("node1/singleton")))
                .build(new DelegatingServiceContainer(context.getServiceTarget(), context.getServiceRegistry()))

  3. Create a file named org.jboss.msc.service.ServiceActivator in the application’s META-INF/services/ directory and add a line containing the fully qualified name of the ServiceActivator class created in the previous step.

    META-INF/services/org.jboss.msc.service.ServiceActivator File Example

  4. Create a Scheduler interface that contains the initialize() and stop() methods.

    Scheduler Interface Code Example

    public interface Scheduler {
        void initialize(String info);
        void stop();

  5. Create a Singleton bean that implements the Scheduler interface. This bean is used as the cluster-wide singleton timer.


    The Singleton bean must not have a remote interface and you must not reference its local interface from another EJB in any application. This prevents a lookup by a client or other component and ensures the HATimerService has total control of the Singleton.

    Singleton Bean Code Example

    public class SchedulerBean implements Scheduler {
        private static Logger LOGGER = Logger.getLogger(SchedulerBean.class.toString());
        private TimerService timerService;
        public void scheduler(Timer timer) {
  "HASingletonTimer: Info=" + timer.getInfo());
        public void initialize(String info) {
            ScheduleExpression sexpr = new ScheduleExpression();
            // set schedule to every 10 seconds for demonstration
            // persistent must be false because the timer is started by the HASingleton service
            timerService.createCalendarTimer(sexpr, new TimerConfig(info, false));
        public void stop() {
  "Stop all existing HASingleton timers");
            for (Timer timer : timerService.getTimers()) {
                LOGGER.fine("Stop HASingleton timer: " + timer.getInfo());

See the cluster-ha-singleton quickstart that ships with JBoss EAP for a complete working example of this application. The quickstart provides detailed instructions to build and deploy the application.

6.5. HA Singleton Deployments

JBoss EAP 7 adds the ability to deploy a given application as a singleton deployment.

When deployed to a group of clustered servers, a singleton deployment will only deploy on a single node at any given time. If the node on which the deployment is active stops or fails, the deployment will automatically start on another node.

The policies for controlling HA singleton behavior are managed by a new singleton subsystem. A deployment may either specify a specific singleton policy or use the default subsystem policy. A deployment identifies itself as singleton deployment via a /META-INF/singleton-deployment.xml deployment descriptor which is most easily applied to an existing deployment as a deployment overlay. Alternatively, the requisite singleton configuration can be embedded within an existing jboss-all.xml.

Defining or Choosing a Singleton Deployment

  • To define a deployment as a singleton deployment, include a /META-INF/singleton-deployment.xml descriptor in your application archive.

    Example: Singleton Deployment Descriptor

    <?xml version="1.0" encoding="UTF-8"?>
    <singleton-deployment xmlns="urn:jboss:singleton-deployment:1.0"/>

    Example: Singleton Deployment Descriptor with a Specific Singleton Policy

    <?xml version="1.0" encoding="UTF-8"?>
    <singleton-deployment policy="my-new-policy" xmlns="urn:jboss:singleton-deployment:1.0"/>

  • Alternatively, you can also add a singleton-deployment element to your jboss-all.xml descriptor.

    Example: Defining singleton-deployment in jboss-all.xml

    <?xml version="1.0" encoding="UTF-8"?>
    <jboss xmlns="urn:jboss:1.0">
        <singleton-deployment xmlns="urn:jboss:singleton-deployment:1.0"/>

    Example: Defining singleton-deployment in jboss-all.xml with a Specific Singleton Policy

    <?xml version="1.0" encoding="UTF-8"?>
    <jboss xmlns="urn:jboss:1.0">
        <singleton-deployment policy="my-new-policy" xmlns="urn:jboss:singleton-deployment:1.0"/>

Creating a Singleton Deployment

JBoss EAP provides two election policies:

  • Simple Election Policy

    The simple-election-policy chooses a specific member, indicated by the position attribute, on which a given application will be deployed. The position attribute determines the index of the node to be elected from a list of candidates sorted by descending age, where 0 indicates the oldest node, 1 indicates the second oldest node, -1 indicates the youngest node, -2 indicates the second youngest node, and so on. If the specified position exceeds the number of candidates, a modulus operation is applied.

    Example: Create a New Singleton Policy with a simple-election-policy and Position Set to -1, Using the Management CLI



    To set the newly created policy my-new-policy as the default, run this command:

    /subsystem=singleton:write-attribute(name=default, value=my-new-policy)

    Example: Configure a simple-election-policy with Position Set to -1 Using standalone-ha.xml

    <subsystem xmlns="urn:jboss:domain:singleton:1.0">
       <singleton-policies default="my-new-policy">
          <singleton-policy name="my-new-policy" cache-container="server">
             <simple-election-policy position="-1"/>

  • Random Election Policy

    The random-election-policy chooses a random member on which a given application will be deployed.

    Example: Creating a New Singleton Policy with a random-election-policy, Using the Management CLI


    Example: Configure a random-election-policy Using standalone-ha.xml

    <subsystem xmlns="urn:jboss:domain:singleton:1.0">
       <singleton-policies default="my-other-new-policy">
          <singleton-policy name="my-other-new-policy" cache-container="server">


    The default-cache attribute of the cache-container needs to be defined before trying to add the policy. Without this, if you are using a custom cache container, you might end up getting error messages.


Additionally, any singleton election policy may indicate a preference for one or more members of a cluster. Preferences may be defined either via node name or via outbound socket binding name. Node preferences always take precedent over the results of an election policy.

Example: Indicate Preference in the Existing Singleton Policy Using the Management CLI

/subsystem=singleton/singleton-policy=foo/election-policy=simple:list-add(name=name-preferences, value=nodeA)

/subsystem=singleton/singleton-policy=bar/election-policy=random:list-add(name=socket-binding-preferences, value=binding1)

Example: Create a New Singleton Policy with a simple-election-policy and name-preferences, Using the Management CLI

/subsystem=singleton/singleton-policy=my-new-policy/election-policy=simple:add(name-preferences=[node1, node2, node3, node4])


To set the newly created policy my-new-policy as the default, run this command:

/subsystem=singleton:write-attribute(name=default, value=my-new-policy)

Example: Configure a random-election-policy with socket-binding-preferences Using standalone-ha.xml

<subsystem xmlns="urn:jboss:domain:singleton:1.0">
   <singleton-policies default="my-other-new-policy">
      <singleton-policy name="my-other-new-policy" cache-container="server">
            <socket-binding-preferences>binding1 binding2 binding3 binding4</socket-binding-preferences>


Network partitions are particularly problematic for singleton deployments, since they can trigger multiple singleton providers for the same deployment to run at the same time. To defend against this scenario, a singleton policy may define a quorum that requires a minimum number of nodes to be present before a singleton provider election can take place. A typical deployment scenario uses a quorum of N/2 + 1, where N is the anticipated cluster size. This value can be updated at runtime, and will immediately affect any singleton deployments using the respective singleton policy.

Example: Quorum Declaration in the standalone-ha.xml File

<subsystem xmlns="urn:jboss:domain:singleton:1.0">
   <singleton-policies default="default">
      <singleton-policy name="default" cache-container="server" quorum="4">

Example: Quorum Declaration Using the Management CLI

/subsystem=singleton/singleton-policy=foo:write-attribute(name=quorum, value=3)

6.6. Apache mod_cluster-manager Application

6.6.1. About mod_cluster-manager Application

The mod_cluster-manager application is an administration web page, which is available on Apache HTTP Server. It is used for monitoring the connected worker nodes and performing various administration tasks, such as enabling or disabling contexts, and configuring the load-balancing properties of worker nodes in a cluster.

Exploring mod_cluster-manager Application

The mod_cluster-manager application can be used for performing various administration tasks on worker nodes.

mod_cluster Administration Web Page Figure - mod_cluster Administration Web Page

  • [1] mod_cluster/1.3.1.Final: The version of the mod_cluster native library.
  • [2] ajp:// The protocol used (either AJP, HTTP, or HTTPS), hostname or IP address of the worker node, and the port.
  • [3] jboss-eap-7.0-2: The worker node’s JVMRoute.
  • [4] Virtual Host 1: The virtual host(s) configured on the worker node.
  • [5] Disable: An administration option that can be used to disable the creation of new sessions on the particular context. However, the ongoing sessions do not get disabled and remain intact.
  • [6] Stop: An administration option that can be used to stop the routing of session requests to the context. The remaining sessions will failover to another node unless the sticky-session-force property is set to true.
  • [7] Enable Contexts Disable Contexts Stop Contexts: The operations that can be performed on the whole node. Selecting one of these options affects all the contexts of a node in all its virtual hosts.
  • [8] Load balancing group (LBGroup): The load-balancing-group property is set in the modcluster subsystem in JBoss EAP configuration to group all worker nodes into custom load balancing groups. Load balancing group (LBGroup) is an informational field that gives information about all set load balancing groups. If this field is not set, then all worker nodes are grouped into a single default load balancing group.


    This is only an informational field and thus cannot be used to set load-balancing-group property. The property has to be set in modcluster subsystem in JBoss EAP configuration.

  • [9] Load (value): The load factor on the worker node. The load factor(s) are evaluated as below:

    -load > 0 : A load factor with value 1 indicates that the worker node is overloaded. A load factor of 100 denotes a free and not-loaded node.
    -load = 0 : A load factor of value 0 indicates that the worker node is in standby mode. This means that no session requests will be routed to this node until and unless the other worker nodes are unavailable.
    -load = -1 : A load factor of value -1 indicates that the worker node is in an error state.
    -load = -2 : A load factor of value -2 indicates that the worker node is undergoing CPing/CPong and is in a transition state.

For JBoss EAP 7.0, it is also possible to use Undertow as load balancer.