The clustering features in JBoss Enterprise Web Platform are built on top of lower level libraries that provide much of the core functionality.
JGroups
|-- JBoss Cache
| |-- HTTP Session
| |-- Hibernate
|
|-- HAPartition
|-- HA-JNDI
|-- HA Singleton
|-- Client Proxies
JGroups is a toolkit for reliable point-to-point and point-to-multipoint communication. JGroups is used for all clustering-related communications between nodes in a JBoss Enterprise Web Platform cluster.
JBoss Cache is a highly flexible clustered transactional caching library. Many Enterprise Web Platform clustering services need to cache some state in memory while ensuring that a backup of that state is available on another node, and the data cached on each node in the cluster is consistent. JBoss Cache handles these concerns for most JBoss Enterprise Web Platform clustered services. JBoss Cache uses JGroups to handle its group communication requirements. POJO Cache is an extension of the core JBoss Cache that JBoss Enterprise Web Platform uses to support fine-grained replication of clustered web session state. See Section 17.1.3, “Distributed Caching with JBoss Cache” for more on how JBoss Enterprise Web Platform uses JBoss Cache and POJO Cache.
HAPartition is an adapter on top of a JGroups channel that allows multiple services to use the channel. HAPartition also supports a distributed registry of which HAPartition-based services are running on which cluster members. It provides notifications to interested listeners when the cluster membership changes or the clustered service registry changes. See Section 17.1.4, “The HAPartition Service” for more details on HAPartition.
The other higher level clustering services make use of JBoss Cache or HAPartition, or, in the case of HA-JNDI, both.
JGroups provides the underlying group communication support for JBoss Enterprise Web Platform clusters. Services deployed on JBoss Enterprise Web Platform which need group communication with their peers will obtain a JGroups
Channel and use it to communicate. The Channel handles such tasks as managing which nodes are members of the group, detecting node failures, ensuring lossless, first-in-first-out delivery of messages to all group members, and providing flow control to ensure fast message senders cannot overwhelm slow message receivers.
The characteristics of a JGroups
Channel are determined by the set of protocols that compose it. Each protocol handles a single aspect of the overall group communication task; for example the UDP protocol handles the details of sending and receiving UDP datagrams. A Channel that uses the UDP protocol is capable of communicating with UDP unicast and multicast; alternatively one that uses the TCP protocol uses TCP unicast for all messages. JGroups supports a wide variety of different protocols (see Section 23.1, “Configuring a JGroups Channel's Protocol Stack” for details), but the Enterprise Web Platform ships with a default set of channel configurations that should meet most needs.
By default, all JGroups channels of the Enterprise Application Platform use the UDP multicast (an exception to this is a JBoss Messaging channel, which is TCP-based). To change the default multicast type for a server, in
$JBOSS_HOME/bin/ create run.conf. Open the file and add the following: JAVA_OPTS="$JAVA_OPTS -Djboss.default.jgroups.stack=<METHOD>".
It is important to note that the JGroups channels needed by clustering services (for example, a channel used by a distributed
HttpSession cache) are not configured in detail as part of the consuming service's configuration, and are not directly instantiated by the consuming service. Instead, a new ChannelFactory service is used as a registry for named channel configurations and as a factory for Channel instances. A service that needs a channel requests the channel from the ChannelFactory, passing in the name of the desired configuration.
The
ChannelFactory service is deployed in the server/production/deploy/cluster/jgroups-channelfactory.sar. On startup the ChannelFactory service parses the server/production/deploy/cluster/jgroups-channelfactory.sar/META-INF/jgroups-channelfactory-stacks.xml file, which includes various standard JGroups configurations identified by name (for example, UDP or TCP). Services that require a channel access the channel factory and request a channel with a particular named configuration.
Note
If several services request a channel with the same configuration name from the
ChannelFactory, they are not handed a reference to the same underlying Channel. Each receives its own Channel, but the channels will have an identical configuration. A logical question is how those channels avoid forming a group with each other if each, for example, is using the same multicast address and port. The answer is that when a consuming service connects its Channel, it passes a unique-to-that-service cluster_name argument to the Channel.connect(String cluster_name) method. The channel uses that cluster_name as one of the factors that determine whether a particular message received over the network is intended for that particular channel.
The standard protocol stack configurations that ship with Enterprise Web Platform 5 are described below. Note that not all of these are actually used; many are included as a convenience to users who may wish to alter the default server configuration profile. The configuration used in a pristine Enterprise Web Platform 5
production configuration is udp.
You can add a new stack configuration by adding a new
stack element to the server/all/deploy/cluster/jgroups-channelfactory.sar/META-INF/jgroups-channelfactory-stacks.xml file. You can alter the behavior of an existing configuration by editing this file. Before doing this though, see the other standard configurations shipped with the Enterprise Web Platform to determine whether they will meet your requirements.
Important
Before you edit a configuration, you should understand which services use that configuration. It is important that any changes you make are appropriate for all affected services. If the change is not appropriate for a particular service, it may be better to create a new configuration and alter some services to use the new configuration.
udp- UDP multicast based stack meant to be shared between different channels. Message bundling is disabled, as it can add latency to synchronous group remote procedure calls (RPCs). Services that only make asynchronous RPCs (for example, JBoss Cache configured for
REPL_ASYNC) and do so in high volume may be able to improve performance by configuring their cache to use theudp-asyncstack below. Services that only make synchronous RPCs (for example, JBoss Cache configured forREPL_SYNCorINVALIDATION_SYNC) may be able to improve performance by using theudp-syncstack below, which does not include flow control. udp-async- Same as the default
udpstack above, except message bundling is enabled in the transport protocol (enable_bundling=true). Useful for services that make high-volume asynchronous RPCs (for example, high volume JBoss Cache instances configured forREPL_ASYNC) where message bundling may improve performance. udp-sync- UDP multicast based stack, without flow control and without message bundling. This can be used instead of
udpif synchronous calls are used and the message volume (rate and size) is not very large. Do not use this configuration if you send messages at a high sustained rate, or you might run out of memory. tcp- TCP based stack, with flow control and message bundling. TCP stacks are usually used when IP multicasting cannot be used in a network (for example, when routers discard multicast).
tcp-sync- TCP based stack, without flow control and without message bundling. TCP stacks are usually used when IP multicasting cannot be used in a network (for example, when routers discard multicast). This configuration should be used instead of
tcpabove when synchronous calls are used and the message volume (rate and size) is not very large. Do not use this configuration if you send messages at a high sustained rate, or you might run out of memory.
As the number of JGroups-based clustering services has risen over the years, the need to share the resources (particularly sockets and threads) used by these channels became a glaring problem. A pristine Enterprise Web Platform 5
production configuration will connect four JGroups channels during startup, and a total of seven or eight will be connected if distributable web applications, clustered EJB3 SFSBs and a clustered JPA or Hibernate second level cache are all used. This many channels can consume a lot of resources, and can be a real configuration nightmare if the network environment requires configuration to ensure cluster isolation.
Beginning with Enterprise Web Platform 5, JGroups supports sharing of transport protocol instances between channels. A JGroups channel is composed of a stack of individual protocols, each of which is responsible for one aspect of the channel's behavior. A transport protocol is a protocol that is responsible for actually sending messages on the network and receiving them from the network. The resources that are most desirable for sharing (sockets and thread pools) are managed by the transport protocol, so sharing a transport protocol between channels efficiently accomplishes JGroups resource sharing.
To configure a transport protocol for sharing, simply add a
singleton_name="someName" attribute to the protocol's configuration. All channels whose transport protocol configuration uses the same singleton_name value will share their transport. All other protocols in the stack will not be shared. Figure 17.2, “Services using a Shared Transport” illustrates four services running in a JVM, each with its own channel. Three of the services are sharing a transport; the fourth is using its own transport.
The protocol stack configurations used by the Enterprise Web Platform 5
ChannelFactory all have a singleton_name configured. In fact, if you add a stack to the ChannelFactory that doesn't include a singleton_name, before creating any channels for that stack, the ChannelFactory will synthetically create a singleton_name by concatenating the stack name to the string unnamed_, for example, unnamed_customStack.
JBoss Cache is a fully featured distributed cache framework that can be used in any application server environment or standalone. JBoss Cache provides the underlying distributed caching support used by many of the standard clustered services in a JBoss Enterprise Web Platform cluster, including:
- replication of clustered web application sessions
- clustered caching of JPA and Hibernate entities
- clustered Single Sign-On
- the HA-JNDI replicated tree
- DistributedStateService
Users can also create their own JBoss Cache and POJO Cache instances for custom use by their applications, see Chapter 24, JBoss Cache Configuration and Deployment for more on this.
Many of the standard clustered services in JBoss Enterprise Web Platform use JBoss Cache to maintain consistent state across the cluster. Different services (for example, web session clustering or second level caching of JPA/Hibernate entities) use different JBoss Cache instances, with each cache configured to meet the needs of the service that uses it.
In JBoss Enterprise Web Platform 5, the scattered cache deployments use the
CacheManager service, deployed via the $JBOSS_HOME/server/$PROFILE/deploy/cluster/jboss-cache-manager.sar. The CacheManager is a factory and registry for JBoss Cache instances. It is configured with a set of named JBoss Cache configurations. Services that require a cache request it by name from the cache manager. The cache manager creates the cache if it does not exist already, and returns it. The cache manager keeps a reference to each cache it has created, so all services that request the same cache configuration name will share the same cache. When a service is done with the cache, it releases it to the cache manager. The cache manager keeps track of how many services are using each cache, and will stop and destroy the cache when all services have released it.
The following standard JBoss Cache configurations ship with JBoss Enterprise Web Platform 5. You can add others to suit your needs, or edit these configurations to adjust cache behavior. Additions or changes are done by editing the
deploy/cluster/jboss-cache-manager.sar/META-INF/jboss-cache-manager-jboss-beans.xml file (see Section 24.2.1, “Deployment Via the CacheManager Service” for details). Note however that these configurations are specifically optimized for their intended use, and except as specifically noted in the documentation chapters for each service in this guide, it is not advisable to change them.
standard-session-cache- Standard cache used for web sessions.
field-granularity-session-cache- Standard cache used for FIELD granularity web sessions.
ha-partition- Used by web tier Clustered Single Sign-On, HA-JNDI, Distributed State.
mvcc-entity- A configuration appropriate for JPA/Hibernate entity/collection caching that uses JBoss Cache's MVCC locking (see notes below).
optimistic-entity- A configuration appropriate for JPA/Hibernate entity/collection caching that uses JBoss Cache's optimistic locking (see notes below).
pessimistic-entity- A configuration appropriate for JPA/Hibernate entity/collection caching that uses JBoss Cache's pessimistic locking (see notes below).
mvcc-entity-repeatable- Same as
mvcc-entitybut uses JBoss Cache'sREPEATABLE_READisolation level instead ofREAD_COMMITTED(see notes below). pessimistic-entity-repeatable- Same as
pessimistic-entitybut uses JBoss Cache'sREPEATABLE_READisolation level instead ofREAD_COMMITTED(see notes below). local-query- A configuration appropriate for JPA/Hibernate query result caching. Does not replicate query results. Do not store the timestamp data Hibernate uses to verify validity of query results in this cache.
replicated-query- A configuration appropriate for JPA/Hibernate query result caching. Replicates query results. DO NOT store the timestamp data Hibernate uses to verify validity of query result in this cache.
timestamps-cache- A configuration appropriate for the timestamp data cached as part of JPA/Hibernate query result caching. A replicated timestamp cache is required if query result caching is used, even if the query results themselves use a non-replicating cache like
local-query. mvcc-shared- A configuration appropriate for a cache that's shared for JPA/Hibernate entity, collection, query result and timestamp caching. Not an advised configuration, since it requires cache mode
REPL_SYNC, which is the least efficient mode. Also requires a full state transfer at startup, which can be expensive. Retained for backwards compatibility. Uses JBoss Cache's MVCC locking. optimistic-shared- A configuration appropriate for a cache that's shared for JPA/Hibernate entity, collection, query result and timestamp caching. Not an advised configuration, since it requires cache mode
REPL_SYNC, which is the least efficient mode. Also requires a full state transfer at startup, which can be expensive. Maintained for backwards compatibility reasons, as a shared cache was the only option in JBoss 4. Uses JBoss Cache's optimistic locking. pessimistic-shared- A configuration appropriate for a cache that's shared for JPA/Hibernate entity, collection, query result and timestamp caching. Not an advised configuration, since it requires cache mode
REPL_SYNC, which is the least efficient mode. Also requires a full state transfer at startup, which can be expensive. Maintained for backwards compatibility reasons, as a shared cache was the only option in JBoss 4. Uses JBoss Cache's pessimistic locking. mvcc-shared-repeatable- Same as
mvcc-sharedbut uses JBoss Cache'sREPEATABLE_READisolation level instead ofREAD_COMMITTED(see notes below). pessimistic-shared-repeatable- Same as
pessimistic-sharedbut uses JBoss Cache'sREPEATABLE_READisolation level instead ofREAD_COMMITTED(see notes below).
Note
For more on JBoss Cache's locking schemes, see Section 24.1.4, “Concurrent Access”)
Note
For JPA/Hibernate second level caching,
REPEATABLE_READ is only useful if the application evicts or clears entities from the Hibernate Entity Manager Session and then expects to repeatably re-read them in the same transaction. Otherwise, the session's internal cache provides a repeatable-read semantic.
The
CacheManager also supports aliasing of caches; that is, allowing caches registered under one name to be looked up under a different name. Aliasing is useful for sharing caches between services whose configuration may specify different cache configuration names.
Aliases can be configured by editing the
CacheManager bean in the jboss-cache-manager-jboss-beans.xml file. The following redacted configuration shows the standard aliases in Enterprise Web Platform 5:
<<<<<<<
<bean name="CacheManager" class="org.jboss.ha.cachemanager.CacheManager">
. . .
<!-- Aliases for cache names. Allows caches to be shared across
services that may expect different cache configuration names. -->
<property name="configAliases">
<map keyClass="java.lang.String" valueClass="java.lang.String">
<!-- Use the HAPartition cache for ClusteredSSO caching -->
<entry>
<key>clustered-sso</key>
<value>ha-partition</value>
</entry>
</map>
</property>
. . .
</bean>HAPartition is a general purpose service used for a variety of tasks in Enterprise Web Platform clustering. At its core, it is an abstraction built on top of a JGroups Channel that provides support for making and receiving RPC invocations from one or more cluster members. HAPartition allows services that use it to share a single Channel and multiplex RPC invocations over it, eliminating the configuration complexity and runtime overhead of having each service create its own Channel. HAPartition also supports a distributed registry of which clustering services are running on which cluster members. It provides notifications to interested listeners when the cluster membership changes or the clustered service registry changes. HAPartition forms the core of many of the clustering services we'll be discussing in the rest of this guide, including smart client-side clustered proxies, farming, HA-JNDI and HA singletons. Custom services can also make use of HAPartition.
The following snippet shows the
HAPartition service definition packaged with the standard JBoss Enterprise Web Platform distribution. This configuration can be found in the server/$PROFILE/deploy/cluster/hapartition-jboss-beans.xml file.
<bean name="HAPartitionCacheHandler" class="org.jboss.ha.framework.server.HAPartitionCacheHandlerImpl">
<property name="cacheManager"><inject bean="CacheManager"/></property>
<property name="cacheConfigName">ha-partition</property>
</bean>
<bean name="HAPartition" class="org.jboss.ha.framework.server.ClusterPartition">
<depends>jboss:service=Naming</depends>
<annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX
(name="jboss:service=HAPartition,partition=${jboss.partition.name:DefaultPartition}", exposedInterface=org.jboss.ha.framework.server.ClusterPartitionMBean.class, registerDirectly=true)</annotation>
<!-- ClusterPartition requires a Cache for state management -->
<property name="cacheHandler"><inject bean="HAPartitionCacheHandler"/></property>
<!-- Name of the partition being built -->
<property name="partitionName">${jboss.partition.name:DefaultPartition}</property>
<!-- The address used to determine the node name -->
<property name="nodeAddress">${jboss.bind.address}</property>
<!-- Max time (in ms) to wait for state transfer to complete. Increase for large states -->
<property name="stateTransferTimeout">30000</property>
<!-- Max time (in ms) to wait for RPC calls to complete. -->
<property name="methodCallTimeout">60000</property>
<!-- Optionally provide a thread source to allow async connect of our channel -->
<property name="threadPool"><inject bean="jboss.system:service=ThreadPool"/></property>
<property name="distributedStateImpl">
<bean name="DistributedState" class="org.jboss.ha.framework.server.DistributedStateImpl">
<annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX
(name="jboss:service=DistributedState,partitionName=${jboss.partition.name:DefaultPartition}", exposedInterface=org.jboss.ha.framework.server.DistributedStateImplMBean.class, registerDirectly=true)</annotation>
<property name="cacheHandler"><inject bean="HAPartitionCacheHandler"/></property>
</bean>
</property>
</bean>
Much of the above is generic; below we'll touch on the key points relevant to end users. There are two beans defined above, the
HAPartitionCacheHandler and the HAPartition itself.
The
HAPartition bean itself exposes the following configuration properties:
partitionName- An optional attribute to specify the name of the cluster. Its default value is
DefaultPartition. Use the-g(or--partition) command line switch to set this value at server startup.The partitionName property on the MCBean:ServerConfig Profile Service Component
If you use the partitionName property on the MCBean:ServerConfig Profile Service component, the system returns a null value for the property. Use the PartionName from the MCBean:HAPartition managed component to obtain the correct value. nodeAddress- This attribute is not used and can be ignored.
stateTransferTimeout- Specifies the timeout in milliseconds for initial application state transfer. State transfer refers to the process of obtaining a serialized copy of initial application state from other already-running cluster members at service startup. Its default value is
30000. methodCallTimeout- Specifies the timeout in milliseconds for obtaining responses to group remote procedure calls from other cluster members. Its default value is
60000.
The
HAPartitionCacheHandler is a small utility service that helps the HAPartition integrate with JBoss Cache (see Section 17.1.3.1, “The JBoss Enterprise Web Platform CacheManager Service”). HAPartition exposes a child service called DistributedState that uses JBoss Cache; the HAPartitionCacheHandler helps ensure consistent configuration between the JGroups Channel used by DistributedState's cache and the one used directly by HAPartition.
cacheConfigName- The name of the JBoss Cache configuration to use for the
HAPartition-related cache. Indirectly, this also specifies the name of the JGroups protocol stack configuration HAPartition should use. See Section 24.1.5, “JGroups Integration” for more on how the JGroups protocol stack is configured.
In order for nodes to form a cluster, they must have the exact same
partitionName and the HAPartitionCacheHandler's cacheConfigName must specify an identical JBoss Cache configuration. Changes in either element on some but not all nodes would prevent proper clustering behavior.
You can view the current cluster information by pointing your browser to the JMX console of any JBoss instance in the cluster (that is,
http://hostname:8080/jmx-console/) and then clicking on the jboss:service=HAPartition,partition=DefaultPartition MBean (change the MBean name to reflect your partition name if you use the -g startup switch). A list of IP addresses for the current cluster members is shown in the CurrentView field.
Note
While it is technically possible to put a JBoss server instance into multiple
HAPartitions at the same time, this practice is generally not recommended, as it increases management complexity.
The
DistributedReplicantManager (DRM) service is a component of the HAPartition service made available to HAPartition users via the HAPartition.getDistributedReplicantManager() method. Generally speaking, JBoss Enterprise Web Platform users will not directly make use of the DRM; we discuss it here as an aid to those who want a deeper understanding of how Enterprise Web Platform clustering internals work.
The DRM is a distributed registry that allows
HAPartition users to register objects under a given key, making the objects registered under that key by various cluster members available to callers. The DRM also provides a notification mechanism so that interested listeners can be notified when the contents of the registry changes.
There are two main uses for the DRM in JBoss Enterprise Web Platform:
Clustered Smart Proxies- Here the keys are the names of the various services that need a clustered smart proxy (see Section 16.2.1, “Client-side interceptor architecture”. The value object each node stores in the DRM is known as a target. This is used by the smart proxy's transport layer to contact the node (for example, an RMI stub, an HTTP URL or a JBoss Remoting
InvokerLocator). The factory that builds clustered smart proxies accesses the DRM to get the set of targets that should be injected into the proxy to allow it to communicate with all the nodes in a cluster. HASingleton- Here the keys are the names of the various services that need to function as High Availablity Singletons. The value object each node stores in the DRM is simply a string that acts as a token to indicate that the node has the service deployed, and thus is a candidate to become the "master" node for the
HASingletonservice.
In both cases, the key under which objects are registered identifies a particular clustered service. It is useful to understand that every node in a cluster need not register an object under every key. Only services that are deployed on a particular node will register something under that service's key, and services don't have to be deployed homogeneously across the cluster. The DRM is thus useful as a mechanism for understanding a service's "topology" around the cluster — which nodes have the service deployed.
Custom services can also use make use of
HAPartition to handle interactions with the cluster. Generally the easiest way to do this is to extend the org.jboss.ha.framework.server.HAServiceImpl base class, or the org.jboss.ha.jxm.HAServiceMBeanSupport class if JMX registration and notification support are desired.