HTTP session replication is used to replicate the state associated with web client sessions to other nodes in a cluster. Thus, in the event one of your nodes crashes, another node in the cluster will be able to recover. Two distinct functions must be performed:
- Session state replication
- Load-balancing HTTP Requests
State replication is directly handled by JBoss. When you run JBoss in the
production configuration, session state replication is enabled by default. Just configure your web application as <distributable> in its web.xml (see Section 21.2, “Configuring HTTP session state replication”), deploy it, and its session state is automatically replicated across all JBoss instances in the cluster.
However, load-balancing is a different story; it is not handled by JBoss itself and requires an external load balancer. This function could be provided by specialized hardware switches or routers (Cisco LoadDirector for example) or by specialized software running on commodity hardware. As a very common scenario, we will demonstrate how to set up a software load balancer using Apache httpd and mod_jk.
Note
A load-balancer tracks HTTP requests and, depending on the session to which the request is linked, it dispatches the request to the appropriate node. This is called load-balancing with sticky-sessions or session affinity: once a session is created on a node, every future request will also be processed by that same node. Using a load-balancer that supports sticky-sessions but not configuring your web application for session replication allows you to scale very well by avoiding the cost of session state replication: each request for a session will always be handled by the same node. But in case a node dies, the state of all client sessions hosted by this node (the shopping carts, for example) will be lost and the clients will most probably need to login on another node and restart with a new session. In many situations, it is acceptable not to replicate HTTP sessions because all critical state is stored in a database or on the client. In other situations, losing a client session is not acceptable and, in this case, session state replication is the price one has to pay.
Apache is a well-known web server which can be extended by plugging in modules. One of these modules, mod_jk has been specifically designed to allow the forwarding of requests from Apache to a Servlet container. Furthermore, it is also able to load-balance HTTP calls to a set of Servlet containers while maintaining sticky sessions, which is what is most interesting for us in this section.
First of all, make sure that you have Apache installed. You can download Apache directly from the Apache web site at http://httpd.apache.org/. Its installation is straightforward and requires no specific configuration. We advise you to use the latest stable version of Apache, which is version 2.2.x. We will assume, for the next sections, that you have installed Apache in the
$APACHE_HOME directory.
Next, download mod_jk binaries. Several versions of mod_jk exist as well. We strongly advise the use of mod_jk 1.2.x, as both earlier versions of mod_jk, and mod_jk2, are deprecated, unsupported and no further development is going on in the community. The mod_jk 1.2.x binary can be downloaded from http://www.apache.org/dist/jakarta/tomcat-connectors/jk/binaries/. Rename the downloaded file to
mod_jk.so and copy it under $APACHE_HOME/modules/.
Modify
$APACHE_HOME/conf/httpd.conf and add a single line at the end of the file:
# Include mod_jk's specific configuration file Include conf/mod-jk.conf
Next, create a new file named
$APACHE_HOME/conf/mod-jk.conf:
# Load mod_jk module
# Specify the filename of the mod_jk lib
LoadModule jk_module modules/mod_jk.so
# Where to find workers.properties
JkWorkersFile conf/workers.properties
# Where to put jk logs
JkLogFile logs/mod_jk.log
# Set the jk log level [debug/error/info]
JkLogLevel info
# Select the log format
JkLogStampFormat "[%a %b %d %H:%M:%S %Y]"
# JkOptions indicates to send SSK KEY SIZE
JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
# JkRequestLogFormat
JkRequestLogFormat "%w %V %T"
# Mount your applications
JkMount /application/* loadbalancer
# You can use external file for mount points.
# It will be checked for updates each 60 seconds.
# The format of the file is: /url=worker
# /examples/*=loadbalancer
JkMountFile conf/uriworkermap.properties
# Add shared memory.
# This directive is present with 1.2.10 and
# later versions of mod_jk, and is needed for
# for load balancing to work properly
JkShmFile logs/jk.shm
# Add jkstatus for managing runtime data
<Location /jkstatus/>
JkMount status
Order deny,allow
Deny from all
Allow from 127.0.0.1
</Location>
Please note that two settings are very important:
- The
LoadModuledirective must reference the mod_jk library you downloaded and placed in$APACHE_HOME/modules/. You must indicate the same name, including themodulesfile path prefix. - The
JkMountdirective tells Apache which URLs it should forward to the mod_jk module (and, in turn, to the Servlet containers). In the above file, all requests with URL path/application/*are sent to the mod_jk load-balancer. This way, you can configure Apache to serve static contents (or PHP contents) directly and only use the load balancer for Java applications. If you only use mod_jk as a load balancer, you can also forward all URLs (that is,/*) to mod_jk.
In addition to the
JkMount directive, you can also use the JkMountFile directive to specify a mount points configuration file, which contains multiple Tomcat forwarding URL mappings. You just need to create a uriworkermap.properties file in the $APACHE_HOME/conf directory. The format of the file is /url=worker_name. To get things started, paste the following example code into the file you created:
# Simple worker configuration file # Mount the Servlet context to the ajp13 worker /jmx-console=loadbalancer /jmx-console/*=loadbalancer /web-console=loadbalancer /web-console/*=loadbalancer
This will configure mod_jk to forward requests made to
/jmx-console and /web-console to Tomcat.
You will most probably not change the other settings in
mod_jk.conf. They are used to tell mod_jk where to put its logging file, which logging level to use and so on.
Next, you need to configure the mod_jk file
conf/workers.properties file. This file specifies where the different Servlet containers are located and how calls should be load-balanced across them. The configuration file contains one section for each target servlet container and one global section. For a two node setup, the file might look like this:
# Define list of workers that will be used # for mapping requests worker.list=loadbalancer,status # Define Node1 # modify the host as your host IP or DNS name. worker.node1.port=8009 worker.node1.host=node1.mydomain.com worker.node1.type=ajp13 worker.node1.lbfactor=1 worker.node1.cachesize=10 # Define Node2 # modify the host as your host IP or DNS name. worker.node2.port=8009 worker.node2.host=node2.mydomain.com worker.node2.type=ajp13 worker.node2.lbfactor=1 worker.node2.cachesize=10 # Load-balancing behaviour worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=node1,node2 worker.loadbalancer.sticky_session=1 #worker.list=loadbalancer # Status worker for managing load balancer worker.status.type=status
The above file configures mod_jk to perform weighted round-robin load balancing with sticky sessions between two servlet containers (JBoss Enterprise Web Platform instances)
node1 and node2 listening on port 8009.
In the
workers.properties file, each node is defined using the worker.XXX naming convention where XXX represents an arbitrary name you choose for each of the target servlet containers. For each worker, you must specify the host name (or IP address) and the port number of the AJP13 connector running in the Servlet container.
The
lbfactor attribute is the load-balancing factor for this specific worker. It is used to define the priority (or weight) a node should have over other nodes. The higher this number is for a given worker relative to the other workers, the more HTTP requests the worker will receive. This setting can be used to differentiate servers with different processing power.
The
cachesize attribute defines the size of the thread pools associated to the servlet container (that is, the number of concurrent requests it will forward to the Servlet container). Make sure this number is not greater than the number of threads configured on the AJP13 connector of the Servlet container. See >http://tomcat.apache.org/connectors-doc/reference/workers.html for comments on cachesize for Apache 1.3.x.
The last part of the
conf/workers.properties file defines the load balancer worker. The only thing you must change is the worker.loadbalancer.balanced_workers line: it must list all workers previously defined in the same file. Load balancing will happen over these workers.
The
sticky_session property specifies the cluster behavior for HTTP sessions. If you specify worker.loadbalancer.sticky_session=0, each request will be load balanced between node1 and node2; that is, different requests for the same session will go to different servers. But when a user opens a session on one server, it is always necessary to always forward this user's requests to the same server, as long as that server is available. This is called a sticky session, as the client is always using the same server he reached on his first request. To enable session stickiness, you need to set worker.loadbalancer.sticky_session to 1.
Note
A non-loadbalanced setup with a single node requires a
worker.list=node1 entry.
Finally, we must configure the JBoss Enterprise Web Platform instances on all clustered nodes so that they can expect requests forwarded from the mod_jk load balancer.
On each clustered JBoss node, we have to name the node according to the name specified in
workers.properties. For instance, on JBoss instance node1, edit the JBOSS_HOME/server/$PROFILE/deploy/jbossweb.sar/server.xml file. Locate the <Engine> element and add an attribute jvmRoute:
<Engine name="jboss.web" defaultHost="localhost" jvmRoute="node1"> </Engine>
You also need to ensure that the AJP connector in
server.xml is enabled (that is, uncommented). It is enabled by default.
<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector protocol="AJP/1.3" port="8009" address="${jboss.bind.address}"
redirectPort="8443" />
At this point, you have a fully working Apache with mod_jk load balancer setup that will balance call to the Servlet containers of your cluster while taking care of session stickiness (clients will always use the same Servlet container).
Note
For more updated information on using mod_jk 1.2 with JBoss Application Server, refer to the JBoss wiki page at http://www.jboss.org/community/wiki/UsingModjk12WithJBoss.
This section shows you how to configure the NSAPI connector to use a JBoss Enterprise Platform as a worker node for a Sun Java System Web Server (SJWS) master node.
Note
Sun Java System Web Server has recently been renamed to the Oracle iPlanet Web Server.
In this section, all of the server instances are on the same machine. To use different machines for each instance, use the
-b switch to bind your instance of JBoss Enterprise Platform to a public IP address. Remember to edit the workers.properties file on the SJWS machine to reflect these changes in IP address.
This section assumes that:
- Your worker node(s) are already installed with a JBoss Enterprise Platform 5.1 or later. The Native components are optional. Refer to the Installation Guide for assistance with this prerequisite.
- Your master node is already installed with any of the following technology combinations, and the appropriate Native binary for its operating system and architecture. Refer to the Installation Guide for assistance with this prerequisite.
- Solaris 9 x86 with Sun Java System Web Server 6.1 SP12
- Solaris 9 SPARC 64 with Sun Java System Web Server 6.1 SP12
- Solaris 10 x86 with Sun Java System Web Server 7.0 U8
- Solaris 10 SPARC 64 with Sun Java System Web Server 7.0 U8
This section shows you how to safely configure your JBoss Enterprise Platform instance as a worker node for use with Sun SJWS.
Procedure 21.1. Configure a JBoss Enterprise Platform instance as a worker node
Create a server profile for each worker node
Make a copy of the server profile that you wish to configure as a worker node. (This procedure uses thedefaultserver profile.)[user@workstation jboss-eap-5.1]$ cd jboss-as/server [user@workstation server]$ cp -r default/ default-01 [user@workstation server]$ cp -r default/ default-02
Give each instance a unique name
Edit the following line in thedeploy/jbossweb.sar/server.xmlfile of each new worker instance:<Engine name="jboss.web" defaultHost="localhost">
Add a uniquejvmRoutevalue, as shown. This value is the identifier for this node in the cluster.For thedefault-01server profile:<Engine name="jboss.web" defaultHost="localhost" jvmRoute="worker01">
For thedefault-02server profile:<Engine name="jboss.web" defaultHost="localhost" jvmRoute="worker02">
Enable session handling
Edit the following line in thedeployers/jbossweb.deployer/META-INF/war-deployers-jboss-beans.xmlfile of each worker node:<property name="useJK">false</property>
This property controls whether special session handling is used to coordinate with mod_jk and other connector variants. Set this property totruein both worker nodes:<property name="useJK">true</property>
Start your worker nodes
Start each worker node in a separate command line interface. Ensure that each node is bound to a different IP address with the-bswitch.[user@workstation jboss-eap-5.1]$ ./jboss-as/bin/run.sh -b 127.0.0.1 -c default-01
[user@workstation jboss-eap-5.1]$ ./jboss-as/bin/run.sh -b 127.0.0.100 -c default-02
The procedures in the following sections assume that the contents of the Native zip appropriate for your operating system and architecture have been extracted to
/tmp/connectors/jboss-ep-native-5.1/. This path is referred to as NATIVE in the procedures that follow. These procedures also assume that the /tmp/connectors directory is used to store logs, properties files and NSAPI locks.
These procedures also assume that your installation of Sun Java System Web Server is in one of the following locations, depending on your version of Solaris:
- for Solaris 9 x86 or SPARC 64:
/opt/SUNWwbsrv61/ - for Solaris 10 x86 or SPARC 64:
/opt/SUNWwbsrv70/
This path is referred to as
SJWS in the procedures that follow.
Procedure 21.2. Initial clustering configuration
Disable servlet mappings
Under Built In Servlet Mappings in thefile, disable the mappings for the following servlets, as shown in the code sample:SJWS/PROFILE/config/default-web.xml- default
- invoker
- jsp
<!-- ==================== Built In Servlet Mappings ===================== --> <!-- The servlet mappings for the built in servlets defined above. --> <!-- The mapping for the default servlet --> <!--servlet-mapping> <servlet-name>default</servlet-name> <url-pattern>/</url-pattern> </servlet-mapping--> <!-- The mapping for the invoker servlet --> <!--servlet-mapping> <servlet-name>invoker</servlet-name> <url-pattern>/servlet/*</url-pattern> </servlet-mapping--> <!-- The mapping for the JSP servlet --> <!--servlet-mapping> <servlet-name>jsp</servlet-name> <url-pattern>*.jsp</url-pattern> </servlet-mapping-->
Load the required modules and properties
Append the following lines to thefile:SJWS/PROFILE/config/magnus.confInit fn="load-modules" funcs="jk_init,jk_service" shlib="
NATIVE/lib/nsapi_redirector.so" shlib_flags="(global|now)" Init fn="jk_init" worker_file="/tmp/connectors/workers.properties" log_level="debug" log_file="/tmp/connectors/nsapi.log" shm_file="/tmp/connectors/jk_shm"These lines define the location of thensapi_redirector.somodule used by thejk_initandjk_servicefunctions, and the location of theworkers.propertiesfile, which defines the worker nodes and their attributes.Note
Thelibdirectory in thepath applies only to 32-bit machines. On 64-bit machines, this directory is calledNATIVE/lib/nsapi_redirector.solib64.
Use the following procedure to configure a basic cluster, where requests for particular paths are forwarded to particular worker nodes. In Procedure 21.3, “Configure a basic cluster with NSAPI”, worker02 serves the
/nc path, while worker01 serves /status and all other paths defined in the first part of the obj.conf file.
Procedure 21.3. Configure a basic cluster with NSAPI
Define the paths to serve via NSAPI
Edit thefile. Define paths that should be served via NSAPI at the end of theSJWS/PROFILE/config/obj.confdefaultObject definition, as shown:<Object name="default"> [...] NameTrans fn="assign-name" from="/status" name="jknsapi" NameTrans fn="assign-name" from="/images(|/*)" name="jknsapi" NameTrans fn="assign-name" from="/css(|/*)" name="jknsapi" NameTrans fn="assign-name" from="/nc(|/*)" name="jknsapi" NameTrans fn="assign-name" from="/jmx-console(|/*)" name="jknsapi" </Object>
You can map the path of any application deployed on your JBoss Enterprise Platform instance in thisobj.conffile. In the example code, the/ncpath is mapped to an application deployed under the namenc.Define the worker that serves each path
Edit thefile and add the followingSJWS/PROFILE/config/obj.confjknsapiObject definition after thedefaultObject definition.<Object name="jknsapi"> ObjectType fn=force-type type=text/plain Service fn="jk_service" worker="worker01" path="/status" Service fn="jk_service" worker="worker02" path="/nc(/*)" Service fn="jk_service" worker="worker01" </Object>
ThisjknsapiObject defines the worker nodes used to serve each path that was assigned toname="jknsapi"in thedefaultObject.In the example code, the third Service definition does not specify apathvalue, so the worker node defined (worker01) serves all of the paths assigned tojknsapiby default. In this case, the first Service definition in the example code, which assigns the/statuspath toworker01, is superfluous.Define the workers and their attributes
Create aworkers.propertiesfile in the location you defined in Step 2. Define the list of worker nodes and each worker node's properties in this file, like so:# An entry that lists all the workers defined worker.list=worker01, worker02 # Entries that define the host and port associated with these workers worker.worker01.host=127.0.0.1 worker.worker01.port=8009 worker.worker01.type=ajp13 worker.worker02.host=127.0.0.100 worker.worker02.port=8009 worker.worker02.type=ajp13
Procedure 21.4. Configure a load-balancing cluster with NSAPI
Define the paths to serve via NSAPI
Edit thefile. Define paths that should be served via NSAPI at the end of theSJWS/PROFILE/config/obj.confdefaultObject definition, as shown:<Object name="default"> [...] NameTrans fn="assign-name" from="/status" name="jknsapi" NameTrans fn="assign-name" from="/images(|/*)" name="jknsapi" NameTrans fn="assign-name" from="/css(|/*)" name="jknsapi" NameTrans fn="assign-name" from="/nc(|/*)" name="jknsapi" NameTrans fn="assign-name" from="/jmx-console(|/*)" name="jknsapi" NameTrans fn="assign-name" from="/jkmanager/*" name="jknsapi" </Object>
You can map the path of any application deployed on your JBoss Enterprise Platform instance in thisobj.conffile. In the example code, the/ncpath is mapped to an application deployed under the namenc.Define the worker that serves each path
Edit thefile and add the followingSJWS/PROFILE/config/obj.confjknsapiObject definition after thedefaultObject definition.<Object name="jknsapi"> ObjectType fn=force-type type=text/plain Service fn="jk_service" worker="status" path="/jkmanager(/*)" Service fn="jk_service" worker="router" </Object>
ThisjknsapiObject defines the worker nodes used to serve each path that was assigned toname="jknsapi"in thedefaultObject.Define the workers and their attributes
Create aworkers.propertiesfile in the location you defined in Step 2. Define the list of worker nodes and each worker node's properties in this file, like so:# The advanced router LB worker worker.list=router,status # Define a worker using ajp13 worker.worker01.port=8009 worker.worker01.host=127.0.0.1 worker.worker01.type=ajp13 worker.worker01.ping_mode=A worker.worker01.socket_timeout=10 worker.worker01.lbfactor=3 # Define another worker using ajp13 worker.worker02.port=8009 worker.worker02.host=127.0.0.100 worker.worker02.type=ajp13 worker.worker02.ping_mode=A worker.worker02.socket_timeout=10 worker.worker02.lbfactor=1 # Define the LB worker worker.router.type=lb worker.router.balance_workers=worker01,worker02 # Define the status worker worker.status.type=status
Once your Sun Java System Web Server instance is configured, restart it so that your changes take effect.
For Sun Java System Web Server 6.1:
SJWS/PROFILE/stopSJWS/PROFILE/start
For Sun Java System Web Server 7.0:
SJWS/PROFILE/bin/stopservSJWS/PROFILE/bin/startserv