Chapter 5. Clustering

When clustering Red Hat JBoss BRMS, consider which components need to be clustered. You can cluster the following:
  • GIT repository: virtual-file-system (VFS) repository that holds the business assets so that all cluster nodes use the same repository
  • Web applications: The web applications need to be clustered so that nodes share the same runtime data.
    For instructions on clustering the application, refer to the container clustering documentation.

GIT Repository Clustering Mechanism

To cluster the GIT repository the following is used:
  • Apache Zookeeper brings all parts together.
  • Apache Helix is the cluster management component that registers all cluster details (the cluster itself, nodes, resources).
  • uberfire framework which provides the backbone of the web applications
A typical clustering setup involves the following:
  • Setting up the cluster itself using Zookeeper and Helix
  • Configuring clustering on your container (this documentation provides only clustering instructions for Red Hat JBoss EAP 6)

Clustering Maven Repositories

Various operations within the Business Central publish JARs to the Business Central's internal Maven Repository.
This repository exists on the application server's file-system as regular files and is not cluster aware. This folder is not synchronized across the various nodes in the cluster and must be synchronized using external tools like rsync.
An alternate to the use of an external synchronization tool is to set the system property org.guvnor.m2repo.dir on each cluster node to point to a SAN or NAS. In this case clustering of the Maven repository folder is not needed.

5.1. Clustering on JBoss EAP

To install JBoss BRMS in clustered mode, we recommend you use the JAR installer, which provides a sample setup that works out of the box. You can, however, set up clustering with the deployable ZIP for EAP as well.

5.1.1. Clustering using JAR Installer

Note

The JAR installer provides sample setup only, adjusting the configuration is necessary for it to suit your project's needs.
Using the JAR installer , you can set up a basic clustering configuration of JBoss BRMS.
The automatic configuration creates three ZooKeeper instances, a Helix cluster that uses these instances, and two Quartz datastores (one managed and one unmanaged). This JBoss BRMS setup consists of two EAP nodes that share a Maven repository, use Quartz for coordinating timed tasks, and have business-central.war, dashbuilder.war, and kie-server.war deployed. To customize the setup to fit your scenario, or to use clustering with the deployable ZIP, see the Section 5.1.2, “Custom Configuration (Deployable ZIP)”. You can also get more information in the JBoss EAP documentation.
Follow the installation process described in Section 1.3, “Installing Red Hat JBoss BRMS Using the Installer” and select Install clustered configuration in Advanced Runtime Configuration. After clicking next, you will go through the following steps:

Note

The steps listed here describe the GUI installation. The steps for the console installation are analogous.

  1. Select JDBC provider

    On this screen, select the JDBC provider from the list. You need to provide the corresponding JDBC driver JAR(s) in one of these ways:
    • Select one or more files on the filesystem
    • Provide one or more URLs. The installer downloads the files automatically.
    The installer then copies the JAR(s) into the appropriate location under the directory $EAP_HOME/modules, where a corresponding module.xml file is also created automatically.

  2. Configure Quartz connection

    On the next screen, provide the data to the database for Quartz. The installer automatically creates the Quartz definition file ($EAP_HOME/domain/configuration/quartz-definition.properties) and two Quartz datasources in the domain configuration file $EAP_HOME/domain/domain.xml. You may edit the files after having finished the installation.

    Note

    During the installation, Quartz DDL scripts will be run on the database selected in this step. These scripts make changes needed for Quartz to operate (adding tables, etc.), and can be found in $EAP_HOME/jboss-brms-bpmsuite-6.2-supplementary-tools/ddl-scripts for reference (You do not need to modify them in any way).
  3. Click next to initiate the installation.

    Important

    When using the JAR installer, the war archives are automatically created from the applications residing in $EAP_HOME/standalone/deployments/. That means additional space is necessary as the applications exist both in uncompressed and compressed state in the storage during the installation.
    Three ZooKeeper instances are automatically created in $EAP_HOME/jboss-brms-bpmsuite-6.2-supplementary-tools/ (directory names zookeeper-one, zookeeper-two, and zookeeper-three).
    In the directory $EAP_HOME/jboss-brms-bpmsuite-6.2-supplementary-tools/helix-core, you can find the default Helix configuration and the scripts to launch the cluster—startCluster.sh for UNIX and startCluster.bat for Windows.
    After the installation finishes, DO NOT select to run the server immediately. You first need to start the cluster by moving to the directory $EAP_HOME/jboss-brms-bpmsuite-6.2-supplementary-tools/helix-core and executing the aforementioned launch script:
    On UNIX systems: 
    ./startCluster.sh
    On Windows: 
    ./startCluster.bat
    This script launches the Helix cluster and the ZooKeeper instances. Only after that, start the EAP server in domain mode by moving to the directory $EAP_HOME/bin and running:
    On UNIX systems: 
    ./domain.sh
    On Windows: 
    ./domain.bat
You now have a fully functioning JBoss BRMS cluster.

5.1.2. Custom Configuration (Deployable ZIP)

When using JBoss EAP clustering, a single JBoss EAP domain controller exists with other JBoss EAP slaves connecting to it as management users. Deployment of Business Central and dashbuilder can be done as a management user on a domain controller, and the WAR deployments will be distributed to other members of the JBoss EAP cluster.
To configure clustering on Red Hat JBoss EAP 6, do the following:
  1. Configure ZooKeeper and Helix according to Section 5.2.1, “Setting up a Cluster”.
  2. Configure individual server nodes in the main-server-group element in the $EAP_HOME/domain/configuration/host.xml file with properties defined in Table 5.1, “Cluster node properties”:
    Note that a when configuring a JBoss EAP cluster with Zookeeper, a different number of JBoss EAP nodes than Zookeeper nodes is possible (keeping in mind that Zookeeper should to have an odd number of nodes). However, having the same node count for both Zookeeper and JBoss EAP is considered best practice.

    Table 5.1. Cluster node properties

    Property name Value Description
    org.uberfire.nio.git.dir /home/jbrm/node[N]/repo
    GIT (VFS) repository location on node[N]
    jboss.node.name nodeOne
    node name unique within the cluster
    org.uberfire.cluster.id brms-cluster
    Helix cluster name
    org.uberfire.cluster.zk server1:2181
    Zookeeper location
    org.uberfire.cluster.local.id nodeOne_12345
    unique ID of the Helix cluster node
    Note that : is replaced with _.
    org.uberfire.cluster.vfs.lock vfs-repo
    name of the resource defined on the Helix cluster
    org.uberfire.nio.git.daemon.port 9418
    port used by the VFS repo to accept client connections
    The port must be unique for each cluster member.
    org.uberfire.metadata.index.dir /home/jbrm/node[N]/index
    location where the index for search is to be created (maintained by Apache Lucene)
    org.uberfire.nio.git.ssh.port 8003 the unique port number for ssh access to the GIT repo for a cluster running on physical machines.
    org.uberfire.nio.git.daemon.host nodeOne the name of the daemon host machine in a physical cluster.
    org.uberfire.nio.git.ssh.host nodeOne the name of the SSH host machine in a physical cluster.
    org.uberfire.nio.git.ssh.hostport and org.uberfire.nio.git.daemon.hostport 8003 and 9418 In a virtualized environment, the outside port to be used.

    Example 5.1. Cluster nodeOne configuration

    <system-properties>

	  <property name="org.uberfire.nio.git.dir" value="/tmp/brms/nodeone" boot-time="false"/>
	  <property name="jboss.node.name" value="nodeOne" boot-time="false"/>
	  <property name="org.uberfire.cluster.id" value="brms-cluster" boot-time="false"/>
	  <property name="org.uberfire.cluster.zk" value="server1:2181,server2:2181,server3:2181" boot-time="false"/>
	  <property name="org.uberfire.cluster.local.id" value="nodeOne_12345" boot-time="false"/>
	  <property name="org.uberfire.cluster.vfs.lock" value="vfs-repo" boot-time="false"/>

	  <property name="org.uberfire.nio.git.daemon.port" value="9418" boot-time="false"/>
	  <property name="org.uberfire.metadata.index.dir" value="/tmp/jbrm/nodeone" boot-time="false"/>
	  <property name="org.uberfire.nio.git.ssh.cert.dir" value="/tmp/jbpm/nodeone" boot-time="false"/>

	  <property name="org.uberfire.nio.git.ssh.port" value="8003" boot-time="false"/>
	  <property name="org.uberfire.nio.git.daemon.host" value="nodeOne" />
	  <property name="org.uberfire.nio.git.ssh.host" value="nodeOne" />
	  <property name="org.uberfire.nio.git.ssh.hostport" value="8003" boot-time="false"/>
	  <property name="org.uberfire.nio.git.daemon.hostport" value="9418" boot-time="false"/>



	</system-properties>

    Example 5.2. Cluster nodeTwo configuration

    <system-properties>

	  <property name="org.uberfire.nio.git.dir" value="/tmp/brms/nodetwo" boot-time="false"/>
	  <property name="jboss.node.name" value="nodeTwo" boot-time="false"/>
	  <property name="org.uberfire.cluster.id" value="brms-cluster" boot-time="false"/>
	  <property name="org.uberfire.cluster.zk" value="server1:2181,server2:2181,server3:2181" boot-time="false"/>
	  <property name="org.uberfire.cluster.local.id" value="nodeTwo_12346" boot-time="false"/>
	  <property name="org.uberfire.cluster.vfs.lock" value="vfs-repo" boot-time="false"/>

	  <property name="org.uberfire.nio.git.daemon.port" value="9418" boot-time="false"/>
	  <property name="org.uberfire.metadata.index.dir" value="/tmp/jbrm/nodetwo" boot-time="false"/>
	  <property name="org.uberfire.nio.git.ssh.cert.dir" value="/tmp/jbpm/nodetwo" boot-time="false"/>

	  <property name="org.uberfire.nio.git.ssh.port" value="8003" boot-time="false"/>
	  <property name="org.uberfire.nio.git.daemon.host" value="nodeTwo" />
	  <property name="org.uberfire.nio.git.ssh.host" value="nodeTwo" />
	  <property name="org.uberfire.nio.git.ssh.hostport" value="8003" boot-time="false"/>
	  <property name="org.uberfire.nio.git.daemon.hostport" value="9418" boot-time="false"/>


	</system-properties>

    Example 5.3. Cluster nodeThree configuration

    <system-properties>

	  <property name="org.uberfire.nio.git.dir" value="/tmp/brms/nodethree" boot-time="false"/>
	  <property name="jboss.node.name" value="nodeThree" boot-time="false"/>
	  <property name="org.uberfire.cluster.id" value="brms-cluster" boot-time="false"/>
	  <property name="org.uberfire.cluster.zk" value="server1:2181,server2:2181,server3:2181" boot-time="false"/>
	  <property name="org.uberfire.cluster.local.id" value="nodeThree_12347" boot-time="false"/>
	  <property name="org.uberfire.cluster.vfs.lock" value="vfs-repo" boot-time="false"/>

	  <property name="org.uberfire.nio.git.daemon.port" value="9418" boot-time="false"/>
	  <property name="org.uberfire.metadata.index.dir" value="/tmp/jbrm/nodethree" boot-time="false"/>
	  <property name="org.uberfire.nio.git.ssh.cert.dir" value="/tmp/jbpm/nodethree" boot-time="false"/>

	  <property name="org.uberfire.nio.git.ssh.port" value="8003" boot-time="false"/>
	  <property name="org.uberfire.nio.git.daemon.host" value="nodeThree" />
	  <property name="org.uberfire.nio.git.ssh.host" value="nodeThree" />
	  <property name="org.uberfire.nio.git.ssh.hostport" value="8003" boot-time="false"/>
	  <property name="org.uberfire.nio.git.daemon.hostport" value="9418" boot-time="false"/>


	</system-properties>
  3. Add management users as instructed in the Administration and Configuration Guide for Red Hat JBoss EAP and application users as instructed in Red Hat JBoss BRMS Administration and Configuration Guide.
  4. Move to the directory $EAP_HOME/bin and start the application server in domain mode:
    On UNIX systems: 
    ./domain.sh
    On Windows: 
    ./domain.bat
  5. Check that the nodes are available.
Deploy the Business Central application to your servers:
  • Log on as the management user to the server Administration console of your domain and add the new deployments using the Runtime view of the console. Once the deployment is added to the domain, assign it to the correct server group (main-server-group).

Note

It is important users explicitly check deployment unit readiness with every cluster member.
When a deployment unit is created on a cluster node, it takes some time before it is distributed among all cluster members. Deployment status can be checked via UI and REST, however if the query goes to the node where the deployment was originally issued, the answer is deployed. Any request targeting this deployment unit sent to a different cluster member fails with DeploymentNotFoundException.