How to configure Zookeeper with multiple JBoss BPMS/BRMS 6 instances?

Resolution

To configure more than one Helix Cluster using the same Zookeeper installation please follow the steps which are described below.

First let's understand how the test environment are distributed:

• In this test, everything is running in the same machine:

  • Helix Controller
  • zookeeper1
  • zookeeper2
  • zookeeper3

• There is a JBoss EAP installation running domain mode with two server groups:

  • brms
    • server-one
    • server-two
  • bpms
    • server-three
    • server-four

Before applying the steps below, please make sure to have a previous cluster setup working.

The steps below describes how to create a new cluster:

• Start all zookeeper instances.
• Creating the new cluster instance:

 ./helix-admin.sh --zkSvr localhost:2181 --addCluster bpms-cluster2

• Register the new cluster's members:

 ./helix-admin.sh --zkSvr localhost:2181 --addNode bpms-cluster2 nodeThree:12347
 ./helix-admin.sh --zkSvr localhost:2181 --addNode bpms-cluster2 nodeFour:12348

• Create the new cluster's vfs-repo:

 ./helix-admin.sh --zkSvr localhost:2181 --addResource bpms-cluster2 vfs-repo1 1 LeaderStandby AUTO_REBALANCE

• Add the created new members to the vfs-repo (2 are the number of members of the cluster):

 ./helix-admin.sh --zkSvr localhost:2181 --rebalance bpms-cluster2 vfs-repo1 2

At this point the new cluster is configured, you can check it and compare the two cluster with the following commands:

• Verifying existent cluster:

./helix-admin.sh --zkSvr localhost:2181 --listClusterInfo bpms-cluster
Existing resources in cluster bpms-cluster:
vfs-repo
Instances in cluster bpms-cluster:
nodeTwo_12346
nodeOne_12345

• New cluster

./helix-admin.sh --zkSvr localhost:2181 --listClusterInfo bpms-cluster2
Existing resources in cluster bpms-cluster2:
vfs-repo1
Instances in cluster bpms-cluster2:
nodeThree_12347
nodeFour_12348

• Verifying the new cluster's members:

./helix-admin.sh --zkSvr localhost:2181 --listInstanceInfo bpms-cluster2 nodeThree_12347
InstanceConfig: {
  "id" : "nodeThree_12347",
  "mapFields" : {
  },
  "listFields" : {
    "HELIX_DISABLED_PARTITION" : [ "vfs-repo1_0" ]
  },
  "simpleFields" : {
    "HELIX_ENABLED" : "true",
    "HELIX_HOST" : "nodeThree",
    "HELIX_PORT" : "12347"
  }
}

./helix-admin.sh --zkSvr localhost:2181 --listInstanceInfo bpms-cluster2 nodeFour_12348
InstanceConfig: {
  "id" : "nodeFour_12348",
  "mapFields" : {
  },
  "listFields" : {
    "HELIX_DISABLED_PARTITION" : [ "vfs-repo1_0" ]
  },
  "simpleFields" : {
    "HELIX_ENABLED" : "true",
    "HELIX_HOST" : "nodeFour",
    "HELIX_PORT" : "12348"
  }
}

• Verifying new cluster information:

./helix-admin.sh --zkSvr localhost:2181 --listResourceInfo bpms-cluster2 vfs-repo1
IdealState for vfs-repo1:
{
  "id" : "vfs-repo1",
  "mapFields" : {
    "vfs-repo1_0" : {
      "nodeFour_12348" : "LEADER",
      "nodeThree_12347" : "STANDBY"
    }
  },
  "listFields" : {
    "vfs-repo1_0" : [ "nodeFour_12348", "nodeThree_12347" ]
  },
  "simpleFields" : {
    "IDEAL_STATE_MODE" : "AUTO",
    "NUM_PARTITIONS" : "1",
    "REBALANCE_MODE" : "SEMI_AUTO",
    "REPLICAS" : "2",
    "STATE_MODEL_DEF_REF" : "LeaderStandby",
    "STATE_MODEL_FACTORY_NAME" : "DEFAULT"
  }
}

• External View for vfs-repo1:

{
  "id" : "vfs-repo1",
  "mapFields" : {
  },
  "listFields" : {
  },
  "simpleFields" : {
    "BUCKET_SIZE" : "0"
  }
}

• Starting all clusters:
• Existent cluster:

./run-helix-controller.sh --zkSvr localhost:2181,localhost:2182,localhost:2183 --cluster bpms-cluster

• New cluster:

./run-helix-controller.sh --zkSvr localhost:2181,localhost:2182,localhost:2183 --cluster bpms-cluster2

At this point we already have two clusters running in the same Zookeeper installation, now we have to configure the JBoss EAP to use these clusters.
See below the EAP's configuration:

• Cluster 1 configuration: Server-group: brms (it will use the bpms-cluster and the vfs-repo), domain.xml file:
  • Servers configurations:

        <server name="server-one" group="brms">
            <system-properties>
                <property name="jboss.node.name" value="nodeOne" boot-time="false"/>
                <property name="org.uberfire.cluster.id" value="bpms-cluster" boot-time="false"/>
                <property name="org.uberfire.cluster.zk" value="localhost:2181,localhost:2182,localhost:2183" boot-time="false"/>
                <property name="org.uberfire.cluster.local.id" value="nodeOne_12345" boot-time="false"/>
                <property name="org.uberfire.cluster.vfs.lock" value="vfs-repo" boot-time="false"/>
                <property name="org.uberfire.cluster.autostart" value="true" boot-time="true"/>
                <property name="org.uberfire.nio.git.daemon.host" value="localhost"/>
                <property name="org.uberfire.nio.git.daemon.port" value="9417" boot-time="false"/>
                <property name="org.uberfire.nio.git.daemon.hostport" value="9417" boot-time="false"/>
                <property name="org.uberfire.nio.git.ssh.host" value="localhost"/>
                <property name="org.uberfire.nio.git.ssh.port" value="8002" boot-time="false"/>
                <property name="org.uberfire.nio.git.ssh.hostport" value="8002" boot-time="false"/>
                <property name="org.uberfire.nio.git.dir" value="//jboss-eap-6.4-domain-cluster-zookeeper-cluster/server-one-data" boot-time="false"/>
                <property name="org.uberfire.metadata.index.dir" value="/jboss-eap-6.4-domain-cluster-zookeeper-cluster/server-one-data" boot-time="false"/>
                <property name="org.uberfire.nio.git.ssh.cert.dir" value="/jboss-eap-6.4-domain-cluster-zookeeper-cluster/server-one-data" boot-time="false"/>
            </system-properties>
        </server>
        <server name="server-two" group="brms" auto-start="true">
            <system-properties>
                <property name="jboss.node.name" value="nodeTwo" boot-time="false"/>
                <property name="org.uberfire.cluster.id" value="bpms-cluster" boot-time="false"/>
                <property name="org.uberfire.cluster.zk" value="localhost:2181,localhost:2182,localhost:2183" boot-time="false"/>
                <property name="org.uberfire.cluster.local.id" value="nodeTwo_12346" boot-time="false"/>
                <property name="org.uberfire.cluster.vfs.lock" value="vfs-repo" boot-time="false"/>
                <property name="org.uberfire.cluster.autostart" value="true" boot-time="true"/>
                <property name="org.uberfire.nio.git.daemon.host" value="localhost"/>
                <property name="org.uberfire.nio.git.daemon.port" value="9418" boot-time="false"/>
                <property name="org.uberfire.nio.git.daemon.hostport" value="9418" boot-time="false"/>
                <property name="org.uberfire.nio.git.ssh.host" value="localhost"/>
                <property name="org.uberfire.nio.git.ssh.port" value="8003" boot-time="false"/>
                <property name="org.uberfire.nio.git.ssh.hostport" value="8003" boot-time="false"/>
                <property name="org.uberfire.nio.git.dir" value="/jboss-eap-6.4-domain-cluster-zookeeper-cluster/server-two-data" boot-time="false"/>
                <property name="org.uberfire.metadata.index.dir" value="/jboss-eap-6.4-domain-cluster-zookeeper-cluster/server-two-data" boot-time="false"/>
                <property name="org.uberfire.nio.git.ssh.cert.dir" value="/jboss-eap-6.4-domain-cluster-zookeeper-cluster/server-two-data" boot-time="false"/>
            </system-properties>
            <socket-bindings socket-binding-group="full-sockets" port-offset="100"/>
        </server>

• Cluster 2: Server-group: bpms (it will use the bpms-cluster2 and the vfs-repo1), domain.xml file:

      <server-group name="bpms" profile="full">
          <jvm name="default">
              <heap size="1000m" max-size="1000m"/>
              <permgen max-size="256m"/>
          </jvm>
          <socket-binding-group ref="full-sockets"/>
          <deployments>
              <deployment name="business-central.war" runtime-name="business-central.war"/>
          </deployments>
      </server-group>
  

  • Servers configurations:

        <server name="server-three" group="bpms" auto-start="true">
            <socket-bindings socket-binding-group="full-sockets" port-offset="300"/>
            <system-properties>
                <property name="jboss.node.name" value="nodeThree" boot-time="false"/>
                <property name="org.uberfire.cluster.id" value="bpms-cluster2" boot-time="false"/>
                <property name="org.uberfire.cluster.zk" value="localhost:2181,localhost:2182,localhost:2183" boot-time="false"/>
                <property name="org.uberfire.cluster.local.id" value="nodeThree_12347" boot-time="false"/>
                <property name="org.uberfire.cluster.vfs.lock" value="vfs-repo1" boot-time="false"/>
                <property name="org.uberfire.cluster.autostart" value="true" boot-time="true"/>
                <property name="org.uberfire.nio.git.daemon.host" value="localhost"/>
                <property name="org.uberfire.nio.git.daemon.port" value="9419" boot-time="false"/>
                <property name="org.uberfire.nio.git.daemon.hostport" value="9419" boot-time="false"/>
                <property name="org.uberfire.nio.git.ssh.host" value="localhost"/>
                <property name="org.uberfire.nio.git.ssh.port" value="8004" boot-time="false"/>
                <property name="org.uberfire.nio.git.ssh.hostport" value="8004" boot-time="false"/>
                <property name="org.uberfire.nio.git.dir" value="/jboss-eap-6.4-domain-cluster-zookeeper-cluster/server-three-data" boot-time="false"/>
                <property name="org.uberfire.metadata.index.dir" value="/jboss-eap-6.4-domain-cluster-zookeeper-cluster/server-three-data" boot-time="false"/>
                <property name="org.uberfire.nio.git.ssh.cert.dir" value="/jboss-eap-6.4-domain-cluster-zookeeper-cluster/server-three-data" boot-time="false"/>
            </system-properties>
        </server>
        <server name="server-four" group="bpms" auto-start="true">
            <socket-bindings socket-binding-group="full-sockets" port-offset="400"/>
            <system-properties>
                <property name="jboss.node.name" value="nodeFour" boot-time="false"/>
                <property name="org.uberfire.cluster.id" value="bpms-cluster2" boot-time="false"/>
                <property name="org.uberfire.cluster.zk" value="localhost:2181,localhost:2182,localhost:2183" boot-time="false"/>
                <property name="org.uberfire.cluster.local.id" value="nodeFour_12348" boot-time="false"/>
                <property name="org.uberfire.cluster.vfs.lock" value="vfs-repo1" boot-time="false"/>
                <property name="org.uberfire.cluster.autostart" value="true" boot-time="true"/>
                <property name="org.uberfire.nio.git.daemon.host" value="localhost"/>
                <property name="org.uberfire.nio.git.daemon.port" value="9420" boot-time="false"/>
                <property name="org.uberfire.nio.git.daemon.hostport" value="9420" boot-time="false"/>
                <property name="org.uberfire.nio.git.ssh.host" value="localhost"/>
                <property name="org.uberfire.nio.git.ssh.port" value="8005" boot-time="false"/>
                <property name="org.uberfire.nio.git.ssh.hostport" value="8005" boot-time="false"/>
                <property name="org.uberfire.nio.git.dir" value="/jboss-eap-6.4-domain-cluster-zookeeper-cluster/server-four-data" boot-time="false"/>
                <property name="org.uberfire.metadata.index.dir" value="/jboss-eap-6.4-domain-cluster-zookeeper-cluster/server-four-data" boot-time="false"/>
                <property name="org.uberfire.nio.git.ssh.cert.dir" value="/jboss-eap-6.4-domain-cluster-zookeeper-cluster/server-four-data" boot-time="false"/>
            </system-properties>
        </server>

After applying these configurations you should be able to have two clusters in the same Zookeeper environment.