How to setup EAP in Domain Mode with Remote Host Controllers ?

Solution Verified - Updated -

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 6.x
    • 7.x

Issue

  • We want to configure run JBoss EAP6 or EAP7 in domain mode such that the host controllers will be running in a remote box.
  • Do you have any suggestions for debugging connections between host and domain controllers?
  • Is there a way to use single user for multiple host-slave controllers?
  • Is there a way to use username attribute that can be connected to the domain controller and shared amongst all slave nodes?
  • Is there any documentation with detailed steps to set-up Remote Host Controllers in EAP ?
  • Could not connect to remote domain controller. Host Controller is not attaching with Domain Controller. The Host Controller and Domain Controller in the same machine.

Resolution

This solution will use the same EAP version for all instances, if you are looking whether different instance versions can be used in the same domain please refer to Is it possible to run a mix of different EAP (EAP6 and EAP7) versions within the same domain? .

We will use two machines:

The "master" box, which will run the Domain Controller, hereafter called the DC. A "slave", which will run a Host Controller, hereafter called the HC1. You can use multiple slaves with each an HC. The principle stays the same. But in following steps we use on slave:

Step 1) Unzip jboss-eap-6.x.y.zip on the DC, for example to /opt/jboss-eap-6-Master

Step 2) Unzip jboss-eap-6.x.y.zip on HC1, for example to /opt/jboss-eap-6-Slave

Step 3) On the DC, edit /opt/jboss-eap-6-Master/domain/configuration/host.xml and set its host name as follows: (the name should be unique in the whole domain)

<host name="masterOne" xmlns="urn:jboss:domain:1.3">

Do not confuse this name with the actual hostname of the machine.

Step 4) On HC1, edit /opt/jboss-eap-6-Slave/domain/configuration/host.xml and set its host name like follows: (the name should be unique in the whole domain)

<host name="hostOne" xmlns="urn:jboss:domain:1.3">

and tell it where the DC is - here we need to use the actual hostname of the DC

    <domain-controller>
        <remote host="YourMasterHostName" port="9999" security-realm="ManagementRealm"/>
    </domain-controller>

Note: For EAP 7, you need to mention the protocol type as well through which HC1 will communicate with DC. The default protocol is remote. Apart from remote protocol, http-remoting and https-remoting protocol can be used.
http-remoting allow to connect host controller to domain controller via http management protocol, default port is 9990 and https-remoting allow to connect host controller to domain controller via management https protocol where ssl is enabled, default port is 9993.

    <domain-controller>
         <remote protocol="remote" host="YourMasterHostName" port="9999" security-realm="ManagementRealm"/> 
    </domain-controller>
  • To share single user for multiple host-slave controllers use the below way :
<host name="AnyName" xmlns="urn:jboss:domain:1.3">
     :
     :
    <domain-controller>
        <remote host="YourMasterHostName" port="9999" security-realm="ManagementRealm" username="hostOne"/>
    </domain-controller>
     :
     :
</host>

Note: If you have an internal DNS environment where you register your servers, make sure you configured host attribute using the FQDN (Full-Qualified Domain Name)
Still in /opt/jboss-eap-6-Slave/domain/configuration/host.xml, make the servers section look like:

<servers>
  <server name="TestServer" group="main-server-group" auto-start="true"/>
</servers>

Step 5) Now on the DC you will need to create a user, for each HC you want to use, in the ManagementRealm. So in our scenario, create a user for hostOne as you defined in <host name="hostOne" ...>

Use the following script on the DC: /opt/jboss-eap-6-Master/bin/add-user.sh
Note: In this case we'll use the password "passwordOne"):

$ ./add-user.sh 

What type of user do you wish to add? 
 a) Management User (mgmt-users.properties) 
 b) Application User (application-users.properties)
(a): a

Enter the details of the new user to add.
Realm (ManagementRealm) : ManagementRealm
Username : hostOne
Password : passwordOne
Re-enter Password : passwordOne
About to add user 'hostOne' for realm 'ManagementRealm'
Is this correct yes/no? yes
Added user 'hostOne' to file '/opt/jboss-eap-6-Master/standalone/configuration/mgmt-users.properties'
Added user 'hostOne' to file '/opt/jboss-eap-6-Master/domain/configuration/mgmt-users.properties'
Is this new user going to be used for one AS process to connect to another AS process e.g. slave domain controller?

yes/no? yes

To represent the user add the following to the `server-identities` definition `<secret value="cGFzc3dvcmRPbmU=" />`

Step 6) On HC1 edit /opt/jboss-eap-6-Slave/domain/configuration/host.xml and add the <server-identity> and the <secret value="..."> which was generated in step 5 as follows:

<management>
  <security-realms>
    <security-realm name="ManagementRealm">
      <server-identities>
          <secret value="cGFzc3dvcmRPbmU=" />
      </server-identities>
      <authentication>
        <local default-user="$local" />
        <properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>
      </authentication>
    </security-realm>
    .
    .
    .
</management>

Step 7) Start the DC with the script in /opt/jboss-eap-6-Master/bin as below. Make sure that there are no firewall restrictions so that YourMasterHostName:9999 is accessible from remote HCs.

./domain.sh -b YourMasterHostName -bmanagement YourMasterHostName

Note that YourMasterHostName should be the host name of the machine where the DC is running, as defined in DNS and not the name you gave the JBoss DC itself (masterOne)

Step 8) Start the HC1 with the script in /opt/jboss-eap-6-Slave/bin as below:

./domain.sh -Djboss.domain.master.address=YourMasterHostName -b SlaveHostName -bmanagement SlaveHostName

Again note that SlaveHostName should be the host name of the machine where HC1 is running, as defined in DNS and not the name you gave the JBoss HC1 itself (hostOne)

The result should be that on the DC, in the console log you see the slaves join, while on the slaves you see a server instance start-up.

Diagnostic Steps

  • If you need to troubleshoot connections, take a look at the domain/log/host-controller.log file for information about host controller connections. For example, you should see a message like this on the DC when a HC successfully connects:
17:10:33,671 INFO  [org.jboss.as.domain] (slave-request-threads - 1) JBAS010918: Registered remote slave host "klape", JBoss EAP 6.1.1.GA (AS 7.2.1.Final-redhat-10)
  • All available logging from the domain/host controllers is printed to host-controller.log by default, but if you want additional details about the connections between controllers, you can turn on TRACE logging for org.jboss.remoting. This modification will have to be done in domain/configuration/logging.properties. Here is a sample logging.properties that has TRACE for org.jboss.remoting enabled and also appends to existing host-controller.log files rather than overwriting it every restart:
# Additional logger names to configure (root logger is always configured)
loggers=org.jboss.remoting

# Root logger level
logger.level=${jboss.boot.server.log.level:INFO}
# Root logger handlers
logger.handlers=BOOT_FILE,CONSOLE

# Console handler configuration
handler.CONSOLE=org.jboss.logmanager.handlers.ConsoleHandler
handler.CONSOLE.properties=autoFlush
handler.CONSOLE.level=${jboss.boot.server.log.console.level:INFO}
handler.CONSOLE.autoFlush=true
handler.CONSOLE.formatter=PATTERN

logger.org.jboss.remoting.level=TRACE

# File handler configuration
handler.BOOT_FILE=org.jboss.logmanager.handlers.PeriodicRotatingFileHandler
handler.BOOT_FILE.level=TRACE
handler.BOOT_FILE.constructorProperties=fileName,append
handler.BOOT_FILE.properties=autoFlush,fileName,append
handler.BOOT_FILE.autoFlush=true
handler.BOOT_FILE.fileName=${org.jboss.boot.log.file:boot.log}
handler.BOOT_FILE.append=true
handler.BOOT_FILE.formatter=PATTERN

# Formatter pattern configuration
formatter.PATTERN=org.jboss.logmanager.formatters.PatternFormatter
formatter.PATTERN.properties=pattern
formatter.PATTERN.pattern=%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

8 Comments

This article should have a link to [Host controller authentication failure] (https://access.redhat.com/knowledge/solutions/97413) because those who install from RPMs (like myself) are likely to follow this article and encounter authentication failures.

I got it working by specifying in /etc/sysconfig/jbossas-domain on the DC:

JBOSSSH="$JBOSS_HOME/bin/$JBOSSCONF.sh -bmanagement=0.0.0.0 -bpublic=0.0.0.0 -bunsecure=0.0.0.0"

And in that same file on HC1:

JBOSSSH="$JBOSS_HOME/bin/$JBOSSCONF.sh -bmanagement=0.0.0.0 -bpublic=0.0.0.0 -bunsecure=0.0.0.0 --host-config=host-slave.xml --master-address=YourMasterHostName"

And in /etc/jbossas/domain/host-slave.xml on HC1, adding a `username` attribute to the `remote` element inside the `domain-controller` element. Without `username` I was stuck with authentication failures that I could not figure out until I encountered the above-referenced article.

@Tony, you only need to set the 'username' property ( element) in host-slave.xml if you don't specify the 'name' property in element.

1) I wish the article had all the "CLI" commands required to perform the above mentioned configuration.
2) This article does not address the case, where someone wants to run 2 separate instances of JBoss EAP on the same host. As in the host-controller, we always mentioned the host name of the domain-controller. What if there are 2 different domain-controllers are running on the domain controller host?

stupid wiki - won't let me put in XML tags.. but you get the idea.....

step 4 forgot the following - which is important if the HC is on one box but the DC is on another remote box:

[domain-controller>
[!-- uncomment <local/> and replace with -->
[remote host="${jboss.domain.master.address}" port="${jboss.domain.master.port:9999}" security-realm="ManagementRealm"/>

I'm on EAP 6.2.

There is a note above that instructs us to use the FQDN for element name parameter. Is that actually required? Are there repurcussions for not doing so? I have a 3-host managed domain, and I am not using the FQDN. I am using the short hostname for name= (name=host1), but not the FQDN (name=host1.foo.com). Seems to be working fine.

I am getting following error while run two jboss instance.... I'm using JDK 7 Update 80... EAP 6.3...

[Host Controller] 14:07:48,629 WARN [org.jboss.as.host.controller] (Controller Boot Thread) JBAS010900: Could not connect to remote domain controller at remote://my-host-ip1:9999 -- java.net.ConnectException: JBAS012144: Could not connect to remote://my-host-ip1:9999. The connection timed out

ran commands....

domain.sh -b my-host-ip1 -Djboss.bind.address.management=my-host-ip1
domain.sh -b my-host-ip2 -Djboss.domain.master.address=my-host-ip1 --host-config=host-slave.xml

my-host-ip1 host.xml has following config...

<domain-controller>
<local/>
</domain-controller>

does it needs to be changed...

As per the solution provided here..... I ran the commands and succeeded....

./domain.sh -b my-host-ip1 -bmanagement my-host-ip1
./domain.sh -Djboss.domain.master.address=my-host-ip1 -b my-host-ip2 -bmanagement my-host-ip2

But I have a question...... as command had been given like -b my-host-ip2 -bmanagement my-host-ip2.......... my application in host1 (master) goes down then host2 will it take it up and running..............?

is am I right.......? or it is something else for this config is.....

I need to run my app in cluster.... as per the solution here there can be many hosts attached to first one which acts as DC...... does it mean the App is clustered across DC and HCs (provided right config. is in place concerning Apache and httpd connector is in place modules\system\layers\base\native\lib64\httpd of JBoss EAP)

In domain.xml in all Host Controller (HC) Machines.... host.xml has to modified to connect with Domain Contoller (DC)....

<domain-controller>
       <!-- Alternative remote domain controller configuration with a host and port -->
        <remote host="${jboss.domain.master.address:10.nnn.nn.nn}" port="${jboss.domain.master.port:9999}" security-realm="ManagementRealm"/>
    </domain-controller>

for DC host.xml can be,

<domain-controller>
<local/>
</domain-controller>

If we are using anything other than host.xml it has to mentioned while startup ./domain.sh ... --host-config=...

Other that, which profile you are running has to be configured. And for that profile heap-size has to be adjusted and .ear which we deploy has to mentioned.

For Apache, simple mod cluster configuration can be found in the bottom of the following JBoss.org link....

https://docs.jboss.org/mod_cluster/1.2.0/html/native.config.html

Once Apache is setup,

http://10.nnn.nn.nn:pppp/mod_cluster-manager will give the mod-cluster configuration that how many nodes attached in the context....

In order deploy the application in / context.... i.e., http://10.nnn.nn.nn:pppp/ following things needs to be corrected in domain.xml

1)

<mod-cluster-config excluded-contexts="console" >

by default ROOT also in excluded-contexts hence it needs to be mentioned explicitly and CONSOLE which is possible option to exclude.

2)

<virtual-server name="default-host" enable-welcome-root="false">