The first post of this series focused on the basic concepts behind clustering JBoss AS 7 and EAP 6. We explained how to enable cluster capabilities for a simple Java EE application and setup a basic cluster environment in the standalone operation mode. In this post we will now explain the domain mode and how to manage a cluster environment in managed domains.
Overview
A managed domain spans over multiple hosts with centralized administration and management policies. Each host that runs in domain mode can contain several server instances that belong to one domain. The domain mode is one of two possible operating modes of the EAP 6 respectively the JBoss AS 7. The other mode is the standalone mode. A JBoss application server, which operates in the standalone mode, is an independent process. This mode is pretty similar to the previous versions of the JBoss application server. The different operating modes have no influence to the capabilities of the application server, but rather how to manage one or multiple servers. So, the domain mode is completely independent from clustering. But it provides many useful features for managing clustered environments such as handling of deployments for multiple instances. For example the JBoss AS 7 running in standalone mode does not support any more the previous farming deployment capabilities. Now, this can be done with the domain mode capabilities and of course there are more features such as starting and stopping servers from one single console that makes it easier to handle clustered environments.
Server topology in our managed domain
In the last post we build a cluster with two nodes on one machine. In this post we want to a cluster with four nodes on two different machines in the same network as illustrated in the graphic below. The server nodes will be managed within one domain.
Managed domain architecture
A domain can consist of multiple physical or virtual hosts. Each host requires a host controller that is responsible for managing the lifecycle of the servers. Every domain needs one domain controller for centralized management tasks. One of the host controllers or a dedicated instance can act as domain controller. In our example environment we use a dedicated host as domain controller. We also recommend this architecture from the perspective of availability because the other hosts are thus completely independent of administrative tasks.
In the domain mode it is possible to manage multiple server instances on one host. That is why every host needs a separate host controller, which controls the server instances on that host. The domain controller interacts with the host controllers to manage the server instances. Every server instance of a domain belongs to a virtual server group. The idea is, that all servers of the same server group, perform the same tasks. When you deploy an application you will not deploy it to a server, you will deploy it to a whole server group. It is also possible to mange different server groups in one domain, e.g. production, staging or a test sever group.
The controller and the server instances of each host are separate JVM processes, which will be monitored by a process controller. The process controller is also a separate JVM process that is responsible for spawning the other processes and monitoring their lifecycle. If, the host controller process crashed, the process controller will start up the host controller and each server that is configured with the auto-start parameter. However, if the server process of a server instance is terminated unexpectedly, the process controller will not restart the server instance automatically. For this, an external monitoring is necessary.
Domain setup
Now we can start to setup our clustered domain. First we need a fresh copy of the EAP 6 or respectively the latest JBoss AS 7 on each machine.
Domain controller setup
As already mentioned on one host we will setup our domain controller. The server configuration of the domain will be according to the policies of the domain configuration. These configurations are centralized in the domain.xml file of the domain controller. The domain.xml is located at domain/configuration/. It includes the main configuration for all server instances. This file is only required for the domain controller. A physical host in a managed domain will be configured with the host.xml file which is locatet at domain/configuration too. This file includes host specific configuration such network interface bindings.
In the domain.xml file we will define a cluster-ha server group for the server instances that will belong to this group. The default configuration includes already two server groups. We will replace the configuration with the following server group configuration for our domain:
<domain xmlns="urn:jboss:domain:1.3">
...
<server-groups>
<server-group name="cluster-ha" profile="ha">
<jvm name="default"/>
<socket-binding-group ref="ha-sockets" />
</server-group>
</server-groups>
</domain>
Each server group needs a unique name and a reference to one profile of the domain configuration. The default configuration includes four preconfigured profiles:
- default – Support of Java EE Web-Profile plus some extensions like RESTFul Web Services or support for EJB3 remote invocations
- full – Support of Java EE Full-Profile and all server capabilities without clustering
- ha – default profile with clustering capabilities
- full-ha – full profile with clustering capabilities
A profile contains the configuration of the supported subsystems that is added by an extension. We choose the ha profile as our application does not require additional technologies except the Java EE web profile technologies and cluster capabilities.
The referenced profile will be assigned by the server group to one socket-binding group. A socket-binding group references to logical interface names instead direct to the interfaces of a host. These logical interfaces are defined in the <interfaces> section of the domain.xml configuration file. The logical interfaces and the socket-binding group represent the socket addresses of each host.
In addition to the profile and the socket-binding group configuration of a server group, the domain controller can provide a set of system-wide properties and a reference to a JVM configuration, which is located in the host.xml configuration file.
For the domain controller we use the preconfigured host-master.xml configuration file, that includes the necessary configuration for the host, e.g. the physical network binding of the management interface or the JVM configuration. With the following configuration in the host-master.xml file, the host becomes the domain controller.
<host name="master" xmlns="urn:jboss:domain:1.3">
...
<domain-controller>
<local/>
</domain-controller>
...
</host>
The last step before we start the domain controller is to create a management user for the domain controller. This user is necessary when the host controller needs to establish a connection to the domain controller. For this there is an add-user.sh script in the bin directory of the JBoss AS distribution.
$./add-user.sh What type of user do you wish to add? a) Management User (mgmt-users.properties) b) Application User (application-users.properties) (a): a Enter the details of the new user to add. Realm (ManagementRealm) : Username : domainadmin Password : Re-enter Password : Are you sure you want to add user 'domain' yes/no? y About to add user 'domain for realm 'ManagementRealm' Is this correct yes/no? y Added user 'domain' to file '/standalone/configuration/mgmt-users.properties' Added user 'domain' to file '/domain/configuration/mgmt-users.properties' Is this new user going to be used for one AS process to connect to another AS process e.g. slave domain controller? yes/no? y To represent the user add the following to the server-identities definition
You need to answer the last question with yes or y to indicate that the user will be used to connect to the domain controller from the host controller. The generated secret value is the Base64-encoded password of the new created user.
Now we can start the domain controller with the following command. We will set the physical network bind address to the host configuration with the jboss.bind.address.management property. The management interface must be reachable for all hosts in the domain in order to establish a connection with the domain controller.
./domain.sh \ --host-config=host-master.xml \ -Djboss.bind.address.management=192.168.0.1
Host configuration
After the domain controller is configured and started, the next step is to setup the two hosts. On each host we need also a fresh copy of the JBoss AS distribution.
The main configuration for the hosts is the host.xml file which is, as already mentioned, located at domain/configuration/. In our example we will use the preconfigured slave-host.xml file for the host controller. This file is also located inside the domain configuration directory of the JBoss AS 7 distribution.
The first thing is to choose a unique name for each host in our domain to avoid name conflicts. Otherwise, the default is the host name of the server.
<host name="server1" xmlns="urn:jboss:domain:1.3">
...
</host>
As already mentioned before, the host controller needs to know how to establish a connection to the domain controller. The following configuration on the host specifies where the domain controller is located. Thus, the host controller can register to the domain controller itself. The remote tag must include the username and the security realm of the domain controller.
<host name="master" xmlns="urn:jboss:domain:1.3">
...
<domain-controller>
<remote host="${jboss.domain.master.address}" port="${jboss.domain.master.port:9999}" username="domainadmin" security-realm="ManagementRealm"/>
</domain-controller>
...
</host>
Another thing is to announce the secret value. This is needed for authentication together with the username. The secret value was the last output from add-user.sh script that we executed previously. This value is only the Base64-encoded password of the domain user. We recommend encrypt sensitive values like as passwords with the vault tool script. The script is provided in the bin directory of JBoss AS.
<host name="server1" xmlns="urn:jboss:domain:1.3">
<management>
<security-realms>
<security-realm name="ManagementRealm">
<server-identities>
<!-- Replace this with either a base64 password of your own, or use a vault with a vault expression -->
<secret value="c2VjcmV0"/>
</server-identities>
<authentication>
<local default-user="$local" />
<properties path="mgmt-users.properties" relative-to="jboss.domain.config.dir"/>
</authentication>
</security-realm>
...
</security-realms>
...
</management>
...
</host>
The last step is to configure the server nodes inside the host-slave.xml file on both hosts. We will replace the existing configuration with the following configuration for our two server instances of the cluster-ha group.
<host name="server1" xmlns="urn:jboss:domain:1.3">
...
<servers>
<server name="server-one" group="cluster-ha" auto-start="false"/>
<server name="server-two" group="cluster-ha" auto-start="false">
<!-- server-two avoids port conflicts by incrementing the ports in
the default socket-group declared in the server-group -->
<socket-bindings port-offset="150"/>
</server>
</servers>
</host>
The auto-start flag indicates that the server instances will not be started automatically if the host controller is started.
For the second server a port-offset is configured to avoid port conflicts. With the port offset we can reuse the socket-binding group of the domain configuration for multiple server instances on one host.
That was already the necessary configuration for a host. After you roll out these configuration on both hosts we can start the host controller on each host with the following command:
./domain.sh \ --host-config=host-slave.xml \ -Djboss.domain.master.address=192.168.0.1 \ -Djboss.bind.address=192.168.0.2 \ -Djboss.bind.address.management=192.168.0.2
and respectively
./domain.sh \ --host-config=host-slave.xml \ -Djboss.domain.master.address=192.168.0.1 \ -Djboss.bind.address=192.168.0.3 \ -Djboss.bind.address.management=192.168.0.3
At this time only the host controllers are started on both hosts. If we have a look into the host-controller.log file from the domain controller, we see server1 and server2 are registered as remote slave hosts.
[Host Controller] 23:32:00,082 INFO [org.jboss.as.domain] (slave-request-threads - 1) JBAS010918: Registered remote slave host "server1", JBoss EAP 6.0.0.GA (AS 7.1.2.Final-redhat-1) [Host Controller] 23:32:16,739 INFO [org.jboss.as.domain] (slave-request-threads - 1) JBAS010918: Registered remote slave host "server2", JBoss EAP 6.0.0.GA (AS 7.1.2.Final-redhat-1)
Domain management via the command line interface
To demonstrate how we can now manage our domain and deploy applications we will use the existing application from the last post.
Checkout the code from github and build the project with Maven by execute the mvn package command.
$ git clone https://github.com/akquinet/jbosscc-as7-examples.git Cloning into jbosscc-as7-examples... $ cd cluster-example/ $ mvn package [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Cluster Web Application example 1.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] ... [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 2.512s [INFO] Finished at: Wed Jun 20 22:29:15 CEST 2012 [INFO] Final Memory: 10M/81M [INFO] ------------------------------------------------------------------------
After we built the application we can now deploy the packaged Web-archive via the command line interface to the whole server group and manage our domain. All these management task can also be done with the web console.
First we need to connect the command line interface with the domain controller. With the following commands the command line interface establishes a connection to the domain controller:
./bin/jboss-cli.sh You are disconnected at the moment. Type 'connect' to connect to the server or 'help' for the list of supported commands. [disconnected /] connect 192.168.0.1 Authenticating against security realm: ManagementRealm Username: domainadmin Password: [domain@192.168.0.1:9999 /]
With the deploy command we can now deploy the application to our server group.
[domain@192.168.0.1:9999 /] deploy jbosscc-as7-examples/cluster-example/target/cluster-example-1.0-SNAPSHOT.war --server-groups=cluster-ha
The deployment information will be persisted in the domain.xml file of the domain controller. After a restart the deployment to the server-group will still be available.
<deployments>
<deployment name="cluster-example-1.0-SNAPSHOT.war" runtime-name="cluster-example-1.0-SNAPSHOT.war">
<content sha1="8634e58ce0919e459474c1bcfb4ba888aceae075"/>
</deployment>
</deployments>
<server-groups>
<server-group name="cluster-ha" profile="ha">
…
<deployments>
<deployment name="cluster-example-1.0-SNAPSHOT.war" runtime-name="cluster-example-1.0-SNAPSHOT.war"/>
</deployments>
</server-group>
</server-groups>
After the application is deployed we can now start all the server instances of the cluster-ha server group with the following command:
[domain@192.168.0.1:9999/] /server-group=cluster-ha:start-servers
{
"outcome" => "success",
"result" => undefined,
"server-groups" => undefined
}
The four server instances formed a cluster, because they are all members of the cluster-ha group which is associated to the ha profile of the domain configuration. The ha profile contains the default configuration for a UDP based cluster environment.
In the last post we have already configured the httpd server and mod_cluster for load balancing. If the hosts are in the same UDP multicasting group as the httpd server we can access our application through http://192.168.0.1/cluster and see which server instance process the request.
Let us stop that node, which was processing the request.
[domain@192.168.0.1:9999/] /host=server1/server-config=node-one:stop
{
"outcome" => "success",
"result" => "STOPPING"
}
After hitting the refresh button of the browser, you see there was a failover to another node and the session is still available.
Summary
The domain mode is one of the new powerful features of the EAP 6 and JBoss AS7. The operation mode is independent of the cluster functionality, but it provides some useful features for the management of a cluster environment, such the deployment of applications, centralized configuration or starting and stopping of server instances.
This post was the second post of an series about clustering capabilities of the JBoss AS 7. In the next post we cover how to build more complex cluster-topologies with the JBoss AS 7 and the underlying subsystems.
Nice article. I’d recommend a simplification, whereby you specify the IP of the domain controller directly in the slave-host.xml, e.g.:
<remote host="${jboss.domain.master.address:172.26.130.164}"…
This also allows you to override it if required. In this way, the slaves can simply be run like this:
./domain.sh –host-config=host-slave.xml
I am new in the creating clustering. Please help to understand the article.
1. How many machines you are using.
2. Domain controller in configured in which machine?
3. Host Controller is configured in which machine?
Thanks,
Santanu
After banging my head against the wall for close to a week, thank you for this post! It got me straightened out.
A couple of points:
You “recommend” keeping the domain controller separate from any servers, but I found that keeping the domain controller separate from servers was ESSENTIAL. I could not get clustering to work AT ALL with a remote server as long as I was trying to run the domain controller and servers in the same jboss instance. (as with the supplied host.xml for example). The cluster nodes would never find each other and all thought they were part of different 1-node clusters. And it’s cleaner conceptually.
Later I tried to put servers on the same box as the domain controller (but in a separate jboss instance) and this can work as long as you bind the slave host controller to a management port other than the default (-Djboss.management.native.port=10099). You might also have to change the console log location as I am not sure what would happen if two processes tried to log to the same logfile. And if launching via services you might have to change the pidfile name.
But why do all this? Virtual machines are cheap.
But I could NOT get clustering to work as long as the domain controller and the servers were under the same process controller.
Hi,
I followed the above approach but host unable to ping the domain controller.
org.jboss.as.host.controller] (Controller Boot Thread) JBAS010900: Could not connect to remote domain controller
Pls let me know.
Hi, i have one question: is it possible manage a specific server though CLI if the domain controller is not available? You said that if the domain controller crashes, the host controller and server instances will be available, right? Is it possible manage server instances without domain controller?
Is there a good resource that you have seen that helps configure the VAULT features in a domain setup? The EAP 6 docs seem to not be adequate when explaining how to setup VAULT in a domain configuration.
The documentation is not clear on whether only the domain controller requires a VAULT setup or if each node requires a VAULT setup. I have tried both. In each case it seems that he VAULT can be opened, but the password cannot be retrieved out of the VAULT…
I am using ISAPI load balancer on two IIS webservers and need to cluster Jboss EAP6 servers running only one node on each server.
Please let me know what are all changes need to be done on host.xml, host-master.xml and domain.xml
Hi, i read your blog from time to time and i own a similar one and i was just wondering if you get a lot of spam comments? If so how do you prevent it, any plugin or anything you can recommend? I get so much lately it’s driving me insane so any assistance is very much appreciated.
But how to stop the domain controller and slave ?
I have a similiar problem Mayank mentioned:
I have an ejb-cluster (stateless) with one master server including domain controller and host controller.
This server also acts as an slave node.
Two other slaves are bound to the server group.
When the domain controller stops all the slaves stop. What is the problem?
Is it possible to stop alle server in a domain ?
Yes, you can stop all server of the domain with the following CLI command:
[domain@192.168.0.1:9999 /] :stop-servers
And of course you can start again all with the command:
[domain@192.168.0.1:9999 /] :start-servers
Regards, Heinz
Hi,
Output is showing success but server is not getting stop.
I am facing issue how To start all slave server from master.
Thanks,
Tauseef
Indeed a good post:
I have a query, when the domain controller stops all the slaves cease to respond to http traffic. Do we have a workaround for the same ?
The server instances are independent from the domain controller. If the server is shut down or the domain controller crashes, the host controller and server instances will be still available. Only the management and configuration of the server is not possible at this time. The host controller tried in a certain interval a re-connect. After the domain controller is available, the host controller establishes a new connection.
hi, this is an old comment but here is the question.
can this be done? if my domain controller down, can the ip remain the same and will not lost the connection.
i’m goggled around but no one said its possible except your comment here.
I always used to read article in news papers but now as
I am a user of web thus from now I am using net for content, thanks to web.