Load-balancing and failover of remote EJB clients in EAP6 and JBoss AS7

Dieser Artikel auf Deutsch

In the recent posts of this series about the clustering capabilities of the JBoss EAP6 and the AS7, we covered the basic concepts, managing cluster nodes in domain mode and scalable HA cluster topologies. This post will be about clustering capabilities for remote EJB clients. We will explain how to cluster EJB components and invoke them from a standalone remote client with client-side failover and load balancing.

Overview

Remote invocation of server-side EJB components has changed compared to previous versions of the JBoss Application Server. To invoke a server-side EJB component, the AS7 provides a client library for EJB applications. The library can be used in several ways to invoke on a server-side EJB component:

1. Programmatically, through the direct use of the EJB client API
2. Through the JNDI API
3. Through the JNDI API, backed by the JBoss remote-naming project

Using the API programmatically or through the JNDI API has the benefit of various optimizations, such as the local creation of a proxy for stateless session beans, avoiding network round-trips and serialization of the proxy. The third way, using the remote-naming project, may be necessary if the client needs both, invocations of remote objects exposed via JNDI and EJB invocations. A detailed explanation that helps to decide which approach to take is given in the community documentation. In the scope of this blogpost, we will use the EJB client library through the JNDI API and focus on the clustering-specific configuration options that do not seem to have been documented very well so far.

The example scenario

To demonstrate client-side failover and load balancing of EJB clients, we will use a simple example application. You find the application in our github repository, in the cluster-client-example directory. The project includes the server-side application and a simple Swing-based client application. It is built with Maven by executing the mvn package command. In our scenario, we assume that the EJB components are deployed on two server instances and the session data are replicated between these instances.

The server-side application

The server-side application includes a stateful session bean that counts the invocations, and a stateless session bean that returns the name of the cluster node handling the invocation. Both session beans implement a remote business interface to support remote access. In order to cluster an EJB, it has to be marked as clustered. This can be done directly in the source code with the @Clustered annotation or with the <clustered> element of the JBoss-specific deployment descriptor for EJB components in the META-INF/jboss-ejb3.xml file. The following listing shows as an example the implementation of the stateful session bean:

@Stateful
@Clustered
@Remote(RemoteStateful.class)
public class ClusteredStatefulBean implements RemoteStateful {
  ...
 
  @Override
  public int getAndIncrementCounter() {
    ...
  }
 
  @Override
  public String getNodeName() {
    ...
  }
 
  @Remove
  public void destroy() {
  }
}

Marking a stateless EJB as clustered enables load balancing over the cluster nodes for a remote client. Marking a stateful EJB enables, in addition to load balancing, the replication of session data between the cluster nodes.

Clustering a stateful session bean is more complex than clustering a stateless session bean, because the state of the components must be managed among the cluster nodes. The application server does this, but a developer should be aware of this complexity which can reduce scalability.

The EJB components of the application are packaged in an EJB archive with the module name cluster which is defined in the META-INF/ejb-jar.xml deployment descriptor.

A cluster forms automatically and changes its topology independently without any interference from the client side. All that is necessary to cluster an EJB is to explicitly mark (annotate) the EJB component as clustered and start the servers with an HA profile. If running in standalone mode, the respective server instances must be started with the standalone-ha.xml configuration. If, however, the cluster is managed as a domain, the server groups must belong to the HA profile. You find a detailed explanation in our first blogpost about the basic concepts to set up a simple cluster.

The client-side application

The client is a Swing-based application with a simple UI to invoke the server-side EJB components.

Basic EJB client configuration

As already mentioned, our application uses the EJB client library through the JNDI API. Therefore, we need a minimal jndi.properties file in the classpath of our application with the package prefixes for the class name of the factory class that will create a URL context factory.

java.naming.factory.url.pkgs=org.jboss.ejb.client.naming

The next step is to configure the EJB client context in a file named jboss-ejb-client.properties located in the classpath of the application. The EJB client context holds the information about how a connection to remote server instances can be established.

A minimal jboss-ejb-client.properties file is needed to set up an EJB client context with one EJB receiver:

endpoint.name=client-endpoint
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
 
remote.connections=default
remote.connection.default.host=localhost
remote.connection.default.port=4447
remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false

This minimal configuration is already sufficient to invoke an EJB component on a single node and also for a clustered environment. In order to connect to a cluster, a client has to configure an initial connection, a so-called EJB receiver, as an entry point into the cluster. On the first invocation, the EJB receiver will be informed about the cluster topology. After that, the client is independent from the initial server. If that server crashes, the client knows about the other servers of the cluster and can handle failover and distribute the load.

Taking a look at the client’s console after having increased the log level, the following debug message appears upon reception of a cluster topology message:

DEBUG: Received a cluster node(s) addition message, for cluster named ejb with 2 nodes [ClusterNode{clusterName='ejb', nodeName='jb1', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='127.0.0.1', destinationPort=4447}], resolvedDestination=[Destination address=127.0.0.1, destination port=4447]}, ClusterNode{clusterName='ejb', nodeName='jb2', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='127.0.0.1', destinationPort=4547}], resolvedDestination=[Destination address=127.0.0.1, destination port=4547]}]

After the first contact to the initial server, that server will keep the client informed about the other cluster nodes. For this reason, a stateless EJB should be marked as clustered. The initial node has to be available until the EJB receiver gets to know the cluster topology. A detailed explanation concerning the cluster topology communication can be found in the community documentation.

To avoid a single point of failure, it is possible to configure multiple EJB receivers for the initial connection. This is done by configuring symbolic connection names (aliases) in a comma-separated list as shown in the following listing:

remote.connections=default, other 
remote.connection.default.<option>=...
…
remote.connection.other.<option>=...
…

Be careful with this configuration because it is currently not stable. The client crashes in some cases (JBPAPP-9349).

There are some other configuration options which are of interest for a clustered environment. These will be elaborated on further below.

Lookup and invocation of a remote EJB from the client

The client code will be the same irrespective of whether the environment is clustered or not. The client so far is totally unaware of whether an EJB call is made on a clustered or non-clustered environment. The following code listing contains the implementation to initialize the JNDI context and look up the remote SLSB and SFSB.

class RemoteEJBClient
{
  private final Context context = new InitialContext();
 
  public RemoteStateless lookupRemoteStatelessBean() 
       throws NamingException
  {
    final String appName = "";

    // module-name in ejb-jar.xml
    final String moduleName = "cluster";  

    // distinct-name in jboss-ejb3.xml
    final String distinctName = "";  
    
    // ejb:/cluster//ClusteredStatelessBean
    //      !de.akquinet.jbosscc.cluster.RemoteStateless
   
    final String jndiName = "ejb:" + appName + '/' + moduleName + 
        '/'+ distinctName + '/' +
        ClusteredStatelessBean.class.getSimpleName() + '!' +
        RemoteStateless.class.getName();

    return (RemoteStateless) context.lookup(jndiName);
  }
 
  public RemoteStateful lookupRemoteStatefulBean() 
       throws NamingException
  {
    final String appName = "";

    // module-name in ejb-jar.xml
    final String moduleName = "cluster";  

    // distinct-name in jboss-ejb3.xml
    final String distinctName = "";  
    
    // ejb:/cluster//ClusteredStatefulBean
    //     !de.akquinet.jbosscc.cluster.RemoteStateful?stateful
    final String jndiName = "ejb:" + appName + '/' + moduleName + 
        '/' + distinctName + '/' +
        ClusteredStatefulBean.class.getSimpleName() + '!' +
        RemoteStateful.class.getName() + "?stateful";
    
     return (RemoteStateful) context.lookup(jndiName);
  }
}

Concerning stateless EJBs, an optimization has been introduced by the EJB client library: There is no server roundtrip necessary to look up a stateless bean. The initial request to the server is postponed to the first time a method is invoked on the stateless EJB. Not only does this save a server roundtrip. It also means that the client can be started and initialized regardless of whether the server is running. Only when a method is invoked on the stateless EJB for the first time must the server node be available.

RemoteStateless statelessProxy = remoteEJBClient.lookupRemoteStatelessBean();
 
String nodeName = statelessProxy.getNodeName();
// More invocations on the SLSB...

For stateful beans, the above-mentioned optimization is not applicable, because a server-side session needs to be initialized. This is done during the server-side lookup.

RemoteStateful statefulSession = remoteEJBClient.lookupRemoteStatefulBean();
 
int counterValue = statefulSession.getAndIncrementCounter();
String nodeName = statefulSession.getNodeName();
// More invocations on the SFSB...
 
// destroy the session 
statefulSession.destroy();

Load-balancing and failover

A client relies on an initial connection to a server instance that can handle the EJB invocation. After having established the initial connection, the server node informs the client about the cluster topology. Once that connection to the cluster has been established, any cluster node including the initial node may go down at any time or new server nodes may join the cluster. As long as at least one cluster node is operational, state replication of the session data is guaranteed and the client may fail over to another cluster node without losing the session data.

Remember the example scenario of our cluster topology. A cluster in the context of the EJB client means always several server instances, which replicate the state between the nodes. State replication is not strictly necessary for load balancing between sever instances that can handle the EJB invocation of the client. It is also possible to distribute the load between several server instances without clustering capabilities. For this purpose, the EJB components simply have to be deployed on the server instances. These EJB components also do not have to be marked as clustered, which, however, would have the big disadvantage of losing their failover capabilities.

Load balancing as well as failover is client-side controlled by so-called node selectors. There are two kinds of node selectors:

1. deployment node selectors and
2. cluster node selectors.

Deployment node selectors

The first kind is important for non-clustered environments for load balancing and for clustered environments for the initial connections. A deployment node selector is used to choose a corresponding EJB receiver for an eligible server instance that can handle the EJB invocation, without having failover in mind.

The official description of the DeploymentNodeSelector interface is this:

“A selector which selects and returns a node, from among the passed eligible nodes, that can handle a specific deployment within a[n] EJB client context. Typical usage of DeploymentNodeSelector involves load balancing calls to multiple nodes which can all handle the same deployment. This allows the application to have a deterministic node selection policy while dealing with multiple nodes with [the] same deployment.”

By default, the EJB client library uses a RandomDeploymentNodeSelector. A custom-made deployment node selector has to implement the org.jboss.ejb.client.DeploymentNodeSelector interface. An implementation can be configured in the jboss-ejb-client.properties file as follows:

deployment.node.selector=de.akquinet.jbosscc.cluster.client.RoundRobinDeploymentNodeSelector

Cluster node selectors

Unlike the deployment node selector, a cluster node selector is used to choose a cluster node, which belongs to a cluster that replicates the state across the server instances.

The official description of the ClusterNodeSelector interface is this:

“A selector which selects and returns a node from the available nodes in a cluster. The EJBReceiver corresponding to the selected node will then be used to forward the invocations on a[n] EJB. Typical usage of a ClusterNodeSelector involve[s] load balancing of calls to various nodes in the cluster.”

For example, at the first invocation on an EJB, the deployment node selector determines which node should be chosen. In case of an SFSB, the next invocation has an affinity to the node that processed the previous invocation. Only if this node is not available, the cluster node selector chooses another node within the cluster to fail over to, without losing the session data of the client. In case of an SLSB, it depends on the proxy. A proxy has an affinity to a cluster. If the proxy is reused, the cluster node selector is asked for a node to be selected for the next invocation. Each time a new proxy is created, the deployment node selector will be asked which node to select.

These node selectors determine the cluster node on which the next method invocation will be made. So for every call on an SLSB a node selector will be asked on which cluster node that call should be executed. If, on the other hand, a call on an SFSB is made and the cluster node on which the last call has been executed seems to be down, a node selector is asked to which node to fail over.

By default, the EJB client library uses the RandomClusterNodeSelector implementation. A custom implementation of a cluster node selector needs to implement the org.jboss.ejb.client.ClusterNodeSelector interface. An implementation can be configured in the jboss-ejb-client.properties file as follows:

remote.clusters=ejb
...
remote.cluster.ejb.clusternode.selector=de.akquinet.jbosscc.cluster.client.RoundRobinClusterNodeSelector
...

Note that the property remote.clusters may contain a comma-separated list of different HA-clusters. The cluster name ejb in the above example is important as it has to match the name of the cache container that backs the @Clustered bean. The Name ejb is the default as configured in the Infinispan subsystem and is referred from the cache-container attribute of the cluster-passivation-store element in the EJB3 subsystem.

The configured cluster node selector is only responsible for choosing a server instance, which belongs to the configured cluster with the name ejb.

More detailed configuration

Until now, we have explained the basic concepts and a basic configuration. But for larger applications, you may want to have more control over the cluster or the requirement to operate a more complex scenario such as multiple HA-clusters as shown in the following illustration.

If the client needs more control over the connectivity and interaction with the cluster, the following things can be configured in jboss-ejb-client.properties:

• endpoint and channel configuration
• multiple remote connections
• different timeouts
• security settings, such as username and password and callback handler
• specific connections for a cluster and individual nodes
• node selectors

Let us consider a client that interacts with the cluster topology shown above by using the following example configuration:

endpoint.name=client-endpoint
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
 
# custom deployment selector
deployment.node.selector=de.akquinet.jbosscc.cluster.client.RoundRobinDeploymentNodeSelector
 
# timeout in milliseconds, that will be used for EJB invocations
invocation.timeout=3000
 
# timeout in milliseconds after reconnect tasks are submitted 
reconnect.tasks.timeout=2000
 
# user credentials
username=${username:admin}
password=${password:secret}
 
remote.connections=ejb,other
 
# first remote connection
remote.connection.ejb.host=192.168.0.1
remote.connection.ejb.port=4447
remote.connection.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT=false
remote.connection.ejb.connect.timeout=2000
remote.connection.ejb.username=${username:admin}
remote.connection.ejb.password=${password:secret}
 
# second remote connection
remote.connection.other.host=192.168.0.2
remote.connection.other.port=4447
remote.connection.other.username=${username:admin}
remote.connection.other.password=${password:secret}
 
# cluster configuration
remote.clusters=ejb, ejb-other
 
remote.cluster.ejb.clusternode.selector=de.akquinet.jbosscc.cluster.client.RoundRobinClusterNodeSelector
remote.cluster.ejb.connect.timeout=2500
remote.cluster.ejb.max-allowed-connected-nodes=2
remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.cluster.ejb.connect.options.org.xnio.Options.SSL_ENABLED=false
 
remote.cluster.ejb-other.clusternode.selector=de.akquinet.jbosscc.cluster.client.RoundRobinClusterNodeSelector
remote.cluster.ejb-other.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.cluster.ejb-other.connect.options.org.xnio.Options.SSL_ENABLED=false

Note that it is possible to set up quite sophisticated cluster topologies with several differently named cluster. By default, the name of the cluster is ejb. This is the name of the cache-container that can be changed by renaming the cache-container attribute of the cluster passivation store on the server side (e.g., in the standalone-ha.xml configuration file).

The communication on the transport layer is based on JBoss Remoting, which in turn is based on XNIO. You can find the common XNIO configuration options for the transport in the org.jboss.xnio.Options implementation and the common remoting options in org.jboss.remoting3.RemotingOptions implementation.

As you can see, certain properties can not only be set for the whole cluster environment, but even for a particular server, thus overriding the cluster-wide settings. This allows fine-grained configuration of the cluster connections and individual nodes.

Conclusion

Hopefully, by now you have a basic understanding of the load-balancing and failover mechanisms of the EJB client library for remote EJB invocations.

The EJB client library can also be used if you need to invoke an EJB component from a remote server instance. The main difference is that an EJB client is configured in META-INF/jboss-ejb-client.xml rather than jboss-ejb-client.properties.

If you have any questions feel free to comment on this post or to send us an e-mail:

• lars.kuettner (at) akquinet.de
• heinz.wilming (at) akquinet.de

There will be two more blogposts in this series, the next about Clustered Messaging and the last about JGroups & Cloud issues.