As announced this is the last post of our series about clustering of the Redhat EAP 6 and JBoss AS 7. The other posts of this series were

• Clustering in JBoss AS7/EAP 6
• Managing cluster nodes in domain mode of JBoss AS 7 / EAP 6
• Scalable HA Clustering with JBoss AS 7 / EAP 6
• Load-balancing and failover of remote EJB clients in EAP6 and JBoss AS7
• Clustering of the messaging subsystem HornetQ in JBoss AS7 and EAP 6

Overview

This post will dig deeper into the clustering mechanisms of the EAP 6 and JBoss AS 7. We will show different solutions to multicast problems you will get in most cloud networks as well as some other networks. Infinispan uses JGroups to do its cluster communication. Cluster communication here means multiple things: finding other cluster nodes, providing a reliable transfer, implementing multicast communication even if there is no IP multicast available, identifying dead cluster nodes and a little bit more. In fact JGroups is able to do a lot more but Infinispan does not need all of the opportunities JGroups offers. The upcoming HornetQ version 2.3 which will be included in the EAP 6.1 will use JGroups for server discovery too. This post will explain the basic principles of JGroups and how to configure it in different network setups, especially most cloud networks.

JGroups

JGroups is a java framework for group communication which employs different network protocols like IP multicasts. IP multicasts are very important for clustered environments since they save a lot of time sending the same message to many recipients. On the other hand, IP multicasts lack a basic feature every TCP unicast has: reliability. If a TCP unicast does not reach its destination, it will be resent and if that does not work you will be informed about that. But IP multicasts are more like UDP unicasts, they fire and forget. That is one important point JGroups fixes. It takes some basic transport protocol like TCP or UDP and adds the missing features you would like to have, like reliability, failure detection or encryption.

JGroups messages run through a bunch of protocols that implement different features. These protocols are arranged in a stack that can be configured to your very special needs. At the bottom of the stack there will always be some transport protocol. In JGroups 3 there are three transport protocols available: TCP, UDP and TUNNEL. UDP uses UDP packages for unicast communication and IP multicasts for multicast communication. TCP uses TCP packages for unicast communication and one TCP package to each recipient for multicast communication. As you can see, TCP has a big performance drawback on multicast communication because it needs to send much more packages. TUNNEL is the last possible transport protocol and helps to penetrate firewalls. It uses a central router, called GossipRouter, which will distribute messages to their recipients. TCP packages are used by TUNNEL. The following figure illustrates how a multicast is performed with the different transports:

You can put many different other protocols on top of the transport protocol. You can arrange them in any order you want and even add the same protocol multiple times if needed. The protocols JGroups features can basically be divided into five categories:

• group membership: pbcast.GMS, PING, TCPPING, MPING, TCPGOSSIP, FILE_PING, JDBC_PING
• failure: MERGE2, FD, FD_ALL, FD_SOCK, VERIFY_SUSPECT
• reliable transmission: pbcast.NAKACK, UNICAST, RSVP
• state transfer: pbcast.STATE_TRANSFER, pbcast.STATE, BARRIER
• efficiency: FRAG2, COMPRESS, UFC, MFC, DAISYCHAIN
• various: ENCRYPT, AUTH, SEQUENCER, STABLE, RELAY

As you see there are a lot of protocols available and this is not a comprehensive list. In the next subsection we will explain some of the more interesting protocols. If you want to know about some other protocol, or want to dig in deeper, I suggest to read the JGroups Documentation.

Important Protocols explained

One first thing to mention is that there are sometimes two protocols with pretty much the same effects, one for unicast communication and one for multicast communication. You can put both onto the same stack, the unicast protocols will only apply to unicasts and the multicast protocols will only apply to multicasts.

pbcast.GMS: pbcast stands for probabilistic broadcast and GMS for Group Membership. This protocol will join new members into our group and remove members that left or are suspected to be dead. It is an essential part of every JGroups stack. But it does not implement a means of how new members of a group detect existing members, which is the task of the *PING protocols, which we will discuss later.

FRAG2 will fragmet big packages into multiple smaller ones, COMPRESS on the other hand will use a compression algorithm of configurable efficiency to cope with bigger packets. ENCRYPT, as the name suggests, will encrypt the packet. Note that the position on the stack will determine which protocol headers will get encrypted and which not, if you put it down right on top of the transport protocol nearly everything will be encrypted but this also means a loss of efficiency because more data has to be encrypted. AUTH achieves that only candidates which authenticate themselves correctly can join the group.

I already mentioned that the *PING protocols and TCPGOSSIP take care of detecting existing group members. PING is the most simple of these protocols, it requires UDP or TUNNEL as transport and uses IP multicasts to detect existing members. In the case of TUNNEL it uses simulated multicasts to detect existing members (more on this later). MPING does the same but requires TCP as transport. Both protocols have the advantage, that nobody needs to know anything in advance. TCPPING on the other hand uses a list of initial known hosts which are asked for who the other members of the group are. TCPPING, as its name suggests, requires TCP as transport. PING, MPING and TCPPING in general will not work through firewalls and NAT-routers. TCPGOSSIP is designed to work over the bounds of firewalls. It requires you to set up a so called GossipRouter which needs to be reachable from all hosts. Each host will ask the router for existing hosts. The GossipRouter is also required when you do not use UDP or TCP, but TUNNEL as transport. There are plenty more *PING protocols which use a common storage like a NFS storage, a cloud storage or a database to store a list of the group members. As for the GossipRouter you do not need to know any hosts in advance, but you do need to decide on the common storage.

Cloud issues

First of all, most clouds do not support IP multicasts. This eliminates PING and MPING from the list of possible discovery protocols as well as UDP as the transport protocol because UDP uses IP multicasts for multicast communication. Regarding the discovery protocols, we have got plenty of options with no major drawbacks. But regarding the transport protocol, not using IP multicasts is a major drawback. There are two options left: TCP and TUNNEL. TUNNEL has an interesting feature, most network load is not on the group members but on the GossipRouter because it simulates the multicast. With TCP on the other hand this simulation is done by the individual group members. You should consider the traffic-pricing-model of your cloud provider when choosing one of the two.

TUNNEL can be used in combination with PING. PING would normally send an IP multicast but in combination with TUNNEL it works differently: It sends the ping to the GossipRouter only. The GossipRouter will then distribute the PING to the various hosts, thus simulating the multicast. I. e. for TUNNEL, PING will be the easiest discovery protocol. Other discovery protocols, such as FILE_PING, JDBC_PING or S3_PING/RACKSPACE_PING, could also be a good choice but are not covered in this post. For TCP we will explain both options: TCP+TCPPING and TCP+TCPGOSSIP. And again other discovery protocols could also be good options. These are not covered here but in the JGroups documentation.

TCP+TCPPING

This is the easiest way to get around multicasts: You simply need some initial “known hosts” which are cluster nodes that are always up and listen on the same IP.

<subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="tcpping">
    <stack name="tcpping">
        <transport type="TCP" socket-binding="jgroups-tcp"/>
        <protocol type="TCPPING">
            <!-- A comma-separated list of hosts: host1[port1],host2[port2] the port is the one specified in the hosts jgroups-tcp socket-binding -->  
            <property name="initial_hosts">1.2.3.4[7600],2.3.4.5[7600]</property>
            <property name="num_initial_members">2</property>
            <property name="port_range">0</property>
            <property name="timeout">2000</property>
        </protocol>
        <!-- ... -->
    </stack> 
</subsystem>

TCP+TCPGOSSIP

For TCPGOSSIP you will need a GossipRouter. The GossipRouter is a Java application from the JGroups project which will help you to build a cluster over different networks without multicast support. It is especially important in cases where the nodes are separated by firewalls. When using TUNNEL as transport, every message will be sent to the GossipRouter and the GossipRouter will distribute the message to its recipients. It will also be needed when using TCPGOSSIP to detect initial members. To start a GossipRouter use the following command:

java -classpath $JBOSS_HOME/modules/org/jboss/logging/main/jboss-logging-3.1.1.GA.jar:$JBOSS_HOME/modules/org/jgroups/main/jgroups-3.0.9.Final.jar org.jgroups.stack.GossipRouter -port 12001

Obviously, the last parameter allows you to configure the port on which the GossipRouter will listen for registrations. You can omit the last parameter. The GossipRouter will then use the default port which is 12001. You can also use multiple GossipRouters. Every host would then register itself at all routers but only use the first available router for requests. This adds availability to the GossipRouter.

Apart from starting a GossipRouter you will also need to configure your JGroups stack to use it:

<subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="tcpgossip">
    <stack name="tcpgossip">
        <transport type="TCP" socket-binding="jgroups-tcp"/>
        <protocol type="TCPGOSSIP">
            <!-- A comma-separated list of GossipRouters: router1[port1],router2[port2] --> 
            <property name="initial_hosts">1.2.3.4[12001],2.3.4.5[12001]</property>
            <property name="num_initial_members">2</property>
            <property name="timeout">3000</property>
        </protocol> 
        <!-- ... -->
    </stack> 
</subsystem>

TUNNEL+PING

To use TUNNEL you will again need to start one or more GossipRouters as explained above. The GossipRouter is the key for firewall penetration, it must be reachable from all hosts and will then manage the group’s communication. To show the use of multiple GossipRouters, the following example-configuration defines two of them:

<subsystem xmlns="urn:jboss:domain:jgroups:1.1" default-stack="tunnel">
    <stack name="tunnel">
        <transport type="TUNNEL" shared="false">
            <!-- A comma-separated list of GossipRouters: router1[port1],router2[port2] --> 
            <property name="gossip_router_hosts">1.2.3.4[12001],2.3.4.5[12001]</property>
        </transport>
        <protocol type="PING"/> 
        <!-- ... -->
    </stack> 
</subsystem>

How to choose the best transport+discovery protocols

First of all try to keep everything as simple as possible. If possible UDP in combination with PING will be the best choice. If not, as in the cloud, the transport should be TCP. TUNNEL could perform better when the GossipRouter has a much better network connection than the individual hosts. But your GossipRouter can also become a bottleneck. So in general TCP will be the better choice. If you need to penetrate firewalls TUNNEL will the choice.

For the discovery protocol it is not so simple. When using the TUNNEL protocol, PING will mostly be the best choice because it will work without any additional configuration. For the TCP protocol, it depends: If you have all your servers at the same location and there will always be some specific servers up, TCPPING seems to be a good choice. If this is not the case, it depends on what you have and what option is the most cost-effective. If you already have some storage or database up you can use it for this purpose. TCPGOSSIP may be a possibility when you have a machine with a public IP available for free (you will get the few bytes that will be used in a cloud storage nearly for free). And as already mentioned for UDP the best choice is PING.

Security

The first thing to take into account is that by default, most communication in our clusters is unencrypted. This is especially bad when our load balancer redirects request over the internet to another server farm or if replication traffic traverses insecure networks. You need to secure GossipRouters or “known hosts” for TCPPING so that only authorized members can join your cluster. One option is to encapsulate your whole infrastructure into a VPN. On first glance it may be an easier solution to add AUTH and/or ENCRYPT to your JGroups stack. AUTH will keep unauthorized hosts out of your JGroups communication and ENCRYPT will render the traffic unreadable for attackers. But at least AUTH is not a state of the art implementation and is by now vulnerable to replay attacks. So it seems more secure to leave AUTH/ENCRYPT behind and get back to the VPN solution. This will also put the cryptography computations on hardware optimized for that purpose. Note that often VPNs do not support multicasts over the tunnel. So to get through the VPN tunnel you may need to use a GossipRouter (i.e. use TUNNEL as transport).

Note that if you leave JGroups unsecured, an attacker can read and even modify your Infinispan caches. A GossipRouter as well as a “known host” is a very good target for a DOS attack. The GossipRouter could be an even better target than any known host because if you use TUNNEL and there is only one GossipRouter, an attacker will, by killing it, provoke a fatal split-brain situation for the whole cluster. So your GossipRouter as well as any host running TCPPING should be well secured against DOS attacks (i. e. only allowing traffic from specific origins, incoming traffic limits, maybe deep packet inspection).

Summary

Don’t forget that, if there are IP multicasts available, there can be no better performance than with UDP and if there are no IP multicasts available the best advice is to use what you already have, e. g. a database or a (cloud-) storage. GossipRouters should in most cases be your last consideration, which does not mean they are bad.

Even if there are many interesting topics uncovered, this is likely to be the last post of this series about clustering the EAP 6 or AS 7. Thank you very much for your feedback on the last posts! If there are any questions or comments on this post feel free to leave a comment or write an e-mail to

• heinz.wilming (at) akquinet.de
• immanuel.sims (at) akquinet.de