Introducing Dynamic Discovery into OSGi distributed applications

This post is the second post of the dynamokos series. It improves the application made in the first post by adding dynamic remote service discovery.

In this post

The former post ends with an issue. The application behave strangely when the oracle is stopped. In this post, we start to fix the issue. To achieve that, I describe how to use a discovery protocol in order to avoid to statically import the service. We will see:

  • How A discovery service works
  • How we can use Zookeeper as a discovery protocol
  • How to publish a service on Zookeeper
  • How to discover a remote service using Zookeeper

Background

The dynamokos series aims to build a dynamic distributed web application on the top of OSGi.

The first version relies on Apache CXF DOSGi to distribute OSGi services as web services. The first framework hosts an Oracle. The Prediction Service (provided by the Oracle) is published as a web service using CXF.
The second framework imports this service, and a web application uses it.

Current architecture of the dynamokos application

Current architecture of the dynamokos application

The main issue in this application is the ad-hoc importation. The importer indicates to DOSGi the location of the Prediction service. But nobody checks the availability of the service. So, despite a potential unavailable Prediction Web Service, the web page tries to use it.

Dynamokos Web View

Dynamokos Web View

Architecture

New architecture of the system

New architecture of the system

This version reuses the same technologies as the one from the first post. It just introduces discovery:

  • The Zookeeper Server is a registry containing all exported services
  • The Zookeeper Discovery is a bundle (also developed in Apache CXF DOSGi) interacting with the Zookeeper server to publish and discover remote services.
  • We use the Apache Felix Configuration Admin to configure Zookeeper.

What’s a Discovery Protocol?

A discovery protocol is a way to notify other platforms of the availability of services (remotely accessible). There are a lot of technologies usable as discovery protocol:

  • Multicast events notifying arrivals, departures and supporting probing
  • Simple Service Discovery Protocol (SSDP)
  • WS-Discovery (used in DPWS)

Some rely on centralized registries (like in Jini), some just on network events.

The OSGi Distributed Service specification defines how the Distributed Service implementation (in our case DOSGi) deals with Discovery Services without imposing any discovery protocol. That’s pretty nice as we can use a lot of different discovery protocols (depending of the use case).

Apache Hadoop Zookeeper

In Dynamokos, I decided to use (at least for the time being) Apache Hadoop Zookeeper as discovery protocol. The good news is that DOSGi supports it.

Zookeeper is a high-performance coordination service for distributed applications. It exposes common services – such as naming, configuration management, synchronization, and grouping.

Zookeeper will be used as a global registry publishing services exported remotely. But before being able to use it, we need to launch it.

To this end, download the dynamokos-runtime archive from here. Then unzip it and go (with a terminal) into the runtime folder.

From there launch one of the following commands depending if your using Linux/Mac OS or windows:

#Linux : 
sh ./zookeeper/bin/zkServer.sh start
#Windows: 
zookeeper\bin\zkServer.cmd start

Note: Zookeeper requires Java 6.

Those commands start a preconfigured Zookeeper server (on port 2181). You should see:

JMX enabled by default
Using config: /Users/clement/workspaces/dynamokos/ 
        dynamokos/runtime/zookeeper/bin/../conf/zoo.cfg
Starting zookeeper ... 
STARTED

Publishing the Prediction service in Zookeeper

Let’s now try to publish our Prediction Web Service into Zookeeper. No code required!

The only requirement is to deploy the Zookeeper discovery service and to configure it correctly.

Configuring the Zookeeper discovery service relies on the OSGi Configuration Admin. Don’t panic. The only things to do is to create the org.apache.cxf.dosgi.discovery.zookeeper.cfg file in the oracle-dosgi-zookeeper directory with:

zookeeper.host = 127.0.0.1

This org.apache.cxf.dosgi.discovery.zookeeper.cfg file will be read by Apache Felix File Install and pushed to the OSGi Configuration Admin. Then the configuration admin pushed this configuration to the Zookeeper discovery service. Once received, this service is able to contact the Zookeeper server to register the Prediction service.

Let’s check that.

From the runtime folder, now launch the Oracle platform with:

java \
    -Dfelix.config.properties= \
        file:./conf/oracle-dosgi-zookeeper.properties \
    -jar bin/felix.jar cache/oracle-dosgi-zookeeper \

This launches the Oracle platform with everything configured (Zookeeper is already configured). You should read something like:

INFO: Creating ZooKeeper node: 
    /.../Prediction/172.20.201.81#9090##oracle

Discovering the Prediction Service

It’s time to configure the client platform to discover the remote Prediction service.
No remote-services.xml file!
As we are using Discovery here, there is no static remote service metadata required. So no remote-services.xml file. Discovery will dynamically inform DOSGi of the locations of available remote services. The DOSGi implementation will in turn register proxies for the remote services in the local service registry. So, this completely avoids the static import.

Let’s try that.

Go back to the runtime folder and execute the following command:

java \ 
    -Dfelix.config.properties= \
    file:./conf/client-dosgi-zookeeper.properties \
   -jar bin/felix.jar cache/client-dosgi-zookeeper

This launches the client platform with the zookeeper discovery service (configured with the same configuration). Note, that there is no more the oracle importer bundle, as the static configuration is no more required.
You should see something like:

INFO: Notified - AVAILABLE: [....Prediction] 
    endpoint id: eb461f59-5250-4dd7-8e09-b17544ca4e9b

Open your browser to: http://localhost:8080/dynamokos/index.html
You should see:

Dynamokos Web View

Dynamokos Web View


Now, you can ask metaphysical questions to the oracle.

Let’s play with the dynamism

Ok, that’s nice but we haven’t even seen the really interesting stuff.

On the oracle platform, stop the oracle:

START LEVEL 1
   ID   State         Level  Name
[   0] [Active] [    0] System Bundle (2.0.0)
[   1] [Active] [    1] Distributed OSGi Distribution ...
[   2] [Active] [    1] Apache Felix Bundle Repository (1.4.1)
[   3] [Active] [    1] Apache Felix Configuration Admin Service (1.0.10)
[   4] [Active] [    1] Apache Felix File Install (1.2.0)
[   5] [Active] [    1] Apache Felix iPOJO (1.4.0)
[   6] [Active] [    1] Apache Felix iPOJO Arch Command (1.4.0)
[   7] [Active] [    1] Apache Felix Shell Service (1.4.0)
[   8] [Active] [    1] Apache Felix Shell TUI (1.4.0)
[   9] [Active] [    1] OSGi R4 Compendium Bundle (4.1.0)
[  10] [Active] [    1] Apache Log4J (1.2.15)
[  11] [Active] [    1] Oracle (0.0.1.SNAPSHOT)
[  12] [Active] [    1] CXF Zookeeper-based ...
[  13] [Active] [    1] Prediction Service Interface (0.0.1.SNAPSHOT)
[  14] [Active] [    1] ZooKeeper wrapper bundle (1.1.0.SNAPSHOT)
-: stop 11

You should see:

INFO: Removing ZooKeeper 
    node: /...Prediction/172.20.201.81#9090##oracle

Go back to your browser and update the page … 404!

No web page when the oracle is out of his office

No web page when the oracle is out of his office


What happened is very simple:

  1. The Prediction service is withdrawn from Zookeeper.
  2. Zookeeper notifies the client Zookeeper discovery service of this departure
  3. The discovery service propagates this info to DOSGi
  4. DOSGi deletes the created proxy, and un-registers the Prediction service from the OSGi service registry
  5. The web application is invalidated and the web page is unregistered from the HTTP Server

Now if you restart the oracle:

start 11

And then, refresh the page… our oracle is back!

The Oracle is back

The Oracle is back

Issues and conclusion

Well, this version fixed the oracle availability issue, but the result is not great (maybe worse) as the user now just gets a 404. What would be uber-cool is if we would be able to handle the dynamism inside the web app itself…

That’s the content of the next post. Stay tuned!

If you have questions, feel free to post a comment on this post or to contact us.

11 Responses to Introducing Dynamic Discovery into OSGi distributed applications

  1. [...] second post modifies the first example to use a discovery protocol (to find the remote oracle. The web site is [...]

  2. Really nice article! I put a link to it from the CXF-DOSGi documentation site: http://cxf.apache.org/dosgi-presentations-and-articles.html

  3. [...] Introducing Dynamic Discovery into OSGi distributed applications … [...]

  4. blog09ats says:

    Thanks David !

  5. web proxy says:

    I really enjoyed the article. The information will help me with a redesign of a web app that I was tasked with

  6. weishan says:

    Great post, but I got this problem as attached, may i know how to solve this?

    D:\test\Dynamokos>zookeeper\bin\zkServer.cmd start

    D:\test\Dynamokos>java “-Dzookeeper.log.dir=D:\test\Dynamokos\zookeeper\bin\..”
    “-Dzookeeper.root.logger=INFO,CONSOLE” -cp D:\test\Dynamokos\zookeeper\bin\..\co
    nf;D:\test\Dynamokos\zookeeper\bin\..\zookeeper-3.1.1.jar;D:\test\Dynamokos\zook
    eeper\bin\..\lib\log4j-1.2.15.jar org.apache.zookeeper.server.quorum.QuorumPeerM
    ain D:\test\Dynamokos\zookeeper\bin\..\conf\zoo.cfg start
    Exception in thread “main” java.lang.NoClassDefFoundError: org/apache/zookeeper/
    server/quorum/QuorumPeerMain
    Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.server.quorum.
    QuorumPeerMain
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
    Could not find the main class: org.apache.zookeeper.server.quorum.QuorumPeerMain
    . Program will exit.

    D:\test\Dynamokos>endlocal

  7. Hmmm – looks like there is an issue with your classpath. And it also looks like the …\zoo.cfg argument before ‘start’ is incorrect.

    I got it running with the following command:
    C:\technology\zk\zookeeper-3.1.1\bin>java -Dzookeeper.log.dir=C:\technology\zk\z
    ookeeper-3.1.1\bin\.. -Dzookeeper.root.logger=INFO,CONSOLE -cp C:\technology\zk\
    zookeeper-3.1.1\bin\..\conf;C:\technology\zk\zookeeper-3.1.1\bin\..\zookeeper-3.
    1.1.jar;C:\technology\zk\zookeeper-3.1.1\bin\..\lib\log4j-1.2.15.jar org.apache.
    zookeeper.server.quorum.QuorumPeerMain 2181 start
    2011-03-09 07:57:38,498 – INFO [main:Environment@97] – Server environment:zooke
    eper.version=3.1.1-755636, built on 03/18/2009 16:52 GMT
    … etc …

  8. weishan says:

    Thank you, David. Yes. The thing is that i did not notice that there comes zookeeper 3.2.1.jar inside the zookeeper directory, but in the environment setting, there is 3.1.1.jar is used, that is why I got that problem. (so a small inconsistency in your run time package)

  9. Michele says:

    Nice article!!!!
    Just one more question about the endpoint ID of the remote service. If your remote service is published by a machine within a private network, how do you solve these issues :
    endpoint ID: you should publish a public endpoint instead of a private one
    port forwording: as soon as you publish the remote service, you should also configure the router/firewall in order to accept incoming connection to your brand new endpoint ID

  10. Clement Escoffier says:

    Hi Michele,

    The idea is to trigger a notification when a new service is exposed and to configure the firewall and the router. A lot of firewalls and routers can be configured using web services and rest endpoints making easy to configure remotely.

    We have already deployed such mechanism where the notification was propagated using JMS.

  11. Michele says:

    Hi Clement,

    thank you for your suggestion!!
    But what about the endpoint ID published by the machine exposing the remote service? The ZooKeeper entry should contain a reachable ip address and not a private one. If I set for the property org.apache.cxf.ws.address with gw ip of my machine (that is configured with a private ip) jetty throws an exception related to the specified ip address.
    Any suggestions?

    Thanks again.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 202 other followers

%d bloggers like this: