Setup Camel Message Integrations

The Fedora Repository makes it possible to design custom event-driven application workflows. For instance, a common task involves indexing content in an external search engine or triplestore. Other repositories may wish to generate derivative content such as creating a set of smaller images from a high-resolution master.

Because Fedora publishes modification events on a JMS topic using a local ActiveMQ broker, one can write custom listener applications to handle these various workflows. By default, the repository's JMS broker supports both the OpenWire and STOMP protocols, which means that it is possible to write client listeners (consumers) in a wide variety of languages, including PHP, Python, Ruby and JAVA, among others.

For simple message-consuming applications, writing special-purpose applications may be an excellent choice. In contrast, once a repository begins making use of more complex message-based workflows or when there are multiple listener applications to manage, many repositories use systems such as Apache Camel to simplify the handling of these messages.

Camel makes use of "components" to integrate various services using a terse, domain specific language that can be expressed in JAVA, XML, Scala or Groovy. There exists one such component designed to work specifically with a Fedora4 repository. This makes it possible to model Solr indexing in only a few lines of code like so:

from("activemq:topic:fedora")
  .to("fcrepo:localhost:8080/fedora/rest")
  .filter()
  .xpath("/rdf:RDF/rdf:Description/rdf:type[@rdf:resource='http://fedora.info/definitions/v4/rest-api#indexable']", namespaces)
  .to("fcrepo:localhost:8080/fedora/rest?accept=application/json&transform=default")
  .to("http4:localhost:8080/solr/core/update");

In this example, the XPath filtering predicate is just an example; you can, of course, use many different Predicate languages, including XQuery, SQL or various Scripting Languages.

This same logic can also be expressed using the Spring XML extensions:

<route>
  <from uri="activemq:topic:fedora"/>
  <to uri="fcrepo:localhost:8080/fedora/rest"/>
  <filter>
    <xpath>/rdf:RDF/rdf:Description/rdf:type[@rdf:resource='http://fedora.info/definitions/v4/rest-api#indexable']</xpath>
    <to uri="fcrepo:localhost:8080/fedora/rest?accept=application/json&transform=default"/>
    <to uri="http4:localhost:8080/solr/core/update"/>
  </filter>
</route>

Or, in Scala:

val xpath = new XPathBuilder("/rdf:RDF/rdf:Description/rdf:type[@rdf:resource='http://fedora.info/definitions/v4/rest-api#indexable']")
xpath.namespace("rdf", "http://www.w3.org/1999/02/22-rdf-syntax-ns#")
"activemq:topic:fedora" ==> {
    to("fcrepo:localhost:8080/fedora/rest")
    filter(xpath) {
        to("fcrepo:localhost:8080/fedora/rest?accept=application/json&transform=default")
        to("http4:localhost:8080/solr/core/update")
    }
}

Please note that the hostnames used for fedora and solr here are entirely arbitrary. It is quite likely that these systems will be deployed on separate hosts and that the Camel routes will be deployed on yet another host. Camel makes it easy to distribute applications and replicate data asynchronously across multiple hosts

Examples

Deployment

Camel routes can be deployed in any JVM container. In order to deploy to Jetty or Tomcat, the route must be built as a WAR file. This command will get you started:

  $> mvn archetype:generate \
      -DarchetypeGroupId=org.apache.camel.archetypes \
      -DarchetypeArtifactId=camel-archetype-war \
      -DarchetypeVersion=2.14.0 \
      -DgroupId=org.example.camel \
      -DartifactId=my-camel-route \
      -Dversion=1.0.0-SNAPSHOT \
      -Dpackage=org.example.camel

After the project has been built (mvn install), you will find the WAR file in ./target. That file can simply be copied to the webapps directory of your Jetty/Tomcat server.

Another popular deployment option is Karaf, which is a light-weight OSGi-based JVM container. Karaf has the advantage of supporting hot code swapping, which allows you to make sure that your routes are always running. It also allows you to deploy XML-based routes (Spring or Blueprint) by simply copying the files into a deploy directory.

Karaf can be set up by:

1. downloading Karaf from an apache.org mirror
2. running ./bin/karaf to enter the shell
3. installing required services:
   
   $> feature:repo-add camel 2.14.0
   $> feature:repo-add activemq 5.10.0
   $> feature:install camel
   $> feature:install activemq-camel
   
   # display available camel features
   $> feature:list | grep camel
   
   # install camel features, as needed
   $> feature:install camel-http4
   
   
4. setting up a service wrapper (so that karaf is always running)
   
   $> feature:install wrapper
   $> wrapper:install
   
   
5. following the directions provided by this command

Now, routes can be deployed (and re-deployed) by simply copying JAR files or XML documents to $KARAF_HOME/deploy.