I’m currently looking into an old bug, TAV-480, that seems to have reappeared. This bug is related to Raven, our classloader system based on Maven that allows us to do plugins and network updates. I’ll come with another post about this bug, but first let’s do an introduction of Raven.
The idea of Raven is quite simple, we use Maven to deploy all our modules, and all of our dependencies are also available either from our own or from official third-party Maven repositories. One thing we decided to support from Taverna 1.5 was plugins and dynamic updates over the web. How could we do this?
We decided to reuse the existing Maven infrastructure also at runtime. To install a plugin would then simply be a matter of specifying which Maven artifacts are needed, and from which Maven repositories to get the artifacts, such as in the plugin description for the t2 preview plugin. So we built Raven, which although inspired by Maven doesn’t use any Maven code.
Given a Maven artifact as described in the plugin file:
<artifact groupId="net.sf.taverna.t2" artifactId="biomart-activity" version="0.2.0"/>
Raven fetches the POM and JAR files for
biomart-activity from the repository, parses the Maven artifact description file
biomart-activity-0.2.0.pom, and then does the same for each of the dependencies listed, and with their dependencies again, and so on until everything required has been downloaded.
What remains is to get all these JARs available on the classpath. The normal way to do this is to use a
ClassLoader, such as the
URLClassLoader. You can’t normally modify the classloader Java gives you at startup, because it has been determined from the
-classpath parameter and you can’t officially add new URLs to it once it’s constructed. However you are free to construct a new classloader and load classes from there.
This is what the bootstrapper asks Raven to do. Since we can have several plugins, each plugin would need its own classloader. The plugins can come from different third-parties (you can add plugin sites to Taverna), and they might depend on different versions of common artifacts such as jdom. In some cases these versions are not compatible.
What we ended up with as a solution was that each Maven artifact gets its own
LocalArtifactClassLoader. This classloader also have a list of the dependencies as declared in the POM, and when asked to resolve a classname, it will first search each of it’s dependencies. As the dependencies have the same kind of classloaders, they will use the same logic, and the search will be depth-first. If none of the dependencies have the class, then maybe the JAR file associated with this artifact does, so it does a normal search through its own superclass
URLClassLoader which searches the single JAR file. If this fails, a
ClassNotFoundException is thrown and whoever depended on this artefact would go on to check the next dependency.
When you download Taverna these days, what you get in the zip-file is a tiny bootstrapper class together with a configuration file raven.properties that says which are the global Maven repositories, and which profile should be used for the main program. The profile version 220.127.116.11 basically says what is Taverna 18.104.22.168 by listing all the required artifacts, very similar to the plugin definition. For instance you would find all the different processors listed here.
If we decide that we need to publish an update of a certain processor, we can just deploy the new version to the Maven repositories, and publish a new version of this profile and list it in the profile list that Taverna checks on startup.
Although this solution came with a few quirks we had to work out (for instance many of the official POMs didn’t correctly state what where the true dependencies of the library), in general it’s a nice solution with lots of possibilities, for instance it doesn’t prevent you to have two versions of the same artifact at once, so you could theoretically build a workflow that used both an old and a new version of the
wsdl-activity – if for instance we had fixed a bug that you suspect might have affected some old workflow runs and you want to compare the outputs.
However, one of the difficulties with doing all of the ClassLoader work ourself is that as this is quite low-level and hard-core Java with many tiny little pitfalls to worry about and lots of concepts to understand. I’ll come back in the next post with the example related to our bug TAV-480.
If you are interested in using Raven in your own project, contact us on taverna-hackers!