Writing modular Java applications (a suggestion for a simple, but versatile plugin architecture)

Some things sound trivial to do, but on closer look turn out to be a formidable brainteaser. One of them is writing modular Java programs, where the core program can be extended using plugins. Such a design might be desirable, when extra program functionality is either to be licensed separately, to be contributed by third party, or generally "to be decided later".

In an ideal world, installing a plugin should simply be a matter of downloading a JAR file, dropping it in some directory and afterwards the contained classes would automagically appear ready to use on the classpath. From the application programmer's point of view, there should be no difference between writing code for the core application or a plugin. A plugin would just be another (optional) package, from which classes can be loaded via SPI (Service Provider Interface) or via beans API, if something more sophisticated is required.

To make it this seamless, the application would have to be able to dynamically add JAR files to the classpath. Unfortunately, this is exactly what is not supported by the JRE, so what are the alternatives?

From the programmers point of view, the simplest solution to the problem would be to just ask the user to include all required jarfiles to the classpath. In reality however, this is not going to work. Most users are barely tech savvy enough to click an icon on the desktop. The idea of asking them to open a shell and type something like:

cd whereever_the_jar_file_is
java -cp core.jar:plugin1.jar:plugin2.jar MainApplication

Is doomed to fail miserably. Providing a shellscript wrapper to do the job isn't going help either, because then you'll have to bundle several platform dependant files with the application and ask the user to pick the correct one. Since this is essentially going nowhere, next idea please.

According to Sun's docs, a directory called "ext" exists in $JAVA_HOME. Any jarfiles found in this directory are automatically included into the classpath. This would be perfect, if this was not a system directory. Users may not have write privileges there and it is not really meant to hold application specific extensions at all, so this is not the way to go either.

It seems like going with the flow might be a solution. The JRE contains a class java.net.URLClassLoader, which might do the trick. In fact, a common way to load additional code from within a running application typically employs some variant of the following code:

import java.net.URLClassLoader;
  
public class PluginLoader {
  
  public static URLClassLoader myLoader;
    
  public PluginLoader() {
    myLoader = URLClassLoader.newInstance(scanPluginDir());
  }
   
  private URL[] scanPluginDirectory() {
    // Do your thing
  }
}  

This works nicely at first, but has some annoying shortcommings:

• You always have to carry PluginLoader.myLoader around and remember to use it instead of the system classloader whenever you want something from a plugin jar file. This is not very convenient. It adds complexity, where seamless operation is desired.
• The method ClassLoader.getResource() (or one of it's siblings) is generally used to retrieve resources from JAR files. This method will delegate to the parent classloader first, before trying to find the resource itself. You might end up loading the wrong file from the wrong JAR without noticing it. This is what makes having two active classloaders in the same application error prone.
• The scanDiretory() method above needs to know where the plugin directory is in the first place. In most cases it will be a path relative to the main jar file. However, the JRE does not provide a reliable method to figure out where this might be. The search path for plugins must either be submitted by the user or found in attempt to parse the java.classpath system property.
• You cannot add more URLs to an already instantiated URLClassLoader. This makes adding more plugins at runtime without restarting the application impossible. The only way around this is to give each plugin it's very own classloader, which adds more implications to consider.
• Loading classes from plugins via SPI is especially long winded.

While using java.net.URLClassLoader is a workable solution for the problem of loading plugins, it is also a complicated one with lots of pitfalls and anything but seamless. There should be a more handy way to do it.

Next stop: Jar File Specification. The Manifest file may contain a "Class-Path:" attribute, which lists relative URLs of libraries or extensions needed by the application. Simple enough, whenever a plugin is installed, just update the Manifest file in the main JAR file. Only one problem here: This is not a trivial thing to do. JAR files are essentially ZIP files. They can only be accessed sequentially and the Manifest file is always the first entry in the archive. Updating it more or less requires the whole JAR to be rebuild. This is not exactly hassle free and opens a can of worms.

It seems like none of the methods discussed so far is particularly appealing (though the last one shows potential) and unfortunately they pretty much exhaust the list of available options. Time to reconsider some of the basic requirements and to commit a bit of heresy, then.
Do Java applications really have to be bundled as JAR file? Actually, no! JAR is just the method of choice for deploying applications in a single file. But with the decision to support plugins, this goal cannot be accomplished anymore anyway. So, why not break with the paradigm completely and let (at least some parts of the application) sit as loose files on the native filesystem?

The only part of the application that actually has to be bundled as a JAR file is the application launcher itself, for the single purpose of interfacing with the operation system. All remaining files can then reside in directories of the native filesystem, which are referenced in that JAR. Consider the following Manifest file for illustrating purposes:


Manifest-Version: 1.0
Main-class: Launcher
Class-Path: resources/ code/main/ code/plugins/ code/spiregistry/


Along with this basic directory structure:


. Where the application launcher is.
|-- code
| |-- main Contains the code of the core application.
| |-- plugins Contains the code of the plugins.
| -- spiregistry For registering plugins via SPI.
|-- META-INF
| -- services-- resources Any media files required by the application.


The principle idea here is, that the JAR file only serves as glue between operation system and the actual application. It provides the means, by which the user can start the program, then sets up a suitable classpath and finally passes control over to the real application, which resides as loose class files below the code/main directory. Since the classpath references directories instead of JAR files, plugins can then be installed by copying their files to the code/plugins directory and registering them under code/spiregistry/META-INF/services, using the SPI mechanism. Example code, illustrating this setup can be found here.

Conclusion: The method introduced above allows application programmers to write code for plugins the same way as for the main application, without having to worry about classloader issues. Plugins are simply packages, that can either be present or absent from the classpath. This also makes it easy to incorporate code from a plugin into the core or to phase out code from the core into a plugin. It is also a JRE only solution, no third party libraries required.