Eclipse Equinox with Java Modules All the Way Down

The Eclipse Equinox project provides a community for developing implementations of various OSGi specifications. At the center of Equinox and the OSGi specification is a Dynamic Module System for Java^tm which is provided by the OSGi Framework. This module system is used by many other Eclipse projects to modularize their code and to provide extensibility. An exemplary project is the Eclipse project which uses Equinox to provide an extensible platform for developing Eclipse plug-ins.

With the Java 9 release, the Java Platform Module System (JPMS) is coming. The JPMS will finally modularize the class libraries provided by the JVM. In addition, the JPMS can be used by developers to modularize applications. This allows developers to split their applications into modules. These modules can then specify what other modules they require and what packages they export for use by other modules.

The OSGi specification has been providing a module system for Java applications for a long time already which also allows developers to modularize their applications into modules (a.k.a. bundles). With OSGi, developers have created many modular applications that are also extensible by installing more bundles provided by third parties.

When JPMS is released, developers can start to deliver their own Java modules for the JPMS. What happens when developers want to use Java modules to compose applications which are running on a container that is built using the OSGi module system? Will containers be able to provide APIs in a way that JPMS modules can require and access them? This newsletter focuses on this scenario which is similar to the Eclipse project. Before going into the details of the problem we should first explore the JPMS and the concept of Layers. Note that the details that follow are how things work in JPMS at the time of this writing. The JPMS is still in the process of being developed and some of these details may change.

JPMS Layers and Modules

A layer in JPMS is a static set of resolved modules. Each layer has a single parent layer except the empty layer which has no parent. Layers are hierarchical and can have no cycles. A layer provides the JVM with a graph which determines how classes are located during class loading. Once a layer is created, none the modules within the layer can change. This allows the class loading graph to be locked in when the layer is created. Modules in one layer can require any module provided in a parent layer. This includes all parents in the hierarchy all the way down to the empty layer. In order to update a module, the complete layer in which a module is contained must be thrown away and recreated in order to provide a new resolution graph for the layer. If any module within a layer in the hierarchy needs to be updated, then that layer as well as any children of that layer must be torn down and recreated. If you need to load up another module provided by JVM in the boot layer, then the complete JVM has to be restarted so that the JVM can recreate the boot layer.

This provides a stable and predictable class loading behavior but it poses a problem for containers that are built using the OSGi module system. The OSGi module system is much more dynamic. Modules (bundles) in OSGi are not placed in hierarchical layers which are resolved in orderly stages like the JPMS layers. In OSGi, bundles that have no dependency on each other can be resolved independently in time of each other. Not only that, but new bundles can be installed and existing bundles can be updated or uninstalled. All this without tearing down the Framework or affecting existing bundles within the framework that do not depend on the other bundles being updated, installed or uninstalled.

How would a container built on a dynamic module system be able to provide a JPMS layer which can be used as a parent of a another layer which contains JPMS modules? Modules in JPMS can only load classes from packages exported from modules within their own layer or one of the layers in their parent hierarchy. If a container is providing APIs which are exported by OSGi bundles then any API which can be used by applications composed of JPMS modules must be represented within a JPMS layer somehow. The following diagram illustrates the possible layers with this scenario:

The boot layer contains the JPMS modules which were configured with the JVM when it was launched. In this diagram, the framework launcher has also been migrated to Java 9 in order to have it create a Layer for the class loader used to load the framework implementation. This layer configures a single module named system.bundle. This allows all the classes for the Framework implementation to be associated with the system.bundle module. Next is the bundle layer. This layer is configured to map each bundle class loader to a named module representing the bundle. Finally we have a module layer which uses all the built-in module class loaders of Java 9 for JPMS.

OSGi JPMS Layer

A github project (OSGi-JPMS layer) investigates this approach. One goal of the project is to not require any modifications to the OSGi framework implementation by using only OSGi specified APIs. This approach uses a bottom up strategy for JPMS modules. With that in mind the first thing needed is to modify an OSGi Framework launcher to create the system.bundle module.

The system.bundle Module

In OSGi the system bundle represents the OSGi Framework implementation as a bundle. In an OSGi Layer each bundle should be represented by a named module. This includes the bundle named system.bundle. Without this all classes which implement the OSGi Framework itself would end up associated with something called an unnamed module. An unnamed module has many limitations in JPMS, but the one that impacts the OSGi JPMS Layer the most is the fact that unnamed modules cannot be depended on by other modules in JPMS.

The Framework launcher therefore needs to create a Layer which can contain the system.bundle module. For details required to create the system.bundle layer refer to modifications done to the Equinox launcher here.

The Bundle Layer

The Bundle Layer implementation discovers all resolved host bundles wirings and maps them to a named module. The following information is used from the bundle wiring for the respective JPMS module:

• The bundle symbolic name is used as the module name.
• The bundle version is used as the module version.
• The exported packages are used as the module exports.
• The private packages are also used as the module exports.
• Dependencies on other bundles for class loading must become module requires.

The mappings for module name, version and exports are fairly easy to understand. But the need for private packages to be exports and for OSGi dependencies to become module requires are not. These two mappings are needed to work around some rules enforced by JPMS at runtime.

All private packages must be exported to work around the fact that JPMS will not allow reflection on classes from another module unless that package is exported. And more recently JMPS is enforcing checks for deep reflection to be limited to packages exported as private. Reflection is an important tool used by containers such as the Eclipse extension registry and OSGi declarative services. In order to maintain existing behavior in OSGi all packages contained in a bundle must be exported as private in the bundle Layer.

The OSGi dependencies are mapped to module requires so the bundle class loaders can read the other modules they are wired to during OSGi bundle resolution. The JPMS enforces read access at runtime and will prevent a module from executing code from another module it does not have read access to.

This mapping of private exports and OSGi dependencies does impose some restrictions from JPMS onto the OSGi framework:

1. JPMS does not allow cycles. Representing OSGi resolution cycles will result in an error when the bundle layer is created.
2. JPMS does not allow for split packages. Representing split packages allowed in OSGi will result in an error when the bundle layer is created. The fact that all private packages from OSGi must be exported from the JPMS module makes split packages more likely.
3. JPMS layers have static resolution. OSGi has dynamic package resolution which cannot be known before creating the bundle layer.
4. Private packages must be discovered eagerly at bundle layer creation. Private package information is not always known upfront. It may be costly to discover all private packages upfront.

OSGi Bundle Dynamics

The bundle layer represents a static set of resolved OSGi bundles in a Framework. But the bundles in an OSGi Framework are not static. They can be uninstalled, updated, re-resolved, and new bundles can be installed. How can this dynamic nature be represented in JPMS layers? The approach the OSGi Bundle Layer uses is to create a linear graph of layers where the youngest child layer represents the current state of the bundles. This would look something like this:

This scenario started out with bundle.a and bundle.b resolved in the bundle layer 1. Then module layer 1 is created to resolve jpms.a and jpms.b modules. After that bundle.b is updated and bundle.c is installed and bundle.b is refreshed in order to flush out its old content and class loader. This leaves bundle layer 1 with a "dead" bundle.b module which also makes module layer 1 stale. Now module layer 1 must be discarded and module layer 2 must be created for jpms.a and jpms.b modules. To do that a new bundle layer is created that represents the current set of resolved bundles.

Bundle layer 1 cannot be discarded because it still has at least one valid module bundle.a. The bundle.a module cannot be represented in a new layer because classes may have already loaded from packages contained in bundle.a. Instead of throwing away bundle layer 1 a new bundle layer 2 is created that uses bundle layer 1 as its parent. Bundle layer 2 will contain all the new versions of modules that are not already represented in the parent layers. This allows the new bundle.b to shadow the "dead" bundle.b module in bundle layer 1. The only JPMS module that cannot be shadowed by a child layer is the java.base module. But there is a big issue with this approach.

Discarded modules from a JPMS layer will be pinned in memory until the complete layer is discarded. This ultimately leads to a class loader leak because the stale bundle class loaders cannot be properly freed. It also causes issues for bundles that are uninstalled completely. The "dead" modules for these bundles will continue to be available since nothing is shadowing them from child layers. An empty module could be created that has the same name that exports nothing, but that will still allow modules on top to resolve when they shouldn't.

The code for this approach is currently located in the OSGi-JPMS layer GitHub project in the branch tjwatson/moduleClassLoader.

Conclusion

This approach allows for a pretty accurate representation of a static set of resolved OSGi bundles as JPMS modules. But it is left with several issues that need to be addressed before this can be considered a truly viable solution. It may be decided that these are permanent restrictions of JPMS going forward. But there are some tweaks to JPMS that could go a long ways to making this approach close to a complete solution. Some of these issues are currently being discussed in the JPMS spec mailing list.

1. The issue #ReflectiveAccessToNonExportedTypes is discussed in this thread. The current proposal unfortunately still requires all private packages to be exported in the JPMS layer.
2. Being able to control readability with issue #ReadabilityAddedByLayerCreator is added by this thread.
3. A proposal for #NonHierarchicalLayers is proposed in this thread.

If all three of these issues are solved in JPMS then a one to one mapping could be created between a bundle wiring and a JPMS layer/module. Depending on what a JPMS layer on top required it could be created with one or more bundle layers as its parent layers. This would allow individual bundle layers to be discarded as the dynamic set of resolved bundles changes in the OSGi Framework.