Eclipse 3.0 builder issues

This document outlines the current pressing builder problems, maps out the solution space, and will eventually propose concrete changes for Eclipse 3.0 and if necessary for a 2.1.x maintenance release. Since this is not a committed plan item, no promises are being made at this point.

Problems

P1. Dynamic classpaths, static project references

There are many different PDE self-hosting styles. Some users build and run against plugins outside the workspace, some use binary or source projects in the workspace, and some use a combination of the two. Every time a user switches from building against external libraries to building against internal projects, the classpaths of all downstream plugins need to be updated. When the set of referenced projects on the classpath change, so do the platform project references. These references are used by the platform when computing the default project build order.

Before the Eclipse 2.1 release, JDT introduced the notion of classpath containers. Classpath containers abstract away volatile sets of references so that the .classpath file does not need to change every time these reference sets change. The two common uses of containers are for JRE libraries, and PDE project references. This approach solves the problem of .classpath files constantly changing as the user's self-hosting style changes (or when the JRE changes). The remaining problem is that the project references, manifested in the .project file, do not have a corresponding dynamic container facility. Therefore the .project file continues to change whenever the user switches between external and internal project references.

P2. Builders need more context

Individual project builders are not supplied with enough context information when they are invoked. In particular, they do not know whether the current build is a single project build or a workspace build. Also, there is no consistent hook that builder implementors can use to be notified when the build process is starting and finishing. If this information was available, builder implementors would have more flexibility in optimizing multiple project builds.

Without going into too much detail, consider a simple case of projects A and B, where B depends on A and so must be built after it. When A is built, it could leave behind a "cache" that describes in more detail exactly what changes builder A made. Builder B could then read the cached information and optimize its build accordingly. However, with manual project builds, project A could be built several times before project B is built. In this simple case, builder A could incrementally update this cache until B is finally built. However, if there is also a project C that depends on A, it could be built at a different time. In the general case, it is impossible for a builder to leave behind a cache for dependent builders to read, because it can never know the appropriate times to update or flush that cache.

To further complicate matters, other builders may have been installed on projects A or B that run in between the two builders that are trying to optimize downstream dependency changes. Builders cannot reliably determine if other builders have been run that affect them, or whether changes they see in a delta have been caused by their own builder or some other builder.

P3. Build scopes

Some Eclipse users work with massive workspaces of highly interconnected resources. These workspaces often take a very long time to build when a resource with a large number of downstream references changes. Builder implementors are able to optimize these cases to some extent, but in cases where thousands of referenced resources are genuinely affected by a change, these optimizations can only go so far.

One coping strategy that users have adopted is to only build small subsets of the workspace frequently, and perform full workspace builds less often. This essentially batches changes so that a single long build can replace frequent long builds, thus freeing up more time for users.

The current Eclipse build facilities are not well suited to this approach. Autobuild and the incremental build command (invoked with Ctrl+B), work only at a workspace scope. To build smaller sets of projects, users have to perform cumbersome manual steps involving multi-selecting the set of projects, and invoking "Build Project" from the "Project" context menu. Autobuild has established a high expectation about the transparency of building, making these manual steps unacceptable.

Solutions

S1. Workspace builders

Currently a workspace build runs roughly as follows:

   for each project P(i) in workspace {
      for each builder B(j) installed on P(i) {
         run builder P(i)B(j)
      }
   }

   for each workspace builder B(i) {
      run builder B(i)
   }

   for each project P(j) in workspace {
      if (P(j) has nature for builder B(i) {
         build project builder B(i) on project P(j)
      }
   }

The order in which builders run would be established by declarative rules in the workspace builder extension point. These rules would be resolved by platform as a topological graph ordering with cycles treated as unrecoverable errors.

Pros: With this approach, each builder has complete control over the build order it uses. Project references are no longer needed, so problem P1 goes away. The builder has all the context information it needs, so the optimizations discussed in problem P2 are now possible. It is also no longer possible for other builders to run interleaved with a particular builder as it iterates over the projects. Thus the builder has reasonably complete knowledge of what changes have happened to each project, and what the content of those changes are.

Cons: This is a breaking change to the workspace build order. If project P1 and P2 have builders B1 and B2, the old order is:

P1B1, P1B2, P2B1, P2B2

P1B1, P2B1, P1B2, P2B2

Second, there is a large class of builders that piggyback on other builders, and rely on those builders to establish the build order for them. For example, a client may install a builder that acts as a pre- or post-processor on the java builder, generating extra source files or copying/mutating class files. These builders typically have no knowledge of the correct build project order. In a world of workspace builders, the workspace builder for these secondary builders would need a mechanism for computing a reasonable project build order. The java builder and other "primary" builders would need to expose such an API and all secondary builders would use this (non generic) API to compute a reasonable build order.

In the general case, it would be impossible to provide a backwards compatibility mode for builders implemented against Eclipse 2.1. For example, say a 2.1 workspace is opened in 3.0, and it has projects with builder order {B1, B2}. Say B2 is now a workspace builder that defines its own project build order. B2 was previously using project references, but it immediately removes all project references when its plugin is first loaded since these are no longer used (if it did not do this it would not be solving problem P1 in our list). How does the platform determine an appropriate project order for running builder B1? Since B2 has not run, it can not infer an order from B2. B2 would have to expose some method that pre-declares the order it intends to use. B1, in the absence of its own build order, would use this declared order for its own projects. Now say there are two workspace builders (B2 and B3) on each project, and each declares a different project order. It is now impossible to infer an appropriate build order for B1.

S2. Pluggable build strategy

public abstract class BuildStrategy {
	public abstract void build(IProject[] projects, IProgressMonitor monitor);
	public final void build(IProject project, IProgressMonitor monitor) {}
	public final int getKind() {...}
	... other methods to provide additional context if needed ... 
}

If this solution was carefully implemented, it could also solve the long standing enhancement request to be able to turn off the java builder (in order to replace it with javac or other compiler).

Proposed action

TBD