Summary
Eclipse requires that the contents of each project be stored in a single directory on disk. Every file and folder in that directory tree on disk must belong to the project in the workspace. This restriction makes it difficult to use Eclipse in conjunction with tools that have specific layout requirements for their files, or with users who have legacy file base layouts that they need to maintain. We would like to improve the situation for the 2.1 release of Eclipse. There is a separate, first document describing the nature of the problem. This document explores the space of potential solutions, without drilling down into the details of any of them.
Last Modified: 14:00 October 4, 2002
In this document we explore the space of potential solutions, to better understand where the various aspects of the problem can and cannot be addressed. Although much of the discussion will center around Java, the problems are likely not confined to Java and will affect tools for other languages. In most cases, we will have the option of addressing the problem at the JDT level without changing the Eclipse Platform. This has its pros and cons. Pro: the solution can be specialized for Java. Con: The solution would not apply to C tools, for example. On the other hand, addressing the problem at the level of the Eclipse Platform also has its pros and cons. Pro: The solution would apply to C tools, etc. Con: The change may impact all tools and users. All this must be factored in to the decision-making process.
In the final decision, we may only be able to address a subset of the problems. But here we proceed with an unlimited budget for exploration.
Although the restrictions on the structure of Eclipse projects may come across as inflexible to some, the current scheme has several important virtues:
NetBeans 3.4 (and Sun™ ONE Studio, formerly Forte™ for Java™)
Borland® JBuilder™
[This section is incomplete.]
How could we change Eclipse to make the Java output folder more flexible?
Here are some options (in no particular order):
Option P-1: Change JDT to allow the output folder to be external to the workspace.
JDT allows library JARs to be either local to a Java project (specified by workspace path) or external to the workspace (specified by an absolute file system path). So one option is to be similarly flexible with the location of a Java project's output folder.
This would entail changing JDT Core API as well as JDT UI, but could likely be done in a non-breaking fashion.
The chief drawback of this approach is that there would be no resource deltas for tracking changes to files in a project's output folder in the case where it was external to the workspace. This would have serious repercussions in at least three areas: (a) the JDT debugger's HotSwap facility relies on resource deltas to class files in the output folder; (b) the Java incremental project builder uses builder-supplied resource deltas to track changes to class files in dependent projects, and (c) other incremental project builders on the same project would be unable to track changes to anything in the output folder as well. Without resource deltas, it is hard to see how either could be supported.
A smaller drawback is that any code that creates and reads class files in the output folder would have to be written in such a way that it could deal with resources in the workspace (using resource handles provided by the workspace API) and outside the workspace (using java.io.File API directly).
Option P-2: Change Eclipse Platform workspace to allow arbitrary directory in the file system to be "mounted" as a folder inside a project.
A user would then be able to mount an external file system subdirectory into their Java project, and designate their output folder at (or below) that mount point. The net effect is that the generated class files would be written directly to the desired location. As long as mounted subdirectories are considered to be a regular part of the project's file, the regular resource API and resource deltas apply. JDT would not even need to be aware that the output folder was separated from the rest of the project's files.
The pros and cons of this approach are chiefly those of mount points generally (covered below).
PDE launches a runtime workbench with a special command line option specifying one or more plug-in-relative paths (e.g., -dev bin) that are to be included on each plug-in class loader's class path. PDE includes the project-relative paths of all output folders; this is just the path "bin" when all plug-in projects locate their output folder in the default location. If an external directory is mounted as a Java project's output folder, the existing Eclipse Platform Core Runtime mechanisms would need to be revamped to allow PDE to supply an absolute file system path for each plug-in.
Some other observations:
Option P-3: Advocate use of OS-level symlinks.
Unix file system have symlinks, and Windows 2000 NTFS has an unsupported feature called junctions. These file system mechanisms allow a directory in one part of the file system to be linked in to somewhere else. For example, if the project root directory of Project1 is C:/myws/project1 and the Java output folder is in its standard location, a symlink from C:/myws/project1/bin to C:/tomcat/webapps/mytest/WEB-INF/classes ensures that the files in the output folder are created directly in the web server's subdirectory.
The advantage of this option is that the user can create this arrangement using existing OS-specific tools, and does not require explicit support from the Eclipse. So this option is already supported in 2.0.
Operating systems may have restrictions on symlinks that make this option less workable for some purposes. For instance, NTFS junctions work on Windows 2000 or better, and only work within and between local drives. Furthermore, junctions are an unsupported feature.
Option P-4: Advocate use of external tool builders for copying the output folder to an external location.
Eclipse allows the user to add Ant-scripted steps as an external tool builder. Coping new and changed files from the output folder to another location can be done with one Ant command.
This option is low-tech and works in Eclipse 2.0; however, it only ensures that the files in the output folder end up in the desired location; it does not reduce the network traffic when the project's files are stored on a network drive.
Option Q-1 (orthogonal): Allow one output folder per source folder.
Orthogonal to the other options, flexibility could be improved by allowing different output folders to be specified for each source folder. In other words, allowing the output folder could be partitioned along source folder lines. The changes allows, for example, a single Java project to contain both product source code and test suites partitioned into separate source and output folders.
The additional output folders would need to find their way onto the runtime class path, and the debugger would need to know how to map from generated class file to source file. Neither should be a problem.
This change would break existing clients that assume that all generated class files for a Java project are in a single output folder.
How could we change Eclipse to make Java projects (source folders in particular) be multi-rooted rather than single-rooted?
Here are some options (in no particular order):
Option M-1: Change JDT to allow a Java project source folder to lie inside another project, or to lie outside the workspace.
It would be relatively straightforward to allow a source folder for a Java project to lie inside a different Java project. Because these files are still inside the workspace, resource deltas can still be used for updating the Java model and triggering recompilation.
Supporting a source folder outside the workspace entirely would have many of the same problems as with external output folders. These files would need to be accessed with java.io.File API (not resource handles), and there would be no resource deltas for updating the Java model or triggering recompilation.
Option M-2: Change Eclipse Platform to "mount" a file system directory as a project folder.
We would add a new workspace concept of mount points, and allow a directory in the file system to be grafted into a project's resource tree. For example, mounting C:/JavaSources at /Project1/src would mean that the files and folders under /Project1/src are stored under the subdirectory C:/JavaSources; e.g., the file C:/JavaSources/com/example/Egg.java would appear in the workspace at the resource path /Project1/src/com/example/Egg.java. Any project files not below a mount point would be held within the project's root directory, as they are currently. Tools that navigate the resource tree using the workspace API need not be aware of the file's location.
It would also be possible to allow a portion of one project's resource tree to be directly grafted into another (instead of doing this indirectly through the file system). This implies some form of overlap be allowed.
With the ability to mount a file system directory into the root of a project, a Java project can have its source folders in different locations on disk.
This would be a breaking change for clients of the workspace because it alters the current implicit understanding that all project contents are stored in one file system subtree. However, it should be possible to minimize the effects of this breakage and provide good compatibility with existing 2.0 tools and team (VCM) providers.
Option M-3: Advocate use of OS-level symlinks.
The use of symlinks is discussed above (Output Folder, Option P-3). These are the file system equivalents of the mount points in Option M-2.
The advantage of this option is that the user can create this arrangement using existing OS-specific tools, and does not require explicit support from the Eclipse. So this option is already supported in 2.0.
Operating systems may have restrictions on symlinks that make this option less workable for some purposes. For instance, NTFS junctions work on Windows 2000 or better, and only work within and between local drives. Furthermore, junctions are an unsupported feature.
How could we change Eclipse to make the Java source folders selective rather than all-inclusive?
Here are some options (in no particular order):
Option S-1: Change Eclipse Platform to add support for resource working sets with the capability to browse and manipulate a subset of the workspace where certain files and folders are excluded.
This option does not involve changing the nature of the workspace resource tree and its relation to the file system. It simply adds ways to provide filtered views of that resource tree.
A Java project source folder is currently specified with a project-relative folder path. All *.java source files found in the project subtree are automatically included in the Java model and on the Java compiler's class path. The source folder specification could be enhanced to permit the user to specify which files and folders to exclude (think Ant filesets).
The source folders of a Java project are not currently allowed to coincide or otherwise overlap. Some relaxation of this rule is required to handle the case where a source folder containing tests is nested inside the main source folder (Example 3). One relaxation is to allow source folders to coincide or overlap but use the mechanism for excluding files to prevent the same file from being included in both source folders.
This option would entail changing Eclipse Platform Core to provide infrastructure, possibly in the form of a rule-based resource visitor. We would explore a possible connection to the existing UI notion of working sets. JDT Core API and JDT UI would need to change as well, but could likely be done in a non-breaking manner by adding working sets to classpath entries (IClasspathEntry) and revamping the rules for legal source folder overlap.
Option S-2 (orthogonal): Change JDT to allow overlapping source folders but automatically exclude the overlapped portion from the parent.
A different approach is to provide the kind of selectivity required to handle cases like where a source folder containing tests is nested inside the main source folder (Example 3). The solution is to allow the overlap, but automatically exclude the files accounted for by the more specific folder from the less specific one. For example, if /src/ and /src/tests/ as both specified as source folders, files under /src/tests/ would only appear under the latter source folder, and the former source folder would act as if it had no child named tests.
This option would involve a non-breaking change to JDT to remove the current restrictions that prohibit overlap and define how overlapping source folders are interpreted. This option is orthogonal to the other selectivity options, and could be done in conjunction with any of them.
Option S-3: Radically change the Eclipse Platform policy for adding and removing files from the workspace.
The workspace folder refresh operation (IResource.refreshLocal) automatically creates entries in the workspace resource tree for newly discovered files and folders out in the file system, and discards entries for any files and folders that are no longer present in the file system. (This refresh action is also implicit in a number of workspace operations when the force flag is used). This characteristic makes a project's workspace resource tree a more-or-less faithful mirror of the project's subtree in the file system. This gives workspace projects their all-inclusive feel.
There are other IDEs (including Visual Studio .net) that allow the user to control which files and folders belong in a project; other files and folders may exist in the local file system, but are simply ignored by the IDE if not flagged as belonging to the project. This characteristic means that the IDE's notion of project is more like a "view" onto files in the file system. This alternate conception of workspace could be achieved by changing Eclipse policy so that file and folder entries are never added to or removed from the resource tree due to a refresh operation; new API would be added so that entries could be added or removed explicitly. This would give workspace projects a selective feel.
On the plus side, the resource tree would be smaller if some files and folders were to be excluded, and this would improve performance. It would also reduce workspace clutter and make Eclipse behave more like certain other popular IDEs (Visual Studio .NET).
The downside of this option is that it is impossible to do while maintaining compatibility. Consider an existing tool that is calling IResource.refreshLocal to cause files created directly in the file system to appear in the workspace (a common problem when bridging external programs to Eclipse). Changing the semantics of IResource.refreshLocal so that it does not discover new files would clearly break this tool. On the other hand, if the semantics of IResource.refreshLocal is not changed, then running this tool would automatically admit files to the workspace, even against the user's wishes. Either path leads to a different kind of breakage.
Option S-4: Add Eclipse Platform mechanism to prevent refreshLocal from adding certain files to the workspace resource tree.
Another option is to change the Eclipse Platform refresh mechanism is a more modest way to provide a way to selectively exclude files. If the refresh operation was governed by exclusion rules, it would be possible for the user to explicitly prevent certain files and folders in the file system from being added automatically to the workspace. By default, there would be no exclusion rules, and the workspace would behave exactly as in 2.0. If the user indicates that a particular file or folder is to be excluded, the refresh operation would never add it to the resource tree, even if the file or folder exists in the file system. This would give workspace projects an initially all-inclusive feel which could be made more selective as required.
New API would be required to change the exclusion rule set, to explicitly add a file or folder to the resource tree even if on the excluded list, and to remove a file or folder from the resource tree without deleting the file in the file system. The semantics of IResource.refreshLocal would not have to change in a breaking way. A project's workspace resource tree is a more-or-less faithful mirror of the project's subtree in the file system except for the files and folders that the user has explicitly taken off the table.
On the plus side, the resource tree would be smaller if some files and folders were to be excluded, and this would improve performance. It would also reduce workspace clutter and make Eclipse behave more like certain other popular IDEs. On the minus side, it does mean that in-memory workspace resource tree is not quite as faithful a mirror of the file system.
Option S-5: Add Eclipse Platform mechanism to hide certain files in the workspace resource tree.
Eclipse Platform 2.0 has a "team private member" mechanism for hiding certain resources. These resources are represented by entries in the workspace resource tree, but are explicitly marked as hidden so that clients will not see these resources unless they ask to see them. Imagine that this mechanism was generalized and extended so that other resources could be similarly hidden from clients that do not ask for them. By default, there would be no hidden files, and the workspace would behave exactly as in 2.0. The refresh operation would automatically add and remove files as in 2.0. The user could explicitly control whether they were marked as hidden, and whether hidden files were shown in the UI. This would give workspace projects an all-inclusive feel which the user could make more selective as required.
New API would be required to change the hidden flag and to include or exclude hidden resources when navigating the workspace. The semantics of IResource.refreshLocal would not have to change. A project's workspace resource tree is a more-or-less faithful mirror of the project's subtree in the file system except for the files and folders that the user has explicitly taken off the table.
How could we change Eclipse to allow Java source folders to overlap?
Here are some options (in no particular order):
Option V-1: Change JDT to allow overlap within a project's source folders.
This option is discussed in S-1 and S-2.
Option V-2: Allow "mounted" file system directories to overlap.
One of the design details of option M-2 is whether to allow mounted directories to overlap, either within a single project or across all projects in the workspace.
The simple interpretation of overlapping mount points would mean that the same file on disk might have multiple distinct resource handles. This would create a semantic problem for the IWorkspaceRoot.getContainer/FileForLocation API methods which presuppose that there is at most one resource handle for any candidate file system path.
Other issues if two resources /P1/Foo.txt and /P2/Foo.txt correspond to the same file C:/Sources/Foo.txt:
Options V-1 and V-2 are not necessarily mutually exclusive. There may be reason to allow overlapping source folders regardless of mount points, and vice-versa.