Hi Doug, it's nice to get to know you.
Well, first let me say that I don't have more than some foggy ideas and thoughts.
IMHO, the most useful feature of Eclipse (being based on OSGi) is the modularity. It allows to write small, compact libraries that can be understood, reviewed and tested in isolation. In practice, however, many libraries have grown to larger bodies of code that cover multiple areas and assumptions have infiltrated the APIs. When adding the need to provide backwards compatibility, it only gets worse with time.
For example, I claim that the model part of the language support for most languages is copied and adapted from the JDT. However, that code has many Java-specific details that aren't relevant for all languages, making it difficult to understand what is really needed. Also, copying and adapting the code makes it difficult to import any improvements JDT make to their code. So extracting the generic parts, like Handly does, is a big win.
Projects like DLTK still mix up all the provided functionality in an all-or-nothing package. If one's requirements don't match DLTK's assumptions, then it's not possible to use just parts of DLTK. Similarly, Xtext is exceptional if one can match their assumptions.
So, to return to the topic at hand, I feel that there are more areas that could be separated from the bulk and exposed as generic functionality, areas that otherwise every implementer needs to reinvent (or copy-and-paste from another project). They can't be as standalone as Handly, because some have to depend on a model API.
There is a lot of jface stuff that mixes core and UI: the core part should be possible to extract and let each language implement it, while the UI would be generic. Examples include: content assist, text hover, content formatter, quick assist, reconciler, damager/repairer, projection. At the very least, the non-ui parts should be moved from o.e.jface.text to o.e.text (after fixing the APIs), but it may be useful to have separate bundles. Some of the existing APIs include references to UI details (like ICompletionProposal returning an Image), so these should use some kind of generic token instead.
Then with these generic (as in: language independent) APIs and a generic model, one could implement a "language engine" that would connect the dots at the next level. It could for example include a database over the language constructs (like Xtext's index) with the associated search mechanism (that would power not only searching for references, but also code completion, etc). But people should be able to set up their own, if they need to, while still using the same basic APIs.
If all works well, it should be possible to have completely generic editors and basic views (i.e. what Xtext generates), with some default implementations that do nothing and let language implementers replace those one by one with relevant ones.
I know it is difficult to do this in a language-agnostic way and without imposing a certain structure, but today each language implementer needs to reinvent too many wheels, which raises the difficulty of implementing new languages. IntelliJ IDEA has better (but with room for improvements) APIs (I guess it's the second mover's advantage, they learned from Eclipse's first steps and didn't have to be backwards compatible).
Okay, this has become quite a long rant :-) I hope it makes sense.
regards,
Vlad
