3) semantic -> compile errors, find definition, code complete, find references, extract method, etc.. You need a semantic engine for this, even if not one as complete or accurate as an actual compiler.
From what I understand from this discussion, Textmate grammars help you with layers 1 and 2, but for 3 you need something else. Now, the thing is, virtually anyone who wants to work with a "big" language, they will want a full IDE, with support for layer 3 as well. People are not embedding Java, C++, Rust etc, in other projects like they would with XML, JSON, _javascript_ or even a scripting language like python.
So there needs to be at least some layer 3 support (semantic analysis), even if not as good as JDT. But, for any minimal layer 3 (semantic) support, you need a parser anyways, so you can trivially provide layer 2 functionality if you have any layer 3 support. (I'll give an real example with Rust)
So in effect TextMateonly helps us with layer 1 functionality... however layer 1 functionality is very simple to write (the language specific bits that is). I wrote the syntax highlighting lexing rules for the Eclipse IDEs for Go, Rust, and D, and for each IDE that was like 0.01% of the total effort involved for the whole IDE project. Basically half a day worth of work. Note: I'm talking only about the language-specific code here. Yes, the UI infrastructure (preference pages for each highlighting item, etc.) was much more complex and took more time, but that is framework code, not language-specific code.
As for the Rust example, in RustDT (
https://github.com/RustDT/RustDT) the layer 2 functionality is provided by Rainicorn (
https://github.com/RustDT/Rainicorn) a tool that displays parse errors, and the structure of the source file[1]. Rainicorn is a very simple tool that just re-uses the pre-existing official Rust parser, the only thing it does is basically serialize the compiler output data structures into JSON (or a similar language). Thus, no need to write or maintain new grammar rules or anything like if one was using TextMate.
([1] The source structure can be used for outline and code folding, although in RustDT it
is only used for the outline. Code folding is not supported in the IDE
itself - I never got around to it since I don't find that functionality is that important really - I never use it when working with Java for example)
My point with all this, to summarize, is that if Eclipse were to get built-in support for TextMate grammars, even as a library, I don't see how that would be of any significant help for the Eclipse IDEs for Go, Rust, D, etc.. Even if one was being written from scratch. The impact would be minimal.
On the other hand, having a common language server protocol (a protocol for an external tool providing layer 2 and layer 3 functionality), that would be of immense use. Which is why I'm excited about the VSCode protocol thing. It's even better if part of the Eclipse support for this language server can be put in an Eclipse-agnostic Java library. Which is exactly what
https://github.com/TypeFox/ls-api aims to provide, so that looks very promising too. (I haven't had the time yet to look into it in more detail.)
But it seems clear to me we should reuse things as much as possible. That means:
* Re-use the VS-Code protocol. It looks somewhat comprehensive already, and has more than one implementations - both client/IDE-wise, and language-wise. Like Sven mentioned, if there are some shortcomings we can always add some Eclipse-specific extensions to the protocol (although ideally this would be merged back into the standard protocol).
* Re-use common Java code across other Java-based IDEs, not just the Eclipse IDE. (The Eclipse IDE project has traditionally been very bad at this - but that is a whole other discussion)
I think the next step as far as Eclipse is concerned would be to have a Proof of Concept / prototype of an editor/IDE support using a VSCode Language Server. But who will take ownership of this?