Eclipse Community Forums: IMP » Accommodating established languages (communities, tools)?

Help

Home

Home » Archived » IMP » Accommodating established languages (communities, tools)?

Show: Today's Messages :: Show Polls :: Message Navigator

Accommodating established languages (communities, tools)? [message #7423]

Sun, 30 September 2007 12:19

Leif Frenzel

Messages: 23
Registered: July 2009

Junior Member

Hi everybody,

I'm glad to see this new initiative to help people with supporting more
languages in Eclipse :-) I remember that as early as at the first two
EclipseCONs there was a lot of talking about the subject, but also a lot of
disappointment for many people when nothing materialized subsequently. The
enthusiasm four years ago when one Eclipse-based IDE after the other was
popping up at SF is long gone, and it would be nice to get a little of that
back again :-)

I have one concern about the IMP approach: if I want to support an existing
language with an existing community and a mature base of tools (written in
the target language), how would I get this community involved and these
tools integrated? Potential contributors should come from the existing
community, but they'll often not want to implement stuff in Java (but in the
target language). Likewise, existing tools may have their own parsers, type
checkers, static analyzers, refactorers etc., again already implemented in
the target language. Clearly, there is a lot of work involved in
accommodating this, but generating Java code that pre-implements some IDE
features seems not to be a step in the right direction. Moreover, even for
new languages (that don't have a large community and a lot of development
tools written in them), if they are designed to be general-purpose, an
IMP-based IDE could only be a first step, to be later replaced by a
re-implementation that uses the target language for the 'interesting bits'.

So (somewhat provocatively put) it looks to me that IMP adresses mostly IDE
support for experimental languages where life expectation is short anyway,
and for languages with a focused range of application (i.e. DSLs), but not
for general-purpose languages with a broad field of application. The latter
sort of language tends to be self-hosting, and it doesn't appear that the
IMP approach takes this aspect seriously. Any thoughts about this?

Thanks && ciao,
Leif

Report message to a moderator

Re: Accommodating established languages (communities, tools)? [message #7499 is a reply to message #7423]

Mon, 08 October 2007 15:21

Robert M. Fuhrer

Messages: 294
Registered: July 2009

Senior Member

Leif Frenzel wrote:
> Hi everybody,
>
> I'm glad to see this new initiative to help people with supporting more
> languages in Eclipse :-) I remember that as early as at the first two
> EclipseCONs there was a lot of talking about the subject, but also a lot of
> disappointment for many people when nothing materialized subsequently. The
> enthusiasm four years ago when one Eclipse-based IDE after the other was
> popping up at SF is long gone, and it would be nice to get a little of that
> back again :-)

Agreed!

> I have one concern about the IMP approach: if I want to support an existing
> language with an existing community and a mature base of tools (written in
> the target language), how would I get this community involved and these
> tools integrated? Potential contributors should come from the existing
> community, but they'll often not want to implement stuff in Java (but in the
> target language). Likewise, existing tools may have their own parsers, type
> checkers, static analyzers, refactorers etc., again already implemented in
> the target language. Clearly, there is a lot of work involved in
> accommodating this, but generating Java code that pre-implements some IDE
> features seems not to be a step in the right direction.

You're asking some very good (though difficult) questions. So apologies for the
somewhat long post. :-)

It's worth mentioning that the LPG parser generator (http://sf.net/projects/lpg)
is in fact implemented as a binary C++ executable, so it's already possible to
encapsulate external tools in an IMP-based IDE. The X10DT (the IDE for the X10
programming language) likewise makes use of an existing command-line compiler
written using the Polyglot compiler framework. In that case, the compiler itself
happens to be written in Java, so we're able to directly access AST nodes and
results from the type-checker, name resolver, and so on. Nonetheless, the
compiler wasn't designed or written to be embedded in Eclipse, and yet the
X10DT's encapsulation of it works.

More generally, although IMP wizards generate Java implementation stubs for the
various services, they serve as examples for how to complete the implementation,
but aren't cast in stone. In particular, one can certainly accommodate external
tools by calling out to them, retrieving their results, and translating them into
a form that the Java-implemented Eclipse IDE can use.

That said, IMP doesn't yet have any meta-tooling specifically engineered to
address the integration of external tools, so it requires manually writing the
necessary adapters. It might make a great sub-project to investigate what sort
of direct support would make it easier to integrate external tools. The CDT has
some nice support for specifying the command-line structure (options and such)
of external tools, and (perhaps?) for managing tool invocations that would
certainly help. Beyond that, other things that could help would include tooling
to help with data interchange (e.g. translation specifications), and perhaps a
standard way of identifying tool-generated artifacts (files) to aid in resource
management. For starters.

Anyway, I'd certainly be interested in participating in such discussions.

Are you interested in contributing to such an effort?

=====================================

On the down side, there will naturally be issues with the tool-wrapping approach.
To present the results of some external tool, someone has to write adapters/
wrappers to expose the information and translate it into a form that the IDE can
use. You'll inevitably end up making choices as to, say, how much code gets
analyzed (a single method/routine, a compilation unit, a project, a workspace),
that won't work for use cases you discover later. And that granularity might be
harder to change with the wrapped external tool approach than if it were really
integrated. The data translation itself will cost something at run time. So, in
the end, in some cases, it might even be cheaper to rewrite the external tool.

Also, it turns out that there's a significant difference between a command-line
compiler and a compiler that works well in an interactive context where errors
occur frequently, where changes occur often, and responsiveness is crucial. So
one's success in attempting to re-purpose a command-line compiler as the heart
of an IDE can vary greatly. Mostly it's very tricky.

[BTW, with respect to refactoring, there are actually very few refactoring
engines out there, so in the near term, sadly, there's not much to interoperate
with anyway in that space.]

> Moreover, even for
> new languages (that don't have a large community and a lot of development
> tools written in them), if they are designed to be general-purpose, an
> IMP-based IDE could only be a first step, to be later replaced by a
> re-implementation that uses the target language for the 'interesting bits'.

The question you have to answer is what's to be gained by "re-implementing in
the target language for the interesting bits"? For languages such as X10 that
target parallel platforms, it's not at all clear that one would derive any
benefit from writing the compiler or refactorings in the target language. I
think a similar argument could be made for languages designed for embedded
software platforms, as well as for scientific computation. And so on.

And once you've re-implemented in the target language, what do you do for the
rest of the IDE framework? Build a new one entirely in the target language?

Anyway, by stitching tools together with adapters and translators, you
probably won't end up with an IDE whose quality is on a par with, say, the
JDT. Users will see/feel the seams. True, it might be cheaper to build, but
ultimately more expensive to use. And economically, one is better off
optimizing for the much larger group of IDE users at the expense of the much
smaller group of IDE developers, and not the other way around.

As far as IMP is concerned, you can really go either way. I.e., you can get
something up and running more quickly than if you had only the "raw" Eclipse,
and later rewrite selected tools/analyses, if doing so would produce a better
IDE for your language through tighter integration.

So the question is: do we want to shift IMP's direction altogether in order
to lure IDE developers to Eclipse with what may be an unrealistic promise
("Integrate your existing tools and everything will be perfect")? Or encourage
them to think hard and consider a somewhat more expensive route that ultimately
will probably produce a better IDE, that *their* users will love?

> So (somewhat provocatively put) it looks to me that IMP adresses mostly IDE
> support for experimental languages where life expectation is short anyway,
> and for languages with a focused range of application (i.e. DSLs), but not
> for general-purpose languages with a broad field of application. The latter
> sort of language tends to be self-hosting, and it doesn't appear that the
> IMP approach takes this aspect seriously. Any thoughts about this?

It's interesting to ponder why general-purpose languages tend to be self-
hosting. One could argue that it needn't be so. In fact, it's interesting to
note that Java was initially not self-hosting: the first VMs were implemented
in C/C++, not Java.

--
Cheers,
-- Bob

--------------------------------
Robert M. Fuhrer
Research Staff Member
Programming Technologies Dept.
IBM T.J. Watson Research Center

IMP Team Lead (http://eclipse-imp.sourceforge.net)
X10: Productivity for High-Performance Parallel Programming (http://x10.sf.net)

Report message to a moderator

Re: Accommodating established languages (communities, tools)? [message #7523 is a reply to message #7423]

Mon, 08 October 2007 16:03

Stan Sutton

Messages: 121
Registered: July 2009

Senior Member

Report message to a moderator

Re: Accommodating established languages (communities, tools)? [message #9732 is a reply to message #7499]

Wed, 10 October 2007 20:25

Leif Frenzel

Messages: 23
Registered: July 2009

Junior Member

Hi Bob,

thanks for your long reply :-) I answer in slices: about external tools
I'll do another post.

[snip]
>On the down side, there will naturally be issues with the tool-wrapping
>approach. To present the results of some external tool, someone has to write
>adapters/wrappers to expose the information and translate it into a form that
>the IDE can use. You'll inevitably end up making choices as to, say, how much
>code gets analyzed (a single method/routine, a compilation unit, a project, a
>workspace), that won't work for use cases you discover later. And that
>granularity might be harder to change with the wrapped external tool approach
>than if it were really integrated. The data translation itself will cost
>something at run time. So, in the end, in some cases, it might even be
>cheaper to rewrite the external tool.
>
>Also, it turns out that there's a significant difference between a
>command-line compiler and a compiler that works well in an interactive
>context where errors occur frequently, where changes occur often, and
>responsiveness is crucial. So one's success in attempting to re-purpose a
>command-line compiler as the heart of an IDE can vary greatly. Mostly it's
>very tricky.
That's true, as long as you assume that you use the compiler by running it
in
a separate process and talk to it via a command line interface. But apart
from
that and an implementation in Java (which you can directly call via Java
API
inside the same JVM), there is at least a third option: if the compiler
provides its own API (in the target language, not in Java), then you may
want to use that API, perhaps via a native interface. You would still have
to do some marshalling, but probably less than if you have to go via
command line
flags and command line compiler output as vehicle.

And if one is prepared to go as far as that, then another possibility
emerges:
you can now not only call that API in the target language, but you can
actually
also implement some IDE intelligence in that language. For example, you're
not
only using the compiler API to parse the AST, but you could also implement
an
algorithm for, say, walking it and collect occurrences of some identifier
(for
the Mark Occurrences functionality in the editor). So you're basically in
the
target language until everything you communicate is just a list of source
positions and nothing more.

To make my own background perhaps a bit clearer: I've been working on an
Eclipse-based IDE for Haskell for quite some time (about four years,
although mostly in my spare time), and I have explored several options in
this area. Now Haskell is definitely a language with loads of quite cool
tools that are of course all written in Haskell. In particular, the GHC
compiler API is what one wants to talk to from an IDE.

>[BTW, with respect to refactoring, there are actually very few refactoring
>engines out there, so in the near term, sadly, there's not much to
>interoperate with anyway in that space.]
True. But there is one for Haskell ;-) Generally, there is a lot of source
code analysis and source manipulation functionality there.

>> Moreover, even for
>> new languages (that don't have a large community and a lot of development
>> tools written in them), if they are designed to be general-purpose, an
>> IMP-based IDE could only be a first step, to be later replaced by a
>> re-implementation that uses the target language for the 'interesting bits'.
>
>
>The question you have to answer is what's to be gained by "re-implementing in
>the target language for the interesting bits"? For languages such as X10 that
>target parallel platforms, it's not at all clear that one would derive any
>benefit from writing the compiler or refactorings in the target language. I
>think a similar argument could be made for languages designed for embedded
>software platforms, as well as for scientific computation. And so on.
I was thinking of general-purpose languages. If they grow beyond a small
community, that community will very probably start to write developer
tools.
Somehow developers seem to be particularly fond of writing development
tools (as compared to writing whatever other sorts of applications).
Perhaps that is because most developers have over time learned to often
automate what they are doing. ("If you find yourself doing the same thing
twice, write a script.") We have also seen languages grow beyond their
originally intended area of application (Java itself is an example).

I agree this is probably not the most common case, but it happens. In this
case, much of the language-specific deep know-how is with that community.
And
the bulk of people who are willing to contribute to a big development
effort
is also in that community (also the bulk of people who would be willing to
early-adopt and thereby test and send bug reports, fixes, moral support and
other much needed feedback). So it makes sense to try and have the
interesting
bits within their playing field.

So for an Eclipse-based IDE the division of labor could be this: the
language-neutral layer (Eclipse Platform) may well be re-used by such a
community (in a similar way as OSs or 3D graphics frameworks are re-used
today from nearly every language runtime, or even as Eclipse/SWT re-uses
native widgets via native bindings); but what's language-specific is in
the domain of the target language.

>And once you've re-implemented in the target language, what do you do for the
>rest of the IDE framework? Build a new one entirely in the target language?
Hm, I have some feeling the Haskell community would probably love to do
that
:-) Granted, this isn't realistic given the current situation, but then
there
was a time when not many would have believed something huge like Eclipse
would be written in Java, a language that was for creating nice animations
in web pages ...

>Anyway, by stitching tools together with adapters and translators, you
>probably won't end up with an IDE whose quality is on a par with, say, the
>JDT. Users will see/feel the seams. True, it might be cheaper to build, but
>ultimately more expensive to use. And economically, one is better off
>optimizing for the much larger group of IDE users at the expense of the much
>smaller group of IDE developers, and not the other way around.
That's a good point.

Maybe at least some of the seams may have a positive effect, though. If
you are integrating tools that have been around for a while, you may help
users if you give them something that is familiar in some respects (and if
you help them integrate some legacy mechanisms they rely on). For example,
I've never really liked CDT's 'Standard Makefile' project type, where you
basically just call an external make for your sources, but that was me
coming straight from the JDT world; I guess it was actually good for
projects using C when they switched to Eclipse and some CDT-based tooling.

>
>As far as IMP is concerned, you can really go either way. I.e., you can get
>something up and running more quickly than if you had only the "raw" Eclipse,
>and later rewrite selected tools/analyses, if doing so would produce a better
>IDE for your language through tighter integration.
>
>So the question is: do we want to shift IMP's direction altogether in order
>to lure IDE developers to Eclipse with what may be an unrealistic promise
>("Integrate your existing tools and everything will be perfect")? Or encourage
>them to think hard and consider a somewhat more expensive route that
ultimately
>will probably produce a better IDE, that *their* users will love?
I don't think there's a strict either/or here. As I said, for many
languages
(those which haven't such an extensive existing toolset), the IMP way is a
great new option where they basically had none at all before. For the
languages
I'm talking about, there won't be an easy way anyway (integrating things is
always somewhat difficult and usually means much work).

Still IMP could be very helpful there too. The current Eclipse Platform
APIs
are relatively low-level, and I guess many language IDE projects
redundantly
implement the same stuff over and over again. So if there is an additional
layer of abstraction where you could selectively replace some functionality
with the native tools, that may still save a lot of work. In any case, I
will
look more closely into IMP to see how far it will help with my Haskell IDE
project :-)

>> So (somewhat provocatively put) it looks to me that IMP adresses mostly IDE
>> support for experimental languages where life expectation is short anyway,
>> and for languages with a focused range of application (i.e. DSLs), but not
>> for general-purpose languages with a broad field of application. The latter
>> sort of language tends to be self-hosting, and it doesn't appear that the
>> IMP approach takes this aspect seriously. Any thoughts about this?
>
>
>It's interesting to ponder why general-purpose languages tend to be self-
>hosting. One could argue that it needn't be so. In fact, it's interesting to
>note that Java was initially not self-hosting: the first VMs were implemented
>in C/C++, not Java.
One (soft) factor is probably the aptness of developers to automate their
own
development tasks that I speculated above. Another may be that exactly
writing
language implementations, refactoring algorithms etc. is the sort of task
that
requires the most intimate knowledge of the language and therefore is most
likely found with that language's 'gurus'. The more 'different' the target
language is from the candidate host language, the more likely it is that
their minds have shifted towards the new way of thinking, and thus they
tend to writing their best stuff in that new language. Well, this is
getting too far into amateur psychology, so I think I'll better shut up ;-)

Thanks for your patience && ciao,
Leif

Report message to a moderator

Re: Accommodating established languages (communities, tools)? [message #9755 is a reply to message #7523]

Wed, 10 October 2007 20:49

Leif Frenzel

Messages: 23
Registered: July 2009

Junior Member

Hi Stan,

>I don't think it's right that IMP addresses mostly IDE support for
>experimental languages that will have a short lifetime, but I believe that it
>is mainly true that IMP is not a platform for the development of languages
>with self-hosted IDEs.
>
>Some of the IDEs that we have so far developed using IMP are for DSLs that
>would not be used for hosting an IDE (regardless of the lifetime of those
>DSLs). We do have an IDE for the X10 language, which is an extension of Java
>with enhanced features for parallel computing. That is not a DSL, but being
>an extension of Java it can live perfectly well in a Java-based IDE.
>(Colleagues of ours also have developed an IMP IDE for JavaScript, but this is
>mainly as an environment for analyzing JavaScript programs, not as a
>production IDE for JavaScript development.)
That makes sense; so the actual dividing line is between languages with
and languages without extensive native tooling available ('native' in the
sense of written in the target language). The more tooling there is, the
more of it would have to make up part of the IDE, so the portion that you
can get for free from IMP is reduced. Basically, when faced with the
choice of getting things for free from their own tools and getting things
for free from IMP, established communities would tend towards their own
tools (and rightly so). Where there is only a choice between getting
things for free from IMP and not getting anything - well, that shouldn't
be too difficult to decide ;-) In fact, that's how it is for many
languages, and of course there is a great value in having something like
IMP around to address that situation :-)

>There are facilities in Eclipse, which IMP can draw on, for running
>non-Java-based tools in the context of a Java-based IMP IDE. This would not
>address the development of a full IDE in the other language, but in the
>absence of an integrated platform for a collection of tools in some language,
>it might be a step forward for users of those tools. We could look into the
>possibility of allowing the implementations of some of our services to be
>substituted by implementations developed in other languages.
That would be very interesting to me. As I mentioned further up in the
thread, I'm involved in an Eclipse-based IDE project for Haskell, and I'll
definitely try out IMP on it and see where it could help. You can
definitely count me in as early adopter there.

>One last thought: If you have a community of users of a language X (as
>distinct from X10), and they want to develop an IDE in X, then perhaps
>the IMP IDE architecture and components might provide an useful model of
>what an IDE in X might look like. (Maybe easier than trying to build one
>that looks like the JDT.)
Indeed, good point.

Thanks for your comments, I appreciate how much work you put in discussing
these somewhat peripheral points :-)

Ciao,
Leif

Report message to a moderator