Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Archived » IMP » Extensibility of IMP
Extensibility of IMP [message #7349] Tue, 25 September 2007 19:44 Go to next message
Eclipse User
Originally posted by: olegm.gmail.com

Hello,

I would like to understand the limits of extensibility of IMP.
Obviously, one can do everything that can be configured with the
extension points, but what happens when this is not sufficient? Are
there any mechanisms for such case?

Here are some of examples I could come up with:
* Showing auto-complete after "." is entered without having to press
ctrl-space.
* Using drag & drop on the parts the edited document.
* Showing additional annotations of the program text.
* Automatically folding the parts of the program that are far enough
from the cursor.

I don't mean to imply that the support for arbitrary extensions of IMP
is neccessary, just want to find out the limits :)

Best,
OM
Re: Extensibility of IMP [message #7488 is a reply to message #7349] Mon, 08 October 2007 14:00 Go to previous messageGo to next message
Robert M. Fuhrer is currently offline Robert M. Fuhrer
Messages: 294
Registered: July 2009
Senior Member
Oleg Mürk wrote:
> Hello,
>
> I would like to understand the limits of extensibility of IMP.
> Obviously, one can do everything that can be configured with the
> extension points, but what happens when this is not sufficient? Are
> there any mechanisms for such case?
>
> Here are some of examples I could come up with:
> * Showing auto-complete after "." is entered without having to press
> ctrl-space.
> * Using drag & drop on the parts the edited document.
> * Showing additional annotations of the program text.
> * Automatically folding the parts of the program that are far enough
> from the cursor.

First, the specific ideas you mentioned are all really nice ideas, and
ones that IMP probably should support.

> I don't mean to imply that the support for arbitrary extensions of IMP
> is neccessary, just want to find out the limits :)

More generally, I hope you can appreciate that this is a *very* tough
question. :-) [Hence my delay in answering as I chewed on it...]

Unfortunately, I don't have a particularly good answer, but I wonder whether
any practical answer is possible. This question goes right to the heart of
software extensibility. In my experience, generally, in order for software
to be extensible in any particular fashion, it must be intentionally
designed to be so.

For example, for an OO class to be extensible, the right fields/methods have
to be made visible to derived classes, and the behavior has to be factored
such that one can override the bits that need customization, while permitting
reuse of the common bits. At the higher level, the same goes for Eclipse and
its extension point mechanism: it enables extensibility, but doesn't directly
provide it. If the software wasn't designed to support a particular kind of
extension, it probably won't support it (well, or at all), and you'll have to
modify the software to make it do so.

So, the best answer I can think of is: IMP is not finished, so please join
the effort, and bring your ideas, and we'll work together to make sure it
supports a very good set of IDE features, and is extensible in all the right
ways to enable reasonable customizations.

--
Cheers,
-- Bob

--------------------------------
Robert M. Fuhrer
Research Staff Member
Programming Technologies Dept.
IBM T.J. Watson Research Center

IMP Team Lead (http://eclipse-imp.sourceforge.net)
X10: Productivity for High-Performance Parallel Programming (http://x10.sf.net)
Re: Extensibility of IMP [message #7511 is a reply to message #7349] Mon, 08 October 2007 15:34 Go to previous messageGo to next message
Stan Sutton is currently offline Stan Sutton
Messages: 121
Registered: July 2009
Senior Member
Oleg Mürk wrote:
> Hello,
>
> I would like to understand the limits of extensibility of IMP.
> Obviously, one can do everything that can be configured with the
> extension points, but what happens when this is not sufficient? Are
> there any mechanisms for such case?
>
> Here are some of examples I could come up with:
> * Showing auto-complete after "." is entered without having to press
> ctrl-space.
> * Using drag & drop on the parts the edited document.
> * Showing additional annotations of the program text.
> * Automatically folding the parts of the program that are far enough
> from the cursor.
>
> I don't mean to imply that the support for arbitrary extensions of IMP
> is neccessary, just want to find out the limits :)
>
> Best,
> OM

I generally agree with what Bob has said in his earlier response. We've
started by trying to support extensibility through existing linguistic
or environmental mechanisms and there are limits to the kinds of
extensions that those mechanisms can support for a given IDE instance.
(We might be able to get some additional extensibility using
aspect-oriented techniques, but I won't try to speculate about that here.)

The examples you give are suggestive. I think we could probably support
a couple of them with relatively minor modifications of our code base.

Regarding additional annotations on a program text, I expect that
annotations are typically the output of some tool that runs on the text.
In light of that, having "additional annotations" might mean enabling
an existing tool (such as a parser) to post two distinct sets of
annotations, or allowing additional tools (other than, say, a parser) to
post annotations. In IMP currently, annotations get put onto a program
text by a "MessageHandler" that is provided to the parser. Allowing one
tool to post different kinds of annotations might mean making the the
tool (and its driver) sensitive to the possibility of providing multiple
MessageHandlers, or it might entail generalizing our MessageHandler to
be sensitive to the kind of annotations it is handling (i.e., to be able
to post multiple kinds of annotations). The latter would be a change to
our library class, but a localized one.

Regarding the automatic folding of parts of a text that are at a certain
distance from the cursor, I think we could probably handle that pretty
straightforwardly with an alternative base class for our folding
updater. Of course, then we'd have to modify the wizard for the folding
service to allow the user to choose which type of folding updater they
wanted. Again, this is a change, but a fairly localized one.

Regarding "auto-auto-completion", I think that our existing content
proposer might work fairly well for that (maybe it would take a little
tuning). I think the main thing would be to set up the automated
invocation, which I think we could do by adding a new service that could
be automatically invoked by the editor without manual intervention. For
that I suppose we'd need at least new extension point and a driver (what
we might call an "auto-auto-completion-controller"). We could probably
make it run automatically, given a content proposer, although we might
want to also provide some menu actions to allow the user to turn it on
and off. If it requires a kind of content proposer that would differ
from our usual one, then we'd need a new template or base class and
possibly a wizard for the user to instantiate the service. This might
be a bit more of a change than the previous cases, but it's still fairly
contained.

Regarding drag and drop, I have no idea. Without any knowledge of how
that might work, all I can do is speculate that it might require a new
base class for the editor (i.e., not the TextEditor on which the IMP
editor is based). Presumably that would entail some major work.

I haven't tried to actually work out any of these solutions, but it is
interesting to think about the problems. It seems that these
suggestions touch on two general cases where code needs to be modified
but the modifications are relatively contained: modifying an existing
service, and adding a new service (albeit one that is like an existing
service). If a fairly wide range of desirable and interesting
variations can be accommodated by such localized modifications of code,
then I think we can claim some measure of success for our architecture.
But I'm sure we can do a better job of preparing for this. And this
is a good fodder for writing up some documentation on what IMP
developers (or IMP IDE developers) might do in these cases.

Thank you!

Stan
Re: Extensibility of IMP [message #7532 is a reply to message #7488] Tue, 09 October 2007 11:16 Go to previous messageGo to next message
Eclipse User
Originally posted by: olegm.gmail.com

Hello,

Robert M. Fuhrer wrote:
>> I don't mean to imply that the support for arbitrary extensions of IMP
>> is neccessary, just want to find out the limits :)
>
> More generally, I hope you can appreciate that this is a *very* tough
> question. :-) [Hence my delay in answering as I chewed on it...]
>
> Unfortunately, I don't have a particularly good answer, but I wonder
> whether any practical answer is possible. This question goes right to the heart of
> software extensibility. In my experience, generally, in order for software
> to be extensible in any particular fashion, it must be intentionally
> designed to be so.

True. One way to design for variability is to create a bunch of usage
scenarios that vary as much as possible and make sure the framework
supports them.

> For example, for an OO class to be extensible, the right fields/methods
> have to be made visible to derived classes, and the behavior has to be factored
> such that one can override the bits that need customization, while
> permitting reuse of the common bits. At the higher level, the same goes for Eclipse
> and its extension point mechanism: it enables extensibility, but doesn't
> directly provide it. If the software wasn't designed to support a particular kind of
> extension, it probably won't support it (well, or at all), and you'll have to
> modify the software to make it do so.

Mainly, I would like to understand why Eclipse extension points were
chosen as opposed to say having a toolkit framework with a bunch of base
classes, decorators/mix-ins, and utilities. In the former case
assembling a particular IDE would mean wiring up the pieces of the
toolkit in a desirable manner. In my opinion this approach allows to
accommodate for unexpected variability quite easily at the compile time.

Eclipse extension points are (in my understanding, I'm quite new to
Eclipse) a mechanism for deployment time variability --- i.e. deploy a
shared plugin, later deploy other plugin that customizes it. After all
the Eclipse UI framework (org.eclipse.ui) is not a set of plugins, but a
toolkit and I think for a good reason.

I can still see a case for deployment time variability of IDEs (e.g.
plug in my own token colorer, new refactoring, etc), but this is a
variability of the IDE plugin (which could be achieved using Eclipse
extension points), not of the shared IMP functionality.

The main bottleneck of the current approach, as I see it, is that it is
VERY easy to get a decent IDE with standard functionality (I've just
tried it, even without LPG, although IMP is a bit alpha now), however
once You get out of the variability space determined by the IMP
extension points the only ways to go are:
* Submit a change request for IMP (or a patch)
* Fork-off IMP and do Your own changes
While the toolkit approach would allow to reuse the suitable parts of
the IMP framework and to customize the rest without modifying the IMP
itself.

Keep up the good work,
Oleg Mürk
Re: Extensibility of IMP [message #9778 is a reply to message #7532] Wed, 10 October 2007 22:23 Go to previous messageGo to next message
Stan Sutton is currently offline Stan Sutton
Messages: 121
Registered: July 2009
Senior Member
Oleg Murk wrote:
>
> ...
>
> The main bottleneck of the current approach, as I see it, is that it is
> VERY easy to get a decent IDE with standard functionality (I've just
> tried it, even without LPG, although IMP is a bit alpha now), however
> once You get out of the variability space determined by the IMP
> extension points the only ways to go are:
> * Submit a change request for IMP (or a patch)
> * Fork-off IMP and do Your own changes
> While the toolkit approach would allow to reuse the suitable parts of
> the IMP framework and to customize the rest without modifying the IMP
> itself.
>
> Keep up the good work,
> Oleg Mürk

Hi Oleg,

Now that we have started to "go public" with IMP, we are hearing from
(potential) users, both within and outside of IBM, who are interested in
a variety of IDE development or integration scenarios, many of which do
not fit precisely with our initial design points for variability.
Actually, we ourselves are considering how to vary some of the IMP
framework for different scenarios.

I think it would be fair to say that IMP provides something of a
toolkit, but one in which the most obvious pieces are oriented toward
service implementations. Many IMP services are implemented as
extensions of Eclipse extension points but actually some are not.
Still, it is true that the style of these is more or less extension-like
in that they define a locus of customization that extends the IMP
framework in a standard way.

Of the extension points used by IMP, some are defined by Eclipse and
some are defined by IMP. One way for an IDE developer to further
customize an IMP IDE might be to add their own extension points for
their own extensions (for services or whatever). Still, this doesn't
get you into customization of the more fundamental parts of the IMP
framework.

For that you might have to fork off your own IMP variant. As I said
in a separate response in this thread, it may be that several kinds of
framework-level customizations may be possible without too much pain (or
at least without making very extensive changes). Actually, we have our
own experience with this in relation to Eclipse. At least through
Eclipse 3.2 the facilities for working with the extensions model and the
plug-in dependencies model are not suited to the kind of incremental
elaboration that is necessary for IMP IDE development. For versions of
IMP that worked with Eclipse up to version 3.1 we actually had our own
version of the org.eclipse.pde.core plug-in, mostly unchanged, but with
half-a-dozen key modifications. Now for Eclipse version 3.2 we have our
own representation of the plug-in extension and dependency models that
allow us to work with the Eclipse models in appropriate ways (these are
just a few classes). So we have been maintaining our own fork of
Eclipse. We didn't have to write so much code to achieve the desired
effect, but understanding what Eclipse did and how to adapt to that took
quite a bit of work.

IMP does not aim for universal extensibility, but we can strive for a
design that would make it easy for IMP users to fork IMP in important
respects.

Regards,

Stan
Re: Extensibility of IMP [message #14831 is a reply to message #7511] Thu, 08 November 2007 23:53 Go to previous messageGo to next message
Stephan Herrmann is currently offline Stephan Herrmann
Messages: 1024
Registered: July 2009
Senior Member
Hi Stan!
Hi Bob!

Stanley Sutton wrote:
> (We might be able to get some additional extensibility using
> aspect-oriented techniques, but I won't try to speculate about that here.)

Ok, this looks like the perfect cue for my part ;-)
i.e., I believe I can offer some help here because we have ...
+ the technology to use a disciplined version of AOP for Eclipse plug-ins
+ experience of using this technology for
- extending the JDT
- adapting and composing generated pieces of software.

Given the availability of aspect technology to support unanticipated
adaptation, this could (at least for the time being) disburden the
IMP project from the commitment to get every dimension of extensibility
right in the first place.
Wouldn't it be more beneficial if the IMP project (for the time being)
would focus more on completeness, so we can soonish generate IDEs that can
actually kind-of compete with the JDT in terms of feature-richness (is
that an english word?) and convenience??

IMP users wanting to fine tune their IDEs can do so now without forking
by using an aspect plug-in. After several users have done similar
extensions using aspects, it should be much easier to migrate that
variability down into the core.

To stop philosophical rambling on what should or shouldn't be done,
let me give some details about what we have to offer:

Technology:
The language ObjectTeams/Java (OT/J, see www.objectteams.org) has been
integrated with Equinox (yielding OT/Equinox) as described in this article:
http://www.jot.fm/issues/issue_2007_10/paper6.pdf

Experience #1:
The Object Teams Development Tooling was developed by forking the
JDT core (sigh) and by adapting other plugins, most prominently the
JDT UI using aspect plug-ins written in OT/J. The resulting IDE supports
all JDT features even for OT/J code and seamlessly integrates new features
(see www.objectteams.org/distrib/features.html).
By this we demonstrate that OT/Equinox is powerful enough to do very
effective adaptations and we have experienced that this kind of
architecture is tremendously easier to maintain
(e.g., when moving from one version of Eclipse to another)
than any other solution we could see.

Experience #2:
We are currently experimenting how OT/Equinox can profitably be
combined with model driven/generative techniques. More specifically,
we are using GMF and GMF-based graphical editors, to create an editor
for a custom extension of the UML. Integrating the different pieces
is done using OT/Equinox. Initial results look very promising for this
kind of use cases, too.

My feelings are that adapting an IMP-generated IDE using OT/Equinox
should be fairly easy and produce working, maintainable solutions.

Since I don't assume IMP users to be familiar with OT/J and OT/Equinox
I'd offer to assist anybody trying the combination of OT/Equinox and IMP.
I'm pretty sure the battle for *complete* built-in extensibility is
lost the very moment you start fighting. But I don't think you have
to start fighting that battle ;-)

cheers,
Stephan

--
Stephan Herrmann
Re: Extensibility of IMP [message #14861 is a reply to message #14831] Fri, 09 November 2007 18:24 Go to previous messageGo to next message
Stan Sutton is currently offline Stan Sutton
Messages: 121
Registered: July 2009
Senior Member
Hi Stephan!

Thanks for your general interest and specific response on this. I think
that what you are proposing makes a lot of sense. IMP is really a
project to investigate development of IDEs, not a project to investigate
extensibility. As a practical matter it has adopted some
extensibility mechanisms but that is in support of the primary goal of
building customized IDEs. Since we want the IDEs to be customizable,
naturally we have to think sometimes about whether the extensibility
mechanisms we use are the most effective for that. In any case, as you
say, complete built-in extensibility is a lost cause from the outset,
but the mechanisms we use do work well enough to allow us to move
forward with creating more complete IDEs.

The soundness of the OT approach seems to be well supported, so your
suggestion makes a lot of sense (in my opinion). I'm sure it would be
interesting to us to see what kinds of modifications to an IMP IDE you
or others would want to apply using AO techniques. Also, we might have
a conversation sometime (off-line) about what you did in adapting the
JDT core and UI. It that won't give you too much of a headache!

Regards,

Stan


Stephan Herrmann wrote:
> Hi Stan!
> Hi Bob!
>
> Stanley Sutton wrote:
>> (We might be able to get some additional extensibility using
>> aspect-oriented techniques, but I won't try to speculate about that here.)
>
> Ok, this looks like the perfect cue for my part ;-)
> i.e., I believe I can offer some help here because we have ...
> + the technology to use a disciplined version of AOP for Eclipse plug-ins
> + experience of using this technology for
> - extending the JDT
> - adapting and composing generated pieces of software.
>
> Given the availability of aspect technology to support unanticipated
> adaptation, this could (at least for the time being) disburden the
> IMP project from the commitment to get every dimension of extensibility
> right in the first place.
> Wouldn't it be more beneficial if the IMP project (for the time being)
> would focus more on completeness, so we can soonish generate IDEs that can
> actually kind-of compete with the JDT in terms of feature-richness (is
> that an english word?) and convenience??
>
> IMP users wanting to fine tune their IDEs can do so now without forking
> by using an aspect plug-in. After several users have done similar
> extensions using aspects, it should be much easier to migrate that
> variability down into the core.
>
> To stop philosophical rambling on what should or shouldn't be done,
> let me give some details about what we have to offer:
>
> Technology:
> The language ObjectTeams/Java (OT/J, see www.objectteams.org) has been
> integrated with Equinox (yielding OT/Equinox) as described in this article:
> http://www.jot.fm/issues/issue_2007_10/paper6.pdf
>
> Experience #1:
> The Object Teams Development Tooling was developed by forking the
> JDT core (sigh) and by adapting other plugins, most prominently the
> JDT UI using aspect plug-ins written in OT/J. The resulting IDE supports
> all JDT features even for OT/J code and seamlessly integrates new features
> (see www.objectteams.org/distrib/features.html).
> By this we demonstrate that OT/Equinox is powerful enough to do very
> effective adaptations and we have experienced that this kind of
> architecture is tremendously easier to maintain
> (e.g., when moving from one version of Eclipse to another)
> than any other solution we could see.
>
> Experience #2:
> We are currently experimenting how OT/Equinox can profitably be
> combined with model driven/generative techniques. More specifically,
> we are using GMF and GMF-based graphical editors, to create an editor
> for a custom extension of the UML. Integrating the different pieces
> is done using OT/Equinox. Initial results look very promising for this
> kind of use cases, too.
>
> My feelings are that adapting an IMP-generated IDE using OT/Equinox
> should be fairly easy and produce working, maintainable solutions.
>
> Since I don't assume IMP users to be familiar with OT/J and OT/Equinox
> I'd offer to assist anybody trying the combination of OT/Equinox and IMP.
> I'm pretty sure the battle for *complete* built-in extensibility is
> lost the very moment you start fighting. But I don't think you have
> to start fighting that battle ;-)
>
> cheers,
> Stephan
>
Re: Extensibility of IMP [message #14891 is a reply to message #14861] Sun, 11 November 2007 23:16 Go to previous message
Stephan Herrmann is currently offline Stephan Herrmann
Messages: 1024
Registered: July 2009
Senior Member
Stan,

> Thanks for your general interest and specific response on this.
I've always been excited about good development environments
(something we used to call names like SEE or PSEE etc, right?)
and the IMP sure looks very promising
(just too bad it comes too late for Object Teams ;-/ )

> I'm sure it would be
> interesting to us to see what kinds of modifications to an IMP IDE you
> or others would want to apply using AO techniques.
I suggest to use OT/Equinox for *any* desired extensibility outside the main
focus of IMP, i.e., where you cannot or don't (currently) want to provide
explicit extensibility.
I haven't yet seen a desirable scenario, where extending a plug-in using OT
did not succeed ;-)

> Also, we might have
> a conversation sometime (off-line) about what you did in adapting the
> JDT core and UI. If that won't give you too much of a headache!
No problem. Before switching to direct communication I'll give an overview
of what we've done (for the records):

Our adaptation of the JDT core is perhaps not on the focus of this
discussion because this is made as a fork of the original sources.
Just for completeness:
We extended more or less all features of the JDT core to cope with those
constructs and relationships that OT/J adds to Java. This adaptation
comprises approx. 1900 individual patch chunks plus 178 additional classes.
We haven't investigated yet, whether this adaptation could possibly be done
by aspects, too. The mere size and complexity of our edits, however,
suggests, that *no* technology could buy us true loose coupling between the
original source and our adaptations.


Other than the JDT core, we've adapted several plug-ins using aspects, only.
Here's a brief summary of our aspects (36 "team" classes in 6 plug-ins):
(explanations are kept at a minimum just to give an overview,
some features might be clearer when also consulting
http://www.objectteams.org/distrib/features.html )

Adapting org.eclipse.ui.workbench:
+ adapt the "about bundles" dialog to display which plug-in is being adapted
by which aspect-bundle.

Adapting org.eclipse.jdt.core and org.eclipse.pde.core:
+ managing incremental building for OT/J and mediating between pde.core and
jdt.core as to enforce specific rules when compiling OT/Equinox projects.
(We're adapting the jdt.core with aspects although we "own" a fork of it.
Doing so helps as to significantly reduce scattering of the concern of
incremental building).

Adapting org.eclipse.jdt.ui and a few others:
+ adapting structure viewers:
- structure: also display OT/J-elements, filter some generated elements.
- icons: icons for new elements plus new decorations for existing elements
(incl. Java Type Indicators).
+ adapting the package explorer to optionally show the physical or logical
structure of role files (physical meaning: compilation units in packages;
logical meaning: roles as contained in a team, hiding compilation units of
nested elements).
+ adapting the type hierarchy view to respect two kinds of inheritance
(plus the same for method override indicators).
+ adapting the call hierarchy to respect aspect-induced control flows
+ adapting completion and quick-fixes
- to support new elements as locations for completion and quick-fix
- to support new completion proposals directly based on information from
the compiler.
- to support new quick-fixes for OT/J-specific problems
(this is to a large part be done using the regular extension point)
+ adapt rewriting operations, most specifically "organize imports" to
distinguish two different kinds of imports depending on the usage context.
+ adapt the quick-outline to include even more elements when ctrl-o is
pressed twice.

Adapting refactoring related stuff in org.eclipse.jdt.ui:
+ tell some processors/utilities to handle new elements and respect semantic
relationships introcuded by OT/J.

Adapting org.eclipse.pde.ui:
+ Perform some more actions when using the create sample wizard to create
a OT-sample project.
+ Perform additional initialization when creating a plug-in project
(iff it is created as an OT/Equinox project).


Furthermore we have back-ported some patches in Eclipse HEAD into the
current stable version by using aspects. Also a set of patches that I
contributed to the JDT UI was first developed as an aspect.

OK, I don't know if just this list is of help for anybody. Should you be
interested about any specific kind of adaptation feel free to ask me
directly.


best,
Stephan
Re: Extensibility of IMP [message #566941 is a reply to message #7349] Mon, 08 October 2007 14:00 Go to previous message
Robert M. Fuhrer is currently offline Robert M. Fuhrer
Messages: 294
Registered: July 2009
Senior Member
Oleg Mürk wrote:
> Hello,
>
> I would like to understand the limits of extensibility of IMP.
> Obviously, one can do everything that can be configured with the
> extension points, but what happens when this is not sufficient? Are
> there any mechanisms for such case?
>
> Here are some of examples I could come up with:
> * Showing auto-complete after "." is entered without having to press
> ctrl-space.
> * Using drag & drop on the parts the edited document.
> * Showing additional annotations of the program text.
> * Automatically folding the parts of the program that are far enough
> from the cursor.

First, the specific ideas you mentioned are all really nice ideas, and
ones that IMP probably should support.

> I don't mean to imply that the support for arbitrary extensions of IMP
> is neccessary, just want to find out the limits :)

More generally, I hope you can appreciate that this is a *very* tough
question. :-) [Hence my delay in answering as I chewed on it...]

Unfortunately, I don't have a particularly good answer, but I wonder whether
any practical answer is possible. This question goes right to the heart of
software extensibility. In my experience, generally, in order for software
to be extensible in any particular fashion, it must be intentionally
designed to be so.

For example, for an OO class to be extensible, the right fields/methods have
to be made visible to derived classes, and the behavior has to be factored
such that one can override the bits that need customization, while permitting
reuse of the common bits. At the higher level, the same goes for Eclipse and
its extension point mechanism: it enables extensibility, but doesn't directly
provide it. If the software wasn't designed to support a particular kind of
extension, it probably won't support it (well, or at all), and you'll have to
modify the software to make it do so.

So, the best answer I can think of is: IMP is not finished, so please join
the effort, and bring your ideas, and we'll work together to make sure it
supports a very good set of IDE features, and is extensible in all the right
ways to enable reasonable customizations.

--
Cheers,
-- Bob

--------------------------------
Robert M. Fuhrer
Research Staff Member
Programming Technologies Dept.
IBM T.J. Watson Research Center

IMP Team Lead (http://eclipse-imp.sourceforge.net)
X10: Productivity for High-Performance Parallel Programming (http://x10.sf.net)
Re: Extensibility of IMP [message #566993 is a reply to message #7349] Mon, 08 October 2007 15:34 Go to previous message
Stan Sutton is currently offline Stan Sutton
Messages: 121
Registered: July 2009
Senior Member
Oleg Mürk wrote:
> Hello,
>
> I would like to understand the limits of extensibility of IMP.
> Obviously, one can do everything that can be configured with the
> extension points, but what happens when this is not sufficient? Are
> there any mechanisms for such case?
>
> Here are some of examples I could come up with:
> * Showing auto-complete after "." is entered without having to press
> ctrl-space.
> * Using drag & drop on the parts the edited document.
> * Showing additional annotations of the program text.
> * Automatically folding the parts of the program that are far enough
> from the cursor.
>
> I don't mean to imply that the support for arbitrary extensions of IMP
> is neccessary, just want to find out the limits :)
>
> Best,
> OM

I generally agree with what Bob has said in his earlier response. We've
started by trying to support extensibility through existing linguistic
or environmental mechanisms and there are limits to the kinds of
extensions that those mechanisms can support for a given IDE instance.
(We might be able to get some additional extensibility using
aspect-oriented techniques, but I won't try to speculate about that here.)

The examples you give are suggestive. I think we could probably support
a couple of them with relatively minor modifications of our code base.

Regarding additional annotations on a program text, I expect that
annotations are typically the output of some tool that runs on the text.
In light of that, having "additional annotations" might mean enabling
an existing tool (such as a parser) to post two distinct sets of
annotations, or allowing additional tools (other than, say, a parser) to
post annotations. In IMP currently, annotations get put onto a program
text by a "MessageHandler" that is provided to the parser. Allowing one
tool to post different kinds of annotations might mean making the the
tool (and its driver) sensitive to the possibility of providing multiple
MessageHandlers, or it might entail generalizing our MessageHandler to
be sensitive to the kind of annotations it is handling (i.e., to be able
to post multiple kinds of annotations). The latter would be a change to
our library class, but a localized one.

Regarding the automatic folding of parts of a text that are at a certain
distance from the cursor, I think we could probably handle that pretty
straightforwardly with an alternative base class for our folding
updater. Of course, then we'd have to modify the wizard for the folding
service to allow the user to choose which type of folding updater they
wanted. Again, this is a change, but a fairly localized one.

Regarding "auto-auto-completion", I think that our existing content
proposer might work fairly well for that (maybe it would take a little
tuning). I think the main thing would be to set up the automated
invocation, which I think we could do by adding a new service that could
be automatically invoked by the editor without manual intervention. For
that I suppose we'd need at least new extension point and a driver (what
we might call an "auto-auto-completion-controller"). We could probably
make it run automatically, given a content proposer, although we might
want to also provide some menu actions to allow the user to turn it on
and off. If it requires a kind of content proposer that would differ
from our usual one, then we'd need a new template or base class and
possibly a wizard for the user to instantiate the service. This might
be a bit more of a change than the previous cases, but it's still fairly
contained.

Regarding drag and drop, I have no idea. Without any knowledge of how
that might work, all I can do is speculate that it might require a new
base class for the editor (i.e., not the TextEditor on which the IMP
editor is based). Presumably that would entail some major work.

I haven't tried to actually work out any of these solutions, but it is
interesting to think about the problems. It seems that these
suggestions touch on two general cases where code needs to be modified
but the modifications are relatively contained: modifying an existing
service, and adding a new service (albeit one that is like an existing
service). If a fairly wide range of desirable and interesting
variations can be accommodated by such localized modifications of code,
then I think we can claim some measure of success for our architecture.
But I'm sure we can do a better job of preparing for this. And this
is a good fodder for writing up some documentation on what IMP
developers (or IMP IDE developers) might do in these cases.

Thank you!

Stan
Re: Extensibility of IMP [message #567042 is a reply to message #7488] Tue, 09 October 2007 11:16 Go to previous message
Eclipse User
Originally posted by: olegm.gmail.com

Hello,

Robert M. Fuhrer wrote:
>> I don't mean to imply that the support for arbitrary extensions of IMP
>> is neccessary, just want to find out the limits :)
>
> More generally, I hope you can appreciate that this is a *very* tough
> question. :-) [Hence my delay in answering as I chewed on it...]
>
> Unfortunately, I don't have a particularly good answer, but I wonder
> whether any practical answer is possible. This question goes right to the heart of
> software extensibility. In my experience, generally, in order for software
> to be extensible in any particular fashion, it must be intentionally
> designed to be so.

True. One way to design for variability is to create a bunch of usage
scenarios that vary as much as possible and make sure the framework
supports them.

> For example, for an OO class to be extensible, the right fields/methods
> have to be made visible to derived classes, and the behavior has to be factored
> such that one can override the bits that need customization, while
> permitting reuse of the common bits. At the higher level, the same goes for Eclipse
> and its extension point mechanism: it enables extensibility, but doesn't
> directly provide it. If the software wasn't designed to support a particular kind of
> extension, it probably won't support it (well, or at all), and you'll have to
> modify the software to make it do so.

Mainly, I would like to understand why Eclipse extension points were
chosen as opposed to say having a toolkit framework with a bunch of base
classes, decorators/mix-ins, and utilities. In the former case
assembling a particular IDE would mean wiring up the pieces of the
toolkit in a desirable manner. In my opinion this approach allows to
accommodate for unexpected variability quite easily at the compile time.

Eclipse extension points are (in my understanding, I'm quite new to
Eclipse) a mechanism for deployment time variability --- i.e. deploy a
shared plugin, later deploy other plugin that customizes it. After all
the Eclipse UI framework (org.eclipse.ui) is not a set of plugins, but a
toolkit and I think for a good reason.

I can still see a case for deployment time variability of IDEs (e.g.
plug in my own token colorer, new refactoring, etc), but this is a
variability of the IDE plugin (which could be achieved using Eclipse
extension points), not of the shared IMP functionality.

The main bottleneck of the current approach, as I see it, is that it is
VERY easy to get a decent IDE with standard functionality (I've just
tried it, even without LPG, although IMP is a bit alpha now), however
once You get out of the variability space determined by the IMP
extension points the only ways to go are:
* Submit a change request for IMP (or a patch)
* Fork-off IMP and do Your own changes
While the toolkit approach would allow to reuse the suitable parts of
the IMP framework and to customize the rest without modifying the IMP
itself.

Keep up the good work,
Oleg Mürk
Re: Extensibility of IMP [message #567099 is a reply to message #7532] Wed, 10 October 2007 22:23 Go to previous message
Stan Sutton is currently offline Stan Sutton
Messages: 121
Registered: July 2009
Senior Member
Oleg Murk wrote:
>
> ...
>
> The main bottleneck of the current approach, as I see it, is that it is
> VERY easy to get a decent IDE with standard functionality (I've just
> tried it, even without LPG, although IMP is a bit alpha now), however
> once You get out of the variability space determined by the IMP
> extension points the only ways to go are:
> * Submit a change request for IMP (or a patch)
> * Fork-off IMP and do Your own changes
> While the toolkit approach would allow to reuse the suitable parts of
> the IMP framework and to customize the rest without modifying the IMP
> itself.
>
> Keep up the good work,
> Oleg Mürk

Hi Oleg,

Now that we have started to "go public" with IMP, we are hearing from
(potential) users, both within and outside of IBM, who are interested in
a variety of IDE development or integration scenarios, many of which do
not fit precisely with our initial design points for variability.
Actually, we ourselves are considering how to vary some of the IMP
framework for different scenarios.

I think it would be fair to say that IMP provides something of a
toolkit, but one in which the most obvious pieces are oriented toward
service implementations. Many IMP services are implemented as
extensions of Eclipse extension points but actually some are not.
Still, it is true that the style of these is more or less extension-like
in that they define a locus of customization that extends the IMP
framework in a standard way.

Of the extension points used by IMP, some are defined by Eclipse and
some are defined by IMP. One way for an IDE developer to further
customize an IMP IDE might be to add their own extension points for
their own extensions (for services or whatever). Still, this doesn't
get you into customization of the more fundamental parts of the IMP
framework.

For that you might have to fork off your own IMP variant. As I said
in a separate response in this thread, it may be that several kinds of
framework-level customizations may be possible without too much pain (or
at least without making very extensive changes). Actually, we have our
own experience with this in relation to Eclipse. At least through
Eclipse 3.2 the facilities for working with the extensions model and the
plug-in dependencies model are not suited to the kind of incremental
elaboration that is necessary for IMP IDE development. For versions of
IMP that worked with Eclipse up to version 3.1 we actually had our own
version of the org.eclipse.pde.core plug-in, mostly unchanged, but with
half-a-dozen key modifications. Now for Eclipse version 3.2 we have our
own representation of the plug-in extension and dependency models that
allow us to work with the Eclipse models in appropriate ways (these are
just a few classes). So we have been maintaining our own fork of
Eclipse. We didn't have to write so much code to achieve the desired
effect, but understanding what Eclipse did and how to adapt to that took
quite a bit of work.

IMP does not aim for universal extensibility, but we can strive for a
design that would make it easy for IMP users to fork IMP in important
respects.

Regards,

Stan
Re: Extensibility of IMP [message #568845 is a reply to message #7511] Thu, 08 November 2007 23:53 Go to previous message
Stephan Herrmann is currently offline Stephan Herrmann
Messages: 1024
Registered: July 2009
Senior Member
Hi Stan!
Hi Bob!

Stanley Sutton wrote:
> (We might be able to get some additional extensibility using
> aspect-oriented techniques, but I won't try to speculate about that here.)

Ok, this looks like the perfect cue for my part ;-)
i.e., I believe I can offer some help here because we have ...
+ the technology to use a disciplined version of AOP for Eclipse plug-ins
+ experience of using this technology for
- extending the JDT
- adapting and composing generated pieces of software.

Given the availability of aspect technology to support unanticipated
adaptation, this could (at least for the time being) disburden the
IMP project from the commitment to get every dimension of extensibility
right in the first place.
Wouldn't it be more beneficial if the IMP project (for the time being)
would focus more on completeness, so we can soonish generate IDEs that can
actually kind-of compete with the JDT in terms of feature-richness (is
that an english word?) and convenience??

IMP users wanting to fine tune their IDEs can do so now without forking
by using an aspect plug-in. After several users have done similar
extensions using aspects, it should be much easier to migrate that
variability down into the core.

To stop philosophical rambling on what should or shouldn't be done,
let me give some details about what we have to offer:

Technology:
The language ObjectTeams/Java (OT/J, see www.objectteams.org) has been
integrated with Equinox (yielding OT/Equinox) as described in this article:
http://www.jot.fm/issues/issue_2007_10/paper6.pdf

Experience #1:
The Object Teams Development Tooling was developed by forking the
JDT core (sigh) and by adapting other plugins, most prominently the
JDT UI using aspect plug-ins written in OT/J. The resulting IDE supports
all JDT features even for OT/J code and seamlessly integrates new features
(see www.objectteams.org/distrib/features.html).
By this we demonstrate that OT/Equinox is powerful enough to do very
effective adaptations and we have experienced that this kind of
architecture is tremendously easier to maintain
(e.g., when moving from one version of Eclipse to another)
than any other solution we could see.

Experience #2:
We are currently experimenting how OT/Equinox can profitably be
combined with model driven/generative techniques. More specifically,
we are using GMF and GMF-based graphical editors, to create an editor
for a custom extension of the UML. Integrating the different pieces
is done using OT/Equinox. Initial results look very promising for this
kind of use cases, too.

My feelings are that adapting an IMP-generated IDE using OT/Equinox
should be fairly easy and produce working, maintainable solutions.

Since I don't assume IMP users to be familiar with OT/J and OT/Equinox
I'd offer to assist anybody trying the combination of OT/Equinox and IMP.
I'm pretty sure the battle for *complete* built-in extensibility is
lost the very moment you start fighting. But I don't think you have
to start fighting that battle ;-)

cheers,
Stephan

--
Stephan Herrmann
Re: Extensibility of IMP [message #568876 is a reply to message #14831] Fri, 09 November 2007 18:24 Go to previous message
Stan Sutton is currently offline Stan Sutton
Messages: 121
Registered: July 2009
Senior Member
Hi Stephan!

Thanks for your general interest and specific response on this. I think
that what you are proposing makes a lot of sense. IMP is really a
project to investigate development of IDEs, not a project to investigate
extensibility. As a practical matter it has adopted some
extensibility mechanisms but that is in support of the primary goal of
building customized IDEs. Since we want the IDEs to be customizable,
naturally we have to think sometimes about whether the extensibility
mechanisms we use are the most effective for that. In any case, as you
say, complete built-in extensibility is a lost cause from the outset,
but the mechanisms we use do work well enough to allow us to move
forward with creating more complete IDEs.

The soundness of the OT approach seems to be well supported, so your
suggestion makes a lot of sense (in my opinion). I'm sure it would be
interesting to us to see what kinds of modifications to an IMP IDE you
or others would want to apply using AO techniques. Also, we might have
a conversation sometime (off-line) about what you did in adapting the
JDT core and UI. It that won't give you too much of a headache!

Regards,

Stan


Stephan Herrmann wrote:
> Hi Stan!
> Hi Bob!
>
> Stanley Sutton wrote:
>> (We might be able to get some additional extensibility using
>> aspect-oriented techniques, but I won't try to speculate about that here.)
>
> Ok, this looks like the perfect cue for my part ;-)
> i.e., I believe I can offer some help here because we have ...
> + the technology to use a disciplined version of AOP for Eclipse plug-ins
> + experience of using this technology for
> - extending the JDT
> - adapting and composing generated pieces of software.
>
> Given the availability of aspect technology to support unanticipated
> adaptation, this could (at least for the time being) disburden the
> IMP project from the commitment to get every dimension of extensibility
> right in the first place.
> Wouldn't it be more beneficial if the IMP project (for the time being)
> would focus more on completeness, so we can soonish generate IDEs that can
> actually kind-of compete with the JDT in terms of feature-richness (is
> that an english word?) and convenience??
>
> IMP users wanting to fine tune their IDEs can do so now without forking
> by using an aspect plug-in. After several users have done similar
> extensions using aspects, it should be much easier to migrate that
> variability down into the core.
>
> To stop philosophical rambling on what should or shouldn't be done,
> let me give some details about what we have to offer:
>
> Technology:
> The language ObjectTeams/Java (OT/J, see www.objectteams.org) has been
> integrated with Equinox (yielding OT/Equinox) as described in this article:
> http://www.jot.fm/issues/issue_2007_10/paper6.pdf
>
> Experience #1:
> The Object Teams Development Tooling was developed by forking the
> JDT core (sigh) and by adapting other plugins, most prominently the
> JDT UI using aspect plug-ins written in OT/J. The resulting IDE supports
> all JDT features even for OT/J code and seamlessly integrates new features
> (see www.objectteams.org/distrib/features.html).
> By this we demonstrate that OT/Equinox is powerful enough to do very
> effective adaptations and we have experienced that this kind of
> architecture is tremendously easier to maintain
> (e.g., when moving from one version of Eclipse to another)
> than any other solution we could see.
>
> Experience #2:
> We are currently experimenting how OT/Equinox can profitably be
> combined with model driven/generative techniques. More specifically,
> we are using GMF and GMF-based graphical editors, to create an editor
> for a custom extension of the UML. Integrating the different pieces
> is done using OT/Equinox. Initial results look very promising for this
> kind of use cases, too.
>
> My feelings are that adapting an IMP-generated IDE using OT/Equinox
> should be fairly easy and produce working, maintainable solutions.
>
> Since I don't assume IMP users to be familiar with OT/J and OT/Equinox
> I'd offer to assist anybody trying the combination of OT/Equinox and IMP.
> I'm pretty sure the battle for *complete* built-in extensibility is
> lost the very moment you start fighting. But I don't think you have
> to start fighting that battle ;-)
>
> cheers,
> Stephan
>
Re: Extensibility of IMP [message #568894 is a reply to message #14861] Sun, 11 November 2007 23:16 Go to previous message
Stephan Herrmann is currently offline Stephan Herrmann
Messages: 1024
Registered: July 2009
Senior Member
Stan,

> Thanks for your general interest and specific response on this.
I've always been excited about good development environments
(something we used to call names like SEE or PSEE etc, right?)
and the IMP sure looks very promising
(just too bad it comes too late for Object Teams ;-/ )

> I'm sure it would be
> interesting to us to see what kinds of modifications to an IMP IDE you
> or others would want to apply using AO techniques.
I suggest to use OT/Equinox for *any* desired extensibility outside the main
focus of IMP, i.e., where you cannot or don't (currently) want to provide
explicit extensibility.
I haven't yet seen a desirable scenario, where extending a plug-in using OT
did not succeed ;-)

> Also, we might have
> a conversation sometime (off-line) about what you did in adapting the
> JDT core and UI. If that won't give you too much of a headache!
No problem. Before switching to direct communication I'll give an overview
of what we've done (for the records):

Our adaptation of the JDT core is perhaps not on the focus of this
discussion because this is made as a fork of the original sources.
Just for completeness:
We extended more or less all features of the JDT core to cope with those
constructs and relationships that OT/J adds to Java. This adaptation
comprises approx. 1900 individual patch chunks plus 178 additional classes.
We haven't investigated yet, whether this adaptation could possibly be done
by aspects, too. The mere size and complexity of our edits, however,
suggests, that *no* technology could buy us true loose coupling between the
original source and our adaptations.


Other than the JDT core, we've adapted several plug-ins using aspects, only.
Here's a brief summary of our aspects (36 "team" classes in 6 plug-ins):
(explanations are kept at a minimum just to give an overview,
some features might be clearer when also consulting
http://www.objectteams.org/distrib/features.html )

Adapting org.eclipse.ui.workbench:
+ adapt the "about bundles" dialog to display which plug-in is being adapted
by which aspect-bundle.

Adapting org.eclipse.jdt.core and org.eclipse.pde.core:
+ managing incremental building for OT/J and mediating between pde.core and
jdt.core as to enforce specific rules when compiling OT/Equinox projects.
(We're adapting the jdt.core with aspects although we "own" a fork of it.
Doing so helps as to significantly reduce scattering of the concern of
incremental building).

Adapting org.eclipse.jdt.ui and a few others:
+ adapting structure viewers:
- structure: also display OT/J-elements, filter some generated elements.
- icons: icons for new elements plus new decorations for existing elements
(incl. Java Type Indicators).
+ adapting the package explorer to optionally show the physical or logical
structure of role files (physical meaning: compilation units in packages;
logical meaning: roles as contained in a team, hiding compilation units of
nested elements).
+ adapting the type hierarchy view to respect two kinds of inheritance
(plus the same for method override indicators).
+ adapting the call hierarchy to respect aspect-induced control flows
+ adapting completion and quick-fixes
- to support new elements as locations for completion and quick-fix
- to support new completion proposals directly based on information from
the compiler.
- to support new quick-fixes for OT/J-specific problems
(this is to a large part be done using the regular extension point)
+ adapt rewriting operations, most specifically "organize imports" to
distinguish two different kinds of imports depending on the usage context.
+ adapt the quick-outline to include even more elements when ctrl-o is
pressed twice.

Adapting refactoring related stuff in org.eclipse.jdt.ui:
+ tell some processors/utilities to handle new elements and respect semantic
relationships introcuded by OT/J.

Adapting org.eclipse.pde.ui:
+ Perform some more actions when using the create sample wizard to create
a OT-sample project.
+ Perform additional initialization when creating a plug-in project
(iff it is created as an OT/Equinox project).


Furthermore we have back-ported some patches in Eclipse HEAD into the
current stable version by using aspects. Also a set of patches that I
contributed to the JDT UI was first developed as an aspect.

OK, I don't know if just this list is of help for anybody. Should you be
interested about any specific kind of adaptation feel free to ask me
directly.


best,
Stephan
Previous Topic:Eclipse IMP project creation review
Next Topic:LPG
Goto Forum:
  


Current Time: Sun Oct 26 08:03:31 GMT 2014

Powered by FUDForum. Page generated in 0.14914 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software