Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Language IDEs » Objectteams » IDE for a language embedded in Java
IDE for a language embedded in Java [message #567839] Thu, 21 January 2010 04:50 Go to next message
Stephan Herrmann is currently offline Stephan Herrmann
Messages: 995
Registered: July 2009
Senior Member
In http://www.eclipse.org/forums/eclipse.tools.imp a question caught my attention: what's the easiest way to write an IDE for a language embedded into Java?

My natural answer is: "Don't write the IDE, take the JDT and just add what you want to add".

In order to validate if OT/Equinox is indeed the technology that solves this and to see how easily this can really be done I wrote a tiny little plugin, which can be found at
http://www.objectteams.org/distrib/demos/embedded-expression -language.zip

This plugin implements the most simple expression language that I could think of: literals for small ints can be written in English, so zero means 0, one means 1, two means 2.

When embedding this into Java you can write:

public class EmbeddingTest {
private static int foo() {
return <% one %>;
}

public static void main(String[] args) {
System.out.println(foo());
}
}

Guess what, when you run this program it prints:
1
:)

Follows: description of how this is implemented:

Class CustomIntLiteral implements the "language": Fields store the source ("one") and the source position. Method resolve(..) looks up the source in an array of number words and uses the index as the literal's value. Nothing magic. Plain Java extension of org.eclipse.jdt.internal.compiler.ast.IntLiteral.

All magic is in class SyntaxAdaptor: This team contains two roles for the two stages of adaptation:

Stage 1

Role ScannerAdaptor is responsible for intercepting the detection of a '<' during scanning. This is the full source code of this role:
protected class ScannerAdaptor playedBy Scanner {

// access fields from Scanner ("callout bindings"):
int getCurrentPosition() -> get int currentPosition;
void setCurrentPosition(int currentPosition) -> set int currentPosition;
char[] getSource() -> get char[] source;

// intercept this method from Scanner ("callin binding"):
int getNextToken() <- replace int getNextToken();

callin int getNextToken() throws InvalidInputException {
// invoke the original method:
int token = base.getNextToken();
if (token == TerminalTokens.TokenNameLESS) {
char[] source = getSource();
int pos = getCurrentPosition();
int start = pos+1;
if (source[pos] == '%') {
try {
while (source[++pos] != '%' || source[++pos] != '>')
; // empty
} catch (ArrayIndexOutOfBoundsException aioobe) {
return token;
}
int end = pos-1;
char[] fragment = CharOperation.subarray(source, start, end);
setCurrentPosition(pos + 1);
// prepare an inner adaptor to intercept the expected parser action
new InnerAdaptor(fragment, start, end).activate();
return TerminalTokens.TokenNamenull;
}
}
return token;
}
}
Hints for the OT/J novice:


"class ScannerAdaptor playedBy Scanner" denotes that this class is a role adapting the given class Scanner.
lines containing "-> get" or "-> set" define a "callout" binding whereby the role can access members of its base class Scanner, here get-/set-accessors for some fields are defined.
the line containing "<- replace" defines a "callin" binding whereby the role intercepts calls to the method at the right hand side, and replaces these calls with calls to the method at the left hand side (side note: for a "replace" binding the role method bound here has to be marked with a "callin" modifier).
inside the role method "getNextToken" the only special syntax is "base.getNextToken()" which is used to invoke the original method being intercepted (here: from Scanner)

The logic in this method simply checks if we found a LESS token, if it is followed by '%', and searches the terminating '%' '>'. The source fragment between the special delimiters is extracted and passed to the second stage, which is the responsibility of an InternalAdaptor.

Stage 2

Class InternalAdaptor is both a role of SyntaxAdaptor and also a team with a nested role. The reason for making this a team is: now it has the capability to be activated at certain points during runtime (see the activate() call immediately following the instantion of InnerAdaptor). I.e., when stage 1 has detected a <% something %> string, it activates a fresh instance of InnerAdaptor which enables a callin binding in the nested role ParserAdaptor. This stage is responsible for feeding some hand-crafted AST into the parser. The InnerAdaptor with its nested role looks like this:
protected team class InnerAdaptor {
char[] source;
int start, end;
protected InnerAdaptor(char[] source, int start, int end) {
this.source = source;
this.start = start;
this.end = end;
}

/** This inner role does the real work of the InnerAdaptor. */
protected class ParserAdaptor playedBy Parser {
// import methods from Parser ("callout bindings"):
@SuppressWarnings("decapsulation")
void pushOnExpressionStack(Expression expr) -> void pushOnExpressionStack(Expression expr);

// intercept this method from Parser ("callin binding"):
void consumeToken(int type) <- replace void consumeToken(int type);

@SuppressWarnings("basecall")
callin void consumeToken(int type) {
if (type == TerminalTokens.TokenNamenull) {
InnerAdaptor.this.deactivate(); // this inner adaptor has done its job, no longer intercept
// TODO analyse source to find what AST should be created
Expression replacement = new CustomIntLiteral(source, start, end);
// feed custom AST into the parser:
this.pushOnExpressionStack(replacement);
return;
}
// shouldn't happen: only activated when scanner returns TokenNamenull
base.consumeToken(type);
}
}
}
We see the same constructs (playedBy, callout, callin, base-call).
The logic is: after the ScannerAdaptor has detected the special tokens it activated an InnerAdaptor and returns a null token, which tells the Parser we saw a legal expression token. This null token is now intercepted in consumeToken(int). Instead of creating a null literal a CustomIntLiteral is constructed and pushed on the expression stack. Well, at this point the instance of InnerAdaptor has done its one-shot job, so it is deactivated again leaving it as prey for the garbage collector.

We're done programming!

Now we only have to tell OT/Equinox about our teams using the following snippet in plugin.xml:
<extension
point="org.objectteams.otequinox.aspectBindings">
<aspectBinding
icon=" platform:/plugin/org.objectteams.otdt.ui/icons/ot/calloutbin ding_obj.gif ">
<basePlugin
icon=" platform:/plugin/org.eclipse.pde.ui/icons/obj16/plugin_obj.g if "
id="org.eclipse.jdt.core">
</basePlugin>
<team
activation="ALL_THREADS"
class="embedding.jdt.SyntaxAdaptor"
icon=" platform:/plugin/org.objectteams.otdt.ui/icons/ot/team_obj.g if ">
</team>
<team
activation="NONE"
class="embedding.jdt.SyntaxAdaptor$__OT__InnerAdaptor"
icon=" platform:/plugin/org.objectteams.otdt.ui/icons/ot/team_obj.g if ">
</team>
</aspectBinding>
</extension>
This anounces that base plugin org.eclipse.jdt.core will be adapted by two teams, SyntaxAdaptor and InnerAdaptor (ignore the man behind the curtain, the __OT__ prefix shouldn't be needed here, to be fixed soon).
The first team is globally activated (ALL_THREADS) while the second awaits programmatic activation (NONE).

Running


Import this plugin into a workspace of your OTDT installation.
Launch a runtime workbench
In the runtime workbench

create a Java project containing the above class EmbeddingTest
run as Java Application
1



When playing with this I accidentally declared foo to return char[]. Guess what: the JDT found that '<% one %>' is of type int and offered to adjust foo() to returning int. Quickfix offered at the place of your special syntax :)

I think this was fun! Starting from here, integrating more serious sub-languages into the JDT should be mostly straight-forward.

best,
Stephan
Re: IDE for a language embedded in Java [message #567873 is a reply to message #567839] Thu, 21 January 2010 08:28 Go to previous messageGo to next message
giovanni  is currently offline giovanni
Messages: 18
Registered: November 2009
Junior Member
Hi Stephan,
I tried your example and it is potentially wonderful to get what we have in mind.
Of course I know that this is a first step but a lot of aspects will be involved to have a complete editor..anyway I'm going to study more about OT/J and AST. I'll definitively come back to you, hoping in your help.
Thanks,
Giovanni
Re: IDE for a language embedded in Java [message #567903 is a reply to message #567873] Thu, 21 January 2010 12:01 Go to previous messageGo to next message
Stephan Herrmann is currently offline Stephan Herrmann
Messages: 995
Registered: July 2009
Senior Member
Giovanni,

> I tried your example and it is potentially wonderful to get what we have
> in mind.

I'm glad you like it :)

> Of course I know that this is a first step but a lot of aspects will be
> involved to have a complete editor..anyway I'm going to study more about
> OT/J and AST. I'll definitively come back to you, hoping in your help.

sure, feel free to ask,
Stephan
Re: IDE for a language embedded in Java [message #568387 is a reply to message #567839] Wed, 17 February 2010 07:22 Go to previous messageGo to next message
Jan Marc Hoffmann is currently offline Jan Marc Hoffmann
Messages: 32
Registered: July 2009
Member
Hi all,

I have been working in the area. But in my case it would be also interesting to adjust the org.eclipse.jdt.core.dom AST, since this is the AST which other plugins use to manipulate the source.

Following Stephans example i have been adapting the dom AST with OT/J.

The primary class to adapt is the ASTConverter. This class converts the compiler AST to a dom AST. My goal was to create my own special NumberLiteral Node (DomCustomIntLiteral). Due to access restrictions in the dom package I had to adapt the NumberLiteral Node with a Role instead of just extending it. The second step was to let the ASTConverter create DomCustomIntLiteral instead of NumberLiteral for each CustomIntLiteral Node in the compiler AST.

Here is the code:


protected class DomConverterAdaptor playedBy ASTConverter {
/*
* Callout
*/
void removeLeadingAndTrailingCommentsFromLiteral(
DomCustomIntLiteral node)
->
void removeLeadingAndTrailingCommentsFromLiteral(ASTNode node);
void recordNodes(DomCustomIntLiteral node,
org.eclipse.jdt.internal.compiler.ast.ASTNode oldASTNode) ->
void recordNodes(ASTNode node,
org.eclipse.jdt.internal.compiler.ast.ASTNode oldASTNode);
boolean getResolveBindings() -> get boolean resolveBindings;
char[] getCompilationUnitSource() -> get char[] compilationUnitSource;
AST getAst() -> get AST ast;

Expression convert(org.eclipse.jdt.internal.compiler.ast.Expression expression) <- replace Expression convert(org.eclipse.jdt.internal.compiler.ast.Expression expression);

callin Expression convert(org.eclipse.jdt.internal.compiler.ast.Expression expression){
if (expression instanceof CustomIntLiteral) {
return convertCustomIntLiteral((CustomIntLiteral) expression);
}
return base.convert(expression);
}

// HERE IS THE INTERESTING PART
private DomCustomIntLiteral convertCustomIntLiteral(CustomIntLiteral expression){
int length = expression.sourceEnd - expression.sourceStart + 1;
int sourceStart = expression.sourceStart;
final DomCustomIntLiteral literal = new DomCustomIntLiteral(this.getAst());
literal.internalSetToken(new String(this.getCompilationUnitSource(), sourceStart, length));
if (this.getResolveBindings()) {
this.recordNodes(literal, expression);
}
literal.setSourceRange(sourceStart, length);
removeLeadingAndTrailingCommentsFromLiteral(literal);
return literal;
}
}




protected class DomCustomIntLiteral playedBy NumberLiteral {

/*
* Callout
*/
internalSetToken -> internalSetToken;
setSourceRange -> setSourceRange;

public DomCustomIntLiteral(AST ast){
base(ast);
}

protected abstract void internalSetToken(String string);
protected abstract void setSourceRange(int sourceStart, int length);
}


To use this modified AST you would either need to store all information in Properties to make them available for all programs using this AST or you would need to offer an API to make the Roles available.

This works fine. But there is one problem left:

The DomCustomIntLiteral adapts all NumberLiterals, which it shouldn't. The DomCustomIntLiteral should only be played by the NumberLiterals which got created by the ASTConverter. Is there a way to fix this? I have been looking into Baseguards, but those would need an information in the Base to decide whether this Node is a regular NumberLiteral or a DomCustomIntLiteral.

Thanks and Greetings

Jan Marc
Re: IDE for a language embedded in Java [message #568544 is a reply to message #567839] Thu, 18 February 2010 11:08 Go to previous message
Jan Marc Hoffmann is currently offline Jan Marc Hoffmann
Messages: 32
Registered: July 2009
Member
Thanks for your hint. This pattern was what i was looking for. Sure, feel free to use and modify my example.

greetings
Jan Marc
Previous Topic:IDE for a language embedded in Java
Next Topic:Some questions on design decisions
Goto Forum:
  


Current Time: Thu Aug 21 14:17:20 EDT 2014

Powered by FUDForum. Page generated in 0.02232 seconds