Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Language IDEs » Objectteams » IDE for a language embedded in Java
IDE for a language embedded in Java [message #509051] Thu, 21 January 2010 09:50 Go to next message
Stephan Herrmann is currently offline Stephan HerrmannFriend
Messages: 1853
Registered: July 2009
Senior Member
In http://www.eclipse.org/forums/eclipse.tools.imp a question caught my attention: what's the easiest way to write an IDE for a language embedded into Java?

My natural answer is: "Don't write the IDE, take the JDT and just add what you want to add".

In order to validate if OT/Equinox is indeed the technology that solves this and to see how easily this can really be done I wrote a tiny little plugin, which can be found at
http://www.objectteams.org/distrib/demos/embedded-expression -language.zip

This plugin implements the most simple expression language that I could think of: literals for small ints can be written in English, so zero means 0, one means 1, two means 2.

When embedding this into Java you can write:

public class EmbeddingTest {
	private static int foo() {
		return <% one %>;
	}

	public static void main(String[] args) {
		System.out.println(foo());
	}
}


Guess what, when you run this program it prints:
1

Smile

Follows: description of how this is implemented:

Class CustomIntLiteral implements the "language": Fields store the source ("one") and the source position. Method resolve(..) looks up the source in an array of number words and uses the index as the literal's value. Nothing magic. Plain Java extension of org.eclipse.jdt.internal.compiler.ast.IntLiteral.

All magic is in class SyntaxAdaptor: This team contains two roles for the two stages of adaptation:

Stage 1

Role ScannerAdaptor is responsible for intercepting the detection of a '<' during scanning. This is the full source code of this role:
	protected class ScannerAdaptor playedBy Scanner {

		// access fields from Scanner ("callout bindings"):
		int getCurrentPosition() 	             -> get int currentPosition;
		void setCurrentPosition(int currentPosition) -> set int currentPosition;
		char[] getSource()                           -> get char[] source;

		// intercept this method from Scanner ("callin binding"):
		int getNextToken() <- replace int getNextToken();

		callin int getNextToken() throws InvalidInputException {
			// invoke the original method:
			int token = base.getNextToken();
			if (token == TerminalTokens.TokenNameLESS) {
				char[] source = getSource();
				int pos = getCurrentPosition();
				int start = pos+1; 
				if (source[pos] == '%') {
					try {
						while (source[++pos] != '%' || source[++pos] != '>') 
							; // empty
					} catch (ArrayIndexOutOfBoundsException aioobe) {
						return token;
					}
					int end = pos-1;
					char[] fragment = CharOperation.subarray(source, start, end);
					setCurrentPosition(pos + 1);
					// prepare an inner adaptor to intercept the expected parser action
					new InnerAdaptor(fragment, start, end).activate();
					return TerminalTokens.TokenNamenull;
				}
			}
			return token;
		}
	}

Hints for the OT/J novice:


  1. "class ScannerAdaptor playedBy Scanner" denotes that this class is a role adapting the given class Scanner.
  2. lines containing "-> get" or "-> set" define a "callout" binding whereby the role can access members of its base class Scanner, here get-/set-accessors for some fields are defined.
  3. the line containing "<- replace" defines a "callin" binding whereby the role intercepts calls to the method at the right hand side, and replaces these calls with calls to the method at the left hand side (side note: for a "replace" binding the role method bound here has to be marked with a "callin" modifier).
  4. inside the role method "getNextToken" the only special syntax is "base.getNextToken()" which is used to invoke the original method being intercepted (here: from Scanner)

The logic in this method simply checks if we found a LESS token, if it is followed by '%', and searches the terminating '%' '>'. The source fragment between the special delimiters is extracted and passed to the second stage, which is the responsibility of an InternalAdaptor.

Stage 2

Class InternalAdaptor is both a role of SyntaxAdaptor and also a team with a nested role. The reason for making this a team is: now it has the capability to be activated at certain points during runtime (see the activate() call immediately following the instantion of InnerAdaptor). I.e., when stage 1 has detected a <% something %> string, it activates a fresh instance of InnerAdaptor which enables a callin binding in the nested role ParserAdaptor. This stage is responsible for feeding some hand-crafted AST into the parser. The InnerAdaptor with its nested role looks like this:
protected team class InnerAdaptor {		
	char[] source;
	int start, end;
	protected InnerAdaptor(char[] source, int start, int end) {
		this.source = source;
		this.start = start;
		this.end = end;
	}
	
	/** This inner role does the real work of the InnerAdaptor. */
	protected class ParserAdaptor playedBy Parser {
		// import methods from Parser ("callout bindings"):
		@SuppressWarnings("decapsulation")
		void pushOnExpressionStack(Expression expr) -> void pushOnExpressionStack(Expression expr);
			
		// intercept this method from Parser ("callin binding"):
		void consumeToken(int type) <- replace void consumeToken(int type);
		
		@SuppressWarnings("basecall")
		callin void consumeToken(int type) {
			if (type == TerminalTokens.TokenNamenull) {
				InnerAdaptor.this.deactivate(); // this inner adaptor has done its job, no longer intercept
				// TODO analyse source to find what AST should be created
				Expression replacement = new CustomIntLiteral(source, start, end);
				// feed custom AST into the parser:
				this.pushOnExpressionStack(replacement);
				return;
			}
			// shouldn't happen: only activated when scanner returns TokenNamenull
			base.consumeToken(type);
		}
	}		
}

We see the same constructs (playedBy, callout, callin, base-call).
The logic is: after the ScannerAdaptor has detected the special tokens it activated an InnerAdaptor and returns a null token, which tells the Parser we saw a legal expression token. This null token is now intercepted in consumeToken(int). Instead of creating a null literal a CustomIntLiteral is constructed and pushed on the expression stack. Well, at this point the instance of InnerAdaptor has done its one-shot job, so it is deactivated again leaving it as prey for the garbage collector.

We're done programming!

Now we only have to tell OT/Equinox about our teams using the following snippet in plugin.xml:
<extension
         point="org.objectteams.otequinox.aspectBindings">
      <aspectBinding
            icon="platform:/plugin/org.objectteams.otdt.ui/icons/ot/calloutbinding_obj.gif">
         <basePlugin
               icon="platform:/plugin/org.eclipse.pde.ui/icons/obj16/plugin_obj.gif"
               id="org.eclipse.jdt.core">
         </basePlugin>
         <team
               activation="ALL_THREADS"
               class="embedding.jdt.SyntaxAdaptor"
               icon="platform:/plugin/org.objectteams.otdt.ui/icons/ot/team_obj.gif">
         </team>
         <team
               activation="NONE"
               class="embedding.jdt.SyntaxAdaptor$__OT__InnerAdaptor"
               icon="platform:/plugin/org.objectteams.otdt.ui/icons/ot/team_obj.gif">
         </team>
      </aspectBinding>
   </extension>

This anounces that base plugin org.eclipse.jdt.core will be adapted by two teams, SyntaxAdaptor and InnerAdaptor (ignore the man behind the curtain, the __OT__ prefix shouldn't be needed here, to be fixed soon).
The first team is globally activated (ALL_THREADS) while the second awaits programmatic activation (NONE).

Running


  1. Import this plugin into a workspace of your OTDT installation.
  2. Launch a runtime workbench
  3. In the runtime workbench

    1. create a Java project containing the above class EmbeddingTest
    2. run as Java Application
      1




When playing with this I accidentally declared foo to return char[]. Guess what: the JDT found that '<% one %>' is of type int and offered to adjust foo() to returning int. Quickfix offered at the place of your special syntax Smile

I think this was fun! Starting from here, integrating more serious sub-languages into the JDT should be mostly straight-forward.

best,
Stephan
Re: IDE for a language embedded in Java [message #509132 is a reply to message #509051] Thu, 21 January 2010 13:28 Go to previous messageGo to next message
giovanni  is currently offline giovanni Friend
Messages: 18
Registered: November 2009
Junior Member
Hi Stephan,
I tried your example and it is potentially wonderful to get what we have in mind.
Of course I know that this is a first step but a lot of aspects will be involved to have a complete editor..anyway I'm going to study more about OT/J and AST. I'll definitively come back to you, hoping in your help.
Thanks,
Giovanni
Re: IDE for a language embedded in Java [message #509213 is a reply to message #509132] Thu, 21 January 2010 17:01 Go to previous messageGo to next message
Stephan Herrmann is currently offline Stephan HerrmannFriend
Messages: 1853
Registered: July 2009
Senior Member
Giovanni,

> I tried your example and it is potentially wonderful to get what we have
> in mind.

I'm glad you like it :)

> Of course I know that this is a first step but a lot of aspects will be
> involved to have a complete editor..anyway I'm going to study more about
> OT/J and AST. I'll definitively come back to you, hoping in your help.

sure, feel free to ask,
Stephan
Re: IDE for a language embedded in Java [message #514984 is a reply to message #509051] Wed, 17 February 2010 12:22 Go to previous messageGo to next message
Jan Marc Hoffmann is currently offline Jan Marc HoffmannFriend
Messages: 32
Registered: July 2009
Member
Hi all,

I have been working on a similar problem. In my case it would be also interesting to adjust the org.eclipse.jdt.core.dom AST, since this is the AST which other plugins use to manipulate the source.

Following Stephans example i have been adapting the dom AST with OT/J.

The primary class to adapt is the ASTConverter. This class converts the compiler AST to a dom AST. My goal was to create my own special NumberLiteral Node (DomCustomIntLiteral). Due to access restrictions in the dom package I had to adapt the NumberLiteral Node with a Role instead of just extending it. The second step was to let the ASTConverter create DomCustomIntLiteral instead of NumberLiteral for each CustomIntLiteral Node in the compiler AST.

Here is the code:

	protected class DomConverterAdaptor playedBy ASTConverter {
		/*
		 * Callout
		 */
		void removeLeadingAndTrailingCommentsFromLiteral(
				DomCustomIntLiteral node)
		->
		void removeLeadingAndTrailingCommentsFromLiteral(ASTNode node);
		void recordNodes(DomCustomIntLiteral node,
				org.eclipse.jdt.internal.compiler.ast.ASTNode oldASTNode)		->
		void recordNodes(ASTNode node,
				org.eclipse.jdt.internal.compiler.ast.ASTNode oldASTNode);
		boolean getResolveBindings() -> get boolean resolveBindings;
		char[] getCompilationUnitSource() -> get char[] compilationUnitSource;
		AST getAst() -> get AST ast;

		Expression convert(org.eclipse.jdt.internal.compiler.ast.Expression expression) <- replace Expression convert(org.eclipse.jdt.internal.compiler.ast.Expression expression);
		
		callin Expression convert(org.eclipse.jdt.internal.compiler.ast.Expression expression){
			if (expression instanceof CustomIntLiteral) {
				return convertCustomIntLiteral((CustomIntLiteral) expression);
			}
			return base.convert(expression);
		}
		
                // HERE IS THE INTERESTING PART
		private DomCustomIntLiteral convertCustomIntLiteral(CustomIntLiteral expression){
			int length = expression.sourceEnd - expression.sourceStart + 1;	
			int sourceStart = expression.sourceStart;
			final DomCustomIntLiteral literal = new DomCustomIntLiteral(this.getAst());
			literal.internalSetToken(new String(this.getCompilationUnitSource(), sourceStart, length));
			if (this.getResolveBindings()) {
				this.recordNodes(literal, expression);
			}
			literal.setSourceRange(sourceStart, length);
			removeLeadingAndTrailingCommentsFromLiteral(literal);
			return literal;			
		}
	}



	protected class DomCustomIntLiteral playedBy NumberLiteral {
		
		/*
		 * Callout
		 */
		internalSetToken -> internalSetToken;
		setSourceRange -> setSourceRange;
		
		public DomCustomIntLiteral(AST ast){
			base(ast);
		}
			
		protected abstract void internalSetToken(String string);
		protected abstract void setSourceRange(int sourceStart, int length);
	}


To use this modified AST you would either need to store all information in Properties to make them available for all programs using this AST or you would need to offer an API to make the Roles available.

This works fine. But there is one problem left:

The DomCustomIntLiteral adapts all NumberLiterals, which it shouldn't. The DomCustomIntLiteral should only be played by the NumberLiterals which got created by the ASTConverter. Is there a way to fix this? I have been looking into Baseguards, but those would need an information in the Base to decide whether this Node is a regular NumberLiteral or a DomCustomIntLiteral.

Thanks and Greetings

Jan Marc

[Updated on: Wed, 17 February 2010 12:23]

Report message to a moderator

Re: IDE for a language embedded in Java [message #515017 is a reply to message #514984] Wed, 17 February 2010 14:00 Go to previous messageGo to next message
Stephan Herrmann is currently offline Stephan HerrmannFriend
Messages: 1853
Registered: July 2009
Senior Member
Jan Marc Hoffmann wrote on Wed, 17 February 2010 07:22

The DomCustomIntLiteral adapts all NumberLiterals, which it shouldn't. The DomCustomIntLiteral should only be played by the NumberLiterals which got created by the ASTConverter. Is there a way to fix this? I have been looking into Baseguards, but those would need an information in the Base to decide whether this Node is a regular NumberLiteral or a DomCustomIntLiteral.



That looks pretty successful so far, and yes, you're exactly on that track
I would recommend. Have you looked at our Object Registration Pattern?
I haven't yet looked at the details in your code, but it seems that this pattern
might just do the trick for your situation?

HTH
Stephan

PS: I was thinking of making a blog post about all this. Would you mind
if I use your DomConverterAdaptor in that post?
Re: IDE for a language embedded in Java [message #515388 is a reply to message #509051] Thu, 18 February 2010 16:08 Go to previous message
Jan Marc Hoffmann is currently offline Jan Marc HoffmannFriend
Messages: 32
Registered: July 2009
Member
Thanks for your hint. This pattern was what i was looking for. Sure, feel free to use and modify my example.

greetings
Jan Marc
Re: IDE for a language embedded in Java [message #568406 is a reply to message #514984] Wed, 17 February 2010 14:00 Go to previous message
Stephan Herrmann is currently offline Stephan HerrmannFriend
Messages: 1853
Registered: July 2009
Senior Member
Jan Marc Hoffmann wrote on Wed, 17 February 2010 07:22
> The DomCustomIntLiteral adapts all NumberLiterals, which it shouldn't. The DomCustomIntLiteral should only be played by the NumberLiterals which got created by the ASTConverter. Is there a way to fix this? I have been looking into Baseguards, but those would need an information in the Base to decide whether this Node is a regular NumberLiteral or a DomCustomIntLiteral.


That looks pretty successful so far, and yes, you're exactly on that track
I would recommend. Have you looked at our http://trac.objectteams.org/ot/wiki/OtPatterns/ObjectRegistr ation?
I haven't yet looked at the details in your code, but it seems that this pattern
might just do the trick for your situation?

HTH
Stephan

PS: I was thinking of making a blog post about all this. Would you mind
if I use your DomConverterAdaptor in that post?
Previous Topic:A question on design motivation for callout syntax
Next Topic:IDE for a language embedded in Java
Goto Forum:
  


Current Time: Fri Apr 19 03:43:50 GMT 2024

Powered by FUDForum. Page generated in 0.02811 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top