Eclipse Community Forums: TMF (Xtext) » Customized InternalMyDSLLexer

Help

Home

Home » Modeling » TMF (Xtext) » Customized InternalMyDSLLexer(how?)

Show: Today's Messages :: Show Polls :: Message Navigator

Customized InternalMyDSLLexer [message #735752]

Wed, 12 October 2011 16:14

Mathieu Garcia

Messages: 14
Registered: March 2011
Location: Tououse, France

Junior Member

Hi developers,

I need to add some validation stuff in the InternalMyDSLLexer code.
Because this class is regenerated each time the DSL is updated, I'd like to have a CustomMyDSLLexer that extends InternalMyDSLLexer class.

I found a way to declare this lexer in both modules, but the fact is I need to override some methods declared final in InternalMyDSLLexer class such as mRULE_T23().

My first idea was to override template that builds InternalMyDSLLexer by adding some hooks and get something like :

//InternalMyDSLLexer
public final void mRULE_T23() {
  try {
        {
        preRULE_T23();
        match('STEP'); 
        }
  }
  finally {
  }
}

public void preRULE_T23() {
    //to be overriden by sub class
}

//CustomMyDSLLexer extends InternalMyDSLLexer

public void preRULE_T23() {
    if (getCharPositionInLine() != 0) {
       throw new RecognitionException(...);
    }
}

I'm not able to locate the template responsible for this class generation.

Someone could help me?

Is there a better way to do that?

Thanks.
Mathieu

Report message to a moderator

Re: Customized InternalMyDSLLexer [message #735770 is a reply to message #735752]

Wed, 12 October 2011 16:50

Henrik Lindberg

Messages: 2509
Registered: July 2009

Senior Member

You could always use an external lexer (written using antlr). This is
supported by the Xtext workflow.

I used an override approach in Eclipse/b3 (similar to what you suggest),
but then learned about the external lexer, which I am now using (much
better). Example use in cloudsmith/geppetto at github.

Regards
- henrik

On 10/12/11 6:14 PM, Mathieu Garcia wrote:
> Hi developers,
>
> I need to add some validation stuff in the InternalMyDSLLexer code.
> Because this class is regenerated each time the DSL is updated, I'd like
> to have a CustomMyDSLLexer that extends InternalMyDSLLexer class.
>
> I found a way to declare this lexer in both modules, but the fact is I
> need to override some methods declared final in InternalMyDSLLexer class
> such as mRULE_T23().
>
> My first idea was to override template that builds InternalMyDSLLexer by
> adding some hooks and get something like :
>
>
> //InternalMyDSLLexer
> public final void mRULE_T23() {
> try {
> {
> preRULE_T23();
> match('STEP'); }
> }
> finally {
> }
> }
>
> public void preRULE_T23() {
> //to be overriden by sub class
> }
>
>
>
>
> //CustomMyDSLLexer extends InternalMyDSLLexer
>
> public void preRULE_T23() {
> if (getCharPositionInLine() != 0) {
> throw new RecognitionException(...);
> }
> }
>
>
>
> I'm not able to locate the template responsible for this class generation.
>
> Someone could help me?
>
> Is there a better way to do that?
> Thanks.
> Mathieu

Report message to a moderator

Re: Customized InternalMyDSLLexer [message #735806 is a reply to message #735770]

Wed, 12 October 2011 19:01

Mathieu Garcia

Messages: 14
Registered: March 2011
Location: Tououse, France

Junior Member

Yes, I found some threads about the external lexer approach and some leads to geppetto.

But, it's very difficult to understand how this works.

What is the purpose? How this external lexer is related to the internal lexer?

If you may describe the different steps, I would really appreciate.

Thank you.
Mathieu

Report message to a moderator

Re: Customized InternalMyDSLLexer [message #735934 is a reply to message #735806]

Thu, 13 October 2011 06:58

Daniel Missing name

Messages: 101
Registered: July 2011

Senior Member

Simply inherit your own lexer and put it in your src folder:

class CustomMyDslLexer extends InternalMyDslLexer { .. }

then bind it to the injection context by adding this method to your MyDslRuntimeModule.java and MyDslUiModule.java

public Class<? extends org.eclipse.xtext.parser.antlr.Lexer> bindLexer() {
	return at.package.mydsl.CustomMyDslLexer.class;
}

This way xtext will create an instance of your own lexer and use it.

Report message to a moderator

Re: Customized InternalMyDSLLexer [message #735947 is a reply to message #735934]

Thu, 13 October 2011 07:35

Mathieu Garcia

Messages: 14
Registered: March 2011
Location: Tououse, France

Junior Member

Yes Daniel,
As I wrote above, I did it, but some methods in InternalMyDSLLexer are final. So, this solution is not very convenient.

Using an external lexer seems to be the good solution but I don't know how implement it.

Mathieu

Report message to a moderator

Re: Customized InternalMyDSLLexer [message #736001 is a reply to message #735947]

Thu, 13 October 2011 10:16

Henrik Lindberg

Messages: 2509
Registered: July 2009

Senior Member

On 10/13/11 9:35 AM, Mathieu Garcia wrote:
> Yes Daniel,
> As I wrote above, I did it, but some methods in InternalMyDSLLexer are
> final. So, this solution is not very convenient.
>
> Using an external lexer seems to be the good solution but I don't know
> how implement it.
>
> Mathieu

Basically, you write the lexer.g file (in antlr syntax) and configure
the mwe workflow for your DSL to use the external lexer that is then
generated.

If you are looking at Geppetto - start in the runtime module:

// contributed by
org.eclipse.xtext.generator.parser.antlr.ex.rt.AntlrGeneratorFragment
public com.google.inject.Provider<PPOverridingLexer>
providePPOverridingLexer() {
return
org.eclipse.xtext.parser.antlr.LexerProvider.create(PPOverridingLexer.class);
}

Look at PPOverridingLexer - which is needed to keep track of last seen
token. It inherits from the generated PPLexer. If you don't need to do
something special like this, then use the generated lexer (i.e. in my
case PPLexer) directly.

The package org.cloudsmith.geppetto.pp.dsl.lexer contains the .g file
and some surrounding supporting stuff.

The mwe workflow contains this:
// Use externally specified lexer
fragment = parser.antlr.ex.ExternalAntlrLexerFragment {
lexerGrammar = "org.cloudsmith.geppetto.pp.dsl.lexer.PPLexer"
runtime = true
antlrParam = "-lib"
antlrParam =
"${runtimeProject}/src-gen/org/cloudsmith/geppetto/pp/dsl/parser/antlr/lexer"
}

Hope that helps.

If you don't know how the lexer works, I suggest you write a very small
grammar and look at what is generated.

The only thing I found to be a bit tricky is that the token values have
to be kept in sync. This is a bit messy if you are adding/removing
keywords or terminals in your grammar.

In short, you keep the terminals and keywords in your grammar. The
definitions of the terminals will never be used, but the generated token
values for your keywords and terminals are. The tokens are just a number
(and the text that was lexed for that token), so if the grammar thinks
"if" is 34 and your lexer delivers 35 when seeing "if" you get very
strange results.

It should be possible to figure out how it works in general from looking
at the Geppetto logic - it is not very complicated, but if you really
try to dig into the string expression interpolation logic you need to
know that double quoted strings can contain ${expr}, $varName or
${varName}, and that it is possible to escape these constructs with \.

You can also look at mwe2 - it is also using an external lexer, and it
is where I picked up how to do some of this.

- henrik

Report message to a moderator