Eclipse Community Forums: TMF (Xtext)

Help

Home

Home » Modeling » TMF (Xtext) » Nested comments

Show: Today's Messages :: Show Polls :: Message Navigator

Nested comments [message #666623]

Thu, 21 April 2011 11:11

costa

Messages: 10
Registered: April 2011

Junior Member

How can I define in xtext nested multiline comments like /* /* */ */ ?

Report message to a moderator

Re: Nested comments [message #666740 is a reply to message #666623]

Fri, 22 April 2011 06:08

Alexander Nittka

Messages: 1193
Registered: July 2009

Senior Member

Hi,

not as terminal rule as they are basically regular expressions. You will have to use datatype or parser rules or write your own lexer.

Alex

Report message to a moderator

Re: Nested comments [message #666741 is a reply to message #666740]

Fri, 22 April 2011 06:22

costa

Messages: 10
Registered: April 2011

Junior Member

Thanks for your answer.

[Updated on: Fri, 22 April 2011 06:22]

Report message to a moderator

Re: Nested comments [message #666761 is a reply to message #666740]

Fri, 22 April 2011 09:28

Sebastian Zarnekow

Messages: 3118
Registered: July 2009

Senior Member

Hi Alex,

did you try something like:

ML_COMMENT: '/*' (.|ML_COMMENT)* '*/'?;

I'm pretty sure this snippet won't work but it should be perfectly ok to
call terminal rules from within terminal rules.

Regards,
Sebastian
--
Need professional support for Eclipse Modeling?
Go visit: http://xtext.itemis.com

Am 22.04.11 08:08, schrieb Alexander Nittka:
> Hi,
>
> not as terminal rule as they are basically regular expressions. You will
> have to use datatype or parser rules or write your own lexer.
>
> Alex

Report message to a moderator

Re: Nested comments [message #666764 is a reply to message #666761]

Fri, 22 April 2011 10:53

costa

Messages: 10
Registered: April 2011

Junior Member

Sebastian Zarnekow wrote on Fri, 22 April 2011 05:28

ML_COMMENT: '/*' (.|ML_COMMENT)* '*/'?;

I try:

warning(200): ../org.xtext.aps/src-gen/org/xtext/example/aps/parser/antlr/internal/InternalAp.g:93:42: Decision can match input such as "'*'<EOT>" using multiple alternatives: 1, 3
As a result, alternative(s) 3 were disabled for that input
warning(200): ../org.xtext.aps/src-gen/org/xtext/example/aps/parser/antlr/internal/InternalAp.g:93:42: Decision can match input such as "'/'<EOT>" using multiple alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
error(201): ../org.xtext.aps/src-gen/org/xtext/example/aps/parser/antlr/internal/InternalAp.g:93:42: The following alternatives can never be matched: 2

Report message to a moderator

Re: Nested comments [message #666767 is a reply to message #666764]

Fri, 22 April 2011 11:34

Sebastian Zarnekow

Messages: 3118
Registered: July 2009

Senior Member

That's why I said "I'm pretty sure this snippet won't work" ;-)

Should be pretty straight forward to figure out what's wrong if you use
AntlrWorks to debug the generated grammar.

Regards,
Sebastian
--
Need professional support for Eclipse Modeling?
Go visit: http://xtext.itemis.com

Am 22.04.11 12:53, schrieb costa:
> Sebastian Zarnekow wrote on Fri, 22 April 2011 05:28
>> ML_COMMENT: '/*' (.|ML_COMMENT)* '*/'?;
> I try:warning(200):
> ../org.xtext.aps/src-gen/org/xtext/example/aps/parser/antlr/ internal/InternalAp.g:93:42:
> Decision can match input such as "'*'<EOT>" using multiple alternatives:
> 1, 3
> As a result, alternative(s) 3 were disabled for that input
> warning(200):
> ../org.xtext.aps/src-gen/org/xtext/example/aps/parser/antlr/ internal/InternalAp.g:93:42:
> Decision can match input such as "'/'<EOT>" using multiple alternatives:
> 1, 2
> As a result, alternative(s) 2 were disabled for that input
> error(201):
> ../org.xtext.aps/src-gen/org/xtext/example/aps/parser/antlr/ internal/InternalAp.g:93:42:
> The following alternatives can never be matched: 2

Report message to a moderator

Re: Nested comments [message #666770 is a reply to message #666767]

Fri, 22 April 2011 12:16

costa

Messages: 10
Registered: April 2011

Junior Member

Sebastian Zarnekow wrote on Fri, 22 April 2011 07:34

That's why I said "I'm pretty sure this snippet won't work" Wink

Should be pretty straight forward to figure out what's wrong if you use
AntlrWorks to debug the generated grammar.

Regards,
Sebastian

Error 201 - the ML_COMMENT in (. | ML_COMMENT) can never be matched.

[Updated on: Fri, 22 April 2011 12:18]

Report message to a moderator

Re: Nested comments [message #666771 is a reply to message #666770]

Fri, 22 April 2011 12:22

Henrik Lindberg

Messages: 2509
Registered: July 2009

Senior Member

I would override the lexer, and lex the ML_COMMENT manually. No need for a
full external lexer, just a simple derived class that looks for ml comments
in front of the regular lexer. Don't use datatypes, then you loose all the
default processing for ml comments.

- henrik

costa <costa.eclipse@mail.ru> wrote:
> Sebastian Zarnekow wrote on Fri, 22 April 2011 07:34
>> That's why I said "I'm pretty sure this snippet won't work" ;)
>>> Should be pretty straight forward to figure out what's wrong if you use
>>> > AntlrWorks to debug the generated grammar.
>>> Regards,
>> Sebastian
> Error - the ML_COMMENT in (. | ML_COMMENT) can never be matched.

--
- henrik

Report message to a moderator

Re: Nested comments [message #666775 is a reply to message #666771]

Fri, 22 April 2011 12:45

costa

Messages: 10
Registered: April 2011

Junior Member

Henrik Lindberg wrote on Fri, 22 April 2011 08:22

Thank you. I should change a file InternalApLexer.java or can change some file in the src folder?

[Updated on: Fri, 22 April 2011 12:46]

Report message to a moderator

Re: Nested comments [message #666786 is a reply to message #666775]

Fri, 22 April 2011 14:42

Henrik Lindberg

Messages: 2509
Registered: July 2009

Senior Member

Create your own class derived from the generated lexer, and then bind
your lexer instead of the default. Then override the appropriate
method(s). Do *not* change the generated lexer (changes will be
overwritten next time you generate).

A suggestion is to create a very simple MyDSL grammar consisting of a
single statement and support for comments. It is easier to get a grip on
the inner workings of the lexer that way.

What you do is override the "mTokens" method.
Depending on the grammar and terminals, the lexer looks a bit different.
In one of my grammars, it starts like this:

public void mTokens() throws RecognitionException {
int alt25=164;
alt25 = dfa25.predict(input);
switch (alt25) {
case 1 :
{
mT21();
}
break;
// etc. etc. etc.

case 158 :
{
mRULE_ML_COMMENT();
}

Now, the mRULE_ML_COMMENT() is final, so it can not be overridden (that
is what we would have wanted to override), so instead we must do
something like:

public void mTokens() throws RecognitionException {
if(dfa.25.predict(input) == 158)
MY_mRULE_ML_COMMENT();
else
super.mTokens();
}

And in MY_mRULE_ML_COMMENT() you mimic the behavior or the original
rule, but modify it to count opening and closing nested ML comments.

This is a bit of a hack and you have to change the code if you make any
changes to any of the terminals (it may be a different number). I don't
recommend this approach for any extensive lexing work - you are then far
better of by writing a separate external lexer using Antlr (there is
support for this in Xtext).

Hope that helps.
Regards
- henrik

On 4/22/11 2:45 PM, costa wrote:
> Henrik Lindberg wrote on Fri, 22 April 2011 08:22
>> I would override the lexer, and lex the ML_COMMENT manually. No need
>> for a
>> full external lexer, just a simple derived class that looks for ml
>> comments
>> in front of the regular lexer. Don't use datatypes, then you loose all
>> the
>> default processing for ml comments.
>>
>> - henrik
> Thank you. I should change a file InternalApLexer.java or can change
> some file in src folder?

Report message to a moderator

Re: Nested comments [message #666794 is a reply to message #666786]

Fri, 22 April 2011 15:47

costa

Messages: 10
Registered: April 2011

Junior Member

Henrik Lindberg wrote on Fri, 22 April 2011 10:42

I don't recommend this approach for any extensive lexing work - you are then far better of by writing a separate external lexer using Antlr (there is support for this in Xtext).

Hope that helps.
Regards
- henrik

Thank you for details. But usage of external lexer means that I can't use xtext, isn't it?

Report message to a moderator

Re: Nested comments [message #666801 is a reply to message #666794]

Fri, 22 April 2011 16:18

Henrik Lindberg

Messages: 2509
Registered: July 2009

Senior Member

The alternative to just deriving class that overrides the generated
lexer is to use an external lexer. Such a lexer is supported by Xtext
(it is configured in the mwe workflow, and its use is automatically
configured). By using an external lexer you take over the responsability
of all the terminal rules. All you have to do is to write a "mylexer.g"
and then do the configuration. An additional chore is that you have to
keep this lexer in sync with any changes you make to the grammar
terminals. I recommend using an external lexer if you need to do more
advanced lexing as you can use all of antlr's predicates, have your own
java functions etc.

I use an external lexer in the Geppetto "pp" project (found at
Cloudsmith/Geppetto on Github) if you want a working example to look at.

- henrik

On 4/22/11 5:47 PM, costa wrote:
> Henrik Lindberg wrote on Fri, 22 April 2011 10:42
>> I don't recommend this approach for any extensive lexing work - you
>> are then far better of by writing a separate external lexer using
>> Antlr (there is support for this in Xtext).
>>
>> Hope that helps.
>> Regards
>> - henrik
> Thank you for details. But usage of external lexer means that I can't
> use xtext, isn't it?

Report message to a moderator

Re: Nested comments [message #666816 is a reply to message #666801]

Fri, 22 April 2011 18:49

costa

Messages: 10
Registered: April 2011

Junior Member

Henrik Lindberg wrote on Fri, 22 April 2011 12:18

Many thanks.

Report message to a moderator

Re: Nested comments [message #881309 is a reply to message #666816]

Mon, 04 June 2012 10:18

Cash Ko

Messages: 21
Registered: May 2012

Junior Member

... deleted

[Updated on: Wed, 06 June 2012 20:58]

Report message to a moderator

Re: Nested comments [message #882610 is a reply to message #881309]

Wed, 06 June 2012 20:57

Cash Ko

Messages: 21
Registered: May 2012

Junior Member

Here finally Wink

is the solution for a nested multiline comments like "(* .. (* .. *) .. *)" using terminal rules (it was really not so easy to get it working properly) ...

For "/* .. /* .. */ .. */" the solution should be similar ...


  // +++ Support for nested multiline comment blocks +++ //
  
  terminal fragment MLC_OTH      : !( '*' | '(' | ')' );
  terminal fragment MLC_OTH_RP   : !( '*' | '(' );
  terminal fragment MLC_LP_TOK   : '('+ ( '*' MLC_BODY | MLC_OTH_RP );
  terminal fragment MLC_X_TOK    : '*'+ ( MLC_OTH | MLC_LP_TOK );
  terminal fragment MLC_BODY     : ( MLC_OTH_RP | MLC_LP_TOK | MLC_X_TOK )* '*'+ ')';
  terminal ML_COMMENT            : '(*' MLC_BODY;

[Updated on: Tue, 19 June 2012 10:06]

Report message to a moderator

Re: Nested comments [message #1061492 is a reply to message #882610]

Sat, 01 June 2013 01:19

David Pace

Messages: 19
Registered: March 2013

Junior Member

Here is a version for /* .. /* .. */ .. */-Style Multiline comments:

terminal fragment MLC_ANY    : !( '*' | '/' );
terminal fragment MLC_SLASH  : '/'+ ( '*' MLC_BODY | MLC_ANY );
terminal fragment MLC_STAR   : '*'+ ( MLC_ANY );
terminal fragment MLC_BODY   : ( MLC_ANY | MLC_SLASH | MLC_STAR )* '*'+ '/';
terminal ML_COMMENT          : '/*' MLC_BODY;

Report message to a moderator

Previous Topic:	Bug in commons.types???
Next Topic:	implicit variable declaration in dsl editor with xbase

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Tue Apr 16 17:50:11 GMT 2024

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter