Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Nested comments
Nested comments [message #666623] Thu, 21 April 2011 07:11 Go to next message
costa is currently offline costa
Messages: 10
Registered: April 2011
Junior Member
How can I define in xtext nested multiline comments like /* /* */ */ ?
Re: Nested comments [message #666740 is a reply to message #666623] Fri, 22 April 2011 02:08 Go to previous messageGo to next message
Alexander Nittka is currently offline Alexander Nittka
Messages: 1151
Registered: July 2009
Senior Member
Hi,

not as terminal rule as they are basically regular expressions. You will have to use datatype or parser rules or write your own lexer.

Alex
Re: Nested comments [message #666741 is a reply to message #666740] Fri, 22 April 2011 02:22 Go to previous messageGo to next message
costa is currently offline costa
Messages: 10
Registered: April 2011
Junior Member
Thanks for your answer.

[Updated on: Fri, 22 April 2011 02:22]

Report message to a moderator

Re: Nested comments [message #666761 is a reply to message #666740] Fri, 22 April 2011 05:28 Go to previous messageGo to next message
Sebastian Zarnekow is currently offline Sebastian Zarnekow
Messages: 2809
Registered: July 2009
Senior Member
Hi Alex,

did you try something like:

ML_COMMENT: '/*' (.|ML_COMMENT)* '*/'?;

I'm pretty sure this snippet won't work but it should be perfectly ok to
call terminal rules from within terminal rules.

Regards,
Sebastian
--
Need professional support for Eclipse Modeling?
Go visit: http://xtext.itemis.com

Am 22.04.11 08:08, schrieb Alexander Nittka:
> Hi,
>
> not as terminal rule as they are basically regular expressions. You will
> have to use datatype or parser rules or write your own lexer.
>
> Alex
Re: Nested comments [message #666764 is a reply to message #666761] Fri, 22 April 2011 06:53 Go to previous messageGo to next message
costa is currently offline costa
Messages: 10
Registered: April 2011
Junior Member
Sebastian Zarnekow wrote on Fri, 22 April 2011 05:28
ML_COMMENT: '/*' (.|ML_COMMENT)* '*/'?;
I try:
warning(200): ../org.xtext.aps/src-gen/org/xtext/example/aps/parser/antlr/internal/InternalAp.g:93:42: Decision can match input such as "'*'<EOT>" using multiple alternatives: 1, 3
As a result, alternative(s) 3 were disabled for that input
warning(200): ../org.xtext.aps/src-gen/org/xtext/example/aps/parser/antlr/internal/InternalAp.g:93:42: Decision can match input such as "'/'<EOT>" using multiple alternatives: 1, 2
As a result, alternative(s) 2 were disabled for that input
error(201): ../org.xtext.aps/src-gen/org/xtext/example/aps/parser/antlr/internal/InternalAp.g:93:42: The following alternatives can never be matched: 2
Re: Nested comments [message #666767 is a reply to message #666764] Fri, 22 April 2011 07:34 Go to previous messageGo to next message
Sebastian Zarnekow is currently offline Sebastian Zarnekow
Messages: 2809
Registered: July 2009
Senior Member
That's why I said "I'm pretty sure this snippet won't work" ;-)

Should be pretty straight forward to figure out what's wrong if you use
AntlrWorks to debug the generated grammar.

Regards,
Sebastian
--
Need professional support for Eclipse Modeling?
Go visit: http://xtext.itemis.com

Am 22.04.11 12:53, schrieb costa:
> Sebastian Zarnekow wrote on Fri, 22 April 2011 05:28
>> ML_COMMENT: '/*' (.|ML_COMMENT)* '*/'?;
> I try:warning(200):
> ../org.xtext.aps/src-gen/org/xtext/example/aps/parser/antlr/ internal/InternalAp.g:93:42:
> Decision can match input such as "'*'<EOT>" using multiple alternatives:
> 1, 3
> As a result, alternative(s) 3 were disabled for that input
> warning(200):
> ../org.xtext.aps/src-gen/org/xtext/example/aps/parser/antlr/ internal/InternalAp.g:93:42:
> Decision can match input such as "'/'<EOT>" using multiple alternatives:
> 1, 2
> As a result, alternative(s) 2 were disabled for that input
> error(201):
> ../org.xtext.aps/src-gen/org/xtext/example/aps/parser/antlr/ internal/InternalAp.g:93:42:
> The following alternatives can never be matched: 2
Re: Nested comments [message #666770 is a reply to message #666767] Fri, 22 April 2011 08:16 Go to previous messageGo to next message
costa is currently offline costa
Messages: 10
Registered: April 2011
Junior Member
Sebastian Zarnekow wrote on Fri, 22 April 2011 07:34
That's why I said "I'm pretty sure this snippet won't work" Wink

Should be pretty straight forward to figure out what's wrong if you use
AntlrWorks to debug the generated grammar.

Regards,
Sebastian
Error 201 - the ML_COMMENT in (. | ML_COMMENT) can never be matched.

[Updated on: Fri, 22 April 2011 08:18]

Report message to a moderator

Re: Nested comments [message #666771 is a reply to message #666770] Fri, 22 April 2011 08:22 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik Lindberg
Messages: 2499
Registered: July 2009
Senior Member
I would override the lexer, and lex the ML_COMMENT manually. No need for a
full external lexer, just a simple derived class that looks for ml comments
in front of the regular lexer. Don't use datatypes, then you loose all the
default processing for ml comments.

- henrik

costa <costa.eclipse@mail.ru> wrote:
> Sebastian Zarnekow wrote on Fri, 22 April 2011 07:34
>> That's why I said "I'm pretty sure this snippet won't work" ;)
>>> Should be pretty straight forward to figure out what's wrong if you use
>>> > AntlrWorks to debug the generated grammar.
>>> Regards,
>> Sebastian
> Error - the ML_COMMENT in (. | ML_COMMENT) can never be matched.


--
- henrik
Re: Nested comments [message #666775 is a reply to message #666771] Fri, 22 April 2011 08:45 Go to previous messageGo to next message
costa is currently offline costa
Messages: 10
Registered: April 2011
Junior Member
Henrik Lindberg wrote on Fri, 22 April 2011 08:22
I would override the lexer, and lex the ML_COMMENT manually. No need for a
full external lexer, just a simple derived class that looks for ml comments
in front of the regular lexer. Don't use datatypes, then you loose all the
default processing for ml comments.

- henrik
Thank you. I should change a file InternalApLexer.java or can change some file in the src folder?

[Updated on: Fri, 22 April 2011 08:46]

Report message to a moderator

Re: Nested comments [message #666786 is a reply to message #666775] Fri, 22 April 2011 10:42 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik Lindberg
Messages: 2499
Registered: July 2009
Senior Member
Create your own class derived from the generated lexer, and then bind
your lexer instead of the default. Then override the appropriate
method(s). Do *not* change the generated lexer (changes will be
overwritten next time you generate).

A suggestion is to create a very simple MyDSL grammar consisting of a
single statement and support for comments. It is easier to get a grip on
the inner workings of the lexer that way.

What you do is override the "mTokens" method.
Depending on the grammar and terminals, the lexer looks a bit different.
In one of my grammars, it starts like this:

public void mTokens() throws RecognitionException {
int alt25=164;
alt25 = dfa25.predict(input);
switch (alt25) {
case 1 :
{
mT21();
}
break;
// etc. etc. etc.

case 158 :
{
mRULE_ML_COMMENT();
}

Now, the mRULE_ML_COMMENT() is final, so it can not be overridden (that
is what we would have wanted to override), so instead we must do
something like:

public void mTokens() throws RecognitionException {
if(dfa.25.predict(input) == 158)
MY_mRULE_ML_COMMENT();
else
super.mTokens();
}

And in MY_mRULE_ML_COMMENT() you mimic the behavior or the original
rule, but modify it to count opening and closing nested ML comments.

This is a bit of a hack and you have to change the code if you make any
changes to any of the terminals (it may be a different number). I don't
recommend this approach for any extensive lexing work - you are then far
better of by writing a separate external lexer using Antlr (there is
support for this in Xtext).

Hope that helps.
Regards
- henrik

On 4/22/11 2:45 PM, costa wrote:
> Henrik Lindberg wrote on Fri, 22 April 2011 08:22
>> I would override the lexer, and lex the ML_COMMENT manually. No need
>> for a
>> full external lexer, just a simple derived class that looks for ml
>> comments
>> in front of the regular lexer. Don't use datatypes, then you loose all
>> the
>> default processing for ml comments.
>>
>> - henrik
> Thank you. I should change a file InternalApLexer.java or can change
> some file in src folder?
Re: Nested comments [message #666794 is a reply to message #666786] Fri, 22 April 2011 11:47 Go to previous messageGo to next message
costa is currently offline costa
Messages: 10
Registered: April 2011
Junior Member
Henrik Lindberg wrote on Fri, 22 April 2011 10:42
I don't recommend this approach for any extensive lexing work - you are then far better of by writing a separate external lexer using Antlr (there is support for this in Xtext).

Hope that helps.
Regards
- henrik
Thank you for details. But usage of external lexer means that I can't use xtext, isn't it?
Re: Nested comments [message #666801 is a reply to message #666794] Fri, 22 April 2011 12:18 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik Lindberg
Messages: 2499
Registered: July 2009
Senior Member
The alternative to just deriving class that overrides the generated
lexer is to use an external lexer. Such a lexer is supported by Xtext
(it is configured in the mwe workflow, and its use is automatically
configured). By using an external lexer you take over the responsability
of all the terminal rules. All you have to do is to write a "mylexer.g"
and then do the configuration. An additional chore is that you have to
keep this lexer in sync with any changes you make to the grammar
terminals. I recommend using an external lexer if you need to do more
advanced lexing as you can use all of antlr's predicates, have your own
java functions etc.

I use an external lexer in the Geppetto "pp" project (found at
Cloudsmith/Geppetto on Github) if you want a working example to look at.

- henrik

On 4/22/11 5:47 PM, costa wrote:
> Henrik Lindberg wrote on Fri, 22 April 2011 10:42
>> I don't recommend this approach for any extensive lexing work - you
>> are then far better of by writing a separate external lexer using
>> Antlr (there is support for this in Xtext).
>>
>> Hope that helps.
>> Regards
>> - henrik
> Thank you for details. But usage of external lexer means that I can't
> use xtext, isn't it?
Re: Nested comments [message #666816 is a reply to message #666801] Fri, 22 April 2011 14:49 Go to previous messageGo to next message
costa is currently offline costa
Messages: 10
Registered: April 2011
Junior Member
Henrik Lindberg wrote on Fri, 22 April 2011 12:18
The alternative to just deriving class that overrides the generated
lexer is to use an external lexer. Such a lexer is supported by Xtext
(it is configured in the mwe workflow, and its use is automatically
configured). By using an external lexer you take over the responsability
of all the terminal rules. All you have to do is to write a "mylexer.g"
and then do the configuration. An additional chore is that you have to
keep this lexer in sync with any changes you make to the grammar
terminals. I recommend using an external lexer if you need to do more
advanced lexing as you can use all of antlr's predicates, have your own
java functions etc.

I use an external lexer in the Geppetto "pp" project (found at
Cloudsmith/Geppetto on Github) if you want a working example to look at.

- henrik
Many thanks.
Re: Nested comments [message #881309 is a reply to message #666816] Mon, 04 June 2012 06:18 Go to previous messageGo to next message
Cash Ko is currently offline Cash Ko
Messages: 21
Registered: May 2012
Junior Member
... deleted

[Updated on: Wed, 06 June 2012 16:58]

Report message to a moderator

Re: Nested comments [message #882610 is a reply to message #881309] Wed, 06 June 2012 16:57 Go to previous messageGo to next message
Cash Ko is currently offline Cash Ko
Messages: 21
Registered: May 2012
Junior Member
Here finally Wink is the solution for a nested multiline comments like "(* .. (* .. *) .. *)" using terminal rules (it was really not so easy to get it working properly) ...

For "/* .. /* .. */ .. */" the solution should be similar ...

  // +++ Support for nested multiline comment blocks +++ //
  
  terminal fragment MLC_OTH      : !( '*' | '(' | ')' );
  terminal fragment MLC_OTH_RP   : !( '*' | '(' );
  terminal fragment MLC_LP_TOK   : '('+ ( '*' MLC_BODY | MLC_OTH_RP );
  terminal fragment MLC_X_TOK    : '*'+ ( MLC_OTH | MLC_LP_TOK );
  terminal fragment MLC_BODY     : ( MLC_OTH_RP | MLC_LP_TOK | MLC_X_TOK )* '*'+ ')';
  terminal ML_COMMENT            : '(*' MLC_BODY;

[Updated on: Tue, 19 June 2012 06:06]

Report message to a moderator

Re: Nested comments [message #1061492 is a reply to message #882610] Fri, 31 May 2013 21:19 Go to previous message
David Hofmann is currently offline David Hofmann
Messages: 14
Registered: March 2013
Junior Member
Here is a version for /* .. /* .. */ .. */-Style Multiline comments:

terminal fragment MLC_ANY    : !( '*' | '/' );
terminal fragment MLC_SLASH  : '/'+ ( '*' MLC_BODY | MLC_ANY );
terminal fragment MLC_STAR   : '*'+ ( MLC_ANY );
terminal fragment MLC_BODY   : ( MLC_ANY | MLC_SLASH | MLC_STAR )* '*'+ '/';
terminal ML_COMMENT          : '/*' MLC_BODY;
Previous Topic:Bug in commons.types???
Next Topic:implicit variable declaration in dsl editor with xbase
Goto Forum:
  


Current Time: Wed Aug 20 22:22:59 EDT 2014

Powered by FUDForum. Page generated in 0.09766 seconds