Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Archived » IMP » How to get LPG to treat comments as terminals
How to get LPG to treat comments as terminals [message #25538] Wed, 19 November 2008 20:02 Go to next message
Eclipse User
Originally posted by: aker1121.yahoo.com

By looking into how JDT handles JavaDoc I have
started to understand that the parser must produce
terminals for comments. In the DSL we have created
an IDE for comments can occur 'everyware' (like it is
in java, c,c++ etc), so the way forward can not be
to add comments token in the lexical rule file.

Can someone please give me an hint here?
I can see a solution where occurences of whitespace also
could mean a comment, but how do I do that?

I'm fairly new to LPG and DSL so please have oversight
in my lack of understanding for this.


regards,
Aker Br
Re: How to get LPG to treat comments as terminals [message #25654 is a reply to message #25538] Wed, 19 November 2008 21:56 Go to previous messageGo to next message
Stan Sutton is currently offline Stan Sutton
Messages: 121
Registered: July 2009
Senior Member
Hello Acker,

I am not sure that I understand what you are trying to do.

I could try to guess, but I think I would be wrong.

Can you please clarify what you already have and what you are trying to
achieve?

Thanks,

Stan



Aker wrote:
> By looking into how JDT handles JavaDoc I have
> started to understand that the parser must produce
> terminals for comments. In the DSL we have created
> an IDE for comments can occur 'everyware' (like it is
> in java, c,c++ etc), so the way forward can not be
> to add comments token in the lexical rule file.
>
> Can someone please give me an hint here?
> I can see a solution where occurences of whitespace also
> could mean a comment, but how do I do that?
>
> I'm fairly new to LPG and DSL so please have oversight
> in my lack of understanding for this.
>
>
> regards,
> Aker Brückler
>
>
Re: How to get LPG to treat comments as terminals [message #25730 is a reply to message #25654] Thu, 20 November 2008 16:54 Go to previous messageGo to next message
Eclipse User
Originally posted by: aker1121.yahoo.com

Hi,

Trying to clarify myself:

We have implemented the LPG rules (lexer.gi) so that all the comments are
skipped (using skipToken).
So naturally the comments will not end up as a production in the AST. But I
would like to be able to
use comments the same way that JDT uses JavaDoc for method declaration. So
when I hover over a
method invocation I would like to see the comments that was defined prior to
the method declaration part.

I started to use the ASTView provided by the JDT team
(http://www.eclipse.org/jdt/ui/astview/index.php)
and I noticed that the AST for my file contains also all the comments.

So to make it short; I would like to be able to get a production in the AST
for the comments that (might) exist prior to
the method declaration in our DSL. Or if that is easier all the comments in
the file as productions in the AST.


Regards,
Aker



"Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
news:gg221k$a4q$1@build.eclipse.org...
> Hello Acker,
>
> I am not sure that I understand what you are trying to do.
>
> I could try to guess, but I think I would be wrong.
>
> Can you please clarify what you already have and what you are trying to
> achieve?
>
> Thanks,
>
> Stan
>
>
>
> Aker wrote:
>> By looking into how JDT handles JavaDoc I have
>> started to understand that the parser must produce
>> terminals for comments. In the DSL we have created
>> an IDE for comments can occur 'everyware' (like it is
>> in java, c,c++ etc), so the way forward can not be
>> to add comments token in the lexical rule file.
>>
>> Can someone please give me an hint here?
>> I can see a solution where occurences of whitespace also
>> could mean a comment, but how do I do that?
>>
>> I'm fairly new to LPG and DSL so please have oversight
>> in my lack of understanding for this.
>>
>>
>> regards,
>> Aker Br
Re: How to get LPG to treat comments as terminals [message #25765 is a reply to message #25730] Fri, 21 November 2008 20:47 Go to previous messageGo to next message
Stan Sutton is currently offline Stan Sutton
Messages: 121
Registered: July 2009
Senior Member
Hi Aker,

That's clearer. I am not an expert on the workings of LPG, but I can
tell you a few things that might help.

The LPG representation of the parse stream contains both tokens and
adjuncts, where adjuncts represent comments. Typically, as a file is
parsed by an LPG parser, tokens go into a tokens array, comments go into
an adjuncts array, and a mapping is maintained between them. The
interface lpg.runtime.IParseStream contains methods
IToken[] getFollowingAdjuncts(int i);
IToken[] getPrecedingAdjuncts(int i);
where the argument is the index of a token in the token stream.

Of course, in order to get the comments into the adjuncts array in the
parse stream, you have to specify the grammar or lexer rules
appropriately. In our typical examples, we have rules for single line
comments; this is from a generated Lexer.gi file:

%Rules
Token ::= identifier /. checkForKeyWord();./
| number /. makeToken($_NUMBER);./
| DoubleLiteral /. makeToken($_DoubleLiteral);./
| white /. skipToken();./
| slc /. makeComment($_SINGLE_LINE_COMMENT);./
| ';' /. makeToken($_SEMICOLON);./
...

Here we do not skipToken for the comment but instead use
makeComment(..). makeComment(..) is the method that adds the comment
token to the adjuncts array.

If you are recognizing comments in your grammar rules (.g file) then I
think you should be able to invoke makeComment(..) from there as well.

Does this seem like it can support what you want to do?

Regards,

Stan



Aker wrote:
> Hi,
>
> Trying to clarify myself:
>
> We have implemented the LPG rules (lexer.gi) so that all the comments are
> skipped (using skipToken).
> So naturally the comments will not end up as a production in the AST. But I
> would like to be able to
> use comments the same way that JDT uses JavaDoc for method declaration. So
> when I hover over a
> method invocation I would like to see the comments that was defined prior to
> the method declaration part.
>
> I started to use the ASTView provided by the JDT team
> (http://www.eclipse.org/jdt/ui/astview/index.php)
> and I noticed that the AST for my file contains also all the comments.
>
> So to make it short; I would like to be able to get a production in the AST
> for the comments that (might) exist prior to
> the method declaration in our DSL. Or if that is easier all the comments in
> the file as productions in the AST.
>
>
> Regards,
> Aker
>
>
>
> "Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
> news:gg221k$a4q$1@build.eclipse.org...
>> Hello Acker,
>>
>> I am not sure that I understand what you are trying to do.
>>
>> I could try to guess, but I think I would be wrong.
>>
>> Can you please clarify what you already have and what you are trying to
>> achieve?
>>
>> Thanks,
>>
>> Stan
>>
>>
>>
>> Aker wrote:
>>> By looking into how JDT handles JavaDoc I have
>>> started to understand that the parser must produce
>>> terminals for comments. In the DSL we have created
>>> an IDE for comments can occur 'everyware' (like it is
>>> in java, c,c++ etc), so the way forward can not be
>>> to add comments token in the lexical rule file.
>>>
>>> Can someone please give me an hint here?
>>> I can see a solution where occurences of whitespace also
>>> could mean a comment, but how do I do that?
>>>
>>> I'm fairly new to LPG and DSL so please have oversight
>>> in my lack of understanding for this.
>>>
>>>
>>> regards,
>>> Aker Brückler
>>>
>
Re: How to get LPG to treat comments as terminals [message #25776 is a reply to message #25765] Sat, 22 November 2008 10:15 Go to previous messageGo to next message
Eclipse User
Originally posted by: aker1121.yahoo.com

Hi Stan,

Not only does it support what I wanted, but it is exactly what I wanted!
Thanks for valuable help. Much appreciated!!

I missed that makeComment ('macro') exists in a sample definition file, but
is there some LPG document (or web page) that covers this (and other
macros)?


BR

Aker



"Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
news:gg76p7$g5k$1@build.eclipse.org...
> Hi Aker,
>
> That's clearer. I am not an expert on the workings of LPG, but I can tell
> you a few things that might help.
>
> The LPG representation of the parse stream contains both tokens and
> adjuncts, where adjuncts represent comments. Typically, as a file is
> parsed by an LPG parser, tokens go into a tokens array, comments go into
> an adjuncts array, and a mapping is maintained between them. The
> interface lpg.runtime.IParseStream contains methods
> IToken[] getFollowingAdjuncts(int i);
> IToken[] getPrecedingAdjuncts(int i);
> where the argument is the index of a token in the token stream.
>
> Of course, in order to get the comments into the adjuncts array in the
> parse stream, you have to specify the grammar or lexer rules
> appropriately. In our typical examples, we have rules for single line
> comments; this is from a generated Lexer.gi file:
>
> %Rules
> Token ::= identifier /. checkForKeyWord();./
> | number /. makeToken($_NUMBER);./
> | DoubleLiteral /. makeToken($_DoubleLiteral);./
> | white /. skipToken();./
> | slc /. makeComment($_SINGLE_LINE_COMMENT);./ |
> ';' /. makeToken($_SEMICOLON);./
> ...
>
> Here we do not skipToken for the comment but instead use makeComment(..).
> makeComment(..) is the method that adds the comment token to the adjuncts
> array.
>
> If you are recognizing comments in your grammar rules (.g file) then I
> think you should be able to invoke makeComment(..) from there as well.
>
> Does this seem like it can support what you want to do?
>
> Regards,
>
> Stan
>
>
>
> Aker wrote:
>> Hi,
>>
>> Trying to clarify myself:
>>
>> We have implemented the LPG rules (lexer.gi) so that all the comments are
>> skipped (using skipToken).
>> So naturally the comments will not end up as a production in the AST. But
>> I would like to be able to
>> use comments the same way that JDT uses JavaDoc for method declaration.
>> So when I hover over a
>> method invocation I would like to see the comments that was defined prior
>> to the method declaration part.
>>
>> I started to use the ASTView provided by the JDT team
>> (http://www.eclipse.org/jdt/ui/astview/index.php)
>> and I noticed that the AST for my file contains also all the comments.
>>
>> So to make it short; I would like to be able to get a production in the
>> AST for the comments that (might) exist prior to
>> the method declaration in our DSL. Or if that is easier all the comments
>> in the file as productions in the AST.
>>
>>
>> Regards,
>> Aker
>>
>>
>>
>> "Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
>> news:gg221k$a4q$1@build.eclipse.org...
>>> Hello Acker,
>>>
>>> I am not sure that I understand what you are trying to do.
>>>
>>> I could try to guess, but I think I would be wrong.
>>>
>>> Can you please clarify what you already have and what you are trying to
>>> achieve?
>>>
>>> Thanks,
>>>
>>> Stan
>>>
>>>
>>>
>>> Aker wrote:
>>>> By looking into how JDT handles JavaDoc I have
>>>> started to understand that the parser must produce
>>>> terminals for comments. In the DSL we have created
>>>> an IDE for comments can occur 'everyware' (like it is
>>>> in java, c,c++ etc), so the way forward can not be
>>>> to add comments token in the lexical rule file.
>>>>
>>>> Can someone please give me an hint here?
>>>> I can see a solution where occurences of whitespace also
>>>> could mean a comment, but how do I do that?
>>>>
>>>> I'm fairly new to LPG and DSL so please have oversight
>>>> in my lack of understanding for this.
>>>>
>>>>
>>>> regards,
>>>> Aker Br
Re: How to get LPG to treat comments as terminals [message #25802 is a reply to message #25776] Mon, 24 November 2008 19:49 Go to previous messageGo to next message
Stan Sutton is currently offline Stan Sutton
Messages: 121
Registered: July 2009
Senior Member
Hi Aker,

Unfortunately, the macros were developed for the templates after the LPG
documentation was written.

It isn't hard to write your own macros, though. At the end of the
lexer.gi file you can introduce a "headers" section, e.g.,

%Headers
/.
private void sayHello() {
System.out.println("Hello world!");
}
./
%end

(There is a larger and more useful example at the end of the parser.g
file.) In the "headers" section you can write code for any "macros"
(classes, methods, fields) that you want to use with your rules. Then
you can refer to these in the code that is associated with the rules:


%Rules
Token ::= identifier /. checkForKeyWord();./
...
| slc /. makeComment($_SINGLE_LINE_COMMENT);
sayHello();
./
...

Also feel free to use any of the macros that are used in the examples in
any of the templates.

Regards,

Stan

Aker wrote:
> Hi Stan,
>
> Not only does it support what I wanted, but it is exactly what I wanted!
> Thanks for valuable help. Much appreciated!!
>
> I missed that makeComment ('macro') exists in a sample definition file, but
> is there some LPG document (or web page) that covers this (and other
> macros)?
>
>
> BR
>
> Aker
>
>
>
> "Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
> news:gg76p7$g5k$1@build.eclipse.org...
>> Hi Aker,
>>
>> That's clearer. I am not an expert on the workings of LPG, but I can tell
>> you a few things that might help.
>>
>> The LPG representation of the parse stream contains both tokens and
>> adjuncts, where adjuncts represent comments. Typically, as a file is
>> parsed by an LPG parser, tokens go into a tokens array, comments go into
>> an adjuncts array, and a mapping is maintained between them. The
>> interface lpg.runtime.IParseStream contains methods
>> IToken[] getFollowingAdjuncts(int i);
>> IToken[] getPrecedingAdjuncts(int i);
>> where the argument is the index of a token in the token stream.
>>
>> Of course, in order to get the comments into the adjuncts array in the
>> parse stream, you have to specify the grammar or lexer rules
>> appropriately. In our typical examples, we have rules for single line
>> comments; this is from a generated Lexer.gi file:
>>
>> %Rules
>> Token ::= identifier /. checkForKeyWord();./
>> | number /. makeToken($_NUMBER);./
>> | DoubleLiteral /. makeToken($_DoubleLiteral);./
>> | white /. skipToken();./
>> | slc /. makeComment($_SINGLE_LINE_COMMENT);./ |
>> ';' /. makeToken($_SEMICOLON);./
>> ...
>>
>> Here we do not skipToken for the comment but instead use makeComment(..).
>> makeComment(..) is the method that adds the comment token to the adjuncts
>> array.
>>
>> If you are recognizing comments in your grammar rules (.g file) then I
>> think you should be able to invoke makeComment(..) from there as well.
>>
>> Does this seem like it can support what you want to do?
>>
>> Regards,
>>
>> Stan
>>
>>
>>
>> Aker wrote:
>>> Hi,
>>>
>>> Trying to clarify myself:
>>>
>>> We have implemented the LPG rules (lexer.gi) so that all the comments are
>>> skipped (using skipToken).
>>> So naturally the comments will not end up as a production in the AST. But
>>> I would like to be able to
>>> use comments the same way that JDT uses JavaDoc for method declaration.
>>> So when I hover over a
>>> method invocation I would like to see the comments that was defined prior
>>> to the method declaration part.
>>>
>>> I started to use the ASTView provided by the JDT team
>>> (http://www.eclipse.org/jdt/ui/astview/index.php)
>>> and I noticed that the AST for my file contains also all the comments.
>>>
>>> So to make it short; I would like to be able to get a production in the
>>> AST for the comments that (might) exist prior to
>>> the method declaration in our DSL. Or if that is easier all the comments
>>> in the file as productions in the AST.
>>>
>>>
>>> Regards,
>>> Aker
>>>
>>>
>>>
>>> "Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
>>> news:gg221k$a4q$1@build.eclipse.org...
>>>> Hello Acker,
>>>>
>>>> I am not sure that I understand what you are trying to do.
>>>>
>>>> I could try to guess, but I think I would be wrong.
>>>>
>>>> Can you please clarify what you already have and what you are trying to
>>>> achieve?
>>>>
>>>> Thanks,
>>>>
>>>> Stan
>>>>
>>>>
>>>>
>>>> Aker wrote:
>>>>> By looking into how JDT handles JavaDoc I have
>>>>> started to understand that the parser must produce
>>>>> terminals for comments. In the DSL we have created
>>>>> an IDE for comments can occur 'everyware' (like it is
>>>>> in java, c,c++ etc), so the way forward can not be
>>>>> to add comments token in the lexical rule file.
>>>>>
>>>>> Can someone please give me an hint here?
>>>>> I can see a solution where occurences of whitespace also
>>>>> could mean a comment, but how do I do that?
>>>>>
>>>>> I'm fairly new to LPG and DSL so please have oversight
>>>>> in my lack of understanding for this.
>>>>>
>>>>>
>>>>> regards,
>>>>> Aker Brückler
>>>>>
>
>
Re: How to get LPG to treat comments as terminals [message #25841 is a reply to message #25730] Tue, 25 November 2008 06:16 Go to previous messageGo to next message
Ed Willink is currently offline Ed Willink
Messages: 4086
Registered: July 2009
Senior Member
Hi Aker

> So naturally the comments will not end up as a production in the AST. But I
> would like to be able to
> use comments the same way that JDT uses JavaDoc for method declaration.

I've been thinking about this from a different requirement: refactoring
and a reverse engineeering update.

In the AST it is easy to identify all semantically equivalent re-uses of
the same name, so the AST can be changed, but that must be reverse
engineered into the source text again, which requires preservation of
the comments.

I think it's a mistake to make comments first class terminals, since
then every production must have COMMENTopt between every terminal.

Instead it may be better to use an enhanced Token to which a
partial-line single-line-comment is attached as a suffix and to which
other paragraph and full-line single-line-comments are attached as
prefixes. The Tokens then parse normally but persist the comments for
those activities that require them.

Regards

Ed Willink
Re: How to get LPG to treat comments as terminals [message #26892 is a reply to message #25841] Mon, 02 March 2009 06:57 Go to previous message
Ed Willink is currently offline Ed Willink
Messages: 4086
Registered: July 2009
Senior Member
Ed Willink wrote:

> Instead it may be better to use an enhanced Token to which a
> partial-line single-line-comment is attached as a suffix and to which
> other paragraph and full-line single-line-comments are attached as
> prefixes. The Tokens then parse normally but persist the comments for
> those activities that require them.

Actually there's no need to do anything. Leave comments out of the grammar,
they can always be recovered later by a special purpose comment, newline,
whitespace analyzer.

The comments for 'currentToken' are those between 'previousToken'.end
and 'currentToken'.start and/or between 'currentToken'.end
and 'nextToken'.start. Working in this way will allow you to use
syntactic context to decide how much of A,B,C,D,E in

int a; // A
// B
/* C */
/**
* D
*/
// E
int e;

belongs to int a, and how much to int e; It also gives
you more scope for inserting tabs for refactoring and other issues.

Regards

Ed Willink
Re: How to get LPG to treat comments as terminals [message #574325 is a reply to message #25538] Wed, 19 November 2008 21:56 Go to previous message
Stan Sutton is currently offline Stan Sutton
Messages: 121
Registered: July 2009
Senior Member
Hello Acker,

I am not sure that I understand what you are trying to do.

I could try to guess, but I think I would be wrong.

Can you please clarify what you already have and what you are trying to
achieve?

Thanks,

Stan



Aker wrote:
> By looking into how JDT handles JavaDoc I have
> started to understand that the parser must produce
> terminals for comments. In the DSL we have created
> an IDE for comments can occur 'everyware' (like it is
> in java, c,c++ etc), so the way forward can not be
> to add comments token in the lexical rule file.
>
> Can someone please give me an hint here?
> I can see a solution where occurences of whitespace also
> could mean a comment, but how do I do that?
>
> I'm fairly new to LPG and DSL so please have oversight
> in my lack of understanding for this.
>
>
> regards,
> Aker Brückler
>
>
Re: How to get LPG to treat comments as terminals [message #574407 is a reply to message #25654] Thu, 20 November 2008 16:54 Go to previous message
Eclipse User
Originally posted by: aker1121.yahoo.com

Hi,

Trying to clarify myself:

We have implemented the LPG rules (lexer.gi) so that all the comments are
skipped (using skipToken).
So naturally the comments will not end up as a production in the AST. But I
would like to be able to
use comments the same way that JDT uses JavaDoc for method declaration. So
when I hover over a
method invocation I would like to see the comments that was defined prior to
the method declaration part.

I started to use the ASTView provided by the JDT team
(http://www.eclipse.org/jdt/ui/astview/index.php)
and I noticed that the AST for my file contains also all the comments.

So to make it short; I would like to be able to get a production in the AST
for the comments that (might) exist prior to
the method declaration in our DSL. Or if that is easier all the comments in
the file as productions in the AST.


Regards,
Aker



"Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
news:gg221k$a4q$1@build.eclipse.org...
> Hello Acker,
>
> I am not sure that I understand what you are trying to do.
>
> I could try to guess, but I think I would be wrong.
>
> Can you please clarify what you already have and what you are trying to
> achieve?
>
> Thanks,
>
> Stan
>
>
>
> Aker wrote:
>> By looking into how JDT handles JavaDoc I have
>> started to understand that the parser must produce
>> terminals for comments. In the DSL we have created
>> an IDE for comments can occur 'everyware' (like it is
>> in java, c,c++ etc), so the way forward can not be
>> to add comments token in the lexical rule file.
>>
>> Can someone please give me an hint here?
>> I can see a solution where occurences of whitespace also
>> could mean a comment, but how do I do that?
>>
>> I'm fairly new to LPG and DSL so please have oversight
>> in my lack of understanding for this.
>>
>>
>> regards,
>> Aker Br
Re: How to get LPG to treat comments as terminals [message #574435 is a reply to message #25730] Fri, 21 November 2008 20:47 Go to previous message
Stan Sutton is currently offline Stan Sutton
Messages: 121
Registered: July 2009
Senior Member
Hi Aker,

That's clearer. I am not an expert on the workings of LPG, but I can
tell you a few things that might help.

The LPG representation of the parse stream contains both tokens and
adjuncts, where adjuncts represent comments. Typically, as a file is
parsed by an LPG parser, tokens go into a tokens array, comments go into
an adjuncts array, and a mapping is maintained between them. The
interface lpg.runtime.IParseStream contains methods
IToken[] getFollowingAdjuncts(int i);
IToken[] getPrecedingAdjuncts(int i);
where the argument is the index of a token in the token stream.

Of course, in order to get the comments into the adjuncts array in the
parse stream, you have to specify the grammar or lexer rules
appropriately. In our typical examples, we have rules for single line
comments; this is from a generated Lexer.gi file:

%Rules
Token ::= identifier /. checkForKeyWord();./
| number /. makeToken($_NUMBER);./
| DoubleLiteral /. makeToken($_DoubleLiteral);./
| white /. skipToken();./
| slc /. makeComment($_SINGLE_LINE_COMMENT);./
| ';' /. makeToken($_SEMICOLON);./
...

Here we do not skipToken for the comment but instead use
makeComment(..). makeComment(..) is the method that adds the comment
token to the adjuncts array.

If you are recognizing comments in your grammar rules (.g file) then I
think you should be able to invoke makeComment(..) from there as well.

Does this seem like it can support what you want to do?

Regards,

Stan



Aker wrote:
> Hi,
>
> Trying to clarify myself:
>
> We have implemented the LPG rules (lexer.gi) so that all the comments are
> skipped (using skipToken).
> So naturally the comments will not end up as a production in the AST. But I
> would like to be able to
> use comments the same way that JDT uses JavaDoc for method declaration. So
> when I hover over a
> method invocation I would like to see the comments that was defined prior to
> the method declaration part.
>
> I started to use the ASTView provided by the JDT team
> (http://www.eclipse.org/jdt/ui/astview/index.php)
> and I noticed that the AST for my file contains also all the comments.
>
> So to make it short; I would like to be able to get a production in the AST
> for the comments that (might) exist prior to
> the method declaration in our DSL. Or if that is easier all the comments in
> the file as productions in the AST.
>
>
> Regards,
> Aker
>
>
>
> "Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
> news:gg221k$a4q$1@build.eclipse.org...
>> Hello Acker,
>>
>> I am not sure that I understand what you are trying to do.
>>
>> I could try to guess, but I think I would be wrong.
>>
>> Can you please clarify what you already have and what you are trying to
>> achieve?
>>
>> Thanks,
>>
>> Stan
>>
>>
>>
>> Aker wrote:
>>> By looking into how JDT handles JavaDoc I have
>>> started to understand that the parser must produce
>>> terminals for comments. In the DSL we have created
>>> an IDE for comments can occur 'everyware' (like it is
>>> in java, c,c++ etc), so the way forward can not be
>>> to add comments token in the lexical rule file.
>>>
>>> Can someone please give me an hint here?
>>> I can see a solution where occurences of whitespace also
>>> could mean a comment, but how do I do that?
>>>
>>> I'm fairly new to LPG and DSL so please have oversight
>>> in my lack of understanding for this.
>>>
>>>
>>> regards,
>>> Aker Brückler
>>>
>
Re: How to get LPG to treat comments as terminals [message #574465 is a reply to message #25765] Sat, 22 November 2008 10:15 Go to previous message
Eclipse User
Originally posted by: aker1121.yahoo.com

Hi Stan,

Not only does it support what I wanted, but it is exactly what I wanted!
Thanks for valuable help. Much appreciated!!

I missed that makeComment ('macro') exists in a sample definition file, but
is there some LPG document (or web page) that covers this (and other
macros)?


BR

Aker



"Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
news:gg76p7$g5k$1@build.eclipse.org...
> Hi Aker,
>
> That's clearer. I am not an expert on the workings of LPG, but I can tell
> you a few things that might help.
>
> The LPG representation of the parse stream contains both tokens and
> adjuncts, where adjuncts represent comments. Typically, as a file is
> parsed by an LPG parser, tokens go into a tokens array, comments go into
> an adjuncts array, and a mapping is maintained between them. The
> interface lpg.runtime.IParseStream contains methods
> IToken[] getFollowingAdjuncts(int i);
> IToken[] getPrecedingAdjuncts(int i);
> where the argument is the index of a token in the token stream.
>
> Of course, in order to get the comments into the adjuncts array in the
> parse stream, you have to specify the grammar or lexer rules
> appropriately. In our typical examples, we have rules for single line
> comments; this is from a generated Lexer.gi file:
>
> %Rules
> Token ::= identifier /. checkForKeyWord();./
> | number /. makeToken($_NUMBER);./
> | DoubleLiteral /. makeToken($_DoubleLiteral);./
> | white /. skipToken();./
> | slc /. makeComment($_SINGLE_LINE_COMMENT);./ |
> ';' /. makeToken($_SEMICOLON);./
> ...
>
> Here we do not skipToken for the comment but instead use makeComment(..).
> makeComment(..) is the method that adds the comment token to the adjuncts
> array.
>
> If you are recognizing comments in your grammar rules (.g file) then I
> think you should be able to invoke makeComment(..) from there as well.
>
> Does this seem like it can support what you want to do?
>
> Regards,
>
> Stan
>
>
>
> Aker wrote:
>> Hi,
>>
>> Trying to clarify myself:
>>
>> We have implemented the LPG rules (lexer.gi) so that all the comments are
>> skipped (using skipToken).
>> So naturally the comments will not end up as a production in the AST. But
>> I would like to be able to
>> use comments the same way that JDT uses JavaDoc for method declaration.
>> So when I hover over a
>> method invocation I would like to see the comments that was defined prior
>> to the method declaration part.
>>
>> I started to use the ASTView provided by the JDT team
>> (http://www.eclipse.org/jdt/ui/astview/index.php)
>> and I noticed that the AST for my file contains also all the comments.
>>
>> So to make it short; I would like to be able to get a production in the
>> AST for the comments that (might) exist prior to
>> the method declaration in our DSL. Or if that is easier all the comments
>> in the file as productions in the AST.
>>
>>
>> Regards,
>> Aker
>>
>>
>>
>> "Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
>> news:gg221k$a4q$1@build.eclipse.org...
>>> Hello Acker,
>>>
>>> I am not sure that I understand what you are trying to do.
>>>
>>> I could try to guess, but I think I would be wrong.
>>>
>>> Can you please clarify what you already have and what you are trying to
>>> achieve?
>>>
>>> Thanks,
>>>
>>> Stan
>>>
>>>
>>>
>>> Aker wrote:
>>>> By looking into how JDT handles JavaDoc I have
>>>> started to understand that the parser must produce
>>>> terminals for comments. In the DSL we have created
>>>> an IDE for comments can occur 'everyware' (like it is
>>>> in java, c,c++ etc), so the way forward can not be
>>>> to add comments token in the lexical rule file.
>>>>
>>>> Can someone please give me an hint here?
>>>> I can see a solution where occurences of whitespace also
>>>> could mean a comment, but how do I do that?
>>>>
>>>> I'm fairly new to LPG and DSL so please have oversight
>>>> in my lack of understanding for this.
>>>>
>>>>
>>>> regards,
>>>> Aker Br
Re: How to get LPG to treat comments as terminals [message #574494 is a reply to message #25776] Mon, 24 November 2008 19:49 Go to previous message
Stan Sutton is currently offline Stan Sutton
Messages: 121
Registered: July 2009
Senior Member
Hi Aker,

Unfortunately, the macros were developed for the templates after the LPG
documentation was written.

It isn't hard to write your own macros, though. At the end of the
lexer.gi file you can introduce a "headers" section, e.g.,

%Headers
/.
private void sayHello() {
System.out.println("Hello world!");
}
./
%end

(There is a larger and more useful example at the end of the parser.g
file.) In the "headers" section you can write code for any "macros"
(classes, methods, fields) that you want to use with your rules. Then
you can refer to these in the code that is associated with the rules:


%Rules
Token ::= identifier /. checkForKeyWord();./
...
| slc /. makeComment($_SINGLE_LINE_COMMENT);
sayHello();
./
...

Also feel free to use any of the macros that are used in the examples in
any of the templates.

Regards,

Stan

Aker wrote:
> Hi Stan,
>
> Not only does it support what I wanted, but it is exactly what I wanted!
> Thanks for valuable help. Much appreciated!!
>
> I missed that makeComment ('macro') exists in a sample definition file, but
> is there some LPG document (or web page) that covers this (and other
> macros)?
>
>
> BR
>
> Aker
>
>
>
> "Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
> news:gg76p7$g5k$1@build.eclipse.org...
>> Hi Aker,
>>
>> That's clearer. I am not an expert on the workings of LPG, but I can tell
>> you a few things that might help.
>>
>> The LPG representation of the parse stream contains both tokens and
>> adjuncts, where adjuncts represent comments. Typically, as a file is
>> parsed by an LPG parser, tokens go into a tokens array, comments go into
>> an adjuncts array, and a mapping is maintained between them. The
>> interface lpg.runtime.IParseStream contains methods
>> IToken[] getFollowingAdjuncts(int i);
>> IToken[] getPrecedingAdjuncts(int i);
>> where the argument is the index of a token in the token stream.
>>
>> Of course, in order to get the comments into the adjuncts array in the
>> parse stream, you have to specify the grammar or lexer rules
>> appropriately. In our typical examples, we have rules for single line
>> comments; this is from a generated Lexer.gi file:
>>
>> %Rules
>> Token ::= identifier /. checkForKeyWord();./
>> | number /. makeToken($_NUMBER);./
>> | DoubleLiteral /. makeToken($_DoubleLiteral);./
>> | white /. skipToken();./
>> | slc /. makeComment($_SINGLE_LINE_COMMENT);./ |
>> ';' /. makeToken($_SEMICOLON);./
>> ...
>>
>> Here we do not skipToken for the comment but instead use makeComment(..).
>> makeComment(..) is the method that adds the comment token to the adjuncts
>> array.
>>
>> If you are recognizing comments in your grammar rules (.g file) then I
>> think you should be able to invoke makeComment(..) from there as well.
>>
>> Does this seem like it can support what you want to do?
>>
>> Regards,
>>
>> Stan
>>
>>
>>
>> Aker wrote:
>>> Hi,
>>>
>>> Trying to clarify myself:
>>>
>>> We have implemented the LPG rules (lexer.gi) so that all the comments are
>>> skipped (using skipToken).
>>> So naturally the comments will not end up as a production in the AST. But
>>> I would like to be able to
>>> use comments the same way that JDT uses JavaDoc for method declaration.
>>> So when I hover over a
>>> method invocation I would like to see the comments that was defined prior
>>> to the method declaration part.
>>>
>>> I started to use the ASTView provided by the JDT team
>>> (http://www.eclipse.org/jdt/ui/astview/index.php)
>>> and I noticed that the AST for my file contains also all the comments.
>>>
>>> So to make it short; I would like to be able to get a production in the
>>> AST for the comments that (might) exist prior to
>>> the method declaration in our DSL. Or if that is easier all the comments
>>> in the file as productions in the AST.
>>>
>>>
>>> Regards,
>>> Aker
>>>
>>>
>>>
>>> "Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
>>> news:gg221k$a4q$1@build.eclipse.org...
>>>> Hello Acker,
>>>>
>>>> I am not sure that I understand what you are trying to do.
>>>>
>>>> I could try to guess, but I think I would be wrong.
>>>>
>>>> Can you please clarify what you already have and what you are trying to
>>>> achieve?
>>>>
>>>> Thanks,
>>>>
>>>> Stan
>>>>
>>>>
>>>>
>>>> Aker wrote:
>>>>> By looking into how JDT handles JavaDoc I have
>>>>> started to understand that the parser must produce
>>>>> terminals for comments. In the DSL we have created
>>>>> an IDE for comments can occur 'everyware' (like it is
>>>>> in java, c,c++ etc), so the way forward can not be
>>>>> to add comments token in the lexical rule file.
>>>>>
>>>>> Can someone please give me an hint here?
>>>>> I can see a solution where occurences of whitespace also
>>>>> could mean a comment, but how do I do that?
>>>>>
>>>>> I'm fairly new to LPG and DSL so please have oversight
>>>>> in my lack of understanding for this.
>>>>>
>>>>>
>>>>> regards,
>>>>> Aker Brückler
>>>>>
>
>
Re: How to get LPG to treat comments as terminals [message #574513 is a reply to message #25730] Tue, 25 November 2008 06:16 Go to previous message
Ed Willink is currently offline Ed Willink
Messages: 4086
Registered: July 2009
Senior Member
Hi Aker

> So naturally the comments will not end up as a production in the AST. But I
> would like to be able to
> use comments the same way that JDT uses JavaDoc for method declaration.

I've been thinking about this from a different requirement: refactoring
and a reverse engineeering update.

In the AST it is easy to identify all semantically equivalent re-uses of
the same name, so the AST can be changed, but that must be reverse
engineered into the source text again, which requires preservation of
the comments.

I think it's a mistake to make comments first class terminals, since
then every production must have COMMENTopt between every terminal.

Instead it may be better to use an enhanced Token to which a
partial-line single-line-comment is attached as a suffix and to which
other paragraph and full-line single-line-comments are attached as
prefixes. The Tokens then parse normally but persist the comments for
those activities that require them.

Regards

Ed Willink
Re: How to get LPG to treat comments as terminals [message #575068 is a reply to message #25841] Mon, 02 March 2009 06:57 Go to previous message
Ed Willink is currently offline Ed Willink
Messages: 4086
Registered: July 2009
Senior Member
Ed Willink wrote:

> Instead it may be better to use an enhanced Token to which a
> partial-line single-line-comment is attached as a suffix and to which
> other paragraph and full-line single-line-comments are attached as
> prefixes. The Tokens then parse normally but persist the comments for
> those activities that require them.

Actually there's no need to do anything. Leave comments out of the grammar,
they can always be recovered later by a special purpose comment, newline,
whitespace analyzer.

The comments for 'currentToken' are those between 'previousToken'.end
and 'currentToken'.start and/or between 'currentToken'.end
and 'nextToken'.start. Working in this way will allow you to use
syntactic context to decide how much of A,B,C,D,E in

int a; // A
// B
/* C */
/**
* D
*/
// E
int e;

belongs to int a, and how much to int e; It also gives
you more scope for inserting tabs for refactoring and other issues.

Regards

Ed Willink
Previous Topic:Using IMP for LPG in non-IMP target
Next Topic:Cannot find a solution satisfying the following requirements org.eclipse.ltk.core.refactoring
Goto Forum:
  


Current Time: Tue Oct 21 00:47:50 GMT 2014

Powered by FUDForum. Page generated in 0.02487 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software