Home » Archived » IMP » How to get LPG to treat comments as terminals
| How to get LPG to treat comments as terminals [message #25538] |
Wed, 19 November 2008 20:02  |
Eclipse User |
|
|
|
Originally posted by: aker1121.yahoo.com
By looking into how JDT handles JavaDoc I have
started to understand that the parser must produce
terminals for comments. In the DSL we have created
an IDE for comments can occur 'everyware' (like it is
in java, c,c++ etc), so the way forward can not be
to add comments token in the lexical rule file.
Can someone please give me an hint here?
I can see a solution where occurences of whitespace also
could mean a comment, but how do I do that?
I'm fairly new to LPG and DSL so please have oversight
in my lack of understanding for this.
regards,
Aker Br
|
|
| |
| Re: How to get LPG to treat comments as terminals [message #25730 is a reply to message #25654] |
Thu, 20 November 2008 16:54  |
| Eclipse User |
|
|
|
Originally posted by: aker1121.yahoo.com
Hi,
Trying to clarify myself:
We have implemented the LPG rules (lexer.gi) so that all the comments are
skipped (using skipToken).
So naturally the comments will not end up as a production in the AST. But I
would like to be able to
use comments the same way that JDT uses JavaDoc for method declaration. So
when I hover over a
method invocation I would like to see the comments that was defined prior to
the method declaration part.
I started to use the ASTView provided by the JDT team
(http://www.eclipse.org/jdt/ui/astview/index.php)
and I noticed that the AST for my file contains also all the comments.
So to make it short; I would like to be able to get a production in the AST
for the comments that (might) exist prior to
the method declaration in our DSL. Or if that is easier all the comments in
the file as productions in the AST.
Regards,
Aker
"Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
news:gg221k$a4q$1@build.eclipse.org...
> Hello Acker,
>
> I am not sure that I understand what you are trying to do.
>
> I could try to guess, but I think I would be wrong.
>
> Can you please clarify what you already have and what you are trying to
> achieve?
>
> Thanks,
>
> Stan
>
>
>
> Aker wrote:
>> By looking into how JDT handles JavaDoc I have
>> started to understand that the parser must produce
>> terminals for comments. In the DSL we have created
>> an IDE for comments can occur 'everyware' (like it is
>> in java, c,c++ etc), so the way forward can not be
>> to add comments token in the lexical rule file.
>>
>> Can someone please give me an hint here?
>> I can see a solution where occurences of whitespace also
>> could mean a comment, but how do I do that?
>>
>> I'm fairly new to LPG and DSL so please have oversight
>> in my lack of understanding for this.
>>
>>
>> regards,
>> Aker Br
|
|
|
| Re: How to get LPG to treat comments as terminals [message #25765 is a reply to message #25730] |
Fri, 21 November 2008 20:47 |
 Stan Sutton
Messages: 121
Registered: July 2009 |
Senior Member |
|
|
Hi Aker,
That's clearer. I am not an expert on the workings of LPG, but I can
tell you a few things that might help.
The LPG representation of the parse stream contains both tokens and
adjuncts, where adjuncts represent comments. Typically, as a file is
parsed by an LPG parser, tokens go into a tokens array, comments go into
an adjuncts array, and a mapping is maintained between them. The
interface lpg.runtime.IParseStream contains methods
IToken[] getFollowingAdjuncts(int i);
IToken[] getPrecedingAdjuncts(int i);
where the argument is the index of a token in the token stream.
Of course, in order to get the comments into the adjuncts array in the
parse stream, you have to specify the grammar or lexer rules
appropriately. In our typical examples, we have rules for single line
comments; this is from a generated Lexer.gi file:
%Rules
Token ::= identifier /. checkForKeyWord();./
| number /. makeToken($_NUMBER);./
| DoubleLiteral /. makeToken($_DoubleLiteral);./
| white /. skipToken();./
| slc /. makeComment($_SINGLE_LINE_COMMENT);./
| ';' /. makeToken($_SEMICOLON);./
...
Here we do not skipToken for the comment but instead use
makeComment(..). makeComment(..) is the method that adds the comment
token to the adjuncts array.
If you are recognizing comments in your grammar rules (.g file) then I
think you should be able to invoke makeComment(..) from there as well.
Does this seem like it can support what you want to do?
Regards,
Stan
Aker wrote:
> Hi,
>
> Trying to clarify myself:
>
> We have implemented the LPG rules (lexer.gi) so that all the comments are
> skipped (using skipToken).
> So naturally the comments will not end up as a production in the AST. But I
> would like to be able to
> use comments the same way that JDT uses JavaDoc for method declaration. So
> when I hover over a
> method invocation I would like to see the comments that was defined prior to
> the method declaration part.
>
> I started to use the ASTView provided by the JDT team
> (http://www.eclipse.org/jdt/ui/astview/index.php)
> and I noticed that the AST for my file contains also all the comments.
>
> So to make it short; I would like to be able to get a production in the AST
> for the comments that (might) exist prior to
> the method declaration in our DSL. Or if that is easier all the comments in
> the file as productions in the AST.
>
>
> Regards,
> Aker
>
>
>
> "Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
> news:gg221k$a4q$1@build.eclipse.org...
>> Hello Acker,
>>
>> I am not sure that I understand what you are trying to do.
>>
>> I could try to guess, but I think I would be wrong.
>>
>> Can you please clarify what you already have and what you are trying to
>> achieve?
>>
>> Thanks,
>>
>> Stan
>>
>>
>>
>> Aker wrote:
>>> By looking into how JDT handles JavaDoc I have
>>> started to understand that the parser must produce
>>> terminals for comments. In the DSL we have created
>>> an IDE for comments can occur 'everyware' (like it is
>>> in java, c,c++ etc), so the way forward can not be
>>> to add comments token in the lexical rule file.
>>>
>>> Can someone please give me an hint here?
>>> I can see a solution where occurences of whitespace also
>>> could mean a comment, but how do I do that?
>>>
>>> I'm fairly new to LPG and DSL so please have oversight
>>> in my lack of understanding for this.
>>>
>>>
>>> regards,
>>> Aker Brückler
>>>
>
|
|
|
| Re: How to get LPG to treat comments as terminals [message #25776 is a reply to message #25765] |
Sat, 22 November 2008 10:15 |
| Eclipse User |
|
|
|
Originally posted by: aker1121.yahoo.com
Hi Stan,
Not only does it support what I wanted, but it is exactly what I wanted!
Thanks for valuable help. Much appreciated!!
I missed that makeComment ('macro') exists in a sample definition file, but
is there some LPG document (or web page) that covers this (and other
macros)?
BR
Aker
"Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
news:gg76p7$g5k$1@build.eclipse.org...
> Hi Aker,
>
> That's clearer. I am not an expert on the workings of LPG, but I can tell
> you a few things that might help.
>
> The LPG representation of the parse stream contains both tokens and
> adjuncts, where adjuncts represent comments. Typically, as a file is
> parsed by an LPG parser, tokens go into a tokens array, comments go into
> an adjuncts array, and a mapping is maintained between them. The
> interface lpg.runtime.IParseStream contains methods
> IToken[] getFollowingAdjuncts(int i);
> IToken[] getPrecedingAdjuncts(int i);
> where the argument is the index of a token in the token stream.
>
> Of course, in order to get the comments into the adjuncts array in the
> parse stream, you have to specify the grammar or lexer rules
> appropriately. In our typical examples, we have rules for single line
> comments; this is from a generated Lexer.gi file:
>
> %Rules
> Token ::= identifier /. checkForKeyWord();./
> | number /. makeToken($_NUMBER);./
> | DoubleLiteral /. makeToken($_DoubleLiteral);./
> | white /. skipToken();./
> | slc /. makeComment($_SINGLE_LINE_COMMENT);./ |
> ';' /. makeToken($_SEMICOLON);./
> ...
>
> Here we do not skipToken for the comment but instead use makeComment(..).
> makeComment(..) is the method that adds the comment token to the adjuncts
> array.
>
> If you are recognizing comments in your grammar rules (.g file) then I
> think you should be able to invoke makeComment(..) from there as well.
>
> Does this seem like it can support what you want to do?
>
> Regards,
>
> Stan
>
>
>
> Aker wrote:
>> Hi,
>>
>> Trying to clarify myself:
>>
>> We have implemented the LPG rules (lexer.gi) so that all the comments are
>> skipped (using skipToken).
>> So naturally the comments will not end up as a production in the AST. But
>> I would like to be able to
>> use comments the same way that JDT uses JavaDoc for method declaration.
>> So when I hover over a
>> method invocation I would like to see the comments that was defined prior
>> to the method declaration part.
>>
>> I started to use the ASTView provided by the JDT team
>> (http://www.eclipse.org/jdt/ui/astview/index.php)
>> and I noticed that the AST for my file contains also all the comments.
>>
>> So to make it short; I would like to be able to get a production in the
>> AST for the comments that (might) exist prior to
>> the method declaration in our DSL. Or if that is easier all the comments
>> in the file as productions in the AST.
>>
>>
>> Regards,
>> Aker
>>
>>
>>
>> "Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
>> news:gg221k$a4q$1@build.eclipse.org...
>>> Hello Acker,
>>>
>>> I am not sure that I understand what you are trying to do.
>>>
>>> I could try to guess, but I think I would be wrong.
>>>
>>> Can you please clarify what you already have and what you are trying to
>>> achieve?
>>>
>>> Thanks,
>>>
>>> Stan
>>>
>>>
>>>
>>> Aker wrote:
>>>> By looking into how JDT handles JavaDoc I have
>>>> started to understand that the parser must produce
>>>> terminals for comments. In the DSL we have created
>>>> an IDE for comments can occur 'everyware' (like it is
>>>> in java, c,c++ etc), so the way forward can not be
>>>> to add comments token in the lexical rule file.
>>>>
>>>> Can someone please give me an hint here?
>>>> I can see a solution where occurences of whitespace also
>>>> could mean a comment, but how do I do that?
>>>>
>>>> I'm fairly new to LPG and DSL so please have oversight
>>>> in my lack of understanding for this.
>>>>
>>>>
>>>> regards,
>>>> Aker Br
|
|
|
| Re: How to get LPG to treat comments as terminals [message #25802 is a reply to message #25776] |
Mon, 24 November 2008 19:49 |
Stan Sutton
Messages: 121
Registered: July 2009 |
Senior Member |
|
|
Hi Aker,
Unfortunately, the macros were developed for the templates after the LPG
documentation was written.
It isn't hard to write your own macros, though. At the end of the
lexer.gi file you can introduce a "headers" section, e.g.,
%Headers
/.
private void sayHello() {
System.out.println("Hello world!");
}
./
%end
(There is a larger and more useful example at the end of the parser.g
file.) In the "headers" section you can write code for any "macros"
(classes, methods, fields) that you want to use with your rules. Then
you can refer to these in the code that is associated with the rules:
%Rules
Token ::= identifier /. checkForKeyWord();./
...
| slc /. makeComment($_SINGLE_LINE_COMMENT);
sayHello();
./
...
Also feel free to use any of the macros that are used in the examples in
any of the templates.
Regards,
Stan
Aker wrote:
> Hi Stan,
>
> Not only does it support what I wanted, but it is exactly what I wanted!
> Thanks for valuable help. Much appreciated!!
>
> I missed that makeComment ('macro') exists in a sample definition file, but
> is there some LPG document (or web page) that covers this (and other
> macros)?
>
>
> BR
>
> Aker
>
>
>
> "Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
> news:gg76p7$g5k$1@build.eclipse.org...
>> Hi Aker,
>>
>> That's clearer. I am not an expert on the workings of LPG, but I can tell
>> you a few things that might help.
>>
>> The LPG representation of the parse stream contains both tokens and
>> adjuncts, where adjuncts represent comments. Typically, as a file is
>> parsed by an LPG parser, tokens go into a tokens array, comments go into
>> an adjuncts array, and a mapping is maintained between them. The
>> interface lpg.runtime.IParseStream contains methods
>> IToken[] getFollowingAdjuncts(int i);
>> IToken[] getPrecedingAdjuncts(int i);
>> where the argument is the index of a token in the token stream.
>>
>> Of course, in order to get the comments into the adjuncts array in the
>> parse stream, you have to specify the grammar or lexer rules
>> appropriately. In our typical examples, we have rules for single line
>> comments; this is from a generated Lexer.gi file:
>>
>> %Rules
>> Token ::= identifier /. checkForKeyWord();./
>> | number /. makeToken($_NUMBER);./
>> | DoubleLiteral /. makeToken($_DoubleLiteral);./
>> | white /. skipToken();./
>> | slc /. makeComment($_SINGLE_LINE_COMMENT);./ |
>> ';' /. makeToken($_SEMICOLON);./
>> ...
>>
>> Here we do not skipToken for the comment but instead use makeComment(..).
>> makeComment(..) is the method that adds the comment token to the adjuncts
>> array.
>>
>> If you are recognizing comments in your grammar rules (.g file) then I
>> think you should be able to invoke makeComment(..) from there as well.
>>
>> Does this seem like it can support what you want to do?
>>
>> Regards,
>>
>> Stan
>>
>>
>>
>> Aker wrote:
>>> Hi,
>>>
>>> Trying to clarify myself:
>>>
>>> We have implemented the LPG rules (lexer.gi) so that all the comments are
>>> skipped (using skipToken).
>>> So naturally the comments will not end up as a production in the AST. But
>>> I would like to be able to
>>> use comments the same way that JDT uses JavaDoc for method declaration.
>>> So when I hover over a
>>> method invocation I would like to see the comments that was defined prior
>>> to the method declaration part.
>>>
>>> I started to use the ASTView provided by the JDT team
>>> (http://www.eclipse.org/jdt/ui/astview/index.php)
>>> and I noticed that the AST for my file contains also all the comments.
>>>
>>> So to make it short; I would like to be able to get a production in the
>>> AST for the comments that (might) exist prior to
>>> the method declaration in our DSL. Or if that is easier all the comments
>>> in the file as productions in the AST.
>>>
>>>
>>> Regards,
>>> Aker
>>>
>>>
>>>
>>> "Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
>>> news:gg221k$a4q$1@build.eclipse.org...
>>>> Hello Acker,
>>>>
>>>> I am not sure that I understand what you are trying to do.
>>>>
>>>> I could try to guess, but I think I would be wrong.
>>>>
>>>> Can you please clarify what you already have and what you are trying to
>>>> achieve?
>>>>
>>>> Thanks,
>>>>
>>>> Stan
>>>>
>>>>
>>>>
>>>> Aker wrote:
>>>>> By looking into how JDT handles JavaDoc I have
>>>>> started to understand that the parser must produce
>>>>> terminals for comments. In the DSL we have created
>>>>> an IDE for comments can occur 'everyware' (like it is
>>>>> in java, c,c++ etc), so the way forward can not be
>>>>> to add comments token in the lexical rule file.
>>>>>
>>>>> Can someone please give me an hint here?
>>>>> I can see a solution where occurences of whitespace also
>>>>> could mean a comment, but how do I do that?
>>>>>
>>>>> I'm fairly new to LPG and DSL so please have oversight
>>>>> in my lack of understanding for this.
>>>>>
>>>>>
>>>>> regards,
>>>>> Aker Brückler
>>>>>
>
>
|
|
| | | |
| Re: How to get LPG to treat comments as terminals [message #574407 is a reply to message #25654] |
Thu, 20 November 2008 16:54 |
| Eclipse User |
|
|
|
Originally posted by: aker1121.yahoo.com
Hi,
Trying to clarify myself:
We have implemented the LPG rules (lexer.gi) so that all the comments are
skipped (using skipToken).
So naturally the comments will not end up as a production in the AST. But I
would like to be able to
use comments the same way that JDT uses JavaDoc for method declaration. So
when I hover over a
method invocation I would like to see the comments that was defined prior to
the method declaration part.
I started to use the ASTView provided by the JDT team
(http://www.eclipse.org/jdt/ui/astview/index.php)
and I noticed that the AST for my file contains also all the comments.
So to make it short; I would like to be able to get a production in the AST
for the comments that (might) exist prior to
the method declaration in our DSL. Or if that is easier all the comments in
the file as productions in the AST.
Regards,
Aker
"Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
news:gg221k$a4q$1@build.eclipse.org...
> Hello Acker,
>
> I am not sure that I understand what you are trying to do.
>
> I could try to guess, but I think I would be wrong.
>
> Can you please clarify what you already have and what you are trying to
> achieve?
>
> Thanks,
>
> Stan
>
>
>
> Aker wrote:
>> By looking into how JDT handles JavaDoc I have
>> started to understand that the parser must produce
>> terminals for comments. In the DSL we have created
>> an IDE for comments can occur 'everyware' (like it is
>> in java, c,c++ etc), so the way forward can not be
>> to add comments token in the lexical rule file.
>>
>> Can someone please give me an hint here?
>> I can see a solution where occurences of whitespace also
>> could mean a comment, but how do I do that?
>>
>> I'm fairly new to LPG and DSL so please have oversight
>> in my lack of understanding for this.
>>
>>
>> regards,
>> Aker Br
|
|
|
| Re: How to get LPG to treat comments as terminals [message #574435 is a reply to message #25730] |
Fri, 21 November 2008 20:47 |
Stan Sutton
Messages: 121
Registered: July 2009 |
Senior Member |
|
|
Hi Aker,
That's clearer. I am not an expert on the workings of LPG, but I can
tell you a few things that might help.
The LPG representation of the parse stream contains both tokens and
adjuncts, where adjuncts represent comments. Typically, as a file is
parsed by an LPG parser, tokens go into a tokens array, comments go into
an adjuncts array, and a mapping is maintained between them. The
interface lpg.runtime.IParseStream contains methods
IToken[] getFollowingAdjuncts(int i);
IToken[] getPrecedingAdjuncts(int i);
where the argument is the index of a token in the token stream.
Of course, in order to get the comments into the adjuncts array in the
parse stream, you have to specify the grammar or lexer rules
appropriately. In our typical examples, we have rules for single line
comments; this is from a generated Lexer.gi file:
%Rules
Token ::= identifier /. checkForKeyWord();./
| number /. makeToken($_NUMBER);./
| DoubleLiteral /. makeToken($_DoubleLiteral);./
| white /. skipToken();./
| slc /. makeComment($_SINGLE_LINE_COMMENT);./
| ';' /. makeToken($_SEMICOLON);./
...
Here we do not skipToken for the comment but instead use
makeComment(..). makeComment(..) is the method that adds the comment
token to the adjuncts array.
If you are recognizing comments in your grammar rules (.g file) then I
think you should be able to invoke makeComment(..) from there as well.
Does this seem like it can support what you want to do?
Regards,
Stan
Aker wrote:
> Hi,
>
> Trying to clarify myself:
>
> We have implemented the LPG rules (lexer.gi) so that all the comments are
> skipped (using skipToken).
> So naturally the comments will not end up as a production in the AST. But I
> would like to be able to
> use comments the same way that JDT uses JavaDoc for method declaration. So
> when I hover over a
> method invocation I would like to see the comments that was defined prior to
> the method declaration part.
>
> I started to use the ASTView provided by the JDT team
> (http://www.eclipse.org/jdt/ui/astview/index.php)
> and I noticed that the AST for my file contains also all the comments.
>
> So to make it short; I would like to be able to get a production in the AST
> for the comments that (might) exist prior to
> the method declaration in our DSL. Or if that is easier all the comments in
> the file as productions in the AST.
>
>
> Regards,
> Aker
>
>
>
> "Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
> news:gg221k$a4q$1@build.eclipse.org...
>> Hello Acker,
>>
>> I am not sure that I understand what you are trying to do.
>>
>> I could try to guess, but I think I would be wrong.
>>
>> Can you please clarify what you already have and what you are trying to
>> achieve?
>>
>> Thanks,
>>
>> Stan
>>
>>
>>
>> Aker wrote:
>>> By looking into how JDT handles JavaDoc I have
>>> started to understand that the parser must produce
>>> terminals for comments. In the DSL we have created
>>> an IDE for comments can occur 'everyware' (like it is
>>> in java, c,c++ etc), so the way forward can not be
>>> to add comments token in the lexical rule file.
>>>
>>> Can someone please give me an hint here?
>>> I can see a solution where occurences of whitespace also
>>> could mean a comment, but how do I do that?
>>>
>>> I'm fairly new to LPG and DSL so please have oversight
>>> in my lack of understanding for this.
>>>
>>>
>>> regards,
>>> Aker Brückler
>>>
>
|
|
|
| Re: How to get LPG to treat comments as terminals [message #574465 is a reply to message #25765] |
Sat, 22 November 2008 10:15 |
| Eclipse User |
|
|
|
Originally posted by: aker1121.yahoo.com
Hi Stan,
Not only does it support what I wanted, but it is exactly what I wanted!
Thanks for valuable help. Much appreciated!!
I missed that makeComment ('macro') exists in a sample definition file, but
is there some LPG document (or web page) that covers this (and other
macros)?
BR
Aker
"Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
news:gg76p7$g5k$1@build.eclipse.org...
> Hi Aker,
>
> That's clearer. I am not an expert on the workings of LPG, but I can tell
> you a few things that might help.
>
> The LPG representation of the parse stream contains both tokens and
> adjuncts, where adjuncts represent comments. Typically, as a file is
> parsed by an LPG parser, tokens go into a tokens array, comments go into
> an adjuncts array, and a mapping is maintained between them. The
> interface lpg.runtime.IParseStream contains methods
> IToken[] getFollowingAdjuncts(int i);
> IToken[] getPrecedingAdjuncts(int i);
> where the argument is the index of a token in the token stream.
>
> Of course, in order to get the comments into the adjuncts array in the
> parse stream, you have to specify the grammar or lexer rules
> appropriately. In our typical examples, we have rules for single line
> comments; this is from a generated Lexer.gi file:
>
> %Rules
> Token ::= identifier /. checkForKeyWord();./
> | number /. makeToken($_NUMBER);./
> | DoubleLiteral /. makeToken($_DoubleLiteral);./
> | white /. skipToken();./
> | slc /. makeComment($_SINGLE_LINE_COMMENT);./ |
> ';' /. makeToken($_SEMICOLON);./
> ...
>
> Here we do not skipToken for the comment but instead use makeComment(..).
> makeComment(..) is the method that adds the comment token to the adjuncts
> array.
>
> If you are recognizing comments in your grammar rules (.g file) then I
> think you should be able to invoke makeComment(..) from there as well.
>
> Does this seem like it can support what you want to do?
>
> Regards,
>
> Stan
>
>
>
> Aker wrote:
>> Hi,
>>
>> Trying to clarify myself:
>>
>> We have implemented the LPG rules (lexer.gi) so that all the comments are
>> skipped (using skipToken).
>> So naturally the comments will not end up as a production in the AST. But
>> I would like to be able to
>> use comments the same way that JDT uses JavaDoc for method declaration.
>> So when I hover over a
>> method invocation I would like to see the comments that was defined prior
>> to the method declaration part.
>>
>> I started to use the ASTView provided by the JDT team
>> (http://www.eclipse.org/jdt/ui/astview/index.php)
>> and I noticed that the AST for my file contains also all the comments.
>>
>> So to make it short; I would like to be able to get a production in the
>> AST for the comments that (might) exist prior to
>> the method declaration in our DSL. Or if that is easier all the comments
>> in the file as productions in the AST.
>>
>>
>> Regards,
>> Aker
>>
>>
>>
>> "Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
>> news:gg221k$a4q$1@build.eclipse.org...
>>> Hello Acker,
>>>
>>> I am not sure that I understand what you are trying to do.
>>>
>>> I could try to guess, but I think I would be wrong.
>>>
>>> Can you please clarify what you already have and what you are trying to
>>> achieve?
>>>
>>> Thanks,
>>>
>>> Stan
>>>
>>>
>>>
>>> Aker wrote:
>>>> By looking into how JDT handles JavaDoc I have
>>>> started to understand that the parser must produce
>>>> terminals for comments. In the DSL we have created
>>>> an IDE for comments can occur 'everyware' (like it is
>>>> in java, c,c++ etc), so the way forward can not be
>>>> to add comments token in the lexical rule file.
>>>>
>>>> Can someone please give me an hint here?
>>>> I can see a solution where occurences of whitespace also
>>>> could mean a comment, but how do I do that?
>>>>
>>>> I'm fairly new to LPG and DSL so please have oversight
>>>> in my lack of understanding for this.
>>>>
>>>>
>>>> regards,
>>>> Aker Br
|
|
|
| Re: How to get LPG to treat comments as terminals [message #574494 is a reply to message #25776] |
Mon, 24 November 2008 19:49 |
Stan Sutton
Messages: 121
Registered: July 2009 |
Senior Member |
|
|
Hi Aker,
Unfortunately, the macros were developed for the templates after the LPG
documentation was written.
It isn't hard to write your own macros, though. At the end of the
lexer.gi file you can introduce a "headers" section, e.g.,
%Headers
/.
private void sayHello() {
System.out.println("Hello world!");
}
./
%end
(There is a larger and more useful example at the end of the parser.g
file.) In the "headers" section you can write code for any "macros"
(classes, methods, fields) that you want to use with your rules. Then
you can refer to these in the code that is associated with the rules:
%Rules
Token ::= identifier /. checkForKeyWord();./
...
| slc /. makeComment($_SINGLE_LINE_COMMENT);
sayHello();
./
...
Also feel free to use any of the macros that are used in the examples in
any of the templates.
Regards,
Stan
Aker wrote:
> Hi Stan,
>
> Not only does it support what I wanted, but it is exactly what I wanted!
> Thanks for valuable help. Much appreciated!!
>
> I missed that makeComment ('macro') exists in a sample definition file, but
> is there some LPG document (or web page) that covers this (and other
> macros)?
>
>
> BR
>
> Aker
>
>
>
> "Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
> news:gg76p7$g5k$1@build.eclipse.org...
>> Hi Aker,
>>
>> That's clearer. I am not an expert on the workings of LPG, but I can tell
>> you a few things that might help.
>>
>> The LPG representation of the parse stream contains both tokens and
>> adjuncts, where adjuncts represent comments. Typically, as a file is
>> parsed by an LPG parser, tokens go into a tokens array, comments go into
>> an adjuncts array, and a mapping is maintained between them. The
>> interface lpg.runtime.IParseStream contains methods
>> IToken[] getFollowingAdjuncts(int i);
>> IToken[] getPrecedingAdjuncts(int i);
>> where the argument is the index of a token in the token stream.
>>
>> Of course, in order to get the comments into the adjuncts array in the
>> parse stream, you have to specify the grammar or lexer rules
>> appropriately. In our typical examples, we have rules for single line
>> comments; this is from a generated Lexer.gi file:
>>
>> %Rules
>> Token ::= identifier /. checkForKeyWord();./
>> | number /. makeToken($_NUMBER);./
>> | DoubleLiteral /. makeToken($_DoubleLiteral);./
>> | white /. skipToken();./
>> | slc /. makeComment($_SINGLE_LINE_COMMENT);./ |
>> ';' /. makeToken($_SEMICOLON);./
>> ...
>>
>> Here we do not skipToken for the comment but instead use makeComment(..).
>> makeComment(..) is the method that adds the comment token to the adjuncts
>> array.
>>
>> If you are recognizing comments in your grammar rules (.g file) then I
>> think you should be able to invoke makeComment(..) from there as well.
>>
>> Does this seem like it can support what you want to do?
>>
>> Regards,
>>
>> Stan
>>
>>
>>
>> Aker wrote:
>>> Hi,
>>>
>>> Trying to clarify myself:
>>>
>>> We have implemented the LPG rules (lexer.gi) so that all the comments are
>>> skipped (using skipToken).
>>> So naturally the comments will not end up as a production in the AST. But
>>> I would like to be able to
>>> use comments the same way that JDT uses JavaDoc for method declaration.
>>> So when I hover over a
>>> method invocation I would like to see the comments that was defined prior
>>> to the method declaration part.
>>>
>>> I started to use the ASTView provided by the JDT team
>>> (http://www.eclipse.org/jdt/ui/astview/index.php)
>>> and I noticed that the AST for my file contains also all the comments.
>>>
>>> So to make it short; I would like to be able to get a production in the
>>> AST for the comments that (might) exist prior to
>>> the method declaration in our DSL. Or if that is easier all the comments
>>> in the file as productions in the AST.
>>>
>>>
>>> Regards,
>>> Aker
>>>
>>>
>>>
>>> "Stanley Sutton" <suttons@us.ibm.com> skrev i meddelandet
>>> news:gg221k$a4q$1@build.eclipse.org...
>>>> Hello Acker,
>>>>
>>>> I am not sure that I understand what you are trying to do.
>>>>
>>>> I could try to guess, but I think I would be wrong.
>>>>
>>>> Can you please clarify what you already have and what you are trying to
>>>> achieve?
>>>>
>>>> Thanks,
>>>>
>>>> Stan
>>>>
>>>>
>>>>
>>>> Aker wrote:
>>>>> By looking into how JDT handles JavaDoc I have
>>>>> started to understand that the parser must produce
>>>>> terminals for comments. In the DSL we have created
>>>>> an IDE for comments can occur 'everyware' (like it is
>>>>> in java, c,c++ etc), so the way forward can not be
>>>>> to add comments token in the lexical rule file.
>>>>>
>>>>> Can someone please give me an hint here?
>>>>> I can see a solution where occurences of whitespace also
>>>>> could mean a comment, but how do I do that?
>>>>>
>>>>> I'm fairly new to LPG and DSL so please have oversight
>>>>> in my lack of understanding for this.
>>>>>
>>>>>
>>>>> regards,
>>>>> Aker Brückler
>>>>>
>
>
|
|
| | |
Goto Forum:
Current Time: Tue Apr 23 06:50:49 GMT 2024
Powered by FUDForum. Page generated in 0.05132 seconds |
] 
Search
Help
Register
Login
Home
