Home » Modeling » TMF (Xtext) » Design considerations supporting Heredoc with Xtext/antlr lexer
| Design considerations supporting Heredoc with Xtext/antlr lexer [message #1053391] |
Sun, 05 May 2013 00:37  |
 Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
Hi,
In the Puppet language, there is going to be support for Ruby like
Heredoc expressions. In simple form (but also showing some complexity)
it looks like this.
$a = @(END).upcase
This is the text that is in the heredoc,
and it will be turned to uppercase.
END
I have implemented this in the Puppet language itself, so I know what
needs to be done (it is almost all in the lexer). My questions are about
if there are things I need to think about in Xtext when I do a similar
solution using the custom ANTLR based lexer in Geppetto.
Basically, when seeing '@(' there is lookahead in the lexer that:
a) slurps everything to the closing ')' and checks if this is a valid
heredoc tag (there are some options and parameters too that I did not
show above).
b) skip to end of the line, then slurp everything to the END marker.
c) record the position where regular lexer should continue (after the
END), and at what position the skipping start (at the end of the line).
d) return to the position after the closing ')' of the heredoc tag.
On subsequent calls to the lexer when advancing to the recorded position
for the heredoc start, the lexer skips to the recorded end of the heredoc.
My qustions are: Can I have a queue of tokens in the lexer. In the
runtime implementation this makes it possible to first produce a HEREDOC
token, and then a STRING token with the text, or should I try to creat a
more complex token? The problem is that it contains a discontinuous
sequence of bytes.
Secondly, will this work in Xtext? Basically tokens will appear in a non
linear fashion - i.e. tokens from offsets on line 5 will appear before
say offsets on line 3. Does that matter?
I am greatful if anyone have ideas or advise how to proceed with this.
Regards
- henrik
|
|
|
| Re: Design considerations supporting Heredoc with Xtext/antlr lexer [message #1053392 is a reply to message #1053391] |
Sun, 05 May 2013 01:09  |
Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
I wonder especially how Xtext uses parser/lexer to process partial
input, this since the lexer now needs to have state. (In my current
implementation I also maintain state (the previous significant token is
remembered since some tokens may only appear after a certain other
token) and this seems to work fine. Now I need something a bit more
elaborate.
So, details on this would be much appreciated. (Or pointers to where to
look in the source).
Regards
- henrik
On 2013-05-05 2:37, Henrik Lindberg wrote:
> Hi,
> In the Puppet language, there is going to be support for Ruby like
> Heredoc expressions. In simple form (but also showing some complexity)
> it looks like this.
>
> $a = @(END).upcase
> This is the text that is in the heredoc,
> and it will be turned to uppercase.
> END
>
> I have implemented this in the Puppet language itself, so I know what
> needs to be done (it is almost all in the lexer). My questions are about
> if there are things I need to think about in Xtext when I do a similar
> solution using the custom ANTLR based lexer in Geppetto.
>
> Basically, when seeing '@(' there is lookahead in the lexer that:
> a) slurps everything to the closing ')' and checks if this is a valid
> heredoc tag (there are some options and parameters too that I did not
> show above).
> b) skip to end of the line, then slurp everything to the END marker.
> c) record the position where regular lexer should continue (after the
> END), and at what position the skipping start (at the end of the line).
> d) return to the position after the closing ')' of the heredoc tag.
>
> On subsequent calls to the lexer when advancing to the recorded position
> for the heredoc start, the lexer skips to the recorded end of the heredoc.
>
> My qustions are: Can I have a queue of tokens in the lexer. In the
> runtime implementation this makes it possible to first produce a HEREDOC
> token, and then a STRING token with the text, or should I try to creat a
> more complex token? The problem is that it contains a discontinuous
> sequence of bytes.
>
> Secondly, will this work in Xtext? Basically tokens will appear in a non
> linear fashion - i.e. tokens from offsets on line 5 will appear before
> say offsets on line 3. Does that matter?
>
> I am greatful if anyone have ideas or advise how to proceed with this.
>
> Regards
> - henrik
|
|
| |
| Re: Design considerations supporting Heredoc with Xtext/antlr lexer [message #1053523 is a reply to message #1053393] |
Mon, 06 May 2013 11:18 |
Sebastian Zarnekow
Messages: 3118
Registered: July 2009 |
Senior Member |
|
|
Hi Henrik,
the described approach should work fine with Xtext, as long as you
customize the partial parsing accordingly, e.g. expand the partially
parsed region or initialize the lexer in a proper state.
Regards,
Sebastian
--
Looking for professional support for Xtext, Xtend or Eclipse Modeling?
Go visit: http://xtext.itemis.com
Am 05.05.13 03:28, schrieb Henrik Lindberg:
> On 2013-05-05 2:37, Henrik Lindberg wrote:
>
>> My qustions are: Can I have a queue of tokens in the lexer. In the
>> runtime implementation this makes it possible to first produce a HEREDOC
>> token, and then a STRING token with the text, or should I try to creat a
>> more complex token? The problem is that it contains a discontinuous
>> sequence of bytes.
>>
> Answer found here:
> http://www.antlr.org/wiki/pages/viewpage.action?pageId=3604497
>
> That is straight forward (I already have half of that in my current
> override).
>
> - henrik
>
|
|
| |
| Re: Design considerations supporting Heredoc with Xtext/antlr lexer [message #1053649 is a reply to message #1053598] |
Tue, 07 May 2013 07:06 |
Sebastian Zarnekow
Messages: 3118
Registered: July 2009 |
Senior Member |
|
|
Hi Henrik,
it's PartialParsingHelper.reparse(IParser, IParseResult, ReplaceRegion)
Regards,
Sebastian
--
Looking for professional support for Xtext, Xtend or Eclipse Modeling?
Go visit: http://xtext.itemis.com
Am 06.05.13 18:33, schrieb Henrik Lindberg:
> On 2013-06-05 13:18, Sebastian Zarnekow wrote:
>> Hi Henrik,
>>
>> the described approach should work fine with Xtext, as long as you
>> customize the partial parsing accordingly, e.g. expand the partially
>> parsed region or initialize the lexer in a proper state.
>>
>> Regards,
>> Sebastian
>
> How do I find the places where partial parsing takes place (I don't have
> any that I have introduced afaik) ?
>
> - henrik
|
|
| | |
Goto Forum:
Current Time: Tue Apr 16 10:17:07 GMT 2024
Powered by FUDForum. Page generated in 0.13252 seconds |
] 
Search
Help
Register
Login
Home
