Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Design considerations supporting Heredoc with Xtext/antlr lexer
Design considerations supporting Heredoc with Xtext/antlr lexer [message #1053391] Sun, 05 May 2013 00:37 Go to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2501
Registered: July 2009
Senior Member
Hi,
In the Puppet language, there is going to be support for Ruby like
Heredoc expressions. In simple form (but also showing some complexity)
it looks like this.

$a = @(END).upcase
This is the text that is in the heredoc,
and it will be turned to uppercase.
END

I have implemented this in the Puppet language itself, so I know what
needs to be done (it is almost all in the lexer). My questions are about
if there are things I need to think about in Xtext when I do a similar
solution using the custom ANTLR based lexer in Geppetto.

Basically, when seeing '@(' there is lookahead in the lexer that:
a) slurps everything to the closing ')' and checks if this is a valid
heredoc tag (there are some options and parameters too that I did not
show above).
b) skip to end of the line, then slurp everything to the END marker.
c) record the position where regular lexer should continue (after the
END), and at what position the skipping start (at the end of the line).
d) return to the position after the closing ')' of the heredoc tag.

On subsequent calls to the lexer when advancing to the recorded position
for the heredoc start, the lexer skips to the recorded end of the heredoc.

My qustions are: Can I have a queue of tokens in the lexer. In the
runtime implementation this makes it possible to first produce a HEREDOC
token, and then a STRING token with the text, or should I try to creat a
more complex token? The problem is that it contains a discontinuous
sequence of bytes.

Secondly, will this work in Xtext? Basically tokens will appear in a non
linear fashion - i.e. tokens from offsets on line 5 will appear before
say offsets on line 3. Does that matter?

I am greatful if anyone have ideas or advise how to proceed with this.

Regards
- henrik
Re: Design considerations supporting Heredoc with Xtext/antlr lexer [message #1053392 is a reply to message #1053391] Sun, 05 May 2013 01:09 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2501
Registered: July 2009
Senior Member
I wonder especially how Xtext uses parser/lexer to process partial
input, this since the lexer now needs to have state. (In my current
implementation I also maintain state (the previous significant token is
remembered since some tokens may only appear after a certain other
token) and this seems to work fine. Now I need something a bit more
elaborate.

So, details on this would be much appreciated. (Or pointers to where to
look in the source).

Regards
- henrik

On 2013-05-05 2:37, Henrik Lindberg wrote:
> Hi,
> In the Puppet language, there is going to be support for Ruby like
> Heredoc expressions. In simple form (but also showing some complexity)
> it looks like this.
>
> $a = @(END).upcase
> This is the text that is in the heredoc,
> and it will be turned to uppercase.
> END
>
> I have implemented this in the Puppet language itself, so I know what
> needs to be done (it is almost all in the lexer). My questions are about
> if there are things I need to think about in Xtext when I do a similar
> solution using the custom ANTLR based lexer in Geppetto.
>
> Basically, when seeing '@(' there is lookahead in the lexer that:
> a) slurps everything to the closing ')' and checks if this is a valid
> heredoc tag (there are some options and parameters too that I did not
> show above).
> b) skip to end of the line, then slurp everything to the END marker.
> c) record the position where regular lexer should continue (after the
> END), and at what position the skipping start (at the end of the line).
> d) return to the position after the closing ')' of the heredoc tag.
>
> On subsequent calls to the lexer when advancing to the recorded position
> for the heredoc start, the lexer skips to the recorded end of the heredoc.
>
> My qustions are: Can I have a queue of tokens in the lexer. In the
> runtime implementation this makes it possible to first produce a HEREDOC
> token, and then a STRING token with the text, or should I try to creat a
> more complex token? The problem is that it contains a discontinuous
> sequence of bytes.
>
> Secondly, will this work in Xtext? Basically tokens will appear in a non
> linear fashion - i.e. tokens from offsets on line 5 will appear before
> say offsets on line 3. Does that matter?
>
> I am greatful if anyone have ideas or advise how to proceed with this.
>
> Regards
> - henrik
Re: Design considerations supporting Heredoc with Xtext/antlr lexer [message #1053393 is a reply to message #1053391] Sun, 05 May 2013 01:28 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2501
Registered: July 2009
Senior Member
On 2013-05-05 2:37, Henrik Lindberg wrote:

> My qustions are: Can I have a queue of tokens in the lexer. In the
> runtime implementation this makes it possible to first produce a HEREDOC
> token, and then a STRING token with the text, or should I try to creat a
> more complex token? The problem is that it contains a discontinuous
> sequence of bytes.
>
Answer found here:
http://www.antlr.org/wiki/pages/viewpage.action?pageId=3604497

That is straight forward (I already have half of that in my current
override).

- henrik
Re: Design considerations supporting Heredoc with Xtext/antlr lexer [message #1053523 is a reply to message #1053393] Mon, 06 May 2013 11:18 Go to previous messageGo to next message
Sebastian Zarnekow is currently offline Sebastian ZarnekowFriend
Messages: 2961
Registered: July 2009
Senior Member
Hi Henrik,

the described approach should work fine with Xtext, as long as you
customize the partial parsing accordingly, e.g. expand the partially
parsed region or initialize the lexer in a proper state.

Regards,
Sebastian
--
Looking for professional support for Xtext, Xtend or Eclipse Modeling?
Go visit: http://xtext.itemis.com

Am 05.05.13 03:28, schrieb Henrik Lindberg:
> On 2013-05-05 2:37, Henrik Lindberg wrote:
>
>> My qustions are: Can I have a queue of tokens in the lexer. In the
>> runtime implementation this makes it possible to first produce a HEREDOC
>> token, and then a STRING token with the text, or should I try to creat a
>> more complex token? The problem is that it contains a discontinuous
>> sequence of bytes.
>>
> Answer found here:
> http://www.antlr.org/wiki/pages/viewpage.action?pageId=3604497
>
> That is straight forward (I already have half of that in my current
> override).
>
> - henrik
>
Re: Design considerations supporting Heredoc with Xtext/antlr lexer [message #1053598 is a reply to message #1053523] Mon, 06 May 2013 16:33 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2501
Registered: July 2009
Senior Member
On 2013-06-05 13:18, Sebastian Zarnekow wrote:
> Hi Henrik,
>
> the described approach should work fine with Xtext, as long as you
> customize the partial parsing accordingly, e.g. expand the partially
> parsed region or initialize the lexer in a proper state.
>
> Regards,
> Sebastian

How do I find the places where partial parsing takes place (I don't have
any that I have introduced afaik) ?

- henrik
Re: Design considerations supporting Heredoc with Xtext/antlr lexer [message #1053649 is a reply to message #1053598] Tue, 07 May 2013 07:06 Go to previous messageGo to next message
Sebastian Zarnekow is currently offline Sebastian ZarnekowFriend
Messages: 2961
Registered: July 2009
Senior Member
Hi Henrik,

it's PartialParsingHelper.reparse(IParser, IParseResult, ReplaceRegion)

Regards,
Sebastian
--
Looking for professional support for Xtext, Xtend or Eclipse Modeling?
Go visit: http://xtext.itemis.com

Am 06.05.13 18:33, schrieb Henrik Lindberg:
> On 2013-06-05 13:18, Sebastian Zarnekow wrote:
>> Hi Henrik,
>>
>> the described approach should work fine with Xtext, as long as you
>> customize the partial parsing accordingly, e.g. expand the partially
>> parsed region or initialize the lexer in a proper state.
>>
>> Regards,
>> Sebastian
>
> How do I find the places where partial parsing takes place (I don't have
> any that I have introduced afaik) ?
>
> - henrik
Re: Design considerations supporting Heredoc with Xtext/antlr lexer [message #1053732 is a reply to message #1053649] Tue, 07 May 2013 12:23 Go to previous message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2501
Registered: July 2009
Senior Member
On 2013-07-05 9:06, Sebastian Zarnekow wrote:
> PartialParsingHelper.reparse(IParser, IParseResult, ReplaceRegion)
Thanks
- henrik
Previous Topic:strange behavior of XImportSection
Next Topic:Formating with Xtend
Goto Forum:
  


Current Time: Fri Dec 19 22:44:49 GMT 2014

Powered by FUDForum. Page generated in 0.01862 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software