Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » How do I capture all the text between two tokens?(How do I capture all the text between two tokens?)
icon5.gif  How do I capture all the text between two tokens? [message #638585] Thu, 11 November 2010 17:03 Go to next message
Chris Ainsley is currently offline Chris AinsleyFriend
Messages: 78
Registered: March 2010
Location: UK
Member
Hi,

I'm wondering what is the best way to consume the text between two tokens. The STRING terminator does not seem appropriate. I simply want all the characters until the '}%' token is reached. Do I need to define a new terminator category or is there some other way?

Scriptlet:
    '%{' text=STRING '}%';



Thanks,

Chris
Re: How do I capture all the text between two tokens? [message #638622 is a reply to message #638585] Thu, 11 November 2010 19:44 Go to previous messageGo to next message
Meinte Boersma is currently offline Meinte BoersmaFriend
Messages: 434
Registered: July 2009
Location: Leiden, Netherlands
Senior Member
What are these "terminators" I keep hearing about? Wink

If you're used to other parsing frameworks, I can understand the instinct to get down to business with terminals but Xtext is a little more nuanced than that. In this case, you probably want to use a datatype rule to capture the content between %{ }%. It's (easy to find) in the User Guide which is something to keep handy anyways.


Re: How do I capture all the text between two tokens? [message #638666 is a reply to message #638585] Fri, 12 November 2010 03:09 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
Hi,
Search this forum for the topic "terminals gets my knickers in a twist
every time" where Sebastian helped me with something similar.

The question is if you want everything between a START and and END to be
the raw input (all spaces, newlines, etc. except END) in one string, or
if you want a sequence of strings/tokens separated by any amount of
(ignored) whitespace.

Sebastian helped with terminal definitions that I use for templates -
corresponding to %{ }% in your example, but where it is possible to
escape inside the %{ }%.

If you do not need to escape inside the %{ }% you could try the ->
operator (everything from lhs to rhs inclusive) in the terminal - as in
this rule:

terminal JAVADOC : '/**' -> '*/' ;

If you are interested in looking at the implementation in the b3
project, start with the rule called Template in BeeLang.xtext and drill
down towards the terminals. (This implementation is more advanced than
just getting the "raw" input between two special tokes, but may be of
value if you plan to switch between "modes").

If you want a sequence of tokens you can instead use a (non terminal)
rule - something like this:

Rule : '%{' (ID | INT | STRING | Keyword1 | Keyword2 | ...)* '%}' ;

with the drawback that you have to list all keywords that may appear.

- henrik

On 11/11/10 6:03 PM, Chris Ainsley wrote:
> Hi,
>
> I'm wondering what is the best way to consume the text between two
> tokens. The STRING terminator does not seem appropriate. I simply want
> all the characters until the '}%' token is reached. Do I need to define
> a new terminator category or is there some other way?
>
> Scriptlet:
> '%{' text=STRING '}%';
>
>
> Thanks,
>
> Chris
>
Re: How do I capture all the text between two tokens? [message #638952 is a reply to message #638666] Sat, 13 November 2010 19:38 Go to previous message
Chris Ainsley is currently offline Chris AinsleyFriend
Messages: 78
Registered: March 2010
Location: UK
Member
Thanks...

At the moment I have managed to get it working using the terminal approach but will file this away in case I do get some side effects as you have pointed out.
Previous Topic:Code Template does not respect formatting / indentation?
Next Topic:Is it possible to export the Xtext parser?
Goto Forum:
  


Current Time: Thu Apr 25 11:13:03 GMT 2024

Powered by FUDForum. Page generated in 0.03329 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top