Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Token Splitting via custom lexer?(Inquiring about customizing lexer to extend some features.)
Token Splitting via custom lexer? [message #1822406] Thu, 05 March 2020 17:30 Go to next message
Jonathan Eugenio is currently offline Jonathan EugenioFriend
Messages: 9
Registered: March 2020
Junior Member
Hi everyone.

I'm looking to implement token splitting (i.e., replacing a single token with multiple tokens) but I'm confused with how to implement it correctly. I have found some repositories that provide examples of custom lexer files but I have also noticed that there is a lack of information on token splitting in general. Generally I'm stuck in trying to implement something that works the way I want it to.

What I would like to do is develop a DSL that sees this token:
"2J"
And interprets it as:
"J J"
It should be able to handle other cases like 3J -> J J J, and so on.

Can anyone point me in the right direction on how to implement this kind of functionality? Thanks
Re: Token Splitting via custom lexer? [message #1822679 is a reply to message #1822406] Wed, 11 March 2020 09:59 Go to previous messageGo to next message
Karsten Thoms is currently offline Karsten ThomsFriend
Messages: 739
Registered: July 2009
Location: Dortmund, Germany
Senior Member

Token splitting is no common use case in Xtext languages. The default lexer can't do this, so searching the links to use a custom lexer is the right starting point. From there this is out of the scope of Xtext.
Re: Token Splitting via custom lexer? [message #1822698 is a reply to message #1822679] Wed, 11 March 2020 15:24 Go to previous messageGo to next message
Jonathan Eugenio is currently offline Jonathan EugenioFriend
Messages: 9
Registered: March 2020
Junior Member
Thank you for your points. I understand that doing this is out of the scope of Xtext, but are there specific resources to look into that will provide more information on the custom lexer? Thanks.
Re: Token Splitting via custom lexer? [message #1822717 is a reply to message #1822698] Thu, 12 March 2020 06:41 Go to previous message
Tamas Miklossy is currently offline Tamas MiklossyFriend
Messages: 114
Registered: February 2016
Senior Member
Hello Jonathan,

you can find more information about a custom lexer in the following forum thread:
https://www.eclipse.org/forums/index.php?t=msg&th=1101305&goto=1817099&#msg_1817099

Hope that helps,
Tamás
Previous Topic:Use Language Server Xtext
Next Topic:Understanding lambdas
Goto Forum:
  


Current Time: Fri Oct 23 06:25:39 GMT 2020

Powered by FUDForum. Page generated in 0.01715 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top