Eclipse Community Forums: TMF (Xtext) » Token Splitting via custom lexer?

Home » Modeling » TMF (Xtext) » Token Splitting via custom lexer?(Inquiring about customizing lexer to extend some features.)

Show: Today's Messages :: Show Polls :: Message Navigator

Token Splitting via custom lexer? [message #1822406]

Thu, 05 March 2020 17:30

Jonathan Eugenio

Messages: 11
Registered: March 2020

Junior Member

Hi everyone.

I'm looking to implement token splitting (i.e., replacing a single token with multiple tokens) but I'm confused with how to implement it correctly. I have found some repositories that provide examples of custom lexer files but I have also noticed that there is a lack of information on token splitting in general. Generally I'm stuck in trying to implement something that works the way I want it to.

What I would like to do is develop a DSL that sees this token:
"2J"
And interprets it as:
"J J"
It should be able to handle other cases like 3J -> J J J, and so on.

Can anyone point me in the right direction on how to implement this kind of functionality? Thanks

Report message to a moderator

Re: Token Splitting via custom lexer? [message #1822679 is a reply to message #1822406]

Wed, 11 March 2020 09:59

Karsten Thoms

Messages: 762
Registered: July 2009
Location: Dortmund, Germany

Senior Member

Token splitting is no common use case in Xtext languages. The default lexer can't do this, so searching the links to use a custom lexer is the right starting point. From there this is out of the scope of Xtext.

Report message to a moderator

Re: Token Splitting via custom lexer? [message #1822698 is a reply to message #1822679]

Wed, 11 March 2020 15:24

Jonathan Eugenio

Messages: 11
Registered: March 2020

Junior Member

Thank you for your points. I understand that doing this is out of the scope of Xtext, but are there specific resources to look into that will provide more information on the custom lexer? Thanks.

Report message to a moderator

Re: Token Splitting via custom lexer? [message #1822717 is a reply to message #1822698]

Thu, 12 March 2020 06:41

Tamas Miklossy

Messages: 157
Registered: February 2016

Senior Member

Hello Jonathan,

you can find more information about a custom lexer in the following forum thread:
https://www.eclipse.org/forums/index.php?t=msg&th=1101305&goto=1817099&#msg_1817099

Hope that helps,
Tamás

Report message to a moderator

Previous Topic:	Use Language Server Xtext
Next Topic:	Understanding lambdas

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Thu Apr 25 01:55:00 GMT 2024

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter