Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Xtext lexer ambiguity(Question on how to solve an Xtext lexer problem I have)
Xtext lexer ambiguity [message #1744159] Thu, 22 September 2016 22:06 Go to next message
Larry Cousin is currently offline Larry CousinFriend
Messages: 15
Registered: September 2016
Junior Member
I am working on porting an existing Bison-based language to Xtext/Antlr. I have several lexer rules that are conflicting and not giving me the desired tokens. Here are the rules:

fragment UCLETR: ('A'..'Z');
fragment LCLETR: ('a'..'z');
fragment DIGIT: ('0'..'9');
TEMPLATE: ('$'|'*'|'+'|'.'|'@'|'-'|':'|'&'|UCLETR|LCLETR|'_'|DIGIT)+'?';
IDENTIFIER: (LCLETR|UCLETR)(LCLETR|UCLETR|DIGIT|'_')*;
POINTER: '->';
ASSIGN: ':=';

When I get the tokens for:

c->priority := DX_VERY_HIGH_PRIORITY;

it tries to read the "c-" as a TEMPLATE (not even trying IDENTIFIER), then it says it found a ">" and was expecting a "?" (from TEMPLATE) and dumped the "c->". Shouldn't TEMPATE fail and then have IDENTIFIER match "c" and then POINTER match "->"? I don't know how this works and I don't know how to fix it. Any help would be greatly appreciated! Sorry if this is not the place to ask such things...
Re: Xtext lexer ambiguity [message #1744862 is a reply to message #1744159] Fri, 30 September 2016 20:48 Go to previous messageGo to next message
Larry Cousin is currently offline Larry CousinFriend
Messages: 15
Registered: September 2016
Junior Member
I had a hard time finding information on how to do lexer backtracking, but by perusing the Web I found that I could include these options in my mwe2 file:

backtrackLexer = true
classSplitting = true
fieldsPerClass = "200"
methodsPerClass= "100"

This made it so that the lexer backtracked and found "c" as an IDENTIFIER instead of a failed TEMPLATE "c-". I had to include the other 3 options because with just backtrackLexer=true, Xtext created a java method larger than the largest one accepted. I also had to redefine "terminal ANY_OTHER:" in my grammar since org.eclipse.xtext.common.Terminals set it to catch a single character. When I redefined ANY_OTHER to an "unfindable" token, the lexer then started using the terminals I defined.

Now my new question--I was under the impression that Antrl3 could not backtrack the lexer. Since Xtext is using Antlr3, how is backtrackLexer=true working? Also, is there a way to get Antlr3 (AntlrWorks) to backtrack the lexer?

Thanks!
Larry
Re: Xtext lexer ambiguity [message #1744874 is a reply to message #1744862] Sat, 01 October 2016 07:47 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14661
Registered: July 2009
Senior Member
Can you give the minimal grammar you used .
Maybe tricking around with terminal and datatype rules solves the problem


Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Xtext lexer ambiguity [message #1744881 is a reply to message #1744874] Sat, 01 October 2016 09:37 Go to previous message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 14661
Registered: July 2009
Senior Member
(e.g. by splitting the terminals up into TEMPLATE_WITHOUT_MINUS, "-", ">" and use datatype rules to stick them together

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Previous Topic:Xtext code generation using other JVM languages, e.g. Scala
Next Topic:Xtext standalone validation issue
Goto Forum:
  


Current Time: Fri Mar 29 07:03:47 GMT 2024

Powered by FUDForum. Page generated in 0.02364 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top