Eclipse Community Forums: TMF (Xtext) » Xtext lexer ambiguity

Help

Home

Home » Modeling » TMF (Xtext) » Xtext lexer ambiguity(Question on how to solve an Xtext lexer problem I have)

Show: Today's Messages :: Show Polls :: Message Navigator

Xtext lexer ambiguity [message #1744159]

Thu, 22 September 2016 22:06

Larry Cousin

Messages: 15
Registered: September 2016

Junior Member

I am working on porting an existing Bison-based language to Xtext/Antlr. I have several lexer rules that are conflicting and not giving me the desired tokens. Here are the rules:

fragment UCLETR: ('A'..'Z');
fragment LCLETR: ('a'..'z');
fragment DIGIT: ('0'..'9');
TEMPLATE: ('$'|'*'|'+'|'.'|'@'|'-'|':'|'&'|UCLETR|LCLETR|'_'|DIGIT)+'?';
IDENTIFIER: (LCLETR|UCLETR)(LCLETR|UCLETR|DIGIT|'_')*;
POINTER: '->';
ASSIGN: ':=';

When I get the tokens for:

c->priority := DX_VERY_HIGH_PRIORITY;

it tries to read the "c-" as a TEMPLATE (not even trying IDENTIFIER), then it says it found a ">" and was expecting a "?" (from TEMPLATE) and dumped the "c->". Shouldn't TEMPATE fail and then have IDENTIFIER match "c" and then POINTER match "->"? I don't know how this works and I don't know how to fix it. Any help would be greatly appreciated! Sorry if this is not the place to ask such things...

Report message to a moderator

Re: Xtext lexer ambiguity [message #1744862 is a reply to message #1744159]

Fri, 30 September 2016 20:48

Larry Cousin

Messages: 15
Registered: September 2016

Junior Member

I had a hard time finding information on how to do lexer backtracking, but by perusing the Web I found that I could include these options in my mwe2 file:

backtrackLexer = true
classSplitting = true
fieldsPerClass = "200"
methodsPerClass= "100"

This made it so that the lexer backtracked and found "c" as an IDENTIFIER instead of a failed TEMPLATE "c-". I had to include the other 3 options because with just backtrackLexer=true, Xtext created a java method larger than the largest one accepted. I also had to redefine "terminal ANY_OTHER:" in my grammar since org.eclipse.xtext.common.Terminals set it to catch a single character. When I redefined ANY_OTHER to an "unfindable" token, the lexer then started using the terminals I defined.

Now my new question--I was under the impression that Antrl3 could not backtrack the lexer. Since Xtext is using Antlr3, how is backtrackLexer=true working? Also, is there a way to get Antlr3 (AntlrWorks) to backtrack the lexer?

Thanks!
Larry

Report message to a moderator

Re: Xtext lexer ambiguity [message #1744874 is a reply to message #1744862]

Sat, 01 October 2016 07:47

Christian Dietrich

Messages: 14498
Registered: July 2009

Senior Member

Can you give the minimal grammar you used .
Maybe tricking around with terminal and datatype rules solves the problem

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: Xtext lexer ambiguity [message #1744881 is a reply to message #1744874]

Sat, 01 October 2016 09:37

Christian Dietrich

Messages: 14498
Registered: July 2009

Senior Member

(e.g. by splitting the terminals up into TEMPLATE_WITHOUT_MINUS, "-", ">" and use datatype rules to stick them together

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Previous Topic:	Xtext code generation using other JVM languages, e.g. Scala
Next Topic:	Xtext standalone validation issue

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Fri Jun 09 12:45:07 GMT 2023

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter