Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Odd problem with terminal ID(ID returning a single character if similar terminal defined)
Odd problem with terminal ID [message #1745596] Wed, 12 October 2016 21:45 Go to next message
Larry Cousin is currently offline Larry CousinFriend
Messages: 15
Registered: September 2016
Junior Member
I have a very odd problem with the use of ID. Here is my input text:

bit bit32 b32,b31;

Here is my lexer grammar:

terminal BIT: 'bit';
terminal IDENTIFIER: ID;

The "," and the ";" are defined in the grammar productions. Xtext LEXER gives me an error on the "b" in "bit32", so I wanted to see exactly what tokens are being found. So I run the program shown below and get the following tokens:

Token type: 33 text: 'bit'
Token type: 193 text: ' '
Token type: 0 text: 'b'
Token type: 195 text: 'it32'
Token type: 193 text: ' '
Token type: 195 text: 'b32'
Token type: 10 text: ','
Token type: 195 text: 'b31'
Token type: 14 text: ';'

As you can see, it is pulling the "b" from "bit32" and return it as an error token. Token type 195 is IDENTIFIER and it is then returning the rest, "it32", as an IDENTIFIER. Also, as you can see, this only happens for IDENTIFIER when it finds a token that starts with another terminal like "bit." I have other examples that do the same thing. If I have a keyword terminal defined like "do" and then I use a variable like "dog", it will error out on the "d" and give "og" as the IDENTIFIER. As you can see "b32" and "B31" above were not split apart. BTW, I am doing backtracking for both the lexer and parser.

Can anyone help me unravel this mystery?

Thanks,
Larry

Program I used to print the tokens:

public static void main(String[] args)
{
String fileName = "test6.f";
File file = new File(fileName);
FileInputStream fis = null;

try {
// Open the input file stream
fis = new FileInputStream(file);

// Create a CharStream that reads from standard input
ANTLRInputStream input = new ANTLRInputStream(fis);

// Create a lexer that feeds off of input CharStream
InternalDomainLexer lexer = new InternalDomainLexer(input);

// Create a buffer of tokens pulled from the lexer
CommonTokenStream tokens = new CommonTokenStream(lexer);
List<?> tokenLst = tokens.getTokens();

for(int i = 0;i < tokenLst.size(); i++)
{
Token aToken = (Token) tokenLst.get(i);
System.out.println( "Token type: " + aToken.getType() + " text: '" + aToken.getText() + "'" );
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
Re: Odd problem with terminal ID [message #1745600 is a reply to message #1745596] Thu, 13 October 2016 03:27 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13883
Registered: July 2009
Senior Member
Hi,

this is caused by backtracking lexer


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Odd problem with terminal ID [message #1745620 is a reply to message #1745600] Thu, 13 October 2016 13:10 Go to previous messageGo to next message
Larry Cousin is currently offline Larry CousinFriend
Messages: 15
Registered: September 2016
Junior Member
Any suggestions for fixing this without turning off backtractLexer? I need to have backtrackLexer = true or the lexer won't find other terminal tokens that I have defined.
Re: Odd problem with terminal ID [message #1745632 is a reply to message #1745620] Thu, 13 October 2016 15:40 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13883
Registered: July 2009
Senior Member
Sry I have no idea on that. Maybe not use antlr as lexer helps

https://github.com/TypeFox/xtext-jflex


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Odd problem with terminal ID [message #1745636 is a reply to message #1745632] Thu, 13 October 2016 16:17 Go to previous message
Larry Cousin is currently offline Larry CousinFriend
Messages: 15
Registered: September 2016
Junior Member
Thank-you for the suggestion.
Previous Topic:Terminal Rule Token based on previous crossreference
Next Topic:Symbol
Goto Forum:
  


Current Time: Fri Apr 16 07:51:51 GMT 2021

Powered by FUDForum. Page generated in 0.02077 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top