| Odd problem with terminal ID [message #1745596] |
Wed, 12 October 2016 21:45  |
 Larry Cousin
Messages: 15
Registered: September 2016 |
Junior Member |
|
|
I have a very odd problem with the use of ID. Here is my input text:
bit bit32 b32,b31;
Here is my lexer grammar:
terminal BIT: 'bit';
terminal IDENTIFIER: ID;
The "," and the ";" are defined in the grammar productions. Xtext LEXER gives me an error on the "b" in "bit32", so I wanted to see exactly what tokens are being found. So I run the program shown below and get the following tokens:
Token type: 33 text: 'bit'
Token type: 193 text: ' '
Token type: 0 text: 'b'
Token type: 195 text: 'it32'
Token type: 193 text: ' '
Token type: 195 text: 'b32'
Token type: 10 text: ','
Token type: 195 text: 'b31'
Token type: 14 text: ';'
As you can see, it is pulling the "b" from "bit32" and return it as an error token. Token type 195 is IDENTIFIER and it is then returning the rest, "it32", as an IDENTIFIER. Also, as you can see, this only happens for IDENTIFIER when it finds a token that starts with another terminal like "bit." I have other examples that do the same thing. If I have a keyword terminal defined like "do" and then I use a variable like "dog", it will error out on the "d" and give "og" as the IDENTIFIER. As you can see "b32" and "B31" above were not split apart. BTW, I am doing backtracking for both the lexer and parser.
Can anyone help me unravel this mystery?
Thanks,
Larry
Program I used to print the tokens:
public static void main(String[] args)
{
String fileName = "test6.f";
File file = new File(fileName);
FileInputStream fis = null;
try {
// Open the input file stream
fis = new FileInputStream(file);
// Create a CharStream that reads from standard input
ANTLRInputStream input = new ANTLRInputStream(fis);
// Create a lexer that feeds off of input CharStream
InternalDomainLexer lexer = new InternalDomainLexer(input);
// Create a buffer of tokens pulled from the lexer
CommonTokenStream tokens = new CommonTokenStream(lexer);
List<?> tokenLst = tokens.getTokens();
for(int i = 0;i < tokenLst.size(); i++)
{
Token aToken = (Token) tokenLst.get(i);
System.out.println( "Token type: " + aToken.getType() + " text: '" + aToken.getText() + "'" );
}
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
|
|
|
|
|
|
|
Powered by
FUDForum. Page generated in 0.02315 seconds
] 
Search
Help
Register
Login
Home
