Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » need help with terminal/keyword problem
need help with terminal/keyword problem [message #653722] Thu, 10 February 2011 19:44 Go to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
Hi,
I have this terminal rule

terminal FQN :
(':' ':')? ('0'..'9'|'a'..'z'|'A'..'Z'|'_'|'.'|'-')+
((':' ':') ('0'..'9'|'a'..'z'|'A'..'Z'|'_'|'.'|'-')+)*;

and a set of keywords such as 'default', 'case', etc.
with the intent of allowing FQN to include keywords if also including
'::' as in '::default', or 'default::default', or naturally a::b::c::d etc.

I get problems when the lexer sees a sequence that ends with a single
':', it will then not recognize what precedes it as a keyword. e.g.

// false error on space after ':' (expected ':')
default:

// ok
default::default

// ok - kw 'default' followd by ':'
default :

I use an external lexer so I can include "antlr tricks", but at the
moment I don't understand how to make the lexer do the same when it sees
a ':' that is not followed by a another ':' as it does when it sees say
a space.

I do not want to use parser rules for this (moving this to the lexer is
done to reduce the complexity in the grammar).

Hope someone can help me...
Regards
- henrik
Re: need help with terminal/keyword problem [message #653839 is a reply to message #653722] Fri, 11 February 2011 13:43 Go to previous message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
I managed to solve this in the end, but wonder why it has to be so darned
complicated...

I ended up with a rule using a fragment for the :: part, and it required a
predicate of (':' ':')=>. this got me as far as not greedily lexing a
single colon. To make it perform as if terminated by a space (the dfa
prediction generated by antlr does not seem to consider the predicated
colons...), I had to write an action last in the rule to check if parsed
text is a keyword and return the keyword token instead.

- henrik

Henrik Lindberg <henrik.lindberg@cloudsmith.com> wrote:
> Hi,
> I have this terminal rule
>
> terminal FQN :
> (':' ':')? ('0'..'9'|'a'..'z'|'A'..'Z'|'_'|'.'|'-')+
> ((':' ':') ('0'..'9'|'a'..'z'|'A'..'Z'|'_'|'.'|'-')+)*;
>
> and a set of keywords such as 'default', 'case', etc.
> with the intent of allowing FQN to include keywords if also including
> '::' as in '::default', or 'default::default', or naturally a::b::c::d etc.
>
> I get problems when the lexer sees a sequence that ends with a single
> ':', it will then not recognize what precedes it as a keyword. e.g.
>
> // false error on space after ':' (expected ':')
> default:
>
> // ok
> default::default
>
> // ok - kw 'default' followd by ':'
> default :
>
> I use an external lexer so I can include "antlr tricks", but at the
> moment I don't understand how to make the lexer do the same when it sees
> a ':' that is not followed by a another ':' as it does when it sees say a space.
>
> I do not want to use parser rules for this (moving this to the lexer is
> done to reduce the complexity in the grammar).
>
> Hope someone can help me...
> Regards
> - henrik


--
- henrik
Previous Topic:Help with IF grammar
Next Topic:NullPointerException at LazyURIEncoder.resolveShortFragment
Goto Forum:
  


Current Time: Sat Apr 27 03:31:21 GMT 2024

Powered by FUDForum. Page generated in 0.03063 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top