Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Problem with whitespace and terminals
Problem with whitespace and terminals [message #882792] Thu, 07 June 2012 08:03 Go to next message
Axel Guckelsberger is currently offline Axel GuckelsbergerFriend
Messages: 347
Registered: July 2009
Senior Member
Hi all,

I am working on a grammar which has just been imported from an existing
ecore file.

When generating the language infrastructure I receive dozens of these
warnings:

warning(209): ../mydsl/src-
gen/mydsl/parser/antlr/internal/InternalMyDsl.g:16194:1: Multiple token
rules can match input such as "'D'": T__94, T__123, T__127, T__187, T__189,
T__194, T__206, T__221, T__229, T__230, T__265, T__283, T__284, RULE_ID

As a result, token(s)
T__123,T__127,T__187,T__189,T__194,T__206,T__221,T__229,T__230,T__265,T__283,T__284,RULE_ID
were disabled for that input

So as soon as multiple keywords start with the same letter they seem to
become competitors. Probably a lexer problem?


When I removed the Terminals grammar and just used some needed rules of it,
like ID and STRING, suddenly everything worked fine. Except that I had no
comments functionality anymore. If I copy the whitespace declarations
(hidden()) as well the problems occur again.

Any idea how to prevent the whitespace rules breaking my grammar?

TIA,
Axel
Re: Problem with whitespace and terminals [message #882962 is a reply to message #882792] Thu, 07 June 2012 14:31 Go to previous messageGo to next message
Axel Guckelsberger is currently offline Axel GuckelsbergerFriend
Messages: 347
Registered: July 2009
Senior Member
I don't know why but the problem went away after regeneration.
Re: Problem with whitespace and terminals [message #883256 is a reply to message #882792] Fri, 08 June 2012 06:42 Go to previous messageGo to next message
Karsten Thoms is currently offline Karsten ThomsFriend
Messages: 740
Registered: July 2009
Location: Dortmund, Germany
Senior Member

It usually is a sign that your grammar became ambigious. This can partially be solved by semantic predicates, or by enabling backtracking. Backtracking should be avoided when possible, since it can result in unpredictable results.

~Karsten


Need professional support for Xtext, EMF, Eclipse IDE?
Go to: http://devhub.karakun.com
Twitter : @kthoms
Blog : www.karsten-thoms.de
Re: Problem with whitespace and terminals [message #884663 is a reply to message #883256] Mon, 11 June 2012 15:20 Go to previous messageGo to next message
Axel Guckelsberger is currently offline Axel GuckelsbergerFriend
Messages: 347
Registered: July 2009
Senior Member
Karsten,

thanks for the pointer. Indeed the warning messages do sometimes appear and
sometimes not, it seems like a random choice.

Any idea how to examine the grammar for finding the ambigious parts? I tried
opening and checking the .g file with antlrworks, but the grammar validated
fine there.

Greetings,
Axel

Karsten Thoms wrote:

> It usually is a sign that your grammar became ambigious. This can
> partially be solved by semantic predicates, or by enabling backtracking.
> Backtracking should be avoided when possible, since it can result in
> unpredictable results.
>
> ~Karsten
Re: Problem with whitespace and terminals [message #884674 is a reply to message #884663] Mon, 11 June 2012 15:33 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik LindbergFriend
Messages: 2509
Registered: July 2009
Senior Member
On 2012-11-06 17:20, Axel Guckelsberger wrote:
> Karsten,
>
> thanks for the pointer. Indeed the warning messages do sometimes appear and
> sometimes not, it seems like a random choice.
>
> Any idea how to examine the grammar for finding the ambigious parts? I tried
> opening and checking the .g file with antlrworks, but the grammar validated
> fine there.
>
> Greetings,
> Axel
>
> Karsten Thoms wrote:
>
>> It usually is a sign that your grammar became ambigious. This can
>> partially be solved by semantic predicates, or by enabling backtracking.
>> Backtracking should be avoided when possible, since it can result in
>> unpredictable results.
>>
>> ~Karsten
>
Do you have backtracking turned on? That can mask lots of grammar problems.

I had similar issues with a complex (backtracking) grammar, and I had to
do two things - make sure the grammar generator had plenty of memory,
and that it did not time out (a parameter can be set in the workflow for
that). I also had to make sure nothing heavy was running on my machine
at the same time.

Later I reduced the complexity (moving things to an external lexer)
which had several positive effects, one being that a lot less resources
were required due to big reduction in tokens/rules.

Regards
- henrik
Re: Problem with whitespace and terminals [message #885275 is a reply to message #884674] Tue, 12 June 2012 18:13 Go to previous message
Axel Guckelsberger is currently offline Axel GuckelsbergerFriend
Messages: 347
Registered: July 2009
Senior Member
Hi Henrik,

> Do you have backtracking turned on? That can mask lots of grammar
> problems.
No, there is no backtracking enabled.

> make sure the grammar generator had plenty of memory,
> and that it did not time out (a parameter can be set in the workflow for
> that). I also had to make sure nothing heavy was running on my machine
> at the same time.
Thanks, will look for these aspects.

Axel
Previous Topic:Predicate or Left-factoring with Ecore objects
Next Topic:XtextResource loading with JvmTypes and importURI
Goto Forum:
  


Current Time: Fri Oct 30 14:48:18 GMT 2020

Powered by FUDForum. Page generated in 0.02096 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top