Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Antlr failure on complicated grammar(When my grammar is large and my machine is heavily loaded, Antlr fails)
Antlr failure on complicated grammar [message #524723] Fri, 02 April 2010 03:17 Go to next message
Thom Boyer is currently offline Thom BoyerFriend
Messages: 7
Registered: April 2010
Junior Member
I am trying to write a grammar for a Java-like language using xtext. Things have been going quite well, but I seem to have reached a threshold of complexity where Antlr fails. The interesting thing is that as I modify my grammar, simplifying and complexifying it, I'm finding that Antlr's results are not deterministic: at a certain point of grammar complexity, Antlr manages to generate a lexer and parser for the grammar but only if I'm not otherwise tasking my CPU. If I play music while Antlr is running, then Antlr fails:

warning(205): ../com.sample.xtext.sky.ui/src-gen/com/sample/ui/contentassi st/antlr/internal/InternalSky.g:1:8: ANTLR could not analyze this decision in rule Tokens; often this is because of recursive rule references visible from the left edge of alternatives. ANTLR will re-analyze the decision with a fixed lookahead of k=1. Consider using "options {k=1;}" for that decision and possibly adding a syntactic predicate.
warning(209): ../com.sample.xtext.sky.ui/src-gen/com/sample/ui/contentassi st/antlr/internal/InternalSky.g:10594:1: Multiple token rules can match input such as "'s'": T23, T89, RULE_ID, RULE_ANY_OTHER
As a result, tokens(s) T89,RULE_ID,RULE_ANY_OTHER were disabled for that input

The second and third line ("Multiple token rules can match input such as..." and "As a result, tokens(s) ... were disabled for that input") are repeated many times, for different tokens.

It appears (from looking at http://www.antlr.org/pipermail/antlr-interest/2009-May/03456 3.html) that Antlr actually uses a timer to decide when to quit its attempt to generate a DFA from an NFA. And there is a commandline option for changing the timer duration, -Xconversiontimeout.

It appears that I need to get xtext to pass something like "-Xconverstiontimeout 10000" to antlr when I run my GenerateLang.mwe2 as "MWE2 Workflow". Can anybody tell me how to do that?

Or perhaps I'm completely wrong about what's going on here. Any tips would be appreciated.
=thom
Re: Antlr failure on complicated grammar [message #524732 is a reply to message #524723] Fri, 02 April 2010 05:43 Go to previous messageGo to next message
Sven Efftinge is currently offline Sven EfftingeFriend
Messages: 1823
Registered: July 2009
Senior Member
Thom Boyer schrieb:
> I am trying to write a grammar for a Java-like language using xtext.
> Things have been going quite well, but I seem to have reached a
> threshold of complexity where Antlr fails. The interesting thing is that
> as I modify my grammar, simplifying and complexifying it, I'm finding
> that Antlr's results are not deterministic: at a certain point of
> grammar complexity, Antlr manages to generate a lexer and parser for the
> grammar but only if I'm not otherwise tasking my CPU. If I play music
> while Antlr is running, then Antlr fails:

What kind of music? ... (just kidding, hihi)

> It appears that I need to get xtext to pass something like
> "-Xconverstiontimeout 10000" to antlr when I run my GenerateLang.mwe2 as
> "MWE2 Workflow". Can anybody tell me how to do that?

You need to create a subclass of
de.itemis.xtext.antlr.XtextAntlrGeneratorFragment and override generate.
Just copy the five lines from the original body and modify the call to
AntlrToolRunner,run() to use
de.itemis.xtext.antlr.AntlrToolRunner.runWithParams(String, String...)
and pass the needed arguments.

You also need to override to
@Override
protected String getTemplate() {
return XtextAntlrGeneratorFragment.class.getName().replaceAll("\\. ",
"::");
}

>
> Or perhaps I'm completely wrong about what's going on here. Any tips
> would be appreciated.

Sounds strange indeed and you're the first with this effect, but it
makes sense from your description.

Sven


--
Need professional support for Xtext and EMF?
Go to: http://xtext.itemis.com
Twitter : @svenefftinge
Blog : blog.efftinge.de
Re: Antlr failure on complicated grammar [message #524745 is a reply to message #524723] Fri, 02 April 2010 07:59 Go to previous messageGo to next message
Sylvain EVEILLARD is currently offline Sylvain EVEILLARDFriend
Messages: 556
Registered: July 2009
Senior Member
You could also try to allocate more heap to the mwe process.
Re: Antlr failure on complicated grammar [message #524790 is a reply to message #524745] Fri, 02 April 2010 14:20 Go to previous message
Thom Boyer is currently offline Thom BoyerFriend
Messages: 7
Registered: April 2010
Junior Member
Increasing the heap allocated to the MWE2 process was sufficient to solve the problem.

I put "-Xmx2000m" into the "VM arguments" field of the "Arguments" tab of the "Run Configurations..." context menu of "GenerateMyLang.mwe2", and the full grammar processed without errors.

Thanks so much!
=thom


=thom
Previous Topic:StateBased container manager (0.7.2 -> 1.0.0)
Next Topic:List of references
Goto Forum:
  


Current Time: Mon May 16 13:02:32 GMT 2022

Powered by FUDForum. Page generated in 0.01464 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top