Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » 64K limit on InternalParser(How to get around 64K limit present in ANTLR 4.0x and previous)
64K limit on InternalParser [message #1068029] Thu, 11 July 2013 18:44 Go to next message
Michael Card is currently offline Michael CardFriend
Messages: 7
Registered: June 2013
Junior Member
Hello everyone-

The Xtext development team directed me here from their Twitter feed. I have a DSL that is fairly small (~120 production rules including enum declarations) but it is apparently complex enough that it generates too many "DFAs" in the InternalParser class in the content assist package. A DFA is a static internal class that looks like this:

public static final BitSet FOLLOW_ruleradar_signal_description_in_entryRuleradar_signal_description67 = new BitSet(new long[]{0x0000000000000000L});

The ANTLR compiler generates a LOT of these and it blows the infamous Java 64K static initializer limit.

The Xtext development team suggest via Twitter that I try the SplitClasses option but that did nothing for this problem.

ANTLR 4.1 solves this problem by redesigning how it generates DFAs so that the 64K limit cannot be hit (they are probably all generated as separate classes that are statically imported).

My questions are:

(1) Is there any way around this problem with the current delivery of Xtext?
(2) If not, when will ANTLR 4.1 be incorporated into Xtext?

Thanks for any help you can provide!

-Mike
Re: 64K limit on InternalParser [message #1068032 is a reply to message #1068029] Thu, 11 July 2013 18:51 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 6550
Registered: July 2009
Senior Member
Hi,

can you please post the configs of the parser fragments in your project? (with classsplitting set)
Re: 64K limit on InternalParser [message #1068033 is a reply to message #1068032] Thu, 11 July 2013 18:56 Go to previous messageGo to next message
Michael Card is currently offline Michael CardFriend
Messages: 7
Registered: June 2013
Junior Member
Hey Christian-

OK I'll try. This is my MWE2 file where I put the fragment for classplitting

-Mike

Re: 64K limit on InternalParser [message #1068034 is a reply to message #1068033] Thu, 11 July 2013 18:58 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 6550
Registered: July 2009
Senior Member
ähhhmmmm
seems you did not see

fragment = parser.antlr.XtextAntlrUiGeneratorFragment auto-inject {}

did you?
Re: 64K limit on InternalParser [message #1068035 is a reply to message #1068034] Thu, 11 July 2013 18:59 Go to previous messageGo to next message
Michael Card is currently offline Michael CardFriend
Messages: 7
Registered: June 2013
Junior Member
Hey Christian-

Well I saw it but the contents of an MWE2 file are Greek to me. Is a change to it needed?

-Mike
Re: 64K limit on InternalParser [message #1068036 is a reply to message #1068035] Thu, 11 July 2013 19:01 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 6550
Registered: July 2009
Senior Member
HI,

xtext generates 2 parsers, a runtime parser and a content assist parser.
you have to activate class splitting for both of course.
Re: 64K limit on InternalParser [message #1068037 is a reply to message #1068036] Thu, 11 July 2013 19:06 Go to previous messageGo to next message
Michael Card is currently offline Michael CardFriend
Messages: 7
Registered: June 2013
Junior Member
Hey Christian-

OK near the bottom of my MWE2 file I did find and change this statement and it seems to have worked, is this what you were referring to?


// generates a more lightweight Antlr parser and lexer tailored for content assist
fragment = parser.antlr.XtextAntlrUiGeneratorFragment auto-inject {
options = {
classSplitting = true
}
}


-Mike
Re: 64K limit on InternalParser [message #1068038 is a reply to message #1068037] Thu, 11 July 2013 19:07 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 6550
Registered: July 2009
Senior Member
yes this i exaclty what i mean.
Re: 64K limit on InternalParser [message #1068040 is a reply to message #1068038] Thu, 11 July 2013 19:11 Go to previous messageGo to next message
Michael Card is currently offline Michael CardFriend
Messages: 7
Registered: June 2013
Junior Member
OK thanks Christian I think that has solved the problem. I have yet to go in and try the content assist, but it worked fine with some simple DSLs I was playing with before starting work on the "real" DSL so I am sure it will be OK. As a newcomer to Xtext I found it fairly easy to get started, the only "mysterious" realm to me is all the MWE2 stuff which thankfully I haven't had to delve into to do my DSL work (except of course for this work-around). Thanks again for your help!
Re: 64K limit on InternalParser [message #1068041 is a reply to message #1068040] Thu, 11 July 2013 19:16 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 6550
Registered: July 2009
Senior Member
Hi,

dont be afraid from mwe, it is just a "i dont wont to compose java object graphs with xml" dsl.
Re: 64K limit on InternalParser [message #1068045 is a reply to message #1068041] Thu, 11 July 2013 19:22 Go to previous messageGo to next message
Michael Card is currently offline Michael CardFriend
Messages: 7
Registered: June 2013
Junior Member
OK. BTW, I just tried content assist and it is working fine, so the SplitClasses option did indeed fix whatever was being generated by the ANTLR 3.x parser embedded in Xtext.
Re: 64K limit on InternalParser [message #1071072 is a reply to message #1068045] Fri, 19 July 2013 09:34 Go to previous messageGo to next message
Ingo Meyer is currently offline Ingo MeyerFriend
Messages: 127
Registered: July 2009
Senior Member
If you have a very big DSL classplitting does not help anymore.... even with both splitting params "fieldsPerClass" and "methodsPerClass" set to "50"

I just want to follow up here and asking if Xtext is planning to integrate ANTLR 4.1 in the future?

[Updated on: Fri, 19 July 2013 09:36]

Report message to a moderator

Re: 64K limit on InternalParser [message #1073160 is a reply to message #1071072] Wed, 24 July 2013 08:41 Go to previous message
Jan Koehnlein is currently offline Jan KoehnleinFriend
Messages: 658
Registered: July 2009
Senior Member
That's not planned. It would break a lot of existing API.

Am 19.07.13 11:34, schrieb Ingo Meyer:
> If you have a very big DSL classplitting does not help anymore.... even
> with both splitting params set to "50"
>
> I just want to follow up here and asking if Xtext is planning to
> integrate ANTLR 4.1 in the future?


--
Need professional support for Eclipse Modeling?
Go visit: http://xtext.itemis.com
Previous Topic:No EObjectDescription could be found in Scope
Next Topic:migration guide to Xtext 2.4.2
Goto Forum:
  


Current Time: Sat Dec 20 01:47:14 GMT 2014

Powered by FUDForum. Page generated in 0.02103 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software