Eclipse Community Forums: TMF (Xtext) » 64K limit on InternalParser

Home » Modeling » TMF (Xtext) » 64K limit on InternalParser(How to get around 64K limit present in ANTLR 4.0x and previous)

Show: Today's Messages :: Show Polls :: Message Navigator

64K limit on InternalParser [message #1068029]

Thu, 11 July 2013 18:44

Michael Card

Messages: 7
Registered: June 2013

Junior Member

Hello everyone-

The Xtext development team directed me here from their Twitter feed. I have a DSL that is fairly small (~120 production rules including enum declarations) but it is apparently complex enough that it generates too many "DFAs" in the InternalParser class in the content assist package. A DFA is a static internal class that looks like this:

public static final BitSet FOLLOW_ruleradar_signal_description_in_entryRuleradar_signal_description67 = new BitSet(new long[]{0x0000000000000000L});

The ANTLR compiler generates a LOT of these and it blows the infamous Java 64K static initializer limit.

The Xtext development team suggest via Twitter that I try the SplitClasses option but that did nothing for this problem.

ANTLR 4.1 solves this problem by redesigning how it generates DFAs so that the 64K limit cannot be hit (they are probably all generated as separate classes that are statically imported).

My questions are:

(1) Is there any way around this problem with the current delivery of Xtext?
(2) If not, when will ANTLR 4.1 be incorporated into Xtext?

Thanks for any help you can provide!

-Mike

Report message to a moderator

Re: 64K limit on InternalParser [message #1068032 is a reply to message #1068029]

Thu, 11 July 2013 18:51

Christian Dietrich

Messages: 14665
Registered: July 2009

Senior Member

Hi,

can you please post the configs of the parser fragments in your project? (with classsplitting set)

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: 64K limit on InternalParser [message #1068033 is a reply to message #1068032]

Thu, 11 July 2013 18:56

Michael Card

Messages: 7
Registered: June 2013

Junior Member

Hey Christian-

OK I'll try. This is my MWE2 file where I put the fragment for classplitting

-Mike

Attachment: GenerateRSDL.mwe2
(Size: 4.64KB, Downloaded 228 times)

Report message to a moderator

Re: 64K limit on InternalParser [message #1068034 is a reply to message #1068033]

Thu, 11 July 2013 18:58

Christian Dietrich

Messages: 14665
Registered: July 2009

Senior Member

ähhhmmmm
seems you did not see

fragment = parser.antlr.XtextAntlrUiGeneratorFragment auto-inject {}

did you?

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: 64K limit on InternalParser [message #1068035 is a reply to message #1068034]

Thu, 11 July 2013 18:59

Michael Card

Messages: 7
Registered: June 2013

Junior Member

Hey Christian-

Well I saw it but the contents of an MWE2 file are Greek to me. Is a change to it needed?

-Mike

Report message to a moderator

Re: 64K limit on InternalParser [message #1068036 is a reply to message #1068035]

Thu, 11 July 2013 19:01

Christian Dietrich

Messages: 14665
Registered: July 2009

Senior Member

HI,

xtext generates 2 parsers, a runtime parser and a content assist parser.
you have to activate class splitting for both of course.

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: 64K limit on InternalParser [message #1068037 is a reply to message #1068036]

Thu, 11 July 2013 19:06

Michael Card

Messages: 7
Registered: June 2013

Junior Member

Hey Christian-

OK near the bottom of my MWE2 file I did find and change this statement and it seems to have worked, is this what you were referring to?

// generates a more lightweight Antlr parser and lexer tailored for content assist
fragment = parser.antlr.XtextAntlrUiGeneratorFragment auto-inject {
options = {
classSplitting = true
}
}

-Mike

Report message to a moderator

Re: 64K limit on InternalParser [message #1068038 is a reply to message #1068037]

Thu, 11 July 2013 19:07

Christian Dietrich

Messages: 14665
Registered: July 2009

Senior Member

yes this i exaclty what i mean.

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: 64K limit on InternalParser [message #1068040 is a reply to message #1068038]

Thu, 11 July 2013 19:11

Michael Card

Messages: 7
Registered: June 2013

Junior Member

OK thanks Christian I think that has solved the problem. I have yet to go in and try the content assist, but it worked fine with some simple DSLs I was playing with before starting work on the "real" DSL so I am sure it will be OK. As a newcomer to Xtext I found it fairly easy to get started, the only "mysterious" realm to me is all the MWE2 stuff which thankfully I haven't had to delve into to do my DSL work (except of course for this work-around). Thanks again for your help!

Report message to a moderator

Re: 64K limit on InternalParser [message #1068041 is a reply to message #1068040]

Thu, 11 July 2013 19:16

Christian Dietrich

Messages: 14665
Registered: July 2009

Senior Member

Hi,

dont be afraid from mwe, it is just a "i dont wont to compose java object graphs with xml" dsl.

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: 64K limit on InternalParser [message #1068045 is a reply to message #1068041]

Thu, 11 July 2013 19:22

Michael Card

Messages: 7
Registered: June 2013

Junior Member

OK. BTW, I just tried content assist and it is working fine, so the SplitClasses option did indeed fix whatever was being generated by the ANTLR 3.x parser embedded in Xtext.

Report message to a moderator

Re: 64K limit on InternalParser [message #1071072 is a reply to message #1068045]

Fri, 19 July 2013 09:34

Ingo Meyer

Messages: 162
Registered: July 2009

Senior Member

If you have a very big DSL classplitting does not help anymore.... even with both splitting params "fieldsPerClass" and "methodsPerClass" set to "50"

I just want to follow up here and asking if Xtext is planning to integrate ANTLR 4.1 in the future?

[Updated on: Fri, 19 July 2013 09:36]

Report message to a moderator

Re: 64K limit on InternalParser [message #1073160 is a reply to message #1071072]

Wed, 24 July 2013 08:41

Jan Koehnlein

Messages: 760
Registered: July 2009
Location: Hamburg

Senior Member

That's not planned. It would break a lot of existing API.

Am 19.07.13 11:34, schrieb Ingo Meyer:
> If you have a very big DSL classplitting does not help anymore.... even
> with both splitting params set to "50"
>
> I just want to follow up here and asking if Xtext is planning to
> integrate ANTLR 4.1 in the future?

--
Need professional support for Eclipse Modeling?
Go visit: http://xtext.itemis.com

---
Get professional support from the Xtext committers at www.typefox.io

Report message to a moderator

Previous Topic:	No EObjectDescription could be found in Scope
Next Topic:	migration guide to Xtext 2.4.2

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Fri Apr 19 21:09:56 GMT 2024

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter