Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » XText 2.12 blocking issues with Large Grammars(Workflow Errors block the ability to generate the infrastructure)
XText 2.12 blocking issues with Large Grammars [message #1770309] Wed, 09 August 2017 21:49 Go to next message
Joseph Landers is currently offline Joseph LandersFriend
Messages: 12
Registered: February 2017
Junior Member
Hi there,

I'm working with Xtext 2.12 to try and create a DSL with a fairly large grammar. At first, I wasn't able to generate the infrastructure at all due to jvm out of memory errors. I was able to fix this by increasing the heap size in the jvm to 2gb via the flag -Xms2g (1 gig wasn't enough).

Anyway, after this I was able to generate the grammar, but the resulting files had errors as follows:

    In my XxxxParser.java file - "The code of constructor x() is exceeding the 65535 bytes limit."
    and in my InternalXxxxParser.java file "Too many constants, the constant pool for InternalXxxxParser would exceed 65536 entries."


Reading a few other posts led me to find Bug 34992 , which I read and mostly understood. ;-) Based on the recommendations in that ticket, I tried adding the following to my workflow:
fragment = parser.antlr.XtextAntlrGeneratorFragment2 auto-inject {
    combinedGrammar = true // <-- I tried both true and false here.  No luck. 
    options = {
        classSplitting=true
        fieldsPerClass = "500" 
    }
}

Note, the writeup in that ticket calls for using both parser.antlr.XtextAntlrGeneratorFragment and parser.antlr.XtextAntlrUIGeneratorFragment. However, neither of those classes are available to use, and parser.antlr.XtextAntlrGeneratorFragment2 is the closest thing I could find. Is there a replacement for parser.antlr.XtextAntlrUIGeneratorFragment?

Anyway, after adding the fragment above, the workflow is now failing with the following error:
[ERROR]: GeneratorException: (Element: -UNKNOWN-; Reported by: XtextGenerator)
	 java.lang.IllegalStateException: The given EMF object already contains an adapter for CombinedGrammarMarker
	at org.eclipse.xtext.xtext.generator.parser.antlr.CombinedGrammarMarker.attachToEmfObject(CombinedGrammarMarker.java:66)
	at org.eclipse.xtext.xtext.generator.parser.antlr.XtextAntlrGeneratorFragment2.doGenerate(XtextAntlrGeneratorFragment2.java:148)
	at org.eclipse.xtext.xtext.generator.parser.antlr.AbstractAntlrGeneratorFragment2.generate(AbstractAntlrGeneratorFragment2.java:109)
	at org.eclipse.xtext.xtext.generator.CompositeGeneratorFragment2.generate(CompositeGeneratorFragment2.java:50)
	at org.eclipse.xtext.xtext.generator.XtextGenerator.invokeInternal(XtextGenerator.java:230)
	at org.eclipse.emf.mwe.core.lib.AbstractWorkflowComponent.invoke(AbstractWorkflowComponent.java:126)
	at org.eclipse.emf.mwe.core.lib.Mwe2Bridge.invoke(Mwe2Bridge.java:34)
	at org.eclipse.emf.mwe.core.lib.AbstractWorkflowComponent.invoke(AbstractWorkflowComponent.java:201)
	at org.eclipse.emf.mwe2.runtime.workflow.AbstractCompositeWorkflowComponent.invoke(AbstractCompositeWorkflowComponent.java:35)
	at org.eclipse.emf.mwe2.runtime.workflow.Workflow.run(Workflow.java:19)
	at org.eclipse.emf.mwe2.launch.runtime.Mwe2Runner.run(Mwe2Runner.java:102)
	at org.eclipse.emf.mwe2.launch.runtime.Mwe2Runner.run(Mwe2Runner.java:62)
	at org.eclipse.emf.mwe2.launch.runtime.Mwe2Runner.run(Mwe2Runner.java:52)
	at org.eclipse.emf.mwe2.launch.runtime.Mwe2Launcher.run(Mwe2Launcher.java:78)
	at org.eclipse.emf.mwe2.launch.runtime.Mwe2Launcher.main(Mwe2Launcher.java:36)


I don't fully understand the error, but since it's complaining about a duplicate adapter for "CombinedGrammarMarker" I tried twiddling the "combinedGrammar" flag in my config, but neither setting it to true nor false made any difference.

What do I have to do to get around these 64k errors?

    - In my xxxParser.java file - "The code of constructor x() is exceeding the 65535 bytes limit."
    - and in my InternalXxxParser.java file "Too many constants, the constant pool for InternalHtmlParser would exceed 65536 entries."


If using classSpliting = true is no longer supported, what else can I do? If this is still the correct way to get around these issues, what am I missing?

Thanks!
Re: XText 2.12 blocking issues with Large Grammars [message #1770310 is a reply to message #1770309] Wed, 09 August 2017 21:54 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 11390
Registered: July 2009
Senior Member
The right section to put that stuff
Is.

parserGenerator = {
...
}

You should play with other stuff like methods per class as well.

If this does not help it would be interesting to have
A (if not possible synthetic ) reproducible grammar


Need professional support for Xtext, Xpand, EMF?
Go to: http://xtext.itemis.com
Twitter : @chrdietrich
Blog : christiandietrich.wordpress.com
Re: XText 2.12 blocking issues with Large Grammars [message #1770313 is a reply to message #1770309] Wed, 09 August 2017 22:15 Go to previous messageGo to next message
Joseph Landers is currently offline Joseph LandersFriend
Messages: 12
Registered: February 2017
Junior Member
One quick update. I switched from using StandardLanguage to using XtextGeneratorLanguage in my workflow, and now the classSplitting=true option is working without throwing the exception. However, this only seems to solve one of my problems:

    Solved - In my InternalXxxParser.java file "Too many constants, the constant pool for InternalHtmlParser would exceed 65536 entries."
    Not Solved - In my XxxxParser.java file - "The code of constructor x() is exceeding the 65535 bytes limit."


Any advice how to get that constructor under control? I tried setting some of the other options in the XtextAntlrGeneratorFragment2:
fragment = parser.antlr.XtextAntlrGeneratorFragment2 {
    options = {
        classSplitting = true
        casesPerSpecialStateSwitch= "200"
        fieldsPerClass = "200"
        methodsPerClass = "200"
    }
}

but none of these options helped.
Re: XText 2.12 blocking issues with Large Grammars [message #1770314 is a reply to message #1770309] Wed, 09 August 2017 22:15 Go to previous messageGo to next message
Joseph Landers is currently offline Joseph LandersFriend
Messages: 12
Registered: February 2017
Junior Member
One quick update. I switched from using StandardLanguage to using XtextGeneratorLanguage in my workflow, and now the classSplitting=true option is working without throwing the exception. However, this only seems to solve one of my problems:

    Solved - In my InternalXxxParser.java file "Too many constants, the constant pool for InternalHtmlParser would exceed 65536 entries."
    Not Solved - In my XxxxParser.java file - "The code of constructor x() is exceeding the 65535 bytes limit."


Any advice how to get that constructor under control? I tried setting some of the other options in the XtextAntlrGeneratorFragment2:
fragment = parser.antlr.XtextAntlrGeneratorFragment2 {
    options = {
        classSplitting = true
        casesPerSpecialStateSwitch= "200"
        fieldsPerClass = "200"
        methodsPerClass = "200"
    }
}

but none of these options helped.
Re: XText 2.12 blocking issues with Large Grammars [message #1770315 is a reply to message #1770314] Wed, 09 August 2017 22:24 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 11390
Registered: July 2009
Senior Member
I don't know if the code in the generator has a splitting strategy for the constructor
You would have to digg into that code


Need professional support for Xtext, Xpand, EMF?
Go to: http://xtext.itemis.com
Twitter : @chrdietrich
Blog : christiandietrich.wordpress.com
Re: XText 2.12 blocking issues with Large Grammars [message #1770316 is a reply to message #1770315] Wed, 09 August 2017 22:30 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 11390
Registered: July 2009
Senior Member
can you give some hints how the big contructor looks like?
is it the one that populates the hashmap?


Need professional support for Xtext, Xpand, EMF?
Go to: http://xtext.itemis.com
Twitter : @chrdietrich
Blog : christiandietrich.wordpress.com

[Updated on: Wed, 09 August 2017 22:32]

Report message to a moderator

Re: XText 2.12 blocking issues with Large Grammars [message #1770318 is a reply to message #1770316] Wed, 09 August 2017 22:39 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 11390
Registered: July 2009
Senior Member
if it is the nameMappings map you should file a ticket at github.com/eclipse/xtext-core

i have no idea how to solve this

(depending on how much you are above the limit you may save some bytes by changing the grammar - using shorter names etc)


Need professional support for Xtext, Xpand, EMF?
Go to: http://xtext.itemis.com
Twitter : @chrdietrich
Blog : christiandietrich.wordpress.com
Re: XText 2.12 blocking issues with Large Grammars [message #1770329 is a reply to message #1770318] Thu, 10 August 2017 05:28 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 11390
Registered: July 2009
Senior Member
the following generation pattern would fix the issue

	@Override
	protected String getRuleName(AbstractElement element) {
		if (nameMappings == null) {
			nameMappings = new HashMap<AbstractElement, String>() {
				private static final long serialVersionUID = 1L;
				{
					fillMap0();
					fillMap1();
					fillMap2();
					fillMap3();
				}
				private void fillMap0() {
					put(grammarAccess.getGreetingAccess().getGroup(), "rule__Greeting__Group__0");
					put(grammarAccess.getModelAccess().getGreetingsAssignment(), "rule__Model__GreetingsAssignment");
					put(grammarAccess.getGreetingAccess().getName1Assignment_1(), "rule__Greeting__Name1Assignment_1");
				}
				private void fillMap1() {
					put(grammarAccess.getGreetingAccess().getName2Assignment_2(), "rule__Greeting__Name2Assignment_2");
					put(grammarAccess.getGreetingAccess().getName3Assignment_3(), "rule__Greeting__Name3Assignment_3");
					put(grammarAccess.getGreetingAccess().getName4Assignment_4(), "rule__Greeting__Name4Assignment_4");
				}
				private void fillMap2() {
					put(grammarAccess.getGreetingAccess().getName5Assignment_5(), "rule__Greeting__Name5Assignment_5");
					put(grammarAccess.getGreetingAccess().getName6Assignment_6(), "rule__Greeting__Name6Assignment_6");
					put(grammarAccess.getGreetingAccess().getName7Assignment_7(), "rule__Greeting__Name7Assignment_7");
				}
				private void fillMap3() {
					put(grammarAccess.getGreetingAccess().getName8Assignment_8(), "rule__Greeting__Name8Assignment_8");
					put(grammarAccess.getGreetingAccess().getName9Assignment_9(), "rule__Greeting__Name9Assignment_9");
					put(grammarAccess.getGreetingAccess().getName0Assignment_10(), "rule__Greeting__Name0Assignment_10");
				}
			};
		}
		return nameMappings.get(element);
	}


Need professional support for Xtext, Xpand, EMF?
Go to: http://xtext.itemis.com
Twitter : @chrdietrich
Blog : christiandietrich.wordpress.com
Re: XText 2.12 blocking issues with Large Grammars [message #1770379 is a reply to message #1770329] Thu, 10 August 2017 15:18 Go to previous messageGo to next message
Joseph Landers is currently offline Joseph LandersFriend
Messages: 12
Registered: February 2017
Junior Member
It is in-fact the constructor of the HashMap, and I've been manually using this exact pattern to manually fix it. I'll file a bug later today, when I have some time.

So, since we all seem to be on the same page, anyone care to point me at towards the code that generates this? I'd be happy to try and fix it myself, and submit a PR ;-)
Re: XText 2.12 blocking issues with Large Grammars [message #1770380 is a reply to message #1770329] Thu, 10 August 2017 15:18 Go to previous messageGo to next message
Joseph Landers is currently offline Joseph LandersFriend
Messages: 12
Registered: February 2017
Junior Member
It is in-fact the constructor of the HashMap, and I've been using this exact pattern to manually fix it after each workflow run. I'll file a bug later today, when I have some time.

So, since we all seem to be on the same page, anyone care to point me at towards the code that generates this? I'd be happy to try and fix it myself, and submit a PR ;-)
Re: XText 2.12 blocking issues with Large Grammars [message #1770381 is a reply to message #1770380] Thu, 10 August 2017 15:22 Go to previous messageGo to next message
Joseph Landers is currently offline Joseph LandersFriend
Messages: 12
Registered: February 2017
Junior Member
PS. I've found that breaking this into 2000 line functions seems to do the trick ;-)
Re: XText 2.12 blocking issues with Large Grammars [message #1770382 is a reply to message #1770381] Thu, 10 August 2017 15:24 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 11390
Registered: July 2009
Senior Member
Can you please file an issue

Need professional support for Xtext, Xpand, EMF?
Go to: http://xtext.itemis.com
Twitter : @chrdietrich
Blog : christiandietrich.wordpress.com
Re: XText 2.12 blocking issues with Large Grammars [message #1770396 is a reply to message #1770382] Thu, 10 August 2017 16:58 Go to previous messageGo to next message
Joseph Landers is currently offline Joseph LandersFriend
Messages: 12
Registered: February 2017
Junior Member
Issue filed. https://github.com/eclipse/xtext-core/issues/419
Re: XText 2.12 blocking issues with Large Grammars [message #1770397 is a reply to message #1770396] Thu, 10 August 2017 17:03 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 11390
Registered: July 2009
Senior Member
thank you

Need professional support for Xtext, Xpand, EMF?
Go to: http://xtext.itemis.com
Twitter : @chrdietrich
Blog : christiandietrich.wordpress.com
Re: XText 2.12 blocking issues with Large Grammars [message #1770407 is a reply to message #1770397] Thu, 10 August 2017 22:10 Go to previous messageGo to next message
Joseph Landers is currently offline Joseph LandersFriend
Messages: 12
Registered: February 2017
Junior Member
Thank you for your quick response. I see you've already submitted a PR with a fix for this issue. Thank you very much! :-)
Re: XText 2.12 blocking issues with Large Grammars [message #1770570 is a reply to message #1770407] Mon, 14 August 2017 15:06 Go to previous messageGo to next message
David BlackFriend
Messages: 33
Registered: June 2017
Member
Wow, this has been an interesting thread., thanks for posting it. I can see this kind of problems coming on me because I'm also manipulating a very large grammar and I've already had out-of-memory-errors :/
Re: XText 2.12 blocking issues with Large Grammars [message #1770626 is a reply to message #1770570] Tue, 15 August 2017 09:12 Go to previous message
Jens von Pilgrim is currently offline Jens von PilgrimFriend
Messages: 311
Registered: July 2009
Senior Member
We also have a rather large grammar (JavaScript with types) and a lot of fancy features. For that we had to "massage" the generation process quite a bit. If you are interested in that, have a look at

https://github.com/eclipse/n4js/blob/master/plugins/org.eclipse.n4js/src/org/eclipse/n4js/GenerateN4JS.mwe2

as a starting point. The customized generator stuff can be found at

https://github.com/eclipse/n4js/tree/master/releng/org.eclipse.n4js.releng.utils/src/org/eclipse/n4js/antlr

I have explained some of that stuff at an EclipseCon talk in 2013: https://www.youtube.com/watch?v=Xm-7aE1UMGY&feature=youtu.be

Cheers,
Jens
Previous Topic:Formatting Multiple Consecutive Keywords
Next Topic:Multiple Sort Contributions (or Comparators) in Outline
Goto Forum:
  


Current Time: Thu Nov 23 16:50:31 GMT 2017

Powered by FUDForum. Page generated in 0.03084 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software