Eclipse Community Forums: TMF (Xtext) » Xtext Build and Indexing taking more time

Help

Home

Home » Modeling » TMF (Xtext) » Xtext Build and Indexing taking more time(Xtext index performance issue )

Show: Today's Messages :: Show Polls :: Message Navigator

Xtext Build and Indexing taking more time [message #1832350]

Thu, 17 September 2020 05:08

Shashi Mising name

Messages: 47
Registered: August 2011

Member

Hello All,

When project build happen, indexing taking lot of time. build time increased, not allowed to work on the DSL until build completes.

The grammar size little big[1100 lines] and contains some cross reference rules.

Project also contains more number of DSLs[more than 3000 dsl files], so when build happen on this project it takes around 18-20 min for complete build
including [parsing+linking+validation+generation]. if i exclude generation it reduces for 2 min.

Need some ideas/best methodologies to improvise the build and index performance.

Thank you

Regards
Shashi

Report message to a moderator

Re: Xtext Build and Indexing taking more time [message #1832351 is a reply to message #1832350]

Thu, 17 September 2020 05:11

Christian Dietrich

Messages: 13709
Registered: July 2009

Senior Member

hi,

you should first start profiling to see "where" the performance is lost.
is it calculation or is it writing file

(e.g. writing thousands of files to a network drive might become terrible slow and there is nothing to optimize there)

same for reading files

=> measure where the time is spent

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

[Updated on: Thu, 17 September 2020 05:11]

Report message to a moderator

Re: Xtext Build and Indexing taking more time [message #1832353 is a reply to message #1832351]

Thu, 17 September 2020 05:44

Shashi Mising name

Messages: 47
Registered: August 2011

Member

Hello Christain,

Thanks for your suggestion.

No not writing any files to network drive and not reading any resources from network drive.

Did a profiling and identified that the doUpdate method of ClusteringBuilderState is taking more time.

Thank you

Report message to a moderator

Re: Xtext Build and Indexing taking more time [message #1832357 is a reply to message #1832353]

Thu, 17 September 2020 06:06

Christian Dietrich

Messages: 13709
Registered: July 2009

Senior Member

doUpdate is all
- parsing
- linking
-validation

we need to know the subtaks like

- writeNewResourceDescriotion
- updateMarkers
- resolveLazyCrossReferences

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: Xtext Build and Indexing taking more time [message #1832361 is a reply to message #1832357]

Thu, 17 September 2020 06:58

Shashi Mising name

Messages: 47
Registered: August 2011

Member

Hello Christain,

Thank you very much.

Number of DSL files in the current project = 1850
The writeNewResourceDescriotion method is taking around 32 seconds.

Aafter this there is a execution step for

- resolveLazyCrossReferences
- updateMarkers

// Step 6: Iteratively got through the queue. For each resource, create a new resource description and queue all depending
// resources that are not yet in the delta. Validate resources. Do this in chunks.

Both the methods are not taking much time to process the individual resource.

But there is a while loop to process all the resources for crossReferences resolution and marker update, so this loop actually taking very long time.

The below loop taking more time, it takes more than 10 min.
Loop :
Queue<URI> queue = buildData.getURIQueue();
while (!queue.isEmpty()) {
- resolveLazyCrossReferences
- updateMarkers
}

Thank you

Report message to a moderator

Re: Xtext Build and Indexing taking more time [message #1832362 is a reply to message #1832361]

Thu, 17 September 2020 07:04

Ed Willink

Messages: 6985
Registered: July 2009

Senior Member

Hi

I find that the Xtext indexing is very enthusiastic about indexing anything that might just possibly be a model, regardless of whether there is any active client that could use it.

I therefore minimize the number of projects that have an Xtext nature by migrating all the Xtext mwe2/build functionality into a separate ...build plugin. This also has the benefit of reducing the footprint of the delivered Xtext editor.

But don't let me discourage your profiling. It seems that you may be close to a repro for index/update performance issues that have been outstanding for far too long.

Regards

Ed Willink

Report message to a moderator

Re: Xtext Build and Indexing taking more time [message #1832364 is a reply to message #1832362]

Thu, 17 September 2020 07:15

Shashi Mising name

Messages: 47
Registered: August 2011

Member

Hello Ed,

Thank you,

What does it mean. could you please elaborate more on your below statement.

Minimize the number of projects that have an Xtext nature by migrating all the Xtext mwe2/build functionality into a separate ...build plugin.

How that actually helps for performance improvement? any examples will definitely help for better understanding.

Thank you

Report message to a moderator

Re: Xtext Build and Indexing taking more time [message #1832394 is a reply to message #1832364]

Thu, 17 September 2020 11:55

Shashi Mising name

Messages: 47
Registered: August 2011

Member

Hi,

Any inputs/suggestions are highly appreciated.

Thank you

Report message to a moderator

Re: Xtext Build and Indexing taking more time [message #1832396 is a reply to message #1832394]

Thu, 17 September 2020 12:02

Christian Dietrich

Messages: 13709
Registered: July 2009

Senior Member

again: can you provide actual measured data on what on the build takes time.
there can be

- parallel parsing
- reduce number of index object / tree traversing on indexing
- fix bad O(nsquared) validation algos
- fix bad scoping
etc

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: Xtext Build and Indexing taking more time [message #1832475 is a reply to message #1832396]

Fri, 18 September 2020 13:55

Ed Willink

Messages: 6985
Registered: July 2009

Senior Member

Hi

See https://bugs.eclipse.org/bugs/show_bug.cgi?id=420710#c2 and https://bugs.eclipse.org/bugs/show_bug.cgi?id=486083 and its see alsoes for some history on this problem. The problem seems so old that a number of Bugzillas have been auto-CLOSED WONTFIX.

My attempts to debug the build recursions failed because I couldn't understand the code, or install a debuggable version to instrument usefully or find the developer's debug ".options".

I therefore gave up and adjusted my working practices to minimize the pain caused by the defective index, that I never use, and/or defective auto-builder that should only build once. I also minimize my use of Xtend to just M2T StringTemplates..

You can see my restructuring in the org.eclipse.ocl.examples.build plugin that houses the MWE2 build procedures and dependencies for the five/six org.eclipse.ocl.xtext.... editors. The editor plugins do NOT have an Xtext nature. Test plugins do NOT have an Xtext nature to prevent *.uml files burning time even though I know of no Xtext-based UML tooling that requires an index. Only org.eclipse.ocl.examples.build has an Xtext nature which much reduces the opportunities for Xtext build loops. Beware of checking out org.eclipse.emf.core since that now has an Xtext nature.

My observations suggested that the dreadful performance was intermittent, perhaps caused by accidental GIT checkout timing, perhaps caused by indeterminate Set ordering in worklists. If you have a solid repro please put it forward for investigation.

The solution might be as trivial as adding a validation warning to alert the user to erroneous Xtext/Java build nature ordering. Since Xtext depends on the Index which depends on Java, I never know what the correct order is or even if there is a correct order. Perhaps the bug is differently a failure of a builder to detect nothing was actually built and so that no timestamps are to be changed and so provoke a loop.

Regards

Ed Willink

Report message to a moderator

Previous Topic:	Xtend template expressions are not resolved using Xtend Maven plugin
Next Topic:	Model-Transformation: How to resolve custom generated Proxy-Objects.

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Mon Jan 18 01:08:13 GMT 2021

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter