Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Xtext Build and Indexing taking more time(Xtext index performance issue )
Xtext Build and Indexing taking more time [message #1832350] Thu, 17 September 2020 05:08 Go to next message
Shashi Mising name is currently offline Shashi Mising nameFriend
Messages: 47
Registered: August 2011
Member
Hello All,

When project build happen, indexing taking lot of time. build time increased, not allowed to work on the DSL until build completes.

The grammar size little big[1100 lines] and contains some cross reference rules.

Project also contains more number of DSLs[more than 3000 dsl files], so when build happen on this project it takes around 18-20 min for complete build
including [parsing+linking+validation+generation]. if i exclude generation it reduces for 2 min.

Need some ideas/best methodologies to improvise the build and index performance.

Thank you

Regards
Shashi
Re: Xtext Build and Indexing taking more time [message #1832351 is a reply to message #1832350] Thu, 17 September 2020 05:11 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13645
Registered: July 2009
Senior Member
hi,

you should first start profiling to see "where" the performance is lost.
is it calculation or is it writing file

(e.g. writing thousands of files to a network drive might become terrible slow and there is nothing to optimize there)

same for reading files

=> measure where the time is spent


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

[Updated on: Thu, 17 September 2020 05:11]

Report message to a moderator

Re: Xtext Build and Indexing taking more time [message #1832353 is a reply to message #1832351] Thu, 17 September 2020 05:44 Go to previous messageGo to next message
Shashi Mising name is currently offline Shashi Mising nameFriend
Messages: 47
Registered: August 2011
Member
Hello Christain,

Thanks for your suggestion.

No not writing any files to network drive and not reading any resources from network drive.

Did a profiling and identified that the doUpdate method of ClusteringBuilderState is taking more time.

Thank you

Re: Xtext Build and Indexing taking more time [message #1832357 is a reply to message #1832353] Thu, 17 September 2020 06:06 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13645
Registered: July 2009
Senior Member
doUpdate is all
- parsing
- linking
-validation

we need to know the subtaks like

- writeNewResourceDescriotion
- updateMarkers
- resolveLazyCrossReferences


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Xtext Build and Indexing taking more time [message #1832361 is a reply to message #1832357] Thu, 17 September 2020 06:58 Go to previous messageGo to next message
Shashi Mising name is currently offline Shashi Mising nameFriend
Messages: 47
Registered: August 2011
Member
Hello Christain,

Thank you very much.

Number of DSL files in the current project = 1850
The writeNewResourceDescriotion method is taking around 32 seconds.

Aafter this there is a execution step for

- resolveLazyCrossReferences
- updateMarkers

// Step 6: Iteratively got through the queue. For each resource, create a new resource description and queue all depending
// resources that are not yet in the delta. Validate resources. Do this in chunks.

Both the methods are not taking much time to process the individual resource.

But there is a while loop to process all the resources for crossReferences resolution and marker update, so this loop actually taking very long time.

The below loop taking more time, it takes more than 10 min.
Loop :
Queue<URI> queue = buildData.getURIQueue();
while (!queue.isEmpty()) {
- resolveLazyCrossReferences
- updateMarkers
}


Thank you
Re: Xtext Build and Indexing taking more time [message #1832362 is a reply to message #1832361] Thu, 17 September 2020 07:04 Go to previous messageGo to next message
Ed Willink is currently offline Ed WillinkFriend
Messages: 6880
Registered: July 2009
Senior Member
Hi

I find that the Xtext indexing is very enthusiastic about indexing anything that might just possibly be a model, regardless of whether there is any active client that could use it.

I therefore minimize the number of projects that have an Xtext nature by migrating all the Xtext mwe2/build functionality into a separate ...build plugin. This also has the benefit of reducing the footprint of the delivered Xtext editor.

But don't let me discourage your profiling. It seems that you may be close to a repro for index/update performance issues that have been outstanding for far too long.

Regards

Ed Willink
Re: Xtext Build and Indexing taking more time [message #1832364 is a reply to message #1832362] Thu, 17 September 2020 07:15 Go to previous messageGo to next message
Shashi Mising name is currently offline Shashi Mising nameFriend
Messages: 47
Registered: August 2011
Member
Hello Ed,

Thank you,

What does it mean. could you please elaborate more on your below statement.

Minimize the number of projects that have an Xtext nature by migrating all the Xtext mwe2/build functionality into a separate ...build plugin.

How that actually helps for performance improvement? any examples will definitely help for better understanding.


Thank you
Re: Xtext Build and Indexing taking more time [message #1832394 is a reply to message #1832364] Thu, 17 September 2020 11:55 Go to previous messageGo to next message
Shashi Mising name is currently offline Shashi Mising nameFriend
Messages: 47
Registered: August 2011
Member
Hi,

Any inputs/suggestions are highly appreciated.

Thank you
Re: Xtext Build and Indexing taking more time [message #1832396 is a reply to message #1832394] Thu, 17 September 2020 12:02 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13645
Registered: July 2009
Senior Member
again: can you provide actual measured data on what on the build takes time.
there can be

- parallel parsing
- reduce number of index object / tree traversing on indexing
- fix bad O(nsquared) validation algos
- fix bad scoping
etc


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Xtext Build and Indexing taking more time [message #1832475 is a reply to message #1832396] Fri, 18 September 2020 13:55 Go to previous message
Ed Willink is currently offline Ed WillinkFriend
Messages: 6880
Registered: July 2009
Senior Member
Hi

See https://bugs.eclipse.org/bugs/show_bug.cgi?id=420710#c2 and https://bugs.eclipse.org/bugs/show_bug.cgi?id=486083 and its see alsoes for some history on this problem. The problem seems so old that a number of Bugzillas have been auto-CLOSED WONTFIX.

My attempts to debug the build recursions failed because I couldn't understand the code, or install a debuggable version to instrument usefully or find the developer's debug ".options".

I therefore gave up and adjusted my working practices to minimize the pain caused by the defective index, that I never use, and/or defective auto-builder that should only build once. I also minimize my use of Xtend to just M2T StringTemplates..

You can see my restructuring in the org.eclipse.ocl.examples.build plugin that houses the MWE2 build procedures and dependencies for the five/six org.eclipse.ocl.xtext.... editors. The editor plugins do NOT have an Xtext nature. Test plugins do NOT have an Xtext nature to prevent *.uml files burning time even though I know of no Xtext-based UML tooling that requires an index. Only org.eclipse.ocl.examples.build has an Xtext nature which much reduces the opportunities for Xtext build loops. Beware of checking out org.eclipse.emf.core since that now has an Xtext nature.

My observations suggested that the dreadful performance was intermittent, perhaps caused by accidental GIT checkout timing, perhaps caused by indeterminate Set ordering in worklists. If you have a solid repro please put it forward for investigation.

The solution might be as trivial as adding a validation warning to alert the user to erroneous Xtext/Java build nature ordering. Since Xtext depends on the Index which depends on Java, I never know what the correct order is or even if there is a correct order. Perhaps the bug is differently a failure of a builder to detect nothing was actually built and so that no timestamps are to be changed and so provoke a loop.

Regards

Ed Willink
Previous Topic:Xtend template expressions are not resolved using Xtend Maven plugin
Next Topic:Model-Transformation: How to resolve custom generated Proxy-Objects.
Goto Forum:
  


Current Time: Sat Oct 31 07:20:10 GMT 2020

Powered by FUDForum. Page generated in 0.37708 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top