Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Reduce resource consumption in project indexing(Scope, Indexing)
Reduce resource consumption in project indexing [message #1705950] Thu, 20 August 2015 11:29 Go to next message
Anakreontas Mentis is currently offline Anakreontas MentisFriend
Messages: 85
Registered: October 2014
Member
In a project with approximately 700 source files of my DSL, indexing takes a lot of resources. When the memory is exhausted, eclipse is almost idle (10% CPU) and I assume it is waiting for the garbage collector to release whatever memory it can. Until eclipse uses the available memory CPU usage is quite high. In the end, the index is never built completely. In projects with fewer files the index is built in reasonable time.

I provided a specific IDefaultResourceDescriptionStrategy implementation that indexes only objects that might be referenced by other elements of the grammar. This helped a bit but not much. I think the problem is that although my DSL uses includes and *not* namespaces, all files are indexed regardless if another file includes them.

How could I prevent the indexing of files not included by any other file? Is this the responsibility of the Scope? My Scope implementation extends AbstractDeclarativeScopeProvider. Should I extend some other class instead?

Thank you in advance for your help.
Re: Reduce resource consumption in project indexing [message #1705951 is a reply to message #1705950] Thu, 20 August 2015 11:42 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13642
Registered: July 2009
Senior Member
Hi,

this is not possible. in the indexing Phase the files no not know what is left or right of them.
what you may do is to enable clustinging when indexing (DynamicResourceClusteringPolicy as IResourceClusteringPolicy )


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Reduce resource consumption in project indexing [message #1705956 is a reply to message #1705951] Thu, 20 August 2015 12:18 Go to previous messageGo to next message
Stefan Oehme is currently offline Stefan OehmeFriend
Messages: 159
Registered: April 2010
Location: Kiel
Senior Member

Hey Anakreontas,

first of all: Why don't you use namespaces? They are both more efficient and more user friendly than uri imports.

700 files doesn't sound like that much. Have you actually profiled to see where the time is spent?

How many elements are in the index? How much memory do you give to the workbench?

Also note that the index only needs to contain elements that are *globally* referencable. For example: For Java you don't need to index fields and methods, because they are always referenced in the context of a type. So only top-level types need to be indexed.

Cheers,
Stefan
Re: Reduce resource consumption in project indexing [message #1705979 is a reply to message #1705956] Thu, 20 August 2015 16:03 Go to previous messageGo to next message
Anakreontas Mentis is currently offline Anakreontas MentisFriend
Messages: 85
Registered: October 2014
Member
Hello Stefan.

The DSL is quite old and I can't change the design.

Could you point to documentation about querying the index for the number of elements it contains?
I launch the workbench with these parameters "-Xms40m -Xmx4048m -XX:MaxPermSize=1024m".
Re: Reduce resource consumption in project indexing [message #1705981 is a reply to message #1705979] Thu, 20 August 2015 16:15 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13642
Registered: July 2009
Senior Member
You can use the open model element dialog for that

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Reduce resource consumption in project indexing [message #1705985 is a reply to message #1705981] Thu, 20 August 2015 16:48 Go to previous messageGo to next message
Anakreontas Mentis is currently offline Anakreontas MentisFriend
Messages: 85
Registered: October 2014
Member
It reports 11.010 objects. The index building was not complete but I opened the dialog multiple times and was reporting the same number.

I am trying to profile eclipse running the plugin. Got a snapshot of the memory with jmap but jhat run out of memory when processing the file (1.3 GB). jmap -histo:live reported that most of the memory was allocated for LeafNode and CompositeNode instances. I am running jhat again with more memory to get better results on memory usage,
Re: Reduce resource consumption in project indexing [message #1705987 is a reply to message #1705985] Thu, 20 August 2015 17:23 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13642
Registered: July 2009
Senior Member
did you try to use

package org.xtext.example.mydsl.ui;

import org.eclipse.xtext.resource.clustering.DynamicResourceClusteringPolicy;
import org.eclipse.xtext.resource.clustering.IResourceClusteringPolicy;

import com.google.inject.Binder;
import com.google.inject.Module;

@SuppressWarnings("restriction")
public class OverridingGuiceModule implements Module {

	@Override
	public void configure(Binder binder) {
		binder.bind(IResourceClusteringPolicy.class).to(DynamicResourceClusteringPolicy.class);

	}

}



 <extension
        point="org.eclipse.xtext.ui.shared.overridingGuiceModule">
     <module
           class="org.xtext.example.mydsl.ui.OverridingGuiceModule">
     </module>
  </extension>


package org.xtext.example.mydsl.ui;

import org.eclipse.ui.plugin.AbstractUIPlugin;
import org.eclipse.xtext.resource.clustering.DynamicResourceClusteringPolicy;
import org.eclipse.xtext.resource.clustering.IResourceClusteringPolicy;

@SuppressWarnings("restriction")
public class MyDslUiModule extends org.xtext.example.mydsl.ui.AbstractMyDslUiModule {
	public MyDslUiModule(AbstractUIPlugin plugin) {
		super(plugin);
	}
	
	public Class<? extends IResourceClusteringPolicy> bindIResourceClusteringPolicy() {
		return DynamicResourceClusteringPolicy.class;
	}
}

as i proposed


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Reduce resource consumption in project indexing [message #1706060 is a reply to message #1705987] Fri, 21 August 2015 13:26 Go to previous messageGo to next message
Anakreontas Mentis is currently offline Anakreontas MentisFriend
Messages: 85
Registered: October 2014
Member
Hello Chrisitan.

I tried that but still the indexing would not complete. The DSL uses file extensions as a convention for files that are meant to be included. Myy implementation of DefaultResourceDescriptionStrategy returns false in the createEObjectDescriptions and createReferenceDescriptions methods for files that have a different extension. Since file extensions are just a convention I did not enforce this rule before. Now indexing is quite faster because it has to process fewer files. However, whenever the user edits a file and saves it, the index start building again and the user is interrupted from continuing to edit the file because eclipse either becomes unresponsive or very sluggish for 1-2 seconds.

Would you suggest any resources about profiling xtext languages or eclipse plugins? I need to investigate which parts of the code consumes most resources.
Re: Reduce resource consumption in project indexing [message #1706081 is a reply to message #1706060] Fri, 21 August 2015 22:06 Go to previous messageGo to next message
Anakreontas Mentis is currently offline Anakreontas MentisFriend
Messages: 85
Registered: October 2014
Member
Perhaps I have misunderstood the role of the Scope.

My scope implementation derives from AbstractDeclarativeScopeProvider. I replaced it with an empty descendant of AbstractDeclarativeScopeProvider and surprisingly almost everything worked fine. I could for example jump to the implementation of function or the declaration of variables, a service I thought was possible because of my Scope implementation. Moreover, memory consumption is fairly reasonable and Eclipse is responsive.

Perhaps a specialized scope implementation is needed for namespace based resolution of methods and variables.
Re: Reduce resource consumption in project indexing [message #1706084 is a reply to message #1706081] Fri, 21 August 2015 22:33 Go to previous messageGo to next message
Stefan Oehme is currently offline Stefan OehmeFriend
Messages: 159
Registered: April 2010
Location: Kiel
Senior Member

Xtext has quite nice out of the box behavior for scoping. So it's normal that many things work even with no special scope implementation =)

It seems your scope provider contains some very inefficient code. For more suggestions we'd need to see the full Grammar and ScopeProvider.
Re: Reduce resource consumption in project indexing [message #1818260 is a reply to message #1706084] Wed, 11 December 2019 06:51 Go to previous messageGo to next message
Renganathan S is currently offline Renganathan SFriend
Messages: 1
Registered: December 2019
Junior Member
Hello Anakreontas ,
I am also facing same kind of memory issue. Can you tell me more about empty descendant of AbstractDeclarativeScopeProvider?
Re: Reduce resource consumption in project indexing [message #1818336 is a reply to message #1818260] Thu, 12 December 2019 17:28 Go to previous message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13642
Registered: July 2009
Senior Member
Simply have a look at the scope provider you have and the one the new xtext project wizard creates

Are you sure you have a subclass of abstractdeclarativescopeprovider

If not you won't profit

If you might change it the way the wizard creates it for new projects

(You have to customize getScope and ifelse
Instead of using scope_ methods

Please also measure where your memory is actually spent


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Previous Topic:Xtext / Xtend 2.20 release
Next Topic:XbaseCompiler does not generate lambdas when running from xtext-maven-plugin
Goto Forum:
  


Current Time: Wed Oct 28 18:01:20 GMT 2020

Powered by FUDForum. Page generated in 0.02388 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top