Eclipse Community Forums: TMF (Xtext) » Exclude generated JAR files from indexing

Home » Modeling » TMF (Xtext) » Exclude generated JAR files from indexing(JAR files containing DSL are added to index, would like to filter out)

Show: Today's Messages :: Show Polls :: Message Navigator

Exclude generated JAR files from indexing [message #1774706]

Wed, 18 October 2017 21:30

Oleg Cohen

Messages: 28
Registered: August 2016

Junior Member

Greetings,

I have a number of Xtext projects that depend on each other. These projects have JAR files that are built by a custom builder and put into a generated folder.

I implemented MyDslResourceServiceProvider and MyDslResourceUIServiceProvider, but they don't even get the JAR file resources in the canHandle method.

Is there a way to filter out either all JAR files or the folder where they reside? I also suspect that the overall classpath is being scanned. I don't need that added to the index so I can bypass the entire classpath or individual JARs.

I would appreciate guidance on the subject!

Thank you,
Oleg

Report message to a moderator

Re: Exclude generated JAR files from indexing [message #1774717 is a reply to message #1774706]

Thu, 19 October 2017 04:38

Christian Dietrich

Messages: 14062
Registered: July 2009

Senior Member

What do you mean by indexed?

Your resource service provider gets the files with your file extensions
So I wonder if the are reallly not called ....????
Others get files with their file extensions
Class,files are handled by Xtext itself

So which of this cases do you want to exclude?

Besides this have a look e.g. at https://github.com/eclipse/xtext-eclipse/blob/242b5fcc83f4697f6381f9264f710773e8435ce8/org.eclipse.xtext.ui/src/org/eclipse/xtext/ui/containers/JavaProjectsStateHelper.java

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

[Updated on: Thu, 19 October 2017 04:39]

Report message to a moderator

Re: Exclude generated JAR files from indexing [message #1774771 is a reply to message #1774717]

Thu, 19 October 2017 13:48

Oleg Cohen

Messages: 28
Registered: August 2016

Junior Member

Thank you for answering and the link!

In my case I have two DSL/Xbase driven projects: ProjectA and ProjectB.

The OSGi framework I am using (bndtools) creates a JAR file for each project in the folder called generated. The folder is at the top level of each projects. So, ProjectA has ./generated/ProjectA.jar and ProjectB has ./generated/ProjectB.jar.

ProjectB uses services from ProjectA, so in its classpath it refers directly to ProjectA/generated/ProjectA.jar.

Based on what I see in canHandle() method (DSL and UI both), the ProjectA DSL is found multiple times and one of them points to the JAR file (ProjectA/generated/ProjectA.jar). In general it is not an issue, but on Windows, it seems the fact that Xtext looked at the JAR and discovered ProjectA's DSL model in it, locks the file and my OSGi framework fails to remove and rebuild this JAR on subsequent project builds and rebuilds. I am not sure if this locking issue has anything to do with Xtext or Eclipse's JDT. I am trying to work around the issue.

My workaround is to have Xtext skip the JAR file and not even look in it for the DSL files.

My options are:
- Not look in JARs at all as I am only relying on src level DSL
- Not look for JARs in a particular location (these are in a folder called generated).

Thank you,
Oleg

Report message to a moderator

Re: Exclude generated JAR files from indexing [message #1774774 is a reply to message #1774771]

Thu, 19 October 2017 13:55

Christian Dietrich

Messages: 14062
Registered: July 2009

Senior Member

have a look at JavaProjectsStateHelper / Storage2UriMapperJavaImpl

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: Exclude generated JAR files from indexing [message #1774775 is a reply to message #1774774]

Thu, 19 October 2017 13:56

Christian Dietrich

Messages: 14062
Registered: July 2009

Senior Member

but: why is the jars on the classpath?

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: Exclude generated JAR files from indexing [message #1774886 is a reply to message #1774775]

Fri, 20 October 2017 19:48

Oleg Cohen

Messages: 28
Registered: August 2016

Junior Member

Thank you! I was able to extend JavaProjectsStateHelper and exclude the generated JAR files, but it didn't help yet. So, I am still looking for a solution.

The reason that the JARs are there is because that is how the OSGi framework we are using operates (BndTools). It creates and re-creates the bundle JARs on the fly via a custom builder. These JARs are then added to the classpath via a custom container.

I added logic in the getPackageFragmentRootHandles() method to exclude certain jars and not return them in the list. The question is whether the JAR files I am excluding have already been used/locked?

Report message to a moderator

Re: Exclude generated JAR files from indexing [message #1774923 is a reply to message #1774886]

Sat, 21 October 2017 07:42

Christian Dietrich

Messages: 14062
Registered: July 2009

Senior Member

sorry i still cant follow you.
what exactly do you want to prevent? the jar from being looked at, the jar from beeing on the classpath, the jar from beeing on the index?

why not implement

e.g.

class CustomResourceUIServiceProvider extends DefaultResourceUIServiceProvider {

@Inject
new(IResourceServiceProvider delegate) {
super(delegate)
}

override canHandle(URI uri, IStorage storage) {
println(uri)
if (uri.isArchive) {
return false
}
super.canHandle(uri, storage)
}

}

(you know you need to remove and readd the jar to get rid of them in the index?)

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

[Updated on: Sat, 21 October 2017 08:17]

Report message to a moderator

Re: Exclude generated JAR files from indexing [message #1775123 is a reply to message #1774923]

Wed, 25 October 2017 01:19

Oleg Cohen

Messages: 28
Registered: August 2016

Junior Member

This is very helpful, thank you! I have implemented what you suggested, but I am still experiencing problem. I have more info to share to clarify.

My projects have a directory called generated where are JAR files I would like to exclude from the cache that is built in the core Xtext plugin.

Here is what I see:

There is a module called SharedContributionWithJDT.

It binds components in the following way:

binder.bind(IEagerContribution.class).to(JavaCoreListenerRegistrar.class);

binder.bind(IStorage2UriMapperJdtExtensions.class).to(Storage2UriMapperJavaImpl.class);
binder.bind(IStorage2UriMapperContribution.class).to(Storage2UriMapperJavaImpl.class);
binder.bind(Storage2UriMapperJavaImpl.class).in(Scopes.SINGLETON);

Upon startup JavaCoreListenerRegistrar calls storage2UriMapperJavaImpl.asyncInitializeCache();

As part of this process the JAR files in the generated folder are opened to have elements added to the cache. Eclipse holds a RW lock on these JAR file in Windows and my project rebuild/redeploy to an OSGi container fails.

I am wondering if I could bind my own ExtendedStorage2UriMapperJavaImpl.class that would override isRejected() method to allow me also exclude the generated folders?

So far I was only able to override the entire XtextBuilder, but not sure how it might help ...

I would really appreciate guidance on this ... And if I cna provide more info I would be happy to.

Best,
Oleg

Report message to a moderator

Re: Exclude generated JAR files from indexing [message #1775127 is a reply to message #1775123]

Wed, 25 October 2017 05:26

Christian Dietrich

Messages: 14062
Registered: July 2009

Senior Member

of course you can do that but it has implications since the mapper is used by all dsls in your target eclipse/rcp
thus i wonder if you can do the same by different means like eclipse resource filters https://help.eclipse.org/luna/index.jsp?topic=%2Forg.eclipse.platform.doc.user%2Fconcepts%2Fresourcefilters.htm
(you have to test that)

here is the hacky solution

<extension
       point="org.eclipse.xtext.ui.shared.overridingGuiceModule">
    <module
          class="org.xtext.example.mydsl5.ui.MyDslOverridingModule">
    </module>
 </extension>

package org.xtext.example.mydsl5.ui;

import org.eclipse.xtext.ui.shared.contribution.ISharedStateContributionRegistry;

import com.google.inject.Binder;
import com.google.inject.Module;
import com.google.inject.Scopes;

public class MyDslOverridingModule implements Module{

	@Override
	public void configure(Binder binder) {
		 binder.bind(ISharedStateContributionRegistry.class)
         .to(MyDslSharedStateContributionRegistryImpl.class)
         .in(Scopes.SINGLETON);
	}

}

package org.xtext.example.mydsl5.ui;

import org.eclipse.xtext.ui.shared.internal.SharedContributionWithJDT;
import org.eclipse.xtext.ui.shared.internal.SharedStateContributionRegistryImpl;

import com.google.inject.Inject;
import com.google.inject.Injector;
import com.google.inject.Module;
import com.google.inject.util.Modules;

@SuppressWarnings("restriction")
public class MyDslSharedStateContributionRegistryImpl extends SharedStateContributionRegistryImpl {

	@Inject
	public MyDslSharedStateContributionRegistryImpl(Injector injector) {
		super(injector);
	}

	@Override
	protected Module getWrappedModule(Module childModule) {
		if (childModule instanceof SharedContributionWithJDT) {
			return super.getWrappedModule(
					Modules.override(childModule).with(new CustomSharedContributionWithJDTModule()));
		}
		return super.getWrappedModule(childModule);
	}

}

package org.xtext.example.mydsl5.ui;

import org.eclipse.xtext.ui.resource.IStorage2UriMapperContribution;
import org.eclipse.xtext.ui.resource.IStorage2UriMapperJdtExtensions;

import com.google.inject.Binder;
import com.google.inject.Module;
import com.google.inject.Scopes;

public class CustomSharedContributionWithJDTModule implements Module {

	@Override
	public void configure(Binder binder) {
		binder.install(new Module() {

			@Override
			public void configure(Binder binder) {
				
				binder.bind(IStorage2UriMapperJdtExtensions.class).to(MyDslStorage2UriMapperJavaImpl.class);
				binder.bind(IStorage2UriMapperContribution.class).to(MyDslStorage2UriMapperJavaImpl.class);
binder.bind(Storage2UriMapperJavaImpl.class).to(MyDslStorage2UriMapperJavaImpl.class);				binder.bind(MyDslStorage2UriMapperJavaImpl.class).in(Scopes.SINGLETON); 
			}
			
		});
		
	}

}

package org.xtext.example.mydsl5.ui;

import org.eclipse.core.resources.IFolder;
import org.eclipse.xtext.ui.resource.Storage2UriMapperJavaImpl;

public class MyDslStorage2UriMapperJavaImpl extends Storage2UriMapperJavaImpl {
	
	@Override
	public boolean isRejected(IFolder folder) {
		System.err.println("folder: " + folder);
		return super.isRejected(folder);
	}

}

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

[Updated on: Sat, 28 October 2017 01:24]

Report message to a moderator

Re: Exclude generated JAR files from indexing [message #1775338 is a reply to message #1775127]

Fri, 27 October 2017 20:10

Oleg Cohen

Messages: 28
Registered: August 2016

Junior Member

Thank you for suggesting the solution!

I have tried the resource route and it completely removes the JARs from eclipse and I can't use them as classpath dependencies.

I implemented the override you suggested. I realized that in order to properly skip the JAR files in my /generated folders I need to override the logic in the updateCache() method of Storage2UriMapperJavaImpl. The method is private. I managed to do it by copying code into my class.

This is what I did:

boolean isCachable = root.isArchive() || root.isExternal();

if (isCachable && root.getPath().toString().contains("/generated/")) {

isCachable = false;
}

All this worked well, except right after my code is called I see that another cache building code runs:

at org.eclipse.xtext.ui.resource.Storage2UriMapperJavaImpl.getCachedData(Storage2UriMapperJavaImpl.java:251)
at org.eclipse.xtext.ui.resource.Storage2UriMapperJavaImpl.updateCache(Storage2UriMapperJavaImpl.java:466)
at org.eclipse.xtext.ui.resource.Storage2UriMapperJavaImpl.access$6(Storage2UriMapperJavaImpl.java:459)
at org.eclipse.xtext.ui.resource.Storage2UriMapperJavaImpl$4.run(Storage2UriMapperJavaImpl.java:655)
at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2240)
at org.eclipse.core.internal.resources.Workspace.run(Workspace.java:2267)
at org.eclipse.xtext.ui.resource.Storage2UriMapperJavaImpl.doInitializeCache(Storage2UriMapperJavaImpl.java:667)
at org.eclipse.xtext.ui.resource.Storage2UriMapperJavaImpl.access$2(Storage2UriMapperJavaImpl.java:646)
at org.eclipse.xtext.ui.resource.Storage2UriMapperJavaImpl$3.run(Storage2UriMapperJavaImpl.java:635)
at java.lang.Thread.run(Thread.java:745)

And this again grabs the JAR files and locks them on Windows. I have two questions:

1. Where is this Storage2UriMapperJavaImpl instance coming from?

2. Is there a logic within Storage2UriMapperJavaImpl that leaves files opened in Windows after opening and traversing the content?

Thank you!
Oleg

Report message to a moderator

Re: Exclude generated JAR files from indexing [message #1775345 is a reply to message #1775338]

Sat, 28 October 2017 00:22

Christian Dietrich

Messages: 14062
Registered: July 2009

Senior Member

Would be nice if you can share something that lets me repo this in two instead of thirty minute

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: Exclude generated JAR files from indexing [message #1775347 is a reply to message #1775345]

Sat, 28 October 2017 01:23

Christian Dietrich

Messages: 14062
Registered: July 2009

Senior Member

ps it might be that somebody injects the mapper directly

=>

add

binder.bind(Storage2UriMapperJavaImpl.class).to(MyDslStorage2UriMapperJavaImpl.class);

Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/it-services/methods-and-tools/xtext
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: Exclude generated JAR files from indexing [message #1775404 is a reply to message #1775347]

Sun, 29 October 2017 17:26

Oleg Cohen

Messages: 28
Registered: August 2016

Junior Member

I will try to have a small example to demonstrate this and thank you for the PS suggestion. I will take a look.

On the same subject, I would like to see if there is any way to actually ensure that the files that are opened and inspected during the cache build/update are fully closed?

I am not sure which plugin is responsible for the opening/closing of the files. Xtext or JDT?

There is a discussion here https://bugs.eclipse.org/bugs/show_bug.cgi?id=406170 where people ran into similar issues and the Eclipse folks are suggesting that this call needs to be made:

JavaModelManager.getJavaModelManager().flushZipFiles(this);

My approach at the moment is still a workaround that isn't ideal because I am mocking around with the core of Xtext :-(

Maybe there is a way for me to insert this call somewhere after the caching process is complete?

Report message to a moderator