Eclipse Community Forums: TMF (Xtext) » Resolving imports in standalone generator

Help

Home

Home » Modeling » TMF (Xtext) » Resolving imports in standalone generator

Show: Today's Messages :: Show Polls :: Message Navigator

Resolving imports in standalone generator [message #775090]

Thu, 05 January 2012 09:59

Lasse indg

Messages: 12
Registered: July 2009

Junior Member

Hi,

I realize that this has been asked before, but I just can't find the solution.
So bear with me as I try to ask this question in a simple way instead of posting heaps of code that doesn't work, I'll just post some simpler more naive code that doesn't work either ;-)

I am trying to write a standalone wrapper for my code generator, and I am having trouble with the imports.

I have this simple domainmodel derived model:
* I is not based om XBase
* Uses namespace imports

grammar com.mydsl.SchemaDSL with org.eclipse.xtext.common.Terminals

generate schemaDSL "http://com.mydsl.SchemaDSL"

SchemaDSL:
	PackageDeclaration
	imports+=Import* 
	elements+=Entity*;

PackageDeclaration:
	'namespace' name=QualifiedName;

Import:
	'import' importedNamespace=QualifiedName;

Entity: Element;

Element:
	'element' name=ValidID ('extends' superType=[Entity])? '{'
		properties+=Property*
	'}';

Property:
	name=ValidID ':' type=[Entity];

QualifiedName:
	ID ('.' ID)*;

ValidID:
	ID
;

The above model is slightly simplified. I can post the real one if needed.

Using this model I can create the following simple files:
main.model:

namespace com.mydsl.main

import com.mydsl.ref.ReferencedEntity

element MainEntity {
	x : LocalEntity
	y : ReferencedEntity
} 

element LocalEntity {
	name : String
}

ref.model:

namespace com.mydsl.ref

element ReferencedEntity {
	number : Integer
}

I can launch the generated eclipse UI, and it generates files beautifully.

This standalone generator is inspired by the one from the state machine example.

	public static void main(String[] args) {
		Injector injector = new SchemaDSLStandaloneSetupGenerated().createInjectorAndDoEMFRegistration();
		Main2 main = injector.getInstance(Main2.class);
		// load the resource
		ResourceSet set = main.resourceSetProvider.get();
		Resource resource = set.getResource(URI.createURI(args[0]), true);
		
		// validate the resource
		List<Issue> list = main.validator.validate(resource, CheckMode.ALL, CancelIndicator.NullImpl);
		if (!list.isEmpty()) {
			for (Issue issue : list) {
				System.err.println(issue);
			}
			System.exit(-1);
		}
		
		// configure and start the generator
		main.fileAccess.setOutputPath("src-gen/");
		main.generator.doGenerate(resource, main.fileAccess);
		
		System.out.println("Code generation finished.");
	}

After adding all dependencies to my classpath I can use that to compile ref.model without errors.
When I compile main.model, it fails in the validator:

ERROR:Couldn't resolve reference to Entity 'ReferencedEntity'. (src\main.model line : 7)

I have put ref.model on my classpath, but that didn't make any difference.

http://www.eclipse.org/Xtext/documentation/1_0_1/xtext.html#processing_Xtext_models
Quote:

If one Resource references another Resource, EMF will automatically load that other Resource into the same ResourceSet as soon as the cross-reference is resolved. Resolution of cross-references is done lazy, i.e. on first access.

Somehow I need to get my standalone EMF setup to resolve the resources, but I just can't find out how to do it.

The closest I got was manually scanning my file system and adding all files to the resourceset like this:

for (String modelFile : getSourceFiles(srcFolder) {
    Resource modelResource = set.getResource(URI.createURI(modelFile), true);
    resource.getContents().addAll(modelResource.getContents());
}

I could make this post a lot longer by trying to explain all the things I have tried and failed with. But I think it is already too long.

This must have been solved many times by the community, since a standalone generator for any DSL must be the first requirement for deployment.

I hope that you can help me in the right direction.

Thanks in advance.

Report message to a moderator

Re: Resolving imports in standalone generator [message #775126 is a reply to message #775090]

Thu, 05 January 2012 11:36

Christian Dietrich

Messages: 14665
Registered: July 2009

Senior Member

Hi,

you have the solution in the hand so i dont see what is the problem

Resource resource = set.getResource(URI.createURI(args[0]), true);

you have to do this for all referenced files as well.

another solution: you can instantiate and call a workflow components
from java too (Reader)

~Christian

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: Resolving imports in standalone generator [message #775149 is a reply to message #775126]

Thu, 05 January 2012 12:59

Lasse indg

Messages: 12
Registered: July 2009

Junior Member

Thank you for your reply.

I have been down this path, but I ran into another problem, so I thought that I would backtrack a little in my original question, to find out if I should have gone down that road in the first place.

Ok, so here is my standalone generator that manually loads all referenced files:

    Injector injector = new SchemaDSLStandaloneSetupGenerated().createInjectorAndDoEMFRegistration();
    Main2 main = injector.getInstance(Main2.class);
    // load the resource
    ResourceSet set = main.resourceSetProvider.get();
    String folderName = args[0];
    Resource resource = null;
    for (String fileName : getSourceFiles(folderName)) {
        if (resource == null) {
            resource = set.getResource(URI.createURI(fileName), true);
        } else {
            Resource modelResource = set.getResource(URI.createURI(fileName), true);
            resource.getContents().addAll(modelResource.getContents());					
        }
    }
    
    // validator disabled for now
    main.fileAccess.setOutputPath("src-gen/");
    main.generator.doGenerate(resource, main.fileAccess);

(I pass in 'src' and it discovers 'src/*.model')

This works for the two files main.model and ref.model above.

However if I introduce ref2.model:

namespace com.mydsl.ref2 r2

element ReferencedEntity2 {
    number : Integer
}

and change ref.model to:

namespace com.mydsl.ref r

import com.mydsl.ref2.ReferencedEntity2

element ReferencedEntity {
    number : Integer
    ref2 : ReferencedEntity2
}

0    [main] ERROR xt.linking.lazy.LazyLinkingResource  - resolution of uriFragment 'xtextLink_::0.1.0.2.1::2::/4' failed.
org.eclipse.emf.common.util.BasicEList$BasicIndexOutOfBoundsException: index=0, size=0
        at org.eclipse.emf.common.util.BasicEList.get(BasicEList.java:352)
        at org.eclipse.xtext.linking.lazy.LazyURIEncoder.resolveShortFragment(LazyURIEncoder.java:92)
        at org.eclipse.xtext.linking.lazy.LazyURIEncoder.decode(LazyURIEncoder.java:80)
        at org.eclipse.xtext.linking.lazy.LazyLinkingResource.getEObject(LazyLinkingResource.java:162)
        at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getEObject(ResourceSetImpl.java:219)
        at org.eclipse.emf.ecore.util.EcoreUtil.resolve(EcoreUtil.java:202)
        at org.eclipse.emf.ecore.util.EcoreUtil.resolve(EcoreUtil.java:262)
        at org.eclipse.emf.ecore.impl.BasicEObjectImpl.eResolveProxy(BasicEObjectImpl.java:1475)
        at com.mydsl.schemaDSL.impl.PropertyImpl.getType(PropertyImpl.java:204)

I get the error when I try to access a referenced type:

	def mapType(Property p) {
		var type = p.type // type is a reference to Entity
    }

My guess is that I should read the resources in some different way, but I am stumped on what that way might be.

Report message to a moderator

Re: Resolving imports in standalone generator [message #775297 is a reply to message #775149]

Thu, 05 January 2012 18:34

Christian Dietrich

Messages: 14665
Registered: July 2009

Senior Member

Hi,

why do you do this getcontents add all stuff. this is not neccessary. standalone the global scope is built of ALL resources in the resourceset.
so you do not need to shift stuff between resources. and if you have to you have to first resolve all proxies.

~Christian

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: Resolving imports in standalone generator [message #775345 is a reply to message #775297]

Thu, 05 January 2012 20:16

Lasse indg

Messages: 12
Registered: July 2009

Junior Member

Thank you very much!
Of course it was incredible simple in the end. I don't know why it caused me so much trouble Grin

Below is my standalone generator for the benefit of others that might have the same request in the future.

/Lasse

package com.mydsl.generator;

import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.LinkedHashSet;
import java.util.List;
import java.util.Set;

import org.eclipse.emf.common.util.URI;
import org.eclipse.emf.ecore.resource.Resource;
import org.eclipse.emf.ecore.resource.ResourceSet;
import org.eclipse.xtext.Constants;
import org.eclipse.xtext.generator.IGenerator;
import org.eclipse.xtext.generator.JavaIoFileSystemAccess;
import org.eclipse.xtext.util.CancelIndicator;
import org.eclipse.xtext.validation.CheckMode;
import org.eclipse.xtext.validation.IResourceValidator;
import org.eclipse.xtext.validation.Issue;

import com.google.inject.Inject;
import com.google.inject.Injector;
import com.google.inject.Provider;
import com.google.inject.name.Named;

import com.mydsl.SchemaDSLStandaloneSetupGenerated;

public class Main {

	@Inject 
	private Provider<ResourceSet> resourceSetProvider;
	
	@Inject
	private IResourceValidator validator;
	
	@Inject
	private IGenerator generator;
	
	@Inject 
	private JavaIoFileSystemAccess fileAccess;

	@Inject @Named(Constants.FILE_EXTENSIONS)
	private String fileExtension;
	
	public static void main(String[] args) {
		checkArgs(args);
		try {
			Injector injector = new SchemaDSLStandaloneSetupGenerated().createInjectorAndDoEMFRegistration();
			Main main = injector.getInstance(Main.class);
			main.run(args[0]);
			
		} catch (Exception e) {
			System.err.println("Unexpected error");
			e.printStackTrace();
		}
	}

	private static void checkArgs(String[] args) {
		if (args.length != 1) {
			System.err.println("Name of the source folder expected");
			System.exit(-1);
		} else {
			File folder = new File(args[0]);
			if (!(folder.exists() && folder.isDirectory())) {
				System.err.println("Source folder " + folder.getAbsolutePath() + " not found");
				System.exit(-1);
			}
		}
	}
	
	private void run(String folderName) throws IOException {
		// load the resource
		ResourceSet set = resourceSetProvider.get();
		Set<Resource> resources = new LinkedHashSet<Resource>();
		for (String fileName : getSourceFiles(folderName)) {
			Resource modelResource = set.getResource(URI.createURI(fileName), true);
			resources.add(modelResource);
		}

		// configure and start the generator
		fileAccess.setOutputPath("src-gen/");
		for (Resource resource : resources) {
			// validate the resource
			List<Issue> list = validator.validate(resource, CheckMode.ALL, CancelIndicator.NullImpl);
			if (!list.isEmpty()) {
				for (Issue issue : list) {
					System.err.println(issue);
				}
				System.exit(-2);
			}

			generator.doGenerate(resource, fileAccess);
		}
		
		System.out.println("Code generation finished.");
	}
	
	private List<String> getSourceFiles(String folderName) throws IOException {
		ArrayList<String> result = new ArrayList<String>();
		File folder = new File(folderName);
		for (String s : folder.list()) {
			File f = new File(folder.getAbsolutePath() + '/' + s);
			String relativePath = folderName + '/' + s;
			if (isModelFile(f)) {
				result.add(relativePath);
			}
			else if (f.isDirectory()) {
				result.addAll(getSourceFiles(relativePath));
			}
		}		
		return result;
	}

	private boolean isModelFile(File f) {
		return f.isFile() && f.getName().endsWith("." + fileExtension);
	}
}

Report message to a moderator

Re: Resolving imports in standalone generator [message #885305 is a reply to message #775345]

Tue, 12 June 2012 19:28

Brad Riching

Messages: 20
Registered: May 2012

Junior Member

Lasse,

I tried your example because I was running into the same problem you had. However, when I do, I get the following stack trace when I run my compiler at the command line in cygwin:

$ java edu.byu.ee.phdl.Compile fmc_module
1    [main] ERROR xt.linking.lazy.LazyLinkingResource  - resolution of uriFragment 'xtextLink_::0.3.0.3.5::5::/8' failed.
java.lang.StringIndexOutOfBoundsException: String index out of range: 402
        at java.lang.String.substring(String.java:1934)
        at org.eclipse.xtext.nodemodel.impl.AbstractNode.getText(AbstractNode.java:84)
        at org.eclipse.xtext.nodemodel.util.NodeModelUtils.getTokenText(NodeModelUtils.java:351)
        at org.eclipse.xtext.linking.impl.LinkingHelper.getCrossRefNodeAsString(LinkingHelper.java:51)
        at org.eclipse.xtext.linking.impl.DefaultLinkingService.getCrossRefNodeAsString(DefaultLinkingService.java:132)
        at org.eclipse.xtext.linking.impl.DefaultLinkingService.getLinkedObjects(DefaultLinkingService.java:113)
        at org.eclipse.xtext.linking.lazy.LazyLinkingResource.getEObject(LazyLinkingResource.java:175)
        at org.eclipse.xtext.linking.lazy.LazyLinkingResource.resolveLazyCrossReference(LazyLinkingResource.java:143)
        at org.eclipse.xtext.linking.lazy.LazyLinkingResource.resolveLazyCrossReferences(LazyLinkingResource.java:104)
        at org.eclipse.xtext.EcoreUtil2.resolveLazyCrossReferences(EcoreUtil2.java:491)
        at org.eclipse.xtext.validation.ResourceValidatorImpl.resolveProxies(ResourceValidatorImpl.java:127)
        at org.eclipse.xtext.validation.ResourceValidatorImpl.validate(ResourceValidatorImpl.java:62)
        at edu.byu.ee.phdl.Compile.run(Compile.java:81)
        at edu.byu.ee.phdl.Compile.main(Compile.java:117)
Unexpected error
org.eclipse.emf.common.util.WrappedException: java.lang.StringIndexOutOfBoundsException: String index out of range: 402
        at org.eclipse.xtext.linking.lazy.LazyLinkingResource.getEObject(LazyLinkingResource.java:209)
        at org.eclipse.xtext.linking.lazy.LazyLinkingResource.resolveLazyCrossReference(LazyLinkingResource.java:143)
        at org.eclipse.xtext.linking.lazy.LazyLinkingResource.resolveLazyCrossReferences(LazyLinkingResource.java:104)
        at org.eclipse.xtext.EcoreUtil2.resolveLazyCrossReferences(EcoreUtil2.java:491)
        at org.eclipse.xtext.validation.ResourceValidatorImpl.resolveProxies(ResourceValidatorImpl.java:127)
        at org.eclipse.xtext.validation.ResourceValidatorImpl.validate(ResourceValidatorImpl.java:62)
        at edu.byu.ee.phdl.Compile.run(Compile.java:81)
        at edu.byu.ee.phdl.Compile.main(Compile.java:117)
Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: 402
        at java.lang.String.substring(String.java:1934)
        at org.eclipse.xtext.nodemodel.impl.AbstractNode.getText(AbstractNode.java:84)
        at org.eclipse.xtext.nodemodel.util.NodeModelUtils.getTokenText(NodeModelUtils.java:351)
        at org.eclipse.xtext.linking.impl.LinkingHelper.getCrossRefNodeAsString(LinkingHelper.java:51)
        at org.eclipse.xtext.linking.impl.DefaultLinkingService.getCrossRefNodeAsString(DefaultLinkingService.java:132)
        at org.eclipse.xtext.linking.impl.DefaultLinkingService.getLinkedObjects(DefaultLinkingService.java:113)
        at org.eclipse.xtext.linking.lazy.LazyLinkingResource.getEObject(LazyLinkingResource.java:175)
        ... 7 more

However, when I create a run configuration in Eclipse, give it my main class (from your example) I get expected output in the console window:

Code generation finished.

The program also generates all the output files as expected. As far as I can tell, I have all of the jar dependencies on my classpath for running the standalone application outside of eclipse:

CLASSPATH=.;
C:\Users\brad\eclipse\edu.byu.ee.phdl.xtext\bin;
C:\Program Files\eclipse Indigo\plugins\org.eclipse.xtext_2.2.1.v201112130541.jar;
C:\Program Files\eclipse Indigo\plugins\org.eclipse.emf.ecore_2.7.0.v20120127-1122.jar;
C:\Program Files\eclipse Indigo\plugins\org.eclipse.emf.common_2.7.0.v20120127-1122.jar;
C:\Program Files\eclipse Indigo\plugins\org.eclipse.emf.ecore.xmi_2.7.0.v20120127-1122.jar;
C:\Program Files\eclipse Indigo\plugins\com.google.inject_3.0.0.no_aop.jar;
C:\Program Files\eclipse Indigo\plugins\javax.inject_1.0.0.v20091030.jar;
C:\Program Files\eclipse Indigo\plugins\org.apache.log4j_1.2.15.v201012070815.jar;
C:\Program Files\eclipse Indigo\plugins\org.eclipse.xtext.common.types_2.2.1.v201112130541.jar;
C:\Program Files\eclipse Indigo\plugins\com.google.guava_10.0.1.1.jar;
C:\Program Files\eclipse Indigo\plugins\org.eclipse.xtext.util_2.2.1.v201112130541.jar;
C:\Program Files\eclipse Indigo\plugins\org.eclipse.xtext.xbase.lib_2.2.1.v201112130541.jar;
C:\Program Files\eclipse Indigo\plugins\org.eclipse.emf.mwe.utils_1.2.1.v201112070431.jar;
C:\Program Files\eclipse Indigo\plugins\org.eclipse.emf.mwe.core_1.2.1.v201112070431.jar;
C:\Program Files\eclipse Indigo\plugins\org.apache.commons.logging_1.0.4.v201101211617.jar

I have tried closing my currently running instance of Eclipse to rule out any global registry issues. I end up with the same problem with the linking. My intuition says it has something to do with the URI, since it works inside Eclipse but not from the command line. Do I need to add a line in the main class that sets the platformURI? such as:

new org.eclipse.emf.mwe.utils.StandaloneSetup().setPlatformUri(<cwd>);

where <cwd> is the current working directory (specified by your source folder argument in the main method).

I'm really stumped, as the error reporting doesn't appear to offer me any good direction.

My question is: have you been able to run this outside of the Eclipse platform, truly from the command-line with Eclipse completely closed? If so, would you mind sharing how you did it?

Report message to a moderator

Re: Resolving imports in standalone generator [message #885317 is a reply to message #885305]

Tue, 12 June 2012 19:52

Christian Dietrich

Messages: 14665
Registered: July 2009

Senior Member

Sorry from the stuff you posted i cannot tell anything.
did you try the export -> runnable jar file wizard?

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: Resolving imports in standalone generator [message #885324 is a reply to message #885317]

Tue, 12 June 2012 20:11

Brad Riching

Messages: 20
Registered: May 2012

Junior Member

Christian,

Thank you. That worked for me. I guess I thought exporting into a jar was optional as long as I had all of the dependencies resolved on my classpath. I'm wondering why the xtext standalone howto does not mention this simple step, which would have avoided me several hours of troubleshooting and the hassle of tying up valuable eclipse developer resources.

Best wishes,

Brad

Report message to a moderator

Re: Resolving imports in standalone generator [message #885326 is a reply to message #885324]

Tue, 12 June 2012 20:16

Christian Dietrich

Messages: 14665
Registered: July 2009

Senior Member

Hi,

feel free to file an enhancement request against the docs

~Christian

Twitter : @chrdietrich
Blog : https://www.dietrich-it.de

Report message to a moderator

Re: Resolving imports in standalone generator [message #897077 is a reply to message #775090]

Sat, 21 July 2012 13:27

Hannes Stockner

Messages: 1
Registered: May 2011

Junior Member

Hello,

I am using xtext 2.3.0 and I copied the code from Lasse indg to my standalone generator.

I still get ""ERROR:Couldn't resolve reference to..."

Using single files without references works fine.

Do you have any idea what the problem could be?

Thanks
Hannes

Report message to a moderator