Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Problem using the Xtext serializer
Problem using the Xtext serializer [message #1125474] Fri, 04 October 2013 10:55 Go to next message
Alan DW is currently offline Alan DW
Messages: 77
Registered: March 2012
Member
Hello everyone,

I have an XMI file which contains instances of my DSL. I want to deserialize the XMI and store the EObjects in files in my workspace (programmatically), such that the files contain DSL code again. This works very well as long as there are no cross references, but I run into problems when there are some.

I've attached a minimum working example that uses a slight extension of the "Greetings DSL", just to introduce a few cross references and illustrate the problem. The project "MinmalWorkingExample" contains a JUnit test in "src/minimalworkingexample/TestClass" that can be run as a JUnit-Plugin-Test without any setup required. It shows what I'm trying to do.

The error I get when deserializing a model element with cross references back into DSL code is:

java.lang.RuntimeException: No EObjectDescription found in Scope Greetings.person for Model.fragments[1]->Person'John'



It would be easy to state now that this is a scoping problem. Two facts however indicate that this is not the case:


  • Using the StandaloneSetup of the DSL fixes the problem, but I can't use it inside Eclipse (this is soooo frustrating -.-)
  • I did not implement any custom scoping (or other language customization) at all.


My best guess would be that there is something wrong in the "wiring" of the plug-ins (plugin.xml, build.properties...) but I could not spot any problems (neither did my IDE). I'm using Eclipse Juno SR1 and Xtext 2.4.3.


Any help would be much appreciated!



Alan
Re: Problem using the Xtext serializer [message #1125747 is a reply to message #1125474] Fri, 04 October 2013 17:10 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian Dietrich
Messages: 5245
Registered: July 2009
Senior Member
package minmalworkingexample;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;

import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.util.HashSet;
import java.util.Set;

import javax.inject.Inject;

import org.eclipse.core.resources.IFile;
import org.eclipse.core.resources.IProject;
import org.eclipse.core.resources.IProjectDescription;
import org.eclipse.core.resources.IWorkspaceRoot;
import org.eclipse.core.resources.IncrementalProjectBuilder;
import org.eclipse.core.resources.ResourcesPlugin;
import org.eclipse.core.runtime.CoreException;
import org.eclipse.core.runtime.NullProgressMonitor;
import org.eclipse.core.runtime.jobs.Job;
import org.eclipse.emf.common.util.URI;
import org.eclipse.emf.ecore.EObject;
import org.eclipse.emf.ecore.resource.Resource;
import org.eclipse.emf.ecore.resource.ResourceSet;
import org.eclipse.swt.widgets.Display;
import org.eclipse.xtext.builder.nature.XtextNature;
import org.eclipse.xtext.junit4.InjectWith;
import org.eclipse.xtext.junit4.XtextRunner;
import org.eclipse.xtext.resource.IResourceDescriptions;
import org.eclipse.xtext.ui.XtextProjectHelper;
import org.eclipse.xtext.ui.resource.IResourceSetProvider;
import org.eclipse.xtext.ui.wizard.XtextNewProjectWizard;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.xtext.example.mydsl.MyDslUiInjectorProvider;
import org.xtext.example.mydsl.myDsl.Fragment;
import org.xtext.example.mydsl.myDsl.Greeting;
import org.xtext.example.mydsl.myDsl.Model;
import org.xtext.example.mydsl.myDsl.MyDslFactory;
import org.xtext.example.mydsl.myDsl.Person;

import utils.FileUtils;
import utils.XMIUtils;

@RunWith(XtextRunner.class)
@InjectWith(MyDslUiInjectorProvider.class)
public class TestClass {

	private static final String MODEL_FILE_NAME = "resources/model.xmi";
	private static final String TARGET_PROJECT_NAME = "TARGET";

	@Inject
	IResourceSetProvider resourceSetProvider;
	
	@Inject
	IResourceDescriptions desc;

	@BeforeClass
	public static void setUp() {
		System.out.println("SET UP START");
		// delete our dummy project (that we build during the test) if it exists.
		IWorkspaceRoot workspaceRoot = ResourcesPlugin.getWorkspace().getRoot();
		IProject targetProject = workspaceRoot.getProject(TARGET_PROJECT_NAME);
		if (targetProject.exists()) {
			try {
				targetProject.delete(true, true, new NullProgressMonitor());
			} catch (CoreException e) {
				e.printStackTrace();
				fail();
			}
		}
		waitForJobs();
		System.out.println("SET UP DONE");
	}

	/**
	 * This is the main test method.<br>
	 * It reads the XMI file from "/resources/model.xmi" and tries to save its contents to two different files.
	 */
	@Test
	public void runTest() {
		// enable the next line and the JUnit test will work and produce the expected output!
		// However, it is strongly discouraged in OSGI-Based environments.
		// MyDslStandaloneSetup.doSetup();
		EObject model = this.deserializeTestXMI();
		assertTrue("Could not deserialize XMI!", model != null && model instanceof Model);
		this.partitionAndSerialize((Model) model);
	}

	private EObject deserializeTestXMI() {
		File xmiFile = FileUtils.getBundleResource(Activator.getDefault().getBundle(), MODEL_FILE_NAME);
		return XMIUtils.deserializeFromXMIFile(xmiFile);
	}

	/**
	 * Partitions the model into files and saves them, using the Xtext serializer.<br>
	 * All instances of {@link Person} will go into the file <code>Persons.mydsl</code>, all instances of
	 * {@link Greeting} will go into the file <code>Greetings.mydsl</code>.
	 * 
	 * @param model
	 *            The model to partition and save
	 */
	private void partitionAndSerialize(final Model model) {
		IProject project = this.createTargetProject();
		Set<Person> persons = new HashSet<Person>();
		Set<Greeting> greetings = new HashSet<Greeting>();
		for (Fragment fragment : model.getFragments()) {
			if (fragment instanceof Person) {
				persons.add((Person) fragment);
			}
			if (fragment instanceof Greeting) {
				greetings.add((Greeting) fragment);
			}
		}
		assertEquals(2, greetings.size());
		assertEquals(2, persons.size());
		IFile greetingsFile = project.getFile("Greetings.mydsl");
		IFile personsFile = project.getFile("Persons.mydsl");
		try {
			this.createFile(greetingsFile);
			this.createFile(personsFile);
		} catch (CoreException e) {
			e.printStackTrace();
			fail();
		}
		ResourceSet resSet = this.resourceSetProvider.get(project);
		Resource greetingsResource = resSet.getResource(URI.createPlatformResourceURI(greetingsFile.getFullPath().toString()), true);
		Resource personsResource = resSet.getResource(URI.createPlatformResourceURI(personsFile.getFullPath().toString()), true);
		Model personsModel = MyDslFactory.eINSTANCE.createModel();
		personsModel.getFragments().addAll(persons);
		Model greetingsModel = MyDslFactory.eINSTANCE.createModel();
		greetingsModel.getFragments().addAll(greetings);
		greetingsResource.getContents().add(greetingsModel);
		personsResource.getContents().add(personsModel);
		try {
			personsResource.save(null);
			project.build(IncrementalProjectBuilder.FULL_BUILD, new NullProgressMonitor());
			waitForJobs();
			System.out.println(desc.getAllResourceDescriptions());
			
			
			greetingsResource.save(null);
		} catch (IOException e) {
			e.printStackTrace();
			fail();
		} catch (CoreException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}

	private IProject createTargetProject() {
		IWorkspaceRoot workspaceRoot = ResourcesPlugin.getWorkspace().getRoot();
		IProject project = workspaceRoot.getProject(TARGET_PROJECT_NAME);
		try {
			project.create(new NullProgressMonitor());
			project.open(new NullProgressMonitor());
			IProjectDescription description = project.getDescription();
			String[] natures = description.getNatureIds();
			String[] newnatures = new String[natures.length + 1];
			System.arraycopy(natures, 0, newnatures, 0, natures.length);
			newnatures[natures.length] = XtextProjectHelper.NATURE_ID;
			description.setNatureIds(newnatures);
			System.out.println(newnatures);
			project.setDescription(description, new NullProgressMonitor());
			XtextNature n = new XtextNature();
			n.setProject(project);
			n.configure();
		} catch (CoreException e) {
			e.printStackTrace();
			fail();
		}
		return project;
	}

	private void createFile(final IFile file) throws CoreException {
		file.create(new ByteArrayInputStream(new byte[0]), true, new NullProgressMonitor());
	}

	/**
	 * Perform post-test cleanup.
	 */
	@AfterClass
	public static void tearDown() throws Exception {
		// Dispose of test fixture.
		waitForJobs();
		// Add additional teardown code here.
	}

	/**
	 * Process UI input but do not return for the specified time interval.
	 * 
	 * @param waitTimeMillis
	 *            the number of milliseconds
	 */
	private static void delay(final long waitTimeMillis) {
		Display display = Display.getCurrent();

		// If this is the UI thread,
		// then process input.

		if (display != null) {
			long endTimeMillis = System.currentTimeMillis() + waitTimeMillis;
			while (System.currentTimeMillis() < endTimeMillis) {
				if (!display.readAndDispatch()) {
					display.sleep();
				}
			}
			display.update();
		}
		// Otherwise, perform a simple sleep.
		else {
			try {
				Thread.sleep(waitTimeMillis);
			} catch (InterruptedException e) {
				// Ignored.
			}
		}
	}

	/**
	 * Wait until all background tasks are complete.
	 */
	private static void waitForJobs() {
		while (!Job.getJobManager().isIdle()) {
			delay(1000);
		}
	}
}
Re: Problem using the Xtext serializer [message #1126205 is a reply to message #1125747] Sat, 05 October 2013 05:12 Go to previous messageGo to next message
Alan DW is currently offline Alan DW
Messages: 77
Registered: March 2012
Member
Hi Christian,

thanks for your response and your continued support! I'm ashamed that I forgot to add the Xtext nature in the code, I actually have it in my project.

These lines are interesting, though:

personsResource.save(null);
project.build(IncrementalProjectBuilder.FULL_BUILD, new NullProgressMonitor());
waitForJobs();
System.out.println(desc.getAllResourceDescriptions());
greetingsResource.save(null);



Okay, I may do that if I know that the file "persons" has no cross references at all then this will be perfectly fine. But what about a scenario where I have cyclic cross references? For example, an EObject in File A references an EObject in File B, but there is also an EObject in B that references one in A? Or transitive cycles (A->B->C->A)? How would you resolve such a situation? It is not so uncommon, after all.
Re: Problem using the Xtext serializer [message #1126227 is a reply to message #1126205] Sat, 05 October 2013 05:48 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian Dietrich
Messages: 5245
Registered: July 2009
Senior Member
Sorry have no idea on that. Maybe it is not possible at all (I do not
have the time to digg on that but Xtext indexing is required for the
serialized somehow (I don't know if it is possible to skip that)

Maybe Moritz has an idea

--
Need training, onsite consulting or any other kind of help for Xtext?
Go visit http://xtext.itemis.com or send a mail to xtext at itemis dot de
Re: Problem using the Xtext serializer [message #1126247 is a reply to message #1126227] Sat, 05 October 2013 06:16 Go to previous messageGo to next message
Alan DW is currently offline Alan DW
Messages: 77
Registered: March 2012
Member
Well, basically, all information is available in the model, it is all there. As far as I know, the serializer draws its information from a kind of registry that is managed internally, and an object gets registered there when a resource (containing the object) is saved and the project is built. Rather than the internal registry, the serializer would have to use the instance model (provided by the XMI) directly. In the end, all it really has to do is to insert the (qualified) name of a referenced object whenever it encounters a non-containment reference. Actually, I'm quite surprised that I'm the first one to stumble upon this problem...

Another solution would be to not use the Xtext serializer alltogether and "manually" write the Model-to-Text transformation in Java/Xtend. I tried that, but I'm also getting some weird exceptions when I just write the DSL text into the files and try to build the project afterwards.

If anyone has a possible solution, please let me know. I'll continue to try things out and do research on the web, maybe I'll find something of use.

[Updated on: Sat, 05 October 2013 06:17]

Report message to a moderator

Re: Problem using the Xtext serializer [message #1126309 is a reply to message #1126247] Sat, 05 October 2013 07:51 Go to previous messageGo to next message
Moritz Eysholdt is currently offline Moritz Eysholdt
Messages: 138
Registered: July 2009
Location: Kiel, Germany
Senior Member
Alan DW wrote on Sat, 05 October 2013 11:12
But what about a scenario where I have cyclic cross references? For example, an EObject in File A references an EObject in File B, but there is also an EObject in B that references one in A? Or transitive cycles (A->B->C->A)? How would you resolve such a situation? It is not so uncommon, after all.


That's no problem for the serializer. The serializer does nothing except call the IScopeProvider for every cross reference. As long as the scope provider can provider a scope for a cross reference and the scope can provide a name for the referenced object, the serializer can save the file.

Christian Dietrich wrote on Sat, 05 October 2013 11:48
Sorry have no idea on that. Maybe it is not possible at all (I do not
have the time to dig on that but Xtext indexing is required for the
serialized somehow (I don't know if it is possible to skip that)


As mentioned above, the serializer uses scoping to determine the to-be-serialized names for cross references. However, Xtext's standard scoping implementation for cross-file references is based on the Xtext index. Thereby, serialization indirectly depends on the index.

If you don't like this behavior, you can override
org.eclipse.xtext.serializer.tokens.CrossReferenceSerializer.serializeCrossRef(EObject, CrossReference, EObject, INode, Acceptor)
and put in your own implementation. Maybe one that doesn't make use of scoping and/or the index and instead delegates to the IQualifiedNameProvider.

Alan DW wrote on Sat, 05 October 2013 12:16
Well, basically, all information is available in the model, it is all there. As far as I know, the serializer draws its information from a kind of registry that is managed internally, and an object gets registered there when a resource (containing the object) is saved and the project is built. Rather than the internal registry, the serializer would have to use the instance model (provided by the XMI) directly. In the end, all it really has to do is to insert the (qualified) name of a referenced object whenever it encounters a non-containment reference. Actually, I'm quite surprised that I'm the first one to stumble upon this problem...


What you describe as the "internal Registry" sounds like Xtext's index. There is no other "internal Registry" otherwise.
Yes, when the Xtext Builder runs, it collects QualifiedNames for all model elements and stores them in the index. And the Xtext builder runs when you trigger a full build or save a file (autobuild, which is an incremental build).

Is your use case to save a file that cross-references elements from unsaved files?
Re: Problem using the Xtext serializer [message #1127329 is a reply to message #1126309] Sun, 06 October 2013 10:37 Go to previous message
Alan DW is currently offline Alan DW
Messages: 77
Registered: March 2012
Member
Hi Moritz,

thanks for your response.

Quote:
Is your use case to save a file that cross-references elements from unsaved files?



Yes, it boils down to exactly this problem. I cannot guarantee that all cross-reference targets have been saved (and/or built) because I don't even know the grammar of the language in my use-case (it can be an arbitrary grammar). Of course, I can check all non-containment references in my EObjects and store the ones first that don't have any cross references, build the project, then save the ones which are only cross referencing the EObjects I just saved and built, and so on. But due to the generic nature of the tool I am building, I must consider the case when there are cyclic dependencies. For that reason, I cannot save and build those files first which do not have cross-references.

Do you have any hint on how to do this? If it can be avoided at all, I'd rather not touch on the scoping since I'm totally fine with the default scoping in the editor.


Basically, what happens in my tool is: I get an XMI file that adheres to an Xtext generated Metamodel, and I want to re-build the DSL code from that XMI file. Which model element ends up in which generated file is determined by the user, I have no control over that. In the end, I need to simultaneously save all of them, that's the goal.

[Updated on: Sun, 06 October 2013 10:45]

Report message to a moderator

Previous Topic:using types from non-exported packages
Next Topic:New Maven archetype available: Xtext with multi module Tycho
Goto Forum:
  


Current Time: Mon Oct 07 10:30:12 EDT 2013

Powered by FUDForum. Page generated in 0.01733 seconds