Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Problem using the Xtext serializer
Problem using the Xtext serializer [message #1125474] Fri, 04 October 2013 14:55 Go to next message
Alan DW is currently offline Alan DWFriend
Messages: 119
Registered: March 2012
Senior Member
Hello everyone,

I have an XMI file which contains instances of my DSL. I want to deserialize the XMI and store the EObjects in files in my workspace (programmatically), such that the files contain DSL code again. This works very well as long as there are no cross references, but I run into problems when there are some.

I've attached a minimum working example that uses a slight extension of the "Greetings DSL", just to introduce a few cross references and illustrate the problem. The project "MinmalWorkingExample" contains a JUnit test in "src/minimalworkingexample/TestClass" that can be run as a JUnit-Plugin-Test without any setup required. It shows what I'm trying to do.

The error I get when deserializing a model element with cross references back into DSL code is:

java.lang.RuntimeException: No EObjectDescription found in Scope Greetings.person for Model.fragments[1]->Person'John'



It would be easy to state now that this is a scoping problem. Two facts however indicate that this is not the case:


  • Using the StandaloneSetup of the DSL fixes the problem, but I can't use it inside Eclipse (this is soooo frustrating -.-)
  • I did not implement any custom scoping (or other language customization) at all.


My best guess would be that there is something wrong in the "wiring" of the plug-ins (plugin.xml, build.properties...) but I could not spot any problems (neither did my IDE). I'm using Eclipse Juno SR1 and Xtext 2.4.3.


Any help would be much appreciated!



Alan
Re: Problem using the Xtext serializer [message #1125747 is a reply to message #1125474] Fri, 04 October 2013 21:10 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13264
Registered: July 2009
Senior Member
package minmalworkingexample;

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import static org.junit.Assert.fail;

import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;
import java.util.HashSet;
import java.util.Set;

import javax.inject.Inject;

import org.eclipse.core.resources.IFile;
import org.eclipse.core.resources.IProject;
import org.eclipse.core.resources.IProjectDescription;
import org.eclipse.core.resources.IWorkspaceRoot;
import org.eclipse.core.resources.IncrementalProjectBuilder;
import org.eclipse.core.resources.ResourcesPlugin;
import org.eclipse.core.runtime.CoreException;
import org.eclipse.core.runtime.NullProgressMonitor;
import org.eclipse.core.runtime.jobs.Job;
import org.eclipse.emf.common.util.URI;
import org.eclipse.emf.ecore.EObject;
import org.eclipse.emf.ecore.resource.Resource;
import org.eclipse.emf.ecore.resource.ResourceSet;
import org.eclipse.swt.widgets.Display;
import org.eclipse.xtext.builder.nature.XtextNature;
import org.eclipse.xtext.junit4.InjectWith;
import org.eclipse.xtext.junit4.XtextRunner;
import org.eclipse.xtext.resource.IResourceDescriptions;
import org.eclipse.xtext.ui.XtextProjectHelper;
import org.eclipse.xtext.ui.resource.IResourceSetProvider;
import org.eclipse.xtext.ui.wizard.XtextNewProjectWizard;
import org.junit.AfterClass;
import org.junit.BeforeClass;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.xtext.example.mydsl.MyDslUiInjectorProvider;
import org.xtext.example.mydsl.myDsl.Fragment;
import org.xtext.example.mydsl.myDsl.Greeting;
import org.xtext.example.mydsl.myDsl.Model;
import org.xtext.example.mydsl.myDsl.MyDslFactory;
import org.xtext.example.mydsl.myDsl.Person;

import utils.FileUtils;
import utils.XMIUtils;

@RunWith(XtextRunner.class)
@InjectWith(MyDslUiInjectorProvider.class)
public class TestClass {

	private static final String MODEL_FILE_NAME = "resources/model.xmi";
	private static final String TARGET_PROJECT_NAME = "TARGET";

	@Inject
	IResourceSetProvider resourceSetProvider;
	
	@Inject
	IResourceDescriptions desc;

	@BeforeClass
	public static void setUp() {
		System.out.println("SET UP START");
		// delete our dummy project (that we build during the test) if it exists.
		IWorkspaceRoot workspaceRoot = ResourcesPlugin.getWorkspace().getRoot();
		IProject targetProject = workspaceRoot.getProject(TARGET_PROJECT_NAME);
		if (targetProject.exists()) {
			try {
				targetProject.delete(true, true, new NullProgressMonitor());
			} catch (CoreException e) {
				e.printStackTrace();
				fail();
			}
		}
		waitForJobs();
		System.out.println("SET UP DONE");
	}

	/**
	 * This is the main test method.<br>
	 * It reads the XMI file from "/resources/model.xmi" and tries to save its contents to two different files.
	 */
	@Test
	public void runTest() {
		// enable the next line and the JUnit test will work and produce the expected output!
		// However, it is strongly discouraged in OSGI-Based environments.
		// MyDslStandaloneSetup.doSetup();
		EObject model = this.deserializeTestXMI();
		assertTrue("Could not deserialize XMI!", model != null && model instanceof Model);
		this.partitionAndSerialize((Model) model);
	}

	private EObject deserializeTestXMI() {
		File xmiFile = FileUtils.getBundleResource(Activator.getDefault().getBundle(), MODEL_FILE_NAME);
		return XMIUtils.deserializeFromXMIFile(xmiFile);
	}

	/**
	 * Partitions the model into files and saves them, using the Xtext serializer.<br>
	 * All instances of {@link Person} will go into the file <code>Persons.mydsl</code>, all instances of
	 * {@link Greeting} will go into the file <code>Greetings.mydsl</code>.
	 * 
	 * @param model
	 *            The model to partition and save
	 */
	private void partitionAndSerialize(final Model model) {
		IProject project = this.createTargetProject();
		Set<Person> persons = new HashSet<Person>();
		Set<Greeting> greetings = new HashSet<Greeting>();
		for (Fragment fragment : model.getFragments()) {
			if (fragment instanceof Person) {
				persons.add((Person) fragment);
			}
			if (fragment instanceof Greeting) {
				greetings.add((Greeting) fragment);
			}
		}
		assertEquals(2, greetings.size());
		assertEquals(2, persons.size());
		IFile greetingsFile = project.getFile("Greetings.mydsl");
		IFile personsFile = project.getFile("Persons.mydsl");
		try {
			this.createFile(greetingsFile);
			this.createFile(personsFile);
		} catch (CoreException e) {
			e.printStackTrace();
			fail();
		}
		ResourceSet resSet = this.resourceSetProvider.get(project);
		Resource greetingsResource = resSet.getResource(URI.createPlatformResourceURI(greetingsFile.getFullPath().toString()), true);
		Resource personsResource = resSet.getResource(URI.createPlatformResourceURI(personsFile.getFullPath().toString()), true);
		Model personsModel = MyDslFactory.eINSTANCE.createModel();
		personsModel.getFragments().addAll(persons);
		Model greetingsModel = MyDslFactory.eINSTANCE.createModel();
		greetingsModel.getFragments().addAll(greetings);
		greetingsResource.getContents().add(greetingsModel);
		personsResource.getContents().add(personsModel);
		try {
			personsResource.save(null);
			project.build(IncrementalProjectBuilder.FULL_BUILD, new NullProgressMonitor());
			waitForJobs();
			System.out.println(desc.getAllResourceDescriptions());
			
			
			greetingsResource.save(null);
		} catch (IOException e) {
			e.printStackTrace();
			fail();
		} catch (CoreException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
	}

	private IProject createTargetProject() {
		IWorkspaceRoot workspaceRoot = ResourcesPlugin.getWorkspace().getRoot();
		IProject project = workspaceRoot.getProject(TARGET_PROJECT_NAME);
		try {
			project.create(new NullProgressMonitor());
			project.open(new NullProgressMonitor());
			IProjectDescription description = project.getDescription();
			String[] natures = description.getNatureIds();
			String[] newnatures = new String[natures.length + 1];
			System.arraycopy(natures, 0, newnatures, 0, natures.length);
			newnatures[natures.length] = XtextProjectHelper.NATURE_ID;
			description.setNatureIds(newnatures);
			System.out.println(newnatures);
			project.setDescription(description, new NullProgressMonitor());
			XtextNature n = new XtextNature();
			n.setProject(project);
			n.configure();
		} catch (CoreException e) {
			e.printStackTrace();
			fail();
		}
		return project;
	}

	private void createFile(final IFile file) throws CoreException {
		file.create(new ByteArrayInputStream(new byte[0]), true, new NullProgressMonitor());
	}

	/**
	 * Perform post-test cleanup.
	 */
	@AfterClass
	public static void tearDown() throws Exception {
		// Dispose of test fixture.
		waitForJobs();
		// Add additional teardown code here.
	}

	/**
	 * Process UI input but do not return for the specified time interval.
	 * 
	 * @param waitTimeMillis
	 *            the number of milliseconds
	 */
	private static void delay(final long waitTimeMillis) {
		Display display = Display.getCurrent();

		// If this is the UI thread,
		// then process input.

		if (display != null) {
			long endTimeMillis = System.currentTimeMillis() + waitTimeMillis;
			while (System.currentTimeMillis() < endTimeMillis) {
				if (!display.readAndDispatch()) {
					display.sleep();
				}
			}
			display.update();
		}
		// Otherwise, perform a simple sleep.
		else {
			try {
				Thread.sleep(waitTimeMillis);
			} catch (InterruptedException e) {
				// Ignored.
			}
		}
	}

	/**
	 * Wait until all background tasks are complete.
	 */
	private static void waitForJobs() {
		while (!Job.getJobManager().isIdle()) {
			delay(1000);
		}
	}
}


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/xtext/
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Problem using the Xtext serializer [message #1126205 is a reply to message #1125747] Sat, 05 October 2013 09:12 Go to previous messageGo to next message
Alan DW is currently offline Alan DWFriend
Messages: 119
Registered: March 2012
Senior Member
Hi Christian,

thanks for your response and your continued support! I'm ashamed that I forgot to add the Xtext nature in the code, I actually have it in my project.

These lines are interesting, though:

personsResource.save(null);
project.build(IncrementalProjectBuilder.FULL_BUILD, new NullProgressMonitor());
waitForJobs();
System.out.println(desc.getAllResourceDescriptions());
greetingsResource.save(null);



Okay, I may do that if I know that the file "persons" has no cross references at all then this will be perfectly fine. But what about a scenario where I have cyclic cross references? For example, an EObject in File A references an EObject in File B, but there is also an EObject in B that references one in A? Or transitive cycles (A->B->C->A)? How would you resolve such a situation? It is not so uncommon, after all.
Re: Problem using the Xtext serializer [message #1126227 is a reply to message #1126205] Sat, 05 October 2013 09:48 Go to previous messageGo to next message
Christian Dietrich is currently offline Christian DietrichFriend
Messages: 13264
Registered: July 2009
Senior Member
Sorry have no idea on that. Maybe it is not possible at all (I do not
have the time to digg on that but Xtext indexing is required for the
serialized somehow (I don't know if it is possible to skip that)

Maybe Moritz has an idea

--
Need training, onsite consulting or any other kind of help for Xtext?
Go visit http://xtext.itemis.com or send a mail to xtext at itemis dot de


Need professional support for Xtext, Xpand, EMF?
Go to: https://www.itemis.com/en/xtext/
Twitter : @chrdietrich
Blog : https://www.dietrich-it.de
Re: Problem using the Xtext serializer [message #1126247 is a reply to message #1126227] Sat, 05 October 2013 10:16 Go to previous messageGo to next message
Alan DW is currently offline Alan DWFriend
Messages: 119
Registered: March 2012
Senior Member
Well, basically, all information is available in the model, it is all there. As far as I know, the serializer draws its information from a kind of registry that is managed internally, and an object gets registered there when a resource (containing the object) is saved and the project is built. Rather than the internal registry, the serializer would have to use the instance model (provided by the XMI) directly. In the end, all it really has to do is to insert the (qualified) name of a referenced object whenever it encounters a non-containment reference. Actually, I'm quite surprised that I'm the first one to stumble upon this problem...

Another solution would be to not use the Xtext serializer alltogether and "manually" write the Model-to-Text transformation in Java/Xtend. I tried that, but I'm also getting some weird exceptions when I just write the DSL text into the files and try to build the project afterwards.

If anyone has a possible solution, please let me know. I'll continue to try things out and do research on the web, maybe I'll find something of use.

[Updated on: Sat, 05 October 2013 10:17]

Report message to a moderator

Re: Problem using the Xtext serializer [message #1126309 is a reply to message #1126247] Sat, 05 October 2013 11:51 Go to previous messageGo to next message
Moritz Eysholdt is currently offline Moritz EysholdtFriend
Messages: 161
Registered: July 2009
Location: Kiel, Germany
Senior Member
Alan DW wrote on Sat, 05 October 2013 11:12
But what about a scenario where I have cyclic cross references? For example, an EObject in File A references an EObject in File B, but there is also an EObject in B that references one in A? Or transitive cycles (A->B->C->A)? How would you resolve such a situation? It is not so uncommon, after all.


That's no problem for the serializer. The serializer does nothing except call the IScopeProvider for every cross reference. As long as the scope provider can provider a scope for a cross reference and the scope can provide a name for the referenced object, the serializer can save the file.

Christian Dietrich wrote on Sat, 05 October 2013 11:48
Sorry have no idea on that. Maybe it is not possible at all (I do not
have the time to dig on that but Xtext indexing is required for the
serialized somehow (I don't know if it is possible to skip that)


As mentioned above, the serializer uses scoping to determine the to-be-serialized names for cross references. However, Xtext's standard scoping implementation for cross-file references is based on the Xtext index. Thereby, serialization indirectly depends on the index.

If you don't like this behavior, you can override
org.eclipse.xtext.serializer.tokens.CrossReferenceSerializer.serializeCrossRef(EObject, CrossReference, EObject, INode, Acceptor)
and put in your own implementation. Maybe one that doesn't make use of scoping and/or the index and instead delegates to the IQualifiedNameProvider.

Alan DW wrote on Sat, 05 October 2013 12:16
Well, basically, all information is available in the model, it is all there. As far as I know, the serializer draws its information from a kind of registry that is managed internally, and an object gets registered there when a resource (containing the object) is saved and the project is built. Rather than the internal registry, the serializer would have to use the instance model (provided by the XMI) directly. In the end, all it really has to do is to insert the (qualified) name of a referenced object whenever it encounters a non-containment reference. Actually, I'm quite surprised that I'm the first one to stumble upon this problem...


What you describe as the "internal Registry" sounds like Xtext's index. There is no other "internal Registry" otherwise.
Yes, when the Xtext Builder runs, it collects QualifiedNames for all model elements and stores them in the index. And the Xtext builder runs when you trigger a full build or save a file (autobuild, which is an incremental build).

Is your use case to save a file that cross-references elements from unsaved files?
Re: Problem using the Xtext serializer [message #1127329 is a reply to message #1126309] Sun, 06 October 2013 14:37 Go to previous messageGo to next message
Alan DW is currently offline Alan DWFriend
Messages: 119
Registered: March 2012
Senior Member
Hi Moritz,

thanks for your response.

Quote:
Is your use case to save a file that cross-references elements from unsaved files?



Yes, it boils down to exactly this problem. I cannot guarantee that all cross-reference targets have been saved (and/or built) because I don't even know the grammar of the language in my use-case (it can be an arbitrary grammar). Of course, I can check all non-containment references in my EObjects and store the ones first that don't have any cross references, build the project, then save the ones which are only cross referencing the EObjects I just saved and built, and so on. But due to the generic nature of the tool I am building, I must consider the case when there are cyclic dependencies. For that reason, I cannot save and build those files first which do not have cross-references.

Do you have any hint on how to do this? If it can be avoided at all, I'd rather not touch on the scoping since I'm totally fine with the default scoping in the editor.


Basically, what happens in my tool is: I get an XMI file that adheres to an Xtext generated Metamodel, and I want to re-build the DSL code from that XMI file. Which model element ends up in which generated file is determined by the user, I have no control over that. In the end, I need to simultaneously save all of them, that's the goal.

[Updated on: Sun, 06 October 2013 14:45]

Report message to a moderator

Re: Problem using the Xtext serializer [message #1388544 is a reply to message #1127329] Thu, 26 June 2014 15:45 Go to previous messageGo to next message
Benoit Lelandais is currently offline Benoit LelandaisFriend
Messages: 19
Registered: November 2011
Junior Member
Hi Alan,

I have exactly the same problem and find no solution on forums...
Have you finally find a solution ?

Thanks for your help

Benoît
Re: Problem using the Xtext serializer [message #1389345 is a reply to message #1388544] Fri, 27 June 2014 17:19 Go to previous messageGo to next message
Alan DW is currently offline Alan DWFriend
Messages: 119
Registered: March 2012
Senior Member
Benoit Lelandais wrote on Thu, 26 June 2014 11:45
Hi Alan,

I have exactly the same problem and find no solution on forums...
Have you finally find a solution ?

Thanks for your help

Benoît


Hi,

I would love to share my solution with you, but unfortunately there is none - in the end, I've given up on using the Xtext serializer. There are certain problems associated with it:



  • The runtime performance of the generic serializer can never be as good as the performance of a serializer that is written especially for a single grammar. This is due to the amount of backtracking on the grammar rule tree that the generic serializer has to perform.
  • The generic serializer can't do anything about value conversions associated with your grammar. These converters are essentially one-way converters (from concrete syntax to model only).
  • The generic serializer was never intended to be used that way. It's purpose is to allow smaller changes (i.e. element name refactorings) to be applied to the model instead of the raw text. It is not intended to serialize entire models from scratch.


So in the end, what I did is I manually wrote a simple model-to-text generator using Xtend templates. Works like a charm and has none of the problems described above. The drawback is of course that this component is tightly coupled with the grammar. If you change the grammar, you must make the corresponding changes in your serializer as well. Plus, since the serializer works for *one* particular grammar, there is no way to support arbitrary grammars with it.

Maybe in the meantime there is a better solution from the Xtext dev team? I'm not really up-to-date with this anymore I'm afraid...


Kind regards,


Alan
Re: Problem using the Xtext serializer [message #1391708 is a reply to message #1389345] Tue, 01 July 2014 09:32 Go to previous message
Benoit Lelandais is currently offline Benoit LelandaisFriend
Messages: 19
Registered: November 2011
Junior Member
Hi Alan,

Thanks a lot for your answer.

I will write a serializer too. I waited for your answer before starting this because I have thought to the drawback and I would like to maintain only one code !

My model transformation is EMF-UML to DSL in Xtext. The EMF-UML model comes from Magicdraw tuned with a specific profile very close to the DSL. Even thought I will perhaps try to build a magicdraw plugin to convert the model before going into Eclipse to work on the DSL files.

Thanks again for your answer.

Kind regards,

Benoît
Previous Topic:How can I change IME mode when I start own language?
Next Topic:How to detect parse errors when loading from ResourceSet?
Goto Forum:
  


Current Time: Sun Dec 08 05:17:53 GMT 2019

Powered by FUDForum. Page generated in 0.02494 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top