Eclipse Community Forums: TMF (Xtext) » Using NeoEMF with Xtext

Home » Modeling » TMF (Xtext) » Using NeoEMF with Xtext(Question about using the Neo4j backend as a persistence layer for Xtext resources)

Show: Today's Messages :: Show Polls :: Message Navigator

Using NeoEMF with Xtext [message #1725599]

Fri, 04 March 2016 17:16

Harald Weiner

Messages: 23
Registered: January 2015
Location: Linz, Austria

Junior Member

Dear all,

sorry for the long description, I just wanted to explain my current situation.
I am currently struggling with the integration of NeoEMF (https://github.com/atlanmod/NeoEMF, former name Neo4EMF) with Xtext. I try to develop a MoDisco discoverer https://eclipse.org/MoDisco/ which scans directories and parses the individual source files with Xtext (which works quite well). The EMF root objects (e.g., with the name Foo) which are returned by Xtext are then stored as elements in an EList of the main EMF model. The problem is that the default XMI persistence layer is at some point not able to handle the amount of data any more and raises an OutOfMemory runtime exception. Now I am trying to migrate to NeoEMF which can use the NoSQL Neo4j graph database as a back-end. This also works quite well for my JUnit plug-in test cases. But when running on real code it also throws an OutOfMemory exception which may be preventable.

In my code I basically do the following (error handling and other stuff omitted):

private Foo parseFooUnitFor(final IFile iFile) {
   IResourceFactory resourceFactory = this.injector.getInstance(IResourceFactory.class);
   Resource.Factory.Registry.INSTANCE.getExtensionToFactoryMap().put(
					".foo", resourceFactory);
   final IProject iProject = iFile.getProject();
   final XtextResourceSetProvider provider = this.injector
				.getInstance(XtextResourceSetProvider.class);
   final XtextResourceSet resourceSet = (XtextResourceSet) provider
				.get(iProject);
   resourceSet.addLoadOption(XtextResource.OPTION_RESOLVE_ALL,
				Boolean.TRUE);
   final URI uri = URI.createURI(iFile.getLocationURI().toString());
   final Resource resource = resourceSet.getResource(uri, true);
   final Foo result = (Foo)resource.getContents().get(0);
   return result;
}

private void addFooUnitTo(final Model model, final IFile iFile) {
   final TranslationUnit unit = FooFactory.eINSTANCE
				.createTranslationUnit();
  unit.setPath(iFile.getLocationURI().toString());
  model.add(unit);
  Foo foo = parseFooUnitFor(iFile);

  MyLog.debug(XtextHandler.class, "foo='" + foo + "'");
  MyLog.debug(XtextHandler.class, "foo-res='" + foo.eResource()
				+ "'");
  MyLog.debug(XtextHandler.class, "foo-resSet='"
				+ foo.eResource().getResourceSet() + "'");
		
  MyLog.debug(XtextHandler.class, "unit='" + unit + "'");
  MyLog.debug(XtextHandler.class, "unit-res='" + unit.eResource() + "'");
  MyLog.debug(XtextHandler.class, "unit-resSet='"
				+ unit.eResource().getResourceSet() + "'");
  
  // the next line is where the OutOfMemory exception is thrown
  unit.setFoo(foo);
   
  MyLog.debug(XtextHandler.class, "");
  MyLog.debug(XtextHandler.class, "foo='" + foo + "'");
  MyLog.debug(XtextHandler.class, "foo-res='" + foo.eResource()
				+ "'");
   MyLog.debug(XtextHandler.class, "foo-resSet='"
				+ foo.eResource().getResourceSet() + "'");
  MyLog.debug(XtextHandler.class, "unit='" + unit + "'");
  MyLog.debug(XtextHandler.class, "unit-res='" + unit.eResource() + "'");
  MyLog.debug(XtextHandler.class, "unit-resSet='"
				+ unit.eResource().getResourceSet() + "'");
  
}

I am getting the following output (when no OutOfMemory Exception is thrown:

foo='foo.foo.impl.FooImpl@8a490b9b (eClass: org.eclipse.emf.ecore.impl.EClassImpl@7c2f2675 (name: Foo) (instanceClassName: null) (abstract: false, interface: false))'
foo-res='org.eclipse.xtext.linking.lazy.LazyLinkingResource@9196714 uri='file:/home/harry/workspace/runtime-EclipseApplication/Test/tmp-discover/home/harry/workspace/runtime-EclipseApplication/Test/src/b.foo''
foo-resSet='org.eclipse.xtext.resource.SynchronizedXtextResourceSet@3e2ef95f resources=[org.eclipse.xtext.linking.lazy.LazyLinkingResource@9196714 uri='file:/home/harry/workspace/runtime-EclipseApplication/Test/tmp-discover/home/harry/workspace/runtime-EclipseApplication/Test/src/b.foo']'
unit='foo.foo.impl.TranslationUnitImpl@ddb48f0d (eClass: org.eclipse.emf.ecore.impl.EClassImpl@396b060e (name: TranslationUnit) (instanceClassName: null) (abstract: false, interface: false))'
unit-res='fr.inria.atlanmod.neoemf.resources.impl.PersistentResourceImpl@183c95d2 uri='neo-blueprints:/home/harry/workspace/runtime-EclipseApplication/Test/tmp-discover/discover-2016-03-03-175607_c.neoemf''
unit-resSet='org.eclipse.emf.ecore.resource.impl.ResourceSetImpl@7cea2c4e resources=[fr.inria.atlanmod.neoemf.resources.impl.PersistentResourceImpl@183c95d2 uri='neo-blueprints:/home/harry/workspace/runtime-EclipseApplication/Test/tmp-discover/discover-2016-03-03-175607_c.neoemf']'

foo='foo.foo.impl.FooImpl@8a490b9b (eClass: org.eclipse.emf.ecore.impl.EClassImpl@7c2f2675 (name: Foo) (instanceClassName: null) (abstract: false, interface: false))'
foo-res='fr.inria.atlanmod.neoemf.resources.impl.PersistentResourceImpl@183c95d2 uri='neo-blueprints:/home/harry/workspace/runtime-EclipseApplication/Test/tmp-discover/discover-2016-03-03-175607_c.neoemf''
foo-resSet='org.eclipse.emf.ecore.resource.impl.ResourceSetImpl@7cea2c4e resources=[fr.inria.atlanmod.neoemf.resources.impl.PersistentResourceImpl@183c95d2 uri='neo-blueprints:/home/harry/workspace/runtime-EclipseApplication/Test/tmp-discover/discover-2016-03-03-175607_c.neoemf']'
unit='foo.foo.impl.TranslationUnitImpl@ddb48f0d (eClass: org.eclipse.emf.ecore.impl.EClassImpl@396b060e (name: TranslationUnit) (instanceClassName: null) (abstract: false, interface: false))'
unit-res='fr.inria.atlanmod.neoemf.resources.impl.PersistentResourceImpl@183c95d2 uri='neo-blueprints:/home/harry/workspace/runtime-EclipseApplication/Test/tmp-discover/discover-2016-03-03-175607_c.neoemf''
unit-resSet='org.eclipse.emf.ecore.resource.impl.ResourceSetImpl@7cea2c4e resources=[fr.inria.atlanmod.neoemf.resources.impl.PersistentResourceImpl@183c95d2 uri='neo-blueprints:/home/harry/workspace/runtime-EclipseApplication/Test/tmp-discover/discover-2016-03-03-175607_c.neoemf']'

So I believe that the problem is that somehow the Foo EObject element is copied over from the LazyLinkingResource to the PersistentResourceImpl which results in a Java heap overflow for very large Foo objects.

So my question is: would it be possible to directly create the resource of Foo as a PersistentResourceImpl resource while still being able to parse EMF models from text files?

Report message to a moderator

Re: Using NeoEMF with Xtext [message #1725614 is a reply to message #1725599]

Sat, 05 March 2016 00:05

Harald Weiner

Messages: 23
Registered: January 2015
Location: Linz, Austria

Junior Member

I have already taken a look at https://github.com/xpomul/cdo-xtext which is using CDO as a persistence layer, but unfortunately they are using CDO resources as input AND as output while I am trying to parse text files and store them in a NeoEMF database.

Report message to a moderator

Re: Using NeoEMF with Xtext [message #1725617 is a reply to message #1725599]

Sat, 05 March 2016 03:40

Sven Efftinge

Messages: 83
Registered: January 2016
Location: Kiel

Member

Harald Timeraider wrote on Fri, 04 March 2016 18:16

So my question is: would it be possible to directly create the resource of Foo as a PersistentResourceImpl resource while still being able to parse EMF models from text files?

No, Xtext requires the XtextResource.

May I ask why you need to store all the foos in a database or have them in memory at once? If it is because you need to find them quickly, have you thought about using the index for that?

Sven

Report message to a moderator

Re: Using NeoEMF with Xtext [message #1725639 is a reply to message #1725599]

Sat, 05 March 2016 15:28

Harald Weiner

Messages: 23
Registered: January 2015
Location: Linz, Austria

Junior Member

Quote:

why you need to store all the foos in a database or have them in memory at once?

No, I do not want them to be all in-memory at once. I just want them to be saved/persisted on the hard drive. NeoEMF supports Quote:

Lazy-loading mechanism: a model is loaded part by part while needed

and Quote:

Dirty saving: handle large models that haven't been persisted to avoid memory overhead

. Instead of saving/loading the EMF objects to/from XMI files, they are stored/re-loaded from a graph database. So NeoEMF saves them automatically and unloads the unnecessary parts to prevent OutOfMemory errors.

The problem seems to be that during the creation of the Foo object (as a PersistentResourceImpl) it is copied from the Xtext's SynchronizedXtextResourceSet over to the ResourceSet used by NeoEMF which means that there are two copies of the Foo object in memory at once resulting in an OutOfMemory exception.

The Foo object is anyway getting too big (with over 140.000 lines of code).

Quote:

No, Xtext requires the XtextResource.

So would it be possible to sub-class the XtextResource to let it create fr.inria.atlanmod.neoemf.resources.impl.PersistentResourceImpl resources instead?

Report message to a moderator

Re: Using NeoEMF with Xtext [message #1725718 is a reply to message #1725599]

Mon, 07 March 2016 09:25

Karsten Thoms

Messages: 762
Registered: July 2009
Location: Dortmund, Germany

Senior Member

Isn't it possible to increase memory settings in order to avoid the OOM exception?

How would you implement the behavior of PersistentResourceImpl in a subclass of XtextResource? Sounds as if you would have to merge the behavior into this sublass. And wouldn't NeoEMF expect to have instances of PersistentResourceImpl in the ResourceSet? You can't inherit from both, of course.

~Karsten

Need professional support for Xtext, EMF, Eclipse IDE?
Go to: http://devhub.karakun.com
Twitter : @kthoms
Blog : www.karsten-thoms.de

Report message to a moderator

Re: Using NeoEMF with Xtext [message #1726065 is a reply to message #1725718]

Wed, 09 March 2016 10:57

Harald Weiner

Messages: 23
Registered: January 2015
Location: Linz, Austria

Junior Member

Well, I made a work-around for this problem by doing the following (as my Foo EObject mainly consists of a list of Bar objects:

Foo secondFoo = FooFactory.eINSTANCE.createFoo();
final EList<Bar> list = foo.getBar();
final Iterator<Bar> it = list.getIterator();
for (int i = 0; it.hasNext(); i++) {
   final Bar bar = it.next();
   it.remove();
   secondFoo.getBar().add(bar);
   if ((i % 1000) == 0) {
      secondFoo.eResource().save(new HashMap<>());
   }
}

This solves the OutOfMemory exception problems for me.

I do not know how it would work with sub-classing XtextResource/XtextResource.Factory/XtextResourceSet, but maybe applying the adapter pattern might help... Just some thoughts. Anyway, I think that it would be better to have the persistence layer separated from the Xtext resource handling.

Report message to a moderator

Previous Topic:	Standalone command-line compiler
Next Topic:	generate a model .xmi xtext eclipse

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Thu Apr 25 04:10:08 GMT 2024

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter