Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » TMF (Xtext) » Using NeoEMF with Xtext(Question about using the Neo4j backend as a persistence layer for Xtext resources)
Using NeoEMF with Xtext [message #1725599] Fri, 04 March 2016 17:16 Go to next message
Harald Weiner is currently offline Harald WeinerFriend
Messages: 23
Registered: January 2015
Location: Linz, Austria
Junior Member
Dear all,

sorry for the long description, I just wanted to explain my current situation.
I am currently struggling with the integration of NeoEMF (https://github.com/atlanmod/NeoEMF, former name Neo4EMF) with Xtext. I try to develop a MoDisco discoverer https://eclipse.org/MoDisco/ which scans directories and parses the individual source files with Xtext (which works quite well). The EMF root objects (e.g., with the name Foo) which are returned by Xtext are then stored as elements in an EList of the main EMF model. The problem is that the default XMI persistence layer is at some point not able to handle the amount of data any more and raises an OutOfMemory runtime exception. Now I am trying to migrate to NeoEMF which can use the NoSQL Neo4j graph database as a back-end. This also works quite well for my JUnit plug-in test cases. But when running on real code it also throws an OutOfMemory exception which may be preventable.

In my code I basically do the following (error handling and other stuff omitted):
private Foo parseFooUnitFor(final IFile iFile) {
   IResourceFactory resourceFactory = this.injector.getInstance(IResourceFactory.class);
   Resource.Factory.Registry.INSTANCE.getExtensionToFactoryMap().put(
					".foo", resourceFactory);
   final IProject iProject = iFile.getProject();
   final XtextResourceSetProvider provider = this.injector
				.getInstance(XtextResourceSetProvider.class);
   final XtextResourceSet resourceSet = (XtextResourceSet) provider
				.get(iProject);
   resourceSet.addLoadOption(XtextResource.OPTION_RESOLVE_ALL,
				Boolean.TRUE);
   final URI uri = URI.createURI(iFile.getLocationURI().toString());
   final Resource resource = resourceSet.getResource(uri, true);
   final Foo result = (Foo)resource.getContents().get(0);
   return result;
}

private void addFooUnitTo(final Model model, final IFile iFile) {
   final TranslationUnit unit = FooFactory.eINSTANCE
				.createTranslationUnit();
  unit.setPath(iFile.getLocationURI().toString());
  model.add(unit);
  Foo foo = parseFooUnitFor(iFile);

  MyLog.debug(XtextHandler.class, "foo='" + foo + "'");
  MyLog.debug(XtextHandler.class, "foo-res='" + foo.eResource()
				+ "'");
  MyLog.debug(XtextHandler.class, "foo-resSet='"
				+ foo.eResource().getResourceSet() + "'");
		
  MyLog.debug(XtextHandler.class, "unit='" + unit + "'");
  MyLog.debug(XtextHandler.class, "unit-res='" + unit.eResource() + "'");
  MyLog.debug(XtextHandler.class, "unit-resSet='"
				+ unit.eResource().getResourceSet() + "'");
  
  // the next line is where the OutOfMemory exception is thrown
  unit.setFoo(foo);
   
  MyLog.debug(XtextHandler.class, "");
  MyLog.debug(XtextHandler.class, "foo='" + foo + "'");
  MyLog.debug(XtextHandler.class, "foo-res='" + foo.eResource()
				+ "'");
   MyLog.debug(XtextHandler.class, "foo-resSet='"
				+ foo.eResource().getResourceSet() + "'");
  MyLog.debug(XtextHandler.class, "unit='" + unit + "'");
  MyLog.debug(XtextHandler.class, "unit-res='" + unit.eResource() + "'");
  MyLog.debug(XtextHandler.class, "unit-resSet='"
				+ unit.eResource().getResourceSet() + "'");
  
}


I am getting the following output (when no OutOfMemory Exception is thrown:
foo='foo.foo.impl.FooImpl@8a490b9b (eClass: org.eclipse.emf.ecore.impl.EClassImpl@7c2f2675 (name: Foo) (instanceClassName: null) (abstract: false, interface: false))'
foo-res='org.eclipse.xtext.linking.lazy.LazyLinkingResource@9196714 uri='file:/home/harry/workspace/runtime-EclipseApplication/Test/tmp-discover/home/harry/workspace/runtime-EclipseApplication/Test/src/b.foo''
foo-resSet='org.eclipse.xtext.resource.SynchronizedXtextResourceSet@3e2ef95f resources=[org.eclipse.xtext.linking.lazy.LazyLinkingResource@9196714 uri='file:/home/harry/workspace/runtime-EclipseApplication/Test/tmp-discover/home/harry/workspace/runtime-EclipseApplication/Test/src/b.foo']'
unit='foo.foo.impl.TranslationUnitImpl@ddb48f0d (eClass: org.eclipse.emf.ecore.impl.EClassImpl@396b060e (name: TranslationUnit) (instanceClassName: null) (abstract: false, interface: false))'
unit-res='fr.inria.atlanmod.neoemf.resources.impl.PersistentResourceImpl@183c95d2 uri='neo-blueprints:/home/harry/workspace/runtime-EclipseApplication/Test/tmp-discover/discover-2016-03-03-175607_c.neoemf''
unit-resSet='org.eclipse.emf.ecore.resource.impl.ResourceSetImpl@7cea2c4e resources=[fr.inria.atlanmod.neoemf.resources.impl.PersistentResourceImpl@183c95d2 uri='neo-blueprints:/home/harry/workspace/runtime-EclipseApplication/Test/tmp-discover/discover-2016-03-03-175607_c.neoemf']'

foo='foo.foo.impl.FooImpl@8a490b9b (eClass: org.eclipse.emf.ecore.impl.EClassImpl@7c2f2675 (name: Foo) (instanceClassName: null) (abstract: false, interface: false))'
foo-res='fr.inria.atlanmod.neoemf.resources.impl.PersistentResourceImpl@183c95d2 uri='neo-blueprints:/home/harry/workspace/runtime-EclipseApplication/Test/tmp-discover/discover-2016-03-03-175607_c.neoemf''
foo-resSet='org.eclipse.emf.ecore.resource.impl.ResourceSetImpl@7cea2c4e resources=[fr.inria.atlanmod.neoemf.resources.impl.PersistentResourceImpl@183c95d2 uri='neo-blueprints:/home/harry/workspace/runtime-EclipseApplication/Test/tmp-discover/discover-2016-03-03-175607_c.neoemf']'
unit='foo.foo.impl.TranslationUnitImpl@ddb48f0d (eClass: org.eclipse.emf.ecore.impl.EClassImpl@396b060e (name: TranslationUnit) (instanceClassName: null) (abstract: false, interface: false))'
unit-res='fr.inria.atlanmod.neoemf.resources.impl.PersistentResourceImpl@183c95d2 uri='neo-blueprints:/home/harry/workspace/runtime-EclipseApplication/Test/tmp-discover/discover-2016-03-03-175607_c.neoemf''
unit-resSet='org.eclipse.emf.ecore.resource.impl.ResourceSetImpl@7cea2c4e resources=[fr.inria.atlanmod.neoemf.resources.impl.PersistentResourceImpl@183c95d2 uri='neo-blueprints:/home/harry/workspace/runtime-EclipseApplication/Test/tmp-discover/discover-2016-03-03-175607_c.neoemf']'


So I believe that the problem is that somehow the Foo EObject element is copied over from the LazyLinkingResource to the PersistentResourceImpl which results in a Java heap overflow for very large Foo objects.

So my question is: would it be possible to directly create the resource of Foo as a PersistentResourceImpl resource while still being able to parse EMF models from text files?
Re: Using NeoEMF with Xtext [message #1725614 is a reply to message #1725599] Sat, 05 March 2016 00:05 Go to previous messageGo to next message
Harald Weiner is currently offline Harald WeinerFriend
Messages: 23
Registered: January 2015
Location: Linz, Austria
Junior Member
I have already taken a look at https://github.com/xpomul/cdo-xtext which is using CDO as a persistence layer, but unfortunately they are using CDO resources as input AND as output while I am trying to parse text files and store them in a NeoEMF database.
Re: Using NeoEMF with Xtext [message #1725617 is a reply to message #1725599] Sat, 05 March 2016 03:40 Go to previous messageGo to next message
Sven Efftinge is currently offline Sven EfftingeFriend
Messages: 83
Registered: January 2016
Location: Kiel
Member

Harald Timeraider wrote on Fri, 04 March 2016 18:16

So my question is: would it be possible to directly create the resource of Foo as a PersistentResourceImpl resource while still being able to parse EMF models from text files?


No, Xtext requires the XtextResource.

May I ask why you need to store all the foos in a database or have them in memory at once? If it is because you need to find them quickly, have you thought about using the index for that?

Sven
Re: Using NeoEMF with Xtext [message #1725639 is a reply to message #1725599] Sat, 05 March 2016 15:28 Go to previous messageGo to next message
Harald Weiner is currently offline Harald WeinerFriend
Messages: 23
Registered: January 2015
Location: Linz, Austria
Junior Member
Quote:
why you need to store all the foos in a database or have them in memory at once?

No, I do not want them to be all in-memory at once. I just want them to be saved/persisted on the hard drive. NeoEMF supports Quote:
Lazy-loading mechanism: a model is loaded part by part while needed
and Quote:
Dirty saving: handle large models that haven't been persisted to avoid memory overhead
. Instead of saving/loading the EMF objects to/from XMI files, they are stored/re-loaded from a graph database. So NeoEMF saves them automatically and unloads the unnecessary parts to prevent OutOfMemory errors.

The problem seems to be that during the creation of the Foo object (as a PersistentResourceImpl) it is copied from the Xtext's SynchronizedXtextResourceSet over to the ResourceSet used by NeoEMF which means that there are two copies of the Foo object in memory at once resulting in an OutOfMemory exception.

The Foo object is anyway getting too big (with over 140.000 lines of code).

Quote:
No, Xtext requires the XtextResource.

So would it be possible to sub-class the XtextResource to let it create fr.inria.atlanmod.neoemf.resources.impl.PersistentResourceImpl resources instead?
Re: Using NeoEMF with Xtext [message #1725718 is a reply to message #1725599] Mon, 07 March 2016 09:25 Go to previous messageGo to next message
Karsten Thoms is currently offline Karsten ThomsFriend
Messages: 762
Registered: July 2009
Location: Dortmund, Germany
Senior Member

Isn't it possible to increase memory settings in order to avoid the OOM exception?

How would you implement the behavior of PersistentResourceImpl in a subclass of XtextResource? Sounds as if you would have to merge the behavior into this sublass. And wouldn't NeoEMF expect to have instances of PersistentResourceImpl in the ResourceSet? You can't inherit from both, of course.

~Karsten


Need professional support for Xtext, EMF, Eclipse IDE?
Go to: http://devhub.karakun.com
Twitter : @kthoms
Blog : www.karsten-thoms.de
Re: Using NeoEMF with Xtext [message #1726065 is a reply to message #1725718] Wed, 09 March 2016 10:57 Go to previous message
Harald Weiner is currently offline Harald WeinerFriend
Messages: 23
Registered: January 2015
Location: Linz, Austria
Junior Member
Well, I made a work-around for this problem by doing the following (as my Foo EObject mainly consists of a list of Bar objects:
Foo secondFoo = FooFactory.eINSTANCE.createFoo();
final EList<Bar> list = foo.getBar();
final Iterator<Bar> it = list.getIterator();
for (int i = 0; it.hasNext(); i++) {
   final Bar bar = it.next();
   it.remove();
   secondFoo.getBar().add(bar);
   if ((i % 1000) == 0) {
      secondFoo.eResource().save(new HashMap<>());
   }
}

This solves the OutOfMemory exception problems for me.

I do not know how it would work with sub-classing XtextResource/XtextResource.Factory/XtextResourceSet, but maybe applying the adapter pattern might help... Just some thoughts. Anyway, I think that it would be better to have the persistence layer separated from the Xtext resource handling.
Previous Topic:Standalone command-line compiler
Next Topic: generate a model .xmi xtext eclipse
Goto Forum:
  


Current Time: Fri Dec 09 13:17:18 GMT 2022

Powered by FUDForum. Page generated in 0.02975 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top