Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF "Technology" (Ecore Tools, EMFatic, etc)  » [EMFStore] performances problems & model division
[EMFStore] performances problems & model division [message #643876] Wed, 08 December 2010 15:08 Go to next message
Guillaume P. is currently offline Guillaume P.Friend
Messages: 69
Registered: June 2010
Location: Toulouse, France
Member
Hi !

Today I have two questions (or three)... which are related Smile

First question:

I've spoken a little about it with Maximilian, but I had no example to provide... So I have now an example project showing the problem:

download it here
or here

The problem is: when I create a big model (about 50000 objects), I have big performances problems (Checkout is really slow, the sample interface is really slow too...).
To use my example, you have to launch an EMF Store server including the two plugins, and to launch an eclipse environment with the two plugins. Then, open the "Library Sets Manager" view (into the "Library Set Exemple" category), click on the "connect" button of the opened view, and click on the "+" button to create and fill automatically a model (My model is odd because I wanted to make something which reflect my model in my real project, with several root objects).

I think that all interesting code is into the [EMFStoreExample]/src/example/manager/EMFStoreConnection.jav a file (I can paste it here if you want ?).


Second problem:

My solution has to be able to manage up to 1 000 000 objects... but I see that on the server, everything for a given project is stored into the same xmi file. Is it possible to change the server behavior in order to save data into separated files for each root object ?
Else, is it possible to make an external reference to an object contained by another project ? And is there no problem if I have a big number of projects ?

Bonus problem:

In my example project, when adding a book to a library, I have the following error (while adding others objects to each others or to the project don't raises any exception). (You encounter the problem when clicking the "+" button, which fill automatically the created LibrarySet)

Quote:
!ENTRY org.unicase.metamodel 4 4 2010-12-08 15:52:27.125
!MESSAGE Project Change Observer threw an exception again, but it will not be detached.example.manager.EMFStoreConnection$ProjectChangeLis tener
!STACK 0
java.lang.NullPointerException
at org.unicase.workspace.impl.ProjectChangeTracker.notify(Proje ctChangeTracker.java:203)
at org.unicase.metamodel.impl.ProjectImpl$2.run(ProjectImpl.jav a:436)
at org.unicase.metamodel.impl.ProjectImpl.notifyProjectChangeOb servers(ProjectImpl.java:373)
at org.unicase.metamodel.impl.ProjectImpl.handleEMFNotification (ProjectImpl.java:439)
at org.unicase.metamodel.util.ProjectChangeNotifier.notifyChang ed(ProjectChangeNotifier.java:132)
at org.eclipse.emf.common.notify.impl.BasicNotifierImpl.eNotify (BasicNotifierImpl.java:380)
at org.eclipse.emf.ecore.util.EcoreEList.dispatchNotification(E coreEList.java:255)
at org.eclipse.emf.common.notify.impl.NotifyingListImpl.addUniq ue(NotifyingListImpl.java:300)
at org.eclipse.emf.common.util.AbstractEList.add(AbstractEList. java:307)
at example.manager.EMFStoreConnection.createLibraries(EMFStoreC onnection.java:259)
at example.manager.EMFStoreConnection.access$3(EMFStoreConnecti on.java:242)
at example.manager.EMFStoreConnection$3.doRun(EMFStoreConnectio n.java:230)
at example.manager.EMFStoreConnection$3.doRun(EMFStoreConnectio n.java:1)
at org.unicase.workspace.util.UnicaseCommandWithResult.doExecut e(UnicaseCommandWithResult.java:35)
at org.eclipse.emf.transaction.RecordingCommand.execute(Recordi ngCommand.java:135)
at org.eclipse.emf.common.command.BasicCommandStack.execute(Bas icCommandStack.java:84)
at org.eclipse.emf.transaction.impl.AbstractTransactionalComman dStack.basicExecute(AbstractTransactionalCommandStack.java:2 41)
at org.unicase.workspace.changeTracking.commands.EMFStoreTransa ctionalCommandStack.basicExecute(EMFStoreTransactionalComman dStack.java:49)
at org.eclipse.emf.transaction.impl.TransactionalCommandStackIm pl.doExecute(TransactionalCommandStackImpl.java:63)
at org.eclipse.emf.transaction.impl.AbstractTransactionalComman dStack.execute(AbstractTransactionalCommandStack.java:165)
at org.eclipse.emf.transaction.impl.AbstractTransactionalComman dStack.execute(AbstractTransactionalCommandStack.java:219)
at org.unicase.workspace.util.UnicaseCommandWithResult.run(Unic aseCommandWithResult.java:52)
at example.manager.EMFStoreConnection.createLibrarySet(EMFStore Connection.java:234)
at example.manager.LibrarySetManager.runAddLibrarySet(LibrarySe tManager.java:177)
at example.manager.LibrarySetManager.access$0(LibrarySetManager .java:153)
at example.manager.LibrarySetManager$1.run(LibrarySetManager.ja va:74)
at org.eclipse.jface.action.Action.runWithEvent(Action.java:498 )
at org.eclipse.jface.action.ActionContributionItem.handleWidget Selection(ActionContributionItem.java:584)
at org.eclipse.jface.action.ActionContributionItem.access$2(Act ionContributionItem.java:501)
at org.eclipse.jface.action.ActionContributionItem$6.handleEven t(ActionContributionItem.java:452)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java :84)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1053)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.ja va:4066)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java :3657)
at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.jav a:2640)
at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2604)
at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:24 38)
at org.eclipse.ui.internal.Workbench$7.run(Workbench.java:671)
at org.eclipse.core.databinding.observable.Realm.runWithDefault (Realm.java:332)
at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Work bench.java:664)
at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.j ava:149)
at org.eclipse.ui.internal.ide.application.IDEApplication.start (IDEApplication.java:115)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:196)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:110)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:79)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:369)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 619)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:574)
at org.eclipse.equinox.launcher.Main.run(Main.java:1407)
at org.eclipse.equinox.launcher.Main.main(Main.java:1383



[Edit] : I changed my example, because it was creating only 50 books (while my problem is when much more objects are stored). The new version creates now 50 000 books per library.

[Updated on: Thu, 09 December 2010 08:45]

Report message to a moderator

Re: [EMFStore] performances problems & model division [message #643988 is a reply to message #643876] Wed, 08 December 2010 23:11 Go to previous messageGo to next message
Maximilian Koegel is currently offline Maximilian KoegelFriend
Messages: 253
Registered: July 2009
Senior Member
Hi Guillaume,

Thank you very much for preparing the example, that´s great!
Comments inline...

Cheers,
Maximilian

> I've spoken a little about it with Maximilian, but I had no example to
> provide... So I have now an example project showing the problem:
>
> http://www.4shared.com/file/CJez2ElN/EMF_Store_Example_-_plu gins.html
Unfortunately the provided link does not work for me: "The file link
that you requested is not valid.". Maybe you can upload the file here?:
http://rapidshare.com/


> The problem is: when I create a big model (about 50000 objects), I have
> big performances problems (Checkout is really slow, the sample interface
> is really slow too...).
> To use my example, you have to launch an EMF Store server including the
> two plugins, and to launch an eclipse environment with the two plugins.
> Then, open the "Library Sets Manager" view (into the "Library Set
> Exemple" category), click on the "connect" button of the opened view,
> and click on the "+" button to create and fill automatically a model (My
> model is odd because I wanted to make something which reflect my model
> in my real project, with several root objects).
I have tested the EMFStore with 100.000+ elements recently (although
with a quite simple structure), so this is odd, probably something that
I did not take into consideration. I will look into it, once the example
goes online.

>
> Second problem:
>
> My solution has to be able to manage up to 1 000 000 objects... but I
> see that on the server, everything for a given project is stored into
> the same xmi file. Is it possible to change the server behavior in order
> to save data into separated files for each root object ?
Yes it is possible, there is one method in the server which is
responsible for this. There is no extension point to change it in custom
code yet, but I could add an extension point for this if required.

> Else, is it possible to make an external reference to an object
> contained by another project ? And is there no problem if I have a big
> number of projects ?
This is currently not possible, it is on the feature list, but not yet
scheduled.

> In my example project, when adding a book to a library, I have the
> following error (while adding others objects to each others or to the
> project don't raises any exception). (You encounter the problem when
> clicking the "+" button, which fill automatically the created LibrarySet)
Seems to be you custom class
example.manager.EMFStoreConnection$ProjectChangeListener which throws
the null pointer exception. But I can look into it once the uploaded
example is available.


>
> Quote:
>> !ENTRY org.unicase.metamodel 4 4 2010-12-08 15:52:27.125
>> !MESSAGE Project Change Observer threw an exception again, but it will
>> not be detached.example.manager.EMFStoreConnection$ProjectChangeLis tener
>> !STACK 0
>> java.lang.NullPointerException
>> at org.unicase.workspace.impl.ProjectChangeTracker.notify(Proje
>> ctChangeTracker.java:203)
>> at org.unicase.metamodel.impl.ProjectImpl$2.run(ProjectImpl.jav a:436)
>> at org.unicase.metamodel.impl.ProjectImpl.notifyProjectChangeOb
>> servers(ProjectImpl.java:373)
>> at org.unicase.metamodel.impl.ProjectImpl.handleEMFNotification
>> (ProjectImpl.java:439)
>> at org.unicase.metamodel.util.ProjectChangeNotifier.notifyChang
>> ed(ProjectChangeNotifier.java:132)
>> at org.eclipse.emf.common.notify.impl.BasicNotifierImpl.eNotify
>> (BasicNotifierImpl.java:380)
>> at org.eclipse.emf.ecore.util.EcoreEList.dispatchNotification(E
>> coreEList.java:255)
>> at org.eclipse.emf.common.notify.impl.NotifyingListImpl.addUniq
>> ue(NotifyingListImpl.java:300)
>> at org.eclipse.emf.common.util.AbstractEList.add(AbstractEList. java:307)
>> at example.manager.EMFStoreConnection.createLibraries(EMFStoreC
>> onnection.java:259)
>> at example.manager.EMFStoreConnection.access$3(EMFStoreConnecti
>> on.java:242)
>> at example.manager.EMFStoreConnection$3.doRun(EMFStoreConnectio
>> n.java:230)
>> at example.manager.EMFStoreConnection$3.doRun(EMFStoreConnectio n.java:1)
>> at org.unicase.workspace.util.UnicaseCommandWithResult.doExecut
>> e(UnicaseCommandWithResult.java:35)
>> at org.eclipse.emf.transaction.RecordingCommand.execute(Recordi
>> ngCommand.java:135)
>> at org.eclipse.emf.common.command.BasicCommandStack.execute(Bas
>> icCommandStack.java:84)
>> at org.eclipse.emf.transaction.impl.AbstractTransactionalComman
>> dStack.basicExecute(AbstractTransactionalCommandStack.java:2 41)
>> at org.unicase.workspace.changeTracking.commands.EMFStoreTransa
>> ctionalCommandStack.basicExecute(EMFStoreTransactionalComman
>> dStack.java:49)
>> at org.eclipse.emf.transaction.impl.TransactionalCommandStackIm
>> pl.doExecute(TransactionalCommandStackImpl.java:63)
>> at org.eclipse.emf.transaction.impl.AbstractTransactionalComman
>> dStack.execute(AbstractTransactionalCommandStack.java:165)
>> at org.eclipse.emf.transaction.impl.AbstractTransactionalComman
>> dStack.execute(AbstractTransactionalCommandStack.java:219)
>> at org.unicase.workspace.util.UnicaseCommandWithResult.run(Unic
>> aseCommandWithResult.java:52)
>> at example.manager.EMFStoreConnection.createLibrarySet(EMFStore
>> Connection.java:234)
>> at example.manager.LibrarySetManager.runAddLibrarySet(LibrarySe
>> tManager.java:177)
>> at example.manager.LibrarySetManager.access$0(LibrarySetManager
>> .java:153)
>> at example.manager.LibrarySetManager$1.run(LibrarySetManager.ja va:74)
>> at org.eclipse.jface.action.Action.runWithEvent(Action.java:498 )
>> at org.eclipse.jface.action.ActionContributionItem.handleWidget
>> Selection(ActionContributionItem.java:584)
>> at org.eclipse.jface.action.ActionContributionItem.access$2(Act
>> ionContributionItem.java:501)
>> at org.eclipse.jface.action.ActionContributionItem$6.handleEven
>> t(ActionContributionItem.java:452)
>> at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java :84)
>> at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1053)
>> at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.ja va:4066)
>> at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java :3657)
>> at org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.jav a:2640)
>> at org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2604)
>> at org.eclipse.ui.internal.Workbench.access$4(Workbench.java:24 38)
>> at org.eclipse.ui.internal.Workbench$7.run(Workbench.java:671)
>> at org.eclipse.core.databinding.observable.Realm.runWithDefault
>> (Realm.java:332)
>> at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Work
>> bench.java:664)
>> at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.j ava:149)
>> at org.eclipse.ui.internal.ide.application.IDEApplication.start
>> (IDEApplication.java:115)
>> at org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips
>> eAppHandle.java:196)
>> at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher
>> .runApplication(EclipseAppLauncher.java:110)
>> at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher
>> .start(EclipseAppLauncher.java:79)
>> at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS
>> tarter.java:369)
>> at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS
>> tarter.java:179)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>> at java.lang.reflect.Method.invoke(Unknown Source)
>> at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 619)
>> at org.eclipse.equinox.launcher.Main.basicRun(Main.java:574)
>> at org.eclipse.equinox.launcher.Main.run(Main.java:1407)
>> at org.eclipse.equinox.launcher.Main.main(Main.java:1383)
>
Re: [EMFStore] performances problems & model division [message #644041 is a reply to message #643988] Thu, 09 December 2010 08:44 Go to previous messageGo to next message
Guillaume P. is currently offline Guillaume P.Friend
Messages: 69
Registered: June 2010
Location: Toulouse, France
Member
Hi !

Rapidshare seems to be locked by my enterprise... I tried with another service.

new link here

>> My solution has to be able to manage up to 1 000 000 objects... but I
>> see that on the server, everything for a given project is stored into
>> the same xmi file. Is it possible to change the server behavior in order
>> to save data into separated files for each root object ?
> Yes it is possible, there is one method in the server which is
> responsible for this. There is no extension point to change it in custom
> code yet, but I could add an extension point for this if required.
Yes, I would be really interested by this feature (Can I sill making cross references between root objects (or their sub-objects) even if they are not stored in the same file on server ? Does it have some impact on the revision operation ?).

>> Else, is it possible to make an external reference to an object
>> contained by another project ? And is there no problem if I have a big
>> number of projects ?
> This is currently not possible, it is on the feature list, but not yet
> scheduled.
If the file division (just above), and performances are here, references between projects is not a need for me for now Smile
Re: [EMFStore] performances problems & model division [message #644048 is a reply to message #644041] Thu, 09 December 2010 09:18 Go to previous messageGo to next message
Guillaume P. is currently offline Guillaume P.Friend
Messages: 69
Registered: June 2010
Location: Toulouse, France
Member
> If the file division (just above), and performances are here, references between projects is not a need for me for now Smile

After consideration, it appears that the references between projects is something blocking for us, because our model contains several parts: a main part used by every one, and sub-parts using the main part. But sub-parts have not to be accessed by everyone, so making a global checkout including all sub-parts is not possible (some data are critical, we cannot store them on the computer of non authorized people): we should checkout sup-parts separately... which means into deferent projects with EMF Store, making the link with the main part (in another project) impossible :-/
Perhaps is there another existing solution to make separate checkouts ? But this missing feature is really a problem for us Sad
Re: [EMFStore] performances problems & model division [message #644074 is a reply to message #644048] Thu, 09 December 2010 10:00 Go to previous messageGo to next message
Maximilian Koegel is currently offline Maximilian KoegelFriend
Messages: 253
Registered: July 2009
Senior Member
Hi,

>> If the file division (just above), and performances are here,
>> references between projects is not a need for me for now Smile
>
> After consideration, it appears that the references between projects is
> something blocking for us, because our model contains several parts: a
> main part used by every one, and sub-parts using the main part. But
> sub-parts have not to be accessed by everyone, so making a global
> checkout including all sub-parts is not possible (some data are
> critical, we cannot store them on the computer of non authorized
> people): we should checkout sup-parts separately... which means into
> deferent projects with EMF Store, making the link with the main part (in
> another project) impossible :-/
> Perhaps is there another existing solution to make separate checkouts ?
> But this missing feature is really a problem for us :(
I do not think it is very complicated to implement. We have already
implemented something quite similar. It is more a question of resources.
If you need it fast, we can help you to implement it and you make a
contribution to the EMFStore concerning this feature. What is the time
frame you would need this feature in?


Cheers,
Maximilian
Re: [EMFStore] performances problems & model division [message #644075 is a reply to message #644074] Thu, 09 December 2010 10:13 Go to previous messageGo to next message
Maximilian Koegel is currently offline Maximilian KoegelFriend
Messages: 253
Registered: July 2009
Senior Member
Hi,

> Rapidshare seems to be locked by my enterprise... I tried with
another service.
> new link here
Thanks, now it worked, I will look into it tomorrow and get back to you!


>>> My solution has to be able to manage up to 1 000 000 objects... but I
>>> see that on the server, everything for a given project is stored into
>>> the same xmi file. Is it possible to change the server behavior in
order
>>> to save data into separated files for each root object ?
>> Yes it is possible, there is one method in the server which is
>> responsible for this. There is no extension point to change it in custom
>> code yet, but I could add an extension point for this if required.
> Yes, I would be really interested by this feature (Can I sill making
cross references between root objects (or their sub-objects) even if
they are not stored in the same file on server ? Does it have some
impact on the revision operation ?).
Yes cross references between root nodes are no problem even if they are
in different files, that is already the case on the client side.

Cheers,
Maximilian
Re: [EMFStore] performances problems & model division [message #648325 is a reply to message #644075] Wed, 12 January 2011 12:28 Go to previous message
Maximilian Koegel is currently offline Maximilian KoegelFriend
Messages: 253
Registered: July 2009
Senior Member
Hi,

I looked into your example. There is a two things that I changed:
- You created one model element (e.g. a book) and then added it to the
project directly and repeated this 50.000 times. The EMFStore will track
each addition as a single change, so it records 50.000 changes. It is a
lot faster to first create a library with 50.000 books and then to add
it to the project. It will then be tracked as one change only.
- NPE: You subclassed ProjectChangeTracker and added it as a
ProjectChangeObserver. The ProjectChangeTracker was not initialized
therefore it threw an exception. Anyway it is easier to just implement
the interface ProjectChangeObserver. This is what I did, the NPE is gone.

I have tested the example with 100.000 model elements now and it works
fine. It takes about a minute to initially commit the newly created
100.000 elements. Follow-up commits with less changes take about a
second on the same project.
I uploaded the changed code along with a Team Project Set of the current
trunk version of the EMFStore:
http://rapidshare.com/files/442159179/EMFStoreExampleRevised .zip

Hope this helps!

Cheers,
Maximilian

Am 09.12.2010 11:13, schrieb Maximilian Koegel:
> Hi,
>
> > Rapidshare seems to be locked by my enterprise... I tried with
> another service.
> > new link here
> Thanks, now it worked, I will look into it tomorrow and get back to you!
>
>
> >>> My solution has to be able to manage up to 1 000 000 objects... but I
> >>> see that on the server, everything for a given project is stored into
> >>> the same xmi file. Is it possible to change the server behavior in
> order
> >>> to save data into separated files for each root object ?
> >> Yes it is possible, there is one method in the server which is
> >> responsible for this. There is no extension point to change it in
> custom
> >> code yet, but I could add an extension point for this if required.
> > Yes, I would be really interested by this feature (Can I sill making
> cross references between root objects (or their sub-objects) even if
> they are not stored in the same file on server ? Does it have some
> impact on the revision operation ?).
> Yes cross references between root nodes are no problem even if they are
> in different files, that is already the case on the client side.
>
> Cheers,
> Maximilian
Previous Topic:[Announce] EMF Refactor 0.5.0 is available
Next Topic: Problem: failed to create task or type emf.JETCompiler
Goto Forum:
  


Current Time: Sat Oct 24 16:05:36 GMT 2020

Powered by FUDForum. Page generated in 0.02827 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top