Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF "Technology" (Ecore Tools, EMFatic, etc)  » [EMFStore/ECP] Can't add more than approx. 58.000 model elements to project
[EMFStore/ECP] Can't add more than approx. 58.000 model elements to project [message #1242992] Mon, 10 February 2014 11:06 Go to next message
Steffen Lehnert is currently offline Steffen LehnertFriend
Messages: 22
Registered: October 2013
Junior Member
Hi,

I'm running a ECP (v.1.2.0) application (e3) with EMFStore (v.1.1.0) as a backend.
My current workspace project contains 57.531 EMF models and I can't add more than ~250 additional models to it, without running into "java.lang.OutOfMemoryError: GC overhead limit exceeded"-exceptions and crashing the ECP application.

I also deleted the entire workspace, re-imported my project and tried to add the models again. The exception, however; keeps on re-occurring.

When exported, the entire project-file has a size of about 55mb and I remember working with much larger projects back then with the "old" 0.9.3 EMFStore/ECP releases.

Is something messed up with my project/setup (most likely?) or did I hit a performance barrier of EMF/EMFStore/ECP?

Kind regards,
Steffen

Here is the stack trace that is printed to the console before the application crashes:
!ENTRY org.eclipse.emf.ecp.ui 4 0 2014-02-10 10:59:08.491
!MESSAGE GC overhead limit exceeded
!STACK 0
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.lang.AbstractStringBuilder.<init>(Unknown Source)
	at java.lang.StringBuilder.<init>(Unknown Source)
	at org.eclipse.core.runtime.Plugin.isDebugging(Plugin.java:493)
	at org.eclipse.emf.transaction.internal.Tracing.shouldTrace(Tracing.java:125)
	at org.eclipse.emf.transaction.internal.Tracing.shouldTrace(Tracing.java:137)
	at org.eclipse.emf.transaction.impl.TransactionalEditingDomainImpl.deactivate(TransactionalEditingDomainImpl.java:528)
	at org.eclipse.emf.transaction.impl.TransactionImpl.close(TransactionImpl.java:712)
	at org.eclipse.emf.transaction.impl.TransactionImpl.commit(TransactionImpl.java:474)
	at org.eclipse.emf.transaction.impl.TransactionalCommandStackImpl.doExecute(TransactionalCommandStackImpl.java:70)
	at org.eclipse.emf.transaction.impl.AbstractTransactionalCommandStack.execute(AbstractTransactionalCommandStack.java:165)
	at org.eclipse.emf.transaction.impl.AbstractTransactionalCommandStack.execute(AbstractTransactionalCommandStack.java:219)
	at org.eclipse.emf.emfstore.internal.client.transaction.EMFStoreTransactionalCommandStack.runEMFStoreCommand(EMFStoreTransactionalCommandStack.java:64)
	at org.eclipse.emf.emfstore.internal.client.transaction.EMFStoreTransactionalCommandStack.execute(EMFStoreTransactionalCommandStack.java:49)
	at org.eclipse.emf.emfstore.internal.client.model.util.AbstractEMFStoreCommand.aRun(AbstractEMFStoreCommand.java:108)
	at org.eclipse.emf.emfstore.internal.client.model.util.EMFStoreCommand.run(EMFStoreCommand.java:60)
	at org.eclipse.emf.emfstore.client.util.RunESCommand.run(RunESCommand.java:143)
	at org.eclipse.emf.ecp.emfstore.core.internal.EMFStoreProvider.fillChildren(EMFStoreProvider.java:159)
	at org.eclipse.emf.ecp.internal.core.ECPProviderRegistryImpl$ProviderDescriptor.fillChildren(ECPProviderRegistryImpl.java:213)
	at org.eclipse.emf.ecp.internal.ui.model.ECPContentProvider.fillChildren(ECPContentProvider.java:50)
	at org.eclipse.emf.ecp.internal.ui.model.ModelContentProvider.fillChildren(ModelContentProvider.java:65)
	at org.eclipse.emf.ecp.internal.ui.model.TreeContentProvider.fillChildrenDetectError(TreeContentProvider.java:193)
	at org.eclipse.emf.ecp.internal.ui.model.TreeContentProvider.getChildrenList(TreeContentProvider.java:185)
	at org.eclipse.emf.ecp.internal.ui.model.TreeContentProvider.hasChildren(TreeContentProvider.java:59)
	at org.eclipse.jface.viewers.AbstractTreeViewer.isExpandable(AbstractTreeViewer.java:2146)
	at org.eclipse.jface.viewers.TreeViewer.isExpandable(TreeViewer.java:588)
	at org.eclipse.jface.viewers.AbstractTreeViewer.isExpandable(AbstractTreeViewer.java:2176)
	at org.eclipse.jface.viewers.AbstractTreeViewer.updatePlus(AbstractTreeViewer.java:2858)
	at org.eclipse.jface.viewers.TreeViewer.updatePlus(TreeViewer.java:852)
	at org.eclipse.jface.viewers.AbstractTreeViewer.updateChildren(AbstractTreeViewer.java:2751)
	at org.eclipse.jface.viewers.AbstractTreeViewer.internalRefreshStruct(AbstractTreeViewer.java:1923)
	at org.eclipse.jface.viewers.TreeViewer.internalRefreshStruct(TreeViewer.java:721)
	at org.eclipse.jface.viewers.AbstractTreeViewer.internalRefresh(AbstractTreeViewer.java:1898)

Re: [EMFStore/ECP] Can't add more than approx. 58.000 model elements to project [message #1243010 is a reply to message #1242992] Mon, 10 February 2014 11:35 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 33113
Registered: July 2009
Senior Member
Steffen,

How big is your heap? You can increase it with a -Xmx VM parameter.
You can edit the eclipse.ini to modify that parameter.

On 10/02/2014 12:06 PM, Steffen Lehnert wrote:
> Hi,
>
> I'm running a ECP (v.1.2.0) application (e3) with EMFStore (v.1.1.0)
> as a backend.
> My current workspace project contains 57.531 EMF models and I can't
> add more than ~250 additional models to it, without running into
> "java.lang.OutOfMemoryError: GC overhead limit exceeded"-exceptions
> and crashing the ECP application.
>
> I also deleted the entire workspace, re-imported my project and tried
> to add the models again. The exception, however; keeps on re-occurring.
>
> When exported, the entire project-file has a size of about 55mb and I
> remember working with much larger projects back then with the "old"
> 0.9.3 EMFStore/ECP releases.
>
> Is something messed up with my project/setup (most likely?) or did I
> hit a performance barrier of EMF/EMFStore/ECP?
>
> Kind regards,
> Steffen
>
> Here is the stack trace that is printed to the console before the
> application crashes:
>
> !ENTRY org.eclipse.emf.ecp.ui 4 0 2014-02-10 10:59:08.491
> !MESSAGE GC overhead limit exceeded
> !STACK 0
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.lang.AbstractStringBuilder.<init>(Unknown Source)
> at java.lang.StringBuilder.<init>(Unknown Source)
> at org.eclipse.core.runtime.Plugin.isDebugging(Plugin.java:493)
> at
> org.eclipse.emf.transaction.internal.Tracing.shouldTrace(Tracing.java:125)
> at
> org.eclipse.emf.transaction.internal.Tracing.shouldTrace(Tracing.java:137)
> at
> org.eclipse.emf.transaction.impl.TransactionalEditingDomainImpl.deactivate(TransactionalEditingDomainImpl.java:528)
> at
> org.eclipse.emf.transaction.impl.TransactionImpl.close(TransactionImpl.java:712)
> at
> org.eclipse.emf.transaction.impl.TransactionImpl.commit(TransactionImpl.java:474)
> at
> org.eclipse.emf.transaction.impl.TransactionalCommandStackImpl.doExecute(TransactionalCommandStackImpl.java:70)
> at
> org.eclipse.emf.transaction.impl.AbstractTransactionalCommandStack.execute(AbstractTransactionalCommandStack.java:165)
> at
> org.eclipse.emf.transaction.impl.AbstractTransactionalCommandStack.execute(AbstractTransactionalCommandStack.java:219)
> at
> org.eclipse.emf.emfstore.internal.client.transaction.EMFStoreTransactionalCommandStack.runEMFStoreCommand(EMFStoreTransactionalCommandStack.java:64)
> at
> org.eclipse.emf.emfstore.internal.client.transaction.EMFStoreTransactionalCommandStack.execute(EMFStoreTransactionalCommandStack.java:49)
> at
> org.eclipse.emf.emfstore.internal.client.model.util.AbstractEMFStoreCommand.aRun(AbstractEMFStoreCommand.java:108)
> at
> org.eclipse.emf.emfstore.internal.client.model.util.EMFStoreCommand.run(EMFStoreCommand.java:60)
> at
> org.eclipse.emf.emfstore.client.util.RunESCommand.run(RunESCommand.java:143)
> at
> org.eclipse.emf.ecp.emfstore.core.internal.EMFStoreProvider.fillChildren(EMFStoreProvider.java:159)
> at
> org.eclipse.emf.ecp.internal.core.ECPProviderRegistryImpl$ProviderDescriptor.fillChildren(ECPProviderRegistryImpl.java:213)
> at
> org.eclipse.emf.ecp.internal.ui.model.ECPContentProvider.fillChildren(ECPContentProvider.java:50)
> at
> org.eclipse.emf.ecp.internal.ui.model.ModelContentProvider.fillChildren(ModelContentProvider.java:65)
> at
> org.eclipse.emf.ecp.internal.ui.model.TreeContentProvider.fillChildrenDetectError(TreeContentProvider.java:193)
> at
> org.eclipse.emf.ecp.internal.ui.model.TreeContentProvider.getChildrenList(TreeContentProvider.java:185)
> at
> org.eclipse.emf.ecp.internal.ui.model.TreeContentProvider.hasChildren(TreeContentProvider.java:59)
> at
> org.eclipse.jface.viewers.AbstractTreeViewer.isExpandable(AbstractTreeViewer.java:2146)
> at
> org.eclipse.jface.viewers.TreeViewer.isExpandable(TreeViewer.java:588)
> at
> org.eclipse.jface.viewers.AbstractTreeViewer.isExpandable(AbstractTreeViewer.java:2176)
> at
> org.eclipse.jface.viewers.AbstractTreeViewer.updatePlus(AbstractTreeViewer.java:2858)
> at
> org.eclipse.jface.viewers.TreeViewer.updatePlus(TreeViewer.java:852)
> at
> org.eclipse.jface.viewers.AbstractTreeViewer.updateChildren(AbstractTreeViewer.java:2751)
> at
> org.eclipse.jface.viewers.AbstractTreeViewer.internalRefreshStruct(AbstractTreeViewer.java:1923)
> at
> org.eclipse.jface.viewers.TreeViewer.internalRefreshStruct(TreeViewer.java:721)
> at
> org.eclipse.jface.viewers.AbstractTreeViewer.internalRefresh(AbstractTreeViewer.java:1898)
>
>


Ed Merks
Professional Support: https://www.macromodeling.com/
Re: [EMFStore/ECP] Can't add more than approx. 58.000 model elements to project [message #1243015 is a reply to message #1243010] Mon, 10 February 2014 11:50 Go to previous messageGo to next message
Steffen Lehnert is currently offline Steffen LehnertFriend
Messages: 22
Registered: October 2013
Junior Member
Hi Ed,

I've already increased the heapsize from 256mb to 1024mb in the eclipse.ini
I will try it again with 2048 or more...

Steffen
Re: [EMFStore/ECP] Can't add more than approx. 58.000 model elements to project [message #1243063 is a reply to message #1242992] Mon, 10 February 2014 13:16 Go to previous messageGo to next message
Jonas Helming is currently offline Jonas HelmingFriend
Messages: 699
Registered: July 2009
Senior Member
Hi Steffen,

there should be no "conceptional" limit here, the size is too small for
this. However, there might be a performance problem, I would guess
rather in the UI (e.g. the tree viewer), which your usage reveals.
Would it be possible to create a test case with a generic model (generic
names) which you could contribute? This would help to identify the issue.

Best Regards

Jonas

Am 10.02.2014 12:06, schrieb Steffen Lehnert:
> Hi,
>
> I'm running a ECP (v.1.2.0) application (e3) with EMFStore (v.1.1.0) as
> a backend.
> My current workspace project contains 57.531 EMF models and I can't add
> more than ~250 additional models to it, without running into
> "java.lang.OutOfMemoryError: GC overhead limit exceeded"-exceptions and
> crashing the ECP application.
>
> I also deleted the entire workspace, re-imported my project and tried to
> add the models again. The exception, however; keeps on re-occurring.
>
> When exported, the entire project-file has a size of about 55mb and I
> remember working with much larger projects back then with the "old"
> 0.9.3 EMFStore/ECP releases.
>
> Is something messed up with my project/setup (most likely?) or did I hit
> a performance barrier of EMF/EMFStore/ECP?
>
> Kind regards,
> Steffen
>
> Here is the stack trace that is printed to the console before the
> application crashes:
>
> !ENTRY org.eclipse.emf.ecp.ui 4 0 2014-02-10 10:59:08.491
> !MESSAGE GC overhead limit exceeded
> !STACK 0
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.lang.AbstractStringBuilder.<init>(Unknown Source)
> at java.lang.StringBuilder.<init>(Unknown Source)
> at org.eclipse.core.runtime.Plugin.isDebugging(Plugin.java:493)
> at
> org.eclipse.emf.transaction.internal.Tracing.shouldTrace(Tracing.java:125)
> at
> org.eclipse.emf.transaction.internal.Tracing.shouldTrace(Tracing.java:137)
> at
> org.eclipse.emf.transaction.impl.TransactionalEditingDomainImpl.deactivate(TransactionalEditingDomainImpl.java:528)
>
> at
> org.eclipse.emf.transaction.impl.TransactionImpl.close(TransactionImpl.java:712)
>
> at
> org.eclipse.emf.transaction.impl.TransactionImpl.commit(TransactionImpl.java:474)
>
> at
> org.eclipse.emf.transaction.impl.TransactionalCommandStackImpl.doExecute(TransactionalCommandStackImpl.java:70)
>
> at
> org.eclipse.emf.transaction.impl.AbstractTransactionalCommandStack.execute(AbstractTransactionalCommandStack.java:165)
>
> at
> org.eclipse.emf.transaction.impl.AbstractTransactionalCommandStack.execute(AbstractTransactionalCommandStack.java:219)
>
> at
> org.eclipse.emf.emfstore.internal.client.transaction.EMFStoreTransactionalCommandStack.runEMFStoreCommand(EMFStoreTransactionalCommandStack.java:64)
>
> at
> org.eclipse.emf.emfstore.internal.client.transaction.EMFStoreTransactionalCommandStack.execute(EMFStoreTransactionalCommandStack.java:49)
>
> at
> org.eclipse.emf.emfstore.internal.client.model.util.AbstractEMFStoreCommand.aRun(AbstractEMFStoreCommand.java:108)
>
> at
> org.eclipse.emf.emfstore.internal.client.model.util.EMFStoreCommand.run(EMFStoreCommand.java:60)
>
> at
> org.eclipse.emf.emfstore.client.util.RunESCommand.run(RunESCommand.java:143)
>
> at
> org.eclipse.emf.ecp.emfstore.core.internal.EMFStoreProvider.fillChildren(EMFStoreProvider.java:159)
>
> at
> org.eclipse.emf.ecp.internal.core.ECPProviderRegistryImpl$ProviderDescriptor.fillChildren(ECPProviderRegistryImpl.java:213)
>
> at
> org.eclipse.emf.ecp.internal.ui.model.ECPContentProvider.fillChildren(ECPContentProvider.java:50)
>
> at
> org.eclipse.emf.ecp.internal.ui.model.ModelContentProvider.fillChildren(ModelContentProvider.java:65)
>
> at
> org.eclipse.emf.ecp.internal.ui.model.TreeContentProvider.fillChildrenDetectError(TreeContentProvider.java:193)
>
> at
> org.eclipse.emf.ecp.internal.ui.model.TreeContentProvider.getChildrenList(TreeContentProvider.java:185)
>
> at
> org.eclipse.emf.ecp.internal.ui.model.TreeContentProvider.hasChildren(TreeContentProvider.java:59)
>
> at
> org.eclipse.jface.viewers.AbstractTreeViewer.isExpandable(AbstractTreeViewer.java:2146)
>
> at
> org.eclipse.jface.viewers.TreeViewer.isExpandable(TreeViewer.java:588)
> at
> org.eclipse.jface.viewers.AbstractTreeViewer.isExpandable(AbstractTreeViewer.java:2176)
>
> at
> org.eclipse.jface.viewers.AbstractTreeViewer.updatePlus(AbstractTreeViewer.java:2858)
>
> at org.eclipse.jface.viewers.TreeViewer.updatePlus(TreeViewer.java:852)
> at
> org.eclipse.jface.viewers.AbstractTreeViewer.updateChildren(AbstractTreeViewer.java:2751)
>
> at
> org.eclipse.jface.viewers.AbstractTreeViewer.internalRefreshStruct(AbstractTreeViewer.java:1923)
>
> at
> org.eclipse.jface.viewers.TreeViewer.internalRefreshStruct(TreeViewer.java:721)
>
> at
> org.eclipse.jface.viewers.AbstractTreeViewer.internalRefresh(AbstractTreeViewer.java:1898)
>
>
>
Re: [EMFStore/ECP] Can't add more than approx. 58.000 model elements to project [message #1243165 is a reply to message #1243063] Mon, 10 February 2014 16:02 Go to previous messageGo to next message
Steffen Lehnert is currently offline Steffen LehnertFriend
Messages: 22
Registered: October 2013
Junior Member
Hi Jonas,

as a test case you could use the following code snippet, which I posted in another thread: this one here (at the end of the the post).
You only have to adjust the for-loop of the "RunnableWithProgress" to create the aforementioned problem:
 monitor.beginTask("Add some elements to the project", 60000);
        for(int i = 0; i < 60000; i++)
        {                                      		
            project.getContents().add( MyModelFactory.eINSTANCE.createMyModel() );            
            monitor.worked(1);
        }


I can reproduce this behavior with any kind of "model"; simple ones containing only a few int or string properties or complex ones, like the UML metamodel.
Only the amount of models seems to matter.

I could also provide you with my entire project as a test case.
My project is based on custom meta-models (all open source and already "published" on SourceForge) and the Java meta-model supplied by the MoDisco-project.
Would that be of any help to you?

Steffen

Jonas Helming wrote on Mon, 10 February 2014 08:16
Hi Steffen,

there should be no "conceptional" limit here, the size is too small for
this. However, there might be a performance problem, I would guess
rather in the UI (e.g. the tree viewer), which your usage reveals.
Would it be possible to create a test case with a generic model (generic
names) which you could contribute? This would help to identify the issue.

Best Regards

Jonas
Re: [EMFStore/ECP] Can't add more than approx. 58.000 model elements to project [message #1243826 is a reply to message #1243165] Tue, 11 February 2014 13:53 Go to previous messageGo to next message
Jonas Helming is currently offline Jonas HelmingFriend
Messages: 699
Registered: July 2009
Senior Member
Hi,

if it works with every model, we can use the example model, we use for
all tests (bowling model). Does the issue only occur when the UI is
running or did you do this test headless?

Regards

Jonas

Am 10.02.2014 17:02, schrieb Steffen Lehnert:
> Hi Jonas,
>
> as a test case you could use the following code snippet, which I posted
> in another thread:
> http://www.eclipse.org/forums/index.php/mv/msg/555359/1142797/#msg_1142797.
> You only have to adjust the for-loop of the "RunnableWithProgress" to
> create the aforementioned problem:
>
> monitor.beginTask("Add some elements to the project", 60000);
> for(int i = 0; i < 60000; i++)
> {
> project.getContents().add(
> MyModelFactory.eINSTANCE.createMyModel() );
> monitor.worked(1);
> }
>
>
> I can reproduce this behavior with any kind of "model"; simple ones
> containing only a few int or string properties or complex ones, like the
> UML metamodel.
> Only the amount of models seems to matter.
>
> I could also provide you with my entire project as a test case.
> My project is based on custom meta-models (all open source and already
> "published" on SourceForge) and the Java meta-model supplied by the
> http://eclipse.org/MoDisco/-project.
> Would that be of any help to you?
>
> Steffen
>
> Jonas Helming wrote on Mon, 10 February 2014 08:16
>> Hi Steffen,
>>
>> there should be no "conceptional" limit here, the size is too small for
>> this. However, there might be a performance problem, I would guess
>> rather in the UI (e.g. the tree viewer), which your usage reveals.
>> Would it be possible to create a test case with a generic model (generic
>> names) which you could contribute? This would help to identify the issue.
>>
>> Best Regards
>>
>> Jonas
>
Re: [EMFStore/ECP] Can't add more than approx. 58.000 model elements to project [message #1244794 is a reply to message #1243826] Wed, 12 February 2014 22:25 Go to previous messageGo to next message
Steffen Lehnert is currently offline Steffen LehnertFriend
Messages: 22
Registered: October 2013
Junior Member
Hi Jonas,

I'm sorry for the delayed reply!
So far I have only tested this when the UI is running using a "org.eclipse.emf.ecp.application.e3.application".
Creating the models however does not take place in the UI thread. The models are created in a thread forked with a IRunnableWithProgress and than added to an existing ECP project.

Could this then be related to a concurrency issue I've reported a few months ago:
[ECP] Modifying content of ECPProject in non-UI thread?

Regards

Steffen

Jonas Helming wrote on Tue, 11 February 2014 08:53
Hi,

if it works with every model, we can use the example model, we use for
all tests (bowling model). Does the issue only occur when the UI is
running or did you do this test headless?

Regards

Jonas

[Updated on: Wed, 12 February 2014 22:47]

Report message to a moderator

Re: [EMFStore/ECP] Can't add more than approx. 58.000 model elements to project [message #1245085 is a reply to message #1244794] Thu, 13 February 2014 09:21 Go to previous messageGo to next message
Jonas Helming is currently offline Jonas HelmingFriend
Messages: 699
Registered: July 2009
Senior Member
Hi,

yes, that is what I meant. In general, the number of model elements
should not be an issue at all. However, the case to add this huge number
at once to the root level of a project as single operations seems a
little special. Is it generally possible to wrap this into one command
or composite operation?
There are three different components which could cause the issue here:

The ECP UI, more precisely the model explorer, it has to update the
TreeViewer. As the elements are all shown on the root level, it probably
makes a difference, if the project is unfolded or not. In any case, as
the default tree viewer is not a virtual tree viewer, I think even if
you can add this amount of elements on one level, working with it, will
not be fun.

The ECP core, may be related to some change notification. However, I
would not expect the problem here.

The EMFStore client persistence. For this it would be interesting,
whether you use a shared project or a local (not-shared) project. If it
is a shared project, your changes would all be recorded. While this is
not a problem in general, at least the history and the commit/update
dialogs would not show anything understandable for the user (58.000 add
operations). This could be avoided by wrapping all your operations into
one command.

So please tell me, whether you would be able to test this without the UI
(and also with using one wrapped command) and tell me the result.

Best Regards

Jonas



Am 12.02.2014 23:25, schrieb Steffen Lehnert:
> Hi Jonas,
>
> I'm sorry for the delayed reply!
> So far I have only tested this when the UI is running using a
> "org.eclipse.emf.ecp.application.e3.application".
> I'm not quite sure what exactly you mean with "test this headless"?
> Do you mean setting up a ECP project manually and populating it with the
> models, e.g. in a unit test case?
>
> Regards
>
> Steffen
>
> Jonas Helming wrote on Tue, 11 February 2014 08:53
>> Hi,
>>
>> if it works with every model, we can use the example model, we use for
>> all tests (bowling model). Does the issue only occur when the UI is
>> running or did you do this test headless?
>>
>> Regards
>>
>> Jonas
>
>
Re: [EMFStore/ECP] Can't add more than approx. 58.000 model elements to project [message #1245185 is a reply to message #1245085] Thu, 13 February 2014 12:33 Go to previous messageGo to next message
Steffen Lehnert is currently offline Steffen LehnertFriend
Messages: 22
Registered: October 2013
Junior Member
Hi,

thanks for the explanations!
To answer your questions:

> For this it would be interesting, whether you use a shared project or a local (not-shared) project.
I'm using local projects that have not yet been shared.

> So please tell me, whether you would be able to test this using one wrapped command and tell me the result.
I'm using a Callable for that purpose:
Callable<Void> call = new Callable<Void>()
{
    @Override
    public Void call() throws Exception
    {
        // add the models in here
    }
};
		
RunESCommand.run(call);


> So please tell me, whether you would be able to test this without the UI and tell me the result.
I set up a new JUnit test in which I created a new project and populated it with 6000 models of 20 different classes (300 instances of each class).
The whole test took 30 minutes.
During these 30 minutes, I noticed that after adding about 2700 models the performance dropped significantly.
Adding the first 2500 models was done in less than 3 minutes while adding the other half took about 27 minutes.
I've re-run this test 3 times and the results were the same.
However, there was no OutOfMemory or GC exception.

Now I will try this test again using 60k models.
I will report back in a couple of hours once the test is done...

Steffen
Re: [EMFStore/ECP] Can't add more than approx. 58.000 model elements to project [message #1245813 is a reply to message #1245185] Fri, 14 February 2014 09:56 Go to previous messageGo to next message
Jonas Helming is currently offline Jonas HelmingFriend
Messages: 699
Registered: July 2009
Senior Member
Hi,

that already takes much to long. Could you embed this test in
ECPProjectTests in org.eclipse.emf.ecp.core.test and contribute it (open
a BR)? That would be great! You could use any instance of the
"org.eclipse.emf.emfstore.examplemodel", factory would be "BowlingFactory".

Tahnks in advance!

Regards

Jonas

Am 13.02.2014 13:33, schrieb Steffen Lehnert:
> Hi,
>
> thanks for the explanations!
> To answer your questions:
>
>> For this it would be interesting, whether you use a shared project or
>> a local (not-shared) project.
> I'm using local projects that have not yet been shared.
>
>> So please tell me, whether you would be able to test this using one
>> wrapped command and tell me the result.
> I'm using a Callable for that purpose:
>
> Callable<Void> call = new Callable<Void>()
> {
> @Override
> public Void call() throws Exception
> {
> // add the models in here
> }
> };
>
> RunESCommand.run(call);
>
>
>> So please tell me, whether you would be able to test this without the
>> UI and tell me the result.
> I set up a new JUnit test in which I created a new project and populated
> it with 6000 models of 20 different classes (300 instances of each class).
> The whole test took 30 minutes.
> During these 30 minutes, I noticed that after adding about 2700 models
> the performance dropped significantly.
> Adding the first 2500 models was done in less than 3 minutes while
> adding the other half took about 27 minutes.
> I've re-run this test 3 times and the results were the same.
> However, there was no OutOfMemory or GC exception.
>
> Now I will try this test again using 60k models.
> I will report back in a couple of hours once the test is done...
>
> Steffen
Re: [EMFStore/ECP] Can't add more than approx. 58.000 model elements to project [message #1245998 is a reply to message #1245813] Fri, 14 February 2014 15:17 Go to previous messageGo to next message
Steffen Lehnert is currently offline Steffen LehnertFriend
Messages: 22
Registered: October 2013
Junior Member
Hi Jonas,

I will checkout and setup the ECP projects during the weekend.
Meanwhile, I was able to further simplify my test case while obtaining the same results.
Instead of adding any custom models, it is sufficient to add simple EClass-instances and it still takes about half an hour.
I triple-checked that there are no CPU or memory intensive processes running (all CPU cores are idling on my i7-3610QM with 16GB of RAM).

Steffen

package simpletest;

import java.util.concurrent.Callable;

import junit.framework.TestCase;

import org.eclipse.emf.ecore.EcoreFactory;
import org.eclipse.emf.ecp.core.ECPProject;
import org.eclipse.emf.ecp.core.util.ECPUtil;
import org.eclipse.emf.emfstore.client.util.RunESCommand;
import org.junit.Before;
import org.junit.Test;

public class AddingModelsTest extends TestCase
{	
    protected ECPProject project;
    
    @Before
    @Override
    public void setUp() throws Exception
    {
        super.setUp();
        
    	Callable<Void> call = new Callable<Void>()
    	{
			@Override
			public Void call() throws Exception
			{
				String providerName = "org.eclipse.emf.ecp.emfstore.provider";
		        String projectName = "TestProject";

		        project = ECPUtil.getECPProjectManager().createProject(ECPUtil.getECPProviderRegistry().getProvider(providerName), projectName, ECPUtil.createProperties());		        
		        project.open();
		        
				return null;
			}
		};
		
		RunESCommand.run(call);
    }
    
    @Test
    public void testAddModels()
    {    	
    	Callable<Void> call = new Callable<Void>()
    	{
			@Override
			public Void call() throws Exception
			{
				int numModels = 6000;
		        long time = System.currentTimeMillis();
		        
		        for(int i = 0; i < numModels; i++)
		        {
		        	project.getContents().add(EcoreFactory.eINSTANCE.createEClass());
		        	
		        	if( i%100 == 0 && i > 0 ) System.out.println(i+"/"+numModels+":"+(System.currentTimeMillis() - time)+"ms");
		        }
		        
				return null;
			}
		};
		
		RunESCommand.run(call);
    }
}


Jonas Helming wrote on Fri, 14 February 2014 04:56
Hi,

that already takes much to long. Could you embed this test in
ECPProjectTests in org.eclipse.emf.ecp.core.test and contribute it (open
a BR)? That would be great! You could use any instance of the
"org.eclipse.emf.emfstore.examplemodel", factory would be "BowlingFactory".

Tahnks in advance!

Regards

Jonas

Re: [EMFStore/ECP] Can't add more than approx. 58.000 model elements to project [message #1249432 is a reply to message #1245998] Tue, 18 February 2014 08:49 Go to previous messageGo to next message
Jonas Helming is currently offline Jonas HelmingFriend
Messages: 699
Registered: July 2009
Senior Member
Hi,

we found a potential reason for your issue. While we are working on a
"real" solution, could you kindly try the following work-around:
Please add ECPUtil.getECPProjectManager().getProjects() before you
create the new project?
Please tell me, if this speeds up things for you.

Best regards

Jonas

Am 14.02.2014 16:17, schrieb Steffen Lehnert:
> Hi Jonas,
>
> I will checkout and setup the ECP projects during the weekend.
> Meanwhile, I was able to further simplify my test case while obtaining
> the same results.
> Instead of adding any custom models, it is sufficient to add simple
> EClass-instances and it still takes about half an hour.
> I triple-checked that there are no CPU or memory intensive processes
> running (all CPU cores are idling on my i7-3610QM with 16GB of RAM).
> Steffen
>
>
> package simpletest;
>
> import java.util.concurrent.Callable;
>
> import junit.framework.TestCase;
>
> import org.eclipse.emf.ecore.EcoreFactory;
> import org.eclipse.emf.ecp.core.ECPProject;
> import org.eclipse.emf.ecp.core.util.ECPUtil;
> import org.eclipse.emf.emfstore.client.util.RunESCommand;
> import org.junit.Before;
> import org.junit.Test;
>
> public class AddingModelsTest extends TestCase
> {
> protected ECPProject project;
> @Before
> @Override
> public void setUp() throws Exception
> {
> super.setUp();
> Callable<Void> call = new Callable<Void>()
> {
> @Override
> public Void call() throws Exception
> {
> String providerName =
> "org.eclipse.emf.ecp.emfstore.provider";
> String projectName = "TestProject";
>
> project =
> ECPUtil.getECPProjectManager().createProject(ECPUtil.getECPProviderRegistry().getProvider(providerName),
> projectName, ECPUtil.createProperties());
> project.open();
> return null;
> }
> };
>
> RunESCommand.run(call);
> }
> @Test
> public void testAddModels()
> {
> Callable<Void> call = new Callable<Void>()
> {
> @Override
> public Void call() throws Exception
> {
> int numModels = 6000;
> long time = System.currentTimeMillis();
> for(int i = 0; i < numModels; i++)
> {
>
> project.getContents().add(EcoreFactory.eINSTANCE.createEClass());
>
> if( i%100 == 0 && i > 0 )
> System.out.println(i+"/"+numModels+":"+(System.currentTimeMillis() -
> time)+"ms");
> }
> return null;
> }
> };
>
> RunESCommand.run(call);
> }
> }
>
>
> Jonas Helming wrote on Fri, 14 February 2014 04:56
>> Hi,
>>
>> that already takes much to long. Could you embed this test in
>> ECPProjectTests in org.eclipse.emf.ecp.core.test and contribute it (open
>> a BR)? That would be great! You could use any instance of the
>> "org.eclipse.emf.emfstore.examplemodel", factory would be
>> "BowlingFactory".
>>
>> Tahnks in advance!
>>
>> Regards
>>
>> Jonas
>
>
Re: [EMFStore/ECP] Can't add more than approx. 58.000 model elements to project [message #1249496 is a reply to message #1249432] Tue, 18 February 2014 09:55 Go to previous messageGo to next message
Jonas Helming is currently offline Jonas HelmingFriend
Messages: 699
Registered: July 2009
Senior Member
Additional note:

We did some performance tests with this case. If you add the
work-around, described in the last post (due to this Bug
https://bugs.eclipse.org/bugs/show_bug.cgi?id=428370)
adding 60k or even 600k is no performance problem at all.


Best Regards

Jonas

Am 18.02.2014 09:49, schrieb Jonas Helming:
> Hi,
>
> we found a potential reason for your issue. While we are working on a
> "real" solution, could you kindly try the following work-around:
> Please add ECPUtil.getECPProjectManager().getProjects() before you
> create the new project?
> Please tell me, if this speeds up things for you.
>
> Best regards
>
> Jonas
>
> Am 14.02.2014 16:17, schrieb Steffen Lehnert:
>> Hi Jonas,
>>
>> I will checkout and setup the ECP projects during the weekend.
>> Meanwhile, I was able to further simplify my test case while obtaining
>> the same results.
>> Instead of adding any custom models, it is sufficient to add simple
>> EClass-instances and it still takes about half an hour.
>> I triple-checked that there are no CPU or memory intensive processes
>> running (all CPU cores are idling on my i7-3610QM with 16GB of RAM).
>> Steffen
>>
>>
>> package simpletest;
>>
>> import java.util.concurrent.Callable;
>>
>> import junit.framework.TestCase;
>>
>> import org.eclipse.emf.ecore.EcoreFactory;
>> import org.eclipse.emf.ecp.core.ECPProject;
>> import org.eclipse.emf.ecp.core.util.ECPUtil;
>> import org.eclipse.emf.emfstore.client.util.RunESCommand;
>> import org.junit.Before;
>> import org.junit.Test;
>>
>> public class AddingModelsTest extends TestCase
>> {
>> protected ECPProject project;
>> @Before
>> @Override
>> public void setUp() throws Exception
>> {
>> super.setUp();
>> Callable<Void> call = new Callable<Void>()
>> {
>> @Override
>> public Void call() throws Exception
>> {
>> String providerName =
>> "org.eclipse.emf.ecp.emfstore.provider";
>> String projectName = "TestProject";
>>
>> project =
>> ECPUtil.getECPProjectManager().createProject(ECPUtil.getECPProviderRegistry().getProvider(providerName),
>> projectName, ECPUtil.createProperties());
>> project.open();
>> return null;
>> }
>> };
>>
>> RunESCommand.run(call);
>> }
>> @Test
>> public void testAddModels()
>> {
>> Callable<Void> call = new Callable<Void>()
>> {
>> @Override
>> public Void call() throws Exception
>> {
>> int numModels = 6000;
>> long time = System.currentTimeMillis();
>> for(int i = 0; i < numModels; i++)
>> {
>>
>> project.getContents().add(EcoreFactory.eINSTANCE.createEClass());
>>
>> if( i%100 == 0 && i > 0 )
>> System.out.println(i+"/"+numModels+":"+(System.currentTimeMillis() -
>> time)+"ms");
>> }
>> return null;
>> }
>> };
>>
>> RunESCommand.run(call);
>> }
>> }
>>
>>
>> Jonas Helming wrote on Fri, 14 February 2014 04:56
>>> Hi,
>>>
>>> that already takes much to long. Could you embed this test in
>>> ECPProjectTests in org.eclipse.emf.ecp.core.test and contribute it (open
>>> a BR)? That would be great! You could use any instance of the
>>> "org.eclipse.emf.emfstore.examplemodel", factory would be
>>> "BowlingFactory".
>>>
>>> Tahnks in advance!
>>>
>>> Regards
>>>
>>> Jonas
>>
>>
>
Re: [EMFStore/ECP] Can't add more than approx. 58.000 model elements to project [message #1249497 is a reply to message #1242992] Tue, 18 February 2014 09:57 Go to previous messageGo to next message
Jonas Helming is currently offline Jonas HelmingFriend
Messages: 699
Registered: July 2009
Senior Member
A shortcut for everyone, who does not want to read the whole thread:

This is cause by Bug
https://bugs.eclipse.org/bugs/show_bug.cgi?id=428370 and only occurs, if
the project manager is not initialized yet. Until the bug is fixed,
calling ECPUtil.getECPProjectManager().getProjects() once before adding
elements solves the issue for us. In almost any application based on
ECP, this is already called on start-up (e.g. by the model explorer)

We did some performance tests with this case. Adding 60k or even 600k is
no performance problem at all.


Best Regards

Jonas


Am 10.02.2014 12:06, schrieb Steffen Lehnert:
> Hi,
>
> I'm running a ECP (v.1.2.0) application (e3) with EMFStore (v.1.1.0) as
> a backend.
> My current workspace project contains 57.531 EMF models and I can't add
> more than ~250 additional models to it, without running into
> "java.lang.OutOfMemoryError: GC overhead limit exceeded"-exceptions and
> crashing the ECP application.
>
> I also deleted the entire workspace, re-imported my project and tried to
> add the models again. The exception, however; keeps on re-occurring.
>
> When exported, the entire project-file has a size of about 55mb and I
> remember working with much larger projects back then with the "old"
> 0.9.3 EMFStore/ECP releases.
>
> Is something messed up with my project/setup (most likely?) or did I hit
> a performance barrier of EMF/EMFStore/ECP?
>
> Kind regards,
> Steffen
>
> Here is the stack trace that is printed to the console before the
> application crashes:
>
> !ENTRY org.eclipse.emf.ecp.ui 4 0 2014-02-10 10:59:08.491
> !MESSAGE GC overhead limit exceeded
> !STACK 0
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> at java.lang.AbstractStringBuilder.<init>(Unknown Source)
> at java.lang.StringBuilder.<init>(Unknown Source)
> at org.eclipse.core.runtime.Plugin.isDebugging(Plugin.java:493)
> at
> org.eclipse.emf.transaction.internal.Tracing.shouldTrace(Tracing.java:125)
> at
> org.eclipse.emf.transaction.internal.Tracing.shouldTrace(Tracing.java:137)
> at
> org.eclipse.emf.transaction.impl.TransactionalEditingDomainImpl.deactivate(TransactionalEditingDomainImpl.java:528)
>
> at
> org.eclipse.emf.transaction.impl.TransactionImpl.close(TransactionImpl.java:712)
>
> at
> org.eclipse.emf.transaction.impl.TransactionImpl.commit(TransactionImpl.java:474)
>
> at
> org.eclipse.emf.transaction.impl.TransactionalCommandStackImpl.doExecute(TransactionalCommandStackImpl.java:70)
>
> at
> org.eclipse.emf.transaction.impl.AbstractTransactionalCommandStack.execute(AbstractTransactionalCommandStack.java:165)
>
> at
> org.eclipse.emf.transaction.impl.AbstractTransactionalCommandStack.execute(AbstractTransactionalCommandStack.java:219)
>
> at
> org.eclipse.emf.emfstore.internal.client.transaction.EMFStoreTransactionalCommandStack.runEMFStoreCommand(EMFStoreTransactionalCommandStack.java:64)
>
> at
> org.eclipse.emf.emfstore.internal.client.transaction.EMFStoreTransactionalCommandStack.execute(EMFStoreTransactionalCommandStack.java:49)
>
> at
> org.eclipse.emf.emfstore.internal.client.model.util.AbstractEMFStoreCommand.aRun(AbstractEMFStoreCommand.java:108)
>
> at
> org.eclipse.emf.emfstore.internal.client.model.util.EMFStoreCommand.run(EMFStoreCommand.java:60)
>
> at
> org.eclipse.emf.emfstore.client.util.RunESCommand.run(RunESCommand.java:143)
>
> at
> org.eclipse.emf.ecp.emfstore.core.internal.EMFStoreProvider.fillChildren(EMFStoreProvider.java:159)
>
> at
> org.eclipse.emf.ecp.internal.core.ECPProviderRegistryImpl$ProviderDescriptor.fillChildren(ECPProviderRegistryImpl.java:213)
>
> at
> org.eclipse.emf.ecp.internal.ui.model.ECPContentProvider.fillChildren(ECPContentProvider.java:50)
>
> at
> org.eclipse.emf.ecp.internal.ui.model.ModelContentProvider.fillChildren(ModelContentProvider.java:65)
>
> at
> org.eclipse.emf.ecp.internal.ui.model.TreeContentProvider.fillChildrenDetectError(TreeContentProvider.java:193)
>
> at
> org.eclipse.emf.ecp.internal.ui.model.TreeContentProvider.getChildrenList(TreeContentProvider.java:185)
>
> at
> org.eclipse.emf.ecp.internal.ui.model.TreeContentProvider.hasChildren(TreeContentProvider.java:59)
>
> at
> org.eclipse.jface.viewers.AbstractTreeViewer.isExpandable(AbstractTreeViewer.java:2146)
>
> at
> org.eclipse.jface.viewers.TreeViewer.isExpandable(TreeViewer.java:588)
> at
> org.eclipse.jface.viewers.AbstractTreeViewer.isExpandable(AbstractTreeViewer.java:2176)
>
> at
> org.eclipse.jface.viewers.AbstractTreeViewer.updatePlus(AbstractTreeViewer.java:2858)
>
> at org.eclipse.jface.viewers.TreeViewer.updatePlus(TreeViewer.java:852)
> at
> org.eclipse.jface.viewers.AbstractTreeViewer.updateChildren(AbstractTreeViewer.java:2751)
>
> at
> org.eclipse.jface.viewers.AbstractTreeViewer.internalRefreshStruct(AbstractTreeViewer.java:1923)
>
> at
> org.eclipse.jface.viewers.TreeViewer.internalRefreshStruct(TreeViewer.java:721)
>
> at
> org.eclipse.jface.viewers.AbstractTreeViewer.internalRefresh(AbstractTreeViewer.java:1898)
>
>
>
Re: [EMFStore/ECP] Can't add more than approx. 58.000 model elements to project [message #1249543 is a reply to message #1249432] Tue, 18 February 2014 10:56 Go to previous messageGo to next message
Steffen Lehnert is currently offline Steffen LehnertFriend
Messages: 22
Registered: October 2013
Junior Member
Hi Jonas,

thank you very much for investigating this issue!
I applied your work-around on my unit tests and they work like a charm now.
As you said, there is no difference between adding 6000, 60k or even 600k for me either.

Thanks again!

Best regards

Steffen

Jonas Helming wrote on Tue, 18 February 2014 03:49
Hi,

we found a potential reason for your issue. While we are working on a
"real" solution, could you kindly try the following work-around:
Please add ECPUtil.getECPProjectManager().getProjects() before you
create the new project?
Please tell me, if this speeds up things for you.

Best regards

Jonas

Re: [EMFStore/ECP] Can't add more than approx. 58.000 model elements to project [message #1249636 is a reply to message #1249543] Tue, 18 February 2014 13:10 Go to previous message
Jonas Helming is currently offline Jonas HelmingFriend
Messages: 699
Registered: July 2009
Senior Member
Please watch https://bugs.eclipse.org/bugs/show_bug.cgi?id=428370
to remove the work-arounf, once it is fixed...

Am 18.02.2014 11:56, schrieb Steffen Lehnert:
> Hi Jonas,
>
> thank you very much for investigating this issue!
> I applied your work-around on my unit tests and they work like a charm now.
> As you said, there is no difference between adding 6000, 60k or even
> 600k for me either.
>
> Thanks again!
>
> Best regards
>
> Steffen
>
> Jonas Helming wrote on Tue, 18 February 2014 03:49
>> Hi,
>>
>> we found a potential reason for your issue. While we are working on a
>> "real" solution, could you kindly try the following work-around:
>> Please add ECPUtil.getECPProjectManager().getProjects() before you
>> create the new project?
>> Please tell me, if this speeds up things for you.
>>
>> Best regards
>>
>> Jonas
>
>
Previous Topic:How can i set save option for element as QualifiedName ?
Next Topic:[Teneo] How to get the audit history of a deleted entity?
Goto Forum:
  


Current Time: Fri Mar 29 15:57:08 GMT 2024

Powered by FUDForum. Page generated in 0.04214 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top