Eclipse Community Forums: EMF » [CDO]remove performance

Help

Home

Home » Modeling » EMF » [CDO]remove performance

Show: Today's Messages :: Show Polls :: Message Navigator

[CDO]remove performance [message #426597]

Thu, 15 January 2009 14:25

David Bonneau

Messages: 28
Registered: July 2009

Junior Member

Hi,

I have a little performance problem with CDO. I have a simple model :
one Folder could contain many Files.

With a MemStore I add 30 000 instances of File in one instance of
Folder. It takes around 2 seconds. Now I remove each File of the Folder
it takes ... more than 90 seconds !

Here is the code :

Folder folder = ExempleFactory.eINSTANCE.createFolder();

for(int i = 0; i < 30000; ++i){
Fichier file = RessourceFactory.eINSTANCE.createFichier();
((BasicEStoreEList)(folder.getFiles())).addUnique(file);
}
transaction.commit();
//=>2 seconds
for(int i = 29999; i >= 0; --i){
EcoreUtil.remove(folder.getFiles().get(i));
}
//=>90 seconds
transaction.commit();

Ok I know that I could use a clear instead of 30 000 removes. But in
my real application, in the second loop I have a condition to keep few
elements.

I try to dig in the code and I find that all the time is consumed by
CDOListFeatureDeltaImpl.reconstrucAddedIndices()

If I try to commit the transaction more often (every 1000 remove) it's
faster : around 20 seconds.

I would like to know if I'm doing something wrong ? If I could do
something to improve performance ?

Thanks

David

Report message to a moderator

Re: [CDO]remove performance [message #426599 is a reply to message #426597]

Thu, 15 January 2009 14:44

Simon Mc Duff

Messages: 596
Registered: July 2009

Senior Member

Hi David,

Let me try it and profile to see and we can do something better.
Performance is very important and we didn't have measure that case.

I think you are not doing anything wrong.

I will be back soon with more answers!!!

Simon

David Bonneau wrote:
> Hi,
>
> I have a little performance problem with CDO. I have a simple model :
> one Folder could contain many Files.
>
> With a MemStore I add 30 000 instances of File in one instance of
> Folder. It takes around 2 seconds. Now I remove each File of the Folder
> it takes ... more than 90 seconds !
>
> Here is the code :
>
> Folder folder = ExempleFactory.eINSTANCE.createFolder();
>
> for(int i = 0; i < 30000; ++i){
> Fichier file = RessourceFactory.eINSTANCE.createFichier();
> ((BasicEStoreEList)(folder.getFiles())).addUnique(file);
> }
> transaction.commit();
> //=>2 seconds
> for(int i = 29999; i >= 0; --i){
> EcoreUtil.remove(folder.getFiles().get(i));
> }
> //=>90 seconds
> transaction.commit();
>
> Ok I know that I could use a clear instead of 30 000 removes. But in my
> real application, in the second loop I have a condition to keep few
> elements.
>
> I try to dig in the code and I find that all the time is consumed by
> CDOListFeatureDeltaImpl.reconstrucAddedIndices()
>
> If I try to commit the transaction more often (every 1000 remove) it's
> faster : around 20 seconds.
>
> I would like to know if I'm doing something wrong ? If I could do
> something to improve performance ?
>
> Thanks
>
> David

Report message to a moderator

Re: [CDO]remove performance [message #426603 is a reply to message #426599]

Thu, 15 January 2009 14:55

David Bonneau

Messages: 28
Registered: July 2009

Junior Member

I have a small unit case that reproduce the "problem". If you want I can
send it to you.

David

Simon McDuff a écrit :
> Hi David,
>
> Let me try it and profile to see and we can do something better.
> Performance is very important and we didn't have measure that case.
>
> I think you are not doing anything wrong.
>
> I will be back soon with more answers!!!
>
> Simon
>
>
> David Bonneau wrote:
>> Hi,
>>
>> I have a little performance problem with CDO. I have a simple model :
>> one Folder could contain many Files.
>>
>> With a MemStore I add 30 000 instances of File in one instance of
>> Folder. It takes around 2 seconds. Now I remove each File of the
>> Folder it takes ... more than 90 seconds !
>>
>> Here is the code :
>>
>> Folder folder = ExempleFactory.eINSTANCE.createFolder();
>>
>> for(int i = 0; i < 30000; ++i){
>> Fichier file = RessourceFactory.eINSTANCE.createFichier();
>> ((BasicEStoreEList)(folder.getFiles())).addUnique(file);
>> }
>> transaction.commit();
>> //=>2 seconds
>> for(int i = 29999; i >= 0; --i){
>> EcoreUtil.remove(folder.getFiles().get(i));
>> }
>> //=>90 seconds
>> transaction.commit();
>>
>> Ok I know that I could use a clear instead of 30 000 removes. But in
>> my real application, in the second loop I have a condition to keep few
>> elements.
>>
>> I try to dig in the code and I find that all the time is consumed by
>> CDOListFeatureDeltaImpl.reconstrucAddedIndices()
>>
>> If I try to commit the transaction more often (every 1000 remove)
>> it's faster : around 20 seconds.
>>
>> I would like to know if I'm doing something wrong ? If I could do
>> something to improve performance ?
>>
>> Thanks
>>
>> David

Report message to a moderator

Re: [CDO]remove performance [message #426608 is a reply to message #426603]

Thu, 15 January 2009 15:18

Simon Mc Duff

Messages: 596
Registered: July 2009

Senior Member

Yes please. SO I can see your ecore (It will help me to know how you
defined relation between Folder and FIle.)

My bottleneck seems to be at another place than you.

Also can you provide me the version of CDO you are using ?

Simon

David Bonneau wrote:
> I have a small unit case that reproduce the "problem". If you want I can
> send it to you.
>
> David
>
> Simon McDuff a écrit :
>> Hi David,
>>
>> Let me try it and profile to see and we can do something better.
>> Performance is very important and we didn't have measure that case.
>>
>> I think you are not doing anything wrong.
>>
>> I will be back soon with more answers!!!
>>
>> Simon
>>
>>
>> David Bonneau wrote:
>>> Hi,
>>>
>>> I have a little performance problem with CDO. I have a simple model
>>> : one Folder could contain many Files.
>>>
>>> With a MemStore I add 30 000 instances of File in one instance of
>>> Folder. It takes around 2 seconds. Now I remove each File of the
>>> Folder it takes ... more than 90 seconds !
>>>
>>> Here is the code :
>>>
>>> Folder folder = ExempleFactory.eINSTANCE.createFolder();
>>>
>>> for(int i = 0; i < 30000; ++i){
>>> Fichier file = RessourceFactory.eINSTANCE.createFichier();
>>> ((BasicEStoreEList)(folder.getFiles())).addUnique(file);
>>> }
>>> transaction.commit();
>>> //=>2 seconds
>>> for(int i = 29999; i >= 0; --i){
>>> EcoreUtil.remove(folder.getFiles().get(i));
>>> }
>>> //=>90 seconds
>>> transaction.commit();
>>>
>>> Ok I know that I could use a clear instead of 30 000 removes. But in
>>> my real application, in the second loop I have a condition to keep
>>> few elements.
>>>
>>> I try to dig in the code and I find that all the time is consumed by
>>> CDOListFeatureDeltaImpl.reconstrucAddedIndices()
>>>
>>> If I try to commit the transaction more often (every 1000 remove)
>>> it's faster : around 20 seconds.
>>>
>>> I would like to know if I'm doing something wrong ? If I could do
>>> something to improve performance ?
>>>
>>> Thanks
>>>
>>> David

Report message to a moderator

Re: [CDO]remove performance [message #426609 is a reply to message #426603]

Thu, 15 January 2009 15:26

Simon Mc Duff

Messages: 596
Registered: July 2009

Senior Member

Also

Does it takes 90 seconds or 2 secs ?
for(int i = 29999; i >= 0; --i){
EcoreUtil.remove(folder.getFiles().get(i));
}

I found out that if a EOpposite is defined and Container == true (your
File Class) then it will take a lot of time...it should not...my first
impression is EMF doesn't handle one case....

Anyway, for now I will look if
CDOListFeatureDeltaImpl.reconstrucAddedIndices() is the problem. HOw did
you find CDOListFeatureDeltaImpl.reconstrucAddedIndices()? In my test
case the bottleneck isn't there ?:-)

Simon

David Bonneau wrote:
> I have a small unit case that reproduce the "problem". If you want I can
> send it to you.
>
> David
>

> Simon McDuff a écrit :
>> Hi David,
>>
>> Let me try it and profile to see and we can do something better.
>> Performance is very important and we didn't have measure that case.
>>
>> I think you are not doing anything wrong.
>>
>> I will be back soon with more answers!!!
>>
>> Simon
>>
>>
>> David Bonneau wrote:
>>> Hi,
>>>
>>> I have a little performance problem with CDO. I have a simple model
>>> : one Folder could contain many Files.
>>>
>>> With a MemStore I add 30 000 instances of File in one instance of
>>> Folder. It takes around 2 seconds. Now I remove each File of the
>>> Folder it takes ... more than 90 seconds !
>>>
>>> Here is the code :
>>>
>>> Folder folder = ExempleFactory.eINSTANCE.createFolder();
>>>
>>> for(int i = 0; i < 30000; ++i){
>>> Fichier file = RessourceFactory.eINSTANCE.createFichier();
>>> ((BasicEStoreEList)(folder.getFiles())).addUnique(file);
>>> }
>>> transaction.commit();
>>> //=>2 seconds
>>> for(int i = 29999; i >= 0; --i){
>>> EcoreUtil.remove(folder.getFiles().get(i));
>>> }
>>> //=>90 seconds
>>> transaction.commit();
>>>
>>> Ok I know that I could use a clear instead of 30 000 removes. But in
>>> my real application, in the second loop I have a condition to keep
>>> few elements.
>>>
>>> I try to dig in the code and I find that all the time is consumed by
>>> CDOListFeatureDeltaImpl.reconstrucAddedIndices()
>>>
>>> If I try to commit the transaction more often (every 1000 remove)
>>> it's faster : around 20 seconds.
>>>
>>> I would like to know if I'm doing something wrong ? If I could do
>>> something to improve performance ?
>>>
>>> Thanks
>>>
>>> David

Report message to a moderator

Re: [CDO]remove performance [message #426610 is a reply to message #426603]

Thu, 15 January 2009 15:32

Simon Mc Duff

Messages: 596
Registered: July 2009

Senior Member

I can see that CDOListFeatureDeltaImpl.reconstrucAddedIndices() is a
bottleneck.

Depending of your model definition bottleneck could be at another place
in EMF as well.

I will first concentrate my effort to see what I can do for
CDOListFeatureDeltaImpl.reconstrucAddedIndices().

After I will write another case for EMF bottleneck and see what WE (the
community) can do. :-)

Simon

David Bonneau wrote:
> I have a small unit case that reproduce the "problem". If you want I can
> send it to you.
>
> David
>
> Simon McDuff a écrit :
>> Hi David,
>>
>> Let me try it and profile to see and we can do something better.
>> Performance is very important and we didn't have measure that case.
>>
>> I think you are not doing anything wrong.
>>
>> I will be back soon with more answers!!!
>>
>> Simon
>>
>>
>> David Bonneau wrote:
>>> Hi,
>>>
>>> I have a little performance problem with CDO. I have a simple model
>>> : one Folder could contain many Files.
>>>
>>> With a MemStore I add 30 000 instances of File in one instance of
>>> Folder. It takes around 2 seconds. Now I remove each File of the
>>> Folder it takes ... more than 90 seconds !
>>>
>>> Here is the code :
>>>
>>> Folder folder = ExempleFactory.eINSTANCE.createFolder();
>>>
>>> for(int i = 0; i < 30000; ++i){
>>> Fichier file = RessourceFactory.eINSTANCE.createFichier();
>>> ((BasicEStoreEList)(folder.getFiles())).addUnique(file);
>>> }
>>> transaction.commit();
>>> //=>2 seconds
>>> for(int i = 29999; i >= 0; --i){
>>> EcoreUtil.remove(folder.getFiles().get(i));
>>> }
>>> //=>90 seconds
>>> transaction.commit();
>>>
>>> Ok I know that I could use a clear instead of 30 000 removes. But in
>>> my real application, in the second loop I have a condition to keep
>>> few elements.
>>>
>>> I try to dig in the code and I find that all the time is consumed by
>>> CDOListFeatureDeltaImpl.reconstrucAddedIndices()
>>>
>>> If I try to commit the transaction more often (every 1000 remove)
>>> it's faster : around 20 seconds.
>>>
>>> I would like to know if I'm doing something wrong ? If I could do
>>> something to improve performance ?
>>>
>>> Thanks
>>>
>>> David

Report message to a moderator

Re: [CDO]remove performance [message #426612 is a reply to message #426603]

Thu, 15 January 2009 15:40

Simon Mc Duff

Messages: 596
Registered: July 2009

Senior Member

I think we could cache some information to improve performance.

Can you create a bugzilla to optimize remove (improve
CDOListFeatureDeltaImpl.reconstrucAddedIndices())?

Also Can you still send me your .ecore ? ... because since I'm not sure
yet if you have more than 1 bottleneck...

Once this one is fixed.. I will debug the others... (when EOpposite is
set and COntainer = true)

Simon

David Bonneau wrote:
> I have a small unit case that reproduce the "problem". If you want I can
> send it to you.
>
> David
>
> Simon McDuff a écrit :
>> Hi David,
>>
>> Let me try it and profile to see and we can do something better.
>> Performance is very important and we didn't have measure that case.
>>
>> I think you are not doing anything wrong.
>>
>> I will be back soon with more answers!!!
>>
>> Simon
>>
>>
>> David Bonneau wrote:
>>> Hi,
>>>
>>> I have a little performance problem with CDO. I have a simple model
>>> : one Folder could contain many Files.
>>>
>>> With a MemStore I add 30 000 instances of File in one instance of
>>> Folder. It takes around 2 seconds. Now I remove each File of the
>>> Folder it takes ... more than 90 seconds !
>>>
>>> Here is the code :
>>>
>>> Folder folder = ExempleFactory.eINSTANCE.createFolder();
>>>
>>> for(int i = 0; i < 30000; ++i){
>>> Fichier file = RessourceFactory.eINSTANCE.createFichier();
>>> ((BasicEStoreEList)(folder.getFiles())).addUnique(file);
>>> }
>>> transaction.commit();
>>> //=>2 seconds
>>> for(int i = 29999; i >= 0; --i){
>>> EcoreUtil.remove(folder.getFiles().get(i));
>>> }
>>> //=>90 seconds
>>> transaction.commit();
>>>
>>> Ok I know that I could use a clear instead of 30 000 removes. But in
>>> my real application, in the second loop I have a condition to keep
>>> few elements.
>>>
>>> I try to dig in the code and I find that all the time is consumed by
>>> CDOListFeatureDeltaImpl.reconstrucAddedIndices()
>>>
>>> If I try to commit the transaction more often (every 1000 remove)
>>> it's faster : around 20 seconds.
>>>
>>> I would like to know if I'm doing something wrong ? If I could do
>>> something to improve performance ?
>>>
>>> Thanks
>>>
>>> David

Report message to a moderator

Re: [CDO]remove performance [message #426614 is a reply to message #426612]

Thu, 15 January 2009 15:54

David Bonneau

Messages: 28
Registered: July 2009

Junior Member

This is a multi-part message in MIME format.
--------------020706030800030302080502
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 8bit

Thanks for your help !

I will create a bugzilla. I use CDO 2.0.0M4.

I attach to this mail my .ecore. In my previous example Folder is Main
in the .ecore and File is Fichier. In other words a Main object can
contain one or many Fichier object.

David

Simon McDuff a

Report message to a moderator

Re: [CDO]remove performance [message #426619 is a reply to message #426614]

Thu, 15 January 2009 16:35

Simon Mc Duff

Messages: 596
Registered: July 2009

Senior Member

Thank David!!

Thank you for the explanation... Since I speak normally French and not
English... I should be able to figure it out!! :-)

By the way, my first draft to fix that bugs seems really to work well.
The time is now almost NIL is ReconstructIndices...

Thank you!

David Bonneau wrote:
> Thanks for your help !
>
> I will create a bugzilla. I use CDO 2.0.0M4.
>
> I attach to this mail my .ecore. In my previous example Folder is Main
> in the .ecore and File is Fichier. In other words a Main object can
> contain one or many Fichier object.
>
> David
>
>
> Simon McDuff a écrit :
>> I think we could cache some information to improve performance.
>>
>> Can you create a bugzilla to optimize remove (improve
>> CDOListFeatureDeltaImpl.reconstrucAddedIndices())?
>>
>> Also Can you still send me your .ecore ? ... because since I'm not
>> sure yet if you have more than 1 bottleneck...
>>
>> Once this one is fixed.. I will debug the others... (when EOpposite is
>> set and COntainer = true)
>>
>> Simon
>>
>> David Bonneau wrote:
>>> I have a small unit case that reproduce the "problem". If you want I
>>> can send it to you.
>>>
>>> David
>>>
>>> Simon McDuff a écrit :
>>>> Hi David,
>>>>
>>>> Let me try it and profile to see and we can do something better.
>>>> Performance is very important and we didn't have measure that case.
>>>>
>>>> I think you are not doing anything wrong.
>>>>
>>>> I will be back soon with more answers!!!
>>>>
>>>> Simon
>>>>
>>>>
>>>> David Bonneau wrote:
>>>>> Hi,
>>>>>
>>>>> I have a little performance problem with CDO. I have a simple
>>>>> model : one Folder could contain many Files.
>>>>>
>>>>> With a MemStore I add 30 000 instances of File in one instance of
>>>>> Folder. It takes around 2 seconds. Now I remove each File of the
>>>>> Folder it takes ... more than 90 seconds !
>>>>>
>>>>> Here is the code :
>>>>>
>>>>> Folder folder = ExempleFactory.eINSTANCE.createFolder();
>>>>>
>>>>> for(int i = 0; i < 30000; ++i){
>>>>> Fichier file = RessourceFactory.eINSTANCE.createFichier();
>>>>> ((BasicEStoreEList)(folder.getFiles())).addUnique(file);
>>>>> }
>>>>> transaction.commit();
>>>>> //=>2 seconds
>>>>> for(int i = 29999; i >= 0; --i){
>>>>> EcoreUtil.remove(folder.getFiles().get(i));
>>>>> }
>>>>> //=>90 seconds
>>>>> transaction.commit();
>>>>>
>>>>> Ok I know that I could use a clear instead of 30 000 removes. But
>>>>> in my real application, in the second loop I have a condition to
>>>>> keep few elements.
>>>>>
>>>>> I try to dig in the code and I find that all the time is consumed
>>>>> by CDOListFeatureDeltaImpl.reconstrucAddedIndices()
>>>>>
>>>>> If I try to commit the transaction more often (every 1000 remove)
>>>>> it's faster : around 20 seconds.
>>>>>
>>>>> I would like to know if I'm doing something wrong ? If I could do
>>>>> something to improve performance ?
>>>>>
>>>>> Thanks
>>>>>
>>>>> David
>

Report message to a moderator

Re: [CDO]remove performance [message #426622 is a reply to message #426614]

Thu, 15 January 2009 16:54

Simon Mc Duff

Messages: 596
Registered: July 2009

Senior Member

In my testcase I went from 56 seconds to 2.3 secs.

I will wait until you create the bugzilla to terminate it!!

Now the other problem when Container = true and EOpposite != null (When
we remove)

Simon

David Bonneau wrote:
> Thanks for your help !
>
> I will create a bugzilla. I use CDO 2.0.0M4.
>
> I attach to this mail my .ecore. In my previous example Folder is Main
> in the .ecore and File is Fichier. In other words a Main object can
> contain one or many Fichier object.
>
> David
>
>
> Simon McDuff a écrit :
>> I think we could cache some information to improve performance.
>>
>> Can you create a bugzilla to optimize remove (improve
>> CDOListFeatureDeltaImpl.reconstrucAddedIndices())?
>>
>> Also Can you still send me your .ecore ? ... because since I'm not
>> sure yet if you have more than 1 bottleneck...
>>
>> Once this one is fixed.. I will debug the others... (when EOpposite is
>> set and COntainer = true)
>>
>> Simon
>>
>> David Bonneau wrote:
>>> I have a small unit case that reproduce the "problem". If you want I
>>> can send it to you.
>>>
>>> David
>>>
>>> Simon McDuff a écrit :
>>>> Hi David,
>>>>
>>>> Let me try it and profile to see and we can do something better.
>>>> Performance is very important and we didn't have measure that case.
>>>>
>>>> I think you are not doing anything wrong.
>>>>
>>>> I will be back soon with more answers!!!
>>>>
>>>> Simon
>>>>
>>>>
>>>> David Bonneau wrote:
>>>>> Hi,
>>>>>
>>>>> I have a little performance problem with CDO. I have a simple
>>>>> model : one Folder could contain many Files.
>>>>>
>>>>> With a MemStore I add 30 000 instances of File in one instance of
>>>>> Folder. It takes around 2 seconds. Now I remove each File of the
>>>>> Folder it takes ... more than 90 seconds !
>>>>>
>>>>> Here is the code :
>>>>>
>>>>> Folder folder = ExempleFactory.eINSTANCE.createFolder();
>>>>>
>>>>> for(int i = 0; i < 30000; ++i){
>>>>> Fichier file = RessourceFactory.eINSTANCE.createFichier();
>>>>> ((BasicEStoreEList)(folder.getFiles())).addUnique(file);
>>>>> }
>>>>> transaction.commit();
>>>>> //=>2 seconds
>>>>> for(int i = 29999; i >= 0; --i){
>>>>> EcoreUtil.remove(folder.getFiles().get(i));
>>>>> }
>>>>> //=>90 seconds
>>>>> transaction.commit();
>>>>>
>>>>> Ok I know that I could use a clear instead of 30 000 removes. But
>>>>> in my real application, in the second loop I have a condition to
>>>>> keep few elements.
>>>>>
>>>>> I try to dig in the code and I find that all the time is consumed
>>>>> by CDOListFeatureDeltaImpl.reconstrucAddedIndices()
>>>>>
>>>>> If I try to commit the transaction more often (every 1000 remove)
>>>>> it's faster : around 20 seconds.
>>>>>
>>>>> I would like to know if I'm doing something wrong ? If I could do
>>>>> something to improve performance ?
>>>>>
>>>>> Thanks
>>>>>
>>>>> David
>

Report message to a moderator

Previous Topic:	[CDO] EMF change notifications
Next Topic:	Uri null. How to check?

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Wed Apr 24 19:45:15 GMT 2024

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter