Eclipse Community Forums: EMF » [CDO] Optimization

Help

Home

Home » Modeling » EMF » [CDO] Optimization

Show: Today's Messages :: Show Polls :: Message Navigator

[CDO] Optimization [message #424929]

Mon, 10 November 2008 23:57

Simon Mc Duff

Messages: 596
Registered: July 2009

Senior Member

Looking at the profiler for CDO.

Since the transfer of objects from client to server at commit time take
s around 11-20% of the time (to Write after Read).. maybe we could be
more efficient ?

Basically we could transfer some data of the objects as soon as the
users changed it asynchronously(or when we detect the users do not use
it anymore :-))... that way when we will commit most of the data will be
at the others ends. If the users rollback... we clear what we have
accumulated so far to the server.

We could have on
- Automatic - Transfer data based on algorithm/rules. Delta are transfered.
- Manually - calling flush() will send changes async to the server.

I think we could increase the commit by 5-10 %.
At the moment, it takes 2.2 secs to commit 60000 objects using MEMStore
(Time to execute transaction.commit).

This could be reduces to 1.8 secs.. maybe .. we will need to try it.

Make senses ?

Simon

Report message to a moderator

Re: [CDO] Optimization [message #424947 is a reply to message #424929]

Tue, 11 November 2008 14:23

Stefan Winkler

Messages: 307
Registered: July 2009
Location: Germany

Senior Member

Hi,

comments below ...

Simon McDuff schrieb:
> Since the transfer of objects from client to server at commit time
> take s around 11-20% of the time (to Write after Read).. maybe we
> could be more efficient ?
>
> Basically we could transfer some data of the objects as soon as the
> users changed it asynchronously(or when we detect the users do not use
> it anymore :-))... that way when we will commit most of the data will
> be at the others ends. If the users rollback... we clear what we have
> accumulated so far to the server.
Is there something like a client/transaction context id in CDO? Because
in the way you describe, the server must remember which dirty object
belongs to which transaction. With many users this could also lead to a
memory bottleneck on the server.

In contrast to think I was also thinking that if we pretransmit data to
the server asynchronously, the server could even pre-write data to a
DBStore and if the user does a rollback, we simply rollback on the DB
side (which works as long the CDO server maintains a strict DB
connection to transaction context mapping and the DB itself is
transactional).
> We could have on
> - Automatic - Transfer data based on algorithm/rules. Delta are
> transfered.
> - Manually - calling flush() will send changes async to the server.
>
> I think we could increase the commit by 5-10 %.
> At the moment, it takes 2.2 secs to commit 60000 objects using
> MEMStore (Time to execute transaction.commit).
>
> This could be reduces to 1.8 secs.. maybe .. we will need to try it.
Is that stress testing code in the CDO test project or do you have it
only on your machine?
I'd be interested in the code, as I'd like to compare prepared to
non-prepared statement JDBC performance - so could you point me to /
send me the code (I'm currently to lazy (read: busy) to create the code
myself ;-))

Cheers,
Stefan

Report message to a moderator

Re: [CDO] Optimization [message #424948 is a reply to message #424947]

Tue, 11 November 2008 14:43

Simon Mc Duff

Messages: 596
Registered: July 2009

Senior Member

Hi Stefan,

comments below

Stefan Winkler wrote:
> Hi,
>
> comments below ...
>
> Simon McDuff schrieb:
>> Since the transfer of objects from client to server at commit time
>> take s around 11-20% of the time (to Write after Read).. maybe we
>> could be more efficient ?
>>
>> Basically we could transfer some data of the objects as soon as the
>> users changed it asynchronously(or when we detect the users do not use
>> it anymore :-))... that way when we will commit most of the data will
>> be at the others ends. If the users rollback... we clear what we have
>> accumulated so far to the server.
> Is there something like a client/transaction context id in CDO? Because
> in the way you describe, the server must remember which dirty object
> belongs to which transaction. With many users this could also lead to a
> memory bottleneck on the server.

We could use the commitmanager! So yes we have something like that! :-)
Memory could be an issue but you could have the same problem locally...
if you want to change many objects. Also since this is a feature that
could be turn in or off... I think it is acceptable!
Also if many threads commit at the same time.. the server could runned
out of memory!! :-)

>
> In contrast to think I was also thinking that if we pretransmit data to
> the server asynchronously, the server could even pre-write data to a
> DBStore and if the user does a rollback, we simply rollback on the DB
> side (which works as long the CDO server maintains a strict DB
> connection to transaction context mapping and the DB itself is
> transactional).
I don't like to sent it to the store..
- Leads in DeadLock (One of my stress test is to commit differents
objects at the same time (but from the same class)) MEMStore and
ObjectivityStore passed it successfully.. not DBStore/MySql. It lockes
pages!
- Other threads cannot change that objects even if they are first.

>> We could have on
>> - Automatic - Transfer data based on algorithm/rules. Delta are
>> transfered.
>> - Manually - calling flush() will send changes async to the server.
>>
>> I think we could increase the commit by 5-10 %.
>> At the moment, it takes 2.2 secs to commit 60000 objects using
>> MEMStore (Time to execute transaction.commit).
>>
>> This could be reduces to 1.8 secs.. maybe .. we will need to try it.
> Is that stress testing code in the CDO test project or do you have it
> only on your machine?

It is part of bugs
237158: Provide Performance TestCase
https://bugs.eclipse.org/bugs/show_bug.cgi?id=237158

So it will be committed at some point to the repository

> I'd be interested in the code, as I'd like to compare prepared to
> non-prepared statement JDBC performance - so could you point me to /
> send me the code (I'm currently to lazy (read: busy) to create the code
> myself ;-))
SO far the POerformance framework have a model. Each time you run the
testcase it will save the result in a XMIResource and compare it with
the expected result (same test you runned before). You could also
compare it with another Store Implementation to have a better idea of
the speed of your implementation.

Here some stats with DBStore/Mysql compare to MEM
https://bugs.eclipse.org/bugs/attachment.cgi?id=117505&a ction=edit

I have more.. but need to explain each of them!! :-)

>
> Cheers,
> Stefan

Report message to a moderator

Re: [CDO] Optimization [message #424966 is a reply to message #424947]

Wed, 12 November 2008 08:05

Eike Stepper

Messages: 6682
Registered: July 2009

Senior Member

Stefan,

At the server side the context of a client side transaction is
represented as an ITransaction. It is also stored by the framework in
each IStoreAccessor with isReader() == false.

Cheers
/Eike

----
http://thegordian.blogspot.com

Stefan Winkler schrieb:
> Hi,
>
> comments below ...
>
> Simon McDuff schrieb:
>> Since the transfer of objects from client to server at commit time
>> take s around 11-20% of the time (to Write after Read).. maybe we
>> could be more efficient ?
>>
>> Basically we could transfer some data of the objects as soon as the
>> users changed it asynchronously(or when we detect the users do not
>> use it anymore :-))... that way when we will commit most of the data
>> will be at the others ends. If the users rollback... we clear what we
>> have accumulated so far to the server.
> Is there something like a client/transaction context id in CDO?
> Because in the way you describe, the server must remember which dirty
> object belongs to which transaction. With many users this could also
> lead to a memory bottleneck on the server.
>
> In contrast to think I was also thinking that if we pretransmit data
> to the server asynchronously, the server could even pre-write data to
> a DBStore and if the user does a rollback, we simply rollback on the
> DB side (which works as long the CDO server maintains a strict DB
> connection to transaction context mapping and the DB itself is
> transactional).
>> We could have on
>> - Automatic - Transfer data based on algorithm/rules. Delta are
>> transfered.
>> - Manually - calling flush() will send changes async to the server.
>>
>> I think we could increase the commit by 5-10 %.
>> At the moment, it takes 2.2 secs to commit 60000 objects using
>> MEMStore (Time to execute transaction.commit).
>>
>> This could be reduces to 1.8 secs.. maybe .. we will need to try it.
> Is that stress testing code in the CDO test project or do you have it
> only on your machine?
> I'd be interested in the code, as I'd like to compare prepared to
> non-prepared statement JDBC performance - so could you point me to /
> send me the code (I'm currently to lazy (read: busy) to create the
> code myself ;-))
>
> Cheers,
> Stefan

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper

Report message to a moderator

Re: [CDO] Optimization [message #424967 is a reply to message #424929]

Wed, 12 November 2008 08:10

Eike Stepper

Messages: 6682
Registered: July 2009

Senior Member

Simon,

I think the measure of 2.2 seconds for committing 60,000 objects is far
more than impressing (although it seems to be the MEMStore and a local
TCP connection)!
Making that 0.4 seconds faster doesn't look like a top priority. Perhaps
the numbers in a real networked scenario give a better indication about
the usefulness of such a comparingly complex optimization...

Cheers
/Eike

----
http://thegordian.blogspot.com

Simon McDuff schrieb:
> Looking at the profiler for CDO.
>
> Since the transfer of objects from client to server at commit time
> take s around 11-20% of the time (to Write after Read).. maybe we
> could be more efficient ?
>
> Basically we could transfer some data of the objects as soon as the
> users changed it asynchronously(or when we detect the users do not use
> it anymore :-))... that way when we will commit most of the data will
> be at the others ends. If the users rollback... we clear what we have
> accumulated so far to the server.
>
> We could have on
> - Automatic - Transfer data based on algorithm/rules. Delta are
> transfered.
> - Manually - calling flush() will send changes async to the server.
>
> I think we could increase the commit by 5-10 %.
> At the moment, it takes 2.2 secs to commit 60000 objects using
> MEMStore (Time to execute transaction.commit).
>
> This could be reduces to 1.8 secs.. maybe .. we will need to try it.
>
> Make senses ?
>
>
> Simon

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper

Report message to a moderator

Previous Topic:	[CDO] Being able to retrieve ResourcePath from a MemStore implementation
Next Topic:	code formatting problem

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Fri Apr 26 05:21:06 GMT 2024

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter