Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF "Technology" (Ecore Tools, EMFatic, etc)  » [CDO] reading/writting throughput performances
[CDO] reading/writting throughput performances [message #123583] Thu, 29 May 2008 10:22 Go to next message
Cedric Brun is currently offline Cedric BrunFriend
Messages: 431
Registered: July 2009
Senior Member
Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex one -
250 EClassifiers/300 features and a simple one - 1 Classifier/3 Features)

The test is a "quick and dirty" one so please do not take these figures too
seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.


* First case : building a small model (10000 elements) with the simple MM,
only adding instances in an empty database :
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.

browsing the just commited elements opening a new transaction and getting a
new Resource instance, I get around 30 000 objets/sec rate just iterating
thanks to Resource.getAllContents().

* Second case : I built a big model in the database using the complex
metamodel. It contains 700 000 elements. The creation went pretty nicely (I
did not measured it exactly as it was launched during a night, but it seems
I got around 100 objects/sec rate in writting as I was writting every 700
objects.


Once the model is built the derby database is 222 MB (which - I heard - is
not so big for derby).

Retrieving the distant resource and calling eAllContents() on it, I get a 50
Objects/sec rate at best, which mean my eAllContents() call would take 4
hours !

This is probably linked to the way the mapping to db is done, here's the
config I used (came from CDO-server example if I recall correctly).


<repository name="repo1">
<property name="overrideUUID"
value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
<property name="supportingAudits" value="true"/>
<property name="verifyingRevisions" value="false"/>
<property name="currentLRUCapacity" value="10000"/>
<property name="revisedLRUCapacity" value="100"/>

<store type="db">
<!-- type: horizontal | vertical | <any user-contributed type> -->
<mappingStrategy type="horizontal">
<!-- ONE_TABLE_PER_REFERENCE | ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
<property name="toManyReferences" value="ONE_TABLE_PER_REFERENCE"/>

<!-- LIKE_ATTRIBUTES | LIKE_TO_MANY_REFERENCES -->
<property name="toOneReferences" value="LIKE_ATTRIBUTES"/>

<!-- MODEL | STRATEGY -->
<property name="mappingPrecedence" value="MODEL"/>
</mappingStrategy>

<dbAdapter name="derby-embedded"/>
<dataSource class="org.apache.derby.jdbc.EmbeddedDataSource"
databaseName="/tmp/cdodb1"
createDatabase="create"/>

</store>
</repository>




What do you think about that ? What throughput do you think I should
expect ? What are the main reasons fur such figures ?

I can provide you the "stupid" test case and even profile the code more
closely if you want.


Cédric


http://cedric.brun.io news and articles on eclipse and eclipse modeling.
Re: [CDO] reading/writting throughput performances [message #123598 is a reply to message #123583] Thu, 29 May 2008 11:27 Go to previous messageGo to next message
Eclipse UserFriend
Originally posted by: stepper.sympedia.de

This is a multi-part message in MIME format.
--------------090102080203020800020703
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi Cédric,

That sounds interesting!
Comments inline...


Cédric Brun schrieb:
> Hi Eike,
>
> I've been messing around CDO lately and I hacked a few ugly unit-tests
> measuring read/write throughput when I use CDO (RC2).
>
> I'm using an embeded-derby database, two different metamodels (complex one -
> 250 EClassifiers/300 features and a simple one - 1 Classifier/3 Features)
>
> The test is a "quick and dirty" one so please do not take these figures too
> seriously, I would just like to have your input about the "order of
> magnitude" I should expect or settings affecting seriously these
> performances. I tried both JVM and TCP acceptor but the performances are
> quite similar though JVM is a bit faster.
>
Looks like a sign that the NIO socket channels are efficient ,-)
You used localhost for TCP transport I guess?

>
> * First case : building a small model (10000 elements) with the simple MM,
> only adding instances in an empty database :
> commiting change on every add I get a 2 objects per second rate.
> commiting changes every 10 adds - I get a 30 objects per second rate.
>
Seems to be consistent with the fact that only for commits there is
remote traffic.
And the backend transaction management is another overhead.
So the impact of the number objects/commit starts with bigger transactions.

> browsing the just commited elements opening a new transaction and getting a
> new Resource instance, I get around 30 000 objets/sec rate just iterating
> thanks to Resource.getAllContents().
>
Hehe, this is a sign that client side caching works well:
All views and transactions of a single session share (and cache) their
object state.
Otherwise I'd expect much smaller values.
With his ObjectivityStore implementation Simon (cc'ed) reached rates of
several thousands objects/sec.
But the DBStore (which you used) is not that optimized.

> * Second case : I built a big model in the database using the complex
> metamodel. It contains 700 000 elements. The creation went pretty nicely (I
> did not measured it exactly as it was launched during a night, but it seems
> I got around 100 objects/sec rate in writting as I was writting every 700
> objects.
>
Sounds not too bad.

>
> Once the model is built the derby database is 222 MB (which - I heard - is
> not so big for derby).
>
Hmm, I have no extensive experience with Derby.
To be honest, I'm really no database expert (although I can write simple
SQL statements).

> Retrieving the distant resource and calling eAllContents() on it, I get a 50
> Objects/sec rate at best, which mean my eAllContents() call would take 4
> hours !
>
Currently I'm not sure what exactly eAllContents() does do (iteration
order and so).
There are several CDO features that can have a major impact on load rates!

- Collections (xrefs and containment) can be loaded lazily. I don't mean
the target objects, which are lazy by default anyway, but even the
target IDs that are in the collections. We call that "reference
chunking" (but search for a better name is ongoing).
|*public interface *CDOSession *extends *CDOProtocolSession, IContainer<CDOView>
{
*public **int *getReferenceChunkSize();

*public **void *setReferenceChunkSize(*int *referenceChunkSize);|


- Loading collection elements (target objects) can be batched (the
current name is even worse)
|*public interface *CDOView *extends *CDOProtocolView, INotifier
{
*public **int *getLoadRevisionCollectionChunkSize();

*public **void *setLoadRevisionCollectionChunkSize(*int *loadRevisionCollectionChunkSize);|



- One of the most sophisticated feature is automatic model usage
analyzation. The result are fetch rules that are sent to the server.
When later objects are loaded from the server, it can answer with
additional objects that are likely to be needed by the client in the
near future.
|*public class *CDOViewImpl *extends *org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView, CDOIDProvider,
Adapter.Internal
{
*public *CDOFeatureAnalyzer getFeatureAnalyzer()
{
*return *featureAnalyzer;
}

*public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
{
*this*.featureAnalyzer = featureAnalyzer == *null *? CDOFeatureAnalyzer.NOOP : featureAnalyzer;
}|


- There's a SmartReadAheadThread (if you're curious, please ask Simon).
- Caches can be configured
- ...


Known issues of the DBStore:
- When in auditing mode updates are always written as complete inserts
(by design)
- When configuring toManyReferences to a value different from
LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
- No (few) prepared statements are used (see
https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
- Maybe others...

Please note that the DBStore was not written by me to compete with
Hibernate in terms of performance or mapping capabilities.
Now that Martin and I, well in the end mostly Martin, provide a
HibernateStore and this HibernateStore gets more and more features I
expect that there will be a trend away from the DBStore.

> This is probably linked to the way the mapping to db is done, here's the
> config I used (came from CDO-server example if I recall correctly).
>
>
> <repository name="repo1">
> <property name="overrideUUID"
> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
> <property name="supportingAudits" value="true"/>
> <property name="verifyingRevisions" value="false"/>
> <property name="currentLRUCapacity" value="10000"/>
>
I guess this one could be larger.

> <property name="revisedLRUCapacity" value="100"/>
>
> <store type="db">
> <!-- type: horizontal | vertical | <any user-contributed type> -->
> <mappingStrategy type="horizontal">
> <!-- ONE_TABLE_PER_REFERENCE | ONE_TABLE_PER_CLASS |
> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
> <property name="toManyReferences" value="ONE_TABLE_PER_REFERENCE"/>
>
> <!-- LIKE_ATTRIBUTES | LIKE_TO_MANY_REFERENCES -->
> <property name="toOneReferences" value="LIKE_ATTRIBUTES"/>
>
> <!-- MODEL | STRATEGY -->
> <property name="mappingPrecedence" value="MODEL"/>
>
This one is obsolete.

> </mappingStrategy>
>
> <dbAdapter name="derby-embedded"/>
> <dataSource class="org.apache.derby.jdbc.EmbeddedDataSource"
> databaseName="/tmp/cdodb1"
> createDatabase="create"/>
>
> </store>
> </repository>
>
>
>
>
> What do you think about that ? What throughput do you think I should
> expect ?
As I said that's very difficult to judge.
I believe it's more appropriate to compare the actual values with values
from other measures (other setups, properties, etc. or even other
products) instead of random expectations.
Apologies that I didn't have the time so far to provide some reference
measures myself. Maybe this gets better now that the team starts to grow
slowly ;-)

> What are the main reasons fur such figures ?
>
Hmm, I *suspect* you don't like the figures? I can find no evidence your
assessment ;-)

Ahh,one more note:
Tracing can also have a significant impact on performance and end-to-end
throughput.
Turn it off by all means. And please make sure that it's really turned
off, not only invisible (i.e. console is not a target).

Cheers
/Eike



--------------090102080203020800020703
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Cédric,<br>
<br>
That sounds interesting!<br>
Comments inline...<br>
<br>
<br>
Cédric Brun schrieb:
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex one -
250 EClassifiers/300 features and a simple one - 1 Classifier/3 Features)

The test is a "quick and dirty" one so please do not take these figures too
seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.
</pre>
</blockquote>
Looks like a sign that the NIO socket channels are efficient ,-)<br>
You used localhost for TCP transport I guess?<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">

* First case : building a small model (10000 elements) with the simple MM,
only adding instances in an empty database :
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.
</pre>
</blockquote>
Seems to be consistent with the fact that only for commits there is
remote traffic.<br>
And the backend transaction management is another overhead.<br>
So the impact of the number objects/commit starts with bigger
transactions.<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">
browsing the just commited elements opening a new transaction and getting a
new Resource instance, I get around 30 000 objets/sec rate just iterating
thanks to Resource.getAllContents().
</pre>
</blockquote>
Hehe, this is a sign that client side caching works well:<br>
All views and transactions of a single session share (and cache) their
object state.<br>
Otherwise I'd expect much smaller values.<br>
With his ObjectivityStore implementation Simon (cc'ed) reached rates of
several thousands objects/sec.<br>
But the DBStore (which you used) is not that optimized.<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">
* Second case : I built a big model in the database using the complex
metamodel. It contains 700 000 elements. The creation went pretty nicely (I
did not measured it exactly as it was launched during a night, but it seems
I got around 100 objects/sec rate in writting as I was writting every 700
objects.
</pre>
</blockquote>
Sounds not too bad.<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">

Once the model is built the derby database is 222 MB (which - I heard - is
not so big for derby).
</pre>
</blockquote>
Hmm, I have no extensive experience with Derby.<br>
To be honest, I'm really no database expert (although I can write
simple SQL statements).<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">
Retrieving the distant resource and calling eAllContents() on it, I get a 50
Objects/sec rate at best, which mean my eAllContents() call would take 4
hours !
</pre>
</blockquote>
Currently I'm not sure what exactly eAllContents() does do (iteration
order and so).<br>
There are several CDO features that can have a major impact on load
rates!<br>
<br>
- Collections (xrefs and containment) can be loaded lazily. I don't
mean the target objects, which are lazy by default anyway, but even the
target IDs that are in the collections. We call that "reference
chunking" (but search for a better name is ongoing).<br>
<title></title>
<style type="text/css">
<!--code { font-family: Courier New, Courier; font-size: 10pt; margin: 0px; }-->
</style>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<!-- ======================================================== --><!-- = Java Sourcecode to HTML automatically converted code = --><!-- = Java2Html Converter 5.0 [2006-02-26] by Markus Gebhard markus@jave.de = -->
<!-- = Further information: http://www.java2html.de = -->
<div class="java" align="left">
<table bgcolor="#ffffff" border="0" cellpadding="3" cellspacing="0">
<tbody>
<tr>
<!-- start source code --> <td align="left" nowrap="nowrap"
valign="top"> <code><font color="#7f0055"><b>public interface </b></font><font
color="#000000">CDOSession </font><font color="#7f0055"><b>extends </b></font><font
color="#000000">CDOProtocolSession, IContainer&lt;CDOView&gt; </font><br>
<font color="#000000">{</font><br>
<font color="#ffffff">  </font><font color="#7f0055"><b>public </b></font><font
color="#7f0055"><b>int </b></font><font color="#000000">getReferenceChunkSize</font><font
color="#000000">()</font><font color="#000000">;</font><br>
<font color="#ffffff"></font><br>
<font color="#ffffff">  </font><font color="#7f0055"><b>public </b></font><font
color="#7f0055"><b>void </b></font><font color="#000000">setReferenceChunkSize</font><font
color="#000000">(</font><font color="#7f0055"><b>int </b></font><font
color="#000000">referenceChunkSize</font><font color="#000000">)</font><font
color="#000000">;</font></code> </td>
<!-- end source code --> </tr>
</tbody>
</table>
</div>
<!-- = END of automatically generated HTML code = --><br>
<!-- ======================================================== -->
- Loading collection elements (target objects) can be batched (the
current name is even worse)<br>
<title></title>
<style type="text/css">
<!--code { font-family: Courier New, Courier; font-size: 10pt; margin: 0px; }-->
</style>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<!-- ======================================================== -->
<!-- = Java Sourcecode to HTML automatically converted code = --><!-- = Java2Html Converter 5.0 [2006-02-26] by Markus Gebhard markus@jave.de = -->
<!-- = Further information: http://www.java2html.de = -->
<div class="java" align="left">
<table bgcolor="#ffffff" border="0" cellpadding="3" cellspacing="0">
<tbody>
<tr>
<!-- start source code --> <td align="left" nowrap="nowrap"
valign="top"> <code><font color="#7f0055"><b>public interface </b></font><font
color="#000000">CDOView </font><font color="#7f0055"><b>extends </b></font><font
color="#000000">CDOProtocolView, INotifier</font><br>
<font color="#000000">{</font><br>
<font color="#ffffff"></font><font color="#ffffff">  </font><font
color="#7f0055"><b>public </b></font><font color="#7f0055"><b>int </b></font><font
color="#000000">getLoadRevisionCollectionChunkSize</font> <font
color="#000000">()</font><font color="#000000">;</font><br>
<font color="#ffffff"></font><br>
<font color="#ffffff">  </font><font color="#7f0055"><b>public </b></font><font
color="#7f0055"><b>void </b></font><font color="#000000">setLoadRevisionCollectionChunkSize</font> <font
color="#000000">(</font><font color="#7f0055"><b>int </b></font><font
color="#000000">loadRevisionCollectionChunkSize</font><font
color="#000000">)</font><font color="#000000">;</font></code> </td>
<!-- end source code --> </tr>
</tbody>
</table>
</div>
<!-- = END of automatically generated HTML code = -->
<!-- ======================================================== --><br>
- One of the most sophisticated feature is automatic model usage
analyzation. The result are fetch rules that are sent to the server.
When later objects are loaded from the server, it can answer with
additional objects that are likely to be needed by the client in the
near future.<br>
<title></title>
<style type="text/css">
<!--code { font-family: Courier New, Courier; font-size: 10pt; margin: 0px; }-->
</style>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<!-- ======================================================== -->
<!-- = Java Sourcecode to HTML automatically converted code = --><!-- = Java2Html Converter 5.0 [2006-02-26] by Markus Gebhard markus@jave.de = -->
<!-- = Further information: http://www.java2html.de = -->
<div class="java" align="left">
<table bgcolor="#ffffff" border="0" cellpadding="3" cellspacing="0">
<tbody>
<tr>
<!-- start source code --> <td align="left" nowrap="nowrap"
valign="top"> <code><font color="#7f0055"><b>public class </b></font><font
color="#000000">CDOViewImpl </font><font color="#7f0055"><b>extends </b></font><font
color="#000000">org.eclipse.net4j.internal.util.event.Notifier  </font><font
color="#7f0055"><b>implements </b></font><font color="#000000">CDOView, CDOIDProvider,</font>
<font color="#000000">Adapter.Internal</font><br>
<font color="#000000">{</font><br>
<font color="#ffffff"></font><font color="#ffffff">  </font><font
color="#7f0055"><b>public </b></font><font color="#000000">CDOFeatureAnalyzer getFeatureAnalyzer</font ><font
color="#000000">()</font><br>
<font color="#ffffff">  </font><font color="#000000">{</font><br>
<font color="#ffffff">    </font><font color="#7f0055"><b>return </b></font><font
color="#000000">featureAnalyzer;</font><br>
<font color="#ffffff">  </font><font color="#000000">}</font><br>
<font color="#ffffff"></font><br>
<font color="#ffffff">  </font><font color="#7f0055"><b>public </b></font><font
color="#7f0055"><b>void </b></font><font color="#000000">setFeatureAnalyzer</font><font
color="#000000">(</font><font color="#000000">CDOFeatureAnalyzer featureAnalyzer</font ><font
color="#000000">)</font><br>
<font color="#ffffff">  </font><font color="#000000">{</font><br>
<font color="#ffffff">    </font><font color="#7f0055"><b>this</b></font><font
color="#000000">.featureAnalyzer = featureAnalyzer ==  </font><font
color="#7f0055"><b>null </b></font><font color="#000000">? CDOFeatureAnalyzer.NOOP : featureAnalyzer; </font><br>
<font color="#ffffff">  </font><font color="#000000">}</font></code>
</td>
<!-- end source code --> </tr>
</tbody>
</table>
</div>
<!-- = END of automatically generated HTML code = -->
<!-- ======================================================== --><br>
- There's a SmartReadAheadThread (if you're curious, please ask Simon).<br>
- Caches can be configured<br>
- ...<br>
<br>
<br>
Known issues of the DBStore:<br>
- When in auditing mode updates are always written as complete inserts
(by design)<br>
- When configuring toManyReferences to a value different from
LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.<br>
- No (few) prepared statements are used (see
<a class="moz-txt-link-freetext" href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487">https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487</a>)<br>
- Maybe others...<br>
<br>
Please note that the DBStore was not written by me to compete with
Hibernate in terms of performance or mapping capabilities.<br>
Now that Martin and I, well in the end mostly Martin, provide a
HibernateStore and this HibernateStore gets more and more features I
expect that there will be a trend away from the DBStore. <br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">
This is probably linked to the way the mapping to db is done, here's the
config I used (came from CDO-server example if I recall correctly).


&lt;repository name="repo1"&gt;
&lt;property name="overrideUUID"
value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/&gt;
&lt;property name="supportingAudits" value="true"/&gt;
&lt;property name="verifyingRevisions" value="false"/&gt;
&lt;property name="currentLRUCapacity" value="10000"/&gt;
</pre>
</blockquote>
I guess this one could be larger.<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap=""> &lt;property name="revisedLRUCapacity" value="100"/&gt;

&lt;store type="db"&gt;
&lt;!-- type: horizontal | vertical | &lt;any user-contributed type&gt; --&gt;
&lt;mappingStrategy type="horizontal"&gt;
&lt;!-- ONE_TABLE_PER_REFERENCE | ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES --&gt;
&lt;property name="toManyReferences" value="ONE_TABLE_PER_REFERENCE"/&gt;

&lt;!-- LIKE_ATTRIBUTES | LIKE_TO_MANY_REFERENCES --&gt;
&lt;property name="toOneReferences" value="LIKE_ATTRIBUTES"/&gt;

&lt;!-- MODEL | STRATEGY --&gt;
&lt;property name="mappingPrecedence" value="MODEL"/&gt;
</pre>
</blockquote>
This one is obsolete.<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap=""> &lt;/mappingStrategy&gt;

&lt;dbAdapter name="derby-embedded"/&gt;
&lt;dataSource class="org.apache.derby.jdbc.EmbeddedDataSource"
databaseName="/tmp/cdodb1"
createDatabase="create"/&gt;

&lt;/store&gt;
&lt;/repository&gt;




What do you think about that ? What throughput do you think I should
expect ? </pre>
</blockquote>
As I said that's very difficult to judge.<br>
I believe it's more appropriate to compare the actual values with
values from other measures (other setups, properties, etc. or even
other products) instead of random expectations.<br>
Apologies that I didn't have the time so far to provide some reference
measures myself. Maybe this gets better now that the team starts to
grow slowly ;-)<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">What are the main reasons fur such figures ?
</pre>
</blockquote>
Hmm, I *suspect* you don't like the figures? I can find no evidence
your assessment ;-)<br>
<br>
Ahh,one more note:<br>
Tracing can also have a significant impact on performance and
end-to-end throughput.<br>
Turn it off by all means. And please make sure that it's really turned
off, not only invisible (i.e. console is not a target).<br>
<br>
Cheers<br>
/Eike<br>
<br>
<br>
</body>
</html>

--------------090102080203020800020703--
Re: [CDO] reading/writting throughput performances [message #123610 is a reply to message #123598] Thu, 29 May 2008 13:03 Go to previous messageGo to next message
Cedric Brun is currently offline Cedric BrunFriend
Messages: 431
Registered: July 2009
Senior Member
Eike,

thanks for your detailled answer, I'll digg further and try to profile the
Net4J/CDO stack so that I understand the "50 objects/sec rate" when
retrieving data (which I think is surprisingly low).

Concerning the logging, what can I change to be *sure* that no logging will
occurs ?

Cheers,

Cédric



Eike Stepper wrote:

> Hi Cédric,
>
> That sounds interesting!
> Comments inline...
>
>
> Cédric Brun schrieb:
>> Hi Eike,
>>
>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>> measuring read/write throughput when I use CDO (RC2).
>>
>> I'm using an embeded-derby database, two different metamodels (complex
>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>> Features)
>>
>> The test is a "quick and dirty" one so please do not take these figures
>> too seriously, I would just like to have your input about the "order of
>> magnitude" I should expect or settings affecting seriously these
>> performances. I tried both JVM and TCP acceptor but the performances are
>> quite similar though JVM is a bit faster.
>>
> Looks like a sign that the NIO socket channels are efficient ,-)
> You used localhost for TCP transport I guess?
>
>>
>> * First case : building a small model (10000 elements) with the simple
>> MM, only adding instances in an empty database :
>> commiting change on every add I get a 2 objects per second rate.
>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>
> Seems to be consistent with the fact that only for commits there is
> remote traffic.
> And the backend transaction management is another overhead.
> So the impact of the number objects/commit starts with bigger
> transactions.
>
>> browsing the just commited elements opening a new transaction and getting
>> a new Resource instance, I get around 30 000 objets/sec rate just
>> iterating thanks to Resource.getAllContents().
>>
> Hehe, this is a sign that client side caching works well:
> All views and transactions of a single session share (and cache) their
> object state.
> Otherwise I'd expect much smaller values.
> With his ObjectivityStore implementation Simon (cc'ed) reached rates of
> several thousands objects/sec.
> But the DBStore (which you used) is not that optimized.
>
>> * Second case : I built a big model in the database using the complex
>> metamodel. It contains 700 000 elements. The creation went pretty nicely
>> (I did not measured it exactly as it was launched during a night, but it
>> seems I got around 100 objects/sec rate in writting as I was writting
>> every 700 objects.
>>
> Sounds not too bad.
>
>>
>> Once the model is built the derby database is 222 MB (which - I heard -
>> is not so big for derby).
>>
> Hmm, I have no extensive experience with Derby.
> To be honest, I'm really no database expert (although I can write simple
> SQL statements).
>
>> Retrieving the distant resource and calling eAllContents() on it, I get a
>> 50 Objects/sec rate at best, which mean my eAllContents() call would take
>> 4 hours !
>>
> Currently I'm not sure what exactly eAllContents() does do (iteration
> order and so).
> There are several CDO features that can have a major impact on load rates!
>
> - Collections (xrefs and containment) can be loaded lazily. I don't mean
> the target objects, which are lazy by default anyway, but even the
> target IDs that are in the collections. We call that "reference
> chunking" (but search for a better name is ongoing).
> |*public interface *CDOSession *extends *CDOProtocolSession,
> |IContainer<CDOView>
> {
> *public **int *getReferenceChunkSize();
>
> *public **void *setReferenceChunkSize(*int *referenceChunkSize);|
>
>
> - Loading collection elements (target objects) can be batched (the
> current name is even worse)
> |*public interface *CDOView *extends *CDOProtocolView, INotifier
> {
> *public **int *getLoadRevisionCollectionChunkSize();
>
> *public **void *setLoadRevisionCollectionChunkSize(*int
> *loadRevisionCollectionChunkSize);|
>
>
>
> - One of the most sophisticated feature is automatic model usage
> analyzation. The result are fetch rules that are sent to the server.
> When later objects are loaded from the server, it can answer with
> additional objects that are likely to be needed by the client in the
> near future.
> |*public class *CDOViewImpl *extends
> |*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
> |CDOIDProvider,
> Adapter.Internal
> {
> *public *CDOFeatureAnalyzer getFeatureAnalyzer()
> {
> *return *featureAnalyzer;
> }
>
> *public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
> {
> *this*.featureAnalyzer = featureAnalyzer == *null *?
> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
> }|
>
>
> - There's a SmartReadAheadThread (if you're curious, please ask Simon).
> - Caches can be configured
> - ...
>
>
> Known issues of the DBStore:
> - When in auditing mode updates are always written as complete inserts
> (by design)
> - When configuring toManyReferences to a value different from
> LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
> - No (few) prepared statements are used (see
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
> - Maybe others...
>
> Please note that the DBStore was not written by me to compete with
> Hibernate in terms of performance or mapping capabilities.
> Now that Martin and I, well in the end mostly Martin, provide a
> HibernateStore and this HibernateStore gets more and more features I
> expect that there will be a trend away from the DBStore.
>
>> This is probably linked to the way the mapping to db is done, here's the
>> config I used (came from CDO-server example if I recall correctly).
>>
>>
>> <repository name="repo1">
>> <property name="overrideUUID"
>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>> <property name="supportingAudits" value="true"/>
>> <property name="verifyingRevisions" value="false"/>
>> <property name="currentLRUCapacity" value="10000"/>
>>
> I guess this one could be larger.
>
>> <property name="revisedLRUCapacity" value="100"/>
>>
>> <store type="db">
>> <!-- type: horizontal | vertical | <any
>> user-contributed type> --> <mappingStrategy
>> type="horizontal">
>> <!-- ONE_TABLE_PER_REFERENCE |
>> ONE_TABLE_PER_CLASS |
>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>> <property name="toManyReferences"
>> value="ONE_TABLE_PER_REFERENCE"/>
>>
>> <!-- LIKE_ATTRIBUTES |
>> LIKE_TO_MANY_REFERENCES --> <property
>> name="toOneReferences"
>> value="LIKE_ATTRIBUTES"/>
>>
>> <!-- MODEL | STRATEGY -->
>> <property name="mappingPrecedence"
>> value="MODEL"/>
>>
> This one is obsolete.
>
>> </mappingStrategy>
>>
>> <dbAdapter name="derby-embedded"/>
>> <dataSource
>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>> databaseName="/tmp/cdodb1"
>> createDatabase="create"/>
>>
>> </store>
>> </repository>
>>
>>
>>
>>
>> What do you think about that ? What throughput do you think I should
>> expect ?
> As I said that's very difficult to judge.
> I believe it's more appropriate to compare the actual values with values
> from other measures (other setups, properties, etc. or even other
> products) instead of random expectations.
> Apologies that I didn't have the time so far to provide some reference
> measures myself. Maybe this gets better now that the team starts to grow
> slowly ;-)
>
>> What are the main reasons fur such figures ?
>>
> Hmm, I *suspect* you don't like the figures? I can find no evidence your
> assessment ;-)
>
> Ahh,one more note:
> Tracing can also have a significant impact on performance and end-to-end
> throughput.
> Turn it off by all means. And please make sure that it's really turned
> off, not only invisible (i.e. console is not a target).
>
> Cheers
> /Eike


http://cedric.brun.io news and articles on eclipse and eclipse modeling.
Re: [CDO] reading/writting throughput performances [message #123623 is a reply to message #123598] Thu, 29 May 2008 13:46 Go to previous messageGo to next message
Cedric Brun is currently offline Cedric BrunFriend
Messages: 431
Registered: July 2009
Senior Member
Eike,

As I don't know (yet ;) ) all the internals of CDO I'm not sure this is
relevant, but in the testcase I'm working on (with the complex metamodel,
big model in database and 50 o/sec throughput) my profiler says that most
of the time (like 99.9%) is spent in a readString call from
CDOClassRefImpl :

public CDOClassRefImpl(ExtendedDataInput in, String defaultURI) throws
IOException
{
// TODO Optimize transfer of URIs
packageURI = in.readString();
if (packageURI == null)
{
packageURI = defaultURI;
}

classifierID = in.readInt();
}

The TODO let me think this seems relevant but I'm not sure, what do you
think of it ?

Cédric


Eike Stepper wrote:

> Hi Cédric,
>
> That sounds interesting!
> Comments inline...
>
>
> Cédric Brun schrieb:
>> Hi Eike,
>>
>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>> measuring read/write throughput when I use CDO (RC2).
>>
>> I'm using an embeded-derby database, two different metamodels (complex
>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>> Features)
>>
>> The test is a "quick and dirty" one so please do not take these figures
>> too seriously, I would just like to have your input about the "order of
>> magnitude" I should expect or settings affecting seriously these
>> performances. I tried both JVM and TCP acceptor but the performances are
>> quite similar though JVM is a bit faster.
>>
> Looks like a sign that the NIO socket channels are efficient ,-)
> You used localhost for TCP transport I guess?
>
>>
>> * First case : building a small model (10000 elements) with the simple
>> MM, only adding instances in an empty database :
>> commiting change on every add I get a 2 objects per second rate.
>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>
> Seems to be consistent with the fact that only for commits there is
> remote traffic.
> And the backend transaction management is another overhead.
> So the impact of the number objects/commit starts with bigger
> transactions.
>
>> browsing the just commited elements opening a new transaction and getting
>> a new Resource instance, I get around 30 000 objets/sec rate just
>> iterating thanks to Resource.getAllContents().
>>
> Hehe, this is a sign that client side caching works well:
> All views and transactions of a single session share (and cache) their
> object state.
> Otherwise I'd expect much smaller values.
> With his ObjectivityStore implementation Simon (cc'ed) reached rates of
> several thousands objects/sec.
> But the DBStore (which you used) is not that optimized.
>
>> * Second case : I built a big model in the database using the complex
>> metamodel. It contains 700 000 elements. The creation went pretty nicely
>> (I did not measured it exactly as it was launched during a night, but it
>> seems I got around 100 objects/sec rate in writting as I was writting
>> every 700 objects.
>>
> Sounds not too bad.
>
>>
>> Once the model is built the derby database is 222 MB (which - I heard -
>> is not so big for derby).
>>
> Hmm, I have no extensive experience with Derby.
> To be honest, I'm really no database expert (although I can write simple
> SQL statements).
>
>> Retrieving the distant resource and calling eAllContents() on it, I get a
>> 50 Objects/sec rate at best, which mean my eAllContents() call would take
>> 4 hours !
>>
> Currently I'm not sure what exactly eAllContents() does do (iteration
> order and so).
> There are several CDO features that can have a major impact on load rates!
>
> - Collections (xrefs and containment) can be loaded lazily. I don't mean
> the target objects, which are lazy by default anyway, but even the
> target IDs that are in the collections. We call that "reference
> chunking" (but search for a better name is ongoing).
> |*public interface *CDOSession *extends *CDOProtocolSession,
> |IContainer<CDOView>
> {
> *public **int *getReferenceChunkSize();
>
> *public **void *setReferenceChunkSize(*int *referenceChunkSize);|
>
>
> - Loading collection elements (target objects) can be batched (the
> current name is even worse)
> |*public interface *CDOView *extends *CDOProtocolView, INotifier
> {
> *public **int *getLoadRevisionCollectionChunkSize();
>
> *public **void *setLoadRevisionCollectionChunkSize(*int
> *loadRevisionCollectionChunkSize);|
>
>
>
> - One of the most sophisticated feature is automatic model usage
> analyzation. The result are fetch rules that are sent to the server.
> When later objects are loaded from the server, it can answer with
> additional objects that are likely to be needed by the client in the
> near future.
> |*public class *CDOViewImpl *extends
> |*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
> |CDOIDProvider,
> Adapter.Internal
> {
> *public *CDOFeatureAnalyzer getFeatureAnalyzer()
> {
> *return *featureAnalyzer;
> }
>
> *public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
> {
> *this*.featureAnalyzer = featureAnalyzer == *null *?
> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
> }|
>
>
> - There's a SmartReadAheadThread (if you're curious, please ask Simon).
> - Caches can be configured
> - ...
>
>
> Known issues of the DBStore:
> - When in auditing mode updates are always written as complete inserts
> (by design)
> - When configuring toManyReferences to a value different from
> LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
> - No (few) prepared statements are used (see
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
> - Maybe others...
>
> Please note that the DBStore was not written by me to compete with
> Hibernate in terms of performance or mapping capabilities.
> Now that Martin and I, well in the end mostly Martin, provide a
> HibernateStore and this HibernateStore gets more and more features I
> expect that there will be a trend away from the DBStore.
>
>> This is probably linked to the way the mapping to db is done, here's the
>> config I used (came from CDO-server example if I recall correctly).
>>
>>
>> <repository name="repo1">
>> <property name="overrideUUID"
>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>> <property name="supportingAudits" value="true"/>
>> <property name="verifyingRevisions" value="false"/>
>> <property name="currentLRUCapacity" value="10000"/>
>>
> I guess this one could be larger.
>
>> <property name="revisedLRUCapacity" value="100"/>
>>
>> <store type="db">
>> <!-- type: horizontal | vertical | <any
>> user-contributed type> --> <mappingStrategy
>> type="horizontal">
>> <!-- ONE_TABLE_PER_REFERENCE |
>> ONE_TABLE_PER_CLASS |
>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>> <property name="toManyReferences"
>> value="ONE_TABLE_PER_REFERENCE"/>
>>
>> <!-- LIKE_ATTRIBUTES |
>> LIKE_TO_MANY_REFERENCES --> <property
>> name="toOneReferences"
>> value="LIKE_ATTRIBUTES"/>
>>
>> <!-- MODEL | STRATEGY -->
>> <property name="mappingPrecedence"
>> value="MODEL"/>
>>
> This one is obsolete.
>
>> </mappingStrategy>
>>
>> <dbAdapter name="derby-embedded"/>
>> <dataSource
>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>> databaseName="/tmp/cdodb1"
>> createDatabase="create"/>
>>
>> </store>
>> </repository>
>>
>>
>>
>>
>> What do you think about that ? What throughput do you think I should
>> expect ?
> As I said that's very difficult to judge.
> I believe it's more appropriate to compare the actual values with values
> from other measures (other setups, properties, etc. or even other
> products) instead of random expectations.
> Apologies that I didn't have the time so far to provide some reference
> measures myself. Maybe this gets better now that the team starts to grow
> slowly ;-)
>
>> What are the main reasons fur such figures ?
>>
> Hmm, I *suspect* you don't like the figures? I can find no evidence your
> assessment ;-)
>
> Ahh,one more note:
> Tracing can also have a significant impact on performance and end-to-end
> throughput.
> Turn it off by all means. And please make sure that it's really turned
> off, not only invisible (i.e. console is not a target).
>
> Cheers
> /Eike


http://cedric.brun.io news and articles on eclipse and eclipse modeling.
Re: [CDO] reading/writting throughput performances [message #123651 is a reply to message #123598] Thu, 29 May 2008 16:13 Go to previous messageGo to next message
Simon Mc Duff is currently offline Simon Mc DuffFriend
Messages: 596
Registered: July 2009
Senior Member
This is a multi-part message in MIME format.

------=_NextPart_000_04AC_01C8C185.678D3E20
Content-Type: text/plain;
charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

We noticed a performance degradation in the following case:
=20
When we fetch a collection of objects that contains collections that we =
don't want to use.=20
Ex:
=20
Model:
=20
ClassB
{
int foo1;
List childrens1;
List childrens2;
List childrens3;
};
=20
Code:

Resource.contents contains objects of type ClassB.
=20
By fetching instances of classB, we will also fetched some items in the =
list(childrens1, childrens2, childrens3) The problem we had.. is we do =
not know if we want to use it. SO better not to download them.

So In our production(we used our own IStore), we never download any =
objects from collection.=20
I'm working at the moment to put in place all feature we used in the =
current CDO.
That one will be part of=20
https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214477. To begin =
with,CDORevisionImpl needs to supports list that are 0 even if =
referenceChunkSize =3D=3D 0.
=20
It is true that we obtain good performance.
Worst case :
1500 objects /sec (Real fetch from disk. Page that contains the objects, =
wasn't in the cache of the filesystem.)=20
=20
Best case=20
12 000 objects/Sec (Fetch objects that are already in the case of the =
filesystem)
=20
(These statistics always fetch objects from server to client. The cache =
client isn't used)
=20
We have another optimization one with SmartReadAhead but for now only =
work for UI client. (for now) This is something we will integrate as =
well in CDO. But for now we concentrate on the other things.
=20
I used an Object database. Based on my preliminary test without CDO.
I compare
=20
Hibernate vs Objectivity/EMF :=20
Objectivity/EMF is around 3 to 50 times faster than Hibernate.
=20
The worst.. you will get 30 to 240 objects/sec or
the best... you will get 500 to 4000 objects/sec
=20
Based on your comments... you are getting the worst (with Derby)!! :-)
=20
=20
Another comments about insertion.
=20
We found another problem when we insert data(DOn't know if it is still =
true). Resource.Contents is a unique list. Basically each time we insert =
one element, it needs to fetch all the collection to see if we already =
have this element. Basically.. it is time consuming. In our case we are =
not going through this stage to persist our objects... but the objects =
are not accessible through that list.
=20


If speed is an issue for you!! We (You included :-)) can optimize it. =
What I like is Eike did a really good job doing that framwork and =
optimizations are easy to add.
Also you can change your back-end without affecting you client side. =
(Maybe going to an Object database!!)
THis was one of the purpose of CDO!! :-)


Simon


"Eike Stepper" <stepper@sympedia.de> a =C3=A9crit dans le message de =
news: 483E9332.6050800@sympedia.de...
Hi C=C3=A9dric,

That sounds interesting!
Comments inline...


C=C3=A9dric Brun schrieb:=20
Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex =
one -
250 EClassifiers/300 features and a simple one - 1 Classifier/3 =
Features)

The test is a "quick and dirty" one so please do not take these figures =
too
seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.
Looks like a sign that the NIO socket channels are efficient ,-)
You used localhost for TCP transport I guess?



* First case : building a small model (10000 elements) with the simple =
MM,
only adding instances in an empty database :=20
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.
Seems to be consistent with the fact that only for commits there is =
remote traffic.
And the backend transaction management is another overhead.
So the impact of the number objects/commit starts with bigger =
transactions.


browsing the just commited elements opening a new transaction and =
getting a
new Resource instance, I get around 30 000 objets/sec rate just =
iterating
thanks to Resource.getAllContents().
Hehe, this is a sign that client side caching works well:
All views and transactions of a single session share (and cache) their =
object state.
Otherwise I'd expect much smaller values.
With his ObjectivityStore implementation Simon (cc'ed) reached rates =
of several thousands objects/sec.
But the DBStore (which you used) is not that optimized.


* Second case : I built a big model in the database using the complex
metamodel. It contains 700 000 elements. The creation went pretty nicely =
(I
did not measured it exactly as it was launched during a night, but it =
seems
I got around 100 objects/sec rate in writting as I was writting every =
700
objects.
Sounds not too bad.



Once the model is built the derby database is 222 MB (which - I heard - =
is
not so big for derby).
Hmm, I have no extensive experience with Derby.
To be honest, I'm really no database expert (although I can write =
simple SQL statements).


Retrieving the distant resource and calling eAllContents() on it, I get =
a 50
Objects/sec rate at best, which mean my eAllContents() call would take 4
hours !
Currently I'm not sure what exactly eAllContents() does do (iteration =
order and so).
There are several CDO features that can have a major impact on load =
rates!

- Collections (xrefs and containment) can be loaded lazily. I don't =
mean the target objects, which are lazy by default anyway, but even the =
target IDs that are in the collections. We call that "reference =
chunking" (but search for a better name is ongoing).

public interface CDOSession extends CDOProtocolSession, =
IContainer<CDOView>
{
public int getReferenceChunkSize();

public void setReferenceChunkSize(int referenceChunkSize); =20


- Loading collection elements (target objects) can be batched (the =
current name is even worse)

public interface CDOView extends CDOProtocolView, INotifier
{
public int getLoadRevisionCollectionChunkSize();

public void setLoadRevisionCollectionChunkSize(int =
loadRevisionCollectionChunkSize); =20


- One of the most sophisticated feature is automatic model usage =
analyzation. The result are fetch rules that are sent to the server. =
When later objects are loaded from the server, it can answer with =
additional objects that are likely to be needed by the client in the =
near future.

public class CDOViewImpl extends =
org.eclipse.net4j.internal.util.event.Notifier implements CDOView, =
CDOIDProvider, Adapter.Internal
{
public CDOFeatureAnalyzer getFeatureAnalyzer()
{
return featureAnalyzer;
}

public void setFeatureAnalyzer(CDOFeatureAnalyzer =
featureAnalyzer)
{
this.featureAnalyzer =3D featureAnalyzer =3D=3D null ? =
CDOFeatureAnalyzer.NOOP : featureAnalyzer;
} =20


- There's a SmartReadAheadThread (if you're curious, please ask =
Simon).
- Caches can be configured
- ...


Known issues of the DBStore:
- When in auditing mode updates are always written as complete inserts =
(by design)
- When configuring toManyReferences to a value different from =
LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
- No (few) prepared statements are used (see =
https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214487)
- Maybe others...

Please note that the DBStore was not written by me to compete with =
Hibernate in terms of performance or mapping capabilities.
Now that Martin and I, well in the end mostly Martin, provide a =
HibernateStore and this HibernateStore gets more and more features I =
expect that there will be a trend away from the DBStore.=20


This is probably linked to the way the mapping to db is done, here's the
config I used (came from CDO-server example if I recall correctly).


<repository name=3D"repo1">
<property name=3D"overrideUUID"
value=3D"1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
<property name=3D"supportingAudits" value=3D"true"/>
<property name=3D"verifyingRevisions" value=3D"false"/>
<property name=3D"currentLRUCapacity" value=3D"10000"/>
I guess this one could be larger.


<property name=3D"revisedLRUCapacity" value=3D"100"/>
=20
<store type=3D"db">
<!-- type: horizontal | vertical | <any =
user-contributed type> -->
<mappingStrategy type=3D"horizontal">
<!-- ONE_TABLE_PER_REFERENCE | =
ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
<property name=3D"toManyReferences" =
value=3D"ONE_TABLE_PER_REFERENCE"/>

<!-- LIKE_ATTRIBUTES | =
LIKE_TO_MANY_REFERENCES -->
<property name=3D"toOneReferences" =
value=3D"LIKE_ATTRIBUTES"/>
=20
<!-- MODEL | STRATEGY -->
<property name=3D"mappingPrecedence" =
value=3D"MODEL"/>
This one is obsolete.


</mappingStrategy>
=20
<dbAdapter name=3D"derby-embedded"/>
<dataSource =
class=3D"org.apache.derby.jdbc.EmbeddedDataSource"=20
databaseName=3D"/tmp/cdodb1"
createDatabase=3D"create"/>

</store>
</repository>




What do you think about that ? What throughput do you think I should
expect ? As I said that's very difficult to judge.
I believe it's more appropriate to compare the actual values with =
values from other measures (other setups, properties, etc. or even other =
products) instead of random expectations.
Apologies that I didn't have the time so far to provide some reference =
measures myself. Maybe this gets better now that the team starts to grow =
slowly ;-)


What are the main reasons fur such figures ?
Hmm, I *suspect* you don't like the figures? I can find no evidence =
your assessment ;-)

Ahh,one more note:
Tracing can also have a significant impact on performance and =
end-to-end throughput.
Turn it off by all means. And please make sure that it's really turned =
off, not only invisible (i.e. console is not a target).

Cheers
/Eike



------=_NextPart_000_04AC_01C8C185.678D3E20
Content-Type: text/html;
charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

=EF=BB=BF<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD><TITLE></TITLE>
<META http-equiv=3DContent-Type content=3Dtext/html;charset=3DUTF-8>
<META content=3D"MSHTML 6.00.6000.16608" name=3DGENERATOR></HEAD>
<BODY text=3D#000000 bgColor=3D#ffffff>
<DIV>We noticed a performance degradation in the following=20
case:<BR>&nbsp;<BR>When we fetch a collection of objects that contains=20
collections that we don't want to use.=20
<BR>Ex:<BR>&nbsp;<BR>Model:<BR>&nbsp;<BR>ClassB <BR>{<BR>&nbsp; int=20
foo1;<BR>&nbsp; List childrens1;<BR>&nbsp; List childrens2;<BR>&nbsp; =
List=20
childrens3;<BR>};<BR>&nbsp;<BR>Code:</DIV>
<DIV><BR>Resource.contents contains objects of type =
ClassB.<BR>&nbsp;<BR>By=20
fetching instances of classB, we will also fetched some items in the=20
list(childrens1, childrens2, childrens3)&nbsp;<FONT face=3D"Courier =
New"><FONT=20
color=3D#000000>The problem we had.. is we do not know if we want to use =
it. SO=20
better not to download them.</FONT></FONT></DIV><FONT face=3D"Courier =
New"></FONT>
<DIV><BR><FONT face=3D"Courier New" color=3D#000000>So In our =
production(we used our=20
own IStore), we never download any objects from collection. =
</FONT><BR><FONT=20
face=3D"Courier New" color=3D#000000>I'm working at the moment to put in =
place all=20
feature we used in the current CDO.</FONT><BR><FONT face=3D"Courier New" =

color=3D#000000>That one will be part of </FONT><BR><A=20
href=3D"https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214477" =
target=3D_blank><FONT=20
color=3D#0068cf>https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214477</F=
ONT></A>.=20
To begin with,CDORevisionImpl needs to supports list that are 0 even if =
<FONT=20
face=3D"Courier New" color=3D#000000>referenceChunkSize =3D=3D =
0.</FONT><BR><FONT=20
face=3D"Courier New" color=3D#000000></FONT>&nbsp;<BR><FONT =
face=3D"Courier New"=20
color=3D#000000>It is true that we obtain good =
performance.</FONT><BR><FONT=20
face=3D"Courier New" color=3D#000000>Worst case :</FONT><BR><FONT =
face=3D"Courier New"=20
color=3D#000000>1500 objects /sec (Real fetch from&nbsp;disk. Page that =
contains=20
the objects, wasn't in the cache of the =
filesystem.)&nbsp;</FONT><BR><FONT=20
face=3D"Courier New" color=3D#000000></FONT>&nbsp;<BR><FONT =
face=3D"Courier New"=20
color=3D#000000>Best case </FONT><BR><FONT face=3D"Courier New" =
color=3D#000000>12 000=20
objects/Sec (Fetch objects that are already in the case of the=20
filesystem)</FONT><BR><FONT face=3D"Courier New"=20
color=3D#000000></FONT>&nbsp;<BR>(These statistics always fetch objects =
from=20
server to client. The cache client isn't used)<BR>&nbsp;<BR><FONT=20
face=3D"Courier New" color=3D#000000>We have another optimization one =
with=20
SmartReadAhead but for now only work for UI client. (for now)&nbsp; This =
is=20
something we will integrate as well in CDO. But for now we concentrate =
on the=20
other things.</FONT><BR><FONT face=3D"Courier New"=20
color=3D#000000></FONT>&nbsp;<BR><FONT face=3D"Courier New" =
color=3D#000000>I used an=20
Object database. Based on my preliminary test without =
CDO.</FONT><BR><FONT=20
face=3D"Courier New" color=3D#000000>I compare</FONT><BR><FONT =
face=3D"Courier New"=20
color=3D#000000></FONT>&nbsp;<BR><FONT face=3D"Courier New" =
color=3D#000000>Hibernate=20
vs </FONT><FONT face=3D"Courier New" color=3D#000000>Objectivity/EMF=20
:&nbsp;</FONT><BR><FONT face=3D"Courier New" =
color=3D#000000>Objectivity/EMF is=20
around 3 to 50 times faster than Hibernate.</FONT><BR><FONT =
face=3D"Courier New"=20
color=3D#000000></FONT>&nbsp;<BR><FONT face=3D"Courier New" =
color=3D#000000>The=20
worst.. you will get 30 to 240 objects/sec or</FONT><BR><FONT =
face=3D"Courier New"=20
color=3D#000000>the best... you will get 500 to 4000 =
objects/sec</FONT><BR><FONT=20
face=3D"Courier New" color=3D#000000></FONT>&nbsp;<BR><FONT =
face=3D"Courier New"=20
color=3D#000000>Based on your comments... you are getting the worst =
(with Derby)!!=20
:-)</FONT><BR>&nbsp;<BR>&nbsp;<BR>Another comments about=20
insertion.<BR>&nbsp;<BR>We found another problem when we insert =
data(DOn't know=20
if it is still true). Resource.Contents is&nbsp;a unique list. Basically =
each=20
time we insert one element, it needs to fetch all the collection to see =
if we=20
already have this element. Basically.. it is time consuming. In our case =
we are=20
not going through this stage to persist our objects... but the objects =
are not=20
accessible through that list.<BR>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>If speed is an issue for you!! We (You =
included=20
:-)) can optimize it. What I like is Eike did a really good job doing =
that=20
framwork and optimizations are easy to add.</FONT></DIV>
<DIV><FONT face=3DArial size=3D2>Also you can change your back-end =
without affecting=20
you client side. (Maybe going to an Object database!!)</FONT></DIV>
<DIV><FONT face=3DArial size=3D2>THis was one of the purpose of CDO!!=20
:-)</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><BR>Simon<BR><BR></DIV>
<BLOCKQUOTE=20
style=3D"PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; =
BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">
<DIV>"Eike Stepper" &lt;<A=20
href=3D"mailto:stepper@sympedia.de">stepper@sympedia.de</A>&gt; a =
=C3=A9crit dans le=20
message de news: <A=20
=
href=3D"mailto:483E9332.6050800@sympedia.de">483E9332.6050800@sympedia.de=
</A>...</DIV>Hi=20
C=C3=A9dric,<BR><BR>That sounds interesting!<BR>Comments=20
inline...<BR><BR><BR>C=C3=A9dric Brun schrieb:=20
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex =
one -
250 EClassifiers/300 features and a simple one - 1 Classifier/3 =
Features)

The test is a "quick and dirty" one so please do not take these figures =
too
seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.
</PRE></BLOCKQUOTE>Looks like a sign that the NIO socket channels are=20
efficient ,-)<BR>You used localhost for TCP transport I guess?<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">
* First case : building a small model (10000 elements) with the simple =
MM,
only adding instances in an empty database :=20
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.
</PRE></BLOCKQUOTE>Seems to be consistent with the fact that only for=20
commits there is remote traffic.<BR>And the backend transaction =
management is=20
another overhead.<BR>So the impact of the number objects/commit starts =
with=20
bigger transactions.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">browsing the just commited elements opening =
a new transaction and getting a
new Resource instance, I get around 30 000 objets/sec rate just =
iterating
thanks to Resource.getAllContents().
</PRE></BLOCKQUOTE>Hehe, this is a sign that client side caching works =

well:<BR>All views and transactions of a single session share (and =
cache)=20
their object state.<BR>Otherwise I'd expect much smaller =
values.<BR>With his=20
ObjectivityStore implementation Simon (cc'ed) reached rates of several =

thousands objects/sec.<BR>But the DBStore (which you used) is not that =

optimized.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">* Second case : I built a big model in the =
database using the complex
metamodel. It contains 700 000 elements. The creation went pretty nicely =
(I
did not measured it exactly as it was launched during a night, but it =
seems
I got around 100 objects/sec rate in writting as I was writting every =
700
objects.
</PRE></BLOCKQUOTE>Sounds not too bad.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">
Once the model is built the derby database is 222 MB (which - I heard - =
is
not so big for derby).
</PRE></BLOCKQUOTE>Hmm, I have no extensive experience with =
Derby.<BR>To be=20
honest, I'm really no database expert (although I can write simple SQL =

statements).<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">Retrieving the distant resource and calling =
eAllContents() on it, I get a 50
Objects/sec rate at best, which mean my eAllContents() call would take 4
hours !
</PRE></BLOCKQUOTE>Currently I'm not sure what exactly eAllContents() =
does=20
do (iteration order and so).<BR>There are several CDO features that =
can have a=20
major impact on load rates!<BR><BR>- Collections (xrefs and =
containment) can=20
be loaded lazily. I don't mean the target objects, which are lazy by =
default=20
anyway, but even the target IDs that are in the collections. We call =
that=20
"reference chunking" (but search for a better name is ongoing).<BR>
<STYLE type=3Dtext/css>CODE {
FONT-SIZE: 10pt; MARGIN: 0px; FONT-FAMILY: Courier New, Courier
}
</STYLE>
<!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><!-- =3D Java Sourcecode to HTML automatically =
converted code =3D --><!-- =3D Java2Html Converter 5.0 [2006-02-26] by =
Markus Gebhard markus@jave.de =3D --><!-- =3D Further =
information: http://www.java2html.de =3D -->
<DIV class=3Djava align=3Dleft>
<TABLE cellSpacing=3D0 cellPadding=3D3 bgColor=3D#ffffff border=3D0>
<TBODY>
<TR><!-- start source code -->
<TD vAlign=3Dtop noWrap align=3Dleft><CODE><FONT=20
color=3D#7f0055><B>public&nbsp;interface&nbsp;</B ></FONT><FONT=20
color=3D#000000>CDOSession&nbsp;</FONT><FONT=20
color=3D#7f0055><B>extends&nbsp;</B></FONT><FONT=20
=
color=3D#000000> CDOProtocolSession,&nbsp;IContainer&lt;CDOView&g t; </FONT>=
<BR><FONT=20
color=3D#000000>{</FONT><BR><FONT =
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>int&nbsp;</B></FONT><FONT=20
color=3D#000000>getReferenceChunkSize</FONT><FONT=20
color=3D#000000>()</FONT><FONT color=3D#000000>;</FONT><BR><FONT =

color=3D#ffffff></FONT><BR><FONT =
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>void&nbsp;</B></FONT><FONT=20
color=3D#000000>setReferenceChunkSize</FONT><FONT=20
color=3D#000000>(</FONT><FONT =
color=3D#7f0055><B>int&nbsp;</B></FONT><FONT=20
color=3D#000000>referenceChunkSize</FONT><FONT =
color=3D#000000>)</FONT><FONT=20
color=3D#000000>;</FONT></CODE>=20
</TD><!-- end source code --></TR></TBODY></TABLE></DIV><!-- =3D =
END of automatically generated HTML code =3D --><BR><!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D -->-=20
Loading collection elements (target objects) can be batched (the =
current name=20
is even worse)<BR>
<STYLE type=3Dtext/css>
<!--code { font-family: Courier New, Courier; font-size: 10pt; =
margin: 0px; }-->
</STYLE>
<!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><!-- =3D Java Sourcecode to HTML automatically =
converted code =3D --><!-- =3D Java2Html Converter 5.0 [2006-02-26] by =
Markus Gebhard markus@jave.de =3D --><!-- =3D Further =
information: http://www.java2html.de =3D -->
<DIV class=3Djava align=3Dleft>
<TABLE cellSpacing=3D0 cellPadding=3D3 bgColor=3D#ffffff border=3D0>
<TBODY>
<TR><!-- start source code -->
<TD vAlign=3Dtop noWrap align=3Dleft><CODE><FONT=20
color=3D#7f0055><B>public&nbsp;interface&nbsp;</B ></FONT><FONT=20
color=3D#000000>CDOView&nbsp;</FONT><FONT=20
color=3D#7f0055><B>extends&nbsp;</B></FONT><FONT=20
color=3D#000000>CDOProtocolView,&nbsp;INotifier</FONT> <BR><FONT=20
color=3D#000000>{</FONT><BR><FONT color=3D#ffffff></FONT><FONT=20
color=3D#ffffff>&nbsp; </FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>int&nbsp;</B></FONT><FONT=20
color=3D#000000>getLoadRevisionCollectionChunkSize</FONT><FONT=20
color=3D#000000>()</FONT><FONT color=3D#000000>;</FONT><BR><FONT =

color=3D#ffffff></FONT><BR><FONT =
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>void&nbsp;</B></FONT><FONT=20
color=3D#000000>setLoadRevisionCollectionChunkSize</FONT><FONT=20
color=3D#000000>(</FONT><FONT =
color=3D#7f0055><B>int&nbsp;</B></FONT><FONT=20
color=3D#000000>loadRevisionCollectionChunkSize</FONT><FONT=20
color=3D#000000>)</FONT><FONT color=3D#000000>;</FONT></CODE> =
</TD><!-- end source code --></TR></TBODY></TABLE></DIV><!-- =3D =
END of automatically generated HTML code =3D --><!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><BR>-=20
One of the most sophisticated feature is automatic model usage =
analyzation.=20
The result are fetch rules that are sent to the server. When later =
objects are=20
loaded from the server, it can answer with additional objects that are =
likely=20
to be needed by the client in the near future.<BR>
<STYLE type=3Dtext/css>
<!--code { font-family: Courier New, Courier; font-size: 10pt; =
margin: 0px; }-->
</STYLE>
<!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><!-- =3D Java Sourcecode to HTML automatically =
converted code =3D --><!-- =3D Java2Html Converter 5.0 [2006-02-26] by =
Markus Gebhard markus@jave.de =3D --><!-- =3D Further =
information: http://www.java2html.de =3D -->
<DIV class=3Djava align=3Dleft>
<TABLE cellSpacing=3D0 cellPadding=3D3 bgColor=3D#ffffff border=3D0>
<TBODY>
<TR><!-- start source code -->
<TD vAlign=3Dtop noWrap align=3Dleft><CODE><FONT=20
color=3D#7f0055><B>public&nbsp;class&nbsp;</B></FONT ><FONT=20
color=3D#000000>CDOViewImpl&nbsp;</FONT><FONT=20
color=3D#7f0055><B>extends&nbsp;</B></FONT><FONT=20
=
color=3D#000000>org.eclipse.net4j.internal.util.event.Notifier&nbsp; </FON=
T><FONT=20
color=3D#7f0055><B>implements&nbsp;</B></FONT><FONT=20
color=3D#000000>CDOView,&nbsp;CDOIDProvider,</FONT> <FONT=20
color=3D#000000>Adapter.Internal</FONT><BR><FONT=20
color=3D#000000>{</FONT><BR><FONT color=3D#ffffff></FONT><FONT=20
color=3D#ffffff>&nbsp; </FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
=
color=3D#000000>CDOFeatureAnalyzer&nbsp;getFeatureAnalyzer </FONT><FONT=20
color=3D#000000>()</FONT><BR><FONT =
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#000000>{</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;&nbsp;&nbsp;</FONT ><FONT=20
color=3D#7f0055><B>return&nbsp;</B></FONT><FONT=20
color=3D#000000>featureAnalyzer;</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT =
color=3D#000000>}</FONT><BR><FONT=20
color=3D#ffffff></FONT><BR><FONT =
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>void&nbsp;</B></FONT><FONT=20
color=3D#000000>setFeatureAnalyzer</FONT><FONT =
color=3D#000000>(</FONT><FONT=20
=
color=3D#000000>CDOFeatureAnalyzer&nbsp;featureAnalyzer </FONT><FONT=20
color=3D#000000>)</FONT><BR><FONT =
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#000000>{</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;&nbsp;&nbsp;</FONT ><FONT=20
color=3D#7f0055><B>this</B></FONT><FONT=20
=
color=3D#000000> .featureAnalyzer&nbsp;=3D&nbsp;featureAnalyzer&n bsp;=3D=3D=
&nbsp;</FONT><FONT=20
color=3D#7f0055><B>null&nbsp;</B></FONT><FONT=20
=
color=3D#000000> ?&nbsp;CDOFeatureAnalyzer.NOOP&nbsp;:&nbsp;featu reAnalyze=
r;</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT =
color=3D#000000>}</FONT></CODE> </TD><!-- end source code =
--></TR></TBODY></TABLE></DIV><!-- =3D END of automatically =
generated HTML code =3D --><!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><BR>-=20
There's a SmartReadAheadThread (if you're curious, please ask =
Simon).<BR>-=20
Caches can be configured<BR>- ...<BR><BR><BR>Known issues of the =
DBStore:<BR>-=20
When in auditing mode updates are always written as complete inserts =
(by=20
design)<BR>- When configuring toManyReferences to a value different =
from=20
LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.<BR>- =
No (few)=20
prepared statements are used (see <A class=3Dmoz-txt-link-freetext=20
=
href=3D"https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214487">https://b=
ugs.eclipse.org/bugs/show_bug.cgi?id=3D214487</A>)<BR>-=20
Maybe others...<BR><BR>Please note that the DBStore was not written by =
me to=20
compete with Hibernate in terms of performance or mapping =
capabilities.<BR>Now=20
that Martin and I, well in the end mostly Martin, provide a =
HibernateStore and=20
this HibernateStore gets more and more features I expect that there =
will be a=20
trend away from the DBStore. <BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">This is probably linked to the way the =
mapping to db is done, here's the
config I used (came from CDO-server example if I recall correctly).


&lt;repository name=3D"repo1"&gt;
&lt;property name=3D"overrideUUID"
value=3D"1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/&gt;
&lt;property name=3D"supportingAudits" =
value=3D"true"/&gt;
&lt;property name=3D"verifyingRevisions" =
value=3D"false"/&gt;
&lt;property name=3D"currentLRUCapacity" =
value=3D"10000"/&gt;
</PRE></BLOCKQUOTE>I guess this one could be larger.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D""> &lt;property =
name=3D"revisedLRUCapacity" value=3D"100"/&gt;
=20
&lt;store type=3D"db"&gt;
&lt;!-- type: horizontal | vertical | &lt;any =
user-contributed type&gt; --&gt;
&lt;mappingStrategy type=3D"horizontal"&gt;
&lt;!-- ONE_TABLE_PER_REFERENCE | =
ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES =
--&gt;
&lt;property name=3D"toManyReferences" =
value=3D"ONE_TABLE_PER_REFERENCE"/&gt;

&lt;!-- LIKE_ATTRIBUTES | =
LIKE_TO_MANY_REFERENCES --&gt;
&lt;property name=3D"toOneReferences" =
value=3D"LIKE_ATTRIBUTES"/&gt;
=20
&lt;!-- MODEL | STRATEGY --&gt;
&lt;property name=3D"mappingPrecedence" =
value=3D"MODEL"/&gt;
</PRE></BLOCKQUOTE>This one is obsolete.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D""> =
&lt;/mappingStrategy&gt;
=20
&lt;dbAdapter name=3D"derby-embedded"/&gt;
&lt;dataSource =
class=3D"org.apache.derby.jdbc.EmbeddedDataSource"=20
databaseName=3D"/tmp/cdodb1"
createDatabase=3D"create"/&gt;

&lt;/store&gt;
&lt;/repository&gt;




What do you think about that ? What throughput do you think I should
expect ? </PRE></BLOCKQUOTE>As I said that's very difficult to =
judge.<BR>I=20
believe it's more appropriate to compare the actual values with values =
from=20
other measures (other setups, properties, etc. or even other products) =
instead=20
of random expectations.<BR>Apologies that I didn't have the time so =
far to=20
provide some reference measures myself. Maybe this gets better now =
that the=20
team starts to grow slowly ;-)<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">What are the main reasons fur such figures =
?
</PRE></BLOCKQUOTE>Hmm, I *suspect* you don't like the figures? I can =
find=20
no evidence your assessment ;-)<BR><BR>Ahh,one more note:<BR>Tracing =
can also=20
have a significant impact on performance and end-to-end =
throughput.<BR>Turn it=20
off by all means. And please make sure that it's really turned off, =
not only=20
invisible (i.e. console is not a=20
target).<BR><BR>Cheers<BR>/Eike<BR><BR><BR></BLOCKQUOTE ></BODY></HTML>

------=_NextPart_000_04AC_01C8C185.678D3E20--
Re: [CDO] reading/writting throughput performances [message #123664 is a reply to message #123623] Thu, 29 May 2008 16:23 Go to previous messageGo to next message
Simon Mc Duff is currently offline Simon Mc DuffFriend
Messages: 596
Registered: July 2009
Senior Member
Did you disable the following :
CDOSessionConfigurationImpl.setLegacySupportEnabled(true);

before opening your session ?

This could cause what you are observing ?

If your classes are generating with CDOObjectImpl as the super class, you
can disableLegacySupport by putting it at true.

Simon


"C
Re: [CDO] reading/writting throughput performances [message #123681 is a reply to message #123664] Thu, 29 May 2008 16:28 Go to previous messageGo to next message
Eclipse UserFriend
Originally posted by: stepper.sympedia.de

This is a multi-part message in MIME format.
--------------090803020509090708070909
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 8bit



Simon McDuff schrieb:
> Did you disable the following :
> CDOSessionConfigurationImpl.setLegacySupportEnabled(true);
>
> before opening your session ?
>
> This could cause what you are observing ?
>
> If your classes are generating with CDOObjectImpl as the super class, you
> can disableLegacySupport by putting it at true.
>
Good point in general!

But I added some logic some time ago that ensures that *session legacy
support* can only be enabled if the *legacy system* is available in general:

| *public **void *setDisableLegacyObjects(*boolean *disableLegacyObjects)
{
checkInactive();
*if *(!disableLegacyObjects && !FSMUtil.isLegacySystemAvailable())
{
*throw new *LegacySystemNotAvailableException();
}

*this*.disableLegacyObjects = disableLegacyObjects;
}|


Notice that the whole legacy system is currently (since the work on the
HibernateStore) broken.
The needed CDOWeaver is currently not part of the distro anymore (also
due to issues with the build system).

This altogether seems to exclude the legacy system as root cause for
performance degradation.

Cheers
/Eike



> Simon
>
>
> "C
Re: [CDO] reading/writting throughput performances [message #123696 is a reply to message #123681] Thu, 29 May 2008 16:41 Go to previous messageGo to next message
Simon Mc Duff is currently offline Simon Mc DuffFriend
Messages: 596
Registered: July 2009
Senior Member
This is a multi-part message in MIME format.

------=_NextPart_000_04D9_01C8C189.41CE2B50
Content-Type: text/plain;
charset="ISO-8859-15"
Content-Transfer-Encoding: quoted-printable

Good point Eike,

Cedric,

How do you create your CDOSessionIMpl ?
Are you using your own mechanism ?

Because it could still occured in the case we doesn`t go through =
CDOSessionConfigurationImpl and create directly the CDOSession

Ex:

CDOSession session =3D CDOSessionImpl();

By default disablyLegacySystem is false.

In this case Eike, Do we have something else to avoid that ? Maybe I`m =
not aware of other mechanism.

Simon

"Eike Stepper" <stepper@sympedia.de> a =E9crit dans le message de news: =
483ED9A4.7010302@sympedia.de...


Simon McDuff schrieb:=20
Did you disable the following :
CDOSessionConfigurationImpl.setLegacySupportEnabled(true);

before opening your session ?

This could cause what you are observing ?

If your classes are generating with CDOObjectImpl as the super class, =
you=20
can disableLegacySupport by putting it at true.
Good point in general!

But I added some logic some time ago that ensures that *session legacy =
support* can only be enabled if the *legacy system* is available in =
general:


public void setDisableLegacyObjects(boolean =
disableLegacyObjects)
{
checkInactive();
if (!disableLegacyObjects && =
!FSMUtil.isLegacySystemAvailable())
{
throw new LegacySystemNotAvailableException();
}

this.disableLegacyObjects =3D disableLegacyObjects;
} =20


Notice that the whole legacy system is currently (since the work on =
the HibernateStore) broken.
The needed CDOWeaver is currently not part of the distro anymore (also =
due to issues with the build system).

This altogether seems to exclude the legacy system as root cause for =
performance degradation.

Cheers
/Eike




Simon


"C=E9dric Brun" <cedric.brun@obeo.fr> a =E9crit dans le message de news: =

g1mbhe$2cl$1@build.eclipse.org...
Eike,

As I don't know (yet ;) ) all the internals of CDO I'm not sure this is
relevant, but in the testcase I'm working on (with the complex =
metamodel,
big model in database and 50 o/sec throughput) my profiler says that =
most
of the time (like 99.9%) is spent in a readString call from
CDOClassRefImpl :

public CDOClassRefImpl(ExtendedDataInput in, String defaultURI) throws
IOException
{
// TODO Optimize transfer of URIs
packageURI =3D in.readString();
if (packageURI =3D=3D null)
{
packageURI =3D defaultURI;
}

classifierID =3D in.readInt();
}

The TODO let me think this seems relevant but I'm not sure, what do you
think of it ?

C=E9dric


Eike Stepper wrote:

Hi C=E9dric,

That sounds interesting!
Comments inline...


C=E9dric Brun schrieb:
Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex
one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
Features)

The test is a "quick and dirty" one so please do not take these figures
too seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.

Looks like a sign that the NIO socket channels are efficient ,-)
You used localhost for TCP transport I guess?

* First case : building a small model (10000 elements) with the =
simple
MM, only adding instances in an empty database :
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.

Seems to be consistent with the fact that only for commits there =
is
remote traffic.
And the backend transaction management is another overhead.
So the impact of the number objects/commit starts with bigger
transactions.

browsing the just commited elements opening a new transaction and=20
getting
a new Resource instance, I get around 30 000 objets/sec rate just
iterating thanks to Resource.getAllContents().

Hehe, this is a sign that client side caching works well:
All views and transactions of a single session share (and cache) their
object state.
Otherwise I'd expect much smaller values.
With his ObjectivityStore implementation Simon (cc'ed) reached rates of
several thousands objects/sec.
But the DBStore (which you used) is not that optimized.

* Second case : I built a big model in the database using the =
complex
metamodel. It contains 700 000 elements. The creation went pretty nicely
(I did not measured it exactly as it was launched during a night, but it
seems I got around 100 objects/sec rate in writting as I was writting
every 700 objects.

Sounds not too bad.

Once the model is built the derby database is 222 MB (which - I =
heard -
is not so big for derby).

Hmm, I have no extensive experience with Derby.
To be honest, I'm really no database expert (although I can write simple
SQL statements).

Retrieving the distant resource and calling eAllContents() on it, =
I get=20
a
50 Objects/sec rate at best, which mean my eAllContents() call would=20
take
4 hours !

Currently I'm not sure what exactly eAllContents() does do =
(iteration
order and so).
There are several CDO features that can have a major impact on load=20
rates!

- Collections (xrefs and containment) can be loaded lazily. I don't mean
the target objects, which are lazy by default anyway, but even the
target IDs that are in the collections. We call that "reference
chunking" (but search for a better name is ongoing).
|*public interface *CDOSession *extends *CDOProtocolSession,
|IContainer<CDOView>
{
*public **int *getReferenceChunkSize();

*public **void *setReferenceChunkSize(*int *referenceChunkSize);|


- Loading collection elements (target objects) can be batched (the
current name is even worse)
|*public interface *CDOView *extends *CDOProtocolView, INotifier
{
*public **int *getLoadRevisionCollectionChunkSize();

*public **void *setLoadRevisionCollectionChunkSize(*int
*loadRevisionCollectionChunkSize);|



- One of the most sophisticated feature is automatic model usage
analyzation. The result are fetch rules that are sent to the server.
When later objects are loaded from the server, it can answer with
additional objects that are likely to be needed by the client in the
near future.
|*public class *CDOViewImpl *extends
|*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
|CDOIDProvider,
Adapter.Internal
{
*public *CDOFeatureAnalyzer getFeatureAnalyzer()
{
*return *featureAnalyzer;
}

*public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
{
*this*.featureAnalyzer =3D featureAnalyzer =3D=3D *null *?
CDOFeatureAnalyzer.NOOP : featureAnalyzer;
}|


- There's a SmartReadAheadThread (if you're curious, please ask Simon).
- Caches can be configured
- ...


Known issues of the DBStore:
- When in auditing mode updates are always written as complete inserts
(by design)
- When configuring toManyReferences to a value different from
LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
- No (few) prepared statements are used (see
https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214487)
- Maybe others...

Please note that the DBStore was not written by me to compete with
Hibernate in terms of performance or mapping capabilities.
Now that Martin and I, well in the end mostly Martin, provide a
HibernateStore and this HibernateStore gets more and more features I
expect that there will be a trend away from the DBStore.

This is probably linked to the way the mapping to db is done, =
here's the
config I used (came from CDO-server example if I recall correctly).


<repository name=3D"repo1">
<property name=3D"overrideUUID"
value=3D"1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
<property name=3D"supportingAudits" value=3D"true"/>
<property name=3D"verifyingRevisions" value=3D"false"/>
<property name=3D"currentLRUCapacity" value=3D"10000"/>

I guess this one could be larger.

<property name=3D"revisedLRUCapacity" =
value=3D"100"/>

<store type=3D"db">
<!-- type: horizontal | vertical | <any
user-contributed type> --> <mappingStrategy
type=3D"horizontal">
<!-- ONE_TABLE_PER_REFERENCE |
ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
<property name=3D"toManyReferences"
value=3D"ONE_TABLE_PER_REFERENCE"/>

<!-- LIKE_ATTRIBUTES |
LIKE_TO_MANY_REFERENCES --> <property
name=3D"toOneReferences"
value=3D"LIKE_ATTRIBUTES"/>

<!-- MODEL | STRATEGY -->
<property name=3D"mappingPrecedence"
value=3D"MODEL"/>

This one is obsolete.

</mappingStrategy>

<dbAdapter name=3D"derby-embedded"/>
<dataSource
=
class=3D"org.apache.derby.jdbc.EmbeddedDataSource"
databaseName=3D"/tmp/cdodb1"
createDatabase=3D"create"/>

</store>
</repository>




What do you think about that ? What throughput do you think I should
expect ?
As I said that's very difficult to judge.
I believe it's more appropriate to compare the actual values with values
from other measures (other setups, properties, etc. or even other
products) instead of random expectations.
Apologies that I didn't have the time so far to provide some reference
measures myself. Maybe this gets better now that the team starts to grow
slowly ;-)

What are the main reasons fur such figures ?

Hmm, I *suspect* you don't like the figures? I can find no =
evidence your
assessment ;-)

Ahh,one more note:
Tracing can also have a significant impact on performance and end-to-end
throughput.
Turn it off by all means. And please make sure that it's really turned
off, not only invisible (i.e. console is not a target).

Cheers
/Eike
=20


------=_NextPart_000_04D9_01C8C189.41CE2B50
Content-Type: text/html;
charset="ISO-8859-15"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD><TITLE></TITLE>
<META http-equiv=3DContent-Type =
content=3Dtext/html;charset=3DISO-8859-15>
<META content=3D"MSHTML 6.00.6000.16608" name=3DGENERATOR></HEAD>
<BODY text=3D#000000 bgColor=3D#ffffff>
<DIV><FONT face=3DArial size=3D2>Good point Eike,</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>Cedric,</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>How do you create your CDOSessionIMpl=20
?</FONT></DIV>
<DIV><FONT face=3DArial size=3D2>Are you using your own mechanism =
?</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>Because it could still occured in the =
case&nbsp;we=20
doesn`t go through <FONT size=3D3>CDOSessionConfigurationImpl =
and</FONT><FONT=20
size=3D2> create directly the CDOSession</FONT></FONT></DIV>
<DIV><FONT face=3DArial size=3D2><FONT =
size=3D2></FONT></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>Ex:</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2><FONT size=3D2>CDOSession session =3D=20
CDOSessionImpl();</FONT></FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>By default disablyLegacySystem is=20
false.</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>In this case Eike, Do we have something =
else to=20
avoid that ? Maybe I`m not aware of other mechanism.</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>Simon</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>"Eike Stepper" &lt;<A=20
href=3D"mailto:stepper@sympedia.de">stepper@sympedia.de</A>&gt; a =
=E9crit dans le=20
message de news: <A=20
href=3D"mailto:483ED9A4.7010302@sympedia.de">483ED9A4.7010302@sympedia.de=
</A>...</DIV>
<BLOCKQUOTE dir=3Dltr=20
style=3D"PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; =
BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px"><BR><BR>Simon=20
McDuff schrieb:=20
<BLOCKQUOTE cite=3Dmid:g1ml62$a15$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">Did you disable the following :
CDOSessionConfigurationImpl.setLegacySupportEnabled(true);

before opening your session ?

This could cause what you are observing ?

If your classes are generating with CDOObjectImpl as the super class, =
you=20
can disableLegacySupport by putting it at true.
</PRE></BLOCKQUOTE>Good point in general!<BR><BR>But I added some =
logic some=20
time ago that ensures that *session legacy support* can only be =
enabled if the=20
*legacy system* is available in general:<BR><BR>
<STYLE type=3Dtext/css>CODE {
FONT-SIZE: 10pt; MARGIN: 0px; FONT-FAMILY: Courier New, Courier
}
</STYLE>
<!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><!-- =3D Java Sourcecode to HTML automatically =
converted code =3D --><!-- =3D Java2Html Converter 5.0 [2006-02-26] by =
Markus Gebhard markus@jave.de =3D --><!-- =3D Further =
information: http://www.java2html.de =3D -->
<DIV class=3Djava align=3Dleft>
<TABLE cellSpacing=3D0 cellPadding=3D3 bgColor=3D#ffffff border=3D0>
<TBODY>
<TR><!-- start source code -->
<TD vAlign=3Dtop noWrap align=3Dleft><CODE><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>void&nbsp;</B></FONT><FONT=20
color=3D#000000>setDisableLegacyObjects</FONT><FONT=20
color=3D#000000>(</FONT><FONT=20
color=3D#7f0055><B>boolean&nbsp;</B></FONT><FONT=20
color=3D#000000>disableLegacyObjects</FONT><FONT=20
color=3D#000000>)</FONT><BR><FONT =
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#000000>{</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;&nbsp;&nbsp;</FONT ><FONT=20
color=3D#000000>checkInactive</FONT><FONT =
color=3D#000000>()</FONT><FONT=20
color=3D#000000>;</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;&nbsp;&nbsp;</FONT ><FONT=20
color=3D#7f0055><B>if&nbsp;</B></FONT><FONT =
color=3D#000000>(</FONT><FONT=20
=
color=3D#000000> !disableLegacyObjects&nbsp;&amp;&amp;&nbsp;! FSMUtil.isLeg=
acySystemAvailable</FONT><FONT=20
color=3D#000000>())</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;&nbsp;&nbsp;</FONT ><FONT=20
color=3D#000000>{</FONT><BR><FONT=20
color=3D#ffffff> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </FONT><FONT =

color=3D#7f0055><B>throw&nbsp;new&nbsp;</B></FONT ><FONT=20
color=3D#000000>LegacySystemNotAvailableException</FONT><FONT=20
color=3D#000000>()</FONT><FONT color=3D#000000>;</FONT><BR><FONT =

color=3D#ffffff>&nbsp;&nbsp;&nbsp;&nbsp;</FONT ><FONT=20
color=3D#000000>}</FONT><BR><FONT =
color=3D#ffffff></FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;&nbsp;&nbsp;</FONT ><FONT=20
color=3D#7f0055><B>this</B></FONT><FONT=20
=
color=3D#000000> .disableLegacyObjects&nbsp;=3D&nbsp;disableLegacyObj ects;=
</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT =
color=3D#000000>}</FONT></CODE> </TD><!-- end source code =
--></TR></TBODY></TABLE></DIV><!-- =3D END of automatically =
generated HTML code =3D --><!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><BR>Notice=20
that the whole legacy system is currently (since the work on the=20
HibernateStore) broken.<BR>The needed CDOWeaver is currently not part =
of the=20
distro anymore (also due to issues with the build system).<BR><BR>This =

altogether seems to exclude the legacy system as root cause for =
performance=20
degradation.<BR><BR>Cheers<BR>/Eike<BR><BR><BR><BR>
<BLOCKQUOTE cite=3Dmid:g1ml62$a15$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">Simon


"C=E9dric Brun" <A class=3Dmoz-txt-link-rfc2396E =
href=3D"mailto:cedric.brun@obeo.fr">&lt;cedric.brun@obeo.fr&gt;</A> a =
=E9crit dans le message de news:=20
<A class=3Dmoz-txt-link-abbreviated =
href=3D"mailto:g1mbhe$2cl$1@build.eclipse.org">g1mbhe$2cl$1@build.eclipse=
..org</A>...
</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">Eike,

As I don't know (yet ;) ) all the internals of CDO I'm not sure this is
relevant, but in the testcase I'm working on (with the complex =
metamodel,
big model in database and 50 o/sec throughput) my profiler says that =
most
of the time (like 99.9%) is spent in a readString call from
CDOClassRefImpl :

public CDOClassRefImpl(ExtendedDataInput in, String defaultURI) throws
IOException
{
// TODO Optimize transfer of URIs
packageURI =3D in.readString();
if (packageURI =3D=3D null)
{
packageURI =3D defaultURI;
}

classifierID =3D in.readInt();
}

The TODO let me think this seems relevant but I'm not sure, what do you
think of it ?

C=E9dric


Eike Stepper wrote:

</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">Hi C=E9dric,

That sounds interesting!
Comments inline...


C=E9dric Brun schrieb:
</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex
one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
Features)

The test is a "quick and dirty" one so please do not take these figures
too seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.

</PRE></BLOCKQUOTE><PRE wrap=3D"">Looks like a sign that the NIO =
socket channels are efficient ,-)
You used localhost for TCP transport I guess?

</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">* First case : building =
a small model (10000 elements) with the simple
MM, only adding instances in an empty database :
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.

</PRE></BLOCKQUOTE><PRE wrap=3D"">Seems to be consistent with =
the fact that only for commits there is
remote traffic.
And the backend transaction management is another overhead.
So the impact of the number objects/commit starts with bigger
transactions.

</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">browsing the just =
commited elements opening a new transaction and=20
getting
a new Resource instance, I get around 30 000 objets/sec rate just
iterating thanks to Resource.getAllContents().

</PRE></BLOCKQUOTE><PRE wrap=3D"">Hehe, this is a sign that =
client side caching works well:
All views and transactions of a single session share (and cache) their
object state.
Otherwise I'd expect much smaller values.
With his ObjectivityStore implementation Simon (cc'ed) reached rates of
several thousands objects/sec.
But the DBStore (which you used) is not that optimized.

</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">* Second case : I built =
a big model in the database using the complex
metamodel. It contains 700 000 elements. The creation went pretty nicely
(I did not measured it exactly as it was launched during a night, but it
seems I got around 100 objects/sec rate in writting as I was writting
every 700 objects.

</PRE></BLOCKQUOTE><PRE wrap=3D"">Sounds not too bad.

</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">Once the model is built =
the derby database is 222 MB (which - I heard -
is not so big for derby).

</PRE></BLOCKQUOTE><PRE wrap=3D"">Hmm, I have no extensive =
experience with Derby.
To be honest, I'm really no database expert (although I can write simple
SQL statements).

</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">Retrieving the distant =
resource and calling eAllContents() on it, I get=20
a
50 Objects/sec rate at best, which mean my eAllContents() call would=20
take
4 hours !

</PRE></BLOCKQUOTE><PRE wrap=3D"">Currently I'm not sure what =
exactly eAllContents() does do (iteration
order and so).
There are several CDO features that can have a major impact on load=20
rates!

- Collections (xrefs and containment) can be loaded lazily. I don't mean
the target objects, which are lazy by default anyway, but even the
target IDs that are in the collections. We call that "reference
chunking" (but search for a better name is ongoing).
|*public interface *CDOSession *extends *CDOProtocolSession,
|IContainer&lt;CDOView&gt;
{
*public **int *getReferenceChunkSize();

*public **void *setReferenceChunkSize(*int *referenceChunkSize);|


- Loading collection elements (target objects) can be batched (the
current name is even worse)
|*public interface *CDOView *extends *CDOProtocolView, INotifier
{
*public **int *getLoadRevisionCollectionChunkSize();

*public **void *setLoadRevisionCollectionChunkSize(*int
*loadRevisionCollectionChunkSize);|



- One of the most sophisticated feature is automatic model usage
analyzation. The result are fetch rules that are sent to the server.
When later objects are loaded from the server, it can answer with
additional objects that are likely to be needed by the client in the
near future.
|*public class *CDOViewImpl *extends
|*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
|CDOIDProvider,
Adapter.Internal
{
*public *CDOFeatureAnalyzer getFeatureAnalyzer()
{
*return *featureAnalyzer;
}

*public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
{
*this*.featureAnalyzer =3D featureAnalyzer =3D=3D *null *?
CDOFeatureAnalyzer.NOOP : featureAnalyzer;
}|


- There's a SmartReadAheadThread (if you're curious, please ask Simon).
- Caches can be configured
- ...


Known issues of the DBStore:
- When in auditing mode updates are always written as complete inserts
(by design)
- When configuring toManyReferences to a value different from
LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
- No (few) prepared statements are used (see
<A class=3Dmoz-txt-link-freetext =
href=3D"https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214487">https://b=
ugs.eclipse.org/bugs/show_bug.cgi?id=3D214487</A>)
- Maybe others...

Please note that the DBStore was not written by me to compete with
Hibernate in terms of performance or mapping capabilities.
Now that Martin and I, well in the end mostly Martin, provide a
HibernateStore and this HibernateStore gets more and more features I
expect that there will be a trend away from the DBStore.

</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">This is probably linked =
to the way the mapping to db is done, here's the
config I used (came from CDO-server example if I recall correctly).


&lt;repository name=3D"repo1"&gt;
&lt;property name=3D"overrideUUID"
value=3D"1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/&gt;
&lt;property name=3D"supportingAudits" =
value=3D"true"/&gt;
&lt;property name=3D"verifyingRevisions" =
value=3D"false"/&gt;
&lt;property name=3D"currentLRUCapacity" =
value=3D"10000"/&gt;

</PRE></BLOCKQUOTE><PRE wrap=3D"">I guess this one could be =
larger.

</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D""> =
&lt;property name=3D"revisedLRUCapacity" value=3D"100"/&gt;

&lt;store type=3D"db"&gt;
&lt;!-- type: horizontal | vertical | &lt;any
user-contributed type&gt; --&gt; =
&lt;mappingStrategy
type=3D"horizontal"&gt;
&lt;!-- ONE_TABLE_PER_REFERENCE |
ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES =
--&gt;
&lt;property name=3D"toManyReferences"
value=3D"ONE_TABLE_PER_REFERENCE"/&gt;

&lt;!-- LIKE_ATTRIBUTES |
LIKE_TO_MANY_REFERENCES --&gt; =
&lt;property
name=3D"toOneReferences"
value=3D"LIKE_ATTRIBUTES"/&gt;

&lt;!-- MODEL | STRATEGY --&gt;
&lt;property name=3D"mappingPrecedence"
value=3D"MODEL"/&gt;

</PRE></BLOCKQUOTE><PRE wrap=3D"">This one is obsolete.

</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D""> =
&lt;/mappingStrategy&gt;

&lt;dbAdapter name=3D"derby-embedded"/&gt;
&lt;dataSource
=
class=3D"org.apache.derby.jdbc.EmbeddedDataSource"
databaseName=3D"/tmp/cdodb1"
createDatabase=3D"create"/&gt;

&lt;/store&gt;
&lt;/repository&gt;




What do you think about that ? What throughput do you think I should
expect ?
</PRE></BLOCKQUOTE><PRE wrap=3D"">As I said that's very =
difficult to judge.
I believe it's more appropriate to compare the actual values with values
from other measures (other setups, properties, etc. or even other
products) instead of random expectations.
Apologies that I didn't have the time so far to provide some reference
measures myself. Maybe this gets better now that the team starts to grow
slowly ;-)

</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">What are the main =
reasons fur such figures ?

</PRE></BLOCKQUOTE><PRE wrap=3D"">Hmm, I *suspect* you don't =
like the figures? I can find no evidence your
assessment ;-)

Ahh,one more note:
Tracing can also have a significant impact on performance and end-to-end
throughput.
Turn it off by all means. And please make sure that it's really turned
off, not only invisible (i.e. console is not a target).

Cheers
/Eike
</PRE></BLOCKQUOTE></BLOCKQUOTE><PRE wrap=3D""><!---->

</PRE></BLOCKQUOTE></BLOCKQUOTE></BODY></HTML>

------=_NextPart_000_04D9_01C8C189.41CE2B50--
Re: [CDO] reading/writting throughput performances [message #123729 is a reply to message #123696] Thu, 29 May 2008 16:50 Go to previous messageGo to next message
Eclipse UserFriend
Originally posted by: stepper.sympedia.de

This is a multi-part message in MIME format.
--------------020208050506060501030003
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 8bit

Simon McDuff schrieb:
> Good point Eike,
>
> Cedric,
>
> How do you create your CDOSessionIMpl ?
> Are you using your own mechanism ?
>
> Because it could still occured in the case we doesn`t go through
> CDOSessionConfigurationImpl and create directly the CDOSession
You could be right, too. Basically I hate myself because once I
advocated for default legacy support!!! You remember these discussions?
Since several months now I'd really like to change that (and the
checkbox in the OpenSessionDialog is early evidence). I didn't dare to
do it because formerly disableLegacyObjects was a ctor arg and a
semantic change would have broken all clients without compiler warning.
Now that there are no ctor args anymore IO really consider making this
change a late one for Ganymede. Otherwise we'd have to carry this odd
issue for another year and maybe maintain two different versions of the
semantic. What do you think?

>
> Ex:
>
> CDOSession session = CDOSessionImpl();
>
> By default disablyLegacySystem is false.
>
> In this case Eike, Do we have something else to avoid that ? Maybe I`m
> not aware of other mechanism.
I can add the same check to doBeforeActivate(), too.

Cheers
/Eike

>
> Simon
>
> "Eike Stepper" <stepper@sympedia.de <mailto:stepper@sympedia.de>> a
>
Re: [CDO] reading/writting throughput performances [message #123755 is a reply to message #123729] Thu, 29 May 2008 17:11 Go to previous messageGo to next message
Mark Geib is currently offline Mark GeibFriend
Messages: 432
Registered: July 2009
Senior Member
I would vote to make the change. You will be sorry if you miss this
opportunity. The users can handle the change I think.

Just my thoughts,
Mark.



Eike Stepper wrote:
> Simon McDuff schrieb:
>> Good point Eike,
>>
>> Cedric,
>>
>> How do you create your CDOSessionIMpl ?
>> Are you using your own mechanism ?
>>
>> Because it could still occured in the case we doesn`t go through
>> CDOSessionConfigurationImpl and create directly the CDOSession
> You could be right, too. Basically I hate myself because once I
> advocated for default legacy support!!! You remember these discussions?
> Since several months now I'd really like to change that (and the
> checkbox in the OpenSessionDialog is early evidence). I didn't dare to
> do it because formerly disableLegacyObjects was a ctor arg and a
> semantic change would have broken all clients without compiler warning.
> Now that there are no ctor args anymore IO really consider making this
> change a late one for Ganymede. Otherwise we'd have to carry this odd
> issue for another year and maybe maintain two different versions of the
> semantic. What do you think?
>
>>
>> Ex:
>>
>> CDOSession session = CDOSessionImpl();
>>
>> By default disablyLegacySystem is false.
>>
>> In this case Eike, Do we have something else to avoid that ? Maybe I`m
>> not aware of other mechanism.
> I can add the same check to doBeforeActivate(), too.
>
> Cheers
> /Eike
>
>>
>> Simon
>>
>> "Eike Stepper" <stepper@sympedia.de <mailto:stepper@sympedia.de>> a
>> écrit dans le message de news: 483ED9A4.7010302@sympedia.de
>> <mailto:483ED9A4.7010302@sympedia.de>...
>>
>>
>>
>> Simon McDuff schrieb:
>>> Did you disable the following :
>>> CDOSessionConfigurationImpl.setLegacySupportEnabled(true);
>>>
>>> before opening your session ?
>>>
>>> This could cause what you are observing ?
>>>
>>> If your classes are generating with CDOObjectImpl as the super class, you
>>> can disableLegacySupport by putting it at true.
>>>
>> Good point in general!
>>
>> But I added some logic some time ago that ensures that *session
>> legacy support* can only be enabled if the *legacy system* is
>> available in general:
>>
>> | *public **void *setDisableLegacyObjects(*boolean *disableLegacyObjects)
>> {
>> checkInactive();
>> *if *(!disableLegacyObjects && !FSMUtil.isLegacySystemAvailable())
>> {
>> *throw new *LegacySystemNotAvailableException();
>> }
>>
>> *this*.disableLegacyObjects = disableLegacyObjects;
>> }|
>>
>>
>> Notice that the whole legacy system is currently (since the work
>> on the HibernateStore) broken.
>> The needed CDOWeaver is currently not part of the distro anymore
>> (also due to issues with the build system).
>>
>> This altogether seems to exclude the legacy system as root cause
>> for performance degradation.
>>
>> Cheers
>> /Eike
>>
>>
>>
>>> Simon
>>>
>>>
>>> "Cédric Brun" <cedric.brun@obeo.fr> a écrit dans le message de news:
>>> g1mbhe$2cl$1@build.eclipse..org...
>>>
>>>> Eike,
>>>>
>>>> As I don't know (yet ;) ) all the internals of CDO I'm not sure this is
>>>> relevant, but in the testcase I'm working on (with the complex metamodel,
>>>> big model in database and 50 o/sec throughput) my profiler says that most
>>>> of the time (like 99.9%) is spent in a readString call from
>>>> CDOClassRefImpl :
>>>>
>>>> public CDOClassRefImpl(ExtendedDataInput in, String defaultURI) throws
>>>> IOException
>>>> {
>>>> // TODO Optimize transfer of URIs
>>>> packageURI = in.readString();
>>>> if (packageURI == null)
>>>> {
>>>> packageURI = defaultURI;
>>>> }
>>>>
>>>> classifierID = in.readInt();
>>>> }
>>>>
>>>> The TODO let me think this seems relevant but I'm not sure, what do you
>>>> think of it ?
>>>>
>>>> Cédric
>>>>
>>>>
>>>> Eike Stepper wrote:
>>>>
>>>>
>>>>> Hi Cédric,
>>>>>
>>>>> That sounds interesting!
>>>>> Comments inline...
>>>>>
>>>>>
>>>>> Cédric Brun schrieb:
>>>>>
>>>>>> Hi Eike,
>>>>>>
>>>>>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>>>>>> measuring read/write throughput when I use CDO (RC2).
>>>>>>
>>>>>> I'm using an embeded-derby database, two different metamodels (complex
>>>>>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>>>>>> Features)
>>>>>>
>>>>>> The test is a "quick and dirty" one so please do not take these figures
>>>>>> too seriously, I would just like to have your input about the "order of
>>>>>> magnitude" I should expect or settings affecting seriously these
>>>>>> performances. I tried both JVM and TCP acceptor but the performances are
>>>>>> quite similar though JVM is a bit faster.
>>>>>>
>>>>>>
>>>>> Looks like a sign that the NIO socket channels are efficient ,-)
>>>>> You used localhost for TCP transport I guess?
>>>>>
>>>>>
>>>>>> * First case : building a small model (10000 elements) with the simple
>>>>>> MM, only adding instances in an empty database :
>>>>>> commiting change on every add I get a 2 objects per second rate.
>>>>>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>>>>>
>>>>>>
>>>>> Seems to be consistent with the fact that only for commits there is
>>>>> remote traffic.
>>>>> And the backend transaction management is another overhead.
>>>>> So the impact of the number objects/commit starts with bigger
>>>>> transactions.
>>>>>
>>>>>
>>>>>> browsing the just commited elements opening a new transaction and
>>>>>> getting
>>>>>> a new Resource instance, I get around 30 000 objets/sec rate just
>>>>>> iterating thanks to Resource.getAllContents().
>>>>>>
>>>>>>
>>>>> Hehe, this is a sign that client side caching works well:
>>>>> All views and transactions of a single session share (and cache) their
>>>>> object state.
>>>>> Otherwise I'd expect much smaller values.
>>>>> With his ObjectivityStore implementation Simon (cc'ed) reached rates of
>>>>> several thousands objects/sec.
>>>>> But the DBStore (which you used) is not that optimized.
>>>>>
>>>>>
>>>>>> * Second case : I built a big model in the database using the complex
>>>>>> metamodel. It contains 700 000 elements. The creation went pretty nicely
>>>>>> (I did not measured it exactly as it was launched during a night, but it
>>>>>> seems I got around 100 objects/sec rate in writting as I was writting
>>>>>> every 700 objects.
>>>>>>
>>>>>>
>>>>> Sounds not too bad.
>>>>>
>>>>>
>>>>>> Once the model is built the derby database is 222 MB (which - I heard -
>>>>>> is not so big for derby).
>>>>>>
>>>>>>
>>>>> Hmm, I have no extensive experience with Derby.
>>>>> To be honest, I'm really no database expert (although I can write simple
>>>>> SQL statements).
>>>>>
>>>>>
>>>>>> Retrieving the distant resource and calling eAllContents() on it, I get
>>>>>> a
>>>>>> 50 Objects/sec rate at best, which mean my eAllContents() call would
>>>>>> take
>>>>>> 4 hours !
>>>>>>
>>>>>>
>>>>> Currently I'm not sure what exactly eAllContents() does do (iteration
>>>>> order and so).
>>>>> There are several CDO features that can have a major impact on load
>>>>> rates!
>>>>>
>>>>> - Collections (xrefs and containment) can be loaded lazily. I don't mean
>>>>> the target objects, which are lazy by default anyway, but even the
>>>>> target IDs that are in the collections. We call that "reference
>>>>> chunking" (but search for a better name is ongoing).
>>>>> |*public interface *CDOSession *extends *CDOProtocolSession,
>>>>> |IContainer<CDOView>
>>>>> {
>>>>> *public **int *getReferenceChunkSize();
>>>>>
>>>>> *public **void *setReferenceChunkSize(*int *referenceChunkSize);|
>>>>>
>>>>>
>>>>> - Loading collection elements (target objects) can be batched (the
>>>>> current name is even worse)
>>>>> |*public interface *CDOView *extends *CDOProtocolView, INotifier
>>>>> {
>>>>> *public **int *getLoadRevisionCollectionChunkSize();
>>>>>
>>>>> *public **void *setLoadRevisionCollectionChunkSize(*int
>>>>> *loadRevisionCollectionChunkSize);|
>>>>>
>>>>>
>>>>>
>>>>> - One of the most sophisticated feature is automatic model usage
>>>>> analyzation. The result are fetch rules that are sent to the server.
>>>>> When later objects are loaded from the server, it can answer with
>>>>> additional objects that are likely to be needed by the client in the
>>>>> near future.
>>>>> |*public class *CDOViewImpl *extends
>>>>> |*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
>>>>> |CDOIDProvider,
>>>>> Adapter.Internal
>>>>> {
>>>>> *public *CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>> {
>>>>> *return *featureAnalyzer;
>>>>> }
>>>>>
>>>>> *public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
>>>>> {
>>>>> *this*.featureAnalyzer = featureAnalyzer == *null *?
>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>> }|
>>>>>
>>>>>
>>>>> - There's a SmartReadAheadThread (if you're curious, please ask Simon).
>>>>> - Caches can be configured
>>>>> - ...
>>>>>
>>>>>
>>>>> Known issues of the DBStore:
>>>>> - When in auditing mode updates are always written as complete inserts
>>>>> (by design)
>>>>> - When configuring toManyReferences to a value different from
>>>>> LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
>>>>> - No (few) prepared statements are used (see
>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
>>>>> - Maybe others...
>>>>>
>>>>> Please note that the DBStore was not written by me to compete with
>>>>> Hibernate in terms of performance or mapping capabilities.
>>>>> Now that Martin and I, well in the end mostly Martin, provide a
>>>>> HibernateStore and this HibernateStore gets more and more features I
>>>>> expect that there will be a trend away from the DBStore.
>>>>>
>>>>>
>>>>>> This is probably linked to the way the mapping to db is done, here's the
>>>>>> config I used (came from CDO-server example if I recall correctly).
>>>>>>
>>>>>>
>>>>>> <repository name="repo1">
>>>>>> <property name="overrideUUID"
>>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>>> <property name="supportingAudits" value="true"/>
>>>>>> <property name="verifyingRevisions" value="false"/>
>>>>>> <property name="currentLRUCapacity" value="10000"/>
>>>>>>
>>>>>>
>>>>> I guess this one could be larger.
>>>>>
>>>>>
>>>>>> <property name="revisedLRUCapacity" value="100"/>
>>>>>>
>>>>>> <store type="db">
>>>>>> <!-- type: horizontal | vertical | <any
>>>>>> user-contributed type> --> <mappingStrategy
>>>>>> type="horizontal">
>>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>>> ONE_TABLE_PER_CLASS |
>>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>>>>>> <property name="toManyReferences"
>>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>>
>>>>>> <!-- LIKE_ATTRIBUTES |
>>>>>> LIKE_TO_MANY_REFERENCES --> <property
>>>>>> name="toOneReferences"
>>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>>
>>>>>> <!-- MODEL | STRATEGY -->
>>>>>> <property name="mappingPrecedence"
>>>>>> value="MODEL"/>
>>>>>>
>>>>>>
>>>>> This one is obsolete.
>>>>>
>>>>>
>>>>>> </mappingStrategy>
>>>>>>
>>>>>> <dbAdapter name="derby-embedded"/>
>>>>>> <dataSource
>>>>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>>>> databaseName="/tmp/cdodb1"
>>>>>> createDatabase="create"/>
>>>>>>
>>>>>> </store>
>>>>>> </repository>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> What do you think about that ? What throughput do you think I should
>>>>>> expect ?
>>>>>>
>>>>> As I said that's very difficult to judge.
>>>>> I believe it's more appropriate to compare the actual values with values
>>>>> from other measures (other setups, properties, etc. or even other
>>>>> products) instead of random expectations.
>>>>> Apologies that I didn't have the time so far to provide some reference
>>>>> measures myself. Maybe this gets better now that the team starts to grow
>>>>> slowly ;-)
>>>>>
>>>>>
>>>>>> What are the main reasons fur such figures ?
>>>>>>
>>>>>>
>>>>> Hmm, I *suspect* you don't like the figures? I can find no evidence your
>>>>> assessment ;-)
>>>>>
>>>>> Ahh,one more note:
>>>>> Tracing can also have a significant impact on performance and end-to-end
>>>>> throughput.
>>>>> Turn it off by all means. And please make sure that it's really turned
>>>>> off, not only invisible (i.e. console is not a target).
>>>>>
>>>>> Cheers
>>>>> /Eike
>>>>>
>>>
>>>
>>>
>>
Re: [CDO] reading/writting throughput performances [message #123770 is a reply to message #123755] Thu, 29 May 2008 17:13 Go to previous messageGo to next message
Eclipse UserFriend
Originally posted by: stepper.sympedia.de

Mark,

Thank you to encourage me ;-)
RC3 will have the change.

Cheers
/Eike



Mark Geib schrieb:
> I would vote to make the change. You will be sorry if you miss this
> opportunity. The users can handle the change I think.
>
> Just my thoughts,
> Mark.
>
>
>
> Eike Stepper wrote:
>> Simon McDuff schrieb:
>>> Good point Eike,
>>>
>>> Cedric,
>>>
>>> How do you create your CDOSessionIMpl ?
>>> Are you using your own mechanism ?
>>>
>>> Because it could still occured in the case we doesn`t go through
>>> CDOSessionConfigurationImpl and create directly the CDOSession
>> You could be right, too. Basically I hate myself because once I
>> advocated for default legacy support!!! You remember these
>> discussions? Since several months now I'd really like to change that
>> (and the checkbox in the OpenSessionDialog is early evidence). I
>> didn't dare to do it because formerly disableLegacyObjects was a ctor
>> arg and a semantic change would have broken all clients without
>> compiler warning. Now that there are no ctor args anymore IO really
>> consider making this change a late one for Ganymede. Otherwise we'd
>> have to carry this odd issue for another year and maybe maintain two
>> different versions of the semantic. What do you think?
>>
>>>
>>> Ex:
>>>
>>> CDOSession session = CDOSessionImpl();
>>>
>>> By default disablyLegacySystem is false.
>>>
>>> In this case Eike, Do we have something else to avoid that ? Maybe
>>> I`m not aware of other mechanism.
>> I can add the same check to doBeforeActivate(), too.
>>
>> Cheers
>> /Eike
>>
>>>
>>> Simon
>>>
>>> "Eike Stepper" <stepper@sympedia.de <mailto:stepper@sympedia.de>> a
>>> écrit dans le message de news: 483ED9A4.7010302@sympedia.de
>>> <mailto:483ED9A4.7010302@sympedia.de>...
>>>
>>>
>>>
>>> Simon McDuff schrieb:
>>>> Did you disable the following :
>>>> CDOSessionConfigurationImpl.setLegacySupportEnabled(true);
>>>>
>>>> before opening your session ?
>>>>
>>>> This could cause what you are observing ?
>>>>
>>>> If your classes are generating with CDOObjectImpl as the super
>>>> class, you can disableLegacySupport by putting it at true.
>>>>
>>> Good point in general!
>>>
>>> But I added some logic some time ago that ensures that *session
>>> legacy support* can only be enabled if the *legacy system* is
>>> available in general:
>>>
>>> | *public **void *setDisableLegacyObjects(*boolean
>>> *disableLegacyObjects)
>>> {
>>> checkInactive();
>>> *if *(!disableLegacyObjects &&
>>> !FSMUtil.isLegacySystemAvailable())
>>> {
>>> *throw new *LegacySystemNotAvailableException();
>>> }
>>>
>>> *this*.disableLegacyObjects = disableLegacyObjects;
>>> }|
>>>
>>>
>>> Notice that the whole legacy system is currently (since the work
>>> on the HibernateStore) broken.
>>> The needed CDOWeaver is currently not part of the distro anymore
>>> (also due to issues with the build system).
>>>
>>> This altogether seems to exclude the legacy system as root cause
>>> for performance degradation.
>>>
>>> Cheers
>>> /Eike
>>>
>>>
>>>
>>>> Simon
>>>>
>>>>
>>>> "Cédric Brun" <cedric.brun@obeo.fr> a écrit dans le message de
>>>> news: g1mbhe$2cl$1@build.eclipse..org...
>>>>
>>>>> Eike,
>>>>>
>>>>> As I don't know (yet ;) ) all the internals of CDO I'm not
>>>>> sure this is
>>>>> relevant, but in the testcase I'm working on (with the complex
>>>>> metamodel,
>>>>> big model in database and 50 o/sec throughput) my profiler
>>>>> says that most
>>>>> of the time (like 99.9%) is spent in a readString call from
>>>>> CDOClassRefImpl :
>>>>>
>>>>> public CDOClassRefImpl(ExtendedDataInput in, String
>>>>> defaultURI) throws
>>>>> IOException
>>>>> {
>>>>> // TODO Optimize transfer of URIs
>>>>> packageURI = in.readString();
>>>>> if (packageURI == null)
>>>>> {
>>>>> packageURI = defaultURI;
>>>>> }
>>>>>
>>>>> classifierID = in.readInt();
>>>>> }
>>>>>
>>>>> The TODO let me think this seems relevant but I'm not sure,
>>>>> what do you
>>>>> think of it ?
>>>>>
>>>>> Cédric
>>>>>
>>>>>
>>>>> Eike Stepper wrote:
>>>>>
>>>>>
>>>>>> Hi Cédric,
>>>>>>
>>>>>> That sounds interesting!
>>>>>> Comments inline...
>>>>>>
>>>>>>
>>>>>> Cédric Brun schrieb:
>>>>>>
>>>>>>> Hi Eike,
>>>>>>>
>>>>>>> I've been messing around CDO lately and I hacked a few ugly
>>>>>>> unit-tests
>>>>>>> measuring read/write throughput when I use CDO (RC2).
>>>>>>>
>>>>>>> I'm using an embeded-derby database, two different
>>>>>>> metamodels (complex
>>>>>>> one - 250 EClassifiers/300 features and a simple one - 1
>>>>>>> Classifier/3
>>>>>>> Features)
>>>>>>>
>>>>>>> The test is a "quick and dirty" one so please do not take
>>>>>>> these figures
>>>>>>> too seriously, I would just like to have your input about
>>>>>>> the "order of
>>>>>>> magnitude" I should expect or settings affecting seriously
>>>>>>> these
>>>>>>> performances. I tried both JVM and TCP acceptor but the
>>>>>>> performances are
>>>>>>> quite similar though JVM is a bit faster.
>>>>>>>
>>>>>>>
>>>>>> Looks like a sign that the NIO socket channels are efficient ,-)
>>>>>> You used localhost for TCP transport I guess?
>>>>>>
>>>>>>
>>>>>>> * First case : building a small model (10000 elements) with
>>>>>>> the simple
>>>>>>> MM, only adding instances in an empty database :
>>>>>>> commiting change on every add I get a 2 objects per second
>>>>>>> rate.
>>>>>>> commiting changes every 10 adds - I get a 30 objects per
>>>>>>> second rate.
>>>>>>>
>>>>>>>
>>>>>> Seems to be consistent with the fact that only for commits
>>>>>> there is
>>>>>> remote traffic.
>>>>>> And the backend transaction management is another overhead.
>>>>>> So the impact of the number objects/commit starts with bigger
>>>>>> transactions.
>>>>>>
>>>>>>
>>>>>>> browsing the just commited elements opening a new
>>>>>>> transaction and getting
>>>>>>> a new Resource instance, I get around 30 000 objets/sec rate
>>>>>>> just
>>>>>>> iterating thanks to Resource.getAllContents().
>>>>>>>
>>>>>>>
>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>> All views and transactions of a single session share (and
>>>>>> cache) their
>>>>>> object state.
>>>>>> Otherwise I'd expect much smaller values.
>>>>>> With his ObjectivityStore implementation Simon (cc'ed)
>>>>>> reached rates of
>>>>>> several thousands objects/sec.
>>>>>> But the DBStore (which you used) is not that optimized.
>>>>>>
>>>>>>
>>>>>>> * Second case : I built a big model in the database using
>>>>>>> the complex
>>>>>>> metamodel. It contains 700 000 elements. The creation went
>>>>>>> pretty nicely
>>>>>>> (I did not measured it exactly as it was launched during a
>>>>>>> night, but it
>>>>>>> seems I got around 100 objects/sec rate in writting as I was
>>>>>>> writting
>>>>>>> every 700 objects.
>>>>>>>
>>>>>>>
>>>>>> Sounds not too bad.
>>>>>>
>>>>>>
>>>>>>> Once the model is built the derby database is 222 MB (which
>>>>>>> - I heard -
>>>>>>> is not so big for derby).
>>>>>>>
>>>>>>>
>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>> To be honest, I'm really no database expert (although I can
>>>>>> write simple
>>>>>> SQL statements).
>>>>>>
>>>>>>
>>>>>>> Retrieving the distant resource and calling eAllContents()
>>>>>>> on it, I get a
>>>>>>> 50 Objects/sec rate at best, which mean my eAllContents()
>>>>>>> call would take
>>>>>>> 4 hours !
>>>>>>>
>>>>>>>
>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>> (iteration
>>>>>> order and so).
>>>>>> There are several CDO features that can have a major impact
>>>>>> on load rates!
>>>>>>
>>>>>> - Collections (xrefs and containment) can be loaded lazily. I
>>>>>> don't mean
>>>>>> the target objects, which are lazy by default anyway, but
>>>>>> even the
>>>>>> target IDs that are in the collections. We call that "reference
>>>>>> chunking" (but search for a better name is ongoing).
>>>>>> |*public interface *CDOSession *extends *CDOProtocolSession,
>>>>>> |IContainer<CDOView>
>>>>>> {
>>>>>> *public **int *getReferenceChunkSize();
>>>>>>
>>>>>> *public **void *setReferenceChunkSize(*int
>>>>>> *referenceChunkSize);|
>>>>>>
>>>>>>
>>>>>> - Loading collection elements (target objects) can be batched
>>>>>> (the
>>>>>> current name is even worse)
>>>>>> |*public interface *CDOView *extends *CDOProtocolView, INotifier
>>>>>> {
>>>>>> *public **int *getLoadRevisionCollectionChunkSize();
>>>>>>
>>>>>> *public **void *setLoadRevisionCollectionChunkSize(*int
>>>>>> *loadRevisionCollectionChunkSize);|
>>>>>>
>>>>>>
>>>>>>
>>>>>> - One of the most sophisticated feature is automatic model usage
>>>>>> analyzation. The result are fetch rules that are sent to the
>>>>>> server.
>>>>>> When later objects are loaded from the server, it can answer
>>>>>> with
>>>>>> additional objects that are likely to be needed by the client
>>>>>> in the
>>>>>> near future.
>>>>>> |*public class *CDOViewImpl *extends
>>>>>> |*org.eclipse.net4j.internal.util.event.Notifier *implements
>>>>>> *CDOView,
>>>>>> |CDOIDProvider,
>>>>>> Adapter.Internal
>>>>>> {
>>>>>> *public *CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>>> {
>>>>>> *return *featureAnalyzer;
>>>>>> }
>>>>>>
>>>>>> *public **void *setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>>> featureAnalyzer)
>>>>>> {
>>>>>> *this*.featureAnalyzer = featureAnalyzer == *null *?
>>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>>> }|
>>>>>>
>>>>>>
>>>>>> - There's a SmartReadAheadThread (if you're curious, please
>>>>>> ask Simon).
>>>>>> - Caches can be configured
>>>>>> - ...
>>>>>>
>>>>>>
>>>>>> Known issues of the DBStore:
>>>>>> - When in auditing mode updates are always written as
>>>>>> complete inserts
>>>>>> (by design)
>>>>>> - When configuring toManyReferences to a value different from
>>>>>> LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
>>>>>> - No (few) prepared statements are used (see
>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
>>>>>> - Maybe others...
>>>>>>
>>>>>> Please note that the DBStore was not written by me to compete
>>>>>> with
>>>>>> Hibernate in terms of performance or mapping capabilities.
>>>>>> Now that Martin and I, well in the end mostly Martin, provide a
>>>>>> HibernateStore and this HibernateStore gets more and more
>>>>>> features I
>>>>>> expect that there will be a trend away from the DBStore.
>>>>>>
>>>>>>
>>>>>>> This is probably linked to the way the mapping to db is
>>>>>>> done, here's the
>>>>>>> config I used (came from CDO-server example if I recall
>>>>>>> correctly).
>>>>>>>
>>>>>>>
>>>>>>> <repository name="repo1">
>>>>>>> <property name="overrideUUID"
>>>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>>>> <property name="supportingAudits"
>>>>>>> value="true"/>
>>>>>>> <property name="verifyingRevisions"
>>>>>>> value="false"/>
>>>>>>> <property name="currentLRUCapacity"
>>>>>>> value="10000"/>
>>>>>>>
>>>>>>>
>>>>>> I guess this one could be larger.
>>>>>>
>>>>>>
>>>>>>> <property name="revisedLRUCapacity"
>>>>>>> value="100"/>
>>>>>>>
>>>>>>> <store type="db">
>>>>>>> <!-- type: horizontal | vertical | <any
>>>>>>> user-contributed type> -->
>>>>>>> <mappingStrategy
>>>>>>> type="horizontal">
>>>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>>>> ONE_TABLE_PER_CLASS |
>>>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY |
>>>>>>> LIKE_ATTRIBUTES -->
>>>>>>> <property
>>>>>>> name="toManyReferences"
>>>>>>>
>>>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>>>
>>>>>>> <!-- LIKE_ATTRIBUTES |
>>>>>>> LIKE_TO_MANY_REFERENCES -->
>>>>>>> <property
>>>>>>> name="toOneReferences"
>>>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>>>
>>>>>>> <!-- MODEL | STRATEGY -->
>>>>>>> <property
>>>>>>> name="mappingPrecedence"
>>>>>>> value="MODEL"/>
>>>>>>>
>>>>>>>
>>>>>> This one is obsolete.
>>>>>>
>>>>>>
>>>>>>> </mappingStrategy>
>>>>>>>
>>>>>>> <dbAdapter name="derby-embedded"/>
>>>>>>> <dataSource
>>>>>>>
>>>>>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>>>>> databaseName="/tmp/cdodb1"
>>>>>>> createDatabase="create"/>
>>>>>>>
>>>>>>> </store>
>>>>>>> </repository>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> What do you think about that ? What throughput do you think
>>>>>>> I should
>>>>>>> expect ?
>>>>>>>
>>>>>> As I said that's very difficult to judge.
>>>>>> I believe it's more appropriate to compare the actual values
>>>>>> with values
>>>>>> from other measures (other setups, properties, etc. or even
>>>>>> other
>>>>>> products) instead of random expectations.
>>>>>> Apologies that I didn't have the time so far to provide some
>>>>>> reference
>>>>>> measures myself. Maybe this gets better now that the team
>>>>>> starts to grow
>>>>>> slowly ;-)
>>>>>>
>>>>>>
>>>>>>> What are the main reasons fur such figures ?
>>>>>>>
>>>>>>>
>>>>>> Hmm, I *suspect* you don't like the figures? I can find no
>>>>>> evidence your
>>>>>> assessment ;-)
>>>>>>
>>>>>> Ahh,one more note:
>>>>>> Tracing can also have a significant impact on performance and
>>>>>> end-to-end
>>>>>> throughput.
>>>>>> Turn it off by all means. And please make sure that it's
>>>>>> really turned
>>>>>> off, not only invisible (i.e. console is not a target).
>>>>>>
>>>>>> Cheers
>>>>>> /Eike
>>>>>>
>>>>
>>>>
>>>>
>>>
Re: [CDO] reading/writting throughput performances [message #123787 is a reply to message #123651] Thu, 29 May 2008 17:27 Go to previous messageGo to next message
Eclipse UserFriend
Originally posted by: stepper.sympedia.de

This is a multi-part message in MIME format.
--------------050305010008090907040306
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Guys,

Comments inline...


Simon McDuff schrieb:
> We noticed a performance degradation in the following case:
>
> When we fetch a collection of objects that contains collections that
> we don't want to use.
> Ex:
>
> Model:
>
> ClassB
> {
> int foo1;
> List childrens1;
> List childrens2;
> List childrens3;
> };
>
> Code:
>
> Resource.contents contains objects of type ClassB.
>
> By fetching instances of classB, we will also fetched some items in
> the list(childrens1, childrens2, childrens3) The problem we had.. is
> we do not know if we want to use it. SO better not to download them.
>
> So In our production(we used our own IStore), we never download any
> objects from collection.
> I'm working at the moment to put in place all feature we used in the
> current CDO.
> That one will be part of
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477. To begin
> with,CDORevisionImpl needs to supports list that are 0 even if
> referenceChunkSize == 0.
>
> It is true that we obtain good performance.
> Worst case :
> 1500 objects /sec (Real fetch from disk. Page that contains the
> objects, wasn't in the cache of the filesystem.)
>
> Best case
> 12 000 objects/Sec (Fetch objects that are already in the case of the
> filesystem)
>
> (These statistics always fetch objects from server to client. The
> cache client isn't used)
>
> We have another optimization one with SmartReadAhead but for now only
> work for UI client. (for now) This is something we will integrate as
> well in CDO. But for now we concentrate on the other things.
>
> I used an Object database. Based on my preliminary test without CDO.
> I compare
>
> Hibernate vs Objectivity/EMF :
> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>
> The worst.. you will get 30 to 240 objects/sec or
> the best... you will get 500 to 4000 objects/sec
>
> Based on your comments... you are getting the worst (with Derby)!! :-)
And with the DBStore!!
I'd really like to encourage you to use the HibernateStore.
It is much younger than the DBStore but Hibernate itself should do a
much better job on mapping than the DBStore.

That said, the DBStore will, for multiple reasons, never cease to exist
in CDO.

>
>
> Another comments about insertion.
>
> We found another problem when we insert data(DOn't know if it is still
> true). Resource.Contents is a unique list. Basically each time we
> insert one element, it needs to fetch all the collection to see if we
> already have this element. Basically.. it is time consuming. In our
> case we are not going through this stage to persist our objects... but
> the objects are not accessible through that list.
I was not aware of this. Sounds as if we should offer an alternative
implementation for Resource.Contents.
Could you please open a Bugzilla to discuss/track this?

>
>
>
> If speed is an issue for you!! We (You included :-)) can optimize it.
> What I like is Eike did a really good job doing that framwork and
> optimizations are easy to add.
> Also you can change your back-end without affecting you client side.
> (Maybe going to an Object database!!)
> THis was one of the purpose of CDO!! :-)
Thank you ;-)
Indeed, CDO was intended to focus of the protocol aspects, the
repository and the client side addition to the EMF concepts.
I feel that this will become clearer the more IStore implementations we
get that depend on mature 3rd party code for the mapping part.

Cheers
/Eike


>
>
> Simon
>
> "Eike Stepper" <stepper@sympedia.de <mailto:stepper@sympedia.de>>
> a écrit dans le message de news: 483E9332.6050800@sympedia.de
> <mailto:483E9332.6050800@sympedia.de>...
> Hi Cédric,
>
> That sounds interesting!
> Comments inline...
>
>
> Cédric Brun schrieb:
>> Hi Eike,
>>
>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>> measuring read/write throughput when I use CDO (RC2).
>>
>> I'm using an embeded-derby database, two different metamodels (complex one -
>> 250 EClassifiers/300 features and a simple one - 1 Classifier/3 Features)
>>
>> The test is a "quick and dirty" one so please do not take these figures too
>> seriously, I would just like to have your input about the "order of
>> magnitude" I should expect or settings affecting seriously these
>> performances. I tried both JVM and TCP acceptor but the performances are
>> quite similar though JVM is a bit faster.
>>
> Looks like a sign that the NIO socket channels are efficient ,-)
> You used localhost for TCP transport I guess?
>
>> * First case : building a small model (10000 elements) with the simple MM,
>> only adding instances in an empty database :
>> commiting change on every add I get a 2 objects per second rate.
>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>
> Seems to be consistent with the fact that only for commits there
> is remote traffic.
> And the backend transaction management is another overhead.
> So the impact of the number objects/commit starts with bigger
> transactions.
>
>> browsing the just commited elements opening a new transaction and getting a
>> new Resource instance, I get around 30 000 objets/sec rate just iterating
>> thanks to Resource.getAllContents().
>>
> Hehe, this is a sign that client side caching works well:
> All views and transactions of a single session share (and cache)
> their object state.
> Otherwise I'd expect much smaller values.
> With his ObjectivityStore implementation Simon (cc'ed) reached
> rates of several thousands objects/sec.
> But the DBStore (which you used) is not that optimized.
>
>> * Second case : I built a big model in the database using the complex
>> metamodel. It contains 700 000 elements. The creation went pretty nicely (I
>> did not measured it exactly as it was launched during a night, but it seems
>> I got around 100 objects/sec rate in writting as I was writting every 700
>> objects.
>>
> Sounds not too bad.
>
>> Once the model is built the derby database is 222 MB (which - I heard - is
>> not so big for derby).
>>
> Hmm, I have no extensive experience with Derby.
> To be honest, I'm really no database expert (although I can write
> simple SQL statements).
>
>> Retrieving the distant resource and calling eAllContents() on it, I get a 50
>> Objects/sec rate at best, which mean my eAllContents() call would take 4
>> hours !
>>
> Currently I'm not sure what exactly eAllContents() does do
> (iteration order and so).
> There are several CDO features that can have a major impact on
> load rates!
>
> - Collections (xrefs and containment) can be loaded lazily. I
> don't mean the target objects, which are lazy by default anyway,
> but even the target IDs that are in the collections. We call that
> "reference chunking" (but search for a better name is ongoing).
> |*public interface *CDOSession *extends *CDOProtocolSession, IContainer<CDOView>
> {
> *public **int *getReferenceChunkSize();
>
> *public **void *setReferenceChunkSize(*int *referenceChunkSize);|
>
>
> - Loading collection elements (target objects) can be batched (the
> current name is even worse)
> |*public interface *CDOView *extends *CDOProtocolView, INotifier
> {
> *public **int *getLoadRevisionCollectionChunkSize();
>
> *public **void *setLoadRevisionCollectionChunkSize(*int *loadRevisionCollectionChunkSize);|
>
>
>
> - One of the most sophisticated feature is automatic model usage
> analyzation. The result are fetch rules that are sent to the
> server. When later objects are loaded from the server, it can
> answer with additional objects that are likely to be needed by the
> client in the near future.
> |*public class *CDOViewImpl *extends *org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView, CDOIDProvider,
> Adapter.Internal
> {
> *public *CDOFeatureAnalyzer getFeatureAnalyzer()
> {
> *return *featureAnalyzer;
> }
>
> *public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
> {
> *this*.featureAnalyzer = featureAnalyzer == *null *? CDOFeatureAnalyzer.NOOP : featureAnalyzer;
> }|
>
>
> - There's a SmartReadAheadThread (if you're curious, please ask
> Simon).
> - Caches can be configured
> - ...
>
>
> Known issues of the DBStore:
> - When in auditing mode updates are always written as complete
> inserts (by design)
> - When configuring toManyReferences to a value different from
> LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
> - No (few) prepared statements are used (see
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
> - Maybe others...
>
> Please note that the DBStore was not written by me to compete with
> Hibernate in terms of performance or mapping capabilities.
> Now that Martin and I, well in the end mostly Martin, provide a
> HibernateStore and this HibernateStore gets more and more features
> I expect that there will be a trend away from the DBStore.
>
>> This is probably linked to the way the mapping to db is done, here's the
>> config I used (came from CDO-server example if I recall correctly).
>>
>>
>> <repository name="repo1">
>> <property name="overrideUUID"
>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>> <property name="supportingAudits" value="true"/>
>> <property name="verifyingRevisions" value="false"/>
>> <property name="currentLRUCapacity" value="10000"/>
>>
> I guess this one could be larger.
>
>> <property name="revisedLRUCapacity" value="100"/>
>>
>> <store type="db">
>> <!-- type: horizontal | vertical | <any user-contributed type> -->
>> <mappingStrategy type="horizontal">
>> <!-- ONE_TABLE_PER_REFERENCE | ONE_TABLE_PER_CLASS |
>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>> <property name="toManyReferences" value="ONE_TABLE_PER_REFERENCE"/>
>>
>> <!-- LIKE_ATTRIBUTES | LIKE_TO_MANY_REFERENCES -->
>> <property name="toOneReferences" value="LIKE_ATTRIBUTES"/>
>>
>> <!-- MODEL | STRATEGY -->
>> <property name="mappingPrecedence" value="MODEL"/>
>>
> This one is obsolete.
>
>> </mappingStrategy>
>>
>> <dbAdapter name="derby-embedded"/>
>> <dataSource class="org.apache.derby.jdbc.EmbeddedDataSource"
>> databaseName="/tmp/cdodb1"
>> createDatabase="create"/>
>>
>> </store>
>> </repository>
>>
>>
>>
>>
>> What do you think about that ? What throughput do you think I should
>> expect ?
> As I said that's very difficult to judge.
> I believe it's more appropriate to compare the actual values with
> values from other measures (other setups, properties, etc. or even
> other products) instead of random expectations.
> Apologies that I didn't have the time so far to provide some
> reference measures myself. Maybe this gets better now that the
> team starts to grow slowly ;-)
>
>> What are the main reasons fur such figures ?
>>
> Hmm, I *suspect* you don't like the figures? I can find no
> evidence your assessment ;-)
>
> Ahh,one more note:
> Tracing can also have a significant impact on performance and
> end-to-end throughput.
> Turn it off by all means. And please make sure that it's really
> turned off, not only invisible (i.e. console is not a target).
>
> Cheers
> /Eike
>
>

--------------050305010008090907040306
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Guys,<br>
<br>
Comments inline...<br>
<br>
<br>
Simon McDuff schrieb:
<blockquote cite="mid:g1mkji$h5d$1@build.eclipse.org" type="cite">
<title></title>
<meta http-equiv="Content-Type" content="text/html;charset=UTF-8">
<meta content="MSHTML 6.00.6000.16608" name="GENERATOR">
<div>We noticed a performance degradation in the following case:<br>
 <br>
When we fetch a collection of objects that contains collections that we
don't want to use. <br>
Ex:<br>
 <br>
Model:<br>
 <br>
ClassB<br>
{<br>
  int foo1;<br>
  List childrens1;<br>
  List childrens2;<br>
  List childrens3;<br>
};<br>
 <br>
Code:</div>
<div><br>
Resource.contents contains objects of type ClassB.<br>
 <br>
By fetching instances of classB, we will also fetched some items in the
list(childrens1, childrens2, childrens3) <font face="Courier New"><font
color="#000000">The problem we had.. is we do not know if we want to
use it. SO better not to download them.</font></font></div>
<div><br>
<font color="#000000" face="Courier New">So In our production(we used
our own IStore), we never download any objects from collection. </font><br>
<font color="#000000" face="Courier New">I'm working at the moment to
put in place all feature we used in the current CDO.</font><br>
<font color="#000000" face="Courier New">That one will be part of </font><br>
<a moz-do-not-send="true"
href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477"
target="_blank"><font color="#0068cf">https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477</font></a>.
To begin with,CDORevisionImpl needs to supports list that are 0 even if
<font color="#000000" face="Courier New">referenceChunkSize == 0.</font><br>
 <br>
<font color="#000000" face="Courier New">It is true that we obtain
good performance.</font><br>
<font color="#000000" face="Courier New">Worst case :</font><br>
<font color="#000000" face="Courier New">1500 objects /sec (Real
fetch from disk. Page that contains the objects, wasn't in the cache of
the filesystem.) </font><br>
 <br>
<font color="#000000" face="Courier New">Best case </font><br>
<font color="#000000" face="Courier New">12 000 objects/Sec (Fetch
objects that are already in the case of the filesystem)</font><br>
 <br>
(These statistics always fetch objects from server to client. The cache
client isn't used)<br>
 <br>
<font color="#000000" face="Courier New">We have another optimization
one with SmartReadAhead but for now only work for UI client. (for now) 
This is something we will integrate as well in CDO. But for now we
concentrate on the other things.</font><br>
 <br>
<font color="#000000" face="Courier New">I used an Object database.
Based on my preliminary test without CDO.</font><br>
<font color="#000000" face="Courier New">I compare</font><br>
 <br>
<font color="#000000" face="Courier New">Hibernate vs </font><font
color="#000000" face="Courier New">Objectivity/EMF : </font><br>
<font color="#000000" face="Courier New">Objectivity/EMF is around 3
to 50 times faster than Hibernate.</font><br>
 <br>
<font color="#000000" face="Courier New">The worst.. you will get 30
to 240 objects/sec or</font><br>
<font color="#000000" face="Courier New">the best... you will get 500
to 4000 objects/sec</font><br>
 <br>
<font color="#000000" face="Courier New">Based on your comments...
you are getting the worst (with Derby)!! :-)</font><br>
</div>
</blockquote>
And with the DBStore!!<br>
I'd really like to encourage you to use the HibernateStore. <br>
It is much younger than the DBStore but Hibernate itself should do a
much better job on mapping than the DBStore.<br>
<br>
That said, the DBStore will, for multiple reasons, never cease to exist
in CDO.<br>
<br>
<blockquote cite="mid:g1mkji$h5d$1@build.eclipse.org" type="cite">
<div> <br>
 <br>
Another comments about insertion.<br>
 <br>
We found another problem when we insert data(DOn't know if it is still
true). Resource.Contents is a unique list. Basically each time we
insert one element, it needs to fetch all the collection to see if we
already have this element. Basically.. it is time consuming. In our
case we are not going through this stage to persist our objects... but
the objects are not accessible through that list.<br>
</div>
</blockquote>
I was not aware of this. Sounds as if we should offer an alternative
implementation for Resource.Contents.<br>
Could you please open a Bugzilla to discuss/track this?<br>
<br>
<blockquote cite="mid:g1mkji$h5d$1@build.eclipse.org" type="cite">
<div> </div>
<div> </div>
<div> </div>
<div><font face="Arial" size="2">If speed is an issue for you!! We
(You included :-)) can optimize it. What I like is Eike did a really
good job doing that framwork and optimizations are easy to add.</font></div>
<div><font face="Arial" size="2">Also you can change your back-end
without affecting you client side. (Maybe going to an Object database!!)</font></div>
<div><font face="Arial" size="2">THis was one of the purpose of CDO!!
:-)</font></div>
</blockquote>
Thank you ;-)<br>
Indeed, CDO was intended to focus of the protocol aspects, the
repository and the client side addition to the EMF concepts.<br>
I feel that this will become clearer the more IStore implementations we
get that depend on mature 3rd party code for the mapping part.<br>
<br>
Cheers<br>
/Eike<br>
<br>
<br>
<blockquote cite="mid:g1mkji$h5d$1@build.eclipse.org" type="cite">
<div> </div>
<div><br>
Simon<br>
<br>
</div>
<blockquote
style="border-left: 2px solid rgb(0, 0, 0); padding-right: 0px; padding-left: 5px; margin-left: 5px; margin-right: 0px;">
<div>"Eike Stepper" &lt;<a moz-do-not-send="true"
href="mailto:stepper@sympedia.de">stepper@sympedia.de</a>&gt; a écrit
dans le message de news: <a moz-do-not-send="true"
href="mailto:483E9332.6050800@sympedia.de">483E9332.6050800@sympedia.de</a>...</div>
Hi Cédric,<br>
<br>
That sounds interesting!<br>
Comments inline...<br>
<br>
<br>
Cédric Brun schrieb:
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex one -
250 EClassifiers/300 features and a simple one - 1 Classifier/3 Features)

The test is a "quick and dirty" one so please do not take these figures too
seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.
</pre>
</blockquote>
Looks like a sign that the NIO socket channels are efficient ,-)<br>
You used localhost for TCP transport I guess?<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">* First case : building a small model (10000 elements) with the simple MM,
only adding instances in an empty database :
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.
</pre>
</blockquote>
Seems to be consistent with the fact that only for commits there is
remote traffic.<br>
And the backend transaction management is another overhead.<br>
So the impact of the number objects/commit starts with bigger
transactions.<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">browsing the just commited elements opening a new transaction and getting a
new Resource instance, I get around 30 000 objets/sec rate just iterating
thanks to Resource.getAllContents().
</pre>
</blockquote>
Hehe, this is a sign that client side caching works well:<br>
All views and transactions of a single session share (and cache) their
object state.<br>
Otherwise I'd expect much smaller values.<br>
With his ObjectivityStore implementation Simon (cc'ed) reached rates of
several thousands objects/sec.<br>
But the DBStore (which you used) is not that optimized.<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">* Second case : I built a big model in the database using the complex
metamodel. It contains 700 000 elements. The creation went pretty nicely (I
did not measured it exactly as it was launched during a night, but it seems
I got around 100 objects/sec rate in writting as I was writting every 700
objects.
</pre>
</blockquote>
Sounds not too bad.<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">Once the model is built the derby database is 222 MB (which - I heard - is
not so big for derby).
</pre>
</blockquote>
Hmm, I have no extensive experience with Derby.<br>
To be honest, I'm really no database expert (although I can write
simple SQL statements).<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">Retrieving the distant resource and calling eAllContents() on it, I get a 50
Objects/sec rate at best, which mean my eAllContents() call would take 4
hours !
</pre>
</blockquote>
Currently I'm not sure what exactly eAllContents() does do (iteration
order and so).<br>
There are several CDO features that can have a major impact on load
rates!<br>
<br>
- Collections (xrefs and containment) can be loaded lazily. I don't
mean the target objects, which are lazy by default anyway, but even the
target IDs that are in the collections. We call that "reference
chunking" (but search for a better name is ongoing).<br>
<style type="text/css">CODE {
FONT-SIZE: 10pt; MARGIN: 0px; FONT-FAMILY: Courier New, Courier
}
</style><!-- ======================================================== --><!-- = Java Sourcecode to HTML automatically converted code = --><!-- = Java2Html Converter 5.0 [2006-02-26] by Markus Gebhard markus@jave.de = --><!-- = Further information: http://www.java2html.de = -->
<div class="java" align="left">
<table bgcolor="#ffffff" border="0" cellpadding="3" cellspacing="0">
<tbody>
<tr>
<!-- start source code --> <td align="left" nowrap="nowrap"
valign="top"><code><font color="#7f0055"><b>public interface </b></font><font
color="#000000">CDOSession </font><font color="#7f0055"><b>extends </b></font><font
color="#000000">CDOProtocolSession, IContainer&lt;CDOView&gt; </font><br>
<font color="#000000">{</font><br>
<font color="#ffffff">  </font><font color="#7f0055"><b>public </b></font><font
color="#7f0055"><b>int </b></font><font color="#000000">getReferenceChunkSize</font><font
color="#000000">()</font><font color="#000000">;</font><br>
<br>
<font color="#ffffff">  </font><font color="#7f0055"><b>public </b></font><font
color="#7f0055"><b>void </b></font><font color="#000000">setReferenceChunkSize</font><font
color="#000000">(</font><font color="#7f0055"><b>int </b></font><font
color="#000000">referenceChunkSize</font><font color="#000000">)</font><font
color="#000000">;</font></code> </td>
<!-- end source code --></tr>
</tbody>
</table>
</div>
<!-- = END of automatically generated HTML code = --><br>
<!-- ======================================================== -->-
Loading collection elements (target objects) can be batched (the
current name is even worse)<br>
<style type="text/css">
<!--code { font-family: Courier New, Courier; font-size: 10pt; margin: 0px; }-->
</style><!-- ======================================================== --><!-- = Java Sourcecode to HTML automatically converted code = --><!-- = Java2Html Converter 5.0 [2006-02-26] by Markus Gebhard markus@jave.de = --><!-- = Further information: http://www.java2html.de = -->
<div class="java" align="left">
<table bgcolor="#ffffff" border="0" cellpadding="3" cellspacing="0">
<tbody>
<tr>
<!-- start source code --> <td align="left" nowrap="nowrap"
valign="top"><code><font color="#7f0055"><b>public interface </b></font><font
color="#000000">CDOView </font><font color="#7f0055"><b>extends </b></font><font
color="#000000">CDOProtocolView, INotifier</font><br>
<font color="#000000">{</font><br>
<font color="#ffffff">  </font><font color="#7f0055"><b>public </b></font><font
color="#7f0055"><b>int </b></font><font color="#000000">getLoadRevisionCollectionChunkSize</font> <font
color="#000000">()</font><font color="#000000">;</font><br>
<br>
<font color="#ffffff">  </font><font color="#7f0055"><b>public </b></font><font
color="#7f0055"><b>void </b></font><font color="#000000">setLoadRevisionCollectionChunkSize</font> <font
color="#000000">(</font><font color="#7f0055"><b>int </b></font><font
color="#000000">loadRevisionCollectionChunkSize</font><font
color="#000000">)</font><font color="#000000">;</font></code> </td>
<!-- end source code --></tr>
</tbody>
</table>
</div>
<!-- = END of automatically generated HTML code = --><!-- ======================================================== --><br>
- One of the most sophisticated feature is automatic model usage
analyzation. The result are fetch rules that are sent to the server.
When later objects are loaded from the server, it can answer with
additional objects that are likely to be needed by the client in the
near future.<br>
<style type="text/css">
<!--code { font-family: Courier New, Courier; font-size: 10pt; margin: 0px; }-->
</style><!-- ======================================================== --><!-- = Java Sourcecode to HTML automatically converted code = --><!-- = Java2Html Converter 5.0 [2006-02-26] by Markus Gebhard markus@jave.de = --><!-- = Further information: http://www.java2html.de = -->
<div class="java" align="left">
<table bgcolor="#ffffff" border="0" cellpadding="3" cellspacing="0">
<tbody>
<tr>
<!-- start source code --> <td align="left" nowrap="nowrap"
valign="top"><code><font color="#7f0055"><b>public class </b></font><font
color="#000000">CDOViewImpl </font><font color="#7f0055"><b>extends </b></font><font
color="#000000">org.eclipse.net4j.internal.util.event.Notifier  </font><font
color="#7f0055"><b>implements </b></font><font color="#000000">CDOView, CDOIDProvider,</font>
<font color="#000000">Adapter.Internal</font><br>
<font color="#000000">{</font><br>
<font color="#ffffff">  </font><font color="#7f0055"><b>public </b></font><font
color="#000000">CDOFeatureAnalyzer getFeatureAnalyzer</font ><font
color="#000000">()</font><br>
<font color="#ffffff">  </font><font color="#000000">{</font><br>
<font color="#ffffff">    </font><font color="#7f0055"><b>return </b></font><font
color="#000000">featureAnalyzer;</font><br>
<font color="#ffffff">  </font><font color="#000000">}</font><br>
<br>
<font color="#ffffff">  </font><font color="#7f0055"><b>public </b></font><font
color="#7f0055"><b>void </b></font><font color="#000000">setFeatureAnalyzer</font><font
color="#000000">(</font><font color="#000000">CDOFeatureAnalyzer featureAnalyzer</font ><font
color="#000000">)</font><br>
<font color="#ffffff">  </font><font color="#000000">{</font><br>
<font color="#ffffff">    </font><font color="#7f0055"><b>this</b></font><font
color="#000000">.featureAnalyzer = featureAnalyzer ==  </font><font
color="#7f0055"><b>null </b></font><font color="#000000">? CDOFeatureAnalyzer.NOOP : featureAnalyzer; </font><br>
<font color="#ffffff">  </font><font color="#000000">}</font></code>
</td>
<!-- end source code --></tr>
</tbody>
</table>
</div>
<!-- = END of automatically generated HTML code = --><!-- ======================================================== --><br>
- There's a SmartReadAheadThread (if you're curious, please ask Simon).<br>
- Caches can be configured<br>
- ...<br>
<br>
<br>
Known issues of the DBStore:<br>
- When in auditing mode updates are always written as complete inserts
(by design)<br>
- When configuring toManyReferences to a value different from
LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.<br>
- No (few) prepared statements are used (see <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487">https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487</a>)<br>
- Maybe others...<br>
<br>
Please note that the DBStore was not written by me to compete with
Hibernate in terms of performance or mapping capabilities.<br>
Now that Martin and I, well in the end mostly Martin, provide a
HibernateStore and this HibernateStore gets more and more features I
expect that there will be a trend away from the DBStore. <br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">This is probably linked to the way the mapping to db is done, here's the
config I used (came from CDO-server example if I recall correctly).


&lt;repository name="repo1"&gt;
&lt;property name="overrideUUID"
value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/&gt;
&lt;property name="supportingAudits" value="true"/&gt;
&lt;property name="verifyingRevisions" value="false"/&gt;
&lt;property name="currentLRUCapacity" value="10000"/&gt;
</pre>
</blockquote>
I guess this one could be larger.<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap=""> &lt;property name="revisedLRUCapacity" value="100"/&gt;

&lt;store type="db"&gt;
&lt;!-- type: horizontal | vertical | &lt;any user-contributed type&gt; --&gt;
&lt;mappingStrategy type="horizontal"&gt;
&lt;!-- ONE_TABLE_PER_REFERENCE | ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES --&gt;
&lt;property name="toManyReferences" value="ONE_TABLE_PER_REFERENCE"/&gt;

&lt;!-- LIKE_ATTRIBUTES | LIKE_TO_MANY_REFERENCES --&gt;
&lt;property name="toOneReferences" value="LIKE_ATTRIBUTES"/&gt;

&lt;!-- MODEL | STRATEGY --&gt;
&lt;property name="mappingPrecedence" value="MODEL"/&gt;
</pre>
</blockquote>
This one is obsolete.<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap=""> &lt;/mappingStrategy&gt;

&lt;dbAdapter name="derby-embedded"/&gt;
&lt;dataSource class="org.apache.derby.jdbc.EmbeddedDataSource"
databaseName="/tmp/cdodb1"
createDatabase="create"/&gt;

&lt;/store&gt;
&lt;/repository&gt;




What do you think about that ? What throughput do you think I should
expect ? </pre>
</blockquote>
As I said that's very difficult to judge.<br>
I believe it's more appropriate to compare the actual values with
values from other measures (other setups, properties, etc. or even
other products) instead of random expectations.<br>
Apologies that I didn't have the time so far to provide some reference
measures myself. Maybe this gets better now that the team starts to
grow slowly ;-)<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">What are the main reasons fur such figures ?
</pre>
</blockquote>
Hmm, I *suspect* you don't like the figures? I can find no evidence
your assessment ;-)<br>
<br>
Ahh,one more note:<br>
Tracing can also have a significant impact on performance and
end-to-end throughput.<br>
Turn it off by all means. And please make sure that it's really turned
off, not only invisible (i.e. console is not a target).<br>
<br>
Cheers<br>
/Eike<br>
<br>
<br>
</blockquote>
</blockquote>
</body>
</html>

--------------050305010008090907040306--
Re: [CDO] reading/writting throughput performances [message #123816 is a reply to message #123623] Thu, 29 May 2008 17:34 Go to previous messageGo to next message
Eclipse UserFriend
Originally posted by: stepper.sympedia.de

This is a multi-part message in MIME format.
--------------010307040705010407070206
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi Cedric,

Without your measure I'd say this is unlikely because there are probably
dozens of readString() calls everywhere.
And that would have hit all other users as well.

But maybe it's simply the mass of CDOClassRefImpls that are to be
transfered in your scenario.
Can you please file an enhancement request?
Then I can work on an optimization that transfers single inegers instead
of URI Strings.

Cheers
/Eike



Cédric Brun schrieb:
> Eike,
>
> As I don't know (yet ;) ) all the internals of CDO I'm not sure this is
> relevant, but in the testcase I'm working on (with the complex metamodel,
> big model in database and 50 o/sec throughput) my profiler says that most
> of the time (like 99.9%) is spent in a readString call from
> CDOClassRefImpl :
>
> public CDOClassRefImpl(ExtendedDataInput in, String defaultURI) throws
> IOException
> {
> // TODO Optimize transfer of URIs
> packageURI = in.readString();
> if (packageURI == null)
> {
> packageURI = defaultURI;
> }
>
> classifierID = in.readInt();
> }
>
> The TODO let me think this seems relevant but I'm not sure, what do you
> think of it ?
>
> Cédric
>
>
> Eike Stepper wrote:
>
>
>> Hi Cédric,
>>
>> That sounds interesting!
>> Comments inline...
>>
>>
>> Cédric Brun schrieb:
>>
>>> Hi Eike,
>>>
>>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>>> measuring read/write throughput when I use CDO (RC2).
>>>
>>> I'm using an embeded-derby database, two different metamodels (complex
>>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>>> Features)
>>>
>>> The test is a "quick and dirty" one so please do not take these figures
>>> too seriously, I would just like to have your input about the "order of
>>> magnitude" I should expect or settings affecting seriously these
>>> performances. I tried both JVM and TCP acceptor but the performances are
>>> quite similar though JVM is a bit faster.
>>>
>>>
>> Looks like a sign that the NIO socket channels are efficient ,-)
>> You used localhost for TCP transport I guess?
>>
>>
>>> * First case : building a small model (10000 elements) with the simple
>>> MM, only adding instances in an empty database :
>>> commiting change on every add I get a 2 objects per second rate.
>>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>>
>>>
>> Seems to be consistent with the fact that only for commits there is
>> remote traffic.
>> And the backend transaction management is another overhead.
>> So the impact of the number objects/commit starts with bigger
>> transactions.
>>
>>
>>> browsing the just commited elements opening a new transaction and getting
>>> a new Resource instance, I get around 30 000 objets/sec rate just
>>> iterating thanks to Resource.getAllContents().
>>>
>>>
>> Hehe, this is a sign that client side caching works well:
>> All views and transactions of a single session share (and cache) their
>> object state.
>> Otherwise I'd expect much smaller values.
>> With his ObjectivityStore implementation Simon (cc'ed) reached rates of
>> several thousands objects/sec.
>> But the DBStore (which you used) is not that optimized.
>>
>>
>>> * Second case : I built a big model in the database using the complex
>>> metamodel. It contains 700 000 elements. The creation went pretty nicely
>>> (I did not measured it exactly as it was launched during a night, but it
>>> seems I got around 100 objects/sec rate in writting as I was writting
>>> every 700 objects.
>>>
>>>
>> Sounds not too bad.
>>
>>
>>> Once the model is built the derby database is 222 MB (which - I heard -
>>> is not so big for derby).
>>>
>>>
>> Hmm, I have no extensive experience with Derby.
>> To be honest, I'm really no database expert (although I can write simple
>> SQL statements).
>>
>>
>>> Retrieving the distant resource and calling eAllContents() on it, I get a
>>> 50 Objects/sec rate at best, which mean my eAllContents() call would take
>>> 4 hours !
>>>
>>>
>> Currently I'm not sure what exactly eAllContents() does do (iteration
>> order and so).
>> There are several CDO features that can have a major impact on load rates!
>>
>> - Collections (xrefs and containment) can be loaded lazily. I don't mean
>> the target objects, which are lazy by default anyway, but even the
>> target IDs that are in the collections. We call that "reference
>> chunking" (but search for a better name is ongoing).
>> |*public interface *CDOSession *extends *CDOProtocolSession,
>> |IContainer<CDOView>
>> {
>> *public **int *getReferenceChunkSize();
>>
>> *public **void *setReferenceChunkSize(*int *referenceChunkSize);|
>>
>>
>> - Loading collection elements (target objects) can be batched (the
>> current name is even worse)
>> |*public interface *CDOView *extends *CDOProtocolView, INotifier
>> {
>> *public **int *getLoadRevisionCollectionChunkSize();
>>
>> *public **void *setLoadRevisionCollectionChunkSize(*int
>> *loadRevisionCollectionChunkSize);|
>>
>>
>>
>> - One of the most sophisticated feature is automatic model usage
>> analyzation. The result are fetch rules that are sent to the server.
>> When later objects are loaded from the server, it can answer with
>> additional objects that are likely to be needed by the client in the
>> near future.
>> |*public class *CDOViewImpl *extends
>> |*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
>> |CDOIDProvider,
>> Adapter.Internal
>> {
>> *public *CDOFeatureAnalyzer getFeatureAnalyzer()
>> {
>> *return *featureAnalyzer;
>> }
>>
>> *public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
>> {
>> *this*.featureAnalyzer = featureAnalyzer == *null *?
>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>> }|
>>
>>
>> - There's a SmartReadAheadThread (if you're curious, please ask Simon).
>> - Caches can be configured
>> - ...
>>
>>
>> Known issues of the DBStore:
>> - When in auditing mode updates are always written as complete inserts
>> (by design)
>> - When configuring toManyReferences to a value different from
>> LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
>> - No (few) prepared statements are used (see
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
>> - Maybe others...
>>
>> Please note that the DBStore was not written by me to compete with
>> Hibernate in terms of performance or mapping capabilities.
>> Now that Martin and I, well in the end mostly Martin, provide a
>> HibernateStore and this HibernateStore gets more and more features I
>> expect that there will be a trend away from the DBStore.
>>
>>
>>> This is probably linked to the way the mapping to db is done, here's the
>>> config I used (came from CDO-server example if I recall correctly).
>>>
>>>
>>> <repository name="repo1">
>>> <property name="overrideUUID"
>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>> <property name="supportingAudits" value="true"/>
>>> <property name="verifyingRevisions" value="false"/>
>>> <property name="currentLRUCapacity" value="10000"/>
>>>
>>>
>> I guess this one could be larger.
>>
>>
>>> <property name="revisedLRUCapacity" value="100"/>
>>>
>>> <store type="db">
>>> <!-- type: horizontal | vertical | <any
>>> user-contributed type> --> <mappingStrategy
>>> type="horizontal">
>>> <!-- ONE_TABLE_PER_REFERENCE |
>>> ONE_TABLE_PER_CLASS |
>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>>> <property name="toManyReferences"
>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>
>>> <!-- LIKE_ATTRIBUTES |
>>> LIKE_TO_MANY_REFERENCES --> <property
>>> name="toOneReferences"
>>> value="LIKE_ATTRIBUTES"/>
>>>
>>> <!-- MODEL | STRATEGY -->
>>> <property name="mappingPrecedence"
>>> value="MODEL"/>
>>>
>>>
>> This one is obsolete.
>>
>>
>>> </mappingStrategy>
>>>
>>> <dbAdapter name="derby-embedded"/>
>>> <dataSource
>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>> databaseName="/tmp/cdodb1"
>>> createDatabase="create"/>
>>>
>>> </store>
>>> </repository>
>>>
>>>
>>>
>>>
>>> What do you think about that ? What throughput do you think I should
>>> expect ?
>>>
>> As I said that's very difficult to judge.
>> I believe it's more appropriate to compare the actual values with values
>> from other measures (other setups, properties, etc. or even other
>> products) instead of random expectations.
>> Apologies that I didn't have the time so far to provide some reference
>> measures myself. Maybe this gets better now that the team starts to grow
>> slowly ;-)
>>
>>
>>> What are the main reasons fur such figures ?
>>>
>>>
>> Hmm, I *suspect* you don't like the figures? I can find no evidence your
>> assessment ;-)
>>
>> Ahh,one more note:
>> Tracing can also have a significant impact on performance and end-to-end
>> throughput.
>> Turn it off by all means. And please make sure that it's really turned
>> off, not only invisible (i.e. console is not a target).
>>
>> Cheers
>> /Eike
>>
>
>

--------------010307040705010407070206
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Cedric,<br>
<br>
Without your measure I'd say this is unlikely because there are
probably dozens of readString() calls everywhere.<br>
And that would have hit all other users as well.<br>
<br>
But maybe it's simply the mass of CDOClassRefImpls  that are to be
transfered in your scenario.<br>
Can you please file an enhancement request? <br>
Then I can work on an optimization that transfers single inegers
instead of URI Strings.<br>
<br>
Cheers<br>
/Eike<br>
<br>
<br>
<br>
Cédric Brun schrieb:
<blockquote cite="mid:g1mbhe$2cl$1@build.eclipse.org" type="cite">
<pre wrap="">Eike,

As I don't know (yet ;) ) all the internals of CDO I'm not sure this is
relevant, but in the testcase I'm working on (with the complex metamodel,
big model in database and 50 o/sec throughput) my profiler says that most
of the time (like 99.9%) is spent in a readString call from
CDOClassRefImpl :

public CDOClassRefImpl(ExtendedDataInput in, String defaultURI) throws
IOException
{
// TODO Optimize transfer of URIs
packageURI = in.readString();
if (packageURI == null)
{
packageURI = defaultURI;
}

classifierID = in.readInt();
}

The TODO let me think this seems relevant but I'm not sure, what do you
think of it ?

Cédric


Eike Stepper wrote:

</pre>
<blockquote type="cite">
<pre wrap="">Hi Cédric,

That sounds interesting!
Comments inline...


Cédric Brun schrieb:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex
one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
Features)

The test is a "quick and dirty" one so please do not take these figures
too seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.

</pre>
</blockquote>
<pre wrap="">Looks like a sign that the NIO socket channels are efficient ,-)
You used localhost for TCP transport I guess?

</pre>
<blockquote type="cite">
<pre wrap="">* First case : building a small model (10000 elements) with the simple
MM, only adding instances in an empty database :
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.

</pre>
</blockquote>
<pre wrap="">Seems to be consistent with the fact that only for commits there is
remote traffic.
And the backend transaction management is another overhead.
So the impact of the number objects/commit starts with bigger
transactions.

</pre>
<blockquote type="cite">
<pre wrap="">browsing the just commited elements opening a new transaction and getting
a new Resource instance, I get around 30 000 objets/sec rate just
iterating thanks to Resource.getAllContents().

</pre>
</blockquote>
<pre wrap="">Hehe, this is a sign that client side caching works well:
All views and transactions of a single session share (and cache) their
object state.
Otherwise I'd expect much smaller values.
With his ObjectivityStore implementation Simon (cc'ed) reached rates of
several thousands objects/sec.
But the DBStore (which you used) is not that optimized.

</pre>
<blockquote type="cite">
<pre wrap="">* Second case : I built a big model in the database using the complex
metamodel. It contains 700 000 elements. The creation went pretty nicely
(I did not measured it exactly as it was launched during a night, but it
seems I got around 100 objects/sec rate in writting as I was writting
every 700 objects.

</pre>
</blockquote>
<pre wrap="">Sounds not too bad.

</pre>
<blockquote type="cite">
<pre wrap="">Once the model is built the derby database is 222 MB (which - I heard -
is not so big for derby).

</pre>
</blockquote>
<pre wrap="">Hmm, I have no extensive experience with Derby.
To be honest, I'm really no database expert (although I can write simple
SQL statements).

</pre>
<blockquote type="cite">
<pre wrap="">Retrieving the distant resource and calling eAllContents() on it, I get a
50 Objects/sec rate at best, which mean my eAllContents() call would take
4 hours !

</pre>
</blockquote>
<pre wrap="">Currently I'm not sure what exactly eAllContents() does do (iteration
order and so).
There are several CDO features that can have a major impact on load rates!

- Collections (xrefs and containment) can be loaded lazily. I don't mean
the target objects, which are lazy by default anyway, but even the
target IDs that are in the collections. We call that "reference
chunking" (but search for a better name is ongoing).
|*public interface *CDOSession *extends *CDOProtocolSession,
|IContainer&lt;CDOView&gt;
{
*public **int *getReferenceChunkSize();

*public **void *setReferenceChunkSize(*int *referenceChunkSize);|


- Loading collection elements (target objects) can be batched (the
current name is even worse)
|*public interface *CDOView *extends *CDOProtocolView, INotifier
{
*public **int *getLoadRevisionCollectionChunkSize();

*public **void *setLoadRevisionCollectionChunkSize(*int
*loadRevisionCollectionChunkSize);|



- One of the most sophisticated feature is automatic model usage
analyzation. The result are fetch rules that are sent to the server.
When later objects are loaded from the server, it can answer with
additional objects that are likely to be needed by the client in the
near future.
|*public class *CDOViewImpl *extends
|*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
|CDOIDProvider,
Adapter.Internal
{
*public *CDOFeatureAnalyzer getFeatureAnalyzer()
{
*return *featureAnalyzer;
}

*public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
{
*this*.featureAnalyzer = featureAnalyzer == *null *?
CDOFeatureAnalyzer.NOOP : featureAnalyzer;
}|


- There's a SmartReadAheadThread (if you're curious, please ask Simon).
- Caches can be configured
- ...


Known issues of the DBStore:
- When in auditing mode updates are always written as complete inserts
(by design)
- When configuring toManyReferences to a value different from
LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
- No (few) prepared statements are used (see
<a class="moz-txt-link-freetext" href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487">https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487</a>)
- Maybe others...

Please note that the DBStore was not written by me to compete with
Hibernate in terms of performance or mapping capabilities.
Now that Martin and I, well in the end mostly Martin, provide a
HibernateStore and this HibernateStore gets more and more features I
expect that there will be a trend away from the DBStore.

</pre>
<blockquote type="cite">
<pre wrap="">This is probably linked to the way the mapping to db is done, here's the
config I used (came from CDO-server example if I recall correctly).


&lt;repository name="repo1"&gt;
&lt;property name="overrideUUID"
value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/&gt;
&lt;property name="supportingAudits" value="true"/&gt;
&lt;property name="verifyingRevisions" value="false"/&gt;
&lt;property name="currentLRUCapacity" value="10000"/&gt;

</pre>
</blockquote>
<pre wrap="">I guess this one could be larger.

</pre>
<blockquote type="cite">
<pre wrap=""> &lt;property name="revisedLRUCapacity" value="100"/&gt;

&lt;store type="db"&gt;
&lt;!-- type: horizontal | vertical | &lt;any
user-contributed type&gt; --&gt; &lt;mappingStrategy
type="horizontal"&gt;
&lt;!-- ONE_TABLE_PER_REFERENCE |
ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES --&gt;
&lt;property name="toManyReferences"
value="ONE_TABLE_PER_REFERENCE"/&gt;

&lt;!-- LIKE_ATTRIBUTES |
LIKE_TO_MANY_REFERENCES --&gt; &lt;property
name="toOneReferences"
value="LIKE_ATTRIBUTES"/&gt;

&lt;!-- MODEL | STRATEGY --&gt;
&lt;property name="mappingPrecedence"
value="MODEL"/&gt;

</pre>
</blockquote>
<pre wrap="">This one is obsolete.

</pre>
<blockquote type="cite">
<pre wrap=""> &lt;/mappingStrategy&gt;

&lt;dbAdapter name="derby-embedded"/&gt;
&lt;dataSource
class="org.apache.derby.jdbc.EmbeddedDataSource"
databaseName="/tmp/cdodb1"
createDatabase="create"/&gt;

&lt;/store&gt;
&lt;/repository&gt;




What do you think about that ? What throughput do you think I should
expect ?
</pre>
</blockquote>
<pre wrap="">As I said that's very difficult to judge.
I believe it's more appropriate to compare the actual values with values
from other measures (other setups, properties, etc. or even other
products) instead of random expectations.
Apologies that I didn't have the time so far to provide some reference
measures myself. Maybe this gets better now that the team starts to grow
slowly ;-)

</pre>
<blockquote type="cite">
<pre wrap="">What are the main reasons fur such figures ?

</pre>
</blockquote>
<pre wrap="">Hmm, I *suspect* you don't like the figures? I can find no evidence your
assessment ;-)

Ahh,one more note:
Tracing can also have a significant impact on performance and end-to-end
throughput.
Turn it off by all means. And please make sure that it's really turned
off, not only invisible (i.e. console is not a target).

Cheers
/Eike
</pre>
</blockquote>
<pre wrap=""><!---->
</pre>
</blockquote>
</body>
</html>

--------------010307040705010407070206--
Re: [CDO] reading/writting throughput performances [message #123831 is a reply to message #123610] Thu, 29 May 2008 17:42 Go to previous messageGo to next message
Eclipse UserFriend
Originally posted by: stepper.sympedia.de

This is a multi-part message in MIME format.
--------------030500070800010205060903
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Cédric Brun schrieb:
> Eike,
>
> thanks for your detailled answer, I'll digg further and try to profile the
> Net4J/CDO stack so that I understand the "50 objects/sec rate" when
> retrieving data (which I think is surprisingly low).
>
Looking forward to new insights!

> Concerning the logging, what can I change to be *sure* that no logging will
> occurs ?
>
In Net4j/CDO there's a basic difference between logging and tracing.
Logging is for important production information (I always advise against
using the DEBUG loglevel!) and can't be switched off.

Tracing can be configured in several dimensions, per bundle, per tracer
hierachy, per trace context (usually == classname).
If you launch an OSGi Framework or Eclipse Application you can control
tracing via the Tracing page of the launch config.
In stand-alone mode call OMPlatform.INSTANCE.setDebugging(false).

Simply removing all ITraceHandlers is not enough.
If you set a breakpoint in
org.eclipse.net4j.internal.util.bundle.AbstractPlatform.trac e(OMTraceHandlerEvent)
you can be absolutely sure.

Cheers
/Eike


> Cheers,
>
> Cédric
>
>
>
> Eike Stepper wrote:
>
>
>> Hi Cédric,
>>
>> That sounds interesting!
>> Comments inline...
>>
>>
>> Cédric Brun schrieb:
>>
>>> Hi Eike,
>>>
>>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>>> measuring read/write throughput when I use CDO (RC2).
>>>
>>> I'm using an embeded-derby database, two different metamodels (complex
>>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>>> Features)
>>>
>>> The test is a "quick and dirty" one so please do not take these figures
>>> too seriously, I would just like to have your input about the "order of
>>> magnitude" I should expect or settings affecting seriously these
>>> performances. I tried both JVM and TCP acceptor but the performances are
>>> quite similar though JVM is a bit faster.
>>>
>>>
>> Looks like a sign that the NIO socket channels are efficient ,-)
>> You used localhost for TCP transport I guess?
>>
>>
>>> * First case : building a small model (10000 elements) with the simple
>>> MM, only adding instances in an empty database :
>>> commiting change on every add I get a 2 objects per second rate.
>>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>>
>>>
>> Seems to be consistent with the fact that only for commits there is
>> remote traffic.
>> And the backend transaction management is another overhead.
>> So the impact of the number objects/commit starts with bigger
>> transactions.
>>
>>
>>> browsing the just commited elements opening a new transaction and getting
>>> a new Resource instance, I get around 30 000 objets/sec rate just
>>> iterating thanks to Resource.getAllContents().
>>>
>>>
>> Hehe, this is a sign that client side caching works well:
>> All views and transactions of a single session share (and cache) their
>> object state.
>> Otherwise I'd expect much smaller values.
>> With his ObjectivityStore implementation Simon (cc'ed) reached rates of
>> several thousands objects/sec.
>> But the DBStore (which you used) is not that optimized.
>>
>>
>>> * Second case : I built a big model in the database using the complex
>>> metamodel. It contains 700 000 elements. The creation went pretty nicely
>>> (I did not measured it exactly as it was launched during a night, but it
>>> seems I got around 100 objects/sec rate in writting as I was writting
>>> every 700 objects.
>>>
>>>
>> Sounds not too bad.
>>
>>
>>> Once the model is built the derby database is 222 MB (which - I heard -
>>> is not so big for derby).
>>>
>>>
>> Hmm, I have no extensive experience with Derby.
>> To be honest, I'm really no database expert (although I can write simple
>> SQL statements).
>>
>>
>>> Retrieving the distant resource and calling eAllContents() on it, I get a
>>> 50 Objects/sec rate at best, which mean my eAllContents() call would take
>>> 4 hours !
>>>
>>>
>> Currently I'm not sure what exactly eAllContents() does do (iteration
>> order and so).
>> There are several CDO features that can have a major impact on load rates!
>>
>> - Collections (xrefs and containment) can be loaded lazily. I don't mean
>> the target objects, which are lazy by default anyway, but even the
>> target IDs that are in the collections. We call that "reference
>> chunking" (but search for a better name is ongoing).
>> |*public interface *CDOSession *extends *CDOProtocolSession,
>> |IContainer<CDOView>
>> {
>> *public **int *getReferenceChunkSize();
>>
>> *public **void *setReferenceChunkSize(*int *referenceChunkSize);|
>>
>>
>> - Loading collection elements (target objects) can be batched (the
>> current name is even worse)
>> |*public interface *CDOView *extends *CDOProtocolView, INotifier
>> {
>> *public **int *getLoadRevisionCollectionChunkSize();
>>
>> *public **void *setLoadRevisionCollectionChunkSize(*int
>> *loadRevisionCollectionChunkSize);|
>>
>>
>>
>> - One of the most sophisticated feature is automatic model usage
>> analyzation. The result are fetch rules that are sent to the server.
>> When later objects are loaded from the server, it can answer with
>> additional objects that are likely to be needed by the client in the
>> near future.
>> |*public class *CDOViewImpl *extends
>> |*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
>> |CDOIDProvider,
>> Adapter.Internal
>> {
>> *public *CDOFeatureAnalyzer getFeatureAnalyzer()
>> {
>> *return *featureAnalyzer;
>> }
>>
>> *public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
>> {
>> *this*.featureAnalyzer = featureAnalyzer == *null *?
>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>> }|
>>
>>
>> - There's a SmartReadAheadThread (if you're curious, please ask Simon).
>> - Caches can be configured
>> - ...
>>
>>
>> Known issues of the DBStore:
>> - When in auditing mode updates are always written as complete inserts
>> (by design)
>> - When configuring toManyReferences to a value different from
>> LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
>> - No (few) prepared statements are used (see
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
>> - Maybe others...
>>
>> Please note that the DBStore was not written by me to compete with
>> Hibernate in terms of performance or mapping capabilities.
>> Now that Martin and I, well in the end mostly Martin, provide a
>> HibernateStore and this HibernateStore gets more and more features I
>> expect that there will be a trend away from the DBStore.
>>
>>
>>> This is probably linked to the way the mapping to db is done, here's the
>>> config I used (came from CDO-server example if I recall correctly).
>>>
>>>
>>> <repository name="repo1">
>>> <property name="overrideUUID"
>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>> <property name="supportingAudits" value="true"/>
>>> <property name="verifyingRevisions" value="false"/>
>>> <property name="currentLRUCapacity" value="10000"/>
>>>
>>>
>> I guess this one could be larger.
>>
>>
>>> <property name="revisedLRUCapacity" value="100"/>
>>>
>>> <store type="db">
>>> <!-- type: horizontal | vertical | <any
>>> user-contributed type> --> <mappingStrategy
>>> type="horizontal">
>>> <!-- ONE_TABLE_PER_REFERENCE |
>>> ONE_TABLE_PER_CLASS |
>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>>> <property name="toManyReferences"
>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>
>>> <!-- LIKE_ATTRIBUTES |
>>> LIKE_TO_MANY_REFERENCES --> <property
>>> name="toOneReferences"
>>> value="LIKE_ATTRIBUTES"/>
>>>
>>> <!-- MODEL | STRATEGY -->
>>> <property name="mappingPrecedence"
>>> value="MODEL"/>
>>>
>>>
>> This one is obsolete.
>>
>>
>>> </mappingStrategy>
>>>
>>> <dbAdapter name="derby-embedded"/>
>>> <dataSource
>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>> databaseName="/tmp/cdodb1"
>>> createDatabase="create"/>
>>>
>>> </store>
>>> </repository>
>>>
>>>
>>>
>>>
>>> What do you think about that ? What throughput do you think I should
>>> expect ?
>>>
>> As I said that's very difficult to judge.
>> I believe it's more appropriate to compare the actual values with values
>> from other measures (other setups, properties, etc. or even other
>> products) instead of random expectations.
>> Apologies that I didn't have the time so far to provide some reference
>> measures myself. Maybe this gets better now that the team starts to grow
>> slowly ;-)
>>
>>
>>> What are the main reasons fur such figures ?
>>>
>>>
>> Hmm, I *suspect* you don't like the figures? I can find no evidence your
>> assessment ;-)
>>
>> Ahh,one more note:
>> Tracing can also have a significant impact on performance and end-to-end
>> throughput.
>> Turn it off by all means. And please make sure that it's really turned
>> off, not only invisible (i.e. console is not a target).
>>
>> Cheers
>> /Eike
>>
>
>

--------------030500070800010205060903
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Cédric Brun schrieb:
<blockquote cite="mid:g1m91c$i2q$1@build.eclipse.org" type="cite">
<pre wrap="">Eike,

thanks for your detailled answer, I'll digg further and try to profile the
Net4J/CDO stack so that I understand the "50 objects/sec rate" when
retrieving data (which I think is surprisingly low).
</pre>
</blockquote>
Looking forward to new insights!<br>
<br>
<blockquote cite="mid:g1m91c$i2q$1@build.eclipse.org" type="cite">
<pre wrap="">
Concerning the logging, what can I change to be *sure* that no logging will
occurs ?
</pre>
</blockquote>
In Net4j/CDO there's a basic difference between logging and tracing.<br>
Logging is for important production information (I always advise
against using the DEBUG loglevel!) and can't be switched off.<br>
<br>
Tracing can be configured in several dimensions, per bundle, per tracer
hierachy, per trace context (usually == classname).<br>
If you launch an OSGi Framework or Eclipse Application you can control
tracing via the Tracing page of the launch config.<br>
In stand-alone mode call OMPlatform.INSTANCE.setDebugging(false).<br>
<br>
Simply removing all ITraceHandlers is not enough.<br>
If you set a breakpoint in
org.eclipse.net4j.internal.util.bundle.AbstractPlatform.trac e(OMTraceHandlerEvent)
you can be absolutely sure.<br>
<br>
Cheers<br>
/Eike<br>
<br>
<br>
<blockquote cite="mid:g1m91c$i2q$1@build.eclipse.org" type="cite">
<pre wrap="">
Cheers,

Cédric



Eike Stepper wrote:

</pre>
<blockquote type="cite">
<pre wrap="">Hi Cédric,

That sounds interesting!
Comments inline...


Cédric Brun schrieb:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex
one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
Features)

The test is a "quick and dirty" one so please do not take these figures
too seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.

</pre>
</blockquote>
<pre wrap="">Looks like a sign that the NIO socket channels are efficient ,-)
You used localhost for TCP transport I guess?

</pre>
<blockquote type="cite">
<pre wrap="">* First case : building a small model (10000 elements) with the simple
MM, only adding instances in an empty database :
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.

</pre>
</blockquote>
<pre wrap="">Seems to be consistent with the fact that only for commits there is
remote traffic.
And the backend transaction management is another overhead.
So the impact of the number objects/commit starts with bigger
transactions.

</pre>
<blockquote type="cite">
<pre wrap="">browsing the just commited elements opening a new transaction and getting
a new Resource instance, I get around 30 000 objets/sec rate just
iterating thanks to Resource.getAllContents().

</pre>
</blockquote>
<pre wrap="">Hehe, this is a sign that client side caching works well:
All views and transactions of a single session share (and cache) their
object state.
Otherwise I'd expect much smaller values.
With his ObjectivityStore implementation Simon (cc'ed) reached rates of
several thousands objects/sec.
But the DBStore (which you used) is not that optimized.

</pre>
<blockquote type="cite">
<pre wrap="">* Second case : I built a big model in the database using the complex
metamodel. It contains 700 000 elements. The creation went pretty nicely
(I did not measured it exactly as it was launched during a night, but it
seems I got around 100 objects/sec rate in writting as I was writting
every 700 objects.

</pre>
</blockquote>
<pre wrap="">Sounds not too bad.

</pre>
<blockquote type="cite">
<pre wrap="">Once the model is built the derby database is 222 MB (which - I heard -
is not so big for derby).

</pre>
</blockquote>
<pre wrap="">Hmm, I have no extensive experience with Derby.
To be honest, I'm really no database expert (although I can write simple
SQL statements).

</pre>
<blockquote type="cite">
<pre wrap="">Retrieving the distant resource and calling eAllContents() on it, I get a
50 Objects/sec rate at best, which mean my eAllContents() call would take
4 hours !

</pre>
</blockquote>
<pre wrap="">Currently I'm not sure what exactly eAllContents() does do (iteration
order and so).
There are several CDO features that can have a major impact on load rates!

- Collections (xrefs and containment) can be loaded lazily. I don't mean
the target objects, which are lazy by default anyway, but even the
target IDs that are in the collections. We call that "reference
chunking" (but search for a better name is ongoing).
|*public interface *CDOSession *extends *CDOProtocolSession,
|IContainer&lt;CDOView&gt;
{
*public **int *getReferenceChunkSize();

*public **void *setReferenceChunkSize(*int *referenceChunkSize);|


- Loading collection elements (target objects) can be batched (the
current name is even worse)
|*public interface *CDOView *extends *CDOProtocolView, INotifier
{
*public **int *getLoadRevisionCollectionChunkSize();

*public **void *setLoadRevisionCollectionChunkSize(*int
*loadRevisionCollectionChunkSize);|



- One of the most sophisticated feature is automatic model usage
analyzation. The result are fetch rules that are sent to the server.
When later objects are loaded from the server, it can answer with
additional objects that are likely to be needed by the client in the
near future.
|*public class *CDOViewImpl *extends
|*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
|CDOIDProvider,
Adapter.Internal
{
*public *CDOFeatureAnalyzer getFeatureAnalyzer()
{
*return *featureAnalyzer;
}

*public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
{
*this*.featureAnalyzer = featureAnalyzer == *null *?
CDOFeatureAnalyzer.NOOP : featureAnalyzer;
}|


- There's a SmartReadAheadThread (if you're curious, please ask Simon).
- Caches can be configured
- ...


Known issues of the DBStore:
- When in auditing mode updates are always written as complete inserts
(by design)
- When configuring toManyReferences to a value different from
LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
- No (few) prepared statements are used (see
<a class="moz-txt-link-freetext" href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487">https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487</a>)
- Maybe others...

Please note that the DBStore was not written by me to compete with
Hibernate in terms of performance or mapping capabilities.
Now that Martin and I, well in the end mostly Martin, provide a
HibernateStore and this HibernateStore gets more and more features I
expect that there will be a trend away from the DBStore.

</pre>
<blockquote type="cite">
<pre wrap="">This is probably linked to the way the mapping to db is done, here's the
config I used (came from CDO-server example if I recall correctly).


&lt;repository name="repo1"&gt;
&lt;property name="overrideUUID"
value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/&gt;
&lt;property name="supportingAudits" value="true"/&gt;
&lt;property name="verifyingRevisions" value="false"/&gt;
&lt;property name="currentLRUCapacity" value="10000"/&gt;

</pre>
</blockquote>
<pre wrap="">I guess this one could be larger.

</pre>
<blockquote type="cite">
<pre wrap=""> &lt;property name="revisedLRUCapacity" value="100"/&gt;

&lt;store type="db"&gt;
&lt;!-- type: horizontal | vertical | &lt;any
user-contributed type&gt; --&gt; &lt;mappingStrategy
type="horizontal"&gt;
&lt;!-- ONE_TABLE_PER_REFERENCE |
ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES --&gt;
&lt;property name="toManyReferences"
value="ONE_TABLE_PER_REFERENCE"/&gt;

&lt;!-- LIKE_ATTRIBUTES |
LIKE_TO_MANY_REFERENCES --&gt; &lt;property
name="toOneReferences"
value="LIKE_ATTRIBUTES"/&gt;

&lt;!-- MODEL | STRATEGY --&gt;
&lt;property name="mappingPrecedence"
value="MODEL"/&gt;

</pre>
</blockquote>
<pre wrap="">This one is obsolete.

</pre>
<blockquote type="cite">
<pre wrap=""> &lt;/mappingStrategy&gt;

&lt;dbAdapter name="derby-embedded"/&gt;
&lt;dataSource
class="org.apache.derby.jdbc.EmbeddedDataSource"
databaseName="/tmp/cdodb1"
createDatabase="create"/&gt;

&lt;/store&gt;
&lt;/repository&gt;




What do you think about that ? What throughput do you think I should
expect ?
</pre>
</blockquote>
<pre wrap="">As I said that's very difficult to judge.
I believe it's more appropriate to compare the actual values with values
from other measures (other setups, properties, etc. or even other
products) instead of random expectations.
Apologies that I didn't have the time so far to provide some reference
measures myself. Maybe this gets better now that the team starts to grow
slowly ;-)

</pre>
<blockquote type="cite">
<pre wrap="">What are the main reasons fur such figures ?

</pre>
</blockquote>
<pre wrap="">Hmm, I *suspect* you don't like the figures? I can find no evidence your
assessment ;-)

Ahh,one more note:
Tracing can also have a significant impact on performance and end-to-end
throughput.
Turn it off by all means. And please make sure that it's really turned
off, not only invisible (i.e. console is not a target).

Cheers
/Eike
</pre>
</blockquote>
<pre wrap=""><!---->
</pre>
</blockquote>
</body>
</html>

--------------030500070800010205060903--
Re: [CDO] reading/writting throughput performances [message #123859 is a reply to message #123770] Thu, 29 May 2008 19:00 Go to previous messageGo to next message
Eclipse UserFriend
Originally posted by: stepper.sympedia.de

Eike Stepper schrieb:
> Mark,
>
> Thank you to encourage me ;-)
> RC3 will have the change.
Please cc yourself if you want:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=234702

>
> Cheers
> /Eike
>
>
>
> Mark Geib schrieb:
>> I would vote to make the change. You will be sorry if you miss this
>> opportunity. The users can handle the change I think.
>>
>> Just my thoughts,
>> Mark.
>>
>>
>>
>> Eike Stepper wrote:
>>> Simon McDuff schrieb:
>>>> Good point Eike,
>>>>
>>>> Cedric,
>>>>
>>>> How do you create your CDOSessionIMpl ?
>>>> Are you using your own mechanism ?
>>>>
>>>> Because it could still occured in the case we doesn`t go through
>>>> CDOSessionConfigurationImpl and create directly the CDOSession
>>> You could be right, too. Basically I hate myself because once I
>>> advocated for default legacy support!!! You remember these
>>> discussions? Since several months now I'd really like to change that
>>> (and the checkbox in the OpenSessionDialog is early evidence). I
>>> didn't dare to do it because formerly disableLegacyObjects was a
>>> ctor arg and a semantic change would have broken all clients without
>>> compiler warning. Now that there are no ctor args anymore IO really
>>> consider making this change a late one for Ganymede. Otherwise we'd
>>> have to carry this odd issue for another year and maybe maintain two
>>> different versions of the semantic. What do you think?
>>>
>>>>
>>>> Ex:
>>>>
>>>> CDOSession session = CDOSessionImpl();
>>>>
>>>> By default disablyLegacySystem is false.
>>>>
>>>> In this case Eike, Do we have something else to avoid that ? Maybe
>>>> I`m not aware of other mechanism.
>>> I can add the same check to doBeforeActivate(), too.
>>>
>>> Cheers
>>> /Eike
>>>
>>>>
>>>> Simon
>>>>
>>>> "Eike Stepper" <stepper@sympedia.de <mailto:stepper@sympedia.de>> a
>>>> écrit dans le message de news: 483ED9A4.7010302@sympedia.de
>>>> <mailto:483ED9A4.7010302@sympedia.de>...
>>>>
>>>>
>>>>
>>>> Simon McDuff schrieb:
>>>>> Did you disable the following :
>>>>> CDOSessionConfigurationImpl.setLegacySupportEnabled(true);
>>>>>
>>>>> before opening your session ?
>>>>>
>>>>> This could cause what you are observing ?
>>>>>
>>>>> If your classes are generating with CDOObjectImpl as the super
>>>>> class, you can disableLegacySupport by putting it at true.
>>>>>
>>>> Good point in general!
>>>>
>>>> But I added some logic some time ago that ensures that *session
>>>> legacy support* can only be enabled if the *legacy system* is
>>>> available in general:
>>>>
>>>> | *public **void *setDisableLegacyObjects(*boolean
>>>> *disableLegacyObjects)
>>>> {
>>>> checkInactive();
>>>> *if *(!disableLegacyObjects &&
>>>> !FSMUtil.isLegacySystemAvailable())
>>>> {
>>>> *throw new *LegacySystemNotAvailableException();
>>>> }
>>>>
>>>> *this*.disableLegacyObjects = disableLegacyObjects;
>>>> }|
>>>>
>>>>
>>>> Notice that the whole legacy system is currently (since the work
>>>> on the HibernateStore) broken.
>>>> The needed CDOWeaver is currently not part of the distro anymore
>>>> (also due to issues with the build system).
>>>>
>>>> This altogether seems to exclude the legacy system as root cause
>>>> for performance degradation.
>>>>
>>>> Cheers
>>>> /Eike
>>>>
>>>>
>>>>
>>>>> Simon
>>>>>
>>>>>
>>>>> "Cédric Brun" <cedric.brun@obeo.fr> a écrit dans le message de
>>>>> news: g1mbhe$2cl$1@build.eclipse..org...
>>>>>
>>>>>> Eike,
>>>>>>
>>>>>> As I don't know (yet ;) ) all the internals of CDO I'm not
>>>>>> sure this is
>>>>>> relevant, but in the testcase I'm working on (with the
>>>>>> complex metamodel,
>>>>>> big model in database and 50 o/sec throughput) my profiler
>>>>>> says that most
>>>>>> of the time (like 99.9%) is spent in a readString call from
>>>>>> CDOClassRefImpl :
>>>>>>
>>>>>> public CDOClassRefImpl(ExtendedDataInput in, String
>>>>>> defaultURI) throws
>>>>>> IOException
>>>>>> {
>>>>>> // TODO Optimize transfer of URIs
>>>>>> packageURI = in.readString();
>>>>>> if (packageURI == null)
>>>>>> {
>>>>>> packageURI = defaultURI;
>>>>>> }
>>>>>>
>>>>>> classifierID = in.readInt();
>>>>>> }
>>>>>>
>>>>>> The TODO let me think this seems relevant but I'm not sure,
>>>>>> what do you
>>>>>> think of it ?
>>>>>>
>>>>>> Cédric
>>>>>>
>>>>>>
>>>>>> Eike Stepper wrote:
>>>>>>
>>>>>>
>>>>>>> Hi Cédric,
>>>>>>>
>>>>>>> That sounds interesting!
>>>>>>> Comments inline...
>>>>>>>
>>>>>>>
>>>>>>> Cédric Brun schrieb:
>>>>>>>
>>>>>>>> Hi Eike,
>>>>>>>>
>>>>>>>> I've been messing around CDO lately and I hacked a few ugly
>>>>>>>> unit-tests
>>>>>>>> measuring read/write throughput when I use CDO (RC2).
>>>>>>>>
>>>>>>>> I'm using an embeded-derby database, two different
>>>>>>>> metamodels (complex
>>>>>>>> one - 250 EClassifiers/300 features and a simple one - 1
>>>>>>>> Classifier/3
>>>>>>>> Features)
>>>>>>>>
>>>>>>>> The test is a "quick and dirty" one so please do not take
>>>>>>>> these figures
>>>>>>>> too seriously, I would just like to have your input about
>>>>>>>> the "order of
>>>>>>>> magnitude" I should expect or settings affecting seriously
>>>>>>>> these
>>>>>>>> performances. I tried both JVM and TCP acceptor but the
>>>>>>>> performances are
>>>>>>>> quite similar though JVM is a bit faster.
>>>>>>>>
>>>>>>>>
>>>>>>> Looks like a sign that the NIO socket channels are efficient
>>>>>>> ,-)
>>>>>>> You used localhost for TCP transport I guess?
>>>>>>>
>>>>>>>
>>>>>>>> * First case : building a small model (10000 elements) with
>>>>>>>> the simple
>>>>>>>> MM, only adding instances in an empty database :
>>>>>>>> commiting change on every add I get a 2 objects per second
>>>>>>>> rate.
>>>>>>>> commiting changes every 10 adds - I get a 30 objects per
>>>>>>>> second rate.
>>>>>>>>
>>>>>>>>
>>>>>>> Seems to be consistent with the fact that only for commits
>>>>>>> there is
>>>>>>> remote traffic.
>>>>>>> And the backend transaction management is another overhead.
>>>>>>> So the impact of the number objects/commit starts with bigger
>>>>>>> transactions.
>>>>>>>
>>>>>>>
>>>>>>>> browsing the just commited elements opening a new
>>>>>>>> transaction and getting
>>>>>>>> a new Resource instance, I get around 30 000 objets/sec
>>>>>>>> rate just
>>>>>>>> iterating thanks to Resource.getAllContents().
>>>>>>>>
>>>>>>>>
>>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>>> All views and transactions of a single session share (and
>>>>>>> cache) their
>>>>>>> object state.
>>>>>>> Otherwise I'd expect much smaller values.
>>>>>>> With his ObjectivityStore implementation Simon (cc'ed)
>>>>>>> reached rates of
>>>>>>> several thousands objects/sec.
>>>>>>> But the DBStore (which you used) is not that optimized.
>>>>>>>
>>>>>>>
>>>>>>>> * Second case : I built a big model in the database using
>>>>>>>> the complex
>>>>>>>> metamodel. It contains 700 000 elements. The creation went
>>>>>>>> pretty nicely
>>>>>>>> (I did not measured it exactly as it was launched during a
>>>>>>>> night, but it
>>>>>>>> seems I got around 100 objects/sec rate in writting as I
>>>>>>>> was writting
>>>>>>>> every 700 objects.
>>>>>>>>
>>>>>>>>
>>>>>>> Sounds not too bad.
>>>>>>>
>>>>>>>
>>>>>>>> Once the model is built the derby database is 222 MB (which
>>>>>>>> - I heard -
>>>>>>>> is not so big for derby).
>>>>>>>>
>>>>>>>>
>>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>>> To be honest, I'm really no database expert (although I can
>>>>>>> write simple
>>>>>>> SQL statements).
>>>>>>>
>>>>>>>
>>>>>>>> Retrieving the distant resource and calling eAllContents()
>>>>>>>> on it, I get a
>>>>>>>> 50 Objects/sec rate at best, which mean my eAllContents()
>>>>>>>> call would take
>>>>>>>> 4 hours !
>>>>>>>>
>>>>>>>>
>>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>>> (iteration
>>>>>>> order and so).
>>>>>>> There are several CDO features that can have a major impact
>>>>>>> on load rates!
>>>>>>>
>>>>>>> - Collections (xrefs and containment) can be loaded lazily.
>>>>>>> I don't mean
>>>>>>> the target objects, which are lazy by default anyway, but
>>>>>>> even the
>>>>>>> target IDs that are in the collections. We call that "reference
>>>>>>> chunking" (but search for a better name is ongoing).
>>>>>>> |*public interface *CDOSession *extends *CDOProtocolSession,
>>>>>>> |IContainer<CDOView>
>>>>>>> {
>>>>>>> *public **int *getReferenceChunkSize();
>>>>>>>
>>>>>>> *public **void *setReferenceChunkSize(*int
>>>>>>> *referenceChunkSize);|
>>>>>>>
>>>>>>>
>>>>>>> - Loading collection elements (target objects) can be
>>>>>>> batched (the
>>>>>>> current name is even worse)
>>>>>>> |*public interface *CDOView *extends *CDOProtocolView,
>>>>>>> INotifier
>>>>>>> {
>>>>>>> *public **int *getLoadRevisionCollectionChunkSize();
>>>>>>>
>>>>>>> *public **void *setLoadRevisionCollectionChunkSize(*int
>>>>>>> *loadRevisionCollectionChunkSize);|
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> - One of the most sophisticated feature is automatic model
>>>>>>> usage
>>>>>>> analyzation. The result are fetch rules that are sent to the
>>>>>>> server.
>>>>>>> When later objects are loaded from the server, it can answer
>>>>>>> with
>>>>>>> additional objects that are likely to be needed by the
>>>>>>> client in the
>>>>>>> near future.
>>>>>>> |*public class *CDOViewImpl *extends
>>>>>>> |*org.eclipse.net4j.internal.util.event.Notifier *implements
>>>>>>> *CDOView,
>>>>>>> |CDOIDProvider,
>>>>>>> Adapter.Internal
>>>>>>> {
>>>>>>> *public *CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>>>> {
>>>>>>> *return *featureAnalyzer;
>>>>>>> }
>>>>>>>
>>>>>>> *public **void *setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>>>> featureAnalyzer)
>>>>>>> {
>>>>>>> *this*.featureAnalyzer = featureAnalyzer == *null *?
>>>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>>>> }|
>>>>>>>
>>>>>>>
>>>>>>> - There's a SmartReadAheadThread (if you're curious, please
>>>>>>> ask Simon).
>>>>>>> - Caches can be configured
>>>>>>> - ...
>>>>>>>
>>>>>>>
>>>>>>> Known issues of the DBStore:
>>>>>>> - When in auditing mode updates are always written as
>>>>>>> complete inserts
>>>>>>> (by design)
>>>>>>> - When configuring toManyReferences to a value different from
>>>>>>> LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
>>>>>>> - No (few) prepared statements are used (see
>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
>>>>>>> - Maybe others...
>>>>>>>
>>>>>>> Please note that the DBStore was not written by me to
>>>>>>> compete with
>>>>>>> Hibernate in terms of performance or mapping capabilities.
>>>>>>> Now that Martin and I, well in the end mostly Martin, provide a
>>>>>>> HibernateStore and this HibernateStore gets more and more
>>>>>>> features I
>>>>>>> expect that there will be a trend away from the DBStore.
>>>>>>>
>>>>>>>
>>>>>>>> This is probably linked to the way the mapping to db is
>>>>>>>> done, here's the
>>>>>>>> config I used (came from CDO-server example if I recall
>>>>>>>> correctly).
>>>>>>>>
>>>>>>>>
>>>>>>>> <repository name="repo1">
>>>>>>>> <property name="overrideUUID"
>>>>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>>>>> <property name="supportingAudits"
>>>>>>>> value="true"/>
>>>>>>>> <property name="verifyingRevisions"
>>>>>>>> value="false"/>
>>>>>>>> <property name="currentLRUCapacity"
>>>>>>>> value="10000"/>
>>>>>>>>
>>>>>>>>
>>>>>>> I guess this one could be larger.
>>>>>>>
>>>>>>>
>>>>>>>> <property name="revisedLRUCapacity"
>>>>>>>> value="100"/>
>>>>>>>>
>>>>>>>> <store type="db">
>>>>>>>> <!-- type: horizontal | vertical |
>>>>>>>> <any
>>>>>>>> user-contributed type> -->
>>>>>>>> <mappingStrategy
>>>>>>>> type="horizontal">
>>>>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>>>>> ONE_TABLE_PER_CLASS |
>>>>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY |
>>>>>>>> LIKE_ATTRIBUTES -->
>>>>>>>> <property
>>>>>>>> name="toManyReferences"
>>>>>>>>
>>>>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>>>>
>>>>>>>> <!-- LIKE_ATTRIBUTES |
>>>>>>>> LIKE_TO_MANY_REFERENCES -->
>>>>>>>> <property
>>>>>>>> name="toOneReferences"
>>>>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>>>>
>>>>>>>> <!-- MODEL | STRATEGY -->
>>>>>>>> <property
>>>>>>>> name="mappingPrecedence"
>>>>>>>> value="MODEL"/>
>>>>>>>>
>>>>>>>>
>>>>>>> This one is obsolete.
>>>>>>>
>>>>>>>
>>>>>>>> </mappingStrategy>
>>>>>>>>
>>>>>>>> <dbAdapter name="derby-embedded"/>
>>>>>>>> <dataSource
>>>>>>>>
>>>>>>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>>>>>> databaseName="/tmp/cdodb1"
>>>>>>>> createDatabase="create"/>
>>>>>>>>
>>>>>>>> </store>
>>>>>>>> </repository>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> What do you think about that ? What throughput do you think
>>>>>>>> I should
>>>>>>>> expect ?
>>>>>>>>
>>>>>>> As I said that's very difficult to judge.
>>>>>>> I believe it's more appropriate to compare the actual values
>>>>>>> with values
>>>>>>> from other measures (other setups, properties, etc. or even
>>>>>>> other
>>>>>>> products) instead of random expectations.
>>>>>>> Apologies that I didn't have the time so far to provide some
>>>>>>> reference
>>>>>>> measures myself. Maybe this gets better now that the team
>>>>>>> starts to grow
>>>>>>> slowly ;-)
>>>>>>>
>>>>>>>
>>>>>>>> What are the main reasons fur such figures ?
>>>>>>>>
>>>>>>>>
>>>>>>> Hmm, I *suspect* you don't like the figures? I can find no
>>>>>>> evidence your
>>>>>>> assessment ;-)
>>>>>>>
>>>>>>> Ahh,one more note:
>>>>>>> Tracing can also have a significant impact on performance
>>>>>>> and end-to-end
>>>>>>> throughput.
>>>>>>> Turn it off by all means. And please make sure that it's
>>>>>>> really turned
>>>>>>> off, not only invisible (i.e. console is not a target).
>>>>>>>
>>>>>>> Cheers
>>>>>>> /Eike
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>
Re: [CDO] reading/writting throughput performances [message #124080 is a reply to message #123651] Fri, 30 May 2008 12:43 Go to previous messageGo to next message
Cedric Brun is currently offline Cedric BrunFriend
Messages: 431
Registered: July 2009
Senior Member
Hi Simon and Eike,

thanks for your feedback, now I know what I could expect in using optimized
implementation, I'll have a try on the hibernate store and see if it is
scaling in a better way.

You gave me many pointers to explain the performances issues but in fact it
seems that the SQL execution is taking so much time (not suprisingly in
fact ;) . My first profile was wrong.

So I'm installing the hibernate store retrieving all the source code from
CVS, and I'm trying to launch the unit tests. I added many jars and
everything gets compiled but most of the tests are failling with "

"
org.hibernate.MappingException: Could not determine type for:
org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType, at
table: cdo_features, for columns: [org.hibernate.mapping.Column(type)]
"

any hint ?

Cédric


Simon McDuff wrote:

> We noticed a performance degradation in the following case:
>
> When we fetch a collection of objects that contains collections that we
> don't want to use. Ex:
>
> Model:
>
> ClassB
> {
> int foo1;
> List childrens1;
> List childrens2;
> List childrens3;
> };
>
> Code:
>
> Resource.contents contains objects of type ClassB.
>
> By fetching instances of classB, we will also fetched some items in the
> list(childrens1, childrens2, childrens3) The problem we had.. is we do not
> know if we want to use it. SO better not to download them.
>
> So In our production(we used our own IStore), we never download any
> objects from collection. I'm working at the moment to put in place all
> feature we used in the current CDO. That one will be part of
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477. To begin
> with,CDORevisionImpl needs to supports list that are 0 even if
> referenceChunkSize == 0.
>
> It is true that we obtain good performance.
> Worst case :
> 1500 objects /sec (Real fetch from disk. Page that contains the objects,
> wasn't in the cache of the filesystem.)
>
> Best case
> 12 000 objects/Sec (Fetch objects that are already in the case of the
> filesystem)
>
> (These statistics always fetch objects from server to client. The cache
> client isn't used)
>
> We have another optimization one with SmartReadAhead but for now only work
> for UI client. (for now) This is something we will integrate as well in
> CDO. But for now we concentrate on the other things.
>
> I used an Object database. Based on my preliminary test without CDO.
> I compare
>
> Hibernate vs Objectivity/EMF :
> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>
> The worst.. you will get 30 to 240 objects/sec or
> the best... you will get 500 to 4000 objects/sec
>
> Based on your comments... you are getting the worst (with Derby)!! :-)
>
>
> Another comments about insertion.
>
> We found another problem when we insert data(DOn't know if it is still
> true). Resource.Contents is a unique list. Basically each time we insert
> one element, it needs to fetch all the collection to see if we already
> have this element. Basically.. it is time consuming. In our case we are
> not going through this stage to persist our objects... but the objects are
> not accessible through that list.
>
>
>
> If speed is an issue for you!! We (You included :-)) can optimize it. What
> I like is Eike did a really good job doing that framwork and optimizations
> are easy to add. Also you can change your back-end without affecting you
> client side. (Maybe going to an Object database!!) THis was one of the
> purpose of CDO!! :-)
>
>
> Simon
>
>
> "Eike Stepper" <stepper@sympedia.de> a écrit dans le message de news:
> 483E9332.6050800@sympedia.de... Hi Cédric,
>
> That sounds interesting!
> Comments inline...
>
>
> Cédric Brun schrieb:
> Hi Eike,
>
> I've been messing around CDO lately and I hacked a few ugly unit-tests
> measuring read/write throughput when I use CDO (RC2).
>
> I'm using an embeded-derby database, two different metamodels (complex one
> - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
> Features)
>
> The test is a "quick and dirty" one so please do not take these figures
> too seriously, I would just like to have your input about the "order of
> magnitude" I should expect or settings affecting seriously these
> performances. I tried both JVM and TCP acceptor but the performances are
> quite similar though JVM is a bit faster.
> Looks like a sign that the NIO socket channels are efficient ,-)
> You used localhost for TCP transport I guess?
>
>
>
> * First case : building a small model (10000 elements) with the simple MM,
> only adding instances in an empty database :
> commiting change on every add I get a 2 objects per second rate.
> commiting changes every 10 adds - I get a 30 objects per second rate.
> Seems to be consistent with the fact that only for commits there is
> remote traffic. And the backend transaction management is another
> overhead. So the impact of the number objects/commit starts with bigger
> transactions.
>
>
> browsing the just commited elements opening a new transaction and getting
> a new Resource instance, I get around 30 000 objets/sec rate just
> iterating thanks to Resource.getAllContents().
> Hehe, this is a sign that client side caching works well:
> All views and transactions of a single session share (and cache) their
> object state. Otherwise I'd expect much smaller values.
> With his ObjectivityStore implementation Simon (cc'ed) reached rates of
> several thousands objects/sec. But the DBStore (which you used) is not
> that optimized.
>
>
> * Second case : I built a big model in the database using the complex
> metamodel. It contains 700 000 elements. The creation went pretty nicely
> (I did not measured it exactly as it was launched during a night, but it
> seems I got around 100 objects/sec rate in writting as I was writting
> every 700 objects.
> Sounds not too bad.
>
>
>
> Once the model is built the derby database is 222 MB (which - I heard - is
> not so big for derby).
> Hmm, I have no extensive experience with Derby.
> To be honest, I'm really no database expert (although I can write simple
> SQL statements).
>
>
> Retrieving the distant resource and calling eAllContents() on it, I get a
> 50 Objects/sec rate at best, which mean my eAllContents() call would take
> 4 hours !
> Currently I'm not sure what exactly eAllContents() does do (iteration
> order and so). There are several CDO features that can have a major
> impact on load rates!
>
> - Collections (xrefs and containment) can be loaded lazily. I don't mean
> the target objects, which are lazy by default anyway, but even the
> target IDs that are in the collections. We call that "reference
> chunking" (but search for a better name is ongoing).
>
> public interface CDOSession extends CDOProtocolSession,
> IContainer<CDOView>
> {
> public int getReferenceChunkSize();
>
> public void setReferenceChunkSize(int referenceChunkSize);
>
>
> - Loading collection elements (target objects) can be batched (the
> current name is even worse)
>
> public interface CDOView extends CDOProtocolView, INotifier
> {
> public int getLoadRevisionCollectionChunkSize();
>
> public void setLoadRevisionCollectionChunkSize(int
> loadRevisionCollectionChunkSize);
>
>
> - One of the most sophisticated feature is automatic model usage
> analyzation. The result are fetch rules that are sent to the server.
> When later objects are loaded from the server, it can answer with
> additional objects that are likely to be needed by the client in the
> near future.
>
> public class CDOViewImpl extends
> org.eclipse.net4j.internal.util.event.Notifier implements CDOView,
> CDOIDProvider, Adapter.Internal
> {
> public CDOFeatureAnalyzer getFeatureAnalyzer()
> {
> return featureAnalyzer;
> }
>
> public void setFeatureAnalyzer(CDOFeatureAnalyzer
> featureAnalyzer)
> {
> this.featureAnalyzer = featureAnalyzer == null ?
> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
> }
>
>
> - There's a SmartReadAheadThread (if you're curious, please ask Simon).
> - Caches can be configured
> - ...
>
>
> Known issues of the DBStore:
> - When in auditing mode updates are always written as complete inserts
> (by design) - When configuring toManyReferences to a value different
> from LIKE_ATTRIBUTES (which is not implemented yet) it is very slow. -
> No (few) prepared statements are used (see
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe others...
>
> Please note that the DBStore was not written by me to compete with
> Hibernate in terms of performance or mapping capabilities. Now that
> Martin and I, well in the end mostly Martin, provide a HibernateStore
> and this HibernateStore gets more and more features I expect that there
> will be a trend away from the DBStore.
>
>
> This is probably linked to the way the mapping to db is done, here's the
> config I used (came from CDO-server example if I recall correctly).
>
>
> <repository name="repo1">
> <property name="overrideUUID"
> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
> <property name="supportingAudits" value="true"/>
> <property name="verifyingRevisions" value="false"/>
> <property name="currentLRUCapacity" value="10000"/>
> I guess this one could be larger.
>
>
> <property name="revisedLRUCapacity" value="100"/>
>
> <store type="db">
> <!-- type: horizontal | vertical | <any
> user-contributed type> --> <mappingStrategy
> type="horizontal">
> <!-- ONE_TABLE_PER_REFERENCE |
> ONE_TABLE_PER_CLASS |
> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
> <property name="toManyReferences"
> value="ONE_TABLE_PER_REFERENCE"/>
>
> <!-- LIKE_ATTRIBUTES |
> LIKE_TO_MANY_REFERENCES --> <property
> name="toOneReferences"
> value="LIKE_ATTRIBUTES"/>
>
> <!-- MODEL | STRATEGY -->
> <property name="mappingPrecedence"
> value="MODEL"/>
> This one is obsolete.
>
>
> </mappingStrategy>
>
> <dbAdapter name="derby-embedded"/>
> <dataSource
> class="org.apache.derby.jdbc.EmbeddedDataSource"
> databaseName="/tmp/cdodb1"
> createDatabase="create"/>
>
> </store>
> </repository>
>
>
>
>
> What do you think about that ? What throughput do you think I should
> expect ? As I said that's very difficult to judge.
> I believe it's more appropriate to compare the actual values with values
> from other measures (other setups, properties, etc. or even other
> products) instead of random expectations. Apologies that I didn't have
> the time so far to provide some reference measures myself. Maybe this
> gets better now that the team starts to grow slowly ;-)
>
>
> What are the main reasons fur such figures ?
> Hmm, I *suspect* you don't like the figures? I can find no evidence your
> assessment ;-)
>
> Ahh,one more note:
> Tracing can also have a significant impact on performance and end-to-end
> throughput. Turn it off by all means. And please make sure that it's
> really turned off, not only invisible (i.e. console is not a target).
>
> Cheers
> /Eike


http://cedric.brun.io news and articles on eclipse and eclipse modeling.
Re: [CDO] reading/writting throughput performances [message #124092 is a reply to message #124080] Fri, 30 May 2008 12:39 Go to previous messageGo to next message
Martin Taal is currently offline Martin TaalFriend
Messages: 5468
Registered: July 2009
Senior Member
Hi Cédric,
Most of the time this exception is thrown because hibernate can't find the class. A common cause is
that the hibernate jar files are located in a separate plugin which can not reach the classes in
other plugins. If you set the eclipse buddy policy of the hibernate plugin to dependent then this
can help.

gr. Martin

Cédric Brun wrote:
> Hi Simon and Eike,
>
> thanks for your feedback, now I know what I could expect in using optimized
> implementation, I'll have a try on the hibernate store and see if it is
> scaling in a better way.
>
> You gave me many pointers to explain the performances issues but in fact it
> seems that the SQL execution is taking so much time (not suprisingly in
> fact ;) . My first profile was wrong.
>
> So I'm installing the hibernate store retrieving all the source code from
> CVS, and I'm trying to launch the unit tests. I added many jars and
> everything gets compiled but most of the tests are failling with "
>
> "
> org.hibernate.MappingException: Could not determine type for:
> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType, at
> table: cdo_features, for columns: [org.hibernate.mapping.Column(type)]
> "
>
> any hint ?
>
> Cédric
>
>
> Simon McDuff wrote:
>
>> We noticed a performance degradation in the following case:
>>
>> When we fetch a collection of objects that contains collections that we
>> don't want to use. Ex:
>>
>> Model:
>>
>> ClassB
>> {
>> int foo1;
>> List childrens1;
>> List childrens2;
>> List childrens3;
>> };
>>
>> Code:
>>
>> Resource.contents contains objects of type ClassB.
>>
>> By fetching instances of classB, we will also fetched some items in the
>> list(childrens1, childrens2, childrens3) The problem we had.. is we do not
>> know if we want to use it. SO better not to download them.
>>
>> So In our production(we used our own IStore), we never download any
>> objects from collection. I'm working at the moment to put in place all
>> feature we used in the current CDO. That one will be part of
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477. To begin
>> with,CDORevisionImpl needs to supports list that are 0 even if
>> referenceChunkSize == 0.
>>
>> It is true that we obtain good performance.
>> Worst case :
>> 1500 objects /sec (Real fetch from disk. Page that contains the objects,
>> wasn't in the cache of the filesystem.)
>>
>> Best case
>> 12 000 objects/Sec (Fetch objects that are already in the case of the
>> filesystem)
>>
>> (These statistics always fetch objects from server to client. The cache
>> client isn't used)
>>
>> We have another optimization one with SmartReadAhead but for now only work
>> for UI client. (for now) This is something we will integrate as well in
>> CDO. But for now we concentrate on the other things.
>>
>> I used an Object database. Based on my preliminary test without CDO.
>> I compare
>>
>> Hibernate vs Objectivity/EMF :
>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>
>> The worst.. you will get 30 to 240 objects/sec or
>> the best... you will get 500 to 4000 objects/sec
>>
>> Based on your comments... you are getting the worst (with Derby)!! :-)
>>
>>
>> Another comments about insertion.
>>
>> We found another problem when we insert data(DOn't know if it is still
>> true). Resource.Contents is a unique list. Basically each time we insert
>> one element, it needs to fetch all the collection to see if we already
>> have this element. Basically.. it is time consuming. In our case we are
>> not going through this stage to persist our objects... but the objects are
>> not accessible through that list.
>>
>>
>>
>> If speed is an issue for you!! We (You included :-)) can optimize it. What
>> I like is Eike did a really good job doing that framwork and optimizations
>> are easy to add. Also you can change your back-end without affecting you
>> client side. (Maybe going to an Object database!!) THis was one of the
>> purpose of CDO!! :-)
>>
>>
>> Simon
>>
>>
>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le message de news:
>> 483E9332.6050800@sympedia.de... Hi Cédric,
>>
>> That sounds interesting!
>> Comments inline...
>>
>>
>> Cédric Brun schrieb:
>> Hi Eike,
>>
>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>> measuring read/write throughput when I use CDO (RC2).
>>
>> I'm using an embeded-derby database, two different metamodels (complex one
>> - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>> Features)
>>
>> The test is a "quick and dirty" one so please do not take these figures
>> too seriously, I would just like to have your input about the "order of
>> magnitude" I should expect or settings affecting seriously these
>> performances. I tried both JVM and TCP acceptor but the performances are
>> quite similar though JVM is a bit faster.
>> Looks like a sign that the NIO socket channels are efficient ,-)
>> You used localhost for TCP transport I guess?
>>
>>
>>
>> * First case : building a small model (10000 elements) with the simple MM,
>> only adding instances in an empty database :
>> commiting change on every add I get a 2 objects per second rate.
>> commiting changes every 10 adds - I get a 30 objects per second rate.
>> Seems to be consistent with the fact that only for commits there is
>> remote traffic. And the backend transaction management is another
>> overhead. So the impact of the number objects/commit starts with bigger
>> transactions.
>>
>>
>> browsing the just commited elements opening a new transaction and getting
>> a new Resource instance, I get around 30 000 objets/sec rate just
>> iterating thanks to Resource.getAllContents().
>> Hehe, this is a sign that client side caching works well:
>> All views and transactions of a single session share (and cache) their
>> object state. Otherwise I'd expect much smaller values.
>> With his ObjectivityStore implementation Simon (cc'ed) reached rates of
>> several thousands objects/sec. But the DBStore (which you used) is not
>> that optimized.
>>
>>
>> * Second case : I built a big model in the database using the complex
>> metamodel. It contains 700 000 elements. The creation went pretty nicely
>> (I did not measured it exactly as it was launched during a night, but it
>> seems I got around 100 objects/sec rate in writting as I was writting
>> every 700 objects.
>> Sounds not too bad.
>>
>>
>>
>> Once the model is built the derby database is 222 MB (which - I heard - is
>> not so big for derby).
>> Hmm, I have no extensive experience with Derby.
>> To be honest, I'm really no database expert (although I can write simple
>> SQL statements).
>>
>>
>> Retrieving the distant resource and calling eAllContents() on it, I get a
>> 50 Objects/sec rate at best, which mean my eAllContents() call would take
>> 4 hours !
>> Currently I'm not sure what exactly eAllContents() does do (iteration
>> order and so). There are several CDO features that can have a major
>> impact on load rates!
>>
>> - Collections (xrefs and containment) can be loaded lazily. I don't mean
>> the target objects, which are lazy by default anyway, but even the
>> target IDs that are in the collections. We call that "reference
>> chunking" (but search for a better name is ongoing).
>>
>> public interface CDOSession extends CDOProtocolSession,
>> IContainer<CDOView>
>> {
>> public int getReferenceChunkSize();
>>
>> public void setReferenceChunkSize(int referenceChunkSize);
>>
>>
>> - Loading collection elements (target objects) can be batched (the
>> current name is even worse)
>>
>> public interface CDOView extends CDOProtocolView, INotifier
>> {
>> public int getLoadRevisionCollectionChunkSize();
>>
>> public void setLoadRevisionCollectionChunkSize(int
>> loadRevisionCollectionChunkSize);
>>
>>
>> - One of the most sophisticated feature is automatic model usage
>> analyzation. The result are fetch rules that are sent to the server.
>> When later objects are loaded from the server, it can answer with
>> additional objects that are likely to be needed by the client in the
>> near future.
>>
>> public class CDOViewImpl extends
>> org.eclipse.net4j.internal.util.event.Notifier implements CDOView,
>> CDOIDProvider, Adapter.Internal
>> {
>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>> {
>> return featureAnalyzer;
>> }
>>
>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>> featureAnalyzer)
>> {
>> this.featureAnalyzer = featureAnalyzer == null ?
>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>> }
>>
>>
>> - There's a SmartReadAheadThread (if you're curious, please ask Simon).
>> - Caches can be configured
>> - ...
>>
>>
>> Known issues of the DBStore:
>> - When in auditing mode updates are always written as complete inserts
>> (by design) - When configuring toManyReferences to a value different
>> from LIKE_ATTRIBUTES (which is not implemented yet) it is very slow. -
>> No (few) prepared statements are used (see
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe others...
>>
>> Please note that the DBStore was not written by me to compete with
>> Hibernate in terms of performance or mapping capabilities. Now that
>> Martin and I, well in the end mostly Martin, provide a HibernateStore
>> and this HibernateStore gets more and more features I expect that there
>> will be a trend away from the DBStore.
>>
>>
>> This is probably linked to the way the mapping to db is done, here's the
>> config I used (came from CDO-server example if I recall correctly).
>>
>>
>> <repository name="repo1">
>> <property name="overrideUUID"
>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>> <property name="supportingAudits" value="true"/>
>> <property name="verifyingRevisions" value="false"/>
>> <property name="currentLRUCapacity" value="10000"/>
>> I guess this one could be larger.
>>
>>
>> <property name="revisedLRUCapacity" value="100"/>
>>
>> <store type="db">
>> <!-- type: horizontal | vertical | <any
>> user-contributed type> --> <mappingStrategy
>> type="horizontal">
>> <!-- ONE_TABLE_PER_REFERENCE |
>> ONE_TABLE_PER_CLASS |
>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>> <property name="toManyReferences"
>> value="ONE_TABLE_PER_REFERENCE"/>
>>
>> <!-- LIKE_ATTRIBUTES |
>> LIKE_TO_MANY_REFERENCES --> <property
>> name="toOneReferences"
>> value="LIKE_ATTRIBUTES"/>
>>
>> <!-- MODEL | STRATEGY -->
>> <property name="mappingPrecedence"
>> value="MODEL"/>
>> This one is obsolete.
>>
>>
>> </mappingStrategy>
>>
>> <dbAdapter name="derby-embedded"/>
>> <dataSource
>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>> databaseName="/tmp/cdodb1"
>> createDatabase="create"/>
>>
>> </store>
>> </repository>
>>
>>
>>
>>
>> What do you think about that ? What throughput do you think I should
>> expect ? As I said that's very difficult to judge.
>> I believe it's more appropriate to compare the actual values with values
>> from other measures (other setups, properties, etc. or even other
>> products) instead of random expectations. Apologies that I didn't have
>> the time so far to provide some reference measures myself. Maybe this
>> gets better now that the team starts to grow slowly ;-)
>>
>>
>> What are the main reasons fur such figures ?
>> Hmm, I *suspect* you don't like the figures? I can find no evidence your
>> assessment ;-)
>>
>> Ahh,one more note:
>> Tracing can also have a significant impact on performance and end-to-end
>> throughput. Turn it off by all means. And please make sure that it's
>> really turned off, not only invisible (i.e. console is not a target).
>>
>> Cheers
>> /Eike
>


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Cell: +31 (0)6 288 48 943
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: [CDO] reading/writting throughput performances [message #124124 is a reply to message #123787] Fri, 30 May 2008 13:08 Go to previous messageGo to next message
Simon Mc Duff is currently offline Simon Mc DuffFriend
Messages: 596
Registered: July 2009
Senior Member
This is a multi-part message in MIME format.

------=_NextPart_000_0540_01C8C234.B8DE3F40
Content-Type: text/plain;
charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Performance problem with resource.content
https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D234839
"Eike Stepper" <stepper@sympedia.de> a =C3=A9crit dans le message de =
news: g1mp23$pu1$4@build.eclipse.org...
Guys,

Comments inline...


Simon McDuff schrieb:=20
We noticed a performance degradation in the following case:
=20
When we fetch a collection of objects that contains collections that =
we don't want to use.=20
Ex:
=20
Model:
=20
ClassB
{
int foo1;
List childrens1;
List childrens2;
List childrens3;
};
=20
Code:

Resource.contents contains objects of type ClassB.
=20
By fetching instances of classB, we will also fetched some items in =
the list(childrens1, childrens2, childrens3) The problem we had.. is we =
do not know if we want to use it. SO better not to download them.

So In our production(we used our own IStore), we never download any =
objects from collection.=20
I'm working at the moment to put in place all feature we used in the =
current CDO.
That one will be part of=20
https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214477. To begin =
with,CDORevisionImpl needs to supports list that are 0 even if =
referenceChunkSize =3D=3D 0.
=20
It is true that we obtain good performance.
Worst case :
1500 objects /sec (Real fetch from disk. Page that contains the =
objects, wasn't in the cache of the filesystem.)=20
=20
Best case=20
12 000 objects/Sec (Fetch objects that are already in the case of =
the filesystem)
=20
(These statistics always fetch objects from server to client. The =
cache client isn't used)
=20
We have another optimization one with SmartReadAhead but for now =
only work for UI client. (for now) This is something we will integrate =
as well in CDO. But for now we concentrate on the other things.
=20
I used an Object database. Based on my preliminary test without CDO.
I compare
=20
Hibernate vs Objectivity/EMF :=20
Objectivity/EMF is around 3 to 50 times faster than Hibernate.
=20
The worst.. you will get 30 to 240 objects/sec or
the best... you will get 500 to 4000 objects/sec
=20
Based on your comments... you are getting the worst (with Derby)!! =
:-)

And with the DBStore!!
I'd really like to encourage you to use the HibernateStore.=20
It is much younger than the DBStore but Hibernate itself should do a =
much better job on mapping than the DBStore.

That said, the DBStore will, for multiple reasons, never cease to =
exist in CDO.



=20
Another comments about insertion.
=20
We found another problem when we insert data(DOn't know if it is =
still true). Resource.Contents is a unique list. Basically each time we =
insert one element, it needs to fetch all the collection to see if we =
already have this element. Basically.. it is time consuming. In our case =
we are not going through this stage to persist our objects... but the =
objects are not accessible through that list.

I was not aware of this. Sounds as if we should offer an alternative =
implementation for Resource.Contents.
Could you please open a Bugzilla to discuss/track this?





If speed is an issue for you!! We (You included :-)) can optimize =
it. What I like is Eike did a really good job doing that framwork and =
optimizations are easy to add.
Also you can change your back-end without affecting you client side. =
(Maybe going to an Object database!!)
THis was one of the purpose of CDO!! :-)
Thank you ;-)
Indeed, CDO was intended to focus of the protocol aspects, the =
repository and the client side addition to the EMF concepts.
I feel that this will become clearer the more IStore implementations =
we get that depend on mature 3rd party code for the mapping part.

Cheers
/Eike





Simon


"Eike Stepper" <stepper@sympedia.de> a =C3=A9crit dans le message =
de news: 483E9332.6050800@sympedia.de...
Hi C=C3=A9dric,

That sounds interesting!
Comments inline...


C=C3=A9dric Brun schrieb:=20
Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex =
one -
250 EClassifiers/300 features and a simple one - 1 Classifier/3 =
Features)

The test is a "quick and dirty" one so please do not take these figures =
too
seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.
Looks like a sign that the NIO socket channels are efficient ,-)
You used localhost for TCP transport I guess?


* First case : building a small model (10000 elements) with the simple =
MM,
only adding instances in an empty database :=20
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.
Seems to be consistent with the fact that only for commits there is =
remote traffic.
And the backend transaction management is another overhead.
So the impact of the number objects/commit starts with bigger =
transactions.


browsing the just commited elements opening a new transaction and =
getting a
new Resource instance, I get around 30 000 objets/sec rate just =
iterating
thanks to Resource.getAllContents().
Hehe, this is a sign that client side caching works well:
All views and transactions of a single session share (and cache) =
their object state.
Otherwise I'd expect much smaller values.
With his ObjectivityStore implementation Simon (cc'ed) reached =
rates of several thousands objects/sec.
But the DBStore (which you used) is not that optimized.


* Second case : I built a big model in the database using the complex
metamodel. It contains 700 000 elements. The creation went pretty nicely =
(I
did not measured it exactly as it was launched during a night, but it =
seems
I got around 100 objects/sec rate in writting as I was writting every =
700
objects.
Sounds not too bad.


Once the model is built the derby database is 222 MB (which - I heard - =
is
not so big for derby).
Hmm, I have no extensive experience with Derby.
To be honest, I'm really no database expert (although I can write =
simple SQL statements).


Retrieving the distant resource and calling eAllContents() on it, I get =
a 50
Objects/sec rate at best, which mean my eAllContents() call would take 4
hours !
Currently I'm not sure what exactly eAllContents() does do (iteration =
order and so).
There are several CDO features that can have a major impact on =
load rates!

- Collections (xrefs and containment) can be loaded lazily. I =
don't mean the target objects, which are lazy by default anyway, but =
even the target IDs that are in the collections. We call that "reference =
chunking" (but search for a better name is ongoing).

public interface CDOSession extends CDOProtocolSession, =
IContainer<CDOView>
{
public int getReferenceChunkSize();

public void setReferenceChunkSize(int referenceChunkSize); =
=20


- Loading collection elements (target objects) can be batched (the =
current name is even worse)

public interface CDOView extends CDOProtocolView, INotifier
{
public int getLoadRevisionCollectionChunkSize();

public void setLoadRevisionCollectionChunkSize(int =
loadRevisionCollectionChunkSize); =20


- One of the most sophisticated feature is automatic model usage =
analyzation. The result are fetch rules that are sent to the server. =
When later objects are loaded from the server, it can answer with =
additional objects that are likely to be needed by the client in the =
near future.

public class CDOViewImpl extends =
org.eclipse.net4j.internal.util.event.Notifier implements CDOView, =
CDOIDProvider, Adapter.Internal
{
public CDOFeatureAnalyzer getFeatureAnalyzer()
{
return featureAnalyzer;
}

public void setFeatureAnalyzer(CDOFeatureAnalyzer =
featureAnalyzer)
{
this.featureAnalyzer =3D featureAnalyzer =3D=3D null ? =
CDOFeatureAnalyzer.NOOP : featureAnalyzer;
} =20


- There's a SmartReadAheadThread (if you're curious, please ask =
Simon).
- Caches can be configured
- ...


Known issues of the DBStore:
- When in auditing mode updates are always written as complete =
inserts (by design)
- When configuring toManyReferences to a value different from =
LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
- No (few) prepared statements are used (see =
https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214487)
- Maybe others...

Please note that the DBStore was not written by me to compete with =
Hibernate in terms of performance or mapping capabilities.
Now that Martin and I, well in the end mostly Martin, provide a =
HibernateStore and this HibernateStore gets more and more features I =
expect that there will be a trend away from the DBStore.=20


This is probably linked to the way the mapping to db is done, here's the
config I used (came from CDO-server example if I recall correctly).


<repository name=3D"repo1">
<property name=3D"overrideUUID"
value=3D"1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
<property name=3D"supportingAudits" value=3D"true"/>
<property name=3D"verifyingRevisions" value=3D"false"/>
<property name=3D"currentLRUCapacity" value=3D"10000"/>
I guess this one could be larger.


<property name=3D"revisedLRUCapacity" value=3D"100"/>
=20
<store type=3D"db">
<!-- type: horizontal | vertical | <any =
user-contributed type> -->
<mappingStrategy type=3D"horizontal">
<!-- ONE_TABLE_PER_REFERENCE | =
ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
<property name=3D"toManyReferences" =
value=3D"ONE_TABLE_PER_REFERENCE"/>

<!-- LIKE_ATTRIBUTES | =
LIKE_TO_MANY_REFERENCES -->
<property name=3D"toOneReferences" =
value=3D"LIKE_ATTRIBUTES"/>
=20
<!-- MODEL | STRATEGY -->
<property name=3D"mappingPrecedence" =
value=3D"MODEL"/>
This one is obsolete.


</mappingStrategy>
=20
<dbAdapter name=3D"derby-embedded"/>
<dataSource =
class=3D"org.apache.derby.jdbc.EmbeddedDataSource"=20
databaseName=3D"/tmp/cdodb1"
createDatabase=3D"create"/>

</store>
</repository>




What do you think about that ? What throughput do you think I should
expect ? As I said that's very difficult to judge.
I believe it's more appropriate to compare the actual values with =
values from other measures (other setups, properties, etc. or even other =
products) instead of random expectations.
Apologies that I didn't have the time so far to provide some =
reference measures myself. Maybe this gets better now that the team =
starts to grow slowly ;-)


What are the main reasons fur such figures ?
Hmm, I *suspect* you don't like the figures? I can find no evidence =
your assessment ;-)

Ahh,one more note:
Tracing can also have a significant impact on performance and =
end-to-end throughput.
Turn it off by all means. And please make sure that it's really =
turned off, not only invisible (i.e. console is not a target).

Cheers
/Eike



------=_NextPart_000_0540_01C8C234.B8DE3F40
Content-Type: text/html;
charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

=EF=BB=BF<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD><TITLE></TITLE>
<META http-equiv=3DContent-Type content=3Dtext/html;charset=3DUTF-8>
<META content=3D"MSHTML 6.00.6000.16608" name=3DGENERATOR></HEAD>
<BODY text=3D#000000 bgColor=3D#ffffff>
<DIV><FONT face=3DArial size=3D2>Performance problem with=20
resource.content</FONT></DIV>
<DIV><FONT face=3DArial size=3D2><A=20
href=3D"https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D234839">https://b=
ugs.eclipse.org/bugs/show_bug.cgi?id=3D234839</A></FONT></DIV >
<BLOCKQUOTE dir=3Dltr=20
style=3D"PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; =
BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">
<DIV>"Eike Stepper" &lt;<A=20
href=3D"mailto:stepper@sympedia.de">stepper@sympedia.de</A>&gt; a =
=C3=A9crit dans le=20
message de news: <A=20
=
href=3D"mailto:g1mp23$pu1$4@build.eclipse.org">g1mp23$pu1$4@build.eclipse=
..org</A>...</DIV>Guys,<BR><BR>Comments=20
inline...<BR><BR><BR>Simon McDuff schrieb:=20
<BLOCKQUOTE cite=3Dmid:g1mkji$h5d$1@build.eclipse.org type=3D"cite">
<META content=3D"MSHTML 6.00.6000.16608" name=3DGENERATOR>
<DIV>We noticed a performance degradation in the following=20
case:<BR>&nbsp;<BR>When we fetch a collection of objects that =
contains=20
collections that we don't want to use.=20
<BR>Ex:<BR>&nbsp;<BR>Model:<BR>&nbsp;<BR>ClassB <BR>{<BR>&nbsp; int=20
foo1;<BR>&nbsp; List childrens1;<BR>&nbsp; List =
childrens2;<BR>&nbsp; List=20
childrens3;<BR>};<BR>&nbsp;<BR>Code:</DIV>
<DIV><BR>Resource.contents contains objects of type =
ClassB.<BR>&nbsp;<BR>By=20
fetching instances of classB, we will also fetched some items in the =

list(childrens1, childrens2, childrens3)&nbsp;<FONT face=3D"Courier =
New"><FONT=20
color=3D#000000>The problem we had.. is we do not know if we want to =
use it.=20
SO better not to download them.</FONT></FONT></DIV>
<DIV><BR><FONT face=3D"Courier New" color=3D#000000>So In our =
production(we used=20
our own IStore), we never download any objects from collection.=20
</FONT><BR><FONT face=3D"Courier New" color=3D#000000>I'm working at =
the moment=20
to put in place all feature we used in the current =
CDO.</FONT><BR><FONT=20
face=3D"Courier New" color=3D#000000>That one will be part of =
</FONT><BR><A=20
href=3D"https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214477" =
target=3D_blank=20
moz-do-not-send=3D"true"><FONT=20
=
color=3D#0068cf>https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214477</F=
ONT></A>.=20
To begin with,CDORevisionImpl needs to supports list that are 0 even =
if=20
<FONT face=3D"Courier New" color=3D#000000>referenceChunkSize =3D=3D =

0.</FONT><BR>&nbsp;<BR><FONT face=3D"Courier New" color=3D#000000>It =
is true=20
that we obtain good performance.</FONT><BR><FONT face=3D"Courier =
New"=20
color=3D#000000>Worst case :</FONT><BR><FONT face=3D"Courier New"=20
color=3D#000000>1500 objects /sec (Real fetch from&nbsp;disk. Page =
that=20
contains the objects, wasn't in the cache of the=20
filesystem.)&nbsp;</FONT><BR>&nbsp;<BR><FONT face=3D"Courier New"=20
color=3D#000000>Best case </FONT><BR><FONT face=3D"Courier New" =
color=3D#000000>12=20
000 objects/Sec (Fetch objects that are already in the case of the=20
filesystem)</FONT><BR>&nbsp;<BR>(These statistics always fetch =
objects from=20
server to client. The cache client isn't used)<BR>&nbsp;<BR><FONT=20
face=3D"Courier New" color=3D#000000>We have another optimization =
one with=20
SmartReadAhead but for now only work for UI client. (for now)&nbsp; =
This is=20
something we will integrate as well in CDO. But for now we =
concentrate on=20
the other things.</FONT><BR>&nbsp;<BR><FONT face=3D"Courier New"=20
color=3D#000000>I used an Object database. Based on my preliminary =
test=20
without CDO.</FONT><BR><FONT face=3D"Courier New" color=3D#000000>I=20
compare</FONT><BR>&nbsp;<BR><FONT face=3D"Courier New" =
color=3D#000000>Hibernate=20
vs </FONT><FONT face=3D"Courier New" color=3D#000000>Objectivity/EMF =

:&nbsp;</FONT><BR><FONT face=3D"Courier New" =
color=3D#000000>Objectivity/EMF is=20
around 3 to 50 times faster than =
Hibernate.</FONT><BR>&nbsp;<BR><FONT=20
face=3D"Courier New" color=3D#000000>The worst.. you will get 30 to =
240=20
objects/sec or</FONT><BR><FONT face=3D"Courier New" =
color=3D#000000>the best...=20
you will get 500 to 4000 objects/sec</FONT><BR>&nbsp;<BR><FONT=20
face=3D"Courier New" color=3D#000000>Based on your comments... you =
are getting=20
the worst (with Derby)!! :-)</FONT><BR></DIV></BLOCKQUOTE>And with =
the=20
DBStore!!<BR>I'd really like to encourage you to use the =
HibernateStore.=20
<BR>It is much younger than the DBStore but Hibernate itself should do =
a much=20
better job on mapping than the DBStore.<BR><BR>That said, the DBStore =
will,=20
for multiple reasons, never cease to exist in CDO.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1mkji$h5d$1@build.eclipse.org type=3D"cite">
<DIV><BR>&nbsp;<BR>Another comments about insertion.<BR>&nbsp;<BR>We =
found=20
another problem when we insert data(DOn't know if it is still true). =

Resource.Contents is&nbsp;a unique list. Basically each time we =
insert one=20
element, it needs to fetch all the collection to see if we already =
have this=20
element. Basically.. it is time consuming. In our case we are not =
going=20
through this stage to persist our objects... but the objects are not =

accessible through that list.<BR></DIV></BLOCKQUOTE>I was not aware =
of this.=20
Sounds as if we should offer an alternative implementation for=20
Resource.Contents.<BR>Could you please open a Bugzilla to =
discuss/track=20
this?<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1mkji$h5d$1@build.eclipse.org type=3D"cite">
<DIV>&nbsp;</DIV>
<DIV>&nbsp;</DIV>
<DIV>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>If speed is an issue for you!! We =
(You included=20
:-)) can optimize it. What I like is Eike did a really good job =
doing that=20
framwork and optimizations are easy to add.</FONT></DIV>
<DIV><FONT face=3DArial size=3D2>Also you can change your back-end =
without=20
affecting you client side. (Maybe going to an Object=20
database!!)</FONT></DIV>
<DIV><FONT face=3DArial size=3D2>THis was one of the purpose of =
CDO!!=20
:-)</FONT></DIV></BLOCKQUOTE>Thank you ;-)<BR>Indeed, CDO was =
intended to=20
focus of the protocol aspects, the repository and the client side =
addition to=20
the EMF concepts.<BR>I feel that this will become clearer the more =
IStore=20
implementations we get that depend on mature 3rd party code for the =
mapping=20
part.<BR><BR>Cheers<BR>/Eike<BR><BR><BR>
<BLOCKQUOTE cite=3Dmid:g1mkji$h5d$1@build.eclipse.org type=3D"cite">
<DIV>&nbsp;</DIV>
<DIV><BR>Simon<BR><BR></DIV>
<BLOCKQUOTE=20
style=3D"PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; =
BORDER-LEFT: rgb(0,0,0) 2px solid; MARGIN-RIGHT: 0px">
<DIV>"Eike Stepper" &lt;<A href=3D"mailto:stepper@sympedia.de"=20
moz-do-not-send=3D"true">stepper@sympedia.de</A>&gt; a =C3=A9crit =
dans le message=20
de news: <A href=3D"mailto:483E9332.6050800@sympedia.de"=20
=
moz-do-not-send=3D"true">483E9332.6050800@sympedia.de</A>...</DIV>Hi=20
C=C3=A9dric,<BR><BR>That sounds interesting!<BR>Comments=20
inline...<BR><BR><BR>C=C3=A9dric Brun schrieb:=20
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex =
one -
250 EClassifiers/300 features and a simple one - 1 Classifier/3 =
Features)

The test is a "quick and dirty" one so please do not take these figures =
too
seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.
</PRE></BLOCKQUOTE>Looks like a sign that the NIO socket channels are=20
efficient ,-)<BR>You used localhost for TCP transport I =
guess?<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">* First case : building a small model =
(10000 elements) with the simple MM,
only adding instances in an empty database :=20
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.
</PRE></BLOCKQUOTE>Seems to be consistent with the fact that only for=20
commits there is remote traffic.<BR>And the backend transaction =
management=20
is another overhead.<BR>So the impact of the number objects/commit =
starts=20
with bigger transactions.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">browsing the just commited elements opening =
a new transaction and getting a
new Resource instance, I get around 30 000 objets/sec rate just =
iterating
thanks to Resource.getAllContents().
</PRE></BLOCKQUOTE>Hehe, this is a sign that client side caching works =

well:<BR>All views and transactions of a single session share (and =
cache)=20
their object state.<BR>Otherwise I'd expect much smaller =
values.<BR>With=20
his ObjectivityStore implementation Simon (cc'ed) reached rates of =
several=20
thousands objects/sec.<BR>But the DBStore (which you used) is not =
that=20
optimized.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">* Second case : I built a big model in the =
database using the complex
metamodel. It contains 700 000 elements. The creation went pretty nicely =
(I
did not measured it exactly as it was launched during a night, but it =
seems
I got around 100 objects/sec rate in writting as I was writting every =
700
objects.
</PRE></BLOCKQUOTE>Sounds not too bad.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">Once the model is built the derby database =
is 222 MB (which - I heard - is
not so big for derby).
</PRE></BLOCKQUOTE>Hmm, I have no extensive experience with =
Derby.<BR>To=20
be honest, I'm really no database expert (although I can write =
simple SQL=20
statements).<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">Retrieving the distant resource and calling =
eAllContents() on it, I get a 50
Objects/sec rate at best, which mean my eAllContents() call would take 4
hours !
</PRE></BLOCKQUOTE>Currently I'm not sure what exactly eAllContents()=20
does do (iteration order and so).<BR>There are several CDO =
features that=20
can have a major impact on load rates!<BR><BR>- Collections (xrefs =
and=20
containment) can be loaded lazily. I don't mean the target =
objects, which=20
are lazy by default anyway, but even the target IDs that are in =
the=20
collections. We call that "reference chunking" (but search for a =
better=20
name is ongoing).<BR>
<STYLE type=3Dtext/css>CODE {
FONT-SIZE: 10pt; MARGIN: 0px; FONT-FAMILY: Courier New, Courier
}
</STYLE>
<!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><!-- =3D Java Sourcecode to HTML automatically =
converted code =3D --><!-- =3D Java2Html Converter 5.0 [2006-02-26] by =
Markus Gebhard markus@jave.de =3D --><!-- =3D Further =
information: http://www.java2html.de =3D -->
<DIV class=3Djava align=3Dleft>
<TABLE cellSpacing=3D0 cellPadding=3D3 bgColor=3D#ffffff =
border=3D0>
<TBODY>
<TR><!-- start source code -->
<TD vAlign=3Dtop noWrap align=3Dleft><CODE><FONT=20
=
color=3D#7f0055><B>public&nbsp;interface&nbsp;</B ></FONT><FONT=20
color=3D#000000>CDOSession&nbsp;</FONT><FONT=20
color=3D#7f0055><B>extends&nbsp;</B></FONT><FONT=20
=
color=3D#000000> CDOProtocolSession,&nbsp;IContainer&lt;CDOView&g t; </FONT>=
<BR><FONT=20
color=3D#000000>{</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>int&nbsp;</B></FONT><FONT=20
color=3D#000000>getReferenceChunkSize</FONT><FONT=20
color=3D#000000>()</FONT><FONT =
color=3D#000000>;</FONT><BR><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>void&nbsp;</B></FONT><FONT=20
color=3D#000000>setReferenceChunkSize</FONT><FONT=20
color=3D#000000>(</FONT><FONT=20
color=3D#7f0055><B>int&nbsp;</B></FONT><FONT=20
color=3D#000000>referenceChunkSize</FONT><FONT=20
color=3D#000000>)</FONT><FONT =
color=3D#000000>;</FONT></CODE> </TD><!-- end source code =
--></TR></TBODY></TABLE></DIV><!-- =3D END of automatically =
generated HTML code =3D --><BR><!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D -->-=20
Loading collection elements (target objects) can be batched (the =
current=20
name is even worse)<BR>
<STYLE type=3Dtext/css>
<!--code { font-family: Courier New, Courier; font-size: 10pt; =
margin: 0px; }-->
</STYLE>
<!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><!-- =3D Java Sourcecode to HTML automatically =
converted code =3D --><!-- =3D Java2Html Converter 5.0 [2006-02-26] by =
Markus Gebhard markus@jave.de =3D --><!-- =3D Further =
information: http://www.java2html.de =3D -->
<DIV class=3Djava align=3Dleft>
<TABLE cellSpacing=3D0 cellPadding=3D3 bgColor=3D#ffffff =
border=3D0>
<TBODY>
<TR><!-- start source code -->
<TD vAlign=3Dtop noWrap align=3Dleft><CODE><FONT=20
=
color=3D#7f0055><B>public&nbsp;interface&nbsp;</B ></FONT><FONT=20
color=3D#000000>CDOView&nbsp;</FONT><FONT=20
color=3D#7f0055><B>extends&nbsp;</B></FONT><FONT=20
=
color=3D#000000>CDOProtocolView,&nbsp;INotifier</FONT> <BR><FONT=20
color=3D#000000>{</FONT><BR><FONT color=3D#ffffff>&nbsp; =
</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>int&nbsp;</B></FONT><FONT=20
=
color=3D#000000>getLoadRevisionCollectionChunkSize</FONT><FONT=20
color=3D#000000>()</FONT><FONT =
color=3D#000000>;</FONT><BR><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>void&nbsp;</B></FONT><FONT=20
=
color=3D#000000>setLoadRevisionCollectionChunkSize</FONT><FONT=20
color=3D#000000>(</FONT><FONT=20
color=3D#7f0055><B>int&nbsp;</B></FONT><FONT=20
color=3D#000000>loadRevisionCollectionChunkSize</FONT><FONT=20
color=3D#000000>)</FONT><FONT =
color=3D#000000>;</FONT></CODE> </TD><!-- end source code =
--></TR></TBODY></TABLE></DIV><!-- =3D END of automatically =
generated HTML code =3D --><!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><BR>-=20
One of the most sophisticated feature is automatic model usage=20
analyzation. The result are fetch rules that are sent to the =
server. When=20
later objects are loaded from the server, it can answer with =
additional=20
objects that are likely to be needed by the client in the near =
future.<BR>
<STYLE type=3Dtext/css>
<!--code { font-family: Courier New, Courier; font-size: 10pt; =
margin: 0px; }-->
</STYLE>
<!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><!-- =3D Java Sourcecode to HTML automatically =
converted code =3D --><!-- =3D Java2Html Converter 5.0 [2006-02-26] by =
Markus Gebhard markus@jave.de =3D --><!-- =3D Further =
information: http://www.java2html.de =3D -->
<DIV class=3Djava align=3Dleft>
<TABLE cellSpacing=3D0 cellPadding=3D3 bgColor=3D#ffffff =
border=3D0>
<TBODY>
<TR><!-- start source code -->
<TD vAlign=3Dtop noWrap align=3Dleft><CODE><FONT=20
color=3D#7f0055><B>public&nbsp;class&nbsp;</B></FONT ><FONT=20
color=3D#000000>CDOViewImpl&nbsp;</FONT><FONT=20
color=3D#7f0055><B>extends&nbsp;</B></FONT><FONT=20
=
color=3D#000000>org.eclipse.net4j.internal.util.event.Notifier&nbsp; </FON=
T><FONT=20
color=3D#7f0055><B>implements&nbsp;</B></FONT><FONT=20
color=3D#000000>CDOView,&nbsp;CDOIDProvider,</FONT> <FONT=20
color=3D#000000>Adapter.Internal</FONT><BR><FONT=20
color=3D#000000>{</FONT><BR><FONT color=3D#ffffff>&nbsp; =
</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
=
color=3D#000000>CDOFeatureAnalyzer&nbsp;getFeatureAnalyzer </FONT><FONT=20
color=3D#000000>()</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#000000>{</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;&nbsp;&nbsp;</FONT ><FONT=20
color=3D#7f0055><B>return&nbsp;</B></FONT><FONT=20
color=3D#000000>featureAnalyzer;</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#000000>}</FONT><BR><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>void&nbsp;</B></FONT><FONT=20
color=3D#000000>setFeatureAnalyzer</FONT><FONT=20
color=3D#000000>(</FONT><FONT=20
=
color=3D#000000>CDOFeatureAnalyzer&nbsp;featureAnalyzer </FONT><FONT=20
color=3D#000000>)</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#000000>{</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;&nbsp;&nbsp;</FONT ><FONT=20
color=3D#7f0055><B>this</B></FONT><FONT=20
=
color=3D#000000> .featureAnalyzer&nbsp;=3D&nbsp;featureAnalyzer&n bsp;=3D=3D=
&nbsp;</FONT><FONT=20
color=3D#7f0055><B>null&nbsp;</B></FONT><FONT=20
=
color=3D#000000> ?&nbsp;CDOFeatureAnalyzer.NOOP&nbsp;:&nbsp;featu reAnalyze=
r;</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT =
color=3D#000000>}</FONT></CODE>=20
</TD><!-- end source code --></TR></TBODY></TABLE></DIV><!-- =
=3D END of automatically generated HTML code =3D --><!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><BR>-=20
There's a SmartReadAheadThread (if you're curious, please ask =
Simon).<BR>-=20
Caches can be configured<BR>- ...<BR><BR><BR>Known issues of the=20
DBStore:<BR>- When in auditing mode updates are always written as =
complete=20
inserts (by design)<BR>- When configuring toManyReferences to a =
value=20
different from LIKE_ATTRIBUTES (which is not implemented yet) it =
is very=20
slow.<BR>- No (few) prepared statements are used (see <A=20
class=3Dmoz-txt-link-freetext=20
href=3D"https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214487"=20
=
moz-do-not-send=3D"true">https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D=
214487</A>)<BR>-=20
Maybe others...<BR><BR>Please note that the DBStore was not =
written by me=20
to compete with Hibernate in terms of performance or mapping=20
capabilities.<BR>Now that Martin and I, well in the end mostly =
Martin,=20
provide a HibernateStore and this HibernateStore gets more and =
more=20
features I expect that there will be a trend away from the =
DBStore.=20
<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">This is probably linked to the way the =
mapping to db is done, here's the
config I used (came from CDO-server example if I recall correctly).


&lt;repository name=3D"repo1"&gt;
&lt;property name=3D"overrideUUID"
value=3D"1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/&gt;
&lt;property name=3D"supportingAudits" =
value=3D"true"/&gt;
&lt;property name=3D"verifyingRevisions" =
value=3D"false"/&gt;
&lt;property name=3D"currentLRUCapacity" =
value=3D"10000"/&gt;
</PRE></BLOCKQUOTE>I guess this one could be larger.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D""> &lt;property =
name=3D"revisedLRUCapacity" value=3D"100"/&gt;
=20
&lt;store type=3D"db"&gt;
&lt;!-- type: horizontal | vertical | &lt;any =
user-contributed type&gt; --&gt;
&lt;mappingStrategy type=3D"horizontal"&gt;
&lt;!-- ONE_TABLE_PER_REFERENCE | =
ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES =
--&gt;
&lt;property name=3D"toManyReferences" =
value=3D"ONE_TABLE_PER_REFERENCE"/&gt;

&lt;!-- LIKE_ATTRIBUTES | =
LIKE_TO_MANY_REFERENCES --&gt;
&lt;property name=3D"toOneReferences" =
value=3D"LIKE_ATTRIBUTES"/&gt;
=20
&lt;!-- MODEL | STRATEGY --&gt;
&lt;property name=3D"mappingPrecedence" =
value=3D"MODEL"/&gt;
</PRE></BLOCKQUOTE>This one is obsolete.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D""> =
&lt;/mappingStrategy&gt;
=20
&lt;dbAdapter name=3D"derby-embedded"/&gt;
&lt;dataSource =
class=3D"org.apache.derby.jdbc.EmbeddedDataSource"=20
databaseName=3D"/tmp/cdodb1"
createDatabase=3D"create"/&gt;

&lt;/store&gt;
&lt;/repository&gt;




What do you think about that ? What throughput do you think I should
expect ? </PRE></BLOCKQUOTE>As I said that's very difficult to=20
judge.<BR>I believe it's more appropriate to compare the actual =
values=20
with values from other measures (other setups, properties, etc. or =
even=20
other products) instead of random expectations.<BR>Apologies that =
I didn't=20
have the time so far to provide some reference measures myself. =
Maybe this=20
gets better now that the team starts to grow slowly ;-)<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">What are the main reasons fur such figures =
?
</PRE></BLOCKQUOTE>Hmm, I *suspect* you don't like the figures? I can=20
find no evidence your assessment ;-)<BR><BR>Ahh,one more =
note:<BR>Tracing=20
can also have a significant impact on performance and end-to-end=20
throughput.<BR>Turn it off by all means. And please make sure that =
it's=20
really turned off, not only invisible (i.e. console is not a=20
=
target).<BR><BR>Cheers<BR>/Eike<BR><BR><BR></BLOCKQUOTE ></BLOCKQUOTE></BL=
OCKQUOTE></BODY></HTML>

------=_NextPart_000_0540_01C8C234.B8DE3F40--
Re: [CDO] reading/writting throughput performances [message #124150 is a reply to message #124092] Fri, 30 May 2008 13:19 Go to previous messageGo to next message
Cedric Brun is currently offline Cedric BrunFriend
Messages: 431
Registered: July 2009
Senior Member
Hi Martin,

and thanks for your quick answer ! what I did is the following :

I used the org.eclipse.emf.cdo.server.hibernate.libraries bundle and added
that :

Bundle-ClassPath: lib/hibernate3.jar,
lib/cglib-2.1.3.jar,
lib/commons-collections-2.1.1.jar,
lib/dom4j.jar,
lib/jta.jar
lib/asm-3.0.jar

and this directive :

Eclipse-BuddyPolicy: dependent

as stated in
http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries

Then I don't get the same error, now when I start my own plugin (JUnit
plugin test) depending on org.eclipse.emf.cdo.server.hibernate.libraries I
get :

java.lang.NoClassDefFoundError: org/objectweb/asm/Type
at net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
at net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
at net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
at
org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)

These jars are such a headache ! Do you know - by chance - what's going
on ?


Cédric


Martin Taal wrote:

> Hi Cédric,
> Most of the time this exception is thrown because hibernate can't find the
> class. A common cause is that the hibernate jar files are located in a
> separate plugin which can not reach the classes in other plugins. If you
> set the eclipse buddy policy of the hibernate plugin to dependent then
> this can help.
>
> gr. Martin
>
> Cédric Brun wrote:
>> Hi Simon and Eike,
>>
>> thanks for your feedback, now I know what I could expect in using
>> optimized implementation, I'll have a try on the hibernate store and see
>> if it is scaling in a better way.
>>
>> You gave me many pointers to explain the performances issues but in fact
>> it seems that the SQL execution is taking so much time (not suprisingly
>> in fact ;) . My first profile was wrong.
>>
>> So I'm installing the hibernate store retrieving all the source code from
>> CVS, and I'm trying to launch the unit tests. I added many jars and
>> everything gets compiled but most of the tests are failling with "
>>
>> "
>> org.hibernate.MappingException: Could not determine type for:
>> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>> at table: cdo_features, for columns: [org.hibernate.mapping.Column(type)]
>> "
>>
>> any hint ?
>>
>> Cédric
>>
>>
>> Simon McDuff wrote:
>>
>>> We noticed a performance degradation in the following case:
>>>
>>> When we fetch a collection of objects that contains collections that we
>>> don't want to use. Ex:
>>>
>>> Model:
>>>
>>> ClassB
>>> {
>>> int foo1;
>>> List childrens1;
>>> List childrens2;
>>> List childrens3;
>>> };
>>>
>>> Code:
>>>
>>> Resource.contents contains objects of type ClassB.
>>>
>>> By fetching instances of classB, we will also fetched some items in the
>>> list(childrens1, childrens2, childrens3) The problem we had.. is we do
>>> not know if we want to use it. SO better not to download them.
>>>
>>> So In our production(we used our own IStore), we never download any
>>> objects from collection. I'm working at the moment to put in place all
>>> feature we used in the current CDO. That one will be part of
>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477. To begin
>>> with,CDORevisionImpl needs to supports list that are 0 even if
>>> referenceChunkSize == 0.
>>>
>>> It is true that we obtain good performance.
>>> Worst case :
>>> 1500 objects /sec (Real fetch from disk. Page that contains the objects,
>>> wasn't in the cache of the filesystem.)
>>>
>>> Best case
>>> 12 000 objects/Sec (Fetch objects that are already in the case of the
>>> filesystem)
>>>
>>> (These statistics always fetch objects from server to client. The cache
>>> client isn't used)
>>>
>>> We have another optimization one with SmartReadAhead but for now only
>>> work
>>> for UI client. (for now) This is something we will integrate as well in
>>> CDO. But for now we concentrate on the other things.
>>>
>>> I used an Object database. Based on my preliminary test without CDO.
>>> I compare
>>>
>>> Hibernate vs Objectivity/EMF :
>>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>>
>>> The worst.. you will get 30 to 240 objects/sec or
>>> the best... you will get 500 to 4000 objects/sec
>>>
>>> Based on your comments... you are getting the worst (with Derby)!! :-)
>>>
>>>
>>> Another comments about insertion.
>>>
>>> We found another problem when we insert data(DOn't know if it is still
>>> true). Resource.Contents is a unique list. Basically each time we insert
>>> one element, it needs to fetch all the collection to see if we already
>>> have this element. Basically.. it is time consuming. In our case we are
>>> not going through this stage to persist our objects... but the objects
>>> are not accessible through that list.
>>>
>>>
>>>
>>> If speed is an issue for you!! We (You included :-)) can optimize it.
>>> What I like is Eike did a really good job doing that framwork and
>>> optimizations are easy to add. Also you can change your back-end without
>>> affecting you client side. (Maybe going to an Object database!!) THis
>>> was one of the purpose of CDO!! :-)
>>>
>>>
>>> Simon
>>>
>>>
>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le message de news:
>>> 483E9332.6050800@sympedia.de... Hi Cédric,
>>>
>>> That sounds interesting!
>>> Comments inline...
>>>
>>>
>>> Cédric Brun schrieb:
>>> Hi Eike,
>>>
>>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>>> measuring read/write throughput when I use CDO (RC2).
>>>
>>> I'm using an embeded-derby database, two different metamodels (complex
>>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>>> Features)
>>>
>>> The test is a "quick and dirty" one so please do not take these figures
>>> too seriously, I would just like to have your input about the "order of
>>> magnitude" I should expect or settings affecting seriously these
>>> performances. I tried both JVM and TCP acceptor but the performances are
>>> quite similar though JVM is a bit faster.
>>> Looks like a sign that the NIO socket channels are efficient ,-)
>>> You used localhost for TCP transport I guess?
>>>
>>>
>>>
>>> * First case : building a small model (10000 elements) with the simple
>>> MM, only adding instances in an empty database :
>>> commiting change on every add I get a 2 objects per second rate.
>>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>> Seems to be consistent with the fact that only for commits there is
>>> remote traffic. And the backend transaction management is another
>>> overhead. So the impact of the number objects/commit starts with
>>> bigger transactions.
>>>
>>>
>>> browsing the just commited elements opening a new transaction and
>>> getting a new Resource instance, I get around 30 000 objets/sec rate
>>> just iterating thanks to Resource.getAllContents().
>>> Hehe, this is a sign that client side caching works well:
>>> All views and transactions of a single session share (and cache) their
>>> object state. Otherwise I'd expect much smaller values.
>>> With his ObjectivityStore implementation Simon (cc'ed) reached rates
>>> of several thousands objects/sec. But the DBStore (which you used) is
>>> not that optimized.
>>>
>>>
>>> * Second case : I built a big model in the database using the complex
>>> metamodel. It contains 700 000 elements. The creation went pretty nicely
>>> (I did not measured it exactly as it was launched during a night, but it
>>> seems I got around 100 objects/sec rate in writting as I was writting
>>> every 700 objects.
>>> Sounds not too bad.
>>>
>>>
>>>
>>> Once the model is built the derby database is 222 MB (which - I heard -
>>> is not so big for derby).
>>> Hmm, I have no extensive experience with Derby.
>>> To be honest, I'm really no database expert (although I can write
>>> simple SQL statements).
>>>
>>>
>>> Retrieving the distant resource and calling eAllContents() on it, I get
>>> a 50 Objects/sec rate at best, which mean my eAllContents() call would
>>> take 4 hours !
>>> Currently I'm not sure what exactly eAllContents() does do (iteration
>>> order and so). There are several CDO features that can have a major
>>> impact on load rates!
>>>
>>> - Collections (xrefs and containment) can be loaded lazily. I don't
>>> mean the target objects, which are lazy by default anyway, but even
>>> the target IDs that are in the collections. We call that "reference
>>> chunking" (but search for a better name is ongoing).
>>>
>>> public interface CDOSession extends CDOProtocolSession,
>>> IContainer<CDOView>
>>> {
>>> public int getReferenceChunkSize();
>>>
>>> public void setReferenceChunkSize(int referenceChunkSize);
>>>
>>>
>>> - Loading collection elements (target objects) can be batched (the
>>> current name is even worse)
>>>
>>> public interface CDOView extends CDOProtocolView, INotifier
>>> {
>>> public int getLoadRevisionCollectionChunkSize();
>>>
>>> public void setLoadRevisionCollectionChunkSize(int
>>> loadRevisionCollectionChunkSize);
>>>
>>>
>>> - One of the most sophisticated feature is automatic model usage
>>> analyzation. The result are fetch rules that are sent to the server.
>>> When later objects are loaded from the server, it can answer with
>>> additional objects that are likely to be needed by the client in the
>>> near future.
>>>
>>> public class CDOViewImpl extends
>>> org.eclipse.net4j.internal.util.event.Notifier implements
>>> CDOView, CDOIDProvider, Adapter.Internal
>>> {
>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>> {
>>> return featureAnalyzer;
>>> }
>>>
>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>> featureAnalyzer)
>>> {
>>> this.featureAnalyzer = featureAnalyzer == null ?
>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>> }
>>>
>>>
>>> - There's a SmartReadAheadThread (if you're curious, please ask
>>> Simon). - Caches can be configured
>>> - ...
>>>
>>>
>>> Known issues of the DBStore:
>>> - When in auditing mode updates are always written as complete inserts
>>> (by design) - When configuring toManyReferences to a value different
>>> from LIKE_ATTRIBUTES (which is not implemented yet) it is very slow. -
>>> No (few) prepared statements are used (see
>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe
>>> others...
>>>
>>> Please note that the DBStore was not written by me to compete with
>>> Hibernate in terms of performance or mapping capabilities. Now that
>>> Martin and I, well in the end mostly Martin, provide a HibernateStore
>>> and this HibernateStore gets more and more features I expect that
>>> there will be a trend away from the DBStore.
>>>
>>>
>>> This is probably linked to the way the mapping to db is done, here's the
>>> config I used (came from CDO-server example if I recall correctly).
>>>
>>>
>>> <repository name="repo1">
>>> <property name="overrideUUID"
>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>> <property name="supportingAudits" value="true"/>
>>> <property name="verifyingRevisions" value="false"/>
>>> <property name="currentLRUCapacity" value="10000"/>
>>> I guess this one could be larger.
>>>
>>>
>>> <property name="revisedLRUCapacity" value="100"/>
>>>
>>> <store type="db">
>>> <!-- type: horizontal | vertical | <any
>>> user-contributed type> --> <mappingStrategy
>>> type="horizontal">
>>> <!-- ONE_TABLE_PER_REFERENCE |
>>> ONE_TABLE_PER_CLASS |
>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>>> <property name="toManyReferences"
>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>
>>> <!-- LIKE_ATTRIBUTES |
>>> LIKE_TO_MANY_REFERENCES --> <property
>>> name="toOneReferences"
>>> value="LIKE_ATTRIBUTES"/>
>>>
>>> <!-- MODEL | STRATEGY -->
>>> <property name="mappingPrecedence"
>>> value="MODEL"/>
>>> This one is obsolete.
>>>
>>>
>>> </mappingStrategy>
>>>
>>> <dbAdapter name="derby-embedded"/>
>>> <dataSource
>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>> databaseName="/tmp/cdodb1"
>>> createDatabase="create"/>
>>>
>>> </store>
>>> </repository>
>>>
>>>
>>>
>>>
>>> What do you think about that ? What throughput do you think I should
>>> expect ? As I said that's very difficult to judge.
>>> I believe it's more appropriate to compare the actual values with
>>> values from other measures (other setups, properties, etc. or even
>>> other products) instead of random expectations. Apologies that I
>>> didn't have the time so far to provide some reference measures myself.
>>> Maybe this gets better now that the team starts to grow slowly ;-)
>>>
>>>
>>> What are the main reasons fur such figures ?
>>> Hmm, I *suspect* you don't like the figures? I can find no evidence
>>> your assessment ;-)
>>>
>>> Ahh,one more note:
>>> Tracing can also have a significant impact on performance and
>>> end-to-end throughput. Turn it off by all means. And please make sure
>>> that it's really turned off, not only invisible (i.e. console is not a
>>> target).
>>>
>>> Cheers
>>> /Eike
>>
>
>


http://cedric.brun.io news and articles on eclipse and eclipse modeling.
Re: [CDO] reading/writting throughput performances [message #124164 is a reply to message #124150] Fri, 30 May 2008 13:21 Go to previous messageGo to next message
Martin Taal is currently offline Martin TaalFriend
Messages: 5468
Registered: July 2009
Senior Member
Hmm, I can understand your headache. Classpath issues are the most annoying errors...

In your junit test source file, what happens when you enter the org.objectweb.asm.Type, can you
source file see this class?
Do you have cglib in some other plugin which does not have asm?

What I also find slightly strange is that cglib is used in this case because normally cdo objects
are not proxied using cglib. Can you post the generated mapping?
How does your cdo initialization code look like?

gr. Martin

Cédric Brun wrote:
> Hi Martin,
>
> and thanks for your quick answer ! what I did is the following :
>
> I used the org.eclipse.emf.cdo.server.hibernate.libraries bundle and added
> that :
>
> Bundle-ClassPath: lib/hibernate3.jar,
> lib/cglib-2.1.3.jar,
> lib/commons-collections-2.1.1.jar,
> lib/dom4j.jar,
> lib/jta.jar
> lib/asm-3.0.jar
>
> and this directive :
>
> Eclipse-BuddyPolicy: dependent
>
> as stated in
> http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>
> Then I don't get the same error, now when I start my own plugin (JUnit
> plugin test) depending on org.eclipse.emf.cdo.server.hibernate.libraries I
> get :
>
> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
> at net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
> at net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
> at net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
> at
> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>
> These jars are such a headache ! Do you know - by chance - what's going
> on ?
>
>
> Cédric
>
>
> Martin Taal wrote:
>
>> Hi Cédric,
>> Most of the time this exception is thrown because hibernate can't find the
>> class. A common cause is that the hibernate jar files are located in a
>> separate plugin which can not reach the classes in other plugins. If you
>> set the eclipse buddy policy of the hibernate plugin to dependent then
>> this can help.
>>
>> gr. Martin
>>
>> Cédric Brun wrote:
>>> Hi Simon and Eike,
>>>
>>> thanks for your feedback, now I know what I could expect in using
>>> optimized implementation, I'll have a try on the hibernate store and see
>>> if it is scaling in a better way.
>>>
>>> You gave me many pointers to explain the performances issues but in fact
>>> it seems that the SQL execution is taking so much time (not suprisingly
>>> in fact ;) . My first profile was wrong.
>>>
>>> So I'm installing the hibernate store retrieving all the source code from
>>> CVS, and I'm trying to launch the unit tests. I added many jars and
>>> everything gets compiled but most of the tests are failling with "
>>>
>>> "
>>> org.hibernate.MappingException: Could not determine type for:
>>> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>> at table: cdo_features, for columns: [org.hibernate.mapping.Column(type)]
>>> "
>>>
>>> any hint ?
>>>
>>> Cédric
>>>
>>>
>>> Simon McDuff wrote:
>>>
>>>> We noticed a performance degradation in the following case:
>>>>
>>>> When we fetch a collection of objects that contains collections that we
>>>> don't want to use. Ex:
>>>>
>>>> Model:
>>>>
>>>> ClassB
>>>> {
>>>> int foo1;
>>>> List childrens1;
>>>> List childrens2;
>>>> List childrens3;
>>>> };
>>>>
>>>> Code:
>>>>
>>>> Resource.contents contains objects of type ClassB.
>>>>
>>>> By fetching instances of classB, we will also fetched some items in the
>>>> list(childrens1, childrens2, childrens3) The problem we had.. is we do
>>>> not know if we want to use it. SO better not to download them.
>>>>
>>>> So In our production(we used our own IStore), we never download any
>>>> objects from collection. I'm working at the moment to put in place all
>>>> feature we used in the current CDO. That one will be part of
>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477. To begin
>>>> with,CDORevisionImpl needs to supports list that are 0 even if
>>>> referenceChunkSize == 0.
>>>>
>>>> It is true that we obtain good performance.
>>>> Worst case :
>>>> 1500 objects /sec (Real fetch from disk. Page that contains the objects,
>>>> wasn't in the cache of the filesystem.)
>>>>
>>>> Best case
>>>> 12 000 objects/Sec (Fetch objects that are already in the case of the
>>>> filesystem)
>>>>
>>>> (These statistics always fetch objects from server to client. The cache
>>>> client isn't used)
>>>>
>>>> We have another optimization one with SmartReadAhead but for now only
>>>> work
>>>> for UI client. (for now) This is something we will integrate as well in
>>>> CDO. But for now we concentrate on the other things.
>>>>
>>>> I used an Object database. Based on my preliminary test without CDO.
>>>> I compare
>>>>
>>>> Hibernate vs Objectivity/EMF :
>>>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>>>
>>>> The worst.. you will get 30 to 240 objects/sec or
>>>> the best... you will get 500 to 4000 objects/sec
>>>>
>>>> Based on your comments... you are getting the worst (with Derby)!! :-)
>>>>
>>>>
>>>> Another comments about insertion.
>>>>
>>>> We found another problem when we insert data(DOn't know if it is still
>>>> true). Resource.Contents is a unique list. Basically each time we insert
>>>> one element, it needs to fetch all the collection to see if we already
>>>> have this element. Basically.. it is time consuming. In our case we are
>>>> not going through this stage to persist our objects... but the objects
>>>> are not accessible through that list.
>>>>
>>>>
>>>>
>>>> If speed is an issue for you!! We (You included :-)) can optimize it.
>>>> What I like is Eike did a really good job doing that framwork and
>>>> optimizations are easy to add. Also you can change your back-end without
>>>> affecting you client side. (Maybe going to an Object database!!) THis
>>>> was one of the purpose of CDO!! :-)
>>>>
>>>>
>>>> Simon
>>>>
>>>>
>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le message de news:
>>>> 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>
>>>> That sounds interesting!
>>>> Comments inline...
>>>>
>>>>
>>>> Cédric Brun schrieb:
>>>> Hi Eike,
>>>>
>>>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>>>> measuring read/write throughput when I use CDO (RC2).
>>>>
>>>> I'm using an embeded-derby database, two different metamodels (complex
>>>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>>>> Features)
>>>>
>>>> The test is a "quick and dirty" one so please do not take these figures
>>>> too seriously, I would just like to have your input about the "order of
>>>> magnitude" I should expect or settings affecting seriously these
>>>> performances. I tried both JVM and TCP acceptor but the performances are
>>>> quite similar though JVM is a bit faster.
>>>> Looks like a sign that the NIO socket channels are efficient ,-)
>>>> You used localhost for TCP transport I guess?
>>>>
>>>>
>>>>
>>>> * First case : building a small model (10000 elements) with the simple
>>>> MM, only adding instances in an empty database :
>>>> commiting change on every add I get a 2 objects per second rate.
>>>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>>> Seems to be consistent with the fact that only for commits there is
>>>> remote traffic. And the backend transaction management is another
>>>> overhead. So the impact of the number objects/commit starts with
>>>> bigger transactions.
>>>>
>>>>
>>>> browsing the just commited elements opening a new transaction and
>>>> getting a new Resource instance, I get around 30 000 objets/sec rate
>>>> just iterating thanks to Resource.getAllContents().
>>>> Hehe, this is a sign that client side caching works well:
>>>> All views and transactions of a single session share (and cache) their
>>>> object state. Otherwise I'd expect much smaller values.
>>>> With his ObjectivityStore implementation Simon (cc'ed) reached rates
>>>> of several thousands objects/sec. But the DBStore (which you used) is
>>>> not that optimized.
>>>>
>>>>
>>>> * Second case : I built a big model in the database using the complex
>>>> metamodel. It contains 700 000 elements. The creation went pretty nicely
>>>> (I did not measured it exactly as it was launched during a night, but it
>>>> seems I got around 100 objects/sec rate in writting as I was writting
>>>> every 700 objects.
>>>> Sounds not too bad.
>>>>
>>>>
>>>>
>>>> Once the model is built the derby database is 222 MB (which - I heard -
>>>> is not so big for derby).
>>>> Hmm, I have no extensive experience with Derby.
>>>> To be honest, I'm really no database expert (although I can write
>>>> simple SQL statements).
>>>>
>>>>
>>>> Retrieving the distant resource and calling eAllContents() on it, I get
>>>> a 50 Objects/sec rate at best, which mean my eAllContents() call would
>>>> take 4 hours !
>>>> Currently I'm not sure what exactly eAllContents() does do (iteration
>>>> order and so). There are several CDO features that can have a major
>>>> impact on load rates!
>>>>
>>>> - Collections (xrefs and containment) can be loaded lazily. I don't
>>>> mean the target objects, which are lazy by default anyway, but even
>>>> the target IDs that are in the collections. We call that "reference
>>>> chunking" (but search for a better name is ongoing).
>>>>
>>>> public interface CDOSession extends CDOProtocolSession,
>>>> IContainer<CDOView>
>>>> {
>>>> public int getReferenceChunkSize();
>>>>
>>>> public void setReferenceChunkSize(int referenceChunkSize);
>>>>
>>>>
>>>> - Loading collection elements (target objects) can be batched (the
>>>> current name is even worse)
>>>>
>>>> public interface CDOView extends CDOProtocolView, INotifier
>>>> {
>>>> public int getLoadRevisionCollectionChunkSize();
>>>>
>>>> public void setLoadRevisionCollectionChunkSize(int
>>>> loadRevisionCollectionChunkSize);
>>>>
>>>>
>>>> - One of the most sophisticated feature is automatic model usage
>>>> analyzation. The result are fetch rules that are sent to the server.
>>>> When later objects are loaded from the server, it can answer with
>>>> additional objects that are likely to be needed by the client in the
>>>> near future.
>>>>
>>>> public class CDOViewImpl extends
>>>> org.eclipse.net4j.internal.util.event.Notifier implements
>>>> CDOView, CDOIDProvider, Adapter.Internal
>>>> {
>>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>>> {
>>>> return featureAnalyzer;
>>>> }
>>>>
>>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>>> featureAnalyzer)
>>>> {
>>>> this.featureAnalyzer = featureAnalyzer == null ?
>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>> }
>>>>
>>>>
>>>> - There's a SmartReadAheadThread (if you're curious, please ask
>>>> Simon). - Caches can be configured
>>>> - ...
>>>>
>>>>
>>>> Known issues of the DBStore:
>>>> - When in auditing mode updates are always written as complete inserts
>>>> (by design) - When configuring toManyReferences to a value different
>>>> from LIKE_ATTRIBUTES (which is not implemented yet) it is very slow. -
>>>> No (few) prepared statements are used (see
>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe
>>>> others...
>>>>
>>>> Please note that the DBStore was not written by me to compete with
>>>> Hibernate in terms of performance or mapping capabilities. Now that
>>>> Martin and I, well in the end mostly Martin, provide a HibernateStore
>>>> and this HibernateStore gets more and more features I expect that
>>>> there will be a trend away from the DBStore.
>>>>
>>>>
>>>> This is probably linked to the way the mapping to db is done, here's the
>>>> config I used (came from CDO-server example if I recall correctly).
>>>>
>>>>
>>>> <repository name="repo1">
>>>> <property name="overrideUUID"
>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>> <property name="supportingAudits" value="true"/>
>>>> <property name="verifyingRevisions" value="false"/>
>>>> <property name="currentLRUCapacity" value="10000"/>
>>>> I guess this one could be larger.
>>>>
>>>>
>>>> <property name="revisedLRUCapacity" value="100"/>
>>>>
>>>> <store type="db">
>>>> <!-- type: horizontal | vertical | <any
>>>> user-contributed type> --> <mappingStrategy
>>>> type="horizontal">
>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>> ONE_TABLE_PER_CLASS |
>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>>>> <property name="toManyReferences"
>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>
>>>> <!-- LIKE_ATTRIBUTES |
>>>> LIKE_TO_MANY_REFERENCES --> <property
>>>> name="toOneReferences"
>>>> value="LIKE_ATTRIBUTES"/>
>>>>
>>>> <!-- MODEL | STRATEGY -->
>>>> <property name="mappingPrecedence"
>>>> value="MODEL"/>
>>>> This one is obsolete.
>>>>
>>>>
>>>> </mappingStrategy>
>>>>
>>>> <dbAdapter name="derby-embedded"/>
>>>> <dataSource
>>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>> databaseName="/tmp/cdodb1"
>>>> createDatabase="create"/>
>>>>
>>>> </store>
>>>> </repository>
>>>>
>>>>
>>>>
>>>>
>>>> What do you think about that ? What throughput do you think I should
>>>> expect ? As I said that's very difficult to judge.
>>>> I believe it's more appropriate to compare the actual values with
>>>> values from other measures (other setups, properties, etc. or even
>>>> other products) instead of random expectations. Apologies that I
>>>> didn't have the time so far to provide some reference measures myself.
>>>> Maybe this gets better now that the team starts to grow slowly ;-)
>>>>
>>>>
>>>> What are the main reasons fur such figures ?
>>>> Hmm, I *suspect* you don't like the figures? I can find no evidence
>>>> your assessment ;-)
>>>>
>>>> Ahh,one more note:
>>>> Tracing can also have a significant impact on performance and
>>>> end-to-end throughput. Turn it off by all means. And please make sure
>>>> that it's really turned off, not only invisible (i.e. console is not a
>>>> target).
>>>>
>>>> Cheers
>>>> /Eike
>>
>


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Cell: +31 (0)6 288 48 943
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: [CDO] reading/writting throughput performances [message #124231 is a reply to message #124164] Fri, 30 May 2008 14:34 Go to previous messageGo to next message
Cedric Brun is currently offline Cedric BrunFriend
Messages: 431
Registered: July 2009
Senior Member
I tried to do the same things as in the CDO/Hibernate unit tests.

Here's a zipped project trying to launch an HibernateServer with a unit test
adding elements.
( http://cedric.brun.free.fr/Tortoose/org.eclipse.cdo-teneo-te st.zip) The
libs I found are included in the same project, but I still have no luck and
failling at :

at
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
... 62 more
Caused by: java.lang.NoClassDefFoundError:
org/hibernate/proxy/HibernateProxy
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
... 68 more

please note that the test project I provide really is a "quick and ugly" one
^^


Cédric



Martin Taal wrote:

> Hmm, I can understand your headache. Classpath issues are the most
> annoying errors...
>
> In your junit test source file, what happens when you enter the
> org.objectweb.asm.Type, can you source file see this class?
> Do you have cglib in some other plugin which does not have asm?
>
> What I also find slightly strange is that cglib is used in this case
> because normally cdo objects are not proxied using cglib. Can you post the
> generated mapping? How does your cdo initialization code look like?
>
> gr. Martin
>
> Cédric Brun wrote:
>> Hi Martin,
>>
>> and thanks for your quick answer ! what I did is the following :
>>
>> I used the org.eclipse.emf.cdo.server.hibernate.libraries bundle and
>> added that :
>>
>> Bundle-ClassPath: lib/hibernate3.jar,
>> lib/cglib-2.1.3.jar,
>> lib/commons-collections-2.1.1.jar,
>> lib/dom4j.jar,
>> lib/jta.jar
>> lib/asm-3.0.jar
>>
>> and this directive :
>>
>> Eclipse-BuddyPolicy: dependent
>>
>> as stated in
>> http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>>
>> Then I don't get the same error, now when I start my own plugin (JUnit
>> plugin test) depending on org.eclipse.emf.cdo.server.hibernate.libraries
>> I get :
>>
>> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
>> at net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
>> at net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
>> at net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
>> at
>>
org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>>
>> These jars are such a headache ! Do you know - by chance - what's going
>> on ?
>>
>>
>> Cédric
>>
>>
>> Martin Taal wrote:
>>
>>> Hi Cédric,
>>> Most of the time this exception is thrown because hibernate can't find
>>> the class. A common cause is that the hibernate jar files are located in
>>> a separate plugin which can not reach the classes in other plugins. If
>>> you set the eclipse buddy policy of the hibernate plugin to dependent
>>> then this can help.
>>>
>>> gr. Martin
>>>
>>> Cédric Brun wrote:
>>>> Hi Simon and Eike,
>>>>
>>>> thanks for your feedback, now I know what I could expect in using
>>>> optimized implementation, I'll have a try on the hibernate store and
>>>> see if it is scaling in a better way.
>>>>
>>>> You gave me many pointers to explain the performances issues but in
>>>> fact it seems that the SQL execution is taking so much time (not
>>>> suprisingly in fact ;) . My first profile was wrong.
>>>>
>>>> So I'm installing the hibernate store retrieving all the source code
>>>> from CVS, and I'm trying to launch the unit tests. I added many jars
>>>> and everything gets compiled but most of the tests are failling with "
>>>>
>>>> "
>>>> org.hibernate.MappingException: Could not determine type for:
>>>> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>>> at table: cdo_features, for columns:
>>>> [org.hibernate.mapping.Column(type)] "
>>>>
>>>> any hint ?
>>>>
>>>> Cédric
>>>>
>>>>
>>>> Simon McDuff wrote:
>>>>
>>>>> We noticed a performance degradation in the following case:
>>>>>
>>>>> When we fetch a collection of objects that contains collections that
>>>>> we don't want to use. Ex:
>>>>>
>>>>> Model:
>>>>>
>>>>> ClassB
>>>>> {
>>>>> int foo1;
>>>>> List childrens1;
>>>>> List childrens2;
>>>>> List childrens3;
>>>>> };
>>>>>
>>>>> Code:
>>>>>
>>>>> Resource.contents contains objects of type ClassB.
>>>>>
>>>>> By fetching instances of classB, we will also fetched some items in
>>>>> the list(childrens1, childrens2, childrens3) The problem we had.. is
>>>>> we do not know if we want to use it. SO better not to download them.
>>>>>
>>>>> So In our production(we used our own IStore), we never download any
>>>>> objects from collection. I'm working at the moment to put in place all
>>>>> feature we used in the current CDO. That one will be part of
>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477. To begin
>>>>> with,CDORevisionImpl needs to supports list that are 0 even if
>>>>> referenceChunkSize == 0.
>>>>>
>>>>> It is true that we obtain good performance.
>>>>> Worst case :
>>>>> 1500 objects /sec (Real fetch from disk. Page that contains the
>>>>> objects, wasn't in the cache of the filesystem.)
>>>>>
>>>>> Best case
>>>>> 12 000 objects/Sec (Fetch objects that are already in the case of the
>>>>> filesystem)
>>>>>
>>>>> (These statistics always fetch objects from server to client. The
>>>>> cache client isn't used)
>>>>>
>>>>> We have another optimization one with SmartReadAhead but for now only
>>>>> work
>>>>> for UI client. (for now) This is something we will integrate as well
>>>>> in CDO. But for now we concentrate on the other things.
>>>>>
>>>>> I used an Object database. Based on my preliminary test without CDO.
>>>>> I compare
>>>>>
>>>>> Hibernate vs Objectivity/EMF :
>>>>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>>>>
>>>>> The worst.. you will get 30 to 240 objects/sec or
>>>>> the best... you will get 500 to 4000 objects/sec
>>>>>
>>>>> Based on your comments... you are getting the worst (with Derby)!! :-)
>>>>>
>>>>>
>>>>> Another comments about insertion.
>>>>>
>>>>> We found another problem when we insert data(DOn't know if it is still
>>>>> true). Resource.Contents is a unique list. Basically each time we
>>>>> insert one element, it needs to fetch all the collection to see if we
>>>>> already have this element. Basically.. it is time consuming. In our
>>>>> case we are not going through this stage to persist our objects... but
>>>>> the objects are not accessible through that list.
>>>>>
>>>>>
>>>>>
>>>>> If speed is an issue for you!! We (You included :-)) can optimize it.
>>>>> What I like is Eike did a really good job doing that framwork and
>>>>> optimizations are easy to add. Also you can change your back-end
>>>>> without affecting you client side. (Maybe going to an Object
>>>>> database!!) THis was one of the purpose of CDO!! :-)
>>>>>
>>>>>
>>>>> Simon
>>>>>
>>>>>
>>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le message de
>>>>> news: 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>>
>>>>> That sounds interesting!
>>>>> Comments inline...
>>>>>
>>>>>
>>>>> Cédric Brun schrieb:
>>>>> Hi Eike,
>>>>>
>>>>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>>>>> measuring read/write throughput when I use CDO (RC2).
>>>>>
>>>>> I'm using an embeded-derby database, two different metamodels (complex
>>>>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>>>>> Features)
>>>>>
>>>>> The test is a "quick and dirty" one so please do not take these
>>>>> figures too seriously, I would just like to have your input about the
>>>>> "order of magnitude" I should expect or settings affecting seriously
>>>>> these performances. I tried both JVM and TCP acceptor but the
>>>>> performances are quite similar though JVM is a bit faster.
>>>>> Looks like a sign that the NIO socket channels are efficient ,-)
>>>>> You used localhost for TCP transport I guess?
>>>>>
>>>>>
>>>>>
>>>>> * First case : building a small model (10000 elements) with the simple
>>>>> MM, only adding instances in an empty database :
>>>>> commiting change on every add I get a 2 objects per second rate.
>>>>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>>>> Seems to be consistent with the fact that only for commits there is
>>>>> remote traffic. And the backend transaction management is another
>>>>> overhead. So the impact of the number objects/commit starts with
>>>>> bigger transactions.
>>>>>
>>>>>
>>>>> browsing the just commited elements opening a new transaction and
>>>>> getting a new Resource instance, I get around 30 000 objets/sec rate
>>>>> just iterating thanks to Resource.getAllContents().
>>>>> Hehe, this is a sign that client side caching works well:
>>>>> All views and transactions of a single session share (and cache)
>>>>> their object state. Otherwise I'd expect much smaller values.
>>>>> With his ObjectivityStore implementation Simon (cc'ed) reached rates
>>>>> of several thousands objects/sec. But the DBStore (which you used)
>>>>> is not that optimized.
>>>>>
>>>>>
>>>>> * Second case : I built a big model in the database using the complex
>>>>> metamodel. It contains 700 000 elements. The creation went pretty
>>>>> nicely (I did not measured it exactly as it was launched during a
>>>>> night, but it seems I got around 100 objects/sec rate in writting as I
>>>>> was writting every 700 objects.
>>>>> Sounds not too bad.
>>>>>
>>>>>
>>>>>
>>>>> Once the model is built the derby database is 222 MB (which - I heard
>>>>> - is not so big for derby).
>>>>> Hmm, I have no extensive experience with Derby.
>>>>> To be honest, I'm really no database expert (although I can write
>>>>> simple SQL statements).
>>>>>
>>>>>
>>>>> Retrieving the distant resource and calling eAllContents() on it, I
>>>>> get a 50 Objects/sec rate at best, which mean my eAllContents() call
>>>>> would take 4 hours !
>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>> (iteration order and so). There are several CDO features that can
>>>>> have a major impact on load rates!
>>>>>
>>>>> - Collections (xrefs and containment) can be loaded lazily. I don't
>>>>> mean the target objects, which are lazy by default anyway, but even
>>>>> the target IDs that are in the collections. We call that "reference
>>>>> chunking" (but search for a better name is ongoing).
>>>>>
>>>>> public interface CDOSession extends CDOProtocolSession,
>>>>> IContainer<CDOView>
>>>>> {
>>>>> public int getReferenceChunkSize();
>>>>>
>>>>> public void setReferenceChunkSize(int referenceChunkSize);
>>>>>
>>>>>
>>>>> - Loading collection elements (target objects) can be batched (the
>>>>> current name is even worse)
>>>>>
>>>>> public interface CDOView extends CDOProtocolView, INotifier
>>>>> {
>>>>> public int getLoadRevisionCollectionChunkSize();
>>>>>
>>>>> public void setLoadRevisionCollectionChunkSize(int
>>>>> loadRevisionCollectionChunkSize);
>>>>>
>>>>>
>>>>> - One of the most sophisticated feature is automatic model usage
>>>>> analyzation. The result are fetch rules that are sent to the server.
>>>>> When later objects are loaded from the server, it can answer with
>>>>> additional objects that are likely to be needed by the client in the
>>>>> near future.
>>>>>
>>>>> public class CDOViewImpl extends
>>>>> org.eclipse.net4j.internal.util.event.Notifier implements
>>>>> CDOView, CDOIDProvider, Adapter.Internal
>>>>> {
>>>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>> {
>>>>> return featureAnalyzer;
>>>>> }
>>>>>
>>>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>> featureAnalyzer)
>>>>> {
>>>>> this.featureAnalyzer = featureAnalyzer == null ?
>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>> }
>>>>>
>>>>>
>>>>> - There's a SmartReadAheadThread (if you're curious, please ask
>>>>> Simon). - Caches can be configured
>>>>> - ...
>>>>>
>>>>>
>>>>> Known issues of the DBStore:
>>>>> - When in auditing mode updates are always written as complete
>>>>> inserts (by design) - When configuring toManyReferences to a value
>>>>> different from LIKE_ATTRIBUTES (which is not implemented yet) it is
>>>>> very slow. - No (few) prepared statements are used (see
>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe
>>>>> others...
>>>>>
>>>>> Please note that the DBStore was not written by me to compete with
>>>>> Hibernate in terms of performance or mapping capabilities. Now that
>>>>> Martin and I, well in the end mostly Martin, provide a
>>>>> HibernateStore and this HibernateStore gets more and more features I
>>>>> expect that there will be a trend away from the DBStore.
>>>>>
>>>>>
>>>>> This is probably linked to the way the mapping to db is done, here's
>>>>> the config I used (came from CDO-server example if I recall
>>>>> correctly).
>>>>>
>>>>>
>>>>> <repository name="repo1">
>>>>> <property name="overrideUUID"
>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>> <property name="supportingAudits" value="true"/>
>>>>> <property name="verifyingRevisions" value="false"/>
>>>>> <property name="currentLRUCapacity" value="10000"/>
>>>>> I guess this one could be larger.
>>>>>
>>>>>
>>>>> <property name="revisedLRUCapacity" value="100"/>
>>>>>
>>>>> <store type="db">
>>>>> <!-- type: horizontal | vertical | <any
>>>>> user-contributed type> --> <mappingStrategy
>>>>> type="horizontal">
>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>> ONE_TABLE_PER_CLASS |
>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>>>>> <property name="toManyReferences"
>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>
>>>>> <!-- LIKE_ATTRIBUTES |
>>>>> LIKE_TO_MANY_REFERENCES --> <property
>>>>> name="toOneReferences"
>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>
>>>>> <!-- MODEL | STRATEGY -->
>>>>> <property name="mappingPrecedence"
>>>>> value="MODEL"/>
>>>>> This one is obsolete.
>>>>>
>>>>>
>>>>> </mappingStrategy>
>>>>>
>>>>> <dbAdapter name="derby-embedded"/>
>>>>> <dataSource
>>>>>
class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>>> databaseName="/tmp/cdodb1"
>>>>> createDatabase="create"/>
>>>>>
>>>>> </store>
>>>>> </repository>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> What do you think about that ? What throughput do you think I should
>>>>> expect ? As I said that's very difficult to judge.
>>>>> I believe it's more appropriate to compare the actual values with
>>>>> values from other measures (other setups, properties, etc. or even
>>>>> other products) instead of random expectations. Apologies that I
>>>>> didn't have the time so far to provide some reference measures
>>>>> myself. Maybe this gets better now that the team starts to grow
>>>>> slowly ;-)
>>>>>
>>>>>
>>>>> What are the main reasons fur such figures ?
>>>>> Hmm, I *suspect* you don't like the figures? I can find no evidence
>>>>> your assessment ;-)
>>>>>
>>>>> Ahh,one more note:
>>>>> Tracing can also have a significant impact on performance and
>>>>> end-to-end throughput. Turn it off by all means. And please make
>>>>> sure that it's really turned off, not only invisible (i.e. console
>>>>> is not a target).
>>>>>
>>>>> Cheers
>>>>> /Eike
>>>
>>
>
>


http://cedric.brun.io news and articles on eclipse and eclipse modeling.
Re: [CDO] reading/writting throughput performances [message #124246 is a reply to message #124231] Fri, 30 May 2008 14:30 Go to previous messageGo to next message
Martin Taal is currently offline Martin TaalFriend
Messages: 5468
Registered: July 2009
Senior Member
I am downloading your zip-file. I don't have time to look at it right away but I will do my best
later this evening.

gr. Martin

Cédric Brun wrote:
> I tried to do the same things as in the CDO/Hibernate unit tests.
>
> Here's a zipped project trying to launch an HibernateServer with a unit test
> adding elements.
> ( http://cedric.brun.free.fr/Tortoose/org.eclipse.cdo-teneo-te st.zip) The
> libs I found are included in the same project, but I still have no luck and
> failling at :
>
> at
> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
> ... 62 more
> Caused by: java.lang.NoClassDefFoundError:
> org/hibernate/proxy/HibernateProxy
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
> ... 68 more
>
> please note that the test project I provide really is a "quick and ugly" one
> ^^
>
>
> Cédric
>
>
>
> Martin Taal wrote:
>
>> Hmm, I can understand your headache. Classpath issues are the most
>> annoying errors...
>>
>> In your junit test source file, what happens when you enter the
>> org.objectweb.asm.Type, can you source file see this class?
>> Do you have cglib in some other plugin which does not have asm?
>>
>> What I also find slightly strange is that cglib is used in this case
>> because normally cdo objects are not proxied using cglib. Can you post the
>> generated mapping? How does your cdo initialization code look like?
>>
>> gr. Martin
>>
>> Cédric Brun wrote:
>>> Hi Martin,
>>>
>>> and thanks for your quick answer ! what I did is the following :
>>>
>>> I used the org.eclipse.emf.cdo.server.hibernate.libraries bundle and
>>> added that :
>>>
>>> Bundle-ClassPath: lib/hibernate3.jar,
>>> lib/cglib-2.1.3.jar,
>>> lib/commons-collections-2.1.1.jar,
>>> lib/dom4j.jar,
>>> lib/jta.jar
>>> lib/asm-3.0.jar
>>>
>>> and this directive :
>>>
>>> Eclipse-BuddyPolicy: dependent
>>>
>>> as stated in
>>> http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>>>
>>> Then I don't get the same error, now when I start my own plugin (JUnit
>>> plugin test) depending on org.eclipse.emf.cdo.server.hibernate.libraries
>>> I get :
>>>
>>> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
>>> at net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
>>> at net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
>>> at net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
>>> at
>>>
> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>>> These jars are such a headache ! Do you know - by chance - what's going
>>> on ?
>>>
>>>
>>> Cédric
>>>
>>>
>>> Martin Taal wrote:
>>>
>>>> Hi Cédric,
>>>> Most of the time this exception is thrown because hibernate can't find
>>>> the class. A common cause is that the hibernate jar files are located in
>>>> a separate plugin which can not reach the classes in other plugins. If
>>>> you set the eclipse buddy policy of the hibernate plugin to dependent
>>>> then this can help.
>>>>
>>>> gr. Martin
>>>>
>>>> Cédric Brun wrote:
>>>>> Hi Simon and Eike,
>>>>>
>>>>> thanks for your feedback, now I know what I could expect in using
>>>>> optimized implementation, I'll have a try on the hibernate store and
>>>>> see if it is scaling in a better way.
>>>>>
>>>>> You gave me many pointers to explain the performances issues but in
>>>>> fact it seems that the SQL execution is taking so much time (not
>>>>> suprisingly in fact ;) . My first profile was wrong.
>>>>>
>>>>> So I'm installing the hibernate store retrieving all the source code
>>>>> from CVS, and I'm trying to launch the unit tests. I added many jars
>>>>> and everything gets compiled but most of the tests are failling with "
>>>>>
>>>>> "
>>>>> org.hibernate.MappingException: Could not determine type for:
>>>>> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>>>> at table: cdo_features, for columns:
>>>>> [org.hibernate.mapping.Column(type)] "
>>>>>
>>>>> any hint ?
>>>>>
>>>>> Cédric
>>>>>
>>>>>
>>>>> Simon McDuff wrote:
>>>>>
>>>>>> We noticed a performance degradation in the following case:
>>>>>>
>>>>>> When we fetch a collection of objects that contains collections that
>>>>>> we don't want to use. Ex:
>>>>>>
>>>>>> Model:
>>>>>>
>>>>>> ClassB
>>>>>> {
>>>>>> int foo1;
>>>>>> List childrens1;
>>>>>> List childrens2;
>>>>>> List childrens3;
>>>>>> };
>>>>>>
>>>>>> Code:
>>>>>>
>>>>>> Resource.contents contains objects of type ClassB.
>>>>>>
>>>>>> By fetching instances of classB, we will also fetched some items in
>>>>>> the list(childrens1, childrens2, childrens3) The problem we had.. is
>>>>>> we do not know if we want to use it. SO better not to download them.
>>>>>>
>>>>>> So In our production(we used our own IStore), we never download any
>>>>>> objects from collection. I'm working at the moment to put in place all
>>>>>> feature we used in the current CDO. That one will be part of
>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477. To begin
>>>>>> with,CDORevisionImpl needs to supports list that are 0 even if
>>>>>> referenceChunkSize == 0.
>>>>>>
>>>>>> It is true that we obtain good performance.
>>>>>> Worst case :
>>>>>> 1500 objects /sec (Real fetch from disk. Page that contains the
>>>>>> objects, wasn't in the cache of the filesystem.)
>>>>>>
>>>>>> Best case
>>>>>> 12 000 objects/Sec (Fetch objects that are already in the case of the
>>>>>> filesystem)
>>>>>>
>>>>>> (These statistics always fetch objects from server to client. The
>>>>>> cache client isn't used)
>>>>>>
>>>>>> We have another optimization one with SmartReadAhead but for now only
>>>>>> work
>>>>>> for UI client. (for now) This is something we will integrate as well
>>>>>> in CDO. But for now we concentrate on the other things.
>>>>>>
>>>>>> I used an Object database. Based on my preliminary test without CDO.
>>>>>> I compare
>>>>>>
>>>>>> Hibernate vs Objectivity/EMF :
>>>>>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>>>>>
>>>>>> The worst.. you will get 30 to 240 objects/sec or
>>>>>> the best... you will get 500 to 4000 objects/sec
>>>>>>
>>>>>> Based on your comments... you are getting the worst (with Derby)!! :-)
>>>>>>
>>>>>>
>>>>>> Another comments about insertion.
>>>>>>
>>>>>> We found another problem when we insert data(DOn't know if it is still
>>>>>> true). Resource.Contents is a unique list. Basically each time we
>>>>>> insert one element, it needs to fetch all the collection to see if we
>>>>>> already have this element. Basically.. it is time consuming. In our
>>>>>> case we are not going through this stage to persist our objects... but
>>>>>> the objects are not accessible through that list.
>>>>>>
>>>>>>
>>>>>>
>>>>>> If speed is an issue for you!! We (You included :-)) can optimize it.
>>>>>> What I like is Eike did a really good job doing that framwork and
>>>>>> optimizations are easy to add. Also you can change your back-end
>>>>>> without affecting you client side. (Maybe going to an Object
>>>>>> database!!) THis was one of the purpose of CDO!! :-)
>>>>>>
>>>>>>
>>>>>> Simon
>>>>>>
>>>>>>
>>>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le message de
>>>>>> news: 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>>>
>>>>>> That sounds interesting!
>>>>>> Comments inline...
>>>>>>
>>>>>>
>>>>>> Cédric Brun schrieb:
>>>>>> Hi Eike,
>>>>>>
>>>>>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>>>>>> measuring read/write throughput when I use CDO (RC2).
>>>>>>
>>>>>> I'm using an embeded-derby database, two different metamodels (complex
>>>>>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>>>>>> Features)
>>>>>>
>>>>>> The test is a "quick and dirty" one so please do not take these
>>>>>> figures too seriously, I would just like to have your input about the
>>>>>> "order of magnitude" I should expect or settings affecting seriously
>>>>>> these performances. I tried both JVM and TCP acceptor but the
>>>>>> performances are quite similar though JVM is a bit faster.
>>>>>> Looks like a sign that the NIO socket channels are efficient ,-)
>>>>>> You used localhost for TCP transport I guess?
>>>>>>
>>>>>>
>>>>>>
>>>>>> * First case : building a small model (10000 elements) with the simple
>>>>>> MM, only adding instances in an empty database :
>>>>>> commiting change on every add I get a 2 objects per second rate.
>>>>>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>>>>> Seems to be consistent with the fact that only for commits there is
>>>>>> remote traffic. And the backend transaction management is another
>>>>>> overhead. So the impact of the number objects/commit starts with
>>>>>> bigger transactions.
>>>>>>
>>>>>>
>>>>>> browsing the just commited elements opening a new transaction and
>>>>>> getting a new Resource instance, I get around 30 000 objets/sec rate
>>>>>> just iterating thanks to Resource.getAllContents().
>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>> All views and transactions of a single session share (and cache)
>>>>>> their object state. Otherwise I'd expect much smaller values.
>>>>>> With his ObjectivityStore implementation Simon (cc'ed) reached rates
>>>>>> of several thousands objects/sec. But the DBStore (which you used)
>>>>>> is not that optimized.
>>>>>>
>>>>>>
>>>>>> * Second case : I built a big model in the database using the complex
>>>>>> metamodel. It contains 700 000 elements. The creation went pretty
>>>>>> nicely (I did not measured it exactly as it was launched during a
>>>>>> night, but it seems I got around 100 objects/sec rate in writting as I
>>>>>> was writting every 700 objects.
>>>>>> Sounds not too bad.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Once the model is built the derby database is 222 MB (which - I heard
>>>>>> - is not so big for derby).
>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>> To be honest, I'm really no database expert (although I can write
>>>>>> simple SQL statements).
>>>>>>
>>>>>>
>>>>>> Retrieving the distant resource and calling eAllContents() on it, I
>>>>>> get a 50 Objects/sec rate at best, which mean my eAllContents() call
>>>>>> would take 4 hours !
>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>> (iteration order and so). There are several CDO features that can
>>>>>> have a major impact on load rates!
>>>>>>
>>>>>> - Collections (xrefs and containment) can be loaded lazily. I don't
>>>>>> mean the target objects, which are lazy by default anyway, but even
>>>>>> the target IDs that are in the collections. We call that "reference
>>>>>> chunking" (but search for a better name is ongoing).
>>>>>>
>>>>>> public interface CDOSession extends CDOProtocolSession,
>>>>>> IContainer<CDOView>
>>>>>> {
>>>>>> public int getReferenceChunkSize();
>>>>>>
>>>>>> public void setReferenceChunkSize(int referenceChunkSize);
>>>>>>
>>>>>>
>>>>>> - Loading collection elements (target objects) can be batched (the
>>>>>> current name is even worse)
>>>>>>
>>>>>> public interface CDOView extends CDOProtocolView, INotifier
>>>>>> {
>>>>>> public int getLoadRevisionCollectionChunkSize();
>>>>>>
>>>>>> public void setLoadRevisionCollectionChunkSize(int
>>>>>> loadRevisionCollectionChunkSize);
>>>>>>
>>>>>>
>>>>>> - One of the most sophisticated feature is automatic model usage
>>>>>> analyzation. The result are fetch rules that are sent to the server.
>>>>>> When later objects are loaded from the server, it can answer with
>>>>>> additional objects that are likely to be needed by the client in the
>>>>>> near future.
>>>>>>
>>>>>> public class CDOViewImpl extends
>>>>>> org.eclipse.net4j.internal.util.event.Notifier implements
>>>>>> CDOView, CDOIDProvider, Adapter.Internal
>>>>>> {
>>>>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>>> {
>>>>>> return featureAnalyzer;
>>>>>> }
>>>>>>
>>>>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>>> featureAnalyzer)
>>>>>> {
>>>>>> this.featureAnalyzer = featureAnalyzer == null ?
>>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>>> }
>>>>>>
>>>>>>
>>>>>> - There's a SmartReadAheadThread (if you're curious, please ask
>>>>>> Simon). - Caches can be configured
>>>>>> - ...
>>>>>>
>>>>>>
>>>>>> Known issues of the DBStore:
>>>>>> - When in auditing mode updates are always written as complete
>>>>>> inserts (by design) - When configuring toManyReferences to a value
>>>>>> different from LIKE_ATTRIBUTES (which is not implemented yet) it is
>>>>>> very slow. - No (few) prepared statements are used (see
>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe
>>>>>> others...
>>>>>>
>>>>>> Please note that the DBStore was not written by me to compete with
>>>>>> Hibernate in terms of performance or mapping capabilities. Now that
>>>>>> Martin and I, well in the end mostly Martin, provide a
>>>>>> HibernateStore and this HibernateStore gets more and more features I
>>>>>> expect that there will be a trend away from the DBStore.
>>>>>>
>>>>>>
>>>>>> This is probably linked to the way the mapping to db is done, here's
>>>>>> the config I used (came from CDO-server example if I recall
>>>>>> correctly).
>>>>>>
>>>>>>
>>>>>> <repository name="repo1">
>>>>>> <property name="overrideUUID"
>>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>>> <property name="supportingAudits" value="true"/>
>>>>>> <property name="verifyingRevisions" value="false"/>
>>>>>> <property name="currentLRUCapacity" value="10000"/>
>>>>>> I guess this one could be larger.
>>>>>>
>>>>>>
>>>>>> <property name="revisedLRUCapacity" value="100"/>
>>>>>>
>>>>>> <store type="db">
>>>>>> <!-- type: horizontal | vertical | <any
>>>>>> user-contributed type> --> <mappingStrategy
>>>>>> type="horizontal">
>>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>>> ONE_TABLE_PER_CLASS |
>>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>>>>>> <property name="toManyReferences"
>>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>>
>>>>>> <!-- LIKE_ATTRIBUTES |
>>>>>> LIKE_TO_MANY_REFERENCES --> <property
>>>>>> name="toOneReferences"
>>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>>
>>>>>> <!-- MODEL | STRATEGY -->
>>>>>> <property name="mappingPrecedence"
>>>>>> value="MODEL"/>
>>>>>> This one is obsolete.
>>>>>>
>>>>>>
>>>>>> </mappingStrategy>
>>>>>>
>>>>>> <dbAdapter name="derby-embedded"/>
>>>>>> <dataSource
>>>>>>
> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>>>> databaseName="/tmp/cdodb1"
>>>>>> createDatabase="create"/>
>>>>>>
>>>>>> </store>
>>>>>> </repository>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> What do you think about that ? What throughput do you think I should
>>>>>> expect ? As I said that's very difficult to judge.
>>>>>> I believe it's more appropriate to compare the actual values with
>>>>>> values from other measures (other setups, properties, etc. or even
>>>>>> other products) instead of random expectations. Apologies that I
>>>>>> didn't have the time so far to provide some reference measures
>>>>>> myself. Maybe this gets better now that the team starts to grow
>>>>>> slowly ;-)
>>>>>>
>>>>>>
>>>>>> What are the main reasons fur such figures ?
>>>>>> Hmm, I *suspect* you don't like the figures? I can find no evidence
>>>>>> your assessment ;-)
>>>>>>
>>>>>> Ahh,one more note:
>>>>>> Tracing can also have a significant impact on performance and
>>>>>> end-to-end throughput. Turn it off by all means. And please make
>>>>>> sure that it's really turned off, not only invisible (i.e. console
>>>>>> is not a target).
>>>>>>
>>>>>> Cheers
>>>>>> /Eike
>>
>


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Cell: +31 (0)6 288 48 943
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: [CDO] reading/writting throughput performances [message #124260 is a reply to message #124246] Fri, 30 May 2008 14:52 Go to previous messageGo to next message
Cedric Brun is currently offline Cedric BrunFriend
Messages: 431
Registered: July 2009
Senior Member
Thanks a lot ! I'm looking forward to your advices !

Cédric

Martin Taal wrote:

> I am downloading your zip-file. I don't have time to look at it right away
> but I will do my best later this evening.
>
> gr. Martin
>
> Cédric Brun wrote:
>> I tried to do the same things as in the CDO/Hibernate unit tests.
>>
>> Here's a zipped project trying to launch an HibernateServer with a unit
>> test adding elements.
>> ( http://cedric.brun.free.fr/Tortoose/org.eclipse.cdo-teneo-te st.zip) The
>> libs I found are included in the same project, but I still have no luck
>> and failling at :
>>
>> at
>>
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>> ... 62 more
>> Caused by: java.lang.NoClassDefFoundError:
>> org/hibernate/proxy/HibernateProxy
>> at java.lang.ClassLoader.defineClass1(Native Method)
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>> ... 68 more
>>
>> please note that the test project I provide really is a "quick and ugly"
>> one ^^
>>
>>
>> Cédric
>>
>>
>>
>> Martin Taal wrote:
>>
>>> Hmm, I can understand your headache. Classpath issues are the most
>>> annoying errors...
>>>
>>> In your junit test source file, what happens when you enter the
>>> org.objectweb.asm.Type, can you source file see this class?
>>> Do you have cglib in some other plugin which does not have asm?
>>>
>>> What I also find slightly strange is that cglib is used in this case
>>> because normally cdo objects are not proxied using cglib. Can you post
>>> the generated mapping? How does your cdo initialization code look like?
>>>
>>> gr. Martin
>>>
>>> Cédric Brun wrote:
>>>> Hi Martin,
>>>>
>>>> and thanks for your quick answer ! what I did is the following :
>>>>
>>>> I used the org.eclipse.emf.cdo.server.hibernate.libraries bundle and
>>>> added that :
>>>>
>>>> Bundle-ClassPath: lib/hibernate3.jar,
>>>> lib/cglib-2.1.3.jar,
>>>> lib/commons-collections-2.1.1.jar,
>>>> lib/dom4j.jar,
>>>> lib/jta.jar
>>>> lib/asm-3.0.jar
>>>>
>>>> and this directive :
>>>>
>>>> Eclipse-BuddyPolicy: dependent
>>>>
>>>> as stated in
>>>> http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>>>>
>>>> Then I don't get the same error, now when I start my own plugin (JUnit
>>>> plugin test) depending on
>>>> org.eclipse.emf.cdo.server.hibernate.libraries I get :
>>>>
>>>> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
>>>> at net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
>>>> at net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
>>>> at net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
>>>> at
>>>>
>>
org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>>>> These jars are such a headache ! Do you know - by chance - what's
>>>> going on ?
>>>>
>>>>
>>>> Cédric
>>>>
>>>>
>>>> Martin Taal wrote:
>>>>
>>>>> Hi Cédric,
>>>>> Most of the time this exception is thrown because hibernate can't find
>>>>> the class. A common cause is that the hibernate jar files are located
>>>>> in a separate plugin which can not reach the classes in other plugins.
>>>>> If you set the eclipse buddy policy of the hibernate plugin to
>>>>> dependent then this can help.
>>>>>
>>>>> gr. Martin
>>>>>
>>>>> Cédric Brun wrote:
>>>>>> Hi Simon and Eike,
>>>>>>
>>>>>> thanks for your feedback, now I know what I could expect in using
>>>>>> optimized implementation, I'll have a try on the hibernate store and
>>>>>> see if it is scaling in a better way.
>>>>>>
>>>>>> You gave me many pointers to explain the performances issues but in
>>>>>> fact it seems that the SQL execution is taking so much time (not
>>>>>> suprisingly in fact ;) . My first profile was wrong.
>>>>>>
>>>>>> So I'm installing the hibernate store retrieving all the source code
>>>>>> from CVS, and I'm trying to launch the unit tests. I added many jars
>>>>>> and everything gets compiled but most of the tests are failling with
>>>>>> "
>>>>>>
>>>>>> "
>>>>>> org.hibernate.MappingException: Could not determine type for:
>>>>>>
org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>>>>> at table: cdo_features, for columns:
>>>>>> [org.hibernate.mapping.Column(type)] "
>>>>>>
>>>>>> any hint ?
>>>>>>
>>>>>> Cédric
>>>>>>
>>>>>>
>>>>>> Simon McDuff wrote:
>>>>>>
>>>>>>> We noticed a performance degradation in the following case:
>>>>>>>
>>>>>>> When we fetch a collection of objects that contains collections that
>>>>>>> we don't want to use. Ex:
>>>>>>>
>>>>>>> Model:
>>>>>>>
>>>>>>> ClassB
>>>>>>> {
>>>>>>> int foo1;
>>>>>>> List childrens1;
>>>>>>> List childrens2;
>>>>>>> List childrens3;
>>>>>>> };
>>>>>>>
>>>>>>> Code:
>>>>>>>
>>>>>>> Resource.contents contains objects of type ClassB.
>>>>>>>
>>>>>>> By fetching instances of classB, we will also fetched some items in
>>>>>>> the list(childrens1, childrens2, childrens3) The problem we had.. is
>>>>>>> we do not know if we want to use it. SO better not to download them.
>>>>>>>
>>>>>>> So In our production(we used our own IStore), we never download any
>>>>>>> objects from collection. I'm working at the moment to put in place
>>>>>>> all feature we used in the current CDO. That one will be part of
>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477. To begin
>>>>>>> with,CDORevisionImpl needs to supports list that are 0 even if
>>>>>>> referenceChunkSize == 0.
>>>>>>>
>>>>>>> It is true that we obtain good performance.
>>>>>>> Worst case :
>>>>>>> 1500 objects /sec (Real fetch from disk. Page that contains the
>>>>>>> objects, wasn't in the cache of the filesystem.)
>>>>>>>
>>>>>>> Best case
>>>>>>> 12 000 objects/Sec (Fetch objects that are already in the case of
>>>>>>> the filesystem)
>>>>>>>
>>>>>>> (These statistics always fetch objects from server to client. The
>>>>>>> cache client isn't used)
>>>>>>>
>>>>>>> We have another optimization one with SmartReadAhead but for now
>>>>>>> only work
>>>>>>> for UI client. (for now) This is something we will integrate as
>>>>>>> well in CDO. But for now we concentrate on the other things.
>>>>>>>
>>>>>>> I used an Object database. Based on my preliminary test without CDO.
>>>>>>> I compare
>>>>>>>
>>>>>>> Hibernate vs Objectivity/EMF :
>>>>>>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>>>>>>
>>>>>>> The worst.. you will get 30 to 240 objects/sec or
>>>>>>> the best... you will get 500 to 4000 objects/sec
>>>>>>>
>>>>>>> Based on your comments... you are getting the worst (with Derby)!!
>>>>>>> :-)
>>>>>>>
>>>>>>>
>>>>>>> Another comments about insertion.
>>>>>>>
>>>>>>> We found another problem when we insert data(DOn't know if it is
>>>>>>> still true). Resource.Contents is a unique list. Basically each time
>>>>>>> we insert one element, it needs to fetch all the collection to see
>>>>>>> if we already have this element. Basically.. it is time consuming.
>>>>>>> In our case we are not going through this stage to persist our
>>>>>>> objects... but the objects are not accessible through that list.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> If speed is an issue for you!! We (You included :-)) can optimize
>>>>>>> it. What I like is Eike did a really good job doing that framwork
>>>>>>> and optimizations are easy to add. Also you can change your back-end
>>>>>>> without affecting you client side. (Maybe going to an Object
>>>>>>> database!!) THis was one of the purpose of CDO!! :-)
>>>>>>>
>>>>>>>
>>>>>>> Simon
>>>>>>>
>>>>>>>
>>>>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le message de
>>>>>>> news: 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>>>>
>>>>>>> That sounds interesting!
>>>>>>> Comments inline...
>>>>>>>
>>>>>>>
>>>>>>> Cédric Brun schrieb:
>>>>>>> Hi Eike,
>>>>>>>
>>>>>>> I've been messing around CDO lately and I hacked a few ugly
>>>>>>> unit-tests measuring read/write throughput when I use CDO (RC2).
>>>>>>>
>>>>>>> I'm using an embeded-derby database, two different metamodels
>>>>>>> (complex one - 250 EClassifiers/300 features and a simple one - 1
>>>>>>> Classifier/3 Features)
>>>>>>>
>>>>>>> The test is a "quick and dirty" one so please do not take these
>>>>>>> figures too seriously, I would just like to have your input about
>>>>>>> the "order of magnitude" I should expect or settings affecting
>>>>>>> seriously these performances. I tried both JVM and TCP acceptor but
>>>>>>> the performances are quite similar though JVM is a bit faster.
>>>>>>> Looks like a sign that the NIO socket channels are efficient ,-)
>>>>>>> You used localhost for TCP transport I guess?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> * First case : building a small model (10000 elements) with the
>>>>>>> simple MM, only adding instances in an empty database :
>>>>>>> commiting change on every add I get a 2 objects per second rate.
>>>>>>> commiting changes every 10 adds - I get a 30 objects per second
>>>>>>> rate.
>>>>>>> Seems to be consistent with the fact that only for commits there
>>>>>>> is remote traffic. And the backend transaction management is
>>>>>>> another overhead. So the impact of the number objects/commit
>>>>>>> starts with bigger transactions.
>>>>>>>
>>>>>>>
>>>>>>> browsing the just commited elements opening a new transaction and
>>>>>>> getting a new Resource instance, I get around 30 000 objets/sec rate
>>>>>>> just iterating thanks to Resource.getAllContents().
>>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>>> All views and transactions of a single session share (and cache)
>>>>>>> their object state. Otherwise I'd expect much smaller values.
>>>>>>> With his ObjectivityStore implementation Simon (cc'ed) reached
>>>>>>> rates of several thousands objects/sec. But the DBStore (which you
>>>>>>> used) is not that optimized.
>>>>>>>
>>>>>>>
>>>>>>> * Second case : I built a big model in the database using the
>>>>>>> complex metamodel. It contains 700 000 elements. The creation went
>>>>>>> pretty nicely (I did not measured it exactly as it was launched
>>>>>>> during a night, but it seems I got around 100 objects/sec rate in
>>>>>>> writting as I was writting every 700 objects.
>>>>>>> Sounds not too bad.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Once the model is built the derby database is 222 MB (which - I
>>>>>>> heard - is not so big for derby).
>>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>>> To be honest, I'm really no database expert (although I can write
>>>>>>> simple SQL statements).
>>>>>>>
>>>>>>>
>>>>>>> Retrieving the distant resource and calling eAllContents() on it, I
>>>>>>> get a 50 Objects/sec rate at best, which mean my eAllContents() call
>>>>>>> would take 4 hours !
>>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>>> (iteration order and so). There are several CDO features that can
>>>>>>> have a major impact on load rates!
>>>>>>>
>>>>>>> - Collections (xrefs and containment) can be loaded lazily. I
>>>>>>> don't mean the target objects, which are lazy by default anyway,
>>>>>>> but even the target IDs that are in the collections. We call that
>>>>>>> "reference chunking" (but search for a better name is ongoing).
>>>>>>>
>>>>>>> public interface CDOSession extends CDOProtocolSession,
>>>>>>> IContainer<CDOView>
>>>>>>> {
>>>>>>> public int getReferenceChunkSize();
>>>>>>>
>>>>>>> public void setReferenceChunkSize(int referenceChunkSize);
>>>>>>>
>>>>>>>
>>>>>>> - Loading collection elements (target objects) can be batched (the
>>>>>>> current name is even worse)
>>>>>>>
>>>>>>> public interface CDOView extends CDOProtocolView, INotifier
>>>>>>> {
>>>>>>> public int getLoadRevisionCollectionChunkSize();
>>>>>>>
>>>>>>> public void setLoadRevisionCollectionChunkSize(int
>>>>>>> loadRevisionCollectionChunkSize);
>>>>>>>
>>>>>>>
>>>>>>> - One of the most sophisticated feature is automatic model usage
>>>>>>> analyzation. The result are fetch rules that are sent to the
>>>>>>> server. When later objects are loaded from the server, it can
>>>>>>> answer with additional objects that are likely to be needed by the
>>>>>>> client in the near future.
>>>>>>>
>>>>>>> public class CDOViewImpl extends
>>>>>>> org.eclipse.net4j.internal.util.event.Notifier implements
>>>>>>> CDOView, CDOIDProvider, Adapter.Internal
>>>>>>> {
>>>>>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>>>> {
>>>>>>> return featureAnalyzer;
>>>>>>> }
>>>>>>>
>>>>>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>>>> featureAnalyzer)
>>>>>>> {
>>>>>>> this.featureAnalyzer = featureAnalyzer == null ?
>>>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> - There's a SmartReadAheadThread (if you're curious, please ask
>>>>>>> Simon). - Caches can be configured
>>>>>>> - ...
>>>>>>>
>>>>>>>
>>>>>>> Known issues of the DBStore:
>>>>>>> - When in auditing mode updates are always written as complete
>>>>>>> inserts (by design) - When configuring toManyReferences to a value
>>>>>>> different from LIKE_ATTRIBUTES (which is not implemented yet) it
>>>>>>> is very slow. - No (few) prepared statements are used (see
>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe
>>>>>>> others...
>>>>>>>
>>>>>>> Please note that the DBStore was not written by me to compete with
>>>>>>> Hibernate in terms of performance or mapping capabilities. Now
>>>>>>> that Martin and I, well in the end mostly Martin, provide a
>>>>>>> HibernateStore and this HibernateStore gets more and more features
>>>>>>> I expect that there will be a trend away from the DBStore.
>>>>>>>
>>>>>>>
>>>>>>> This is probably linked to the way the mapping to db is done, here's
>>>>>>> the config I used (came from CDO-server example if I recall
>>>>>>> correctly).
>>>>>>>
>>>>>>>
>>>>>>> <repository name="repo1">
>>>>>>> <property name="overrideUUID"
>>>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>>>> <property name="supportingAudits" value="true"/>
>>>>>>> <property name="verifyingRevisions" value="false"/>
>>>>>>> <property name="currentLRUCapacity" value="10000"/>
>>>>>>> I guess this one could be larger.
>>>>>>>
>>>>>>>
>>>>>>> <property name="revisedLRUCapacity" value="100"/>
>>>>>>>
>>>>>>> <store type="db">
>>>>>>> <!-- type: horizontal | vertical | <any
>>>>>>> user-contributed type> --> <mappingStrategy
>>>>>>> type="horizontal">
>>>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>>>> ONE_TABLE_PER_CLASS |
>>>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES
>>>>>>> -->
>>>>>>> <property name="toManyReferences"
>>>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>>>
>>>>>>> <!-- LIKE_ATTRIBUTES |
>>>>>>> LIKE_TO_MANY_REFERENCES -->
>>>>>>> <property name="toOneReferences"
>>>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>>>
>>>>>>> <!-- MODEL | STRATEGY -->
>>>>>>> <property name="mappingPrecedence"
>>>>>>> value="MODEL"/>
>>>>>>> This one is obsolete.
>>>>>>>
>>>>>>>
>>>>>>> </mappingStrategy>
>>>>>>>
>>>>>>> <dbAdapter name="derby-embedded"/>
>>>>>>> <dataSource
>>>>>>>
>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>>>>> databaseName="/tmp/cdodb1"
>>>>>>> createDatabase="create"/>
>>>>>>>
>>>>>>> </store>
>>>>>>> </repository>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> What do you think about that ? What throughput do you think I should
>>>>>>> expect ? As I said that's very difficult to judge.
>>>>>>> I believe it's more appropriate to compare the actual values with
>>>>>>> values from other measures (other setups, properties, etc. or even
>>>>>>> other products) instead of random expectations. Apologies that I
>>>>>>> didn't have the time so far to provide some reference measures
>>>>>>> myself. Maybe this gets better now that the team starts to grow
>>>>>>> slowly ;-)
>>>>>>>
>>>>>>>
>>>>>>> What are the main reasons fur such figures ?
>>>>>>> Hmm, I *suspect* you don't like the figures? I can find no
>>>>>>> evidence your assessment ;-)
>>>>>>>
>>>>>>> Ahh,one more note:
>>>>>>> Tracing can also have a significant impact on performance and
>>>>>>> end-to-end throughput. Turn it off by all means. And please make
>>>>>>> sure that it's really turned off, not only invisible (i.e. console
>>>>>>> is not a target).
>>>>>>>
>>>>>>> Cheers
>>>>>>> /Eike
>>>
>>
>
>


http://cedric.brun.io news and articles on eclipse and eclipse modeling.
Re: [CDO] reading/writting throughput performances [message #124481 is a reply to message #124260] Sat, 31 May 2008 23:02 Go to previous messageGo to next message
Martin Taal is currently offline Martin TaalFriend
Messages: 5468
Registered: July 2009
Senior Member
Hi Cédric,
I looked at it for many hours trying different things (using your testcase) but I am kind of stuck
with this (getting the same error as you). The difference I can see with the way I run the
cdo.hibernate testcases (as a junit test) and how your testcase is run (junit plugin test). This for
sure means difference in class loader behavior.

Eike, do you have an idea? I am not sure how classloading works when you start a cdo server in a
plugin. Which plugins are then available in the cdo server? The testcases can be downloaded from the
link below.

gr. Martin

Cédric Brun wrote:
> Thanks a lot ! I'm looking forward to your advices !
>
> Cédric
>
> Martin Taal wrote:
>
>> I am downloading your zip-file. I don't have time to look at it right away
>> but I will do my best later this evening.
>>
>> gr. Martin
>>
>> Cédric Brun wrote:
>>> I tried to do the same things as in the CDO/Hibernate unit tests.
>>>
>>> Here's a zipped project trying to launch an HibernateServer with a unit
>>> test adding elements.
>>> ( http://cedric.brun.free.fr/Tortoose/org.eclipse.cdo-teneo-te st.zip) The
>>> libs I found are included in the same project, but I still have no luck
>>> and failling at :
>>>
>>> at
>>>
> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>> ... 62 more
>>> Caused by: java.lang.NoClassDefFoundError:
>>> org/hibernate/proxy/HibernateProxy
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>>> ... 68 more
>>>
>>> please note that the test project I provide really is a "quick and ugly"
>>> one ^^
>>>
>>>
>>> Cédric
>>>
>>>
>>>
>>> Martin Taal wrote:
>>>
>>>> Hmm, I can understand your headache. Classpath issues are the most
>>>> annoying errors...
>>>>
>>>> In your junit test source file, what happens when you enter the
>>>> org.objectweb.asm.Type, can you source file see this class?
>>>> Do you have cglib in some other plugin which does not have asm?
>>>>
>>>> What I also find slightly strange is that cglib is used in this case
>>>> because normally cdo objects are not proxied using cglib. Can you post
>>>> the generated mapping? How does your cdo initialization code look like?
>>>>
>>>> gr. Martin
>>>>
>>>> Cédric Brun wrote:
>>>>> Hi Martin,
>>>>>
>>>>> and thanks for your quick answer ! what I did is the following :
>>>>>
>>>>> I used the org.eclipse.emf.cdo.server.hibernate.libraries bundle and
>>>>> added that :
>>>>>
>>>>> Bundle-ClassPath: lib/hibernate3.jar,
>>>>> lib/cglib-2.1.3.jar,
>>>>> lib/commons-collections-2.1.1.jar,
>>>>> lib/dom4j.jar,
>>>>> lib/jta.jar
>>>>> lib/asm-3.0.jar
>>>>>
>>>>> and this directive :
>>>>>
>>>>> Eclipse-BuddyPolicy: dependent
>>>>>
>>>>> as stated in
>>>>> http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>>>>>
>>>>> Then I don't get the same error, now when I start my own plugin (JUnit
>>>>> plugin test) depending on
>>>>> org.eclipse.emf.cdo.server.hibernate.libraries I get :
>>>>>
>>>>> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
>>>>> at net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
>>>>> at net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
>>>>> at net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
>>>>> at
>>>>>
> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>>>>> These jars are such a headache ! Do you know - by chance - what's
>>>>> going on ?
>>>>>
>>>>>
>>>>> Cédric
>>>>>
>>>>>
>>>>> Martin Taal wrote:
>>>>>
>>>>>> Hi Cédric,
>>>>>> Most of the time this exception is thrown because hibernate can't find
>>>>>> the class. A common cause is that the hibernate jar files are located
>>>>>> in a separate plugin which can not reach the classes in other plugins.
>>>>>> If you set the eclipse buddy policy of the hibernate plugin to
>>>>>> dependent then this can help.
>>>>>>
>>>>>> gr. Martin
>>>>>>
>>>>>> Cédric Brun wrote:
>>>>>>> Hi Simon and Eike,
>>>>>>>
>>>>>>> thanks for your feedback, now I know what I could expect in using
>>>>>>> optimized implementation, I'll have a try on the hibernate store and
>>>>>>> see if it is scaling in a better way.
>>>>>>>
>>>>>>> You gave me many pointers to explain the performances issues but in
>>>>>>> fact it seems that the SQL execution is taking so much time (not
>>>>>>> suprisingly in fact ;) . My first profile was wrong.
>>>>>>>
>>>>>>> So I'm installing the hibernate store retrieving all the source code
>>>>>>> from CVS, and I'm trying to launch the unit tests. I added many jars
>>>>>>> and everything gets compiled but most of the tests are failling with
>>>>>>> "
>>>>>>>
>>>>>>> "
>>>>>>> org.hibernate.MappingException: Could not determine type for:
>>>>>>>
> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>>>>>> at table: cdo_features, for columns:
>>>>>>> [org.hibernate.mapping.Column(type)] "
>>>>>>>
>>>>>>> any hint ?
>>>>>>>
>>>>>>> Cédric
>>>>>>>
>>>>>>>
>>>>>>> Simon McDuff wrote:
>>>>>>>
>>>>>>>> We noticed a performance degradation in the following case:
>>>>>>>>
>>>>>>>> When we fetch a collection of objects that contains collections that
>>>>>>>> we don't want to use. Ex:
>>>>>>>>
>>>>>>>> Model:
>>>>>>>>
>>>>>>>> ClassB
>>>>>>>> {
>>>>>>>> int foo1;
>>>>>>>> List childrens1;
>>>>>>>> List childrens2;
>>>>>>>> List childrens3;
>>>>>>>> };
>>>>>>>>
>>>>>>>> Code:
>>>>>>>>
>>>>>>>> Resource.contents contains objects of type ClassB.
>>>>>>>>
>>>>>>>> By fetching instances of classB, we will also fetched some items in
>>>>>>>> the list(childrens1, childrens2, childrens3) The problem we had.. is
>>>>>>>> we do not know if we want to use it. SO better not to download them.
>>>>>>>>
>>>>>>>> So In our production(we used our own IStore), we never download any
>>>>>>>> objects from collection. I'm working at the moment to put in place
>>>>>>>> all feature we used in the current CDO. That one will be part of
>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477. To begin
>>>>>>>> with,CDORevisionImpl needs to supports list that are 0 even if
>>>>>>>> referenceChunkSize == 0.
>>>>>>>>
>>>>>>>> It is true that we obtain good performance.
>>>>>>>> Worst case :
>>>>>>>> 1500 objects /sec (Real fetch from disk. Page that contains the
>>>>>>>> objects, wasn't in the cache of the filesystem.)
>>>>>>>>
>>>>>>>> Best case
>>>>>>>> 12 000 objects/Sec (Fetch objects that are already in the case of
>>>>>>>> the filesystem)
>>>>>>>>
>>>>>>>> (These statistics always fetch objects from server to client. The
>>>>>>>> cache client isn't used)
>>>>>>>>
>>>>>>>> We have another optimization one with SmartReadAhead but for now
>>>>>>>> only work
>>>>>>>> for UI client. (for now) This is something we will integrate as
>>>>>>>> well in CDO. But for now we concentrate on the other things.
>>>>>>>>
>>>>>>>> I used an Object database. Based on my preliminary test without CDO.
>>>>>>>> I compare
>>>>>>>>
>>>>>>>> Hibernate vs Objectivity/EMF :
>>>>>>>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>>>>>>>
>>>>>>>> The worst.. you will get 30 to 240 objects/sec or
>>>>>>>> the best... you will get 500 to 4000 objects/sec
>>>>>>>>
>>>>>>>> Based on your comments... you are getting the worst (with Derby)!!
>>>>>>>> :-)
>>>>>>>>
>>>>>>>>
>>>>>>>> Another comments about insertion.
>>>>>>>>
>>>>>>>> We found another problem when we insert data(DOn't know if it is
>>>>>>>> still true). Resource.Contents is a unique list. Basically each time
>>>>>>>> we insert one element, it needs to fetch all the collection to see
>>>>>>>> if we already have this element. Basically.. it is time consuming.
>>>>>>>> In our case we are not going through this stage to persist our
>>>>>>>> objects... but the objects are not accessible through that list.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> If speed is an issue for you!! We (You included :-)) can optimize
>>>>>>>> it. What I like is Eike did a really good job doing that framwork
>>>>>>>> and optimizations are easy to add. Also you can change your back-end
>>>>>>>> without affecting you client side. (Maybe going to an Object
>>>>>>>> database!!) THis was one of the purpose of CDO!! :-)
>>>>>>>>
>>>>>>>>
>>>>>>>> Simon
>>>>>>>>
>>>>>>>>
>>>>>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le message de
>>>>>>>> news: 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>>>>>
>>>>>>>> That sounds interesting!
>>>>>>>> Comments inline...
>>>>>>>>
>>>>>>>>
>>>>>>>> Cédric Brun schrieb:
>>>>>>>> Hi Eike,
>>>>>>>>
>>>>>>>> I've been messing around CDO lately and I hacked a few ugly
>>>>>>>> unit-tests measuring read/write throughput when I use CDO (RC2).
>>>>>>>>
>>>>>>>> I'm using an embeded-derby database, two different metamodels
>>>>>>>> (complex one - 250 EClassifiers/300 features and a simple one - 1
>>>>>>>> Classifier/3 Features)
>>>>>>>>
>>>>>>>> The test is a "quick and dirty" one so please do not take these
>>>>>>>> figures too seriously, I would just like to have your input about
>>>>>>>> the "order of magnitude" I should expect or settings affecting
>>>>>>>> seriously these performances. I tried both JVM and TCP acceptor but
>>>>>>>> the performances are quite similar though JVM is a bit faster.
>>>>>>>> Looks like a sign that the NIO socket channels are efficient ,-)
>>>>>>>> You used localhost for TCP transport I guess?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> * First case : building a small model (10000 elements) with the
>>>>>>>> simple MM, only adding instances in an empty database :
>>>>>>>> commiting change on every add I get a 2 objects per second rate.
>>>>>>>> commiting changes every 10 adds - I get a 30 objects per second
>>>>>>>> rate.
>>>>>>>> Seems to be consistent with the fact that only for commits there
>>>>>>>> is remote traffic. And the backend transaction management is
>>>>>>>> another overhead. So the impact of the number objects/commit
>>>>>>>> starts with bigger transactions.
>>>>>>>>
>>>>>>>>
>>>>>>>> browsing the just commited elements opening a new transaction and
>>>>>>>> getting a new Resource instance, I get around 30 000 objets/sec rate
>>>>>>>> just iterating thanks to Resource.getAllContents().
>>>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>>>> All views and transactions of a single session share (and cache)
>>>>>>>> their object state. Otherwise I'd expect much smaller values.
>>>>>>>> With his ObjectivityStore implementation Simon (cc'ed) reached
>>>>>>>> rates of several thousands objects/sec. But the DBStore (which you
>>>>>>>> used) is not that optimized.
>>>>>>>>
>>>>>>>>
>>>>>>>> * Second case : I built a big model in the database using the
>>>>>>>> complex metamodel. It contains 700 000 elements. The creation went
>>>>>>>> pretty nicely (I did not measured it exactly as it was launched
>>>>>>>> during a night, but it seems I got around 100 objects/sec rate in
>>>>>>>> writting as I was writting every 700 objects.
>>>>>>>> Sounds not too bad.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Once the model is built the derby database is 222 MB (which - I
>>>>>>>> heard - is not so big for derby).
>>>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>>>> To be honest, I'm really no database expert (although I can write
>>>>>>>> simple SQL statements).
>>>>>>>>
>>>>>>>>
>>>>>>>> Retrieving the distant resource and calling eAllContents() on it, I
>>>>>>>> get a 50 Objects/sec rate at best, which mean my eAllContents() call
>>>>>>>> would take 4 hours !
>>>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>>>> (iteration order and so). There are several CDO features that can
>>>>>>>> have a major impact on load rates!
>>>>>>>>
>>>>>>>> - Collections (xrefs and containment) can be loaded lazily. I
>>>>>>>> don't mean the target objects, which are lazy by default anyway,
>>>>>>>> but even the target IDs that are in the collections. We call that
>>>>>>>> "reference chunking" (but search for a better name is ongoing).
>>>>>>>>
>>>>>>>> public interface CDOSession extends CDOProtocolSession,
>>>>>>>> IContainer<CDOView>
>>>>>>>> {
>>>>>>>> public int getReferenceChunkSize();
>>>>>>>>
>>>>>>>> public void setReferenceChunkSize(int referenceChunkSize);
>>>>>>>>
>>>>>>>>
>>>>>>>> - Loading collection elements (target objects) can be batched (the
>>>>>>>> current name is even worse)
>>>>>>>>
>>>>>>>> public interface CDOView extends CDOProtocolView, INotifier
>>>>>>>> {
>>>>>>>> public int getLoadRevisionCollectionChunkSize();
>>>>>>>>
>>>>>>>> public void setLoadRevisionCollectionChunkSize(int
>>>>>>>> loadRevisionCollectionChunkSize);
>>>>>>>>
>>>>>>>>
>>>>>>>> - One of the most sophisticated feature is automatic model usage
>>>>>>>> analyzation. The result are fetch rules that are sent to the
>>>>>>>> server. When later objects are loaded from the server, it can
>>>>>>>> answer with additional objects that are likely to be needed by the
>>>>>>>> client in the near future.
>>>>>>>>
>>>>>>>> public class CDOViewImpl extends
>>>>>>>> org.eclipse.net4j.internal.util.event.Notifier implements
>>>>>>>> CDOView, CDOIDProvider, Adapter.Internal
>>>>>>>> {
>>>>>>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>>>>> {
>>>>>>>> return featureAnalyzer;
>>>>>>>> }
>>>>>>>>
>>>>>>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>>>>> featureAnalyzer)
>>>>>>>> {
>>>>>>>> this.featureAnalyzer = featureAnalyzer == null ?
>>>>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> - There's a SmartReadAheadThread (if you're curious, please ask
>>>>>>>> Simon). - Caches can be configured
>>>>>>>> - ...
>>>>>>>>
>>>>>>>>
>>>>>>>> Known issues of the DBStore:
>>>>>>>> - When in auditing mode updates are always written as complete
>>>>>>>> inserts (by design) - When configuring toManyReferences to a value
>>>>>>>> different from LIKE_ATTRIBUTES (which is not implemented yet) it
>>>>>>>> is very slow. - No (few) prepared statements are used (see
>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe
>>>>>>>> others...
>>>>>>>>
>>>>>>>> Please note that the DBStore was not written by me to compete with
>>>>>>>> Hibernate in terms of performance or mapping capabilities. Now
>>>>>>>> that Martin and I, well in the end mostly Martin, provide a
>>>>>>>> HibernateStore and this HibernateStore gets more and more features
>>>>>>>> I expect that there will be a trend away from the DBStore.
>>>>>>>>
>>>>>>>>
>>>>>>>> This is probably linked to the way the mapping to db is done, here's
>>>>>>>> the config I used (came from CDO-server example if I recall
>>>>>>>> correctly).
>>>>>>>>
>>>>>>>>
>>>>>>>> <repository name="repo1">
>>>>>>>> <property name="overrideUUID"
>>>>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>>>>> <property name="supportingAudits" value="true"/>
>>>>>>>> <property name="verifyingRevisions" value="false"/>
>>>>>>>> <property name="currentLRUCapacity" value="10000"/>
>>>>>>>> I guess this one could be larger.
>>>>>>>>
>>>>>>>>
>>>>>>>> <property name="revisedLRUCapacity" value="100"/>
>>>>>>>>
>>>>>>>> <store type="db">
>>>>>>>> <!-- type: horizontal | vertical | <any
>>>>>>>> user-contributed type> --> <mappingStrategy
>>>>>>>> type="horizontal">
>>>>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>>>>> ONE_TABLE_PER_CLASS |
>>>>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES
>>>>>>>> -->
>>>>>>>> <property name="toManyReferences"
>>>>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>>>>
>>>>>>>> <!-- LIKE_ATTRIBUTES |
>>>>>>>> LIKE_TO_MANY_REFERENCES -->
>>>>>>>> <property name="toOneReferences"
>>>>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>>>>
>>>>>>>> <!-- MODEL | STRATEGY -->
>>>>>>>> <property name="mappingPrecedence"
>>>>>>>> value="MODEL"/>
>>>>>>>> This one is obsolete.
>>>>>>>>
>>>>>>>>
>>>>>>>> </mappingStrategy>
>>>>>>>>
>>>>>>>> <dbAdapter name="derby-embedded"/>
>>>>>>>> <dataSource
>>>>>>>>
>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>>>>>> databaseName="/tmp/cdodb1"
>>>>>>>> createDatabase="create"/>
>>>>>>>>
>>>>>>>> </store>
>>>>>>>> </repository>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> What do you think about that ? What throughput do you think I should
>>>>>>>> expect ? As I said that's very difficult to judge.
>>>>>>>> I believe it's more appropriate to compare the actual values with
>>>>>>>> values from other measures (other setups, properties, etc. or even
>>>>>>>> other products) instead of random expectations. Apologies that I
>>>>>>>> didn't have the time so far to provide some reference measures
>>>>>>>> myself. Maybe this gets better now that the team starts to grow
>>>>>>>> slowly ;-)
>>>>>>>>
>>>>>>>>
>>>>>>>> What are the main reasons fur such figures ?
>>>>>>>> Hmm, I *suspect* you don't like the figures? I can find no
>>>>>>>> evidence your assessment ;-)
>>>>>>>>
>>>>>>>> Ahh,one more note:
>>>>>>>> Tracing can also have a significant impact on performance and
>>>>>>>> end-to-end throughput. Turn it off by all means. And please make
>>>>>>>> sure that it's really turned off, not only invisible (i.e. console
>>>>>>>> is not a target).
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>> /Eike
>>
>


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Cell: +31 (0)6 288 48 943
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: [CDO] reading/writting throughput performances [message #124496 is a reply to message #124481] Sun, 01 June 2008 04:41 Go to previous messageGo to next message
Eclipse UserFriend
Originally posted by: stepper.sympedia.de

This is a multi-part message in MIME format.
--------------050306020307050804040207
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi guys,

I can confirm, making these 3rd party libraries with their own
classloading run in OSGi is a nightmare! I took me hours teach where the
hsqldb driver is. On the way I came over different kinds of classloading
related errors.

First I removed all the jars from the test plugin (the one Cedric
provided). They're already in the cdo.server.hibernate.libraries plugin
which, in fact, needs the buddy policy set to dependent. I think this
last change needs to be committed to CVS! Btw. I'd prefer to use
*registered* buddies instead of making all dependent bundles visible.

The test plugin then needs to provide the database driver. Therefore I
added a dependency on the org.eclipse.net4j.db.hsqldb plugin.

With this setup I can start the server without CNFE, NCDFE, CCI and the
like. At this point I get the following exception, which seems the same
as the one Cedric observed in the end:

net.sf.cglib.core.CodeGenerationException:
java.lang.reflect.InvocationTargetException-->null
at
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:237)
at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
at net.sf.cglib.proxy.Enhancer.createClass(Enhancer.java:317)
at
org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:127)
at
org.hibernate.proxy.pojo.cglib.CGLIBProxyFactory.postInstant iate(CGLIBProxyFactory.java:43)
at
org.hibernate.tuple.entity.PojoEntityTuplizer.buildProxyFact ory(PojoEntityTuplizer.java:162)
at
org.hibernate.tuple.entity.AbstractEntityTuplizer.<init>(AbstractEntityTuplizer.java:135)
at
org.hibernate.tuple.entity.PojoEntityTuplizer.<init>(PojoEntityTuplizer.java:55)
at
org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping . <init>(EntityEntityModeToTuplizerMapping.java:56)
at
org.hibernate.tuple.entity.EntityMetamodel.<init>(EntityMetamodel.java:295)
at
org.hibernate.persister.entity.AbstractEntityPersister.<init >(AbstractEntityPersister.java:434)
at
org.hibernate.persister.entity.JoinedSubclassEntityPersister . <init>(JoinedSubclassEntityPersister.java:91)
at
org.hibernate.persister.PersisterFactory.createClassPersiste r(PersisterFactory.java:58)
at
org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:226)
at
org.hibernate.cfg.Configuration.buildSessionFactory(Configur ation.java:1294)
at
org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getSessionFactory(HibernatePackageHandler.java:335 )
at
org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.readPackageInfos(HibernatePackageHandler.java:293)
at
org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getCDOPackageInfos(HibernatePackageHandler.java:21 5)
at
org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Reader.readPackageInfos(HibernateStoreReader.java:87)
at
org.eclipse.emf.cdo.internal.server.PackageManager.doActivat e(PackageManager.java:72)
at
org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
at
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
at
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
at
org.eclipse.emf.cdo.internal.server.Repository.activateRepos itory(Repository.java:264)
at
org.eclipse.emf.cdo.internal.server.Repository.doActivate(Re pository.java:251)
at
org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
at
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
at
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
at
org.eclipse.emf.cdo.server.CDOServerUtil.addRepository(CDOSe rverUtil.java:67)
at org.eclipse.cdo.bug.LaunchServerTest.setUp(LaunchServerTest. java:54)
at junit.framework.TestCase.runBare(TestCase.java:128)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:120)
at junit.framework.TestSuite.runTest(TestSuite.java:230)
at junit.framework.TestSuite.run(TestSuite.java:225)
at
org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
at
org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
at
org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hodWithException(EclipseAppContainer.java:574)
at
org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
at
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:110)
at
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:79)
at
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:379)
at
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 549)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
at org.eclipse.equinox.launcher.Main.main(Main.java:1212)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at net.sf.cglib.core.ReflectUtils.defineClass(ReflectUtils.java :384)
at
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
... 61 more
Caused by: java.lang.NoClassDefFoundError:
org/hibernate/proxy/HibernateProxy
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
... 67 more

I must admit that I'm still not sure about te fundamental interwork
between Hibernate and the user defined meta model. It seems as if asm
and cglib are somewhere in between but I have no idea why at all. I
thought we have implemented a bunch of static Hibernate interfaces to
make the CDORevision instances known to Hibernate. Please somebody
explain why we need runtime byte code engineering? Is there some
configuration missing to tell Hibernate to use our special tuplizers?

I have changed several things in the code of the test plugin and since
it's quite small now that the jars are removed, I attach my version of
the zip here (including a JUnit Plugin launch config). And don't forget
to update the cdo.server.hibernate.libraries plugin from CVS to get the
new buddy policy!

Cheers
/Eike

Martin Taal schrieb:
> Hi Cédric,
> I looked at it for many hours trying different things (using your
> testcase) but I am kind of stuck with this (getting the same error as
> you). The difference I can see with the way I run the cdo.hibernate
> testcases (as a junit test) and how your testcase is run (junit plugin
> test). This for sure means difference in class loader behavior.
>
> Eike, do you have an idea? I am not sure how classloading works when
> you start a cdo server in a plugin. Which plugins are then available
> in the cdo server? The testcases can be downloaded from the link below.
>
> gr. Martin
>
> Cédric Brun wrote:
>> Thanks a lot ! I'm looking forward to your advices !
>>
>> Cédric
>>
>> Martin Taal wrote:
>>
>>> I am downloading your zip-file. I don't have time to look at it
>>> right away
>>> but I will do my best later this evening.
>>>
>>> gr. Martin
>>>
>>> Cédric Brun wrote:
>>>> I tried to do the same things as in the CDO/Hibernate unit tests.
>>>>
>>>> Here's a zipped project trying to launch an HibernateServer with a
>>>> unit
>>>> test adding elements.
>>>> ( http://cedric.brun.free.fr/Tortoose/org.eclipse.cdo-teneo-te st.zip)
>>>> The
>>>> libs I found are included in the same project, but I still have no
>>>> luck
>>>> and failling at :
>>>>
>>>> at
>>>>
>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>
>>>> ... 62 more
>>>> Caused by: java.lang.NoClassDefFoundError:
>>>> org/hibernate/proxy/HibernateProxy
>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>>>> ... 68 more
>>>>
>>>> please note that the test project I provide really is a "quick and
>>>> ugly"
>>>> one ^^
>>>>
>>>>
>>>> Cédric
>>>>
>>>>
>>>>
>>>> Martin Taal wrote:
>>>>
>>>>> Hmm, I can understand your headache. Classpath issues are the most
>>>>> annoying errors...
>>>>>
>>>>> In your junit test source file, what happens when you enter the
>>>>> org.objectweb.asm.Type, can you source file see this class?
>>>>> Do you have cglib in some other plugin which does not have asm?
>>>>>
>>>>> What I also find slightly strange is that cglib is used in this case
>>>>> because normally cdo objects are not proxied using cglib. Can you
>>>>> post
>>>>> the generated mapping? How does your cdo initialization code look
>>>>> like?
>>>>>
>>>>> gr. Martin
>>>>>
>>>>> Cédric Brun wrote:
>>>>>> Hi Martin,
>>>>>>
>>>>>> and thanks for your quick answer ! what I did is the following :
>>>>>>
>>>>>> I used the org.eclipse.emf.cdo.server.hibernate.libraries bundle and
>>>>>> added that :
>>>>>>
>>>>>> Bundle-ClassPath: lib/hibernate3.jar,
>>>>>> lib/cglib-2.1.3.jar,
>>>>>> lib/commons-collections-2.1.1.jar,
>>>>>> lib/dom4j.jar,
>>>>>> lib/jta.jar
>>>>>> lib/asm-3.0.jar
>>>>>>
>>>>>> and this directive :
>>>>>>
>>>>>> Eclipse-BuddyPolicy: dependent
>>>>>>
>>>>>> as stated in
>>>>>> http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>>>>>>
>>>>>> Then I don't get the same error, now when I start my own plugin
>>>>>> (JUnit
>>>>>> plugin test) depending on
>>>>>> org.eclipse.emf.cdo.server.hibernate.libraries I get :
>>>>>>
>>>>>> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
>>>>>> at net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
>>>>>> at net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
>>>>>> at net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
>>>>>> at
>>>>>>
>> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>>
>>>>>> These jars are such a headache ! Do you know - by chance - what's
>>>>>> going on ?
>>>>>>
>>>>>>
>>>>>> Cédric
>>>>>>
>>>>>>
>>>>>> Martin Taal wrote:
>>>>>>
>>>>>>> Hi Cédric,
>>>>>>> Most of the time this exception is thrown because hibernate
>>>>>>> can't find
>>>>>>> the class. A common cause is that the hibernate jar files are
>>>>>>> located
>>>>>>> in a separate plugin which can not reach the classes in other
>>>>>>> plugins.
>>>>>>> If you set the eclipse buddy policy of the hibernate plugin to
>>>>>>> dependent then this can help.
>>>>>>>
>>>>>>> gr. Martin
>>>>>>>
>>>>>>> Cédric Brun wrote:
>>>>>>>> Hi Simon and Eike,
>>>>>>>>
>>>>>>>> thanks for your feedback, now I know what I could expect in using
>>>>>>>> optimized implementation, I'll have a try on the hibernate
>>>>>>>> store and
>>>>>>>> see if it is scaling in a better way.
>>>>>>>>
>>>>>>>> You gave me many pointers to explain the performances issues
>>>>>>>> but in
>>>>>>>> fact it seems that the SQL execution is taking so much time (not
>>>>>>>> suprisingly in fact ;) . My first profile was wrong.
>>>>>>>>
>>>>>>>> So I'm installing the hibernate store retrieving all the source
>>>>>>>> code
>>>>>>>> from CVS, and I'm trying to launch the unit tests. I added many
>>>>>>>> jars
>>>>>>>> and everything gets compiled but most of the tests are failling
>>>>>>>> with
>>>>>>>> "
>>>>>>>>
>>>>>>>> "
>>>>>>>> org.hibernate.MappingException: Could not determine type for:
>>>>>>>>
>> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>>>>>>> at table: cdo_features, for columns:
>>>>>>>> [org.hibernate.mapping.Column(type)] "
>>>>>>>>
>>>>>>>> any hint ?
>>>>>>>>
>>>>>>>> Cédric
>>>>>>>>
>>>>>>>>
>>>>>>>> Simon McDuff wrote:
>>>>>>>>
>>>>>>>>> We noticed a performance degradation in the following case:
>>>>>>>>>
>>>>>>>>> When we fetch a collection of objects that contains
>>>>>>>>> collections that
>>>>>>>>> we don't want to use. Ex:
>>>>>>>>>
>>>>>>>>> Model:
>>>>>>>>>
>>>>>>>>> ClassB
>>>>>>>>> {
>>>>>>>>> int foo1;
>>>>>>>>> List childrens1;
>>>>>>>>> List childrens2;
>>>>>>>>> List childrens3;
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>> Code:
>>>>>>>>>
>>>>>>>>> Resource.contents contains objects of type ClassB.
>>>>>>>>>
>>>>>>>>> By fetching instances of classB, we will also fetched some
>>>>>>>>> items in
>>>>>>>>> the list(childrens1, childrens2, childrens3) The problem we
>>>>>>>>> had.. is
>>>>>>>>> we do not know if we want to use it. SO better not to download
>>>>>>>>> them.
>>>>>>>>>
>>>>>>>>> So In our production(we used our own IStore), we never
>>>>>>>>> download any
>>>>>>>>> objects from collection. I'm working at the moment to put in
>>>>>>>>> place
>>>>>>>>> all feature we used in the current CDO. That one will be part of
>>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477. To begin
>>>>>>>>> with,CDORevisionImpl needs to supports list that are 0 even if
>>>>>>>>> referenceChunkSize == 0.
>>>>>>>>>
>>>>>>>>> It is true that we obtain good performance.
>>>>>>>>> Worst case :
>>>>>>>>> 1500 objects /sec (Real fetch from disk. Page that contains the
>>>>>>>>> objects, wasn't in the cache of the filesystem.)
>>>>>>>>>
>>>>>>>>> Best case
>>>>>>>>> 12 000 objects/Sec (Fetch objects that are already in the case of
>>>>>>>>> the filesystem)
>>>>>>>>>
>>>>>>>>> (These statistics always fetch objects from server to client. The
>>>>>>>>> cache client isn't used)
>>>>>>>>>
>>>>>>>>> We have another optimization one with SmartReadAhead but for now
>>>>>>>>> only work
>>>>>>>>> for UI client. (for now) This is something we will integrate as
>>>>>>>>> well in CDO. But for now we concentrate on the other things.
>>>>>>>>>
>>>>>>>>> I used an Object database. Based on my preliminary test
>>>>>>>>> without CDO.
>>>>>>>>> I compare
>>>>>>>>>
>>>>>>>>> Hibernate vs Objectivity/EMF :
>>>>>>>>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>>>>>>>>
>>>>>>>>> The worst.. you will get 30 to 240 objects/sec or
>>>>>>>>> the best... you will get 500 to 4000 objects/sec
>>>>>>>>>
>>>>>>>>> Based on your comments... you are getting the worst (with
>>>>>>>>> Derby)!!
>>>>>>>>> :-)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Another comments about insertion.
>>>>>>>>>
>>>>>>>>> We found another problem when we insert data(DOn't know if it is
>>>>>>>>> still true). Resource.Contents is a unique list. Basically
>>>>>>>>> each time
>>>>>>>>> we insert one element, it needs to fetch all the collection to
>>>>>>>>> see
>>>>>>>>> if we already have this element. Basically.. it is time
>>>>>>>>> consuming.
>>>>>>>>> In our case we are not going through this stage to persist our
>>>>>>>>> objects... but the objects are not accessible through that list.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> If speed is an issue for you!! We (You included :-)) can optimize
>>>>>>>>> it. What I like is Eike did a really good job doing that framwork
>>>>>>>>> and optimizations are easy to add. Also you can change your
>>>>>>>>> back-end
>>>>>>>>> without affecting you client side. (Maybe going to an Object
>>>>>>>>> database!!) THis was one of the purpose of CDO!! :-)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Simon
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le message de
>>>>>>>>> news: 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>>>>>>
>>>>>>>>> That sounds interesting!
>>>>>>>>> Comments inline...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cédric Brun schrieb:
>>>>>>>>> Hi Eike,
>>>>>>>>>
>>>>>>>>> I've been messing around CDO lately and I hacked a few ugly
>>>>>>>>> unit-tests measuring read/write throughput when I use CDO (RC2).
>>>>>>>>>
>>>>>>>>> I'm using an embeded-derby database, two different metamodels
>>>>>>>>> (complex one - 250 EClassifiers/300 features and a simple one - 1
>>>>>>>>> Classifier/3 Features)
>>>>>>>>>
>>>>>>>>> The test is a "quick and dirty" one so please do not take these
>>>>>>>>> figures too seriously, I would just like to have your input about
>>>>>>>>> the "order of magnitude" I should expect or settings affecting
>>>>>>>>> seriously these performances. I tried both JVM and TCP
>>>>>>>>> acceptor but
>>>>>>>>> the performances are quite similar though JVM is a bit faster.
>>>>>>>>> Looks like a sign that the NIO socket channels are efficient
>>>>>>>>> ,-)
>>>>>>>>> You used localhost for TCP transport I guess?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> * First case : building a small model (10000 elements) with the
>>>>>>>>> simple MM, only adding instances in an empty database :
>>>>>>>>> commiting change on every add I get a 2 objects per second rate.
>>>>>>>>> commiting changes every 10 adds - I get a 30 objects per second
>>>>>>>>> rate.
>>>>>>>>> Seems to be consistent with the fact that only for commits
>>>>>>>>> there
>>>>>>>>> is remote traffic. And the backend transaction management is
>>>>>>>>> another overhead. So the impact of the number objects/commit
>>>>>>>>> starts with bigger transactions.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> browsing the just commited elements opening a new transaction and
>>>>>>>>> getting a new Resource instance, I get around 30 000
>>>>>>>>> objets/sec rate
>>>>>>>>> just iterating thanks to Resource.getAllContents().
>>>>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>>>>> All views and transactions of a single session share (and
>>>>>>>>> cache)
>>>>>>>>> their object state. Otherwise I'd expect much smaller values.
>>>>>>>>> With his ObjectivityStore implementation Simon (cc'ed) reached
>>>>>>>>> rates of several thousands objects/sec. But the DBStore
>>>>>>>>> (which you
>>>>>>>>> used) is not that optimized.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> * Second case : I built a big model in the database using the
>>>>>>>>> complex metamodel. It contains 700 000 elements. The creation
>>>>>>>>> went
>>>>>>>>> pretty nicely (I did not measured it exactly as it was launched
>>>>>>>>> during a night, but it seems I got around 100 objects/sec rate in
>>>>>>>>> writting as I was writting every 700 objects.
>>>>>>>>> Sounds not too bad.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Once the model is built the derby database is 222 MB (which - I
>>>>>>>>> heard - is not so big for derby).
>>>>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>>>>> To be honest, I'm really no database expert (although I can
>>>>>>>>> write
>>>>>>>>> simple SQL statements).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Retrieving the distant resource and calling eAllContents() on
>>>>>>>>> it, I
>>>>>>>>> get a 50 Objects/sec rate at best, which mean my
>>>>>>>>> eAllContents() call
>>>>>>>>> would take 4 hours !
>>>>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>>>>> (iteration order and so). There are several CDO features
>>>>>>>>> that can
>>>>>>>>> have a major impact on load rates!
>>>>>>>>>
>>>>>>>>> - Collections (xrefs and containment) can be loaded lazily. I
>>>>>>>>> don't mean the target objects, which are lazy by default
>>>>>>>>> anyway,
>>>>>>>>> but even the target IDs that are in the collections. We call
>>>>>>>>> that
>>>>>>>>> "reference chunking" (but search for a better name is ongoing).
>>>>>>>>>
>>>>>>>>> public interface CDOSession extends CDOProtocolSession,
>>>>>>>>> IContainer<CDOView>
>>>>>>>>> {
>>>>>>>>> public int getReferenceChunkSize();
>>>>>>>>>
>>>>>>>>> public void setReferenceChunkSize(int
>>>>>>>>> referenceChunkSize);
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> - Loading collection elements (target objects) can be
>>>>>>>>> batched (the
>>>>>>>>> current name is even worse)
>>>>>>>>>
>>>>>>>>> public interface CDOView extends CDOProtocolView,
>>>>>>>>> INotifier
>>>>>>>>> {
>>>>>>>>> public int getLoadRevisionCollectionChunkSize();
>>>>>>>>>
>>>>>>>>> public void setLoadRevisionCollectionChunkSize(int
>>>>>>>>> loadRevisionCollectionChunkSize);
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> - One of the most sophisticated feature is automatic model
>>>>>>>>> usage
>>>>>>>>> analyzation. The result are fetch rules that are sent to the
>>>>>>>>> server. When later objects are loaded from the server, it can
>>>>>>>>> answer with additional objects that are likely to be needed
>>>>>>>>> by the
>>>>>>>>> client in the near future.
>>>>>>>>>
>>>>>>>>> public class CDOViewImpl extends
>>>>>>>>> org.eclipse.net4j.internal.util.event.Notifier implements
>>>>>>>>> CDOView, CDOIDProvider, Adapter.Internal
>>>>>>>>> {
>>>>>>>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>>>>>> {
>>>>>>>>> return featureAnalyzer;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>>>>>> featureAnalyzer)
>>>>>>>>> {
>>>>>>>>> this.featureAnalyzer = featureAnalyzer == null ?
>>>>>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> - There's a SmartReadAheadThread (if you're curious, please ask
>>>>>>>>> Simon). - Caches can be configured
>>>>>>>>> - ...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Known issues of the DBStore:
>>>>>>>>> - When in auditing mode updates are always written as complete
>>>>>>>>> inserts (by design) - When configuring toManyReferences to a
>>>>>>>>> value
>>>>>>>>> different from LIKE_ATTRIBUTES (which is not implemented
>>>>>>>>> yet) it
>>>>>>>>> is very slow. - No (few) prepared statements are used (see
>>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe
>>>>>>>>> others...
>>>>>>>>>
>>>>>>>>> Please note that the DBStore was not written by me to
>>>>>>>>> compete with
>>>>>>>>> Hibernate in terms of performance or mapping capabilities. Now
>>>>>>>>> that Martin and I, well in the end mostly Martin, provide a
>>>>>>>>> HibernateStore and this HibernateStore gets more and more
>>>>>>>>> features
>>>>>>>>> I expect that there will be a trend away from the DBStore.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is probably linked to the way the mapping to db is done,
>>>>>>>>> here's
>>>>>>>>> the config I used (came from CDO-server example if I recall
>>>>>>>>> correctly).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <repository name="repo1">
>>>>>>>>> <property name="overrideUUID"
>>>>>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>>>>>> <property name="supportingAudits" value="true"/>
>>>>>>>>> <property name="verifyingRevisions"
>>>>>>>>> value="false"/>
>>>>>>>>> <property name="currentLRUCapacity"
>>>>>>>>> value="10000"/>
>>>>>>>>> I guess this one could be larger.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <property name="revisedLRUCapacity" value="100"/>
>>>>>>>>> <store type="db">
>>>>>>>>> <!-- type: horizontal | vertical | <any
>>>>>>>>> user-contributed type> -->
>>>>>>>>> <mappingStrategy
>>>>>>>>> type="horizontal">
>>>>>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>>>>>> ONE_TABLE_PER_CLASS |
>>>>>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY |
>>>>>>>>> LIKE_ATTRIBUTES
>>>>>>>>> -->
>>>>>>>>> <property name="toManyReferences"
>>>>>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>>>>>
>>>>>>>>> <!-- LIKE_ATTRIBUTES |
>>>>>>>>> LIKE_TO_MANY_REFERENCES -->
>>>>>>>>> <property name="toOneReferences"
>>>>>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>>>>>
>>>>>>>>> <!-- MODEL | STRATEGY -->
>>>>>>>>> <property
>>>>>>>>> name="mappingPrecedence"
>>>>>>>>> value="MODEL"/>
>>>>>>>>> This one is obsolete.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> </mappingStrategy>
>>>>>>>>> <dbAdapter
>>>>>>>>> name="derby-embedded"/>
>>>>>>>>> <dataSource
>>>>>>>>>
>>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>>>>>>> databaseName="/tmp/cdodb1"
>>>>>>>>> createDatabase="create"/>
>>>>>>>>>
>>>>>>>>> </store>
>>>>>>>>> </repository>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> What do you think about that ? What throughput do you think I
>>>>>>>>> should
>>>>>>>>> expect ? As I said that's very difficult to judge.
>>>>>>>>> I believe it's more appropriate to compare the actual values
>>>>>>>>> with
>>>>>>>>> values from other measures (other setups, properties, etc.
>>>>>>>>> or even
>>>>>>>>> other products) instead of random expectations. Apologies
>>>>>>>>> that I
>>>>>>>>> didn't have the time so far to provide some reference measures
>>>>>>>>> myself. Maybe this gets better now that the team starts to grow
>>>>>>>>> slowly ;-)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> What are the main reasons fur such figures ?
>>>>>>>>> Hmm, I *suspect* you don't like the figures? I can find no
>>>>>>>>> evidence your assessment ;-)
>>>>>>>>>
>>>>>>>>> Ahh,one more note:
>>>>>>>>> Tracing can also have a significant impact on performance and
>>>>>>>>> end-to-end throughput. Turn it off by all means. And please
>>>>>>>>> make
>>>>>>>>> sure that it's really turned off, not only invisible (i.e.
>>>>>>>>> console
>>>>>>>>> is not a target).
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>> /Eike
>>>
>>
>
>

--------------050306020307050804040207
Content-Type: application/x-zip-compressed;
name="org.eclipse.cdo.bug.zip"
Content-Transfer-Encoding: base64
Content-Disposition: inline;
filename="org.eclipse.cdo.bug.zip"

UEsDBBQAAAAIAAgpwTjj4/0u0gAAAKQBAAAeAAAAb3JnLmVjbGlwc2UuY2Rv LmJ1Zy8uY2xh
c3NwYXRolZDBSgMxEIbPCr7DkvvOWkHwsGsRWcGCVdq1V0mTIR2NkzhJxL69 K1oUD0Vv8w/f
fAx/O3179tUrSqLAnZrAsaqQTbDErlP3w1V9pqbnR4et8TqlqPNmDAffCTnL tnoitp0ygVX1
sexUEAdoPMWE8GgzeF3YbEYnzBb9w+XtfLi4nveL5jdHnFFYe7C4Lg4KfV2i wDJrtlrs6mbY
RmxmJ8u+nsCpav79ULQIJgiC4EshQXvniyNO+1RJzE41jrVD/ju9jwwlx5J3 8Jo+tW3zs+53
UEsDBBQAAAAIAMaDvjhMuH3Q+wAAAEoDAAAcAAAAb3JnLmVjbGlwc2UuY2Rv LmJ1Zy8ucHJv
amVjdL2TzW4CIRSF1/UpzOyFducCx/Qn3dU0sX0AhFuKgQvhx9S3L8yMplNr YtLE3TmXc/ku
Adjyy5rpDkLUDhfNHbltpoDCSY1q0by/Pc/mzbKdMB/cFkR6giiC9qmE28kN Q26hdUEREEb7
CERIRzZZMdqtlIRw1gKmltGDKsVhs1g1/Wk2WRu59iCKGdxjaeMoa+EP3lYm Ilwogu94l4dw
hJcGHlSu1NhbOvKM/iZcgvQSyAtH/QExPVyNuBafYPn1eNxrkpwzsap75GYf dfw3figMN1zI
KQfoo70+mePVZKVx1S1Wbhc6kx89Bryo5exRT4gHWaY9vtnRb/gGUEsDBAoA AAAAAMo0wTgA
AAAAAAAAAAAAAAAeAAAAb3JnLmVjbGlwc2UuY2RvLmJ1Zy8uc2V0dGluZ3Mv UEsDBBQAAAAI
AMaDvjjku112lgAAAEQBAAA4AAAAb3JnLmVjbGlwc2UuY2RvLmJ1Zy8uc2V0 dGluZ3Mvb3Jn
LmVjbGlwc2UuamR0LmNvcmUucHJlZnOVjr0KgzAAhHefItA9qFQQwal06CAU 6gukekpK/rjE
Qt++Gdq5djmO4z7uDuMGMaiXqEtRHbum7OpWnM63MQdlW2AyOkTIQCwg3IQo n2DU3vVV4bnK
b+MxJzl5IosN2oDZzFjhZFJcka5GpcXT9pVsfoI2GK3y2I5yoL8bWKliBNNl hkt60WAP0nMn
DLfZP9HoN34OvgFQSwMEFAAAAAgAuzDBOB0MEFSNAAAA+AAAACQAAABvcmcu ZWNsaXBzZS5j
ZG8uYnVnL2J1aWxkLnByb3BlcnRpZXN1jzkOAjEMRXufIhJTMskFAIkCpCmG ipIGEisEJWPL
SSS4PYGK1d173/ICM7WwxDcJ/lxW0Mi8YhPd4DqAU5h0mGysDrNaKj0/gHqv RA6j+fbjZr/u
h932R8Sx+jb3muLfjIUYpQTMcDlK1pYSh4iaxKE8DoFMVSxq3SCL7T1OH5ua NUC1cC3PrvaK
gTtQSwMEFAAAAAgAxoO+OAyLrPzUAAAATgEAAB0AAABvcmcuZWNsaXBzZS5j ZG8uYnVnL2Rl
cmJ5LmxvZ6WOTU7DMBCF9z7F7AoLu+M4KUkkFkBFN7CiFxj/0EQCO3ImoN4e R1VOwNvMSO/N
N0/K/0lUiK3ERlYdYNc3XY+otK7g9H7uBTynxGO8wDFke4WfkOcxRTgPAZ4m cmV8pE/+pRzg
NS3RE6+23MzblQSNyiit6rLeNQf90NX3PYxxZoougENtWkQvUWuSptZe2saW UqsOlozBVhRs
oZOlOYAfc3Cc8hX2/D3techpuQw8LQwgxHGLvXzRPMNbIh8ylF+Zgy8N/FpK bTDl1tREPDzu
duIPUEsDBBQAAAAIAEAvwThXhVk+yAUAAJUYAAArAAAAb3JnLmVjbGlwc2Uu Y2RvLmJ1Zy9M
YXVuY2hTZXJ2ZXJUZXN0LmxhdW5jaKVZbW/bNhD+PmD/wTD6qVjol7RdBjQr jMTr0jaO4TgF
9smgRFqmQ5EqSfkFRf97SVm2ZelEqxmMAPYd7+7hvfCOzPsPm5i3VlRpJsV1 u4e67RYVoSRM
RNftp+k/F1ftljZYEMyloNdtIdsf/v79t/ccpyJc3EgxZ1GqsLHiLbNN7Aqp IkRDzhJNUUIo
Shn6lApmvhQk2k5FICWnWAyMUSxIDW090+11GycJFQRhFel2a4V5alUaldJ2 x8lou1ZEVRHO
wgzDQaSMYukgIJUKw2KKQqmooXZfBcmOB5N+Di1ZHbTPMdfUK5EaGVu94YCQ F0h9xZwRbGhT
D1h1xjJwchDwLA4XNHymBHQVjecoJPIPgIYSJY0MJQeZmiqbRR4WWrCAKmE3 1WgRYuRknaDm
zbJKQctVDFBNmADU1DDuc38W4zxBq473yKyhVPWu5zKC0gKOV4boiyxl+Kvv a6medYJDOuMy
/NFBMTXYZg3uoISnERO6U64Cl/cd++UiKwcfTkLnOOWmMUhCNeU0NJTMjrBy GGCm7UKyMCZB
rhI1FNtYe3mYxEyc8NxqML0yNWdScLcmxiKSiBJmaqwCmV6AlDCA43YJkElQ u70gJYRRiJN5
LJRxLAW4C7rBccJLoqUqI4F3b2hJgtDnIkkov/T4CKpI/0Z950fBaK/OqCtt 22lAJ9crPZcG
IJ/IEALAzmG/9Oz+vHS/SVojy6YC8g8M2ibgQn/jYDqECwyXUiUCewbo/v0O weaRo6BBGsHo
4q2Fd9Y19ZDWFNdl1Y6F8JLAojGL7FgjYeF9kSGduLMQBk+oCrYAC6zQo9Nh P5b7HFQcgFh2
WnhqK/dCyny9gImQp4Q+JK7/YN50KOHnOxbqZI3o4kDfqeNMm5KyIvgsYbJe hu4H4/HwdjYZ
Pj48TW6Gs/Fg+u9je69jKIza7q2fNEO3d6uko1Xo6J2c3rH0jqPvhtXHzHFT 62K0xCu8A9c5
Qff/4E7/Gw9r4PbqrIG+LtqzGZ2PuzcPo+ngbjSclMZCOMywjs/D4XjyNBrd jT7OBtPppPFE
AKubDh+no8H9sMmgWq9h9vludAuOFcd1XGLiepn7cfkLVnY3G7vORutulIUI tJSnECqnyotM
jScPHyeD+9lg8vHpfjiaPh5MXkjdevXd2PsQNUjqH62LdYGwdgSswsWR5H5Z ouBHkuA/Xorq
0/Bmehp3wAcvUl6o2Jk19PXudggbyS+R7owIqLtCcqztYWEWYyVXzMbYY97J 5rfbg+5L5EsH
26hIGhoQSWmM2i+tV7aficfZIOyZg49XE6+e2S6g3sE6O2gU1TJVoe1QayYu +2hz9e70/P+W
MiE30CC5oPx0WC36PmNge1IvaKG/ZTX2+hJdoT5a9bvdq+5l/8+yjhWja2Di qgLINjBntkNu
taFxYQfuEN5k3ufUvO6jt6ib23vTe9vv9d+c2kxINhMd3ihKzdzty6k+Ieq1 8YGp8g6errJW
WDEcQLM4zfibuDr2ZZyqKpslJ0SpI5Y5gpUtL+e2jdqxYAPsYymDEpY8DRSN bJtRp9OK9Y+t
rgpCQwWtTsYZtTCJxu4dRxXTZRdpbc/lyOZu9LqH3CdPl+673l+9LhS+0jSU 7cOG1Noz7r2p
hBiIz+7JpxzkPKv81aFpmCpmtl5mjY5SGdrd2yOLBYyX9VVCdchbVd4dmktV ul0ZiuPq0mNE
sBDSZGMYkKB0k1iEusLMcqgOE1A0u5xzTw8BE6Q8EOchi+2VigAgvGLHB7uC ++B03ddJIR5W
CrHA/oXpyVIsDOCzPK7WJXOq7GFXKixXcqCAzfSmwHFqFk2O+WZPKHn3PTHu vfRWHtmqtSzA
97O9AvAW57wp87vBrhWAiOqvmq75efBCrGbb4yxQ9gwGr1rgBa1W09nXyDpJ 38CtF3I9O8Rc
Cr5t/lRsFA5tAjV+d0w1vd096f3iA2dZcKAobg7TSo9LA1VBpgP8F8HSfwJQ SwMECgAAAAAA
yjTBOAAAAAAAAAAAAAAAAB0AAABvcmcuZWNsaXBzZS5jZG8uYnVnL01FVEEt SU5GL1BLAwQK
AAAAAADGg7443Tz9/zoAAAA6AAAAIwAAAG9yZy5lY2xpcHNlLmNkby5idWcv TUVUQS1JTkYv
Q0RPLk1GVGhpcyBpcyBhIG1hcmtlciBmaWxlIGZvciBidW5kbGVzIHdpdGgg Q0RPIG5hdGl2
ZSBtb2RlbHMuClBLAwQUAAAACABbMsE4uA/lHVIBAADkAgAAKAAAAG9yZy5l Y2xpcHNlLmNk
by5idWcvTUVUQS1JTkYvTUFOSUZFU1QuTUaVUstOwkAU3fcrGhJ39gaIJqaE hWA3RrSRxP20
vZSL05kyj4by9fYJWNHEVTs5j3vnnFkxQRvUxvtApUkK353A2FlYkXD0Vh14 wqY98soy9N2b
nNuURH3ogXWZRZJT3BKkSgFjTrlGiBMJkU1nmkTK0VR2c6PsSXg5/7zBkjOt Q2a2vgtnpkik
qqcrWVCC6nL+i4wZpyMzjVe7X4+9496SwiQ4YGxrQiAKUlJkKIzvPk/XgTeB e6ejea1qcAmp
EJQVhjK8db5BmG2q/wqfFaQpIk6m9OcK8ZBLZa6Qq0D+QQWNqkA1i9q7FG1c 89EYHmA86jQ7
K+hvPWwpQiWYQTAoUP7wa+IfDTwEmrsdJBFs9Z4n0S+iPufH2FDRNBDWT6H0 Xc6O5QCtG7zy
PGD59LawadgWFzSBeCGLP1laVSGqpnW1W/MFynJ+OlSF8sHWnaXzBVBLAwQK AAAAAADKNME4
AAAAAAAAAAAAAAAAGgAAAG9yZy5lY2xpcHNlLmNkby5idWcvbW9kZWwvUEsD BBQAAAAIAMaD
vjhyMBIWWAEAABkDAAAlAAAAb3JnLmVjbGlwc2UuY2RvLmJ1Zy9tb2RlbC9u YW1lcy5lY29y
ZaVSXWvCMBR991eE7Fmj7mWUVtmcgrCBqIO9xvS2hjVJycfq/v1uW3UWFDaW hzZtzjn3nHsT
Tw+qIJ9gnTQ6oaPBkBLQwqRS5wl92y76D3Q66cUgjIVovuLig+dADkpGZ9IY ST2CC6W0i/As
oXvvy4ixqqoGRuUDY3P2/rqkJ4jrQqr7BjEeDkcIe9mIPSjel9p5rgVcijc+ OlwQhSwdNAKg
slpkzOY1jBLNFYLrp2tFtHtbL2/SlUmhwOBMpIbtQs5aJrJWFjJ5OElNUCuG WcGdk5nENhAM
FPmvEosdG9UcngzMjPZcarANs+ZuvA3CB8uLBXB8wxWJR++t3AXfyYHT2V6C nrnn9Q/yq47c
MTbH0hiRsr9YWUMGFupRHK14vivwI5Ql2CcTdJrQ/qhtcb2OHrHcT3Qi2q0C 7VHABvinB2GK
oDS9Wuzs5EbRmF2OD+83617wSe8bUEsDBBQAAAAIAMaDvjiE2blUBAIAABAF AAAoAAAAb3Jn
LmVjbGlwc2UuY2RvLmJ1Zy9tb2RlbC9uYW1lcy5nZW5tb2RlbMVUXY+bMBB8 z6+w3OeL00iV
KgScqnxUkXp3UdVWfXXMQtwaG9kmyf37rg2kcJdTH8sDAnZmdnbHIr2/1Iqc wDppdEbfzxeU
gBamkLrK6Pdv27uP9D6fpRXo2hSgks+gH8IDudQyufKWyJsRvFBNuwRrGT16 3ySMnc/nuamr
ubEV+/mwoz0EhLEwAYFQsnEQgVCXbLlYLNkmwMbSg5F/UwenlETCWloQ3tjn jDLEXTmiMPND
WzFnxR2Kd70iY6/aSurdOqM38L3qI69xinB3lFhj/ObiQRdupz3YkguYktFc FFitn54Ov9BP
125EXCnu3GuSDHqaqyl7Vzc4noVS4Ys8wRoUVNzHSLxt+8XJujEW6S8nibp9 LehSUgL3rZ3I
fL2KUyIM9pNcC/gCpxDBhyF2YZpnK6uj30pQBfovuXJAcyymJSYoqy6LXIdN zWP4KZtUAnS0
9h/cSn5Q4HKcNruxw5TdRAcZjHHPxW9egSMNbkderhEV0jXGBejempMswG55 fyriwki01rMz
OrL7jsVxOvmYEqrHSh/ZBMtWRnsuNdie1fG23YIdERafYHWUqhiW1Yn1iIzG t2TzyXsrD60H
8oY+C98pu9mlsaYB63G4R6NDgKGfheF0TF2M5n9pAg8BIE28bcKHlf53F8Ko ttaDjZT9TSqe
CzY6GPhPY69+avnsD1BLAwQUAAAACADGg744oyIn2E4AAABgAAAAJQAAAG9y Zy5lY2xpcHNl
LmNkby5idWcvcGx1Z2luLnByb3BlcnRpZXPjUlawSc4vqCzKTM8oseMC8vSR uUABFc8UFS6u
gpzS9Mw8v8TcVAVbBRBVrOCbn5Kaw1VQlF+WmZJaBJUqLy/XS61IzC3ISdXL L0rnAgBQSwME
FAAAAAgAxoO+ONU+MTvjAAAAdgEAAB4AAABvcmcuZWNsaXBzZS5jZG8uYnVn L3BsdWdpbi54
bWxVj0FPxCAQhe/8CpzslaLxYgy0NxMPGg96NhVGliwFAtSu/17o1tXlQHjD m2/mieE4OfqF
KdvgJdx010DRq6CtNxLeXh/YHQw9EQMqZ2PGP+tttdYfIq4YI1SoEL+TNfvS V8H/KUJ3j3pH
GGve6GZjfStSKvBY0DcYjcH6IiEk021zOpw+6zsk7Ax6TGNB/R5HdRgNQt/a G2ErbPJ05mQl
7EuJ95wvy3IGVjifgkZXk3GlA/+YDffjhBku+pUbc5aw/nTP7X7Zxl7Y6lZP jSZhhZ5IbddV
Al93FPycsaXnv/F/AFBLAwQKAAAAAADKNME4AAAAAAAAAAAAAAAAGAAAAG9y Zy5lY2xpcHNl
LmNkby5idWcvc3JjL1BLAwQUAAAACAC0McE4Lk88qGcBAAChAwAAJgAAAG9y Zy5lY2xpcHNl
LmNkby5idWcvc3JjL2FwcC5wcm9wZXJ0aWVznZJdb4IwFIbv+RVNvLZuM9kF CRcodZIIOKnJ
vDKlbQZLoa5Fjfv1K/g9QZP1qj3nfU/OOU/TLOGqICWHaZK/MCYgWZfSoYqb WJcpubLSk0Sn
crvU38Ip1ZpfxKksCk7LTBZwJaVY6uyHO89PltUB3f8f4/bQyJ1PMIgRxn74 FoNRNAN+CAIU
RLMFGMfvE28APBe7AzdG0OpcNMUyIkxTjlSf8CYKa+v+3jwIU9mGqyUVROt9 DTM4S+AXS6hX
55p9ayWcSmPv9XbOc5syedK2uHQVy7mjScteTRtbqZjTnK6gUZnnWXnDhtCU w5WSm4xdz/NH
MiY6LUki+LB6Tg8Gw/AxqXvHUEQfbjCdoCPFGmKwMADO5MA8HEZBgEIM3NAD yPMxwBGY11CB
1XmMyEwP853ZeU0IHhA1G0+MaoPd6wlJiUilLu1+/+m1d8WrpcQRmNG2SM7I LvL3f2WwM1vx
i0J6g+Pn/AVQSwMEFAAAAAgAxoO+OAFLmkPZAQAA7wQAACgAAABvcmcuZWNs aXBzZS5jZG8u
YnVnL3NyYy9sb2c0ai5wcm9wZXJ0aWVzjZRBb9swDIXv/hUEAt8WDS26Yw5Z imIoUjRFdhgw
7CDbrK1UER1KTup/P8pxjTZt4x7jx/eRepQygTUGYKIAlsoSGSzu0UIgQGZi 0K4AEzyQsy3o
ukZXSJHI8wuViOVqo6J7oQOWxO2sc30TNenVvFfU7er+z+z+5iZJJiKD8eCl tZAyBA0Lcp4s
zvsOL+yXjmp+MSMu5bfOK1RH8cTTgxuPHlY6BGS31C014QOYsp3ynvnG+Kkv tt4je0OuN8zS
6RXD3zT8g3T6o4Y0h/QZppBuUyeDHUHHiFVsWplMXJLNMbFktGDy5qOqGUNo hTyZxM3Br4cl
7BrkFmrNXlak82D2Rko+J6tqZ5X2Qc3Xv/suA2/TeFlOhbB+WJ5DiHxqvb3+ uYDMyMWRUfQW
JR4PIp7DhLYeDvq1sqGdl+VtNeBzTRy+N3UhhaPtiKyqsu1lUdhTXkyyyxAC I/qR/E7debxL
Q/hjc3TVp4jA2vlIIPelLb6q/3AVjJ4azuNUu8Z4EwvP8TZFlr8GodOZxe4y PJK1dDCuBGsc
gnkEeQ1w0K57yDJI/gQFHRzk5BwejxAj6OZB/aRLeZuHCp280ki5ZiPP6E47 EXgxeFZMexP/
Z8biGxxqhNSf5z9QSwMECgAAAAAAyjTBOAAAAAAAAAAAAAAAABwAAABvcmcu ZWNsaXBzZS5j
ZG8uYnVnL3NyYy9vcmcvUEsDBAoAAAAAAMo0wTgAAAAAAAAAAAAAAAAkAAAA b3JnLmVjbGlw
c2UuY2RvLmJ1Zy9zcmMvb3JnL2VjbGlwc2UvUEsDBAoAAAAAAMo0wTgAAAAA AAAAAAAAAAAo
AAAAb3JnLmVjbGlwc2UuY2RvLmJ1Zy9zcmMvb3JnL2VjbGlwc2UvY2RvL1BL AwQKAAAAAADK
NME4AAAAAAAAAAAAAAAALAAAAG9yZy5lY2xpcHNlLmNkby5idWcvc3JjL29y Zy9lY2xpcHNl
L2Nkby9idWcvUEsDBBQAAAAIAMaDvjh5kB2zlAEAAOIDAAA9AAAAb3JnLmVj bGlwc2UuY2Rv
LmJ1Zy9zcmMvb3JnL2VjbGlwc2UvY2RvL2J1Zy9DRE9CdWdQbHVnaW4uamF2 Ya2S3WobMRCF
r7VPMSQ3tiG7D7AUQuISHEoT+nNdFO14LSJLYjRyUkrevbPW2vX2h9DSq2Wl c47mm5mozaPu
EQL1NRpnY8LadKF+yH1bVXYbA/H0MhDWlD3bLdb3LvfWt6e6kHpbr0lv8SnQ Y32VfefwOnjG
Z5bEZrGoYAGfNgjasN1pDgTG6ZTAiIiCS8ByGSX5wnpwdo1gvhqHYmuqmB+c NaPhenl3lftS
A0g8+i7B+PutqlTT7N85RK2WlRr9iTXLZ229dvCRyfoe7t99vlm9/7Jawhs4 +007ztofkWmj
CTuwXoK8QcmlAQUPwZPK4tgkJXahVyO+4CambKQBw1lzLO7UPJsLiXoZXi7O mQ/+4lbvdBfM
fH9ymRBfm1A6l8KIZ68NaD6tZBdsB8U5ke1HNchlVBSeErx9NhjZhqHvSqUc keriOygFX5VG
SHt5Y1P7H7CEKsR/hBLj3zAda/fZufYEUnJOGAtSYfqAnMmXdf5lYaBoLmkv +pOm+XljJ4vV
Iy9xrbPjsiZqDDsunFTzUn0HUEsDBBQAAAAIAIMswThuaWhuKAQAAC8MAABK AAAAb3JnLmVj
bGlwc2UuY2RvLmJ1Zy9zcmMvb3JnL2VjbGlwc2UvY2RvL2J1Zy9IYlN0b3Jl UmVwb3NpdG9y
eVByb3ZpZGVyLmphdmGtVttu4kgQfecrSjyZEbTJbjJhkt3VIGAmlpKAuMxq nlBjF7gT39Td
QNCKf99q22AugSXasRDgdp1T1VWnqm1/+nVXCT5BK05WUsx8DZZbgd/q9Wuo mZ8GdMQrwkBj
kqCswneUIY9WVeCRB7H2USpm8M0ggBSvQKJCuUCPwdAXChIZzyQPUwDZA3fd OEyIQ0QzCLlG
KXigDAeXSAsemSy4CPgkQJhHHsoURnahgnia3nTcQCQKoTefBMKFR+FiRLeL K1Y3REtfuP7W
kUBFIIrEE0pLMZlrEUfZBmix8MW1wfpaJ3e2vVwuGWZeWCxndoAzHtiYBLXF VZ35OgyMcZa5
KGONpbozC3Tt5ozyKCKhaZPQ7DmZ2zAJMMRIcxMKYX7ZZZcS7r7yGQIFvd2A 68VsMp/dl0rk
OZZ67yGG09QgLZpkrXZ3kP4baRHcXwBw+pjEStD2VxeZD8gSP0jMejJO1CUg X0xQRqQq5jxs
/j7xJCGxEcdCkJw+RqMxwpgNzff/IRSRNthgh3lLl2fkPZYI9fULI8UG7G9J XtHrvLmYGNkU
gBfScGbzwJVPwd0fP3l/1aQVpaYOITY7mwRf+Vz7sdzT8O76EydABEPOTQuQ 4LIedAOuFDxM
0s0UldvkCPCNMukpOPG89E8JIKdSpi9cmArKF9lLMyiemr2e8/x9/M157MCf UA6zEkxFgGWK
nbBSLCiXG/DpQEREJpGLxBLh8rShVcl492I6TTtD7eTMVoVwZj9As1DPZbT1 eU+L62PWRSw8
UDsE/x194eKkLdvZ6VEAX7ukSklmRTA7/QauREpmsWDlZYh4iFWSQPJHtlDN 6/MXzWelCzkV
4b1nm5hu3uQ/k+yhjcm+gaemLJlrKx0BrNfv9sbdH51+32l3xqOR065Cedq4 atDntvb5xsPa
tdvgtQbe3tSw7uLNl+svjd9vefkc4WDU63X7QyOw5qjtDAeGlE4lvBTV7/xw Bk73edzuPA6b
l8FpE863n7voy3CtUb/feR6OH/ujcavZa7ac4U8CXtXpOgtM3XTaZ4Hc87Zz KR16RU2tlDTr
CgDbhjiXECyF9vNDWunUd4bYD4VeEawDleRceZfsHT3sSIKZ9rLlVPJWpQqb mDZ9lU+BtKPO
7eWkLAvlarlKf7M7yOdRwQF6nzMX9I6TPKdwaMmCmHsWTYyWGZpWhdHfPqp4 Ll1sKgoHeWiV
bRpxrEhmubKlm9IYtrqTF3Q1vOIK7o4c0OoAtVWp5IjNHnaVkfd0hWyrRwQU UX632jUsYliX
im+Xa3rfsrZHE2BlL3Pal/ESDg8wtqQFC3PG9bnJlJZ7v/TbMp085yE8uM/q c/Yo35SM1J2L
Mp1Reyd11gfVQ/oC+dGISPeOeT0weYklFT43UzZ58uauZv4kZG9hsGnS06G9 F9O6tC79C1BL
AwQUAAAACAAgL8E4jF8EPboDAABoDAAAQQAAAG9yZy5lY2xpcHNlLmNkby5i dWcvc3JjL29y
Zy9lY2xpcHNlL2Nkby9idWcvTGF1bmNoU2VydmVyVGVzdC5qYXZhzVbLbus2 EN37KwhtKgMG
gwRd1U3bQPG9MZBYhq10VSCgqbHDXIlUScrxRZF/LynqQdnxA0VRdGPT5Jx5 nDkcuiD0G9kA
EnKDgWasUIBpKvCq3IwHA5YXQmrESQ4KR4JrwjjIcX9/Zj+/EKqF/N5hfIeQ ryun0X28BKWY
4OMzZokkXBmX50wLKbSgIrOYeb02eSpNuFYnkQrkFqRLya6eNcsuAUwXUAjF ThfLuAbJSbZX
9UmSOOgf3zpkafLBIseZ2OC5NNuPYvNAeJp5/F8A15JQcA4SuzzvokLSptd4 Os/KDeOHzT+C
ZAJP46NkeoYmufhpnhG9FjI/TiVQIQFLUKKUppJFvegAb2RLbNSqxqWWQPJx 76yK9kDU6xMp
PjmpdtvtkjON19JI+l3IbzgBpSOibLiiXGWMIpoRpdAjKTl9dcqxNgh2Gniq UAMY/DVAqJBs
SzQgI0dtoGtmOoNWQmRAOEoWd9HkJYlfoni2jB8n6BZpWdpIlwK/TCvUmmTq U5jHCKp00LBj
TH+LTeaSpVDBhAaqIUVbwVKkQD8X4RDpVyneFZrsKBT2IhpLWxRCqiyMMGqz cbXVdRJPZ8vk
bhZNMEnTTrPhnoZxXXWNZ2sU9uoaVtsuHvKzNwVzePdrCwMncr3TQe3uaEL+ HQhbP71dL9aw
dvcxOEyyyb+X5yVRDyLuUVEHM7L82WTB+GaE3PcvtlGFqgmoBb1v0zTEG1NI dstb9LBamhV0
x2Zsbo0OJN6AnlaTk0I4xNQQoD2zMLBuroORy6IO0xuftlQPsD85WkpGXkbD sSv3qLFNa5JB
DlyHweEoIdSqU0hlEgvetrn9SmFNyqwWw8cZtWsg8l6881OCdwMN00wo6Olj 7N2HzlEbtrmP
3QuAHK31r3DYhnDts8mlJdVfpSgL06yDpwPPF/H9c5S8fF3Ez/OxD9XfC3CQ g4fQopI4ih9f
ZndPkx4qBUUlq8o1YMvgT1dXNYG/dm2yb/ytE4CDS9Cl5CjsMhxe1EK/wFGV 88jPwePOjdu6
R0rPzISNRFbm3OZyqlse26r+vt3nvVWv90cD6Z0xrCFYFMC9wwbSvD+GAHsT 9a69KG4/DK7M
yXXQ6LplA9kn9dpA/L9LGFqOnJvWvIlXoeysraoOqp/NlOv7vvk3fN8EvTPT uISsMjsPzN0O
K4thIwBlj61T01flWVw3tVtyRJ4z3QRsuHUX6XivnTZ/J1n5f2/0zX/S6NNc H1D9mT9LK/qz
NKMP/RGgraU2+CfYH85hP+3xx+BvUEsDBBQAAAAIAJcrwTg9kHQ/kgIAAOcG AABIAAAAb3Jn
LmVjbGlwc2UuY2RvLmJ1Zy9zcmMvb3JnL2VjbGlwc2UvY2RvL2J1Zy9TdG9y ZVJlcG9zaXRv
cnlQcm92aWRlci5qYXZhrVTfb9owEH7PX3HqE1TU0GkPU9mmIYo6pFIifkza EzLJNfHqxJbt
gKqp//sucUSglLaa6pckl+/77vPd2d3zj1sBnMNQ6UcjktRBK2rDp17vM1yU jy8wEg8Ic4da
o+nADZqM548d4HkMyqVoLCv5Aymh4lswaNFsMGawSIUFbVRieFYRCA88ilSm SUPkCWTcoRFc
2lKDG6RATJANF5KvJUKRx2gqGuEyC+q++hhFUmiLEBZrKSK4FRHm9Lm5ZL1S aJuKKN0lEmiJ
RE5iYZ0R68IJlfsNULDJxV3JTZ3TV93udrtl6LMwZZKuxITLLmp5sbnssdRl sgT7yuVeVRl7
VQZo7deM6ihy4WiTMAjHPm2mJWaYO15aIc6HrW6gefTAEwQyvdtAFCu2LpJ+ EFBmZdzBT8zu
K0DVNMOG19N59bZ0QvbfQRjPUCsraPuP74LPCYnvQVbA/3DBQqO0bTb7h1rM qOmS/eQ2nXDd
P/5TRYOuPws/eOFSZQ66uB+fcONEDgvOyyGgkvspjCS3FirXjRnyshE0w8Hf AOgoiA0NPNiy
8dEpKI0LAfII4RvkuD0Fa7XJMWn65G9IJujGtWqrTazSDdBJdYXJd/n6FHw6 1twoEYPdE3jL
d5PgBJLt7fA4uVEOI4cx+FGByCAVrXo/9r4bEuZhk9GkRr4k2FipVZsA7cqU N1LOM+xQi/VX
H+iAf36nG8i6crSQuo+2cfISVpcjWPevnrrnGO8QPJTpwrWquWXhbBqu5ssw nM4W47ub1Wz0
azwfT+9W16PbxWDegTNnCjx7jT1czmaju8XqdrZcDQfhYDhe/CbeZY/Wq8Qq 1+j6JPGQSXd+
61lRakjdnIO7hB1V3Jf6oL0dr1437yn4B1BLAwQKAAAAAADKNME4AAAAAAAA AAAAAAAAHAAA
AG9yZy5lY2xpcHNlLmNkby5idWcvc3JjLWdlbi9QSwMECgAAAAAAyjTBOAAA AAAAAAAAAAAA
ACIAAABvcmcuZWNsaXBzZS5jZG8uYnVnL3NyYy1nZW4vbmFtZXMvUEsDBBQA AAAIAMaDvjiN
QfxktAIAAO4KAAAwAAAAb3JnLmVjbGlwc2UuY2RvLmJ1Zy9zcmMtZ2VuL25h bWVzL0NvbnRh
aW5lci5qYXZh1VZdT9swFH1ufsUdILVFa/IOWdWJ7QFpGtPgfXKS2+Dh2JHt wBDiv89fSdou
XWnhYXuqY997fH3O8XWT09MITiHNRf0oaXmr5+4zWfu2UyeXxYn5TaKa5Hek ROCkQnUeRbSq
hdQgZBljzmitMMZqGeeFiC8+XV1lPzHX51ujRFUJHjeasvjzF6pMZJSEkt7N ZpBhSfmsUShn
hchhNnPlfQSJtUSFXBNNBQexBH2LUIkCGQi3JYxTrOZpNr8QJopylGmSzdPE TI7jDh95sYnu
1mq3z43BXArGxAPlJSyR6MbsCkQiqKa2B8LizMU3zCUApIzOnxaM8jtPUNxt f1yi/mqmwNZl
B66W5zQxGS/JvSEZ88lutGf2hWBNxV26H27kp0k4QprULQsLhUHm2Narvnnl PVzAnkxdpGPe
jfCXNqQq6LR3syWaWGLo8hZqMkZzoFyjXJIcoYODP9LhKRpZR4wMzHc0AnDl tL4nrMFW+FZr
T2srMxCtJc0ajbFL3+aoUVDc/l4GJyHhVvNV/E6zFWCgio815AyJfO8ATLix h0TC2COoW9Gw
wuxprCldtQQKVLmktfOtjY3jUF0SShg2pl1ZSEfAlvMP1eehnZDHyvtvcm3W eDntV3ZI/MNl
hXgv9Ghd05EVdeRxIdh8MjVXuVPuGvWQbHvdlP303E5iTSSpQiG2Co4PB7DZ HXOYjHtBC1hn
3G+yRstOQ4er3jk69xxVpvMZjy2Nfbi5Psw0Tl+b7Vj2ywWaIN+sLOhjjTBM 9/Ob3I6+KW2r
8h+6KruL7Sl90RVxgGt3ZBX3w5GWDR4NO8U9e2kHNYe210/2s0rb1v/ulTfR euUF+Q/E3l3t
Rpf0CZNOkj1aZUg9yAj9I9i9169solse/cOt8dqmup8UKzQMM9Y22U3B+k77 DEnS/7uIfgNQ
SwMECgAAAAAAyjTBOAAAAAAAAAAAAAAAACcAAABvcmcuZWNsaXBzZS5jZG8u YnVnL3NyYy1n
ZW4vbmFtZXMvaW1wbC9QSwMEFAAAAAgAxoO+ODzLGO1dAgAAiAgAADkAAABv cmcuZWNsaXBz
ZS5jZG8uYnVnL3NyYy1nZW4vbmFtZXMvaW1wbC9Db250YWluZXJJbXBsLmph dmG9lF9v2jAQ
wJ+TT3HrKhXQkux5RAjG2IREYRpMe6wS5wgejh05Truq4rvPTkJCqLrRafAS 2xff3e98f7xe
z4Ye+ESkj5LGGzUojl7rbETX0+har56dBmQbxAg8SDBzaZKyvm3rRUhVycaC q4BylP22fG6+
X0v1RkfI2EXCaJqhi8naJSJJBHdzRZk7mdFM9V+6iURIdCdjFmTZy/YoVyh5 wFwSCXf8abEI
fyJR05Lbq8J/4zgQYky5k2conUgQcJziKUYcTIyYoA5KUcFBrEFtEBIRIQNR WIMbH5OBHw7q
0H0vHPieFt64tQPk0TPzflosK21wLRgTD5THsMZA5RIzCCQ23jH6UGjkrFAB 8BkdPA0Z5duD
ZDSvb0J8G6Myzw6Gz2wKpp3vadVXGVkFISutFLt/NTMWLE94YafcHhnyvSo6 30v3lTeMUVsI
dPxl/eUhowSIyTq0HAD+UvqNtfQwy80DHlyHJ9syubf+kPz637O8WW0qy2BZ qRRKu8SoDdXp
Gl9WlqcoO92+be3s/+56uLhHKWmEhxRlXwAuTdmS4lCxSNTVxeGwHd0Z1V0S MN28i/lqNJ1P
vl0QVbfonvNzWfpjkXPVxn1/HqCqnJZKms6r2qXtuVP+7OIXVJ2/vNrd3Xx0 O3kHSuZ4pmxX
xPeCRpBVvBU+xwdzLOlxeTruXvFMSV/maarnWfYjkFxzZp2rnJMNki1GV906 omLa+3X3DGA/
d47ScXzvxMSsRh9nF8lMM2XqiXcUQX3jRPTxYvb9dn7Jqqqwm1B0hZSy1xTX nrtRLuF34Hmt
KWn/BlBLAwQUAAAACADGg744RnFH6Z0CAAAiCAAAPAAAAG9yZy5lY2xpcHNl LmNkby5idWcv
c3JjLWdlbi9uYW1lcy9pbXBsL05hbWVzRmFjdG9yeUltcGwuamF2YbVUwU7j MBA9J18xVEik
RY3vFCqq0pV6KYjlB1xnmnrXtSPHoaAV/76246QJLeJCD1E8zmTmvTfPJqNR DCO4Zap41zzf
mqkPSS92W5fL7NK+SVxQ9pfmCJLusEz5rhCTOLYvpU3YGx02lM5TZIIXJaa4 29i10pgu5oKW
5eSbpMf1H2Tmu6ynGs53LR3OdPGLMqP0+7IH+vQPhahyLtOFC5782v5BgloX 4zGs0e6NqxL1
OFMMxmOv3EyCa4U7lIYariSoDZgtwk5lKOB2PQ0Ybsl6mrbFUGZHpe5zlKip wazWvVoLzoA5
6WDlhO6wAXwztkYJXYoHJP0f4F8cOSaRbTLXaDuUHmKGG1oJA5uQ1ieS+vyv qEdfMon6VCLH
JQpkSlea9cFxyU0ydBgjU2ONol6ChdqL7yDpxsPGEumzhVnaGuly9ftltpov 0hxNI1Ay2BpT
3BCy3+/b8VsrED8oLnPCMkXWVU68qQfDCTgkfAPJZwAXdyArIWrMUaTRVFp+ hjlx3z7i+mHU
sC0kizeGhTcJNqtQpGO7A3qh8uSQOAm1QjuJ+yNXJC7pIz6eNrUulVZ8ybDx Zxj6OaZ8DMuT
LKsC9RHEH2p9//iKWvMMWxThPgHmNUjqOwjQvwKgPfdjqfecWfyCbzjq5UMy DLNhtMSaU2O0
+ePqZbZcLZ5vIEwjCd2Gdbe5soeIy0A3isJJu/F+MVut9n5+SyEwp2Km88od u9YeyeDFTqg+
+VcDuIYDQofD6nkNgyvgJUhlgMIrFTyr8z34QWOWMwgd1G0ZwhFjr1ob+4uJ tdl3nnjva61R
ELLNPI9Nug4N04SgahMHAs1gu5+G7j5p034GYYaFRtZAPGHsh07CiWu0w+Ik gZ5xsblbauwf
QMjnwxr/B1BLAwQUAAAACADGg744I4VLk7AGAACPGAAAPAAAAG9yZy5lY2xp cHNlLmNkby5i
dWcvc3JjLWdlbi9uYW1lcy9pbXBsL05hbWVzUGFja2FnZUltcGwuamF2Yc1Y 33MaNxB+hr9i
43QmkIEj6WOcuCH4nN6MDS7gTN484k6AmuPEnIQdkvH/3l1J9wvOOK3jti82 klar3dW3366u
9/JlE17C21Cut6lYLPWJGfYqY5r6JYh+wf+95pqFX9iCQ8JWXHlitY6Pm038 J1Pt5gYy0Uwk
PD2uzg/p7xkLtUy3dUuXVnOhTqYLj4exWCvu8dUcf8uUe35f61TMNpofPyA4 iJlSDwnlpx4W
G/M5T3kSPmgeRSTXGtjw9FyUn3W7MOMLkXQ3iqfdSIbQ7ZqI9xOgjXzFMXZa yATkHPSSw0pG
PIa3sxOn8W1vduLlyngS7al6v+AYe6Z5ZO9rM4tFCCHFAspxJtuAf9WoQ0HZ 4MKS6gb43myQ
J40DrjTutaxRNa1BtjXWqbjBIdirgjBDjhu/g2QTUwDzcwcpR3kFDOOVKM3w Qg5FqgMp2qg0
Xl0Et0IvjZLv72ORfHkAEN7Y7Ey3sDdzB7OtOdLlglGa5cXVOIAbFm+4Z4Ox PhlKzd8YeTwg
5aGGW4b7JYTGmbImEApuBDNTioAQGiVzmzSw4nopo8z+5yIRGuhPq33Xgdul CJfAYiVhzdO5
TFfKbCYBwWLxrYIrd2AHo4Ax0ps0UWa+FK9MxGqZg0w4qkeboy3iBqWU9xgs KM5/9AqKDft0
8ZwPJ9cY9ELmuQ3JYcTtZkKrTfBuqA3GruVUdqBMWR4PhpNpfzjw28fNxl3z yZLBXjzMpIw5
4VwF6A/exzvEQax4TToUMMefLImKO+f2VqtZAQgOnBYWHyZv7DaaZ8kWJO5J FWzWiBcLKwRa
xNdEFR7AxBCE2WwnkRcFJWWKxqPtai7QXMyRkMWI1IU5KwMvqsTZymadQU1h FNEGDpGkbWgK
7sQfHN3Gc6clNREGi9cCnJwQCcaiRn0kKP/ibQclcDZFS7i1hcjB2plnhtCK x/M8jUfzyrqN
M1Jk5SyKQ4ccLM40ATRKluwGESw1bLlGT3lSzjZLKSItCMQGdC5SpR1VGKSY g4uNJi68OI72
oKKkhAHCA+hbicjia/XG3jrXrGspU87+xKAoe/toeZUhrEZ3fnY/eDPWFPxd OqcDShhGTvLt
L1Td
Re: [CDO] reading/writting throughput performances [message #124519 is a reply to message #124496] Sun, 01 June 2008 08:17 Go to previous messageGo to next message
Eclipse UserFriend
Originally posted by: stepper.sympedia.de

This is a multi-part message in MIME format.
--------------000609090607080808010708
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi,

Here is a diagram of the dependencies of the relevant plugins (please
correct me if something is wrong).
The black arrows are ordinary downstream dependencies (Require-Bundle).
The red arrows are upstream dependencies (Eclipse-BuddyPolicy: dependent).



For research purposes I added code to the libraries plugin that tries to
reflectively load CDOFeatureImpl and that is not a problem!
It seems to me as if our problem is not class loading related because
the hibernate.jar should be able to see the needed classes.
I believe this is either a bug in Hibernate or our wrong usage of
Hibernate. I tend to believe the latter ;-)
I put a breakpoint in
org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping .EntityEntityModeToTuplizerMapping(PersistentClass,
EntityMetamodel)
to see how the tuplizers are created. I'm too unexperienced with
Hibernate to get a clue about what happens there...

Btw. I'd prefer to see the application plugin depend on net4j.db instead
of net4j.db.hsqldb.
But it seems as if the BuddyPolicy in net4j.db is not "reexported" so
that the application can see the hsqldb.jar then ;-(
I posted a respective question to equinox...

Cheers
/Eike


Eike Stepper schrieb:
> Hi guys,
>
> I can confirm, making these 3rd party libraries with their own
> classloading run in OSGi is a nightmare! I took me hours teach where
> the hsqldb driver is. On the way I came over different kinds of
> classloading related errors.
>
> First I removed all the jars from the test plugin (the one Cedric
> provided). They're already in the cdo.server.hibernate.libraries
> plugin which, in fact, needs the buddy policy set to dependent. I
> think this last change needs to be committed to CVS! Btw. I'd prefer
> to use *registered* buddies instead of making all dependent bundles
> visible.
>
> The test plugin then needs to provide the database driver. Therefore I
> added a dependency on the org.eclipse.net4j.db.hsqldb plugin.
>
> With this setup I can start the server without CNFE, NCDFE, CCI and
> the like. At this point I get the following exception, which seems the
> same as the one Cedric observed in the end:
>
> net.sf.cglib.core.CodeGenerationException:
> java.lang.reflect.InvocationTargetException-->null
> at
> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:237)
>
> at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
> at net.sf.cglib.proxy.Enhancer.createClass(Enhancer.java:317)
> at
> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:127)
>
> at
> org.hibernate.proxy.pojo.cglib.CGLIBProxyFactory.postInstant iate(CGLIBProxyFactory.java:43)
>
> at
> org.hibernate.tuple.entity.PojoEntityTuplizer.buildProxyFact ory(PojoEntityTuplizer.java:162)
>
> at
> org.hibernate.tuple.entity.AbstractEntityTuplizer.<init>(AbstractEntityTuplizer.java:135)
>
> at
> org.hibernate.tuple.entity.PojoEntityTuplizer.<init>(PojoEntityTuplizer.java:55)
>
> at
> org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping . <init>(EntityEntityModeToTuplizerMapping.java:56)
>
> at
> org.hibernate.tuple.entity.EntityMetamodel.<init>(EntityMetamodel.java:295)
>
> at
> org.hibernate.persister.entity.AbstractEntityPersister.<init >(AbstractEntityPersister.java:434)
>
> at
> org.hibernate.persister.entity.JoinedSubclassEntityPersister . <init>(JoinedSubclassEntityPersister.java:91)
>
> at
> org.hibernate.persister.PersisterFactory.createClassPersiste r(PersisterFactory.java:58)
>
> at
> org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:226)
> at
> org.hibernate.cfg.Configuration.buildSessionFactory(Configur ation.java:1294)
>
> at
> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getSessionFactory(HibernatePackageHandler.java:335 )
>
> at
> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.readPackageInfos(HibernatePackageHandler.java:293)
>
> at
> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getCDOPackageInfos(HibernatePackageHandler.java:21 5)
>
> at
> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Reader.readPackageInfos(HibernateStoreReader.java:87)
>
> at
> org.eclipse.emf.cdo.internal.server.PackageManager.doActivat e(PackageManager.java:72)
>
> at
> org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
>
> at
> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>
> at
> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>
> at
> org.eclipse.emf.cdo.internal.server.Repository.activateRepos itory(Repository.java:264)
>
> at
> org.eclipse.emf.cdo.internal.server.Repository.doActivate(Re pository.java:251)
>
> at
> org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
>
> at
> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>
> at
> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>
> at
> org.eclipse.emf.cdo.server.CDOServerUtil.addRepository(CDOSe rverUtil.java:67)
>
> at
> org.eclipse.cdo.bug.LaunchServerTest.setUp(LaunchServerTest. java:54)
> at junit.framework.TestCase.runBare(TestCase.java:128)
> at junit.framework.TestResult$1.protect(TestResult.java:106)
> at junit.framework.TestResult.runProtected(TestResult.java:124)
> at junit.framework.TestResult.run(TestResult.java:109)
> at junit.framework.TestCase.run(TestCase.java:120)
> at junit.framework.TestSuite.runTest(TestSuite.java:230)
> at junit.framework.TestSuite.run(TestSuite.java:225)
> at
> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
>
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
>
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
>
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
>
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
>
> at
> org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
>
> at
> org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:585)
> at
> org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hodWithException(EclipseAppContainer.java:574)
>
> at
> org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
>
> at
> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:110)
>
> at
> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:79)
>
> at
> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:379)
>
> at
> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:179)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 549)
> at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
> at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
> at org.eclipse.equinox.launcher.Main.main(Main.java:1212)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:585)
> at net.sf.cglib.core.ReflectUtils.defineClass(ReflectUtils.java :384)
> at
> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>
> ... 61 more
> Caused by: java.lang.NoClassDefFoundError:
> org/hibernate/proxy/HibernateProxy
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
> ... 67 more
>
> I must admit that I'm still not sure about te fundamental interwork
> between Hibernate and the user defined meta model. It seems as if asm
> and cglib are somewhere in between but I have no idea why at all. I
> thought we have implemented a bunch of static Hibernate interfaces to
> make the CDORevision instances known to Hibernate. Please somebody
> explain why we need runtime byte code engineering? Is there some
> configuration missing to tell Hibernate to use our special tuplizers?
>
> I have changed several things in the code of the test plugin and since
> it's quite small now that the jars are removed, I attach my version of
> the zip here (including a JUnit Plugin launch config). And don't
> forget to update the cdo.server.hibernate.libraries plugin from CVS to
> get the new buddy policy!
>
> Cheers
> /Eike
>
> Martin Taal schrieb:
>> Hi Cédric,
>> I looked at it for many hours trying different things (using your
>> testcase) but I am kind of stuck with this (getting the same error as
>> you). The difference I can see with the way I run the cdo.hibernate
>> testcases (as a junit test) and how your testcase is run (junit
>> plugin test). This for sure means difference in class loader behavior.
>>
>> Eike, do you have an idea? I am not sure how classloading works when
>> you start a cdo server in a plugin. Which plugins are then available
>> in the cdo server? The testcases can be downloaded from the link below.
>>
>> gr. Martin
>>
>> Cédric Brun wrote:
>>> Thanks a lot ! I'm looking forward to your advices !
>>>
>>> Cédric
>>>
>>> Martin Taal wrote:
>>>
>>>> I am downloading your zip-file. I don't have time to look at it
>>>> right away
>>>> but I will do my best later this evening.
>>>>
>>>> gr. Martin
>>>>
>>>> Cédric Brun wrote:
>>>>> I tried to do the same things as in the CDO/Hibernate unit tests.
>>>>>
>>>>> Here's a zipped project trying to launch an HibernateServer with a
>>>>> unit
>>>>> test adding elements.
>>>>> ( http://cedric.brun.free.fr/Tortoose/org.eclipse.cdo-teneo-te st.zip)
>>>>> The
>>>>> libs I found are included in the same project, but I still have no
>>>>> luck
>>>>> and failling at :
>>>>>
>>>>> at
>>>>>
>>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>>
>>>>> ... 62 more
>>>>> Caused by: java.lang.NoClassDefFoundError:
>>>>> org/hibernate/proxy/HibernateProxy
>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>>>>> ... 68 more
>>>>>
>>>>> please note that the test project I provide really is a "quick and
>>>>> ugly"
>>>>> one ^^
>>>>>
>>>>>
>>>>> Cédric
>>>>>
>>>>>
>>>>>
>>>>> Martin Taal wrote:
>>>>>
>>>>>> Hmm, I can understand your headache. Classpath issues are the most
>>>>>> annoying errors...
>>>>>>
>>>>>> In your junit test source file, what happens when you enter the
>>>>>> org.objectweb.asm.Type, can you source file see this class?
>>>>>> Do you have cglib in some other plugin which does not have asm?
>>>>>>
>>>>>> What I also find slightly strange is that cglib is used in this case
>>>>>> because normally cdo objects are not proxied using cglib. Can you
>>>>>> post
>>>>>> the generated mapping? How does your cdo initialization code look
>>>>>> like?
>>>>>>
>>>>>> gr. Martin
>>>>>>
>>>>>> Cédric Brun wrote:
>>>>>>> Hi Martin,
>>>>>>>
>>>>>>> and thanks for your quick answer ! what I did is the following :
>>>>>>>
>>>>>>> I used the org.eclipse.emf.cdo.server.hibernate.libraries bundle
>>>>>>> and
>>>>>>> added that :
>>>>>>>
>>>>>>> Bundle-ClassPath: lib/hibernate3.jar,
>>>>>>> lib/cglib-2.1.3.jar,
>>>>>>> lib/commons-collections-2.1.1.jar,
>>>>>>> lib/dom4j.jar,
>>>>>>> lib/jta.jar
>>>>>>> lib/asm-3.0.jar
>>>>>>>
>>>>>>> and this directive :
>>>>>>>
>>>>>>> Eclipse-BuddyPolicy: dependent
>>>>>>>
>>>>>>> as stated in
>>>>>>> http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>>>>>>>
>>>>>>>
>>>>>>> Then I don't get the same error, now when I start my own plugin
>>>>>>> (JUnit
>>>>>>> plugin test) depending on
>>>>>>> org.eclipse.emf.cdo.server.hibernate.libraries I get :
>>>>>>>
>>>>>>> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
>>>>>>> at
>>>>>>> net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
>>>>>>> at
>>>>>>> net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
>>>>>>> at net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
>>>>>>> at
>>>>>>>
>>> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>>>
>>>>>>> These jars are such a headache ! Do you know - by chance - what's
>>>>>>> going on ?
>>>>>>>
>>>>>>>
>>>>>>> Cédric
>>>>>>>
>>>>>>>
>>>>>>> Martin Taal wrote:
>>>>>>>
>>>>>>>> Hi Cédric,
>>>>>>>> Most of the time this exception is thrown because hibernate
>>>>>>>> can't find
>>>>>>>> the class. A common cause is that the hibernate jar files are
>>>>>>>> located
>>>>>>>> in a separate plugin which can not reach the classes in other
>>>>>>>> plugins.
>>>>>>>> If you set the eclipse buddy policy of the hibernate plugin to
>>>>>>>> dependent then this can help.
>>>>>>>>
>>>>>>>> gr. Martin
>>>>>>>>
>>>>>>>> Cédric Brun wrote:
>>>>>>>>> Hi Simon and Eike,
>>>>>>>>>
>>>>>>>>> thanks for your feedback, now I know what I could expect in using
>>>>>>>>> optimized implementation, I'll have a try on the hibernate
>>>>>>>>> store and
>>>>>>>>> see if it is scaling in a better way.
>>>>>>>>>
>>>>>>>>> You gave me many pointers to explain the performances issues
>>>>>>>>> but in
>>>>>>>>> fact it seems that the SQL execution is taking so much time (not
>>>>>>>>> suprisingly in fact ;) . My first profile was wrong.
>>>>>>>>>
>>>>>>>>> So I'm installing the hibernate store retrieving all the
>>>>>>>>> source code
>>>>>>>>> from CVS, and I'm trying to launch the unit tests. I added
>>>>>>>>> many jars
>>>>>>>>> and everything gets compiled but most of the tests are
>>>>>>>>> failling with
>>>>>>>>> "
>>>>>>>>>
>>>>>>>>> "
>>>>>>>>> org.hibernate.MappingException: Could not determine type for:
>>>>>>>>>
>>> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>>>>>>>> at table: cdo_features, for columns:
>>>>>>>>> [org.hibernate.mapping.Column(type)] "
>>>>>>>>>
>>>>>>>>> any hint ?
>>>>>>>>>
>>>>>>>>> Cédric
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Simon McDuff wrote:
>>>>>>>>>
>>>>>>>>>> We noticed a performance degradation in the following case:
>>>>>>>>>>
>>>>>>>>>> When we fetch a collection of objects that contains
>>>>>>>>>> collections that
>>>>>>>>>> we don't want to use. Ex:
>>>>>>>>>>
>>>>>>>>>> Model:
>>>>>>>>>>
>>>>>>>>>> ClassB
>>>>>>>>>> {
>>>>>>>>>> int foo1;
>>>>>>>>>> List childrens1;
>>>>>>>>>> List childrens2;
>>>>>>>>>> List childrens3;
>>>>>>>>>> };
>>>>>>>>>>
>>>>>>>>>> Code:
>>>>>>>>>>
>>>>>>>>>> Resource.contents contains objects of type ClassB.
>>>>>>>>>>
>>>>>>>>>> By fetching instances of classB, we will also fetched some
>>>>>>>>>> items in
>>>>>>>>>> the list(childrens1, childrens2, childrens3) The problem we
>>>>>>>>>> had.. is
>>>>>>>>>> we do not know if we want to use it. SO better not to
>>>>>>>>>> download them.
>>>>>>>>>>
>>>>>>>>>> So In our production(we used our own IStore), we never
>>>>>>>>>> download any
>>>>>>>>>> objects from collection. I'm working at the moment to put in
>>>>>>>>>> place
>>>>>>>>>> all feature we used in the current CDO. That one will be part of
>>>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477. To begin
>>>>>>>>>> with,CDORevisionImpl needs to supports list that are 0 even if
>>>>>>>>>> referenceChunkSize == 0.
>>>>>>>>>>
>>>>>>>>>> It is true that we obtain good performance.
>>>>>>>>>> Worst case :
>>>>>>>>>> 1500 objects /sec (Real fetch from disk. Page that contains the
>>>>>>>>>> objects, wasn't in the cache of the filesystem.)
>>>>>>>>>>
>>>>>>>>>> Best case
>>>>>>>>>> 12 000 objects/Sec (Fetch objects that are already in the
>>>>>>>>>> case of
>>>>>>>>>> the filesystem)
>>>>>>>>>>
>>>>>>>>>> (These statistics always fetch objects from server to client.
>>>>>>>>>> The
>>>>>>>>>> cache client isn't used)
>>>>>>>>>>
>>>>>>>>>> We have another optimization one with SmartReadAhead but for now
>>>>>>>>>> only work
>>>>>>>>>> for UI client. (for now) This is something we will integrate as
>>>>>>>>>> well in CDO. But for now we concentrate on the other things.
>>>>>>>>>>
>>>>>>>>>> I used an Object database. Based on my preliminary test
>>>>>>>>>> without CDO.
>>>>>>>>>> I compare
>>>>>>>>>>
>>>>>>>>>> Hibernate vs Objectivity/EMF :
>>>>>>>>>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>>>>>>>>>
>>>>>>>>>> The worst.. you will get 30 to 240 objects/sec or
>>>>>>>>>> the best... you will get 500 to 4000 objects/sec
>>>>>>>>>>
>>>>>>>>>> Based on your comments... you are getting the worst (with
>>>>>>>>>> Derby)!!
>>>>>>>>>> :-)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Another comments about insertion.
>>>>>>>>>>
>>>>>>>>>> We found another problem when we insert data(DOn't know if it is
>>>>>>>>>> still true). Resource.Contents is a unique list. Basically
>>>>>>>>>> each time
>>>>>>>>>> we insert one element, it needs to fetch all the collection
>>>>>>>>>> to see
>>>>>>>>>> if we already have this element. Basically.. it is time
>>>>>>>>>> consuming.
>>>>>>>>>> In our case we are not going through this stage to persist our
>>>>>>>>>> objects... but the objects are not accessible through that list.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> If speed is an issue for you!! We (You included :-)) can
>>>>>>>>>> optimize
>>>>>>>>>> it. What I like is Eike did a really good job doing that
>>>>>>>>>> framwork
>>>>>>>>>> and optimizations are easy to add. Also you can change your
>>>>>>>>>> back-end
>>>>>>>>>> without affecting you client side. (Maybe going to an Object
>>>>>>>>>> database!!) THis was one of the purpose of CDO!! :-)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Simon
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le
>>>>>>>>>> message de
>>>>>>>>>> news: 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>>>>>>>
>>>>>>>>>> That sounds interesting!
>>>>>>>>>> Comments inline...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Cédric Brun schrieb:
>>>>>>>>>> Hi Eike,
>>>>>>>>>>
>>>>>>>>>> I've been messing around CDO lately and I hacked a few ugly
>>>>>>>>>> unit-tests measuring read/write throughput when I use CDO (RC2).
>>>>>>>>>>
>>>>>>>>>> I'm using an embeded-derby database, two different metamodels
>>>>>>>>>> (complex one - 250 EClassifiers/300 features and a simple one
>>>>>>>>>> - 1
>>>>>>>>>> Classifier/3 Features)
>>>>>>>>>>
>>>>>>>>>> The test is a "quick and dirty" one so please do not take these
>>>>>>>>>> figures too seriously, I would just like to have your input
>>>>>>>>>> about
>>>>>>>>>> the "order of magnitude" I should expect or settings affecting
>>>>>>>>>> seriously these performances. I tried both JVM and TCP
>>>>>>>>>> acceptor but
>>>>>>>>>> the performances are quite similar though JVM is a bit faster.
>>>>>>>>>> Looks like a sign that the NIO socket channels are
>>>>>>>>>> efficient ,-)
>>>>>>>>>> You used localhost for TCP transport I guess?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> * First case : building a small model (10000 elements) with the
>>>>>>>>>> simple MM, only adding instances in an empty database :
>>>>>>>>>> commiting change on every add I get a 2 objects per second rate.
>>>>>>>>>> commiting changes every 10 adds - I get a 30 objects per second
>>>>>>>>>> rate.
>>>>>>>>>> Seems to be consistent with the fact that only for commits
>>>>>>>>>> there
>>>>>>>>>> is remote traffic. And the backend transaction management is
>>>>>>>>>> another overhead. So the impact of the number objects/commit
>>>>>>>>>> starts with bigger transactions.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> browsing the just commited elements opening a new transaction
>>>>>>>>>> and
>>>>>>>>>> getting a new Resource instance, I get around 30 000
>>>>>>>>>> objets/sec rate
>>>>>>>>>> just iterating thanks to Resource.getAllContents().
>>>>>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>>>>>> All views and transactions of a single session share (and
>>>>>>>>>> cache)
>>>>>>>>>> their object state. Otherwise I'd expect much smaller values.
>>>>>>>>>> With his ObjectivityStore implementation Simon (cc'ed) reached
>>>>>>>>>> rates of several thousands objects/sec. But the DBStore
>>>>>>>>>> (which you
>>>>>>>>>> used) is not that optimized.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> * Second case : I built a big model in the database using the
>>>>>>>>>> complex metamodel. It contains 700 000 elements. The creation
>>>>>>>>>> went
>>>>>>>>>> pretty nicely (I did not measured it exactly as it was launched
>>>>>>>>>> during a night, but it seems I got around 100 objects/sec
>>>>>>>>>> rate in
>>>>>>>>>> writting as I was writting every 700 objects.
>>>>>>>>>> Sounds not too bad.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Once the model is built the derby database is 222 MB (which - I
>>>>>>>>>> heard - is not so big for derby).
>>>>>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>>>>>> To be honest, I'm really no database expert (although I can
>>>>>>>>>> write
>>>>>>>>>> simple SQL statements).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Retrieving the distant resource and calling eAllContents() on
>>>>>>>>>> it, I
>>>>>>>>>> get a 50 Objects/sec rate at best, which mean my
>>>>>>>>>> eAllContents() call
>>>>>>>>>> would take 4 hours !
>>>>>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>>>>>> (iteration order and so). There are several CDO features
>>>>>>>>>> that can
>>>>>>>>>> have a major impact on load rates!
>>>>>>>>>>
>>>>>>>>>> - Collections (xrefs and containment) can be loaded lazily. I
>>>>>>>>>> don't mean the target objects, which are lazy by default
>>>>>>>>>> anyway,
>>>>>>>>>> but even the target IDs that are in the collections. We
>>>>>>>>>> call that
>>>>>>>>>> "reference chunking" (but search for a better name is
>>>>>>>>>> ongoing).
>>>>>>>>>>
>>>>>>>>>> public interface CDOSession extends CDOProtocolSession,
>>>>>>>>>> IContainer<CDOView>
>>>>>>>>>> {
>>>>>>>>>> public int getReferenceChunkSize();
>>>>>>>>>>
>>>>>>>>>> public void setReferenceChunkSize(int
>>>>>>>>>> referenceChunkSize);
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> - Loading collection elements (target objects) can be
>>>>>>>>>> batched (the
>>>>>>>>>> current name is even worse)
>>>>>>>>>>
>>>>>>>>>> public interface CDOView extends CDOProtocolView,
>>>>>>>>>> INotifier
>>>>>>>>>> {
>>>>>>>>>> public int getLoadRevisionCollectionChunkSize();
>>>>>>>>>>
>>>>>>>>>> public void setLoadRevisionCollectionChunkSize(int
>>>>>>>>>> loadRevisionCollectionChunkSize);
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> - One of the most sophisticated feature is automatic model
>>>>>>>>>> usage
>>>>>>>>>> analyzation. The result are fetch rules that are sent to the
>>>>>>>>>> server. When later objects are loaded from the server, it can
>>>>>>>>>> answer with additional objects that are likely to be needed
>>>>>>>>>> by the
>>>>>>>>>> client in the near future.
>>>>>>>>>>
>>>>>>>>>> public class CDOViewImpl extends
>>>>>>>>>> org.eclipse.net4j.internal.util.event.Notifier
>>>>>>>>>> implements
>>>>>>>>>> CDOView, CDOIDProvider, Adapter.Internal
>>>>>>>>>> {
>>>>>>>>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>>>>>>> {
>>>>>>>>>> return featureAnalyzer;
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>>>>>>> featureAnalyzer)
>>>>>>>>>> {
>>>>>>>>>> this.featureAnalyzer = featureAnalyzer == null ?
>>>>>>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> - There's a SmartReadAheadThread (if you're curious, please
>>>>>>>>>> ask
>>>>>>>>>> Simon). - Caches can be configured
>>>>>>>>>> - ...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Known issues of the DBStore:
>>>>>>>>>> - When in auditing mode updates are always written as complete
>>>>>>>>>> inserts (by design) - When configuring toManyReferences to
>>>>>>>>>> a value
>>>>>>>>>> different from LIKE_ATTRIBUTES (which is not implemented
>>>>>>>>>> yet) it
>>>>>>>>>> is very slow. - No (few) prepared statements are used (see
>>>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe
>>>>>>>>>> others...
>>>>>>>>>>
>>>>>>>>>> Please note that the DBStore was not written by me to
>>>>>>>>>> compete with
>>>>>>>>>> Hibernate in terms of performance or mapping capabilities. Now
>>>>>>>>>> that Martin and I, well in the end mostly Martin, provide a
>>>>>>>>>> HibernateStore and this HibernateStore gets more and more
>>>>>>>>>> features
>>>>>>>>>> I expect that there will be a trend away from the DBStore.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This is probably linked to the way the mapping to db is done,
>>>>>>>>>> here's
>>>>>>>>>> the config I used (came from CDO-server example if I recall
>>>>>>>>>> correctly).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <repository name="repo1">
>>>>>>>>>> <property name="overrideUUID"
>>>>>>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>>>>>>> <property name="supportingAudits" value="true"/>
>>>>>>>>>> <property name="verifyingRevisions"
>>>>>>>>>> value="false"/>
>>>>>>>>>> <property name="currentLRUCapacity"
>>>>>>>>>> value="10000"/>
>>>>>>>>>> I guess this one could be larger.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <property name="revisedLRUCapacity"
>>>>>>>>>> value="100"/>
>>>>>>>>>> <store type="db">
>>>>>>>>>> <!-- type: horizontal | vertical | <any
>>>>>>>>>> user-contributed type> -->
>>>>>>>>>> <mappingStrategy
>>>>>>>>>> type="horizontal">
>>>>>>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>>>>>>> ONE_TABLE_PER_CLASS |
>>>>>>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY |
>>>>>>>>>> LIKE_ATTRIBUTES
>>>>>>>>>> -->
>>>>>>>>>> <property
>>>>>>>>>> name="toManyReferences"
>>>>>>>>>>
>>>>>>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>>>>>>
>>>>>>>>>> <!-- LIKE_ATTRIBUTES |
>>>>>>>>>> LIKE_TO_MANY_REFERENCES -->
>>>>>>>>>> <property name="toOneReferences"
>>>>>>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>>>>>>
>>>>>>>>>> <!-- MODEL | STRATEGY -->
>>>>>>>>>> <property
>>>>>>>>>> name="mappingPrecedence"
>>>>>>>>>> value="MODEL"/>
>>>>>>>>>> This one is obsolete.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> </mappingStrategy>
>>>>>>>>>> <dbAdapter
>>>>>>>>>> name="derby-embedded"/>
>>>>>>>>>> <dataSource
>>>>>>>>>>
>>>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>>>>>>>> databaseName="/tmp/cdodb1"
>>>>>>>>>> createDatabase="create"/>
>>>>>>>>>>
>>>>>>>>>> </store>
>>>>>>>>>> </repository>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> What do you think about that ? What throughput do you think I
>>>>>>>>>> should
>>>>>>>>>> expect ? As I said that's very difficult to judge.
>>>>>>>>>> I believe it's more appropriate to compare the actual
>>>>>>>>>> values with
>>>>>>>>>> values from other measures (other setups, properties, etc.
>>>>>>>>>> or even
>>>>>>>>>> other products) instead of random expectations. Apologies
>>>>>>>>>> that I
>>>>>>>>>> didn't have the time so far to provide some reference measures
>>>>>>>>>> myself. Maybe this gets better now that the team starts to
>>>>>>>>>> grow
>>>>>>>>>> slowly ;-)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> What are the main reasons fur such figures ?
>>>>>>>>>> Hmm, I *suspect* you don't like the figures? I can find no
>>>>>>>>>> evidence your assessment ;-)
>>>>>>>>>>
>>>>>>>>>> Ahh,one more note:
>>>>>>>>>> Tracing can also have a significant impact on performance and
>>>>>>>>>> end-to-end throughput. Turn it off by all means. And please
>>>>>>>>>> make
>>>>>>>>>> sure that it's really turned off, not only invisible (i.e.
>>>>>>>>>> console
>>>>>>>>>> is not a target).
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>> /Eike
>>>>
>>>
>>
>>

--------------000609090607080808010708
Content-Type: multipart/related;
boundary="------------040900020203060703010001"


--------------040900020203060703010001
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi,<br>
<br>
Here is a diagram of the dependencies of the relevant plugins (please
correct me if something is wrong).<br>
The black arrows are ordinary downstream dependencies (Require-Bundle).<br>
The red arrows are upstream dependencies (Eclipse-BuddyPolicy:
dependent).<br>
<br>
<img src="cid:part1.07020404.09020102@sympedia.de" alt=""><br>
<br>
For research purposes I added code to the libraries plugin that tries
to reflectively load CDOFeatureImpl and that is not a problem!<br>
It seems to me as if our problem is not class loading related because
the hibernate.jar should be able to see the needed classes.<br>
I believe this is either a bug in Hibernate or our wrong usage of
Hibernate. I tend to believe the latter ;-)<br>
I put a breakpoint in
org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping .EntityEntityModeToTuplizerMapping(PersistentClass,
EntityMetamodel) <br>
to see how the tuplizers are created. I'm too unexperienced with
Hibernate to get a clue about what happens there...<br>
<br>
Btw. I'd prefer to see the application plugin depend on net4j.db
instead of net4j.db.hsqldb.<br>
But it seems as if the BuddyPolicy in net4j.db is not "reexported" so
that the application can see the hsqldb.jar then ;-(<br>
I posted a respective question to equinox...<br>
<br>
Cheers<br>
/Eike<br>
<br>
<br>
Eike Stepper schrieb:
<blockquote cite="mid:g1t99s$tla$1@build.eclipse.org" type="cite">Hi
guys,
<br>
<br>
I can confirm, making these 3rd party libraries with their own
classloading run in OSGi is a nightmare! I took me hours teach where
the hsqldb driver is. On the way I came over different kinds of
classloading related errors.
<br>
<br>
First I removed all the jars from the test plugin (the one Cedric
provided). They're already in the cdo.server.hibernate.libraries plugin
which, in fact, needs the buddy policy set to dependent. I think this
last change needs to be committed to CVS! Btw. I'd prefer to use
*registered* buddies instead of making all dependent bundles visible.
<br>
<br>
The test plugin then needs to provide the database driver. Therefore I
added a dependency on the org.eclipse.net4j.db.hsqldb plugin.
<br>
<br>
With this setup I can start the server without CNFE, NCDFE, CCI and the
like. At this point I get the following exception, which seems the same
as the one Cedric observed in the end:
<br>
<br>
net.sf.cglib.core.CodeGenerationException:
java.lang.reflect.InvocationTargetException--&gt;null
<br>
   at
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:237)
<br>
   at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
<br>
   at net.sf.cglib.proxy.Enhancer.createClass(Enhancer.java:317)
<br>
   at
org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:127)
<br>
   at
org.hibernate.proxy.pojo.cglib.CGLIBProxyFactory.postInstant iate(CGLIBProxyFactory.java:43)
<br>
   at
org.hibernate.tuple.entity.PojoEntityTuplizer.buildProxyFact ory(PojoEntityTuplizer.java:162)
<br>
   at
org.hibernate.tuple.entity.AbstractEntityTuplizer.&lt;in it&gt;(AbstractEntityTuplizer.java:135)
<br>
   at
org.hibernate.tuple.entity.PojoEntityTuplizer.&lt;init&a mp;gt;(PojoEntityTuplizer.java:55)
<br>
   at
org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping .&lt;init&gt;(EntityEntityModeToTuplizerMapping.java :56)
<br>
   at
org.hibernate.tuple.entity.EntityMetamodel.&lt;init& gt;(EntityMetamodel.java:295)
<br>
   at
org.hibernate.persister.entity.AbstractEntityPersister.& lt;init&gt;(AbstractEntityPersister.java:434)
<br>
   at
org.hibernate.persister.entity.JoinedSubclassEntityPersister .&lt;init&gt;(JoinedSubclassEntityPersister.java:91)
<br>
   at
org.hibernate.persister.PersisterFactory.createClassPersiste r(PersisterFactory.java:58)
<br>
   at
org.hibernate.impl.SessionFactoryImpl.&lt;init&gt;(S essionFactoryImpl.java:226)
<br>
   at
org.hibernate.cfg.Configuration.buildSessionFactory(Configur ation.java:1294)
<br>
   at
org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getSessionFactory(HibernatePackageHandler.java:335 )
<br>
   at
org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.readPackageInfos(HibernatePackageHandler.java:293)
<br>
   at
org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getCDOPackageInfos(HibernatePackageHandler.java:21 5)
<br>
   at
org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Reader.readPackageInfos(HibernateStoreReader.java:87)
<br>
   at
org.eclipse.emf.cdo.internal.server.PackageManager.doActivat e(PackageManager.java:72)
<br>
   at
org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
<br>
   at
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
<br>
   at
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
<br>
   at
org.eclipse.emf.cdo.internal.server.Repository.activateRepos itory(Repository.java:264)
<br>
   at
org.eclipse.emf.cdo.internal.server.Repository.doActivate(Re pository.java:251)
<br>
   at
org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
<br>
   at
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
<br>
   at
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
<br>
   at
org.eclipse.emf.cdo.server.CDOServerUtil.addRepository(CDOSe rverUtil.java:67)
<br>
   at
org.eclipse.cdo.bug.LaunchServerTest.setUp(LaunchServerTest. java:54)
<br>
   at junit.framework.TestCase.runBare(TestCase.java:128)
<br>
   at junit.framework.TestResult$1.protect(TestResult.java:106)
<br>
   at junit.framework.TestResult.runProtected(TestResult.java:124)
<br>
   at junit.framework.TestResult.run(TestResult.java:109)
<br>
   at junit.framework.TestCase.run(TestCase.java:120)
<br>
   at junit.framework.TestSuite.runTest(TestSuite.java:230)
<br>
   at junit.framework.TestSuite.run(TestSuite.java:225)
<br>
   at
org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
<br>
   at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
<br>
   at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
<br>
   at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
<br>
   at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
<br>
   at
org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
<br>
   at
org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
<br>
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
<br>
   at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
<br>
   at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
<br>
   at java.lang.reflect.Method.invoke(Method.java:585)
<br>
   at
org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hodWithException(EclipseAppContainer.java:574)
<br>
   at
org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
<br>
   at
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:110)
<br>
   at
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:79)
<br>
   at
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:379)
<br>
   at
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:179)
<br>
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
<br>
   at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
<br>
   at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
<br>
   at java.lang.reflect.Method.invoke(Method.java:585)
<br>
   at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 549)
<br>
   at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
<br>
   at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
<br>
   at org.eclipse.equinox.launcher.Main.main(Main.java:1212)
<br>
Caused by: java.lang.reflect.InvocationTargetException
<br>
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
<br>
   at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
<br>
   at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
<br>
   at java.lang.reflect.Method.invoke(Method.java:585)
<br>
   at net.sf.cglib.core.ReflectUtils.defineClass(ReflectUtils.java :384)
<br>
   at
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
<br>
   ... 61 more
<br>
Caused by: java.lang.NoClassDefFoundError:
org/hibernate/proxy/HibernateProxy
<br>
   at java.lang.ClassLoader.defineClass1(Native Method)
<br>
   at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
<br>
   ... 67 more
<br>
<br>
I must admit that I'm still not sure about te fundamental interwork
between Hibernate and the user defined meta model. It seems as if asm
and cglib are somewhere in between but I have no idea why at all. I
thought we have implemented a bunch of static Hibernate interfaces to
make the CDORevision instances known to Hibernate. Please somebody
explain why we need runtime byte code engineering? Is there some
configuration missing to tell Hibernate to use our special tuplizers?
<br>
<br>
I have changed several things in the code of the test plugin and since
it's quite small now that the jars are removed, I attach my version of
the zip here (including a JUnit Plugin launch config). And don't forget
to update the cdo.server.hibernate.libraries plugin from CVS to get the
new buddy policy!
<br>
<br>
Cheers
<br>
/Eike
<br>
<br>
Martin Taal schrieb:
<br>
<blockquote type="cite">Hi Cédric,
<br>
I looked at it for many hours trying different things (using your
testcase) but I am kind of stuck with this (getting the same error as
you). The difference I can see with the way I run the cdo.hibernate
testcases (as a junit test) and how your testcase is run (junit plugin
test). This for sure means difference in class loader behavior.
<br>
<br>
Eike, do you have an idea? I am
Re: [CDO] reading/writting throughput performances [message #124535 is a reply to message #124519] Sun, 01 June 2008 10:06 Go to previous messageGo to next message
Martin Taal is currently offline Martin TaalFriend
Messages: 5468
Registered: July 2009
Senior Member
Hi Eike, Cedric,
I can't say to much about net4j.db versus net4j.db.hsqldb.

You are correct that the hibernate.libraries plugin should have Eclipse-BuddyPolicy: dependent (or
another way to use the classpath of the dependent plugins). Btw, I agree that registered is better
as afaik dependent gets slow which larger plugin environments.

The reason that cglib is used is because one of the class mappings in
org.eclipse.emf.cdo.server.hibernate/mappings/meta.hbm/xml did not have the lazy="false" directive.
I added this and now I get another exception (see below). So this 'solves' the problem. If you do a
get latest of this file then I think this problem is 'overcome'.

Overall I still do not see why it went wrong. As far as I can see the cglib library could not find
the hibernate class, which is strange as they are in the same plugin??
The cdo testcases don't have this problem and neither does Teneo.

As it does not occur anymore we can ignore it for now (not very satisfactory but there are other
things to do)... Unless someone else has an idea how to handle this ofcourse.

gr. Martin

java.lang.IllegalStateException: acceptor == null
at org.eclipse.net4j.internal.jvm.JVMClientConnector.doBeforeAc tivate(JVMClientConnector.java:57)
at org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:56)
at org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
at org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
at org.eclipse.net4j.internal.util.container.ManagedContainer.g etElement(ManagedContainer.java:260)
at org.eclipse.emf.internal.cdo.util.ChannelInjector.getConnect or(ChannelInjector.java:74)
at org.eclipse.emf.internal.cdo.util.ChannelInjector.process(Ch annelInjector.java:41)
at
org.eclipse.net4j.internal.util.container.ManagedContainer.p ostProcessElement(ManagedContainer.java:407)
at org.eclipse.net4j.internal.util.container.ManagedContainer.g etElement(ManagedContainer.java:259)
at org.eclipse.cdo.bug.hibernate.HibernateServer.createSession( HibernateServer.java:93)
at org.eclipse.cdo.bug.unit.LaunchServerTest.testStringValue(La unchServerTest.java:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at junit.framework.TestCase.runTest(TestCase.java:164)
at junit.framework.TestCase.runBare(TestCase.java:130)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:120)
at junit.framework.TestSuite.runTest(TestSuite.java:230)
at junit.framework.TestSuite.run(TestSuite.java:225)
at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
at org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
at org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hod(EclipseAppContainer.java:547)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
at
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:106)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:76)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:362)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:175)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 564)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
at org.eclipse.equinox.launcher.Main.run(Main.java:1251)
at org.eclipse.equinox.launcher.Main.main(Main.java:1227)



Eike Stepper wrote:
> Hi,
>
> Here is a diagram of the dependencies of the relevant plugins (please
> correct me if something is wrong).
> The black arrows are ordinary downstream dependencies (Require-Bundle).
> The red arrows are upstream dependencies (Eclipse-BuddyPolicy: dependent).
>
>
>
> For research purposes I added code to the libraries plugin that tries to
> reflectively load CDOFeatureImpl and that is not a problem!
> It seems to me as if our problem is not class loading related because
> the hibernate.jar should be able to see the needed classes.
> I believe this is either a bug in Hibernate or our wrong usage of
> Hibernate. I tend to believe the latter ;-)
> I put a breakpoint in
> org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping .EntityEntityModeToTuplizerMapping(PersistentClass,
> EntityMetamodel)
> to see how the tuplizers are created. I'm too unexperienced with
> Hibernate to get a clue about what happens there...
>
> Btw. I'd prefer to see the application plugin depend on net4j.db instead
> of net4j.db.hsqldb.
> But it seems as if the BuddyPolicy in net4j.db is not "reexported" so
> that the application can see the hsqldb.jar then ;-(
> I posted a respective question to equinox...
>
> Cheers
> /Eike
>
>
> Eike Stepper schrieb:
>> Hi guys,
>>
>> I can confirm, making these 3rd party libraries with their own
>> classloading run in OSGi is a nightmare! I took me hours teach where
>> the hsqldb driver is. On the way I came over different kinds of
>> classloading related errors.
>>
>> First I removed all the jars from the test plugin (the one Cedric
>> provided). They're already in the cdo.server.hibernate.libraries
>> plugin which, in fact, needs the buddy policy set to dependent. I
>> think this last change needs to be committed to CVS! Btw. I'd prefer
>> to use *registered* buddies instead of making all dependent bundles
>> visible.
>>
>> The test plugin then needs to provide the database driver. Therefore I
>> added a dependency on the org.eclipse.net4j.db.hsqldb plugin.
>>
>> With this setup I can start the server without CNFE, NCDFE, CCI and
>> the like. At this point I get the following exception, which seems the
>> same as the one Cedric observed in the end:
>>
>> net.sf.cglib.core.CodeGenerationException:
>> java.lang.reflect.InvocationTargetException-->null
>> at
>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:237)
>>
>> at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
>> at net.sf.cglib.proxy.Enhancer.createClass(Enhancer.java:317)
>> at
>> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:127)
>>
>> at
>> org.hibernate.proxy.pojo.cglib.CGLIBProxyFactory.postInstant iate(CGLIBProxyFactory.java:43)
>>
>> at
>> org.hibernate.tuple.entity.PojoEntityTuplizer.buildProxyFact ory(PojoEntityTuplizer.java:162)
>>
>> at
>> org.hibernate.tuple.entity.AbstractEntityTuplizer.<init>(AbstractEntityTuplizer.java:135)
>>
>> at
>> org.hibernate.tuple.entity.PojoEntityTuplizer.<init>(PojoEntityTuplizer.java:55)
>>
>> at
>> org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping . <init>(EntityEntityModeToTuplizerMapping.java:56)
>>
>> at
>> org.hibernate.tuple.entity.EntityMetamodel.<init>(EntityMetamodel.java:295)
>>
>> at
>> org.hibernate.persister.entity.AbstractEntityPersister.<init >(AbstractEntityPersister.java:434)
>>
>> at
>> org.hibernate.persister.entity.JoinedSubclassEntityPersister . <init>(JoinedSubclassEntityPersister.java:91)
>>
>> at
>> org.hibernate.persister.PersisterFactory.createClassPersiste r(PersisterFactory.java:58)
>>
>> at
>> org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:226)
>> at
>> org.hibernate.cfg.Configuration.buildSessionFactory(Configur ation.java:1294)
>>
>> at
>> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getSessionFactory(HibernatePackageHandler.java:335 )
>>
>> at
>> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.readPackageInfos(HibernatePackageHandler.java:293)
>>
>> at
>> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getCDOPackageInfos(HibernatePackageHandler.java:21 5)
>>
>> at
>> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Reader.readPackageInfos(HibernateStoreReader.java:87)
>>
>> at
>> org.eclipse.emf.cdo.internal.server.PackageManager.doActivat e(PackageManager.java:72)
>>
>> at
>> org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
>>
>> at
>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>>
>> at
>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>>
>> at
>> org.eclipse.emf.cdo.internal.server.Repository.activateRepos itory(Repository.java:264)
>>
>> at
>> org.eclipse.emf.cdo.internal.server.Repository.doActivate(Re pository.java:251)
>>
>> at
>> org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
>>
>> at
>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>>
>> at
>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>>
>> at
>> org.eclipse.emf.cdo.server.CDOServerUtil.addRepository(CDOSe rverUtil.java:67)
>>
>> at
>> org.eclipse.cdo.bug.LaunchServerTest.setUp(LaunchServerTest. java:54)
>> at junit.framework.TestCase.runBare(TestCase.java:128)
>> at junit.framework.TestResult$1.protect(TestResult.java:106)
>> at junit.framework.TestResult.runProtected(TestResult.java:124)
>> at junit.framework.TestResult.run(TestResult.java:109)
>> at junit.framework.TestCase.run(TestCase.java:120)
>> at junit.framework.TestSuite.runTest(TestSuite.java:230)
>> at junit.framework.TestSuite.run(TestSuite.java:225)
>> at
>> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
>>
>> at
>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
>>
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
>>
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
>>
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
>>
>> at
>> org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
>>
>> at
>> org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>
>> at java.lang.reflect.Method.invoke(Method.java:585)
>> at
>> org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hodWithException(EclipseAppContainer.java:574)
>>
>> at
>> org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
>>
>> at
>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:110)
>>
>> at
>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:79)
>>
>> at
>> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:379)
>>
>> at
>> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:179)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>
>> at java.lang.reflect.Method.invoke(Method.java:585)
>> at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 549)
>> at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
>> at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
>> at org.eclipse.equinox.launcher.Main.main(Main.java:1212)
>> Caused by: java.lang.reflect.InvocationTargetException
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>
>> at java.lang.reflect.Method.invoke(Method.java:585)
>> at net.sf.cglib.core.ReflectUtils.defineClass(ReflectUtils.java :384)
>> at
>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>
>> ... 61 more
>> Caused by: java.lang.NoClassDefFoundError:
>> org/hibernate/proxy/HibernateProxy
>> at java.lang.ClassLoader.defineClass1(Native Method)
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>> ... 67 more
>>
>> I must admit that I'm still not sure about te fundamental interwork
>> between Hibernate and the user defined meta model. It seems as if asm
>> and cglib are somewhere in between but I have no idea why at all. I
>> thought we have implemented a bunch of static Hibernate interfaces to
>> make the CDORevision instances known to Hibernate. Please somebody
>> explain why we need runtime byte code engineering? Is there some
>> configuration missing to tell Hibernate to use our special tuplizers?
>>
>> I have changed several things in the code of the test plugin and since
>> it's quite small now that the jars are removed, I attach my version of
>> the zip here (including a JUnit Plugin launch config). And don't
>> forget to update the cdo.server.hibernate.libraries plugin from CVS to
>> get the new buddy policy!
>>
>> Cheers
>> /Eike
>>
>> Martin Taal schrieb:
>>> Hi Cédric,
>>> I looked at it for many hours trying different things (using your
>>> testcase) but I am kind of stuck with this (getting the same error as
>>> you). The difference I can see with the way I run the cdo.hibernate
>>> testcases (as a junit test) and how your testcase is run (junit
>>> plugin test). This for sure means difference in class loader behavior.
>>>
>>> Eike, do you have an idea? I am not sure how classloading works when
>>> you start a cdo server in a plugin. Which plugins are then available
>>> in the cdo server? The testcases can be downloaded from the link below.
>>>
>>> gr. Martin
>>>
>>> Cédric Brun wrote:
>>>> Thanks a lot ! I'm looking forward to your advices !
>>>>
>>>> Cédric
>>>>
>>>> Martin Taal wrote:
>>>>
>>>>> I am downloading your zip-file. I don't have time to look at it
>>>>> right away
>>>>> but I will do my best later this evening.
>>>>>
>>>>> gr. Martin
>>>>>
>>>>> Cédric Brun wrote:
>>>>>> I tried to do the same things as in the CDO/Hibernate unit tests.
>>>>>>
>>>>>> Here's a zipped project trying to launch an HibernateServer with a
>>>>>> unit
>>>>>> test adding elements.
>>>>>> ( http://cedric.brun.free.fr/Tortoose/org.eclipse.cdo-teneo-te st.zip)
>>>>>> The
>>>>>> libs I found are included in the same project, but I still have no
>>>>>> luck
>>>>>> and failling at :
>>>>>>
>>>>>> at
>>>>>>
>>>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>>>
>>>>>> ... 62 more
>>>>>> Caused by: java.lang.NoClassDefFoundError:
>>>>>> org/hibernate/proxy/HibernateProxy
>>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>>>>>> ... 68 more
>>>>>>
>>>>>> please note that the test project I provide really is a "quick and
>>>>>> ugly"
>>>>>> one ^^
>>>>>>
>>>>>>
>>>>>> Cédric
>>>>>>
>>>>>>
>>>>>>
>>>>>> Martin Taal wrote:
>>>>>>
>>>>>>> Hmm, I can understand your headache. Classpath issues are the most
>>>>>>> annoying errors...
>>>>>>>
>>>>>>> In your junit test source file, what happens when you enter the
>>>>>>> org.objectweb.asm.Type, can you source file see this class?
>>>>>>> Do you have cglib in some other plugin which does not have asm?
>>>>>>>
>>>>>>> What I also find slightly strange is that cglib is used in this case
>>>>>>> because normally cdo objects are not proxied using cglib. Can you
>>>>>>> post
>>>>>>> the generated mapping? How does your cdo initialization code look
>>>>>>> like?
>>>>>>>
>>>>>>> gr. Martin
>>>>>>>
>>>>>>> Cédric Brun wrote:
>>>>>>>> Hi Martin,
>>>>>>>>
>>>>>>>> and thanks for your quick answer ! what I did is the following :
>>>>>>>>
>>>>>>>> I used the org.eclipse.emf.cdo.server.hibernate.libraries bundle
>>>>>>>> and
>>>>>>>> added that :
>>>>>>>>
>>>>>>>> Bundle-ClassPath: lib/hibernate3.jar,
>>>>>>>> lib/cglib-2.1.3.jar,
>>>>>>>> lib/commons-collections-2.1.1.jar,
>>>>>>>> lib/dom4j.jar,
>>>>>>>> lib/jta.jar
>>>>>>>> lib/asm-3.0.jar
>>>>>>>>
>>>>>>>> and this directive :
>>>>>>>>
>>>>>>>> Eclipse-BuddyPolicy: dependent
>>>>>>>>
>>>>>>>> as stated in
>>>>>>>> http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>>>>>>>>
>>>>>>>>
>>>>>>>> Then I don't get the same error, now when I start my own plugin
>>>>>>>> (JUnit
>>>>>>>> plugin test) depending on
>>>>>>>> org.eclipse.emf.cdo.server.hibernate.libraries I get :
>>>>>>>>
>>>>>>>> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
>>>>>>>> at
>>>>>>>> net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
>>>>>>>> at
>>>>>>>> net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
>>>>>>>> at net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
>>>>>>>> at
>>>>>>>>
>>>> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>>>>
>>>>>>>> These jars are such a headache ! Do you know - by chance - what's
>>>>>>>> going on ?
>>>>>>>>
>>>>>>>>
>>>>>>>> Cédric
>>>>>>>>
>>>>>>>>
>>>>>>>> Martin Taal wrote:
>>>>>>>>
>>>>>>>>> Hi Cédric,
>>>>>>>>> Most of the time this exception is thrown because hibernate
>>>>>>>>> can't find
>>>>>>>>> the class. A common cause is that the hibernate jar files are
>>>>>>>>> located
>>>>>>>>> in a separate plugin which can not reach the classes in other
>>>>>>>>> plugins.
>>>>>>>>> If you set the eclipse buddy policy of the hibernate plugin to
>>>>>>>>> dependent then this can help.
>>>>>>>>>
>>>>>>>>> gr. Martin
>>>>>>>>>
>>>>>>>>> Cédric Brun wrote:
>>>>>>>>>> Hi Simon and Eike,
>>>>>>>>>>
>>>>>>>>>> thanks for your feedback, now I know what I could expect in using
>>>>>>>>>> optimized implementation, I'll have a try on the hibernate
>>>>>>>>>> store and
>>>>>>>>>> see if it is scaling in a better way.
>>>>>>>>>>
>>>>>>>>>> You gave me many pointers to explain the performances issues
>>>>>>>>>> but in
>>>>>>>>>> fact it seems that the SQL execution is taking so much time (not
>>>>>>>>>> suprisingly in fact ;) . My first profile was wrong.
>>>>>>>>>>
>>>>>>>>>> So I'm installing the hibernate store retrieving all the
>>>>>>>>>> source code
>>>>>>>>>> from CVS, and I'm trying to launch the unit tests. I added
>>>>>>>>>> many jars
>>>>>>>>>> and everything gets compiled but most of the tests are
>>>>>>>>>> failling with
>>>>>>>>>> "
>>>>>>>>>>
>>>>>>>>>> "
>>>>>>>>>> org.hibernate.MappingException: Could not determine type for:
>>>>>>>>>>
>>>> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>>>>>>>>> at table: cdo_features, for columns:
>>>>>>>>>> [org.hibernate.mapping.Column(type)] "
>>>>>>>>>>
>>>>>>>>>> any hint ?
>>>>>>>>>>
>>>>>>>>>> Cédric
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Simon McDuff wrote:
>>>>>>>>>>
>>>>>>>>>>> We noticed a performance degradation in the following case:
>>>>>>>>>>>
>>>>>>>>>>> When we fetch a collection of objects that contains
>>>>>>>>>>> collections that
>>>>>>>>>>> we don't want to use. Ex:
>>>>>>>>>>>
>>>>>>>>>>> Model:
>>>>>>>>>>>
>>>>>>>>>>> ClassB
>>>>>>>>>>> {
>>>>>>>>>>> int foo1;
>>>>>>>>>>> List childrens1;
>>>>>>>>>>> List childrens2;
>>>>>>>>>>> List childrens3;
>>>>>>>>>>> };
>>>>>>>>>>>
>>>>>>>>>>> Code:
>>>>>>>>>>>
>>>>>>>>>>> Resource.contents contains objects of type ClassB.
>>>>>>>>>>>
>>>>>>>>>>> By fetching instances of classB, we will also fetched some
>>>>>>>>>>> items in
>>>>>>>>>>> the list(childrens1, childrens2, childrens3) The problem we
>>>>>>>>>>> had.. is
>>>>>>>>>>> we do not know if we want to use it. SO better not to
>>>>>>>>>>> download them.
>>>>>>>>>>>
>>>>>>>>>>> So In our production(we used our own IStore), we never
>>>>>>>>>>> download any
>>>>>>>>>>> objects from collection. I'm working at the moment to put in
>>>>>>>>>>> place
>>>>>>>>>>> all feature we used in the current CDO. That one will be part of
>>>>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477. To begin
>>>>>>>>>>> with,CDORevisionImpl needs to supports list that are 0 even if
>>>>>>>>>>> referenceChunkSize == 0.
>>>>>>>>>>>
>>>>>>>>>>> It is true that we obtain good performance.
>>>>>>>>>>> Worst case :
>>>>>>>>>>> 1500 objects /sec (Real fetch from disk. Page that contains the
>>>>>>>>>>> objects, wasn't in the cache of the filesystem.)
>>>>>>>>>>>
>>>>>>>>>>> Best case
>>>>>>>>>>> 12 000 objects/Sec (Fetch objects that are already in the
>>>>>>>>>>> case of
>>>>>>>>>>> the filesystem)
>>>>>>>>>>>
>>>>>>>>>>> (These statistics always fetch objects from server to client.
>>>>>>>>>>> The
>>>>>>>>>>> cache client isn't used)
>>>>>>>>>>>
>>>>>>>>>>> We have another optimization one with SmartReadAhead but for now
>>>>>>>>>>> only work
>>>>>>>>>>> for UI client. (for now) This is something we will integrate as
>>>>>>>>>>> well in CDO. But for now we concentrate on the other things.
>>>>>>>>>>>
>>>>>>>>>>> I used an Object database. Based on my preliminary test
>>>>>>>>>>> without CDO.
>>>>>>>>>>> I compare
>>>>>>>>>>>
>>>>>>>>>>> Hibernate vs Objectivity/EMF :
>>>>>>>>>>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>>>>>>>>>>
>>>>>>>>>>> The worst.. you will get 30 to 240 objects/sec or
>>>>>>>>>>> the best... you will get 500 to 4000 objects/sec
>>>>>>>>>>>
>>>>>>>>>>> Based on your comments... you are getting the worst (with
>>>>>>>>>>> Derby)!!
>>>>>>>>>>> :-)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Another comments about insertion.
>>>>>>>>>>>
>>>>>>>>>>> We found another problem when we insert data(DOn't know if it is
>>>>>>>>>>> still true). Resource.Contents is a unique list. Basically
>>>>>>>>>>> each time
>>>>>>>>>>> we insert one element, it needs to fetch all the collection
>>>>>>>>>>> to see
>>>>>>>>>>> if we already have this element. Basically.. it is time
>>>>>>>>>>> consuming.
>>>>>>>>>>> In our case we are not going through this stage to persist our
>>>>>>>>>>> objects... but the objects are not accessible through that list.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> If speed is an issue for you!! We (You included :-)) can
>>>>>>>>>>> optimize
>>>>>>>>>>> it. What I like is Eike did a really good job doing that
>>>>>>>>>>> framwork
>>>>>>>>>>> and optimizations are easy to add. Also you can change your
>>>>>>>>>>> back-end
>>>>>>>>>>> without affecting you client side. (Maybe going to an Object
>>>>>>>>>>> database!!) THis was one of the purpose of CDO!! :-)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Simon
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le
>>>>>>>>>>> message de
>>>>>>>>>>> news: 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>>>>>>>>
>>>>>>>>>>> That sounds interesting!
>>>>>>>>>>> Comments inline...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Cédric Brun schrieb:
>>>>>>>>>>> Hi Eike,
>>>>>>>>>>>
>>>>>>>>>>> I've been messing around CDO lately and I hacked a few ugly
>>>>>>>>>>> unit-tests measuring read/write throughput when I use CDO (RC2).
>>>>>>>>>>>
>>>>>>>>>>> I'm using an embeded-derby database, two different metamodels
>>>>>>>>>>> (complex one - 250 EClassifiers/300 features and a simple one
>>>>>>>>>>> - 1
>>>>>>>>>>> Classifier/3 Features)
>>>>>>>>>>>
>>>>>>>>>>> The test is a "quick and dirty" one so please do not take these
>>>>>>>>>>> figures too seriously, I would just like to have your input
>>>>>>>>>>> about
>>>>>>>>>>> the "order of magnitude" I should expect or settings affecting
>>>>>>>>>>> seriously these performances. I tried both JVM and TCP
>>>>>>>>>>> acceptor but
>>>>>>>>>>> the performances are quite similar though JVM is a bit faster.
>>>>>>>>>>> Looks like a sign that the NIO socket channels are
>>>>>>>>>>> efficient ,-)
>>>>>>>>>>> You used localhost for TCP transport I guess?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> * First case : building a small model (10000 elements) with the
>>>>>>>>>>> simple MM, only adding instances in an empty database :
>>>>>>>>>>> commiting change on every add I get a 2 objects per second rate.
>>>>>>>>>>> commiting changes every 10 adds - I get a 30 objects per second
>>>>>>>>>>> rate.
>>>>>>>>>>> Seems to be consistent with the fact that only for commits
>>>>>>>>>>> there
>>>>>>>>>>> is remote traffic. And the backend transaction management is
>>>>>>>>>>> another overhead. So the impact of the number objects/commit
>>>>>>>>>>> starts with bigger transactions.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> browsing the just commited elements opening a new transaction
>>>>>>>>>>> and
>>>>>>>>>>> getting a new Resource instance, I get around 30 000
>>>>>>>>>>> objets/sec rate
>>>>>>>>>>> just iterating thanks to Resource.getAllContents().
>>>>>>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>>>>>>> All views and transactions of a single session share (and
>>>>>>>>>>> cache)
>>>>>>>>>>> their object state. Otherwise I'd expect much smaller values.
>>>>>>>>>>> With his ObjectivityStore implementation Simon (cc'ed) reached
>>>>>>>>>>> rates of several thousands objects/sec. But the DBStore
>>>>>>>>>>> (which you
>>>>>>>>>>> used) is not that optimized.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> * Second case : I built a big model in the database using the
>>>>>>>>>>> complex metamodel. It contains 700 000 elements. The creation
>>>>>>>>>>> went
>>>>>>>>>>> pretty nicely (I did not measured it exactly as it was launched
>>>>>>>>>>> during a night, but it seems I got around 100 objects/sec
>>>>>>>>>>> rate in
>>>>>>>>>>> writting as I was writting every 700 objects.
>>>>>>>>>>> Sounds not too bad.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Once the model is built the derby database is 222 MB (which - I
>>>>>>>>>>> heard - is not so big for derby).
>>>>>>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>>>>>>> To be honest, I'm really no database expert (although I can
>>>>>>>>>>> write
>>>>>>>>>>> simple SQL statements).
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Retrieving the distant resource and calling eAllContents() on
>>>>>>>>>>> it, I
>>>>>>>>>>> get a 50 Objects/sec rate at best, which mean my
>>>>>>>>>>> eAllContents() call
>>>>>>>>>>> would take 4 hours !
>>>>>>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>>>>>>> (iteration order and so). There are several CDO features
>>>>>>>>>>> that can
>>>>>>>>>>> have a major impact on load rates!
>>>>>>>>>>>
>>>>>>>>>>> - Collections (xrefs and containment) can be loaded lazily. I
>>>>>>>>>>> don't mean the target objects, which are lazy by default
>>>>>>>>>>> anyway,
>>>>>>>>>>> but even the target IDs that are in the collections. We
>>>>>>>>>>> call that
>>>>>>>>>>> "reference chunking" (but search for a better name is
>>>>>>>>>>> ongoing).
>>>>>>>>>>>
>>>>>>>>>>> public interface CDOSession extends CDOProtocolSession,
>>>>>>>>>>> IContainer<CDOView>
>>>>>>>>>>> {
>>>>>>>>>>> public int getReferenceChunkSize();
>>>>>>>>>>>
>>>>>>>>>>> public void setReferenceChunkSize(int
>>>>>>>>>>> referenceChunkSize);
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> - Loading collection elements (target objects) can be
>>>>>>>>>>> batched (the
>>>>>>>>>>> current name is even worse)
>>>>>>>>>>>
>>>>>>>>>>> public interface CDOView extends CDOProtocolView,
>>>>>>>>>>> INotifier
>>>>>>>>>>> {
>>>>>>>>>>> public int getLoadRevisionCollectionChunkSize();
>>>>>>>>>>>
>>>>>>>>>>> public void setLoadRevisionCollectionChunkSize(int
>>>>>>>>>>> loadRevisionCollectionChunkSize);
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> - One of the most sophisticated feature is automatic model
>>>>>>>>>>> usage
>>>>>>>>>>> analyzation. The result are fetch rules that are sent to the
>>>>>>>>>>> server. When later objects are loaded from the server, it can
>>>>>>>>>>> answer with additional objects that are likely to be needed
>>>>>>>>>>> by the
>>>>>>>>>>> client in the near future.
>>>>>>>>>>>
>>>>>>>>>>> public class CDOViewImpl extends
>>>>>>>>>>> org.eclipse.net4j.internal.util.event.Notifier
>>>>>>>>>>> implements
>>>>>>>>>>> CDOView, CDOIDProvider, Adapter.Internal
>>>>>>>>>>> {
>>>>>>>>>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>>>>>>>> {
>>>>>>>>>>> return featureAnalyzer;
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>>>>>>>> featureAnalyzer)
>>>>>>>>>>> {
>>>>>>>>>>> this.featureAnalyzer = featureAnalyzer == null ?
>>>>>>>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> - There's a SmartReadAheadThread (if you're curious, please
>>>>>>>>>>> ask
>>>>>>>>>>> Simon). - Caches can be configured
>>>>>>>>>>> - ...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Known issues of the DBStore:
>>>>>>>>>>> - When in auditing mode updates are always written as complete
>>>>>>>>>>> inserts (by design) - When configuring toManyReferences to
>>>>>>>>>>> a value
>>>>>>>>>>> different from LIKE_ATTRIBUTES (which is not implemented
>>>>>>>>>>> yet) it
>>>>>>>>>>> is very slow. - No (few) prepared statements are used (see
>>>>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe
>>>>>>>>>>> others...
>>>>>>>>>>>
>>>>>>>>>>> Please note that the DBStore was not written by me to
>>>>>>>>>>> compete with
>>>>>>>>>>> Hibernate in terms of performance or mapping capabilities. Now
>>>>>>>>>>> that Martin and I, well in the end mostly Martin, provide a
>>>>>>>>>>> HibernateStore and this HibernateStore gets more and more
>>>>>>>>>>> features
>>>>>>>>>>> I expect that there will be a trend away from the DBStore.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This is probably linked to the way the mapping to db is done,
>>>>>>>>>>> here's
>>>>>>>>>>> the config I used (came from CDO-server example if I recall
>>>>>>>>>>> correctly).
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <repository name="repo1">
>>>>>>>>>>> <property name="overrideUUID"
>>>>>>>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>>>>>>>> <property name="supportingAudits" value="true"/>
>>>>>>>>>>> <property name="verifyingRevisions"
>>>>>>>>>>> value="false"/>
>>>>>>>>>>> <property name="currentLRUCapacity"
>>>>>>>>>>> value="10000"/>
>>>>>>>>>>> I guess this one could be larger.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <property name="revisedLRUCapacity"
>>>>>>>>>>> value="100"/>
>>>>>>>>>>> <store type="db">
>>>>>>>>>>> <!-- type: horizontal | vertical | <any
>>>>>>>>>>> user-contributed type> -->
>>>>>>>>>>> <mappingStrategy
>>>>>>>>>>> type="horizontal">
>>>>>>>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>>>>>>>> ONE_TABLE_PER_CLASS |
>>>>>>>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY |
>>>>>>>>>>> LIKE_ATTRIBUTES
>>>>>>>>>>> -->
>>>>>>>>>>> <property
>>>>>>>>>>> name="toManyReferences"
>>>>>>>>>>>
>>>>>>>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>>>>>>>
>>>>>>>>>>> <!-- LIKE_ATTRIBUTES |
>>>>>>>>>>> LIKE_TO_MANY_REFERENCES -->
>>>>>>>>>>> <property name="toOneReferences"
>>>>>>>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>>>>>>>
>>>>>>>>>>> <!-- MODEL | STRATEGY -->
>>>>>>>>>>> <property
>>>>>>>>>>> name="mappingPrecedence"
>>>>>>>>>>> value="MODEL"/>
>>>>>>>>>>> This one is obsolete.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> </mappingStrategy>
>>>>>>>>>>> <dbAdapter
>>>>>>>>>>> name="derby-embedded"/>
>>>>>>>>>>> <dataSource
>>>>>>>>>>>
>>>>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>>>>>>>>> databaseName="/tmp/cdodb1"
>>>>>>>>>>> createDatabase="create"/>
>>>>>>>>>>>
>>>>>>>>>>> </store>
>>>>>>>>>>> </repository>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> What do you think about that ? What throughput do you think I
>>>>>>>>>>> should
>>>>>>>>>>> expect ? As I said that's very difficult to judge.
>>>>>>>>>>> I believe it's more appropriate to compare the actual
>>>>>>>>>>> values with
>>>>>>>>>>> values from other measures (other setups, properties, etc.
>>>>>>>>>>> or even
>>>>>>>>>>> other products) instead of random expectations. Apologies
>>>>>>>>>>> that I
>>>>>>>>>>> didn't have the time so far to provide some reference measures
>>>>>>>>>>> myself. Maybe this gets better now that the team starts to
>>>>>>>>>>> grow
>>>>>>>>>>> slowly ;-)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> What are the main reasons fur such figures ?
>>>>>>>>>>> Hmm, I *suspect* you don't like the figures? I can find no
>>>>>>>>>>> evidence your assessment ;-)
>>>>>>>>>>>
>>>>>>>>>>> Ahh,one more note:
>>>>>>>>>>> Tracing can also have a significant impact on performance and
>>>>>>>>>>> end-to-end throughput. Turn it off by all means. And please
>>>>>>>>>>> make
>>>>>>>>>>> sure that it's really turned off, not only invisible (i.e.
>>>>>>>>>>> console
>>>>>>>>>>> is not a target).
>>>>>>>>>>>
>>>>>>>>>>> Cheers
>>>>>>>>>>> /Eike
>>>>>
>>>>
>>>
>>>


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Cell: +31 (0)6 288 48 943
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: [CDO] reading/writting throughput performances [message #124561 is a reply to message #124535] Sun, 01 June 2008 12:40 Go to previous messageGo to next message
Eclipse UserFriend
Originally posted by: stepper.sympedia.de

This is a multi-part message in MIME format.
--------------070303090402040107080201
Content-Type: multipart/alternative;
boundary="------------040308010500040802070703"


--------------040308010500040802070703
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi Martin,

Ah, now with the new mapping I get a big step further!
As I said, Cedric's test code contained some bugs, too. IIRC a TCP
acceptor was created but the client uses a JVMConnector, and stuff like
this.
You could try to overcome the exception below by adding the following to
the server code somewhere:

| IPluginContainer.INSTANCE.getElement("org.eclipse.net4j.acceptors ", "jvm", "default");|



Better just use the attached zip because there may be other changes that
I don't remember in detail anymore.
Now I run into another exception:

Thread-4 [debug.signal] ================ Indicating
CommitTransactionIndication
Thread-4 [debug] Created
org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Writer for
repository repo1
Thread-4 [debug] Created
org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Writer for
repository repo1
Thread-4 [debug] Created
org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Writer for
repository repo1
Thread-4 [debug.protocol] Reading 0 new packages
Thread-4 [debug.protocol] Reading 2 new objects
Thread-4 [debug.model] Reading CDOID of type 3 (TEMP_OBJECT)
Thread-4 [debug.model] Reading CDOID of type 3 (TEMP_OBJECT)
Thread-4 [debug.model] Reading CDOID of type 0 (NULL)
Thread-4 [debug.revision] Reading revision: ID=oid1,
classRef=CDOClassRef(http://www.eclipse.org/emf/CDO/resource/1.0.0, 0),
className=CDOResource, version=1, created=0, revised=0, resource=oid1,
container=NULL, feature=0
Thread-4 [debug.revision] Read feature CDOFeature(ID=9, name=path,
type=STRING, referenceType=null): /res2
Thread-4 [debug.revision] Read feature CDOFeature(ID=2, name=contents,
type=OBJECT, referenceType=CDOClass(ID=0, name=CDOObject)): size=1
Thread-4 [debug.model] Reading CDOID of type 3 (TEMP_OBJECT)
Thread-4 [debug.revision] oid2
[ERROR] ClassRef unresolveable:
CDOClassRef(http://www.eclipse.org/modeling/cdo/bug/names, 0)
java.lang.IllegalStateException: ClassRef unresolveable:
CDOClassRef(http://www.eclipse.org/modeling/cdo/bug/names, 0)
at
org.eclipse.emf.cdo.internal.protocol.revision.CDORevisionIm pl. <init>(CDORevisionImpl.java:116)
at
org.eclipse.emf.cdo.protocol.revision.CDORevisionUtil.read(C DORevisionUtil.java:48)
at
org.eclipse.emf.cdo.internal.server.protocol.CommitTransacti onIndication.indicating(CommitTransactionIndication.java:95)
at
org.eclipse.net4j.signal.IndicationWithResponse.execute(Indi cationWithResponse.java:46)
at org.eclipse.net4j.signal.Signal.runSync(Signal.java:143)
at org.eclipse.net4j.signal.Signal.run(Signal.java:124)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Threa dPoolExecutor.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo lExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)

Cheers
/Eike



Martin Taal schrieb:
> Hi Eike, Cedric,
> I can't say to much about net4j.db versus net4j.db.hsqldb.
>
> You are correct that the hibernate.libraries plugin should have
> Eclipse-BuddyPolicy: dependent (or another way to use the classpath of
> the dependent plugins). Btw, I agree that registered is better as
> afaik dependent gets slow which larger plugin environments.
>
> The reason that cglib is used is because one of the class mappings in
> org.eclipse.emf.cdo.server.hibernate/mappings/meta.hbm/xml did not
> have the lazy="false" directive. I added this and now I get another
> exception (see below). So this 'solves' the problem. If you do a get
> latest of this file then I think this problem is 'overcome'.
>
> Overall I still do not see why it went wrong. As far as I can see the
> cglib library could not find the hibernate class, which is strange as
> they are in the same plugin??
> The cdo testcases don't have this problem and neither does Teneo.
>
> As it does not occur anymore we can ignore it for now (not very
> satisfactory but there are other things to do)... Unless someone else
> has an idea how to handle this ofcourse.
>
> gr. Martin
>
> java.lang.IllegalStateException: acceptor == null
> at
> org.eclipse.net4j.internal.jvm.JVMClientConnector.doBeforeAc tivate(JVMClientConnector.java:57)
>
> at
> org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:56)
>
> at
> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>
> at
> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>
> at
> org.eclipse.net4j.internal.util.container.ManagedContainer.g etElement(ManagedContainer.java:260)
>
> at
> org.eclipse.emf.internal.cdo.util.ChannelInjector.getConnect or(ChannelInjector.java:74)
>
> at
> org.eclipse.emf.internal.cdo.util.ChannelInjector.process(Ch annelInjector.java:41)
>
> at
> org.eclipse.net4j.internal.util.container.ManagedContainer.p ostProcessElement(ManagedContainer.java:407)
>
> at
> org.eclipse.net4j.internal.util.container.ManagedContainer.g etElement(ManagedContainer.java:259)
>
> at
> org.eclipse.cdo.bug.hibernate.HibernateServer.createSession( HibernateServer.java:93)
>
> at
> org.eclipse.cdo.bug.unit.LaunchServerTest.testStringValue(La unchServerTest.java:43)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:585)
> at junit.framework.TestCase.runTest(TestCase.java:164)
> at junit.framework.TestCase.runBare(TestCase.java:130)
> at junit.framework.TestResult$1.protect(TestResult.java:106)
> at junit.framework.TestResult.runProtected(TestResult.java:124)
> at junit.framework.TestResult.run(TestResult.java:109)
> at junit.framework.TestCase.run(TestCase.java:120)
> at junit.framework.TestSuite.runTest(TestSuite.java:230)
> at junit.framework.TestSuite.run(TestSuite.java:225)
> at
> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
>
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
>
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
>
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
>
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
>
> at
> org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
>
> at
> org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:585)
> at
> org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hod(EclipseAppContainer.java:547)
>
> at
> org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
>
> at
> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:106)
>
> at
> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:76)
>
> at
> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:362)
>
> at
> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:175)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 564)
> at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
> at org.eclipse.equinox.launcher.Main.run(Main.java:1251)
> at org.eclipse.equinox.launcher.Main.main(Main.java:1227)
>
>
>
> Eike Stepper wrote:
>> Hi,
>>
>> Here is a diagram of the dependencies of the relevant plugins (please
>> correct me if something is wrong).
>> The black arrows are ordinary downstream dependencies (Require-Bundle).
>> The red arrows are upstream dependencies (Eclipse-BuddyPolicy:
>> dependent).
>>
>>
>>
>> For research purposes I added code to the libraries plugin that tries
>> to reflectively load CDOFeatureImpl and that is not a problem!
>> It seems to me as if our problem is not class loading related because
>> the hibernate.jar should be able to see the needed classes.
>> I believe this is either a bug in Hibernate or our wrong usage of
>> Hibernate. I tend to believe the latter ;-)
>> I put a breakpoint in
>> org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping .EntityEntityModeToTuplizerMapping(PersistentClass,
>> EntityMetamodel)
>> to see how the tuplizers are created. I'm too unexperienced with
>> Hibernate to get a clue about what happens there...
>>
>> Btw. I'd prefer to see the application plugin depend on net4j.db
>> instead of net4j.db.hsqldb.
>> But it seems as if the BuddyPolicy in net4j.db is not "reexported" so
>> that the application can see the hsqldb.jar then ;-(
>> I posted a respective question to equinox...
>>
>> Cheers
>> /Eike
>>
>>
>> Eike Stepper schrieb:
>>> Hi guys,
>>>
>>> I can confirm, making these 3rd party libraries with their own
>>> classloading run in OSGi is a nightmare! I took me hours teach where
>>> the hsqldb driver is. On the way I came over different kinds of
>>> classloading related errors.
>>>
>>> First I removed all the jars from the test plugin (the one Cedric
>>> provided). They're already in the cdo.server.hibernate.libraries
>>> plugin which, in fact, needs the buddy policy set to dependent. I
>>> think this last change needs to be committed to CVS! Btw. I'd prefer
>>> to use *registered* buddies instead of making all dependent bundles
>>> visible.
>>>
>>> The test plugin then needs to provide the database driver. Therefore
>>> I added a dependency on the org.eclipse.net4j.db.hsqldb plugin.
>>>
>>> With this setup I can start the server without CNFE, NCDFE, CCI and
>>> the like. At this point I get the following exception, which seems
>>> the same as the one Cedric observed in the end:
>>>
>>> net.sf.cglib.core.CodeGenerationException:
>>> java.lang.reflect.InvocationTargetException-->null
>>> at
>>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:237)
>>>
>>> at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
>>> at net.sf.cglib.proxy.Enhancer.createClass(Enhancer.java:317)
>>> at
>>> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:127)
>>>
>>> at
>>> org.hibernate.proxy.pojo.cglib.CGLIBProxyFactory.postInstant iate(CGLIBProxyFactory.java:43)
>>>
>>> at
>>> org.hibernate.tuple.entity.PojoEntityTuplizer.buildProxyFact ory(PojoEntityTuplizer.java:162)
>>>
>>> at
>>> org.hibernate.tuple.entity.AbstractEntityTuplizer.<init>(AbstractEntityTuplizer.java:135)
>>>
>>> at
>>> org.hibernate.tuple.entity.PojoEntityTuplizer.<init>(PojoEntityTuplizer.java:55)
>>>
>>> at
>>> org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping . <init>(EntityEntityModeToTuplizerMapping.java:56)
>>>
>>> at
>>> org.hibernate.tuple.entity.EntityMetamodel.<init>(EntityMetamodel.java:295)
>>>
>>> at
>>> org.hibernate.persister.entity.AbstractEntityPersister.<init >(AbstractEntityPersister.java:434)
>>>
>>> at
>>> org.hibernate.persister.entity.JoinedSubclassEntityPersister . <init>(JoinedSubclassEntityPersister.java:91)
>>>
>>> at
>>> org.hibernate.persister.PersisterFactory.createClassPersiste r(PersisterFactory.java:58)
>>>
>>> at
>>> org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:226)
>>>
>>> at
>>> org.hibernate.cfg.Configuration.buildSessionFactory(Configur ation.java:1294)
>>>
>>> at
>>> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getSessionFactory(HibernatePackageHandler.java:335 )
>>>
>>> at
>>> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.readPackageInfos(HibernatePackageHandler.java:293)
>>>
>>> at
>>> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getCDOPackageInfos(HibernatePackageHandler.java:21 5)
>>>
>>> at
>>> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Reader.readPackageInfos(HibernateStoreReader.java:87)
>>>
>>> at
>>> org.eclipse.emf.cdo.internal.server.PackageManager.doActivat e(PackageManager.java:72)
>>>
>>> at
>>> org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
>>>
>>> at
>>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>>>
>>> at
>>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>>>
>>> at
>>> org.eclipse.emf.cdo.internal.server.Repository.activateRepos itory(Repository.java:264)
>>>
>>> at
>>> org.eclipse.emf.cdo.internal.server.Repository.doActivate(Re pository.java:251)
>>>
>>> at
>>> org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
>>>
>>> at
>>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>>>
>>> at
>>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>>>
>>> at
>>> org.eclipse.emf.cdo.server.CDOServerUtil.addRepository(CDOSe rverUtil.java:67)
>>>
>>> at
>>> org.eclipse.cdo.bug.LaunchServerTest.setUp(LaunchServerTest. java:54)
>>> at junit.framework.TestCase.runBare(TestCase.java:128)
>>> at junit.framework.TestResult$1.protect(TestResult.java:106)
>>> at junit.framework.TestResult.runProtected(TestResult.java:124)
>>> at junit.framework.TestResult.run(TestResult.java:109)
>>> at junit.framework.TestCase.run(TestCase.java:120)
>>> at junit.framework.TestSuite.runTest(TestSuite.java:230)
>>> at junit.framework.TestSuite.run(TestSuite.java:225)
>>> at
>>> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
>>>
>>> at
>>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
>>>
>>> at
>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
>>>
>>> at
>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
>>>
>>> at
>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
>>>
>>> at
>>> org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
>>>
>>> at
>>> org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>> at
>>> org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hodWithException(EclipseAppContainer.java:574)
>>>
>>> at
>>> org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
>>>
>>> at
>>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:110)
>>>
>>> at
>>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:79)
>>>
>>> at
>>> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:379)
>>>
>>> at
>>> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:179)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>> at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 549)
>>> at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
>>> at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
>>> at org.eclipse.equinox.launcher.Main.main(Main.java:1212)
>>> Caused by: java.lang.reflect.InvocationTargetException
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>> at net.sf.cglib.core.ReflectUtils.defineClass(ReflectUtils.java :384)
>>> at
>>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>>
>>> ... 61 more
>>> Caused by: java.lang.NoClassDefFoundError:
>>> org/hibernate/proxy/HibernateProxy
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>>> ... 67 more
>>>
>>> I must admit that I'm still not sure about te fundamental interwork
>>> between Hibernate and the user defined meta model. It seems as if
>>> asm and cglib are somewhere in between but I have no idea why at
>>> all. I thought we have implemented a bunch of static Hibernate
>>> interfaces to make the CDORevision instances known to Hibernate.
>>> Please somebody explain why we need runtime byte code engineering?
>>> Is there some configuration missing to tell Hibernate to use our
>>> special tuplizers?
>>>
>>> I have changed several things in the code of the test plugin and
>>> since it's quite small now that the jars are removed, I attach my
>>> version of the zip here (including a JUnit Plugin launch config).
>>> And don't forget to update the cdo.server.hibernate.libraries plugin
>>> from CVS to get the new buddy policy!
>>>
>>> Cheers
>>> /Eike
>>>
>>> Martin Taal schrieb:
>>>> Hi Cédric,
>>>> I looked at it for many hours trying different things (using your
>>>> testcase) but I am kind of stuck with this (getting the same error
>>>> as you). The difference I can see with the way I run the
>>>> cdo.hibernate testcases (as a junit test) and how your testcase is
>>>> run (junit plugin test). This for sure means difference in class
>>>> loader behavior.
>>>>
>>>> Eike, do you have an idea? I am not sure how classloading works
>>>> when you start a cdo server in a plugin. Which plugins are then
>>>> available in the cdo server? The testcases can be downloaded from
>>>> the link below.
>>>>
>>>> gr. Martin
>>>>
>>>> Cédric Brun wrote:
>>>>> Thanks a lot ! I'm looking forward to your advices !
>>>>>
>>>>> Cédric
>>>>>
>>>>> Martin Taal wrote:
>>>>>
>>>>>> I am downloading your zip-file. I don't have time to look at it
>>>>>> right away
>>>>>> but I will do my best later this evening.
>>>>>>
>>>>>> gr. Martin
>>>>>>
>>>>>> Cédric Brun wrote:
>>>>>>> I tried to do the same things as in the CDO/Hibernate unit tests.
>>>>>>>
>>>>>>> Here's a zipped project trying to launch an HibernateServer with
>>>>>>> a unit
>>>>>>> test adding elements.
>>>>>>> ( http://cedric.brun.free.fr/Tortoose/org.eclipse.cdo-teneo-te st.zip)
>>>>>>> The
>>>>>>> libs I found are included in the same project, but I still have
>>>>>>> no luck
>>>>>>> and failling at :
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>>>>
>>>>>>> ... 62 more
>>>>>>> Caused by: java.lang.NoClassDefFoundError:
>>>>>>> org/hibernate/proxy/HibernateProxy
>>>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>>>>>>> ... 68 more
>>>>>>>
>>>>>>> please note that the test project I provide really is a "quick
>>>>>>> and ugly"
>>>>>>> one ^^
>>>>>>>
>>>>>>>
>>>>>>> Cédric
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Martin Taal wrote:
>>>>>>>
>>>>>>>> Hmm, I can understand your headache. Classpath issues are the most
>>>>>>>> annoying errors...
>>>>>>>>
>>>>>>>> In your junit test source file, what happens when you enter the
>>>>>>>> org.objectweb.asm.Type, can you source file see this class?
>>>>>>>> Do you have cglib in some other plugin which does not have asm?
>>>>>>>>
>>>>>>>> What I also find slightly strange is that cglib is used in this
>>>>>>>> case
>>>>>>>> because normally cdo objects are not proxied using cglib. Can
>>>>>>>> you post
>>>>>>>> the generated mapping? How does your cdo initialization code
>>>>>>>> look like?
>>>>>>>>
>>>>>>>> gr. Martin
>>>>>>>>
>>>>>>>> Cédric Brun wrote:
>>>>>>>>> Hi Martin,
>>>>>>>>>
>>>>>>>>> and thanks for your quick answer ! what I did is the following :
>>>>>>>>>
>>>>>>>>> I used the org.eclipse.emf.cdo.server.hibernate.libraries
>>>>>>>>> bundle and
>>>>>>>>> added that :
>>>>>>>>>
>>>>>>>>> Bundle-ClassPath: lib/hibernate3.jar,
>>>>>>>>> lib/cglib-2.1.3.jar,
>>>>>>>>> lib/commons-collections-2.1.1.jar,
>>>>>>>>> lib/dom4j.jar,
>>>>>>>>> lib/jta.jar
>>>>>>>>> lib/asm-3.0.jar
>>>>>>>>>
>>>>>>>>> and this directive :
>>>>>>>>>
>>>>>>>>> Eclipse-BuddyPolicy: dependent
>>>>>>>>>
>>>>>>>>> as stated in
>>>>>>>>> http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Then I don't get the same error, now when I start my own
>>>>>>>>> plugin (JUnit
>>>>>>>>> plugin test) depending on
>>>>>>>>> org.eclipse.emf.cdo.server.hibernate.libraries I get :
>>>>>>>>>
>>>>>>>>> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
>>>>>>>>> at
>>>>>>>>> net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
>>>>>>>>> at
>>>>>>>>> net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
>>>>>>>>> at net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
>>>>>>>>> at
>>>>>>>>>
>>>>> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>>>>>
>>>>>>>>> These jars are such a headache ! Do you know - by chance -
>>>>>>>>> what's
>>>>>>>>> going on ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cédric
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Martin Taal wrote:
>>>>>>>>>
>>>>>>>>>> Hi Cédric,
>>>>>>>>>> Most of the time this exception is thrown because hibernate
>>>>>>>>>> can't find
>>>>>>>>>> the class. A common cause is that the hibernate jar files are
>>>>>>>>>> located
>>>>>>>>>> in a separate plugin which can not reach the classes in other
>>>>>>>>>> plugins.
>>>>>>>>>> If you set the eclipse buddy policy of the hibernate plugin to
>>>>>>>>>> dependent then this can help.
>>>>>>>>>>
>>>>>>>>>> gr. Martin
>>>>>>>>>>
>>>>>>>>>> Cédric Brun wrote:
>>>>>>>>>>> Hi Simon and Eike,
>>>>>>>>>>>
>>>>>>>>>>> thanks for your feedback, now I know what I could expect in
>>>>>>>>>>> using
>>>>>>>>>>> optimized implementation, I'll have a try on the hibernate
>>>>>>>>>>> store and
>>>>>>>>>>> see if it is scaling in a better way.
>>>>>>>>>>>
>>>>>>>>>>> You gave me many pointers to explain the performances issues
>>>>>>>>>>> but in
>>>>>>>>>>> fact it seems that the SQL execution is taking so much time
>>>>>>>>>>> (not
>>>>>>>>>>> suprisingly in fact ;) . My first profile was wrong.
>>>>>>>>>>>
>>>>>>>>>>> So I'm installing the hibernate store retrieving all the
>>>>>>>>>>> source code
>>>>>>>>>>> from CVS, and I'm trying to launch the unit tests. I added
>>>>>>>>>>> many jars
>>>>>>>>>>> and everything gets compiled but most of the tests are
>>>>>>>>>>> failling with
>>>>>>>>>>> "
>>>>>>>>>>>
>>>>>>>>>>> "
>>>>>>>>>>> org.hibernate.MappingException: Could not determine type for:
>>>>>>>>>>>
>>>>> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>>>>
>>>>>>>>>>> at table: cdo_features, for columns:
>>>>>>>>>>> [org.hibernate.mapping.Column(type)] "
>>>>>>>>>>>
>>>>>>>>>>> any hint ?
>>>>>>>>>>>
>>>>>>>>>>> Cédric
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Simon McDuff wrote:
>>>>>>>>>>>
>>>>>>>>>>>> We noticed a performance degradation in the following case:
>>>>>>>>>>>>
>>>>>>>>>>>> When we fetch a collection of objects that contains
>>>>>>>>>>>> collections that
>>>>>>>>>>>> we don't want to use. Ex:
>>>>>>>>>>>>
>>>>>>>>>>>> Model:
>>>>>>>>>>>>
>>>>>>>>>>>> ClassB
>>>>>>>>>>>> {
>>>>>>>>>>>> int foo1;
>>>>>>>>>>>> List childrens1;
>>>>>>>>>>>> List childrens2;
>>>>>>>>>>>> List childrens3;
>>>>>>>>>>>> };
>>>>>>>>>>>>
>>>>>>>>>>>> Code:
>>>>>>>>>>>>
>>>>>>>>>>>> Resource.contents contains objects of type ClassB.
>>>>>>>>>>>>
>>>>>>>>>>>> By fetching instances of classB, we will also fetched some
>>>>>>>>>>>> items in
>>>>>>>>>>>> the list(childrens1, childrens2, childrens3) The problem we
>>>>>>>>>>>> had.. is
>>>>>>>>>>>> we do not know if we want to use it. SO better not to
>>>>>>>>>>>> download them.
>>>>>>>>>>>>
>>>>>>>>>>>> So In our production(we used our own IStore), we never
>>>>>>>>>>>> download any
>>>>>>>>>>>> objects from collection. I'm working at the moment to put
>>>>>>>>>>>> in place
>>>>>>>>>>>> all feature we used in the current CDO. That one will be
>>>>>>>>>>>> part of
>>>>>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477. To begin
>>>>>>>>>>>> with,CDORevisionImpl needs to supports list that are 0 even if
>>>>>>>>>>>> referenceChunkSize == 0.
>>>>>>>>>>>>
>>>>>>>>>>>> It is true that we obtain good performance.
>>>>>>>>>>>> Worst case :
>>>>>>>>>>>> 1500 objects /sec (Real fetch from disk. Page that contains
>>>>>>>>>>>> the
>>>>>>>>>>>> objects, wasn't in the cache of the filesystem.)
>>>>>>>>>>>>
>>>>>>>>>>>> Best case
>>>>>>>>>>>> 12 000 objects/Sec (Fetch objects that are already in the
>>>>>>>>>>>> case of
>>>>>>>>>>>> the filesystem)
>>>>>>>>>>>>
>>>>>>>>>>>> (These statistics always fetch objects from server to
>>>>>>>>>>>> client. The
>>>>>>>>>>>> cache client isn't used)
>>>>>>>>>>>>
>>>>>>>>>>>> We have another optimization one with SmartReadAhead but
>>>>>>>>>>>> for now
>>>>>>>>>>>> only work
>>>>>>>>>>>> for UI client. (for now) This is something we will
>>>>>>>>>>>> integrate as
>>>>>>>>>>>> well in CDO. But for now we concentrate on the other things.
>>>>>>>>>>>>
>>>>>>>>>>>> I used an Object database. Based on my preliminary test
>>>>>>>>>>>> without CDO.
>>>>>>>>>>>> I compare
>>>>>>>>>>>>
>>>>>>>>>>>> Hibernate vs Objectivity/EMF :
>>>>>>>>>>>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>>>>>>>>>>>
>>>>>>>>>>>> The worst.. you will get 30 to 240 objects/sec or
>>>>>>>>>>>> the best... you will get 500 to 4000 objects/sec
>>>>>>>>>>>>
>>>>>>>>>>>> Based on your comments... you are getting the worst (with
>>>>>>>>>>>> Derby)!!
>>>>>>>>>>>> :-)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Another comments about insertion.
>>>>>>>>>>>>
>>>>>>>>>>>> We found another problem when we insert data(DOn't know if
>>>>>>>>>>>> it is
>>>>>>>>>>>> still true). Resource.Contents is a unique list. Basically
>>>>>>>>>>>> each time
>>>>>>>>>>>> we insert one element, it needs to fetch all the collection
>>>>>>>>>>>> to see
>>>>>>>>>>>> if we already have this element. Basically.. it is time
>>>>>>>>>>>> consuming.
>>>>>>>>>>>> In our case we are not going through this stage to persist our
>>>>>>>>>>>> objects... but the objects are not accessible through that
>>>>>>>>>>>> list.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> If speed is an issue for you!! We (You included :-)) can
>>>>>>>>>>>> optimize
>>>>>>>>>>>> it. What I like is Eike did a really good job doing that
>>>>>>>>>>>> framwork
>>>>>>>>>>>> and optimizations are easy to add. Also you can change your
>>>>>>>>>>>> back-end
>>>>>>>>>>>> without affecting you client side. (Maybe going to an Object
>>>>>>>>>>>> database!!) THis was one of the purpose of CDO!! :-)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Simon
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le
>>>>>>>>>>>> message de
>>>>>>>>>>>> news: 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>>>>>>>>>
>>>>>>>>>>>> That sounds interesting!
>>>>>>>>>>>> Comments inline...
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Cédric Brun schrieb:
>>>>>>>>>>>> Hi Eike,
>>>>>>>>>>>>
>>>>>>>>>>>> I've been messing around CDO lately and I hacked a few ugly
>>>>>>>>>>>> unit-tests measuring read/write throughput when I use CDO
>>>>>>>>>>>> (RC2).
>>>>>>>>>>>>
>>>>>>>>>>>> I'm using an embeded-derby database, two different metamodels
>>>>>>>>>>>> (complex one - 250 EClassifiers/300 features and a simple
>>>>>>>>>>>> one - 1
>>>>>>>>>>>> Classifier/3 Features)
>>>>>>>>>>>>
>>>>>>>>>>>> The test is a "quick and dirty" one so please do not take
>>>>>>>>>>>> these
>>>>>>>>>>>> figures too seriously, I would just like to have your input
>>>>>>>>>>>> about
>>>>>>>>>>>> the "order of magnitude" I should expect or settings affecting
>>>>>>>>>>>> seriously these performances. I tried both JVM and TCP
>>>>>>>>>>>> acceptor but
>>>>>>>>>>>> the performances are quite similar though JVM is a bit faster.
>>>>>>>>>>>> Looks like a sign that the NIO socket channels are
>>>>>>>>>>>> efficient ,-)
>>>>>>>>>>>> You used localhost for TCP transport I guess?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> * First case : building a small model (10000 elements) with
>>>>>>>>>>>> the
>>>>>>>>>>>> simple MM, only adding instances in an empty database :
>>>>>>>>>>>> commiting change on every add I get a 2 objects per second
>>>>>>>>>>>> rate.
>>>>>>>>>>>> commiting changes every 10 adds - I get a 30 objects per
>>>>>>>>>>>> second
>>>>>>>>>>>> rate.
>>>>>>>>>>>> Seems to be consistent with the fact that only for
>>>>>>>>>>>> commits there
>>>>>>>>>>>> is remote traffic. And the backend transaction management is
>>>>>>>>>>>> another overhead. So the impact of the number objects/commit
>>>>>>>>>>>> starts with bigger transactions.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> browsing the just commited elements opening a new
>>>>>>>>>>>> transaction and
>>>>>>>>>>>> getting a new Resource instance, I get around 30 000
>>>>>>>>>>>> objets/sec rate
>>>>>>>>>>>> just iterating thanks to Resource.getAllContents().
>>>>>>>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>>>>>>>> All views and transactions of a single session share (and
>>>>>>>>>>>> cache)
>>>>>>>>>>>> their object state. Otherwise I'd expect much smaller
>>>>>>>>>>>> values.
>>>>>>>>>>>> With his ObjectivityStore implementation Simon (cc'ed)
>>>>>>>>>>>> reached
>>>>>>>>>>>> rates of several thousands objects/sec. But the DBStore
>>>>>>>>>>>> (which you
>>>>>>>>>>>> used) is not that optimized.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> * Second case : I built a big model in the database using the
>>>>>>>>>>>> complex metamodel. It contains 700 000 elements. The
>>>>>>>>>>>> creation went
>>>>>>>>>>>> pretty nicely (I did not measured it exactly as it was
>>>>>>>>>>>> launched
>>>>>>>>>>>> during a night, but it seems I got around 100 objects/sec
>>>>>>>>>>>> rate in
>>>>>>>>>>>> writting as I was writting every 700 objects.
>>>>>>>>>>>> Sounds not too bad.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Once the model is built the derby database is 222 MB (which
>>>>>>>>>>>> - I
>>>>>>>>>>>> heard - is not so big for derby).
>>>>>>>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>>>>>>>> To be honest, I'm really no database expert (although I
>>>>>>>>>>>> can write
>>>>>>>>>>>> simple SQL statements).
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Retrieving the distant resource and calling eAllContents()
>>>>>>>>>>>> on it, I
>>>>>>>>>>>> get a 50 Objects/sec rate at best, which mean my
>>>>>>>>>>>> eAllContents() call
>>>>>>>>>>>> would take 4 hours !
>>>>>>>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>>>>>>>> (iteration order and so). There are several CDO features
>>>>>>>>>>>> that can
>>>>>>>>>>>> have a major impact on load rates!
>>>>>>>>>>>>
>>>>>>>>>>>> - Collections (xrefs and containment) can be loaded
>>>>>>>>>>>> lazily. I
>>>>>>>>>>>> don't mean the target objects, which are lazy by default
>>>>>>>>>>>> anyway,
>>>>>>>>>>>> but even the target IDs that are in the collections. We
>>>>>>>>>>>> call that
>>>>>>>>>>>> "reference chunking" (but search for a better name is
>>>>>>>>>>>> ongoing).
>>>>>>>>>>>>
>>>>>>>>>>>> public interface CDOSession extends
>>>>>>>>>>>> CDOProtocolSession,
>>>>>>>>>>>> IContainer<CDOView>
>>>>>>>>>>>> {
>>>>>>>>>>>> public int getReferenceChunkSize();
>>>>>>>>>>>>
>>>>>>>>>>>> public void setReferenceChunkSize(int
>>>>>>>>>>>> referenceChunkSize);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> - Loading collection elements (target objects) can be
>>>>>>>>>>>> batched (the
>>>>>>>>>>>> current name is even worse)
>>>>>>>>>>>>
>>>>>>>>>>>> public interface CDOView extends CDOProtocolView,
>>>>>>>>>>>> INotifier
>>>>>>>>>>>> {
>>>>>>>>>>>> public int getLoadRevisionCollectionChunkSize();
>>>>>>>>>>>>
>>>>>>>>>>>> public void setLoadRevisionCollectionChunkSize(int
>>>>>>>>>>>> loadRevisionCollectionChunkSize);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> - One of the most sophisticated feature is automatic
>>>>>>>>>>>> model usage
>>>>>>>>>>>> analyzation. The result are fetch rules that are sent to the
>>>>>>>>>>>> server. When later objects are loaded from the server, it
>>>>>>>>>>>> can
>>>>>>>>>>>> answer with additional objects that are likely to be
>>>>>>>>>>>> needed by the
>>>>>>>>>>>> client in the near future.
>>>>>>>>>>>>
>>>>>>>>>>>> public class CDOViewImpl extends
>>>>>>>>>>>> org.eclipse.net4j.internal.util.event.Notifier
>>>>>>>>>>>> implements
>>>>>>>>>>>> CDOView, CDOIDProvider, Adapter.Internal
>>>>>>>>>>>> {
>>>>>>>>>>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>>>>>>>>> {
>>>>>>>>>>>> return featureAnalyzer;
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>>>>>>>>> featureAnalyzer)
>>>>>>>>>>>> {
>>>>>>>>>>>> this.featureAnalyzer = featureAnalyzer == null ?
>>>>>>>>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> - There's a SmartReadAheadThread (if you're curious,
>>>>>>>>>>>> please ask
>>>>>>>>>>>> Simon). - Caches can be configured
>>>>>>>>>>>> - ...
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Known issues of the DBStore:
>>>>>>>>>>>> - When in auditing mode updates are always written as
>>>>>>>>>>>> complete
>>>>>>>>>>>> inserts (by design) - When configuring toManyReferences
>>>>>>>>>>>> to a value
>>>>>>>>>>>> different from LIKE_ATTRIBUTES (which is not implemented
>>>>>>>>>>>> yet) it
>>>>>>>>>>>> is very slow. - No (few) prepared statements are used (see
>>>>>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) -
>>>>>>>>>>>> Maybe
>>>>>>>>>>>> others...
>>>>>>>>>>>>
>>>>>>>>>>>> Please note that the DBStore was not written by me to
>>>>>>>>>>>> compete with
>>>>>>>>>>>> Hibernate in terms of performance or mapping
>>>>>>>>>>>> capabilities. Now
>>>>>>>>>>>> that Martin and I, well in the end mostly Martin, provide a
>>>>>>>>>>>> HibernateStore and this HibernateStore gets more and more
>>>>>>>>>>>> features
>>>>>>>>>>>> I expect that there will be a trend away from the DBStore.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This is probably linked to the way the mapping to db is
>>>>>>>>>>>> done, here's
>>>>>>>>>>>> the config I used (came from CDO-server example if I recall
>>>>>>>>>>>> correctly).
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> <repository name="repo1">
>>>>>>>>>>>> <property name="overrideUUID"
>>>>>>>>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>>>>>>>>> <property name="supportingAudits"
>>>>>>>>>>>> value="true"/>
>>>>>>>>>>>> <property name="verifyingRevisions"
>>>>>>>>>>>> value="false"/>
>>>>>>>>>>>> <property name="currentLRUCapacity"
>>>>>>>>>>>> value="10000"/>
>>>>>>>>>>>> I guess this one could be larger.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> <property name="revisedLRUCapacity"
>>>>>>>>>>>> value="100"/>
>>>>>>>>>>>> <store type="db">
>>>>>>>>>>>> <!-- type: horizontal | vertical |
>>>>>>>>>>>> <any
>>>>>>>>>>>> user-contributed type> -->
>>>>>>>>>>>> <mappingStrategy
>>>>>>>>>>>> type="horizontal">
>>>>>>>>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>>>>>>>>> ONE_TABLE_PER_CLASS |
>>>>>>>>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY |
>>>>>>>>>>>> LIKE_ATTRIBUTES
>>>>>>>>>>>> -->
>>>>>>>>>>>> <property
>>>>>>>>>>>> name="toManyReferences"
>>>>>>>>>>>>
>>>>>>>>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>>>>>>>>
>>>>>>>>>>>> <!-- LIKE_ATTRIBUTES |
>>>>>>>>>>>> LIKE_TO_MANY_REFERENCES -->
>>>>>>>>>>>> <property
>>>>>>>>>>>> name="toOneReferences"
>>>>>>>>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>>>>>>>>
>>>>>>>>>>>> <!-- MODEL | STRATEGY -->
>>>>>>>>>>>> <property
>>>>>>>>>>>&g
Re: [CDO] reading/writting throughput performances (SUCCESS) [message #124736 is a reply to message #124561] Sun, 01 June 2008 13:05 Go to previous messageGo to next message
Eclipse UserFriend
Originally posted by: stepper.sympedia.de

This is a multi-part message in MIME format.
--------------080000030302050701060709
Content-Type: multipart/alternative;
boundary="------------070501090100080408030009"


--------------070501090100080408030009
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Martin,

In the console log snippet below, please notice that it is the first
commit transaction signal and that there are no new packages contained.
This is caused by the client (test case) that didn't register the used
package with the client session before using it. The following snippet
uses the new automatic registry:

| *private *CDOSession createSession()
{
IConnector connector = (IConnector)IPluginContainer.INSTANCE.getElement("org.eclipse.net4j.connectors ", "jvm", "default");
System.out.println(connector);

CDOSessionConfiguration configuration = CDOUtil.createSessionConfiguration();
configuration.setConnector(connector);
configuration.setRepositoryName(REPO_NAME);
configuration.setDemandPopulatingPackageRegistry();
*return *configuration.openSession();
}|


Now the test cases pass properly!!!
Please see the attached log file.

Cheers
/Eike



Eike Stepper schrieb:
> Hi Martin,
>
> Ah, now with the new mapping I get a big step further!
> As I said, Cedric's test code contained some bugs, too. IIRC a TCP
> acceptor was created but the client uses a JVMConnector, and stuff
> like this.
> You could try to overcome the exception below by adding the following
> to the server code somewhere:
>
> | IPluginContainer.INSTANCE.getElement("org.eclipse.net4j.acceptors ", "jvm", "default");|
>
>
>
> Better just use the attached zip because there may be other changes
> that I don't remember in detail anymore.
> Now I run into another exception:
>
> Thread-4 [debug.signal] ================ Indicating
> CommitTransactionIndication
> Thread-4 [debug] Created
> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Writer for
> repository repo1
> Thread-4 [debug] Created
> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Writer for
> repository repo1
> Thread-4 [debug] Created
> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Writer for
> repository repo1
> Thread-4 [debug.protocol] Reading 0 new packages
> Thread-4 [debug.protocol] Reading 2 new objects
> Thread-4 [debug.model] Reading CDOID of type 3 (TEMP_OBJECT)
> Thread-4 [debug.model] Reading CDOID of type 3 (TEMP_OBJECT)
> Thread-4 [debug.model] Reading CDOID of type 0 (NULL)
> Thread-4 [debug.revision] Reading revision: ID=oid1,
> classRef=CDOClassRef(http://www.eclipse.org/emf/CDO/resource/1.0.0,
> 0), className=CDOResource, version=1, created=0, revised=0,
> resource=oid1, container=NULL, feature=0
> Thread-4 [debug.revision] Read feature CDOFeature(ID=9, name=path,
> type=STRING, referenceType=null): /res2
> Thread-4 [debug.revision] Read feature CDOFeature(ID=2, name=contents,
> type=OBJECT, referenceType=CDOClass(ID=0, name=CDOObject)): size=1
> Thread-4 [debug.model] Reading CDOID of type 3 (TEMP_OBJECT)
> Thread-4 [debug.revision] oid2
> [ERROR] ClassRef unresolveable:
> CDOClassRef(http://www.eclipse.org/modeling/cdo/bug/names, 0)
> java.lang.IllegalStateException: ClassRef unresolveable:
> CDOClassRef(http://www.eclipse.org/modeling/cdo/bug/names, 0)
> at
> org.eclipse.emf.cdo.internal.protocol.revision.CDORevisionIm pl. <init>(CDORevisionImpl.java:116)
> at
> org.eclipse.emf.cdo.protocol.revision.CDORevisionUtil.read(C DORevisionUtil.java:48)
> at
> org.eclipse.emf.cdo.internal.server.protocol.CommitTransacti onIndication.indicating(CommitTransactionIndication.java:95)
> at
> org.eclipse.net4j.signal.IndicationWithResponse.execute(Indi cationWithResponse.java:46)
> at org.eclipse.net4j.signal.Signal.runSync(Signal.java:143)
> at org.eclipse.net4j.signal.Signal.run(Signal.java:124)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Threa dPoolExecutor.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo lExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
>
> Cheers
> /Eike
>
>
>
> Martin Taal schrieb:
>> Hi Eike, Cedric,
>> I can't say to much about net4j.db versus net4j.db.hsqldb.
>>
>> You are correct that the hibernate.libraries plugin should have
>> Eclipse-BuddyPolicy: dependent (or another way to use the classpath
>> of the dependent plugins). Btw, I agree that registered is better as
>> afaik dependent gets slow which larger plugin environments.
>>
>> The reason that cglib is used is because one of the class mappings in
>> org.eclipse.emf.cdo.server.hibernate/mappings/meta.hbm/xml did not
>> have the lazy="false" directive. I added this and now I get another
>> exception (see below). So this 'solves' the problem. If you do a get
>> latest of this file then I think this problem is 'overcome'.
>>
>> Overall I still do not see why it went wrong. As far as I can see the
>> cglib library could not find the hibernate class, which is strange as
>> they are in the same plugin??
>> The cdo testcases don't have this problem and neither does Teneo.
>>
>> As it does not occur anymore we can ignore it for now (not very
>> satisfactory but there are other things to do)... Unless someone else
>> has an idea how to handle this ofcourse.
>>
>> gr. Martin
>>
>> java.lang.IllegalStateException: acceptor == null
>> at
>> org.eclipse.net4j.internal.jvm.JVMClientConnector.doBeforeAc tivate(JVMClientConnector.java:57)
>>
>> at
>> org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:56)
>>
>> at
>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>>
>> at
>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>>
>> at
>> org.eclipse.net4j.internal.util.container.ManagedContainer.g etElement(ManagedContainer.java:260)
>>
>> at
>> org.eclipse.emf.internal.cdo.util.ChannelInjector.getConnect or(ChannelInjector.java:74)
>>
>> at
>> org.eclipse.emf.internal.cdo.util.ChannelInjector.process(Ch annelInjector.java:41)
>>
>> at
>> org.eclipse.net4j.internal.util.container.ManagedContainer.p ostProcessElement(ManagedContainer.java:407)
>>
>> at
>> org.eclipse.net4j.internal.util.container.ManagedContainer.g etElement(ManagedContainer.java:259)
>>
>> at
>> org.eclipse.cdo.bug.hibernate.HibernateServer.createSession( HibernateServer.java:93)
>>
>> at
>> org.eclipse.cdo.bug.unit.LaunchServerTest.testStringValue(La unchServerTest.java:43)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>
>> at java.lang.reflect.Method.invoke(Method.java:585)
>> at junit.framework.TestCase.runTest(TestCase.java:164)
>> at junit.framework.TestCase.runBare(TestCase.java:130)
>> at junit.framework.TestResult$1.protect(TestResult.java:106)
>> at junit.framework.TestResult.runProtected(TestResult.java:124)
>> at junit.framework.TestResult.run(TestResult.java:109)
>> at junit.framework.TestCase.run(TestCase.java:120)
>> at junit.framework.TestSuite.runTest(TestSuite.java:230)
>> at junit.framework.TestSuite.run(TestSuite.java:225)
>> at
>> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
>>
>> at
>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
>>
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
>>
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
>>
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
>>
>> at
>> org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
>>
>> at
>> org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>
>> at java.lang.reflect.Method.invoke(Method.java:585)
>> at
>> org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hod(EclipseAppContainer.java:547)
>>
>> at
>> org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
>>
>> at
>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:106)
>>
>> at
>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:76)
>>
>> at
>> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:362)
>>
>> at
>> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:175)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>
>> at java.lang.reflect.Method.invoke(Method.java:585)
>> at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 564)
>> at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
>> at org.eclipse.equinox.launcher.Main.run(Main.java:1251)
>> at org.eclipse.equinox.launcher.Main.main(Main.java:1227)
>>
>>
>>
>> Eike Stepper wrote:
>>> Hi,
>>>
>>> Here is a diagram of the dependencies of the relevant plugins
>>> (please correct me if something is wrong).
>>> The black arrows are ordinary downstream dependencies (Require-Bundle).
>>> The red arrows are upstream dependencies (Eclipse-BuddyPolicy:
>>> dependent).
>>>
>>>
>>>
>>> For research purposes I added code to the libraries plugin that
>>> tries to reflectively load CDOFeatureImpl and that is not a problem!
>>> It seems to me as if our problem is not class loading related
>>> because the hibernate.jar should be able to see the needed classes.
>>> I believe this is either a bug in Hibernate or our wrong usage of
>>> Hibernate. I tend to believe the latter ;-)
>>> I put a breakpoint in
>>> org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping .EntityEntityModeToTuplizerMapping(PersistentClass,
>>> EntityMetamodel)
>>> to see how the tuplizers are created. I'm too unexperienced with
>>> Hibernate to get a clue about what happens there...
>>>
>>> Btw. I'd prefer to see the application plugin depend on net4j.db
>>> instead of net4j.db.hsqldb.
>>> But it seems as if the BuddyPolicy in net4j.db is not "reexported"
>>> so that the application can see the hsqldb.jar then ;-(
>>> I posted a respective question to equinox...
>>>
>>> Cheers
>>> /Eike
>>>
>>>
>>> Eike Stepper schrieb:
>>>> Hi guys,
>>>>
>>>> I can confirm, making these 3rd party libraries with their own
>>>> classloading run in OSGi is a nightmare! I took me hours teach
>>>> where the hsqldb driver is. On the way I came over different kinds
>>>> of classloading related errors.
>>>>
>>>> First I removed all the jars from the test plugin (the one Cedric
>>>> provided). They're already in the cdo.server.hibernate.libraries
>>>> plugin which, in fact, needs the buddy policy set to dependent. I
>>>> think this last change needs to be committed to CVS! Btw. I'd
>>>> prefer to use *registered* buddies instead of making all dependent
>>>> bundles visible.
>>>>
>>>> The test plugin then needs to provide the database driver.
>>>> Therefore I added a dependency on the org.eclipse.net4j.db.hsqldb
>>>> plugin.
>>>>
>>>> With this setup I can start the server without CNFE, NCDFE, CCI and
>>>> the like. At this point I get the following exception, which seems
>>>> the same as the one Cedric observed in the end:
>>>>
>>>> net.sf.cglib.core.CodeGenerationException:
>>>> java.lang.reflect.InvocationTargetException-->null
>>>> at
>>>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:237)
>>>>
>>>> at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
>>>> at net.sf.cglib.proxy.Enhancer.createClass(Enhancer.java:317)
>>>> at
>>>> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:127)
>>>>
>>>> at
>>>> org.hibernate.proxy.pojo.cglib.CGLIBProxyFactory.postInstant iate(CGLIBProxyFactory.java:43)
>>>>
>>>> at
>>>> org.hibernate.tuple.entity.PojoEntityTuplizer.buildProxyFact ory(PojoEntityTuplizer.java:162)
>>>>
>>>> at
>>>> org.hibernate.tuple.entity.AbstractEntityTuplizer.<init>(AbstractEntityTuplizer.java:135)
>>>>
>>>> at
>>>> org.hibernate.tuple.entity.PojoEntityTuplizer.<init>(PojoEntityTuplizer.java:55)
>>>>
>>>> at
>>>> org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping . <init>(EntityEntityModeToTuplizerMapping.java:56)
>>>>
>>>> at
>>>> org.hibernate.tuple.entity.EntityMetamodel.<init>(EntityMetamodel.java:295)
>>>>
>>>> at
>>>> org.hibernate.persister.entity.AbstractEntityPersister.<init >(AbstractEntityPersister.java:434)
>>>>
>>>> at
>>>> org.hibernate.persister.entity.JoinedSubclassEntityPersister . <init>(JoinedSubclassEntityPersister.java:91)
>>>>
>>>> at
>>>> org.hibernate.persister.PersisterFactory.createClassPersiste r(PersisterFactory.java:58)
>>>>
>>>> at
>>>> org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:226)
>>>>
>>>> at
>>>> org.hibernate.cfg.Configuration.buildSessionFactory(Configur ation.java:1294)
>>>>
>>>> at
>>>> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getSessionFactory(HibernatePackageHandler.java:335 )
>>>>
>>>> at
>>>> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.readPackageInfos(HibernatePackageHandler.java:293)
>>>>
>>>> at
>>>> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getCDOPackageInfos(HibernatePackageHandler.java:21 5)
>>>>
>>>> at
>>>> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Reader.readPackageInfos(HibernateStoreReader.java:87)
>>>>
>>>> at
>>>> org.eclipse.emf.cdo.internal.server.PackageManager.doActivat e(PackageManager.java:72)
>>>>
>>>> at
>>>> org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
>>>>
>>>> at
>>>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>>>>
>>>> at
>>>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>>>>
>>>> at
>>>> org.eclipse.emf.cdo.internal.server.Repository.activateRepos itory(Repository.java:264)
>>>>
>>>> at
>>>> org.eclipse.emf.cdo.internal.server.Repository.doActivate(Re pository.java:251)
>>>>
>>>> at
>>>> org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
>>>>
>>>> at
>>>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>>>>
>>>> at
>>>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>>>>
>>>> at
>>>> org.eclipse.emf.cdo.server.CDOServerUtil.addRepository(CDOSe rverUtil.java:67)
>>>>
>>>> at
>>>> org.eclipse.cdo.bug.LaunchServerTest.setUp(LaunchServerTest. java:54)
>>>> at junit.framework.TestCase.runBare(TestCase.java:128)
>>>> at junit.framework.TestResult$1.protect(TestResult.java:106)
>>>> at junit.framework.TestResult.runProtected(TestResult.java:124)
>>>> at junit.framework.TestResult.run(TestResult.java:109)
>>>> at junit.framework.TestCase.run(TestCase.java:120)
>>>> at junit.framework.TestSuite.runTest(TestSuite.java:230)
>>>> at junit.framework.TestSuite.run(TestSuite.java:225)
>>>> at
>>>> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
>>>>
>>>> at
>>>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
>>>>
>>>> at
>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
>>>>
>>>> at
>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
>>>>
>>>> at
>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
>>>>
>>>> at
>>>> org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
>>>>
>>>> at
>>>> org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
>>>>
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>>
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>> at
>>>> org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hodWithException(EclipseAppContainer.java:574)
>>>>
>>>> at
>>>> org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
>>>>
>>>> at
>>>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:110)
>>>>
>>>> at
>>>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:79)
>>>>
>>>> at
>>>> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:379)
>>>>
>>>> at
>>>> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:179)
>>>>
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>>
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>> at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 549)
>>>> at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
>>>> at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
>>>> at org.eclipse.equinox.launcher.Main.main(Main.java:1212)
>>>> Caused by: java.lang.reflect.InvocationTargetException
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>>
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>> at
>>>> net.sf.cglib.core.ReflectUtils.defineClass(ReflectUtils.java :384)
>>>> at
>>>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>>>
>>>> ... 61 more
>>>> Caused by: java.lang.NoClassDefFoundError:
>>>> org/hibernate/proxy/HibernateProxy
>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>>>> ... 67 more
>>>>
>>>> I must admit that I'm still not sure about te fundamental interwork
>>>> between Hibernate and the user defined meta model. It seems as if
>>>> asm and cglib are somewhere in between but I have no idea why at
>>>> all. I thought we have implemented a bunch of static Hibernate
>>>> interfaces to make the CDORevision instances known to Hibernate.
>>>> Please somebody explain why we need runtime byte code engineering?
>>>> Is there some configuration missing to tell Hibernate to use our
>>>> special tuplizers?
>>>>
>>>> I have changed several things in the code of the test plugin and
>>>> since it's quite small now that the jars are removed, I attach my
>>>> version of the zip here (including a JUnit Plugin launch config).
>>>> And don't forget to update the cdo.server.hibernate.libraries
>>>> plugin from CVS to get the new buddy policy!
>>>>
>>>> Cheers
>>>> /Eike
>>>>
>>>> Martin Taal schrieb:
>>>>> Hi Cédric,
>>>>> I looked at it for many hours trying different things (using your
>>>>> testcase) but I am kind of stuck with this (getting the same error
>>>>> as you). The difference I can see with the way I run the
>>>>> cdo.hibernate testcases (as a junit test) and how your testcase is
>>>>> run (junit plugin test). This for sure means difference in class
>>>>> loader behavior.
>>>>>
>>>>> Eike, do you have an idea? I am not sure how classloading works
>>>>> when you start a cdo server in a plugin. Which plugins are then
>>>>> available in the cdo server? The testcases can be downloaded from
>>>>> the link below.
>>>>>
>>>>> gr. Martin
>>>>>
>>>>> Cédric Brun wrote:
>>>>>> Thanks a lot ! I'm looking forward to your advices !
>>>>>>
>>>>>> Cédric
>>>>>>
>>>>>> Martin Taal wrote:
>>>>>>
>>>>>>> I am downloading your zip-file. I don't have time to look at it
>>>>>>> right away
>>>>>>> but I will do my best later this evening.
>>>>>>>
>>>>>>> gr. Martin
>>>>>>>
>>>>>>> Cédric Brun wrote:
>>>>>>>> I tried to do the same things as in the CDO/Hibernate unit tests.
>>>>>>>>
>>>>>>>> Here's a zipped project trying to launch an HibernateServer
>>>>>>>> with a unit
>>>>>>>> test adding elements.
>>>>>>>> ( http://cedric.brun.free.fr/Tortoose/org.eclipse.cdo-teneo-te st.zip)
>>>>>>>> The
>>>>>>>> libs I found are included in the same project, but I still have
>>>>>>>> no luck
>>>>>>>> and failling at :
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>>>>>
>>>>>>>> ... 62 more
>>>>>>>> Caused by: java.lang.NoClassDefFoundError:
>>>>>>>> org/hibernate/proxy/HibernateProxy
>>>>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>>>>>>>> ... 68 more
>>>>>>>>
>>>>>>>> please note that the test project I provide really is a "quick
>>>>>>>> and ugly"
>>>>>>>> one ^^
>>>>>>>>
>>>>>>>>
>>>>>>>> Cédric
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Martin Taal wrote:
>>>>>>>>
>>>>>>>>> Hmm, I can understand your headache. Classpath issues are the
>>>>>>>>> most
>>>>>>>>> annoying errors...
>>>>>>>>>
>>>>>>>>> In your junit test source file, what happens when you enter the
>>>>>>>>> org.objectweb.asm.Type, can you source file see this class?
>>>>>>>>> Do you have cglib in some other plugin which does not have asm?
>>>>>>>>>
>>>>>>>>> What I also find slightly strange is that cglib is used in
>>>>>>>>> this case
>>>>>>>>> because normally cdo objects are not proxied using cglib. Can
>>>>>>>>> you post
>>>>>>>>> the generated mapping? How does your cdo initialization code
>>>>>>>>> look like?
>>>>>>>>>
>>>>>>>>> gr. Martin
>>>>>>>>>
>>>>>>>>> Cédric Brun wrote:
>>>>>>>>>> Hi Martin,
>>>>>>>>>>
>>>>>>>>>> and thanks for your quick answer ! what I did is the
>>>>>>>>>> following :
>>>>>>>>>>
>>>>>>>>>> I used the org.eclipse.emf.cdo.server.hibernate.libraries
>>>>>>>>>> bundle and
>>>>>>>>>> added that :
>>>>>>>>>>
>>>>>>>>>> Bundle-ClassPath: lib/hibernate3.jar,
>>>>>>>>>> lib/cglib-2.1.3.jar,
>>>>>>>>>> lib/commons-collections-2.1.1.jar,
>>>>>>>>>> lib/dom4j.jar,
>>>>>>>>>> lib/jta.jar
>>>>>>>>>> lib/asm-3.0.jar
>>>>>>>>>>
>>>>>>>>>> and this directive :
>>>>>>>>>>
>>>>>>>>>> Eclipse-BuddyPolicy: dependent
>>>>>>>>>>
>>>>>>>>>> as stated in
>>>>>>>>>> http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Then I don't get the same error, now when I start my own
>>>>>>>>>> plugin (JUnit
>>>>>>>>>> plugin test) depending on
>>>>>>>>>> org.eclipse.emf.cdo.server.hibernate.libraries I get :
>>>>>>>>>>
>>>>>>>>>> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
>>>>>>>>>> at
>>>>>>>>>> net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
>>>>>>>>>> at
>>>>>>>>>> net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
>>>>>>>>>> at
>>>>>>>>>> net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>>>>>>
>>>>>>>>>> These jars are such a headache ! Do you know - by chance -
>>>>>>>>>> what's
>>>>>>>>>> going on ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Cédric
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Martin Taal wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Cédric,
>>>>>>>>>>> Most of the time this exception is thrown because hibernate
>>>>>>>>>>> can't find
>>>>>>>>>>> the class. A common cause is that the hibernate jar files
>>>>>>>>>>> are located
>>>>>>>>>>> in a separate plugin which can not reach the classes in
>>>>>>>>>>> other plugins.
>>>>>>>>>>> If you set the eclipse buddy policy of the hibernate plugin to
>>>>>>>>>>> dependent then this can help.
>>>>>>>>>>>
>>>>>>>>>>> gr. Martin
>>>>>>>>>>>
>>>>>>>>>>> Cédric Brun wrote:
>>>>>>>>>>>> Hi Simon and Eike,
>>>>>>>>>>>>
>>>>>>>>>>>> thanks for your feedback, now I know what I could expect in
>>>>>>>>>>>> using
>>>>>>>>>>>> optimized implementation, I'll have a try on the hibernate
>>>>>>>>>>>> store and
>>>>>>>>>>>> see if it is scaling in a better way.
>>>>>>>>>>>>
>>>>>>>>>>>> You gave me many pointers to explain the performances
>>>>>>>>>>>> issues but in
>>>>>>>>>>>> fact it seems that the SQL execution is taking so much time
>>>>>>>>>>>> (not
>>>>>>>>>>>> suprisingly in fact ;) . My first profile was wrong.
>>>>>>>>>>>>
>>>>>>>>>>>> So I'm installing the hibernate store retrieving all the
>>>>>>>>>>>> source code
>>>>>>>>>>>> from CVS, and I'm trying to launch the unit tests. I added
>>>>>>>>>>>> many jars
>>>>>>>>>>>> and everything gets compiled but most of the tests are
>>>>>>>>>>>> failling with
>>>>>>>>>>>> "
>>>>>>>>>>>>
>>>>>>>>>>>> "
>>>>>>>>>>>> org.hibernate.MappingException: Could not determine type for:
>>>>>>>>>>>>
>>>>>> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>>>>>
>>>>>>>>>>>> at table: cdo_features, for columns:
>>>>>>>>>>>> [org.hibernate.mapping.Column(type)] "
>>>>>>>>>>>>
>>>>>>>>>>>> any hint ?
>>>>>>>>>>>>
>>>>>>>>>>>> Cédric
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Simon McDuff wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> We noticed a performance degradation in the following case:
>>>>>>>>>>>>>
>>>>>>>>>>>>> When we fetch a collection of objects that contains
>>>>>>>>>>>>> collections that
>>>>>>>>>>>>> we don't want to use. Ex:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Model:
>>>>>>>>>>>>>
>>>>>>>>>>>>> ClassB
>>>>>>>>>>>>> {
>>>>>>>>>>>>> int foo1;
>>>>>>>>>>>>> List childrens1;
>>>>>>>>>>>>> List childrens2;
>>>>>>>>>>>>> List childrens3;
>>>>>>>>>>>>> };
>>>>>>>>>>>>>
>>>>>>>>>>>>> Code:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Resource.contents contains objects of type ClassB.
>>>>>>>>>>>>>
>>>>>>>>>>>>> By fetching instances of classB, we will also fetched some
>>>>>>>>>>>>> items in
>>>>>>>>>>>>> the list(childrens1, childrens2, childrens3) The problem
>>>>>>>>>>>>> we had.. is
>>>>>>>>>>>>> we do not know if we want to use it. SO better not to
>>>>>>>>>>>>> download them.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So In our production(we used our own IStore), we never
>>>>>>>>>>>>> download any
>>>>>>>>>>>>> objects from collection. I'm working at the moment to put
>>>>>>>>>>>>> in place
>>>>>>>>>>>>> all feature we used in the current CDO. That one will be
>>>>>>>>>>>>> part of
>>>>>>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477. To
>>>>>>>>>>>>> begin
>>>>>>>>>>>>> with,CDORevisionImpl needs to supports list that are 0
>>>>>>>>>>>>> even if
>>>>>>>>>>>>> referenceChunkSize == 0.
>>>>>>>>>>>>>
>>>>>>>>>>>>> It is true that we obtain good performance.
>>>>>>>>>>>>> Worst case :
>>>>>>>>>>>>> 1500 objects /sec (Real fetch from disk. Page that
>>>>>>>>>>>>> contains the
>>>>>>>>>>>>> objects, wasn't in the cache of the filesystem.)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best case
>>>>>>>>>>>>> 12 000 objects/Sec (Fetch objects that are already in the
>>>>>>>>>>>>> case of
>>>>>>>>>>>>> the filesystem)
>>>>>>>>>>>>>
>>>>>>>>>>>>> (These statistics always fetch objects from server to
>>>>>>>>>>>>> client. The
>>>>>>>>>>>>> cache client isn't used)
>>>>>>>>>>>>>
>>>>>>>>>>>>> We have another optimization one with SmartReadAhead but
>>>>>>>>>>>>> for now
>>>>>>>>>>>>> only work
>>>>>>>>>>>>> for UI client. (for now) This is something we will
>>>>>>>>>>>>> integrate as
>>>>>>>>>>>>> well in CDO. But for now we concentrate on the other things.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I used an Object database. Based on my preliminary test
>>>>>>>>>>>>> without CDO.
>>>>>>>>>>>>> I compare
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hibernate vs Objectivity/EMF :
>>>>>>>>>>>>> Objectivity/EMF is around 3 to 50 times faster than
>>>>>>>>>>>>> Hibernate.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The worst.. you will get 30 to 240 objects/sec or
>>>>>>>>>>>>> the best... you will get 500 to 4000 objects/sec
>>>>>>>>>>>>>
>>>>>>>>>>>>> Based on your comments... you are getting the worst (with
>>>>>>>>>>>>> Derby)!!
>>>>>>>>>>>>> :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Another comments about insertion.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We found another problem when we insert data(DOn't know if
>>>>>>>>>>>>> it is
>>>>>>>>>>>>> still true). Resource.Contents is a unique list. Basically
>>>>>>>>>>>>> each time
>>>>>>>>>>>>> we insert one element, it needs to fetch all the
>>>>>>>>>>>>> collection to see
>>>>>>>>>>>>> if we already have this element. Basically.. it is time
>>>>>>>>>>>>> consuming.
>>>>>>>>>>>>> In our case we are not going through this stage to persist
>>>>>>>>>>>>> our
>>>>>>>>>>>>> objects... but the objects are not accessible through that
>>>>>>>>>>>>> list.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> If speed is an issue for you!! We (You included :-)) can
>>>>>>>>>>>>> optimize
>>>>>>>>>>>>> it. What I like is Eike did a really good job doing that
>>>>>>>>>>>>> framwork
>>>>>>>>>>>>> and optimizations are easy to add. Also you can change
>>>>>>>>>>>>> your back-end
>>>>>>>>>>>>> without affecting you client side. (Maybe going to an Object
>>>>>>>>>>>>> database!!) THis was one of the purpose of CDO!! :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le
>>>>>>>>>>>>> message de
>>>>>>>>>>>>> news: 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>>>>>>>>>>
>>>>>>>>>>>>> That sounds interesting!
>>>>>>>>>>>>> Comments inline...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cédric Brun schrieb:
>>>>>>>>>>>>> Hi Eike,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've been messing around CDO lately and I hacked a few ugly
>>>>>>>>>>>>> unit-tests measuring read/write throughput when I use CDO
>>>>>>>>>>>>> (RC2).
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm using an embeded-derby database, two different metamodels
>>>>>>>>>>>>> (complex one - 250 EClassifiers/300 features and a simple
>>>>>>>>>>>>> one - 1
>>>>>>>>>>>>> Classifier/3 Features)
>>>>>>>>>>>>>
>>>>>>>>>>>>> The test is a "quick and dirty" one so please do not take
>>>>>>>>>>>>> these
>>>>>>>>>>>>> figures too seriously, I would just like to have your
>>>>>>>>>>>>> input about
>>>>>>>>>>>>> the "order of magnitude" I should expect or settings
>>>>>>>>>>>>> affecting
>>>>>>>>>>>>> seriously these performances. I tried both JVM and TCP
>>>>>>>>>>>>> acceptor but
>>>>>>>>>>>>> the performances are quite similar though JVM is a bit
>>>>>>>>>>>>> faster.
>>>>>>>>>>>>> Looks like a sign that the NIO socket channels are
>>>>>>>>>>>>> efficient ,-)
>>>>>>>>>>>>> You used localhost for TCP transport I guess?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> * First case : building a small model (10000 elements)
>>>>>>>>>>>>> with the
>>>>>>>>>>>>> simple MM, only adding instances in an empty database :
>>>>>>>>>>>>> commiting change on every add I get a 2 objects per second
>>>>>>>>>>>>> rate.
>>>>>>>>>>>>> commiting changes every 10 adds - I get a 30 objects per
>>>>>>>>>>>>> second
>>>>>>>>>>>>> rate.
>>>>>>>>>>>>> Seems to be consistent with the fact that only for
>>>>>>>>>>>>> commits there
>>>>>>>>>>>>> is remote traffic. And the backend transaction
>>>>>>>>>>>>> management is
>>>>>>>>>>>>> another overhead. So the impact of the number
>>>>>>>>>>>>> objects/commit
>>>>>>>>>>>>> starts with bigger transactions.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> browsing the just commited elements opening a new
>>>>>>>>>>>>> transaction and
>>>>>>>>>>>>> getting a new Resource instance, I get around 30 000
>>>>>>>>>>>>> objets/sec rate
>>>>>>>>>>>>> just iterating thanks to Resource.getAllContents().
>>>>>>>>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>>>>>>>>> All views and transactions of a single session share
>>>>>>>>>>>>> (and cache)
>>>>>>>>>>>>> their object state. Otherwise I'd expect much smaller
>>>>>>>>>>>>> values.
>>>>>>>>>>>>> With his ObjectivityStore implementation Simon (cc'ed)
>>>>>>>>>>>>> reached
>>>>>>>>>>>>> rates of several thousands objects/sec. But the DBStore
>>>>>>>>>>>>> (which you
>>>>>>>>>>>>> used) is not that optimized.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> * Second case : I built a big model in the database using the
>>>>>>>>>>>>> complex metamodel. It contains 700 000 elements. The
>>>>>>>>>>>>> creation went
>>>>>>>>>>>>> pretty nicely (I did not measured it exactly as it was
>>>>>>>>>>>>> launched
>>>>>>>>>>>>> during a night, but it seems I got around 100 objects/sec
>>>>>>>>>>>>> rate in
>>>>>>>>>>>>> writting as I was writting every 700 objects.
>>>>>>>>>>>>> Sounds not too bad.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Once the model is built the derby database is 222 MB
>>>>>>>>>>>>> (which - I
>>>>>>>>>>>>> heard - is not so big for derby).
>>>>>>>>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>>>>>>>>> To be honest, I'm really no database expert (although I
>>>>>>>>>>>>> can write
>>>>>>>>>>>>> simple SQL statements).
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Retrieving the distant resource and calling eAllContents()
>>>>>>>>>>>>> on it, I
>>>>>>>>>>>>> get a 50 Objects/sec rate at best, which mean my
>>>>>>>>>>>>> eAllContents() call
>>>>>>>>>>>>> would take 4 hours !
>>>>>>>>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>>>>>>>>> (iteration order and so). There are several CDO features
>>>>>>>>>>>>> that can
>>>>>>>>>>>>> have a major impact on load rates!
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Collections (xrefs and containment) can be loaded
>>>>>>>>>>>>> lazily. I
>>>>>>>>>>>>> don't mean the target objects, which are lazy by default
>>>>>>>>>>>>> anyway,
>>>>>>>>>>>>> but even the target IDs that are in the collections. We
>>>>>>>>>>>>> call that
>>>>>>>>>>>>> "reference chunking" (but search for a better name is
>>>>>>>>>>>>> ongoing).
>>>>>>>>>>>>>
>>>>>>>>>>>>> public interface CDOSession extends
>>>>>>>>>>>>> CDOProtocolSession,
>>>>>>>>>>>>> IContainer<CDOView>
>>>>>>>>>>>>> {
>>>>>>>>>>>>> public int getReferenceChunkSize();
>>>>>>>>>>>>>
>>>>>>>>>>>>> public void setReferenceChunkSize(int
>>>>>>>>>>>>> referenceChunkSize);
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Loading collection elements (target objects) can be
>>>>>>>>>>>>> batched (the
>>>>>>>>>>>>> current name is even worse)
>>>>>>>>>>>>>
>>>>>>>>>>>>> public interface CDOView extends CDOProtocolView,
>>>>>>>>>>>>> INotifier
>>>>>>>>>>>>> {
>>>>>>>>>>>>> public int getLoadRevisionCollectionChunkSize();
>>>>>>>>>>>>>
>>>>>>>>>>>>> public void setLoadRevisionCollectionChunkSize(int
>>>>>>>>>>>>> loadRevisionCollectionChunkSize);
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> - One of the most sophisticated feature is automatic
>>>>>>>>>>>>> model usage
>>>>>>>>>>>>> analyzation. The result are fetch rules that are sent to
>>>>>>>>>>>>> the
>>>>>>>>>>>>> server. When later objects are loaded from the server,
>>>>>>>>>>>>> it can
>>>>>>>>>>>>> answer with additional objects that are likely to be
>>>>>>>>>>>>> needed by the
>>>>>>>>>>>>> client in the near future.
>>>>>>>>>>>>>
>>>>>>>>>>>>> public class CDOViewImpl extends
>>>>>>>>>>>>> org.eclipse.net4j.internal.util.event.Notifier
>>>>>>>>>>>>> implements
>>>>>>>>>>>>> CDOView, CDOIDProvider, Adapter.Internal
>>>>>>>>>>>>> {
>>>>>>>>>>>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>>>>>>>>>> {
>>>>>>>>>>>>> return featureAnalyzer;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>>>>>>>>>> featureAnalyzer)
>>>>>>>>>>>>> {
>>>>>>>>>>>>> this.featureAnalyzer = featureAnalyzer == null ?
>>>>>>>>>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> - There's a SmartReadAheadThread (if you're curious,
>>>>>>>>>>>>> please ask
>>>>>>>>>>>>> Simon). - Caches can be configured
>>>>>>>>>>>>> - ...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Known issues of the DBStore:
>>>>>>>>>>>>> - When in auditing mode updates are always written as
>>>>>>>>>>>>> complete
>>>>>>>>>>>>> inserts (by design) - When configuring toManyReferences
>>>>>>>>>>>>> to a value
>>>>>>>>>>>>> different from LIKE_ATTRIBUTES (which is not implemented
>>>>>>>>>>>>> yet) it
>>>>>>>>>>>>> is very slow. - No (few) prepared
Re: [CDO] reading/writting throughput performances (SUCCESS) [message #124902 is a reply to message #124736] Mon, 02 June 2008 12:26 Go to previous messageGo to next message
Cedric Brun is currently offline Cedric BrunFriend
Messages: 431
Registered: July 2009
Senior Member
Hi Eike and Martin,

Thanks a lot for your help, I don't think I would have managed to get
through this "jar hell" without it !

I'm now going to benchmark the hibernate store with the same test case than
the dbstore (complex metamodel - big model - eAllContents throughput) and
gives you the figures, it might be of an interest even if it's a stupid
test for "interaction based" software.

Cédric

Eike Stepper wrote:

> Martin,
>
> In the console log snippet below, please notice that it is the first
> commit transaction signal and that there are no new packages contained.
> This is caused by the client (test case) that didn't register the used
> package with the client session before using it. The following snippet
> uses the new automatic registry:
>
> | *private *CDOSession createSession()
> {
> IConnector connector =
>
(IConnector)IPluginContainer.INSTANCE.getElement("org.eclipse.net4j.connectors ",
> "jvm", "default"); System.out.println(connector);
>
> CDOSessionConfiguration configuration =
> CDOUtil.createSessionConfiguration();
> configuration.setConnector(connector);
> configuration.setRepositoryName(REPO_NAME);
> configuration.setDemandPopulatingPackageRegistry(); *return
> *configuration.openSession();
> }|
>
>
> Now the test cases pass properly!!!
> Please see the attached log file.
>
> Cheers
> /Eike
>
>
>
> Eike Stepper schrieb:
>> Hi Martin,
>>
>> Ah, now with the new mapping I get a big step further!
>> As I said, Cedric's test code contained some bugs, too. IIRC a TCP
>> acceptor was created but the client uses a JVMConnector, and stuff
>> like this.
>> You could try to overcome the exception below by adding the following
>> to the server code somewhere:
>>
>> | IPluginContainer.INSTANCE.getElement("org.eclipse.net4j.acceptors ",
>> | "jvm", "default");|
>>
>>
>>
>> Better just use the attached zip because there may be other changes
>> that I don't remember in detail anymore.
>> Now I run into another exception:
>>
>> Thread-4 [debug.signal] ================ Indicating
>> CommitTransactionIndication
>> Thread-4 [debug] Created
>> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Writer for
>> repository repo1
>> Thread-4 [debug] Created
>> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Writer for
>> repository repo1
>> Thread-4 [debug] Created
>> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Writer for
>> repository repo1
>> Thread-4 [debug.protocol] Reading 0 new packages
>> Thread-4 [debug.protocol] Reading 2 new objects
>> Thread-4 [debug.model] Reading CDOID of type 3 (TEMP_OBJECT)
>> Thread-4 [debug.model] Reading CDOID of type 3 (TEMP_OBJECT)
>> Thread-4 [debug.model] Reading CDOID of type 0 (NULL)
>> Thread-4 [debug.revision] Reading revision: ID=oid1,
>> classRef=CDOClassRef(http://www.eclipse.org/emf/CDO/resource/1.0.0,
>> 0), className=CDOResource, version=1, created=0, revised=0,
>> resource=oid1, container=NULL, feature=0
>> Thread-4 [debug.revision] Read feature CDOFeature(ID=9, name=path,
>> type=STRING, referenceType=null): /res2
>> Thread-4 [debug.revision] Read feature CDOFeature(ID=2, name=contents,
>> type=OBJECT, referenceType=CDOClass(ID=0, name=CDOObject)): size=1
>> Thread-4 [debug.model] Reading CDOID of type 3 (TEMP_OBJECT)
>> Thread-4 [debug.revision] oid2
>> [ERROR] ClassRef unresolveable:
>> CDOClassRef(http://www.eclipse.org/modeling/cdo/bug/names, 0)
>> java.lang.IllegalStateException: ClassRef unresolveable:
>> CDOClassRef(http://www.eclipse.org/modeling/cdo/bug/names, 0)
>> at
>>
org.eclipse.emf.cdo.internal.protocol.revision.CDORevisionIm pl. <init>(CDORevisionImpl.java:116)
>> at
>>
org.eclipse.emf.cdo.protocol.revision.CDORevisionUtil.read(C DORevisionUtil.java:48)
>> at
>>
org.eclipse.emf.cdo.internal.server.protocol.CommitTransacti onIndication.indicating(CommitTransactionIndication.java:95)
>> at
>>
org.eclipse.net4j.signal.IndicationWithResponse.execute(Indi cationWithResponse.java:46)
>> at org.eclipse.net4j.signal.Signal.runSync(Signal.java:143)
>> at org.eclipse.net4j.signal.Signal.run(Signal.java:124)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Threa dPoolExecutor.java:650)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo lExecutor.java:675)
>> at java.lang.Thread.run(Thread.java:595)
>>
>> Cheers
>> /Eike
>>
>>
>>
>> Martin Taal schrieb:
>>> Hi Eike, Cedric,
>>> I can't say to much about net4j.db versus net4j.db.hsqldb.
>>>
>>> You are correct that the hibernate.libraries plugin should have
>>> Eclipse-BuddyPolicy: dependent (or another way to use the classpath
>>> of the dependent plugins). Btw, I agree that registered is better as
>>> afaik dependent gets slow which larger plugin environments.
>>>
>>> The reason that cglib is used is because one of the class mappings in
>>> org.eclipse.emf.cdo.server.hibernate/mappings/meta.hbm/xml did not
>>> have the lazy="false" directive. I added this and now I get another
>>> exception (see below). So this 'solves' the problem. If you do a get
>>> latest of this file then I think this problem is 'overcome'.
>>>
>>> Overall I still do not see why it went wrong. As far as I can see the
>>> cglib library could not find the hibernate class, which is strange as
>>> they are in the same plugin??
>>> The cdo testcases don't have this problem and neither does Teneo.
>>>
>>> As it does not occur anymore we can ignore it for now (not very
>>> satisfactory but there are other things to do)... Unless someone else
>>> has an idea how to handle this ofcourse.
>>>
>>> gr. Martin
>>>
>>> java.lang.IllegalStateException: acceptor == null
>>> at
>>>
org.eclipse.net4j.internal.jvm.JVMClientConnector.doBeforeAc tivate(JVMClientConnector.java:57)
>>>
>>> at
>>>
org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:56)
>>>
>>> at
>>>
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>>>
>>> at
>>>
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>>>
>>> at
>>>
org.eclipse.net4j.internal.util.container.ManagedContainer.g etElement(ManagedContainer.java:260)
>>>
>>> at
>>>
org.eclipse.emf.internal.cdo.util.ChannelInjector.getConnect or(ChannelInjector.java:74)
>>>
>>> at
>>>
org.eclipse.emf.internal.cdo.util.ChannelInjector.process(Ch annelInjector.java:41)
>>>
>>> at
>>>
org.eclipse.net4j.internal.util.container.ManagedContainer.p ostProcessElement(ManagedContainer.java:407)
>>>
>>> at
>>>
org.eclipse.net4j.internal.util.container.ManagedContainer.g etElement(ManagedContainer.java:259)
>>>
>>> at
>>>
org.eclipse.cdo.bug.hibernate.HibernateServer.createSession( HibernateServer.java:93)
>>>
>>> at
>>>
org.eclipse.cdo.bug.unit.LaunchServerTest.testStringValue(La unchServerTest.java:43)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>
>>> at
>>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>> at junit.framework.TestCase.runTest(TestCase.java:164)
>>> at junit.framework.TestCase.runBare(TestCase.java:130)
>>> at junit.framework.TestResult$1.protect(TestResult.java:106)
>>> at junit.framework.TestResult.runProtected(TestResult.java:124)
>>> at junit.framework.TestResult.run(TestResult.java:109)
>>> at junit.framework.TestCase.run(TestCase.java:120)
>>> at junit.framework.TestSuite.runTest(TestSuite.java:230)
>>> at junit.framework.TestSuite.run(TestSuite.java:225)
>>> at
>>>
org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
>>>
>>> at
>>>
org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
>>>
>>> at
>>>
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
>>>
>>> at
>>>
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
>>>
>>> at
>>>
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
>>>
>>> at
>>>
org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
>>>
>>> at
>>>
org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>
>>> at
>>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>> at
>>>
org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hod(EclipseAppContainer.java:547)
>>>
>>> at
>>>
org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
>>>
>>> at
>>>
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:106)
>>>
>>> at
>>>
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:76)
>>>
>>> at
>>>
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:362)
>>>
>>> at
>>>
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:175)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>
>>> at
>>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>> at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 564)
>>> at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
>>> at org.eclipse.equinox.launcher.Main.run(Main.java:1251)
>>> at org.eclipse.equinox.launcher.Main.main(Main.java:1227)
>>>
>>>
>>>
>>> Eike Stepper wrote:
>>>> Hi,
>>>>
>>>> Here is a diagram of the dependencies of the relevant plugins
>>>> (please correct me if something is wrong).
>>>> The black arrows are ordinary downstream dependencies (Require-Bundle).
>>>> The red arrows are upstream dependencies (Eclipse-BuddyPolicy:
>>>> dependent).
>>>>
>>>>
>>>>
>>>> For research purposes I added code to the libraries plugin that
>>>> tries to reflectively load CDOFeatureImpl and that is not a problem!
>>>> It seems to me as if our problem is not class loading related
>>>> because the hibernate.jar should be able to see the needed classes.
>>>> I believe this is either a bug in Hibernate or our wrong usage of
>>>> Hibernate. I tend to believe the latter ;-)
>>>> I put a breakpoint in
>>>>
org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping .EntityEntityModeToTuplizerMapping(PersistentClass,
>>>> EntityMetamodel)
>>>> to see how the tuplizers are created. I'm too unexperienced with
>>>> Hibernate to get a clue about what happens there...
>>>>
>>>> Btw. I'd prefer to see the application plugin depend on net4j.db
>>>> instead of net4j.db.hsqldb.
>>>> But it seems as if the BuddyPolicy in net4j.db is not "reexported"
>>>> so that the application can see the hsqldb.jar then ;-(
>>>> I posted a respective question to equinox...
>>>>
>>>> Cheers
>>>> /Eike
>>>>
>>>>
>>>> Eike Stepper schrieb:
>>>>> Hi guys,
>>>>>
>>>>> I can confirm, making these 3rd party libraries with their own
>>>>> classloading run in OSGi is a nightmare! I took me hours teach
>>>>> where the hsqldb driver is. On the way I came over different kinds
>>>>> of classloading related errors.
>>>>>
>>>>> First I removed all the jars from the test plugin (the one Cedric
>>>>> provided). They're already in the cdo.server.hibernate.libraries
>>>>> plugin which, in fact, needs the buddy policy set to dependent. I
>>>>> think this last change needs to be committed to CVS! Btw. I'd
>>>>> prefer to use *registered* buddies instead of making all dependent
>>>>> bundles visible.
>>>>>
>>>>> The test plugin then needs to provide the database driver.
>>>>> Therefore I added a dependency on the org.eclipse.net4j.db.hsqldb
>>>>> plugin.
>>>>>
>>>>> With this setup I can start the server without CNFE, NCDFE, CCI and
>>>>> the like. At this point I get the following exception, which seems
>>>>> the same as the one Cedric observed in the end:
>>>>>
>>>>> net.sf.cglib.core.CodeGenerationException:
>>>>> java.lang.reflect.InvocationTargetException-->null
>>>>> at
>>>>>
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:237)
>>>>>
>>>>> at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
>>>>> at net.sf.cglib.proxy.Enhancer.createClass(Enhancer.java:317)
>>>>> at
>>>>>
org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:127)
>>>>>
>>>>> at
>>>>>
org.hibernate.proxy.pojo.cglib.CGLIBProxyFactory.postInstant iate(CGLIBProxyFactory.java:43)
>>>>>
>>>>> at
>>>>>
org.hibernate.tuple.entity.PojoEntityTuplizer.buildProxyFact ory(PojoEntityTuplizer.java:162)
>>>>>
>>>>> at
>>>>>
org.hibernate.tuple.entity.AbstractEntityTuplizer.<init>(AbstractEntityTuplizer.java:135)
>>>>>
>>>>> at
>>>>>
org.hibernate.tuple.entity.PojoEntityTuplizer.<init>(PojoEntityTuplizer.java:55)
>>>>>
>>>>> at
>>>>>
org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping . <init>(EntityEntityModeToTuplizerMapping.java:56)
>>>>>
>>>>> at
>>>>>
org.hibernate.tuple.entity.EntityMetamodel.<init>(EntityMetamodel.java:295)
>>>>>
>>>>> at
>>>>>
org.hibernate.persister.entity.AbstractEntityPersister.<init >(AbstractEntityPersister.java:434)
>>>>>
>>>>> at
>>>>>
org.hibernate.persister.entity.JoinedSubclassEntityPersister . <init>(JoinedSubclassEntityPersister.java:91)
>>>>>
>>>>> at
>>>>>
org.hibernate.persister.PersisterFactory.createClassPersiste r(PersisterFactory.java:58)
>>>>>
>>>>> at
>>>>>
org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:226)
>>>>>
>>>>> at
>>>>>
org.hibernate.cfg.Configuration.buildSessionFactory(Configur ation.java:1294)
>>>>>
>>>>> at
>>>>>
org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getSessionFactory(HibernatePackageHandler.java:335 )
>>>>>
>>>>> at
>>>>>
org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.readPackageInfos(HibernatePackageHandler.java:293)
>>>>>
>>>>> at
>>>>>
org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getCDOPackageInfos(HibernatePackageHandler.java:21 5)
>>>>>
>>>>> at
>>>>>
org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Reader.readPackageInfos(HibernateStoreReader.java:87)
>>>>>
>>>>> at
>>>>>
org.eclipse.emf.cdo.internal.server.PackageManager.doActivat e(PackageManager.java:72)
>>>>>
>>>>> at
>>>>>
org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
>>>>>
>>>>> at
>>>>>
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>>>>>
>>>>> at
>>>>>
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>>>>>
>>>>> at
>>>>>
org.eclipse.emf.cdo.internal.server.Repository.activateRepos itory(Repository.java:264)
>>>>>
>>>>> at
>>>>>
org.eclipse.emf.cdo.internal.server.Repository.doActivate(Re pository.java:251)
>>>>>
>>>>> at
>>>>>
org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
>>>>>
>>>>> at
>>>>>
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>>>>>
>>>>> at
>>>>>
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>>>>>
>>>>> at
>>>>>
org.eclipse.emf.cdo.server.CDOServerUtil.addRepository(CDOSe rverUtil.java:67)
>>>>>
>>>>> at
>>>>> org.eclipse.cdo.bug.LaunchServerTest.setUp(LaunchServerTest. java:54)
>>>>> at junit.framework.TestCase.runBare(TestCase.java:128)
>>>>> at junit.framework.TestResult$1.protect(TestResult.java:106)
>>>>> at junit.framework.TestResult.runProtected(TestResult.java:124)
>>>>> at junit.framework.TestResult.run(TestResult.java:109)
>>>>> at junit.framework.TestCase.run(TestCase.java:120)
>>>>> at junit.framework.TestSuite.runTest(TestSuite.java:230)
>>>>> at junit.framework.TestSuite.run(TestSuite.java:225)
>>>>> at
>>>>>
org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
>>>>>
>>>>> at
>>>>>
org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
>>>>>
>>>>> at
>>>>>
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
>>>>>
>>>>> at
>>>>>
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
>>>>>
>>>>> at
>>>>>
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
>>>>>
>>>>> at
>>>>>
org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
>>>>>
>>>>> at
>>>>>
org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
>>>>>
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>>>
>>>>> at
>>>>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>>>
>>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>>> at
>>>>>
org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hodWithException(EclipseAppContainer.java:574)
>>>>>
>>>>> at
>>>>>
org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
>>>>>
>>>>> at
>>>>>
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:110)
>>>>>
>>>>> at
>>>>>
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:79)
>>>>>
>>>>> at
>>>>>
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:379)
>>>>>
>>>>> at
>>>>>
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:179)
>>>>>
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>>>
>>>>> at
>>>>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>>>
>>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>>> at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 549)
>>>>> at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
>>>>> at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
>>>>> at org.eclipse.equinox.launcher.Main.main(Main.java:1212)
>>>>> Caused by: java.lang.reflect.InvocationTargetException
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>>>
>>>>> at
>>>>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>>>
>>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>>> at
>>>>> net.sf.cglib.core.ReflectUtils.defineClass(ReflectUtils.java :384)
>>>>> at
>>>>>
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>>>>
>>>>> ... 61 more
>>>>> Caused by: java.lang.NoClassDefFoundError:
>>>>> org/hibernate/proxy/HibernateProxy
>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>>>>> ... 67 more
>>>>>
>>>>> I must admit that I'm still not sure about te fundamental interwork
>>>>> between Hibernate and the user defined meta model. It seems as if
>>>>> asm and cglib are somewhere in between but I have no idea why at
>>>>> all. I thought we have implemented a bunch of static Hibernate
>>>>> interfaces to make the CDORevision instances known to Hibernate.
>>>>> Please somebody explain why we need runtime byte code engineering?
>>>>> Is there some configuration missing to tell Hibernate to use our
>>>>> special tuplizers?
>>>>>
>>>>> I have changed several things in the code of the test plugin and
>>>>> since it's quite small now that the jars are removed, I attach my
>>>>> version of the zip here (including a JUnit Plugin launch config).
>>>>> And don't forget to update the cdo.server.hibernate.libraries
>>>>> plugin from CVS to get the new buddy policy!
>>>>>
>>>>> Cheers
>>>>> /Eike
>>>>>
>>>>> Martin Taal schrieb:
>>>>>> Hi Cédric,
>>>>>> I looked at it for many hours trying different things (using your
>>>>>> testcase) but I am kind of stuck with this (getting the same error
>>>>>> as you). The difference I can see with the way I run the
>>>>>> cdo.hibernate testcases (as a junit test) and how your testcase is
>>>>>> run (junit plugin test). This for sure means difference in class
>>>>>> loader behavior.
>>>>>>
>>>>>> Eike, do you have an idea? I am not sure how classloading works
>>>>>> when you start a cdo server in a plugin. Which plugins are then
>>>>>> available in the cdo server? The testcases can be downloaded from
>>>>>> the link below.
>>>>>>
>>>>>> gr. Martin
>>>>>>
>>>>>> Cédric Brun wrote:
>>>>>>> Thanks a lot ! I'm looking forward to your advices !
>>>>>>>
>>>>>>> Cédric
>>>>>>>
>>>>>>> Martin Taal wrote:
>>>>>>>
>>>>>>>> I am downloading your zip-file. I don't have time to look at it
>>>>>>>> right away
>>>>>>>> but I will do my best later this evening.
>>>>>>>>
>>>>>>>> gr. Martin
>>>>>>>>
>>>>>>>> Cédric Brun wrote:
>>>>>>>>> I tried to do the same things as in the CDO/Hibernate unit tests.
>>>>>>>>>
>>>>>>>>> Here's a zipped project trying to launch an HibernateServer
>>>>>>>>> with a unit
>>>>>>>>> test adding elements.
>>>>>>>>>
( http://cedric.brun.free.fr/Tortoose/org.eclipse.cdo-teneo-te st.zip)
>>>>>>>>> The
>>>>>>>>> libs I found are included in the same project, but I still have
>>>>>>>>> no luck
>>>>>>>>> and failling at :
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>>>>>>
>>>>>>>>> ... 62 more
>>>>>>>>> Caused by: java.lang.NoClassDefFoundError:
>>>>>>>>> org/hibernate/proxy/HibernateProxy
>>>>>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>>>>>>>>> ... 68 more
>>>>>>>>>
>>>>>>>>> please note that the test project I provide really is a "quick
>>>>>>>>> and ugly"
>>>>>>>>> one ^^
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cédric
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Martin Taal wrote:
>>>>>>>>>
>>>>>>>>>> Hmm, I can understand your headache. Classpath issues are the
>>>>>>>>>> most
>>>>>>>>>> annoying errors...
>>>>>>>>>>
>>>>>>>>>> In your junit test source file, what happens when you enter the
>>>>>>>>>> org.objectweb.asm.Type, can you source file see this class?
>>>>>>>>>> Do you have cglib in some other plugin which does not have asm?
>>>>>>>>>>
>>>>>>>>>> What I also find slightly strange is that cglib is used in
>>>>>>>>>> this case
>>>>>>>>>> because normally cdo objects are not proxied using cglib. Can
>>>>>>>>>> you post
>>>>>>>>>> the generated mapping? How does your cdo initialization code
>>>>>>>>>> look like?
>>>>>>>>>>
>>>>>>>>>> gr. Martin
>>>>>>>>>>
>>>>>>>>>> Cédric Brun wrote:
>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>
>>>>>>>>>>> and thanks for your quick answer ! what I did is the
>>>>>>>>>>> following :
>>>>>>>>>>>
>>>>>>>>>>> I used the org.eclipse.emf.cdo.server.hibernate.libraries
>>>>>>>>>>> bundle and
>>>>>>>>>>> added that :
>>>>>>>>>>>
>>>>>>>>>>> Bundle-ClassPath: lib/hibernate3.jar,
>>>>>>>>>>> lib/cglib-2.1.3.jar,
>>>>>>>>>>> lib/commons-collections-2.1.1.jar,
>>>>>>>>>>> lib/dom4j.jar,
>>>>>>>>>>> lib/jta.jar
>>>>>>>>>>> lib/asm-3.0.jar
>>>>>>>>>>>
>>>>>>>>>>> and this directive :
>>>>>>>>>>>
>>>>>>>>>>> Eclipse-BuddyPolicy: dependent
>>>>>>>>>>>
>>>>>>>>>>> as stated in
>>>>>>>>>>>
http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Then I don't get the same error, now when I start my own
>>>>>>>>>>> plugin (JUnit
>>>>>>>>>>> plugin test) depending on
>>>>>>>>>>> org.eclipse.emf.cdo.server.hibernate.libraries I get :
>>>>>>>>>>>
>>>>>>>>>>> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
>>>>>>>>>>> at
>>>>>>>>>>> net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
>>>>>>>>>>> at
>>>>>>>>>>> net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
>>>>>>>>>>> at
>>>>>>>>>>> net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>
org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>>>>>>>
>>>>>>>>>>> These jars are such a headache ! Do you know - by chance -
>>>>>>>>>>> what's
>>>>>>>>>>> going on ?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Cédric
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Martin Taal wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Cédric,
>>>>>>>>>>>> Most of the time this exception is thrown because hibernate
>>>>>>>>>>>> can't find
>>>>>>>>>>>> the class. A common cause is that the hibernate jar files
>>>>>>>>>>>> are located
>>>>>>>>>>>> in a separate plugin which can not reach the classes in
>>>>>>>>>>>> other plugins.
>>>>>>>>>>>> If you set the eclipse buddy policy of the hibernate plugin to
>>>>>>>>>>>> dependent then this can help.
>>>>>>>>>>>>
>>>>>>>>>>>> gr. Martin
>>>>>>>>>>>>
>>>>>>>>>>>> Cédric Brun wrote:
>>>>>>>>>>>>> Hi Simon and Eike,
>>>>>>>>>>>>>
>>>>>>>>>>>>> thanks for your feedback, now I know what I could expect in
>>>>>>>>>>>>> using
>>>>>>>>>>>>> optimized implementation, I'll have a try on the hibernate
>>>>>>>>>>>>> store and
>>>>>>>>>>>>> see if it is scaling in a better way.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You gave me many pointers to explain the performances
>>>>>>>>>>>>> issues but in
>>>>>>>>>>>>> fact it seems that the SQL execution is taking so much time
>>>>>>>>>>>>> (not
>>>>>>>>>>>>> suprisingly in fact ;) . My first profile was wrong.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So I'm installing the hibernate store retrieving all the
>>>>>>>>>>>>> source code
>>>>>>>>>>>>> from CVS, and I'm trying to launch the unit tests. I added
>>>>>>>>>>>>> many jars
>>>>>>>>>>>>> and everything gets compiled but most of the tests are
>>>>>>>>>>>>> failling with
>>>>>>>>>>>>> "
>>>>>>>>>>>>>
>>>>>>>>>>>>> "
>>>>>>>>>>>>> org.hibernate.MappingException: Could not determine type for:
>>>>>>>>>>>>>
>>>>>>>
org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>>>>>>
>>>>>>>>>>>>> at table: cdo_features, for columns:
>>>>>>>>>>>>> [org.hibernate.mapping.Column(type)] "
>>>>>>>>>>>>>
>>>>>>>>>>>>> any hint ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cédric
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Simon McDuff wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> We noticed a performance degradation in the following case:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When we fetch a collection of objects that contains
>>>>>>>>>>>>>> collections that
>>>>>>>>>>>>>> we don't want to use. Ex:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Model:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ClassB
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>> int foo1;
>>>>>>>>>>>>>> List childrens1;
>>>>>>>>>>>>>> List childrens2;
>>>>>>>>>>>>>> List childrens3;
>>>>>>>>>>>>>> };
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Code:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Resource.contents contains objects of type ClassB.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> By fetching instances of classB, we will also fetched some
>>>>>>>>>>>>>> items in
>>>>>>>>>>>>>> the list(childrens1, childrens2, childrens3) The problem
>>>>>>>>>>>>>> we had.. is
>>>>>>>>>>>>>> we do not know if we want to use it. SO better not to
>>>>>>>>>>>>>> download them.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So In our production(we used our own IStore), we never
>>>>>>>>>>>>>> download any
>>>>>>>>>>>>>> objects from collection. I'm working at the moment to put
>>>>>>>>>>>>>> in place
>>>>>>>>>>>>>> all feature we used in the current CDO. That one will be
>>>>>>>>>>>>>> part of
>>>>>>>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477. To
>>>>>>>>>>>>>> begin
>>>>>>>>>>>>>> with,CDORevisionImpl needs to supports list that are 0
>>>>>>>>>>>>>> even if
>>>>>>>>>>>>>> referenceChunkSize == 0.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It is true that we obtain good performance.
>>>>>>>>>>>>>> Worst case :
>>>>>>>>>>>>>> 1500 objects /sec (Real fetch from disk. Page that
>>>>>>>>>>>>>> contains the
>>>>>>>>>>>>>> objects, wasn't in the cache of the filesystem.)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best case
>>>>>>>>>>>>>> 12 000 objects/Sec (Fetch objects that are already in the
>>>>>>>>>>>>>> case of
>>>>>>>>>>>>>> the filesystem)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (These statistics always fetch objects from server to
>>>>>>>>>>>>>> client. The
>>>>>>>>>>>>>> cache client isn't used)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We have another optimization one with SmartReadAhead but
>>>>>>>>>>>>>> for now
>>>>>>>>>>>>>> only work
>>>>>>>>>>>>>> for UI client. (for now) This is something we will
>>>>>>>>>>>>>> integrate as
>>>>>>>>>>>>>> well in CDO. But for now we concentrate on the other things.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I used an Object database. Based on my preliminary test
>>>>>>>>>>>>>> without CDO.
>>>>>>>>>>>>>> I compare
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hibernate vs Objectivity/EMF :
>>>>>>>>>>>>>> Objectivity/EMF is around 3 to 50 times faster than
>>>>>>>>>>>>>> Hibernate.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The worst.. you will get 30 to 240 objects/sec or
>>>>>>>>>>>>>> the best... you will get 500 to 4000 objects/sec
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Based on your comments... you are getting the worst (with
>>>>>>>>>>>>>> Derby)!!
>>>>>>>>>>>>>> :-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Another comments about insertion.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We found another problem when we insert data(DOn't know if
>>>>>>>>>>>>>> it is
>>>>>>>>>>>>>> still true). Resource.Contents is a unique list. Basically
>>>>>>>>>>>>>> each time
>>>>>>>>>>>>>> we insert one element, it needs to fetch all the
>>>>>>>>>>>>>> collection to see
>>>>>>>>>>>>>> if we already have this element. Basically.. it is time
>>>>>>>>>>>>>> consuming.
>>>>>>>>>>>>>> In our case we are not going through this stage to persist
>>>>>>>>>>>>>> our
>>>>>>>>>>>>>> objects... but the objects are not accessible through that
>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If speed is an issue for you!! We (You included :-)) can
>>>>>>>>>>>>>> optimize
>>>>>>>>>>>>>> it. What I like is Eike did a really good job doing that
>>>>>>>>>>>>>> framwork
>>>>>>>>>>>>>> and optimizations are easy to add. Also you can change
>>>>>>>>>>>>>> your back-end
>>>>>>>>>>>>>> without affecting you client side. (Maybe going to an Object
>>>>>>>>>>>>>> database!!) THis was one of the purpose of CDO!! :-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le
>>>>>>>>>>>>>> message de
>>>>>>>>>>>>>> news: 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That sounds interesting!
>>>>>>>>>>>>>> Comments inline...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cédric Brun schrieb:
>>>>>>>>>>>>>> Hi Eike,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've been messing around CDO lately and I hacked a few ugly
>>>>>>>>>>>>>> unit-tests measuring read/write throughput when I use CDO
>>>>>>>>>>>>>> (RC2).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm using an embeded-derby database, two different metamodels
>>>>>>>>>>>>>> (complex one - 250 EClassifiers/300 features and a simple
>>>>>>>>>>>>>> one - 1
>>>>>>>>>>>>>> Classifier/3 Features)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The test is a "quick and dirty" one so please do not take
>>>>>>>>>>>>>> these
>>>>>>>>>>>>>> figures too seriously, I would just like to have your
>>>>>>>>>>>>>> input about
>>>>>>>>>>>>>> the "order of magnitude" I should expect or settings
>>>>>>>>>>>>>> affecting
>>>>>>>>>>>>>> seriously these performances. I tried both JVM and TCP
>>>>>>>>>>>>>> acceptor but
>>>>>>>>>>>>>> the performances are quite similar though JVM is a bit
>>>>>>>>>>>>>> faster.
>>>>>>>>>>>>>> Looks like a sign that the NIO socket channels are
>>>>>>>>>>>>>> efficient ,-)
>>>>>>>>>>>>>> You used localhost for TCP transport I guess?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * First case : building a small model (10000 elements)
>>>>>>>>>>>>>> with the
>>>>>>>>>>>>>> simple MM, only adding instances in an empty database :
>>>>>>>>>>>>>> commiting change on every add I get a 2 objects per second
>>>>>>>>>>>>>> rate.
>>>>>>>>>>>>>> commiting changes every 10 adds - I get a 30 objects per
>>>>>>>>>>>>>> second
>>>>>>>>>>>>>> rate.
>>>>>>>>>>>>>> Seems to be consistent with the fact that only for
>>>>>>>>>>>>>> commits there
>>>>>>>>>>>>>> is remote traffic. And the backend transaction
>>>>>>>>>>>>>> management is
>>>>>>>>>>>>>> another overhead. So the impact of the number
>>>>>>>>>>>>>> objects/commit
>>>>>>>>>>>>>> starts with bigger transactions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> browsing the just commited elements opening a new
>>>>>>>>>>>>>> transaction and
>>>>>>>>>>>>>> getting a new Resource instance, I get around 30 000
>>>>>>>>>>>>>> objets/sec rate
>>>>>>>>>>>>>> just iterating thanks to Resource.getAllContents().
>>>>>>>>>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>>>>>>>>>> All views and transactions of a single session share
>>>>>>>>>>>>>> (and cache)
>>>>>>>>>>>>>> their object state. Otherwise I'd expect much smaller
>>>>>>>>>>>>>> values.
>>>>>>>>>>>>>> With his ObjectivityStore implementation Simon (cc'ed)
>>>>>>>>>>>>>> reached
>>>>>>>>>>>>>> rates of several thousands objects/sec. But the DBStore
>>>>>>>>>>>>>> (which you
>>>>>>>>>>>>>> used) is not that optimized.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * Second case : I built a big model in the database using the
>>>>>>>>>>>>>> complex metamodel. It contains 700 000 elements. The
>>>>>>>>>>>>>> creation went
>>>>>>>>>>>>>> pretty nicely (I did not measured it exactly as it was
>>>>>>>>>>>>>> launched
>>>>>>>>>>>>>> during a night, but it seems I got around 100 objects/sec
>>>>>>>>>>>>>> rate in
>>>>>>>>>>>>>> writting as I was writting every 700 objects.
>>>>>>>>>>>>>> Sounds not too bad.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Once the model is built the derby database is 222 MB
>>>>>>>>>>>>>> (which - I
>>>>>>>>>>>>>> heard - is not so big for derby).
>>>>>>>>>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>>>>>>>>>> To be honest, I'm really no database expert (although I
>>>>>>>>>>>>>> can write
>>>>>>>>>>>>>> simple SQL statements).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Retrieving the distant resource and calling eAllContents()
>>>>>>>>>>>>>> on it, I
>>>>>>>>>>>>>> get a 50 Objects/sec rate at best, which mean my
>>>>>>>>>>>>>> eAllContents() call
>>>>>>>>>>>>>> would take 4 hours !
>>>>>>>>>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>>>>>>>>>> (iteration order and so). There are several CDO features
>>>>>>>>>>>>>> that can
>>>>>>>>>>>>>> have a major impact on load rates!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Collections (xrefs and containment) can be loaded
>>>>>>>>>>>>>> lazily. I
>>>>>>>>>>>>>> don't mean the target objects, which are lazy by default
>>>>>>>>>>>>>> anyway,
>>>>>>>>>>>>>> but even the target IDs that are in the collections. We
>>>>>>>>>>>>>> call that
>>>>>>>>>>>>>> "reference chunking" (but search for a better name is
>>>>>>>>>>>>>> ongoing).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> public interface CDOSession extends
>>>>>>>>>>>>>> CDOProtocolSession,
>>>>>>>>>>>>>> IContainer<CDOView>
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>> public int getReferenceChunkSize();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> public void setReferenceChunkSize(int
>>>>>>>>>>>>>> referenceChunkSize);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Loading collection elements (target objects) can be
>>>>>>>>>>>>>> batched (the
>>>>>>>>>>>>>> current name is even worse)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> public interface CDOView extends CDOProtocolView,
>>>>>>>>>>>>>> INotifier
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>> public int getLoadRevisionCollectionChunkSize();
>>>>>>>


http://cedric.brun.io news and articles on eclipse and eclipse modeling.
Re: [CDO] reading/writting throughput performances (hibernate mapper) [message #125026 is a reply to message #123598] Tue, 03 June 2008 08:45 Go to previous messageGo to next message
Cedric Brun is currently offline Cedric BrunFriend
Messages: 431
Registered: July 2009
Senior Member
Hi,

here comes a quick report on my measures.

using cdo with the teneo backend (thanks again for your help setting this
one up !) mysql as a database (localhost one) and jvm CDO acceptor.

writing the objects on the database : the process starts at 150 objects/sec
but quickly drop to around 30 objects/sec and fail before the end when 40
000 instances are inserted in the database with:
[ERROR] FATAL: Memory infrastructure corrupted after successful commit
operation of the store
and a TimeOutException on the client.

I can only see "load" issues related to this failure as I'm just copying the
same data over and over on the database so it can't come from the data.

I noticed that my metamodel composed of 250 EClasses is mapped to a big
unique table (for my abstract top level element).

Reading elements : as I did not managed to fill the database like I did not
managed to get the figures.

I also often ends up with:

[ERROR] CommitContext not set
java.lang.IllegalStateException: CommitContext not set
at
org.eclipse.emf.cdo.server.internal.hibernate.HibernateThrea dContext.getCommitContext(HibernateThreadContext.java:69)
at
org.eclipse.emf.cdo.server.internal.hibernate.HibernateUtil. getCDORevision(HibernateUtil.java:116)
at
org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Reader.readRevision(HibernateStoreReader.java:147)
at
org.eclipse.emf.cdo.internal.server.RevisionManager.loadRevi sion(RevisionManager.java:210)
at
org.eclipse.emf.cdo.internal.protocol.revision.CDORevisionRe solverImpl.getRevision(CDORevisionResolverImpl.java:282)
at
org.eclipse.emf.cdo.internal.protocol.revision.CDORevisionRe solverImpl.getRevision(CDORevisionResolverImpl.java:144)
at
org.eclipse.emf.cdo.internal.server.protocol.LoadRevisionInd ication.getRevision(LoadRevisionIndication.java:171)
at
org.eclipse.emf.cdo.internal.server.protocol.LoadRevisionInd ication.responding(LoadRevisionIndication.java:147)
at
org.eclipse.net4j.signal.IndicationWithResponse.execute(Indi cationWithResponse.java:54)
at
org.eclipse.emf.cdo.internal.server.protocol.CDOReadIndicati on.execute(CDOReadIndication.java:38)
at org.eclipse.net4j.signal.Signal.runSync(Signal.java:143)

But I'm not able to identify why and under which circomstances.


I did not managed to use derby-embedded with teneo/hibernate as I only get
an exception just after the initialization process :

org.eclipse.net4j.util.transaction.TransactionException: could not insert:
[org.eclipse.emf.cdo.internal.protocol.model.CDOPackageImpl]
at
org.eclipse.emf.internal.cdo.CDOTransactionImpl.commit(CDOTr ansactionImpl.java:241)
at


It's a shame as I guess I would not have the overhead of the network and
concurrency stack. Derby-embedded fits better with my use cases.
I used the following app.properties:

hibernate.dialect=org.hibernate.dialect.DerbyDialect
hibernate.connection.driver_class=org.apache.derby.jdbc.Embe ddedDriver
hibernate.connection.url=jdbc:derby:/tmp/cdohibernatederby;c reate=true
hibernate.connection.autocommit=true
hibernate.cache.provider_class=org.hibernate.cache.Hashtable CacheProvider



Any hint about this error ?


Cédric


Eike Stepper wrote:

> Hi Cédric,
>
> That sounds interesting!
> Comments inline...
>
>
> Cédric Brun schrieb:
>> Hi Eike,
>>
>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>> measuring read/write throughput when I use CDO (RC2).
>>
>> I'm using an embeded-derby database, two different metamodels (complex
>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>> Features)
>>
>> The test is a "quick and dirty" one so please do not take these figures
>> too seriously, I would just like to have your input about the "order of
>> magnitude" I should expect or settings affecting seriously these
>> performances. I tried both JVM and TCP acceptor but the performances are
>> quite similar though JVM is a bit faster.
>>
> Looks like a sign that the NIO socket channels are efficient ,-)
> You used localhost for TCP transport I guess?
>
>>
>> * First case : building a small model (10000 elements) with the simple
>> MM, only adding instances in an empty database :
>> commiting change on every add I get a 2 objects per second rate.
>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>
> Seems to be consistent with the fact that only for commits there is
> remote traffic.
> And the backend transaction management is another overhead.
> So the impact of the number objects/commit starts with bigger
> transactions.
>
>> browsing the just commited elements opening a new transaction and getting
>> a new Resource instance, I get around 30 000 objets/sec rate just
>> iterating thanks to Resource.getAllContents().
>>
> Hehe, this is a sign that client side caching works well:
> All views and transactions of a single session share (and cache) their
> object state.
> Otherwise I'd expect much smaller values.
> With his ObjectivityStore implementation Simon (cc'ed) reached rates of
> several thousands objects/sec.
> But the DBStore (which you used) is not that optimized.
>
>> * Second case : I built a big model in the database using the complex
>> metamodel. It contains 700 000 elements. The creation went pretty nicely
>> (I did not measured it exactly as it was launched during a night, but it
>> seems I got around 100 objects/sec rate in writting as I was writting
>> every 700 objects.
>>
> Sounds not too bad.
>
>>
>> Once the model is built the derby database is 222 MB (which - I heard -
>> is not so big for derby).
>>
> Hmm, I have no extensive experience with Derby.
> To be honest, I'm really no database expert (although I can write simple
> SQL statements).
>
>> Retrieving the distant resource and calling eAllContents() on it, I get a
>> 50 Objects/sec rate at best, which mean my eAllContents() call would take
>> 4 hours !
>>
> Currently I'm not sure what exactly eAllContents() does do (iteration
> order and so).
> There are several CDO features that can have a major impact on load rates!
>
> - Collections (xrefs and containment) can be loaded lazily. I don't mean
> the target objects, which are lazy by default anyway, but even the
> target IDs that are in the collections. We call that "reference
> chunking" (but search for a better name is ongoing).
> |*public interface *CDOSession *extends *CDOProtocolSession,
> |IContainer<CDOView>
> {
> *public **int *getReferenceChunkSize();
>
> *public **void *setReferenceChunkSize(*int *referenceChunkSize);|
>
>
> - Loading collection elements (target objects) can be batched (the
> current name is even worse)
> |*public interface *CDOView *extends *CDOProtocolView, INotifier
> {
> *public **int *getLoadRevisionCollectionChunkSize();
>
> *public **void *setLoadRevisionCollectionChunkSize(*int
> *loadRevisionCollectionChunkSize);|
>
>
>
> - One of the most sophisticated feature is automatic model usage
> analyzation. The result are fetch rules that are sent to the server.
> When later objects are loaded from the server, it can answer with
> additional objects that are likely to be needed by the client in the
> near future.
> |*public class *CDOViewImpl *extends
> |*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
> |CDOIDProvider,
> Adapter.Internal
> {
> *public *CDOFeatureAnalyzer getFeatureAnalyzer()
> {
> *return *featureAnalyzer;
> }
>
> *public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
> {
> *this*.featureAnalyzer = featureAnalyzer == *null *?
> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
> }|
>
>
> - There's a SmartReadAheadThread (if you're curious, please ask Simon).
> - Caches can be configured
> - ...
>
>
> Known issues of the DBStore:
> - When in auditing mode updates are always written as complete inserts
> (by design)
> - When configuring toManyReferences to a value different from
> LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
> - No (few) prepared statements are used (see
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
> - Maybe others...
>
> Please note that the DBStore was not written by me to compete with
> Hibernate in terms of performance or mapping capabilities.
> Now that Martin and I, well in the end mostly Martin, provide a
> HibernateStore and this HibernateStore gets more and more features I
> expect that there will be a trend away from the DBStore.
>
>> This is probably linked to the way the mapping to db is done, here's the
>> config I used (came from CDO-server example if I recall correctly).
>>
>>
>> <repository name="repo1">
>> <property name="overrideUUID"
>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>> <property name="supportingAudits" value="true"/>
>> <property name="verifyingRevisions" value="false"/>
>> <property name="currentLRUCapacity" value="10000"/>
>>
> I guess this one could be larger.
>
>> <property name="revisedLRUCapacity" value="100"/>
>>
>> <store type="db">
>> <!-- type: horizontal | vertical | <any
>> user-contributed type> --> <mappingStrategy
>> type="horizontal">
>> <!-- ONE_TABLE_PER_REFERENCE |
>> ONE_TABLE_PER_CLASS |
>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>> <property name="toManyReferences"
>> value="ONE_TABLE_PER_REFERENCE"/>
>>
>> <!-- LIKE_ATTRIBUTES |
>> LIKE_TO_MANY_REFERENCES --> <property
>> name="toOneReferences"
>> value="LIKE_ATTRIBUTES"/>
>>
>> <!-- MODEL | STRATEGY -->
>> <property name="mappingPrecedence"
>> value="MODEL"/>
>>
> This one is obsolete.
>
>> </mappingStrategy>
>>
>> <dbAdapter name="derby-embedded"/>
>> <dataSource
>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>> databaseName="/tmp/cdodb1"
>> createDatabase="create"/>
>>
>> </store>
>> </repository>
>>
>>
>>
>>
>> What do you think about that ? What throughput do you think I should
>> expect ?
> As I said that's very difficult to judge.
> I believe it's more appropriate to compare the actual values with values
> from other measures (other setups, properties, etc. or even other
> products) instead of random expectations.
> Apologies that I didn't have the time so far to provide some reference
> measures myself. Maybe this gets better now that the team starts to grow
> slowly ;-)
>
>> What are the main reasons fur such figures ?
>>
> Hmm, I *suspect* you don't like the figures? I can find no evidence your
> assessment ;-)
>
> Ahh,one more note:
> Tracing can also have a significant impact on performance and end-to-end
> throughput.
> Turn it off by all means. And please make sure that it's really turned
> off, not only invisible (i.e. console is not a target).
>
> Cheers
> /Eike


http://cedric.brun.io news and articles on eclipse and eclipse modeling.
Re: [CDO] reading/writting throughput performances (hibernate mapper) [message #125035 is a reply to message #125026] Tue, 03 June 2008 09:43 Go to previous message
Martin Taal is currently offline Martin TaalFriend
Messages: 5468
Registered: July 2009
Senior Member
Hi Cédric,
I can't answer all questions but see my comments inline.

gr. Martin

Cédric Brun wrote:
> Hi,
>
> here comes a quick report on my measures.
>
> using cdo with the teneo backend (thanks again for your help setting this
> one up !) mysql as a database (localhost one) and jvm CDO acceptor.
>
> writing the objects on the database : the process starts at 150 objects/sec
> but quickly drop to around 30 objects/sec and fail before the end when 40
> 000 instances are inserted in the database with:
> [ERROR] FATAL: Memory infrastructure corrupted after successful commit
> operation of the store
> and a TimeOutException on the client.
MT>> Can you post the complete stacktrace for this?

>
> I can only see "load" issues related to this failure as I'm just copying the
> same data over and over on the database so it can't come from the data.
>
> I noticed that my metamodel composed of 250 EClasses is mapped to a big
> unique table (for my abstract top level element).
MT>> This is single table inheritance mapping strategy. You can also choose join-table inheritance
strategy then the database model will become more descriptive. See here:
http://www.elver.org/hibernate/inheritance.html#Supported+in heritance+mapping+strategies
To do join-table, you can do pass the property:
org.eclipse.emf.teneo.mapper.PersistenceMappingOptions.INHER ITANCEMAPPING
with the string value JOIN
You can set this property in the same properties object as the database connection info.

>
> Reading elements : as I did not managed to fill the database like I did not
> managed to get the figures.
>
> I also often ends up with:
>
> [ERROR] CommitContext not set
> java.lang.IllegalStateException: CommitContext not set
> at
> org.eclipse.emf.cdo.server.internal.hibernate.HibernateThrea dContext.getCommitContext(HibernateThreadContext.java:69)
> at
> org.eclipse.emf.cdo.server.internal.hibernate.HibernateUtil. getCDORevision(HibernateUtil.java:116)
> at
> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Reader.readRevision(HibernateStoreReader.java:147)
> at
> org.eclipse.emf.cdo.internal.server.RevisionManager.loadRevi sion(RevisionManager.java:210)
> at
> org.eclipse.emf.cdo.internal.protocol.revision.CDORevisionRe solverImpl.getRevision(CDORevisionResolverImpl.java:282)
> at
> org.eclipse.emf.cdo.internal.protocol.revision.CDORevisionRe solverImpl.getRevision(CDORevisionResolverImpl.java:144)
> at
> org.eclipse.emf.cdo.internal.server.protocol.LoadRevisionInd ication.getRevision(LoadRevisionIndication.java:171)
> at
> org.eclipse.emf.cdo.internal.server.protocol.LoadRevisionInd ication.responding(LoadRevisionIndication.java:147)
> at
> org.eclipse.net4j.signal.IndicationWithResponse.execute(Indi cationWithResponse.java:54)
> at
> org.eclipse.emf.cdo.internal.server.protocol.CDOReadIndicati on.execute(CDOReadIndication.java:38)
> at org.eclipse.net4j.signal.Signal.runSync(Signal.java:143)
>
> But I'm not able to identify why and under which circomstances.
>
>
> I did not managed to use derby-embedded with teneo/hibernate as I only get
> an exception just after the initialization process :
>
> org.eclipse.net4j.util.transaction.TransactionException: could not insert:
> [org.eclipse.emf.cdo.internal.protocol.model.CDOPackageImpl]
> at
> org.eclipse.emf.internal.cdo.CDOTransactionImpl.commit(CDOTr ansactionImpl.java:241)
> at
MT>> Can you post the complete stacktrace? I am not sure if this is the stacktrace visible on the
server or if this is the stacktrace on the client? The one on the server gives the most information.

>
>
> It's a shame as I guess I would not have the overhead of the network and
> concurrency stack. Derby-embedded fits better with my use cases.
> I used the following app.properties:
>
> hibernate.dialect=org.hibernate.dialect.DerbyDialect
> hibernate.connection.driver_class=org.apache.derby.jdbc.Embe ddedDriver
> hibernate.connection.url=jdbc:derby:/tmp/cdohibernatederby;c reate=true
> hibernate.connection.autocommit=true
> hibernate.cache.provider_class=org.hibernate.cache.Hashtable CacheProvider
>
>
>
> Any hint about this error ?
>
>
> Cédric
>
>
> Eike Stepper wrote:
>
>> Hi Cédric,
>>
>> That sounds interesting!
>> Comments inline...
>>
>>
>> Cédric Brun schrieb:
>>> Hi Eike,
>>>
>>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>>> measuring read/write throughput when I use CDO (RC2).
>>>
>>> I'm using an embeded-derby database, two different metamodels (complex
>>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>>> Features)
>>>
>>> The test is a "quick and dirty" one so please do not take these figures
>>> too seriously, I would just like to have your input about the "order of
>>> magnitude" I should expect or settings affecting seriously these
>>> performances. I tried both JVM and TCP acceptor but the performances are
>>> quite similar though JVM is a bit faster.
>>>
>> Looks like a sign that the NIO socket channels are efficient ,-)
>> You used localhost for TCP transport I guess?
>>
>>> * First case : building a small model (10000 elements) with the simple
>>> MM, only adding instances in an empty database :
>>> commiting change on every add I get a 2 objects per second rate.
>>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>>
>> Seems to be consistent with the fact that only for commits there is
>> remote traffic.
>> And the backend transaction management is another overhead.
>> So the impact of the number objects/commit starts with bigger
>> transactions.
>>
>>> browsing the just commited elements opening a new transaction and getting
>>> a new Resource instance, I get around 30 000 objets/sec rate just
>>> iterating thanks to Resource.getAllContents().
>>>
>> Hehe, this is a sign that client side caching works well:
>> All views and transactions of a single session share (and cache) their
>> object state.
>> Otherwise I'd expect much smaller values.
>> With his ObjectivityStore implementation Simon (cc'ed) reached rates of
>> several thousands objects/sec.
>> But the DBStore (which you used) is not that optimized.
>>
>>> * Second case : I built a big model in the database using the complex
>>> metamodel. It contains 700 000 elements. The creation went pretty nicely
>>> (I did not measured it exactly as it was launched during a night, but it
>>> seems I got around 100 objects/sec rate in writting as I was writting
>>> every 700 objects.
>>>
>> Sounds not too bad.
>>
>>> Once the model is built the derby database is 222 MB (which - I heard -
>>> is not so big for derby).
>>>
>> Hmm, I have no extensive experience with Derby.
>> To be honest, I'm really no database expert (although I can write simple
>> SQL statements).
>>
>>> Retrieving the distant resource and calling eAllContents() on it, I get a
>>> 50 Objects/sec rate at best, which mean my eAllContents() call would take
>>> 4 hours !
>>>
>> Currently I'm not sure what exactly eAllContents() does do (iteration
>> order and so).
>> There are several CDO features that can have a major impact on load rates!
>>
>> - Collections (xrefs and containment) can be loaded lazily. I don't mean
>> the target objects, which are lazy by default anyway, but even the
>> target IDs that are in the collections. We call that "reference
>> chunking" (but search for a better name is ongoing).
>> |*public interface *CDOSession *extends *CDOProtocolSession,
>> |IContainer<CDOView>
>> {
>> *public **int *getReferenceChunkSize();
>>
>> *public **void *setReferenceChunkSize(*int *referenceChunkSize);|
>>
>>
>> - Loading collection elements (target objects) can be batched (the
>> current name is even worse)
>> |*public interface *CDOView *extends *CDOProtocolView, INotifier
>> {
>> *public **int *getLoadRevisionCollectionChunkSize();
>>
>> *public **void *setLoadRevisionCollectionChunkSize(*int
>> *loadRevisionCollectionChunkSize);|
>>
>>
>>
>> - One of the most sophisticated feature is automatic model usage
>> analyzation. The result are fetch rules that are sent to the server.
>> When later objects are loaded from the server, it can answer with
>> additional objects that are likely to be needed by the client in the
>> near future.
>> |*public class *CDOViewImpl *extends
>> |*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
>> |CDOIDProvider,
>> Adapter.Internal
>> {
>> *public *CDOFeatureAnalyzer getFeatureAnalyzer()
>> {
>> *return *featureAnalyzer;
>> }
>>
>> *public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
>> {
>> *this*.featureAnalyzer = featureAnalyzer == *null *?
>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>> }|
>>
>>
>> - There's a SmartReadAheadThread (if you're curious, please ask Simon).
>> - Caches can be configured
>> - ...
>>
>>
>> Known issues of the DBStore:
>> - When in auditing mode updates are always written as complete inserts
>> (by design)
>> - When configuring toManyReferences to a value different from
>> LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
>> - No (few) prepared statements are used (see
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
>> - Maybe others...
>>
>> Please note that the DBStore was not written by me to compete with
>> Hibernate in terms of performance or mapping capabilities.
>> Now that Martin and I, well in the end mostly Martin, provide a
>> HibernateStore and this HibernateStore gets more and more features I
>> expect that there will be a trend away from the DBStore.
>>
>>> This is probably linked to the way the mapping to db is done, here's the
>>> config I used (came from CDO-server example if I recall correctly).
>>>
>>>
>>> <repository name="repo1">
>>> <property name="overrideUUID"
>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>> <property name="supportingAudits" value="true"/>
>>> <property name="verifyingRevisions" value="false"/>
>>> <property name="currentLRUCapacity" value="10000"/>
>>>
>> I guess this one could be larger.
>>
>>> <property name="revisedLRUCapacity" value="100"/>
>>>
>>> <store type="db">
>>> <!-- type: horizontal | vertical | <any
>>> user-contributed type> --> <mappingStrategy
>>> type="horizontal">
>>> <!-- ONE_TABLE_PER_REFERENCE |
>>> ONE_TABLE_PER_CLASS |
>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>>> <property name="toManyReferences"
>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>
>>> <!-- LIKE_ATTRIBUTES |
>>> LIKE_TO_MANY_REFERENCES --> <property
>>> name="toOneReferences"
>>> value="LIKE_ATTRIBUTES"/>
>>>
>>> <!-- MODEL | STRATEGY -->
>>> <property name="mappingPrecedence"
>>> value="MODEL"/>
>>>
>> This one is obsolete.
>>
>>> </mappingStrategy>
>>>
>>> <dbAdapter name="derby-embedded"/>
>>> <dataSource
>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>> databaseName="/tmp/cdodb1"
>>> createDatabase="create"/>
>>>
>>> </store>
>>> </repository>
>>>
>>>
>>>
>>>
>>> What do you think about that ? What throughput do you think I should
>>> expect ?
>> As I said that's very difficult to judge.
>> I believe it's more appropriate to compare the actual values with values
>> from other measures (other setups, properties, etc. or even other
>> products) instead of random expectations.
>> Apologies that I didn't have the time so far to provide some reference
>> measures myself. Maybe this gets better now that the team starts to grow
>> slowly ;-)
>>
>>> What are the main reasons fur such figures ?
>>>
>> Hmm, I *suspect* you don't like the figures? I can find no evidence your
>> assessment ;-)
>>
>> Ahh,one more note:
>> Tracing can also have a significant impact on performance and end-to-end
>> throughput.
>> Turn it off by all means. And please make sure that it's really turned
>> off, not only invisible (i.e. console is not a target).
>>
>> Cheers
>> /Eike
>


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Cell: +31 (0)6 288 48 943
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: [CDO] reading/writting throughput performances [message #619063 is a reply to message #123583] Thu, 29 May 2008 11:27 Go to previous message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
This is a multi-part message in MIME format.
--------------090102080203020800020703
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi Cédric,

That sounds interesting!
Comments inline...


Cédric Brun schrieb:
> Hi Eike,
>
> I've been messing around CDO lately and I hacked a few ugly unit-tests
> measuring read/write throughput when I use CDO (RC2).
>
> I'm using an embeded-derby database, two different metamodels (complex one -
> 250 EClassifiers/300 features and a simple one - 1 Classifier/3 Features)
>
> The test is a "quick and dirty" one so please do not take these figures too
> seriously, I would just like to have your input about the "order of
> magnitude" I should expect or settings affecting seriously these
> performances. I tried both JVM and TCP acceptor but the performances are
> quite similar though JVM is a bit faster.
>
Looks like a sign that the NIO socket channels are efficient ,-)
You used localhost for TCP transport I guess?

>
> * First case : building a small model (10000 elements) with the simple MM,
> only adding instances in an empty database :
> commiting change on every add I get a 2 objects per second rate.
> commiting changes every 10 adds - I get a 30 objects per second rate.
>
Seems to be consistent with the fact that only for commits there is
remote traffic.
And the backend transaction management is another overhead.
So the impact of the number objects/commit starts with bigger transactions.

> browsing the just commited elements opening a new transaction and getting a
> new Resource instance, I get around 30 000 objets/sec rate just iterating
> thanks to Resource.getAllContents().
>
Hehe, this is a sign that client side caching works well:
All views and transactions of a single session share (and cache) their
object state.
Otherwise I'd expect much smaller values.
With his ObjectivityStore implementation Simon (cc'ed) reached rates of
several thousands objects/sec.
But the DBStore (which you used) is not that optimized.

> * Second case : I built a big model in the database using the complex
> metamodel. It contains 700 000 elements. The creation went pretty nicely (I
> did not measured it exactly as it was launched during a night, but it seems
> I got around 100 objects/sec rate in writting as I was writting every 700
> objects.
>
Sounds not too bad.

>
> Once the model is built the derby database is 222 MB (which - I heard - is
> not so big for derby).
>
Hmm, I have no extensive experience with Derby.
To be honest, I'm really no database expert (although I can write simple
SQL statements).

> Retrieving the distant resource and calling eAllContents() on it, I get a 50
> Objects/sec rate at best, which mean my eAllContents() call would take 4
> hours !
>
Currently I'm not sure what exactly eAllContents() does do (iteration
order and so).
There are several CDO features that can have a major impact on load rates!

- Collections (xrefs and containment) can be loaded lazily. I don't mean
the target objects, which are lazy by default anyway, but even the
target IDs that are in the collections. We call that "reference
chunking" (but search for a better name is ongoing).
|*public interface *CDOSession *extends *CDOProtocolSession, IContainer<CDOView>
{
*public **int *getReferenceChunkSize();

*public **void *setReferenceChunkSize(*int *referenceChunkSize);|


- Loading collection elements (target objects) can be batched (the
current name is even worse)
|*public interface *CDOView *extends *CDOProtocolView, INotifier
{
*public **int *getLoadRevisionCollectionChunkSize();

*public **void *setLoadRevisionCollectionChunkSize(*int *loadRevisionCollectionChunkSize);|



- One of the most sophisticated feature is automatic model usage
analyzation. The result are fetch rules that are sent to the server.
When later objects are loaded from the server, it can answer with
additional objects that are likely to be needed by the client in the
near future.
|*public class *CDOViewImpl *extends *org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView, CDOIDProvider,
Adapter.Internal
{
*public *CDOFeatureAnalyzer getFeatureAnalyzer()
{
*return *featureAnalyzer;
}

*public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
{
*this*.featureAnalyzer = featureAnalyzer == *null *? CDOFeatureAnalyzer.NOOP : featureAnalyzer;
}|


- There's a SmartReadAheadThread (if you're curious, please ask Simon).
- Caches can be configured
- ...


Known issues of the DBStore:
- When in auditing mode updates are always written as complete inserts
(by design)
- When configuring toManyReferences to a value different from
LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
- No (few) prepared statements are used (see
https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
- Maybe others...

Please note that the DBStore was not written by me to compete with
Hibernate in terms of performance or mapping capabilities.
Now that Martin and I, well in the end mostly Martin, provide a
HibernateStore and this HibernateStore gets more and more features I
expect that there will be a trend away from the DBStore.

> This is probably linked to the way the mapping to db is done, here's the
> config I used (came from CDO-server example if I recall correctly).
>
>
> <repository name="repo1">
> <property name="overrideUUID"
> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
> <property name="supportingAudits" value="true"/>
> <property name="verifyingRevisions" value="false"/>
> <property name="currentLRUCapacity" value="10000"/>
>
I guess this one could be larger.

> <property name="revisedLRUCapacity" value="100"/>
>
> <store type="db">
> <!-- type: horizontal | vertical | <any user-contributed type> -->
> <mappingStrategy type="horizontal">
> <!-- ONE_TABLE_PER_REFERENCE | ONE_TABLE_PER_CLASS |
> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
> <property name="toManyReferences" value="ONE_TABLE_PER_REFERENCE"/>
>
> <!-- LIKE_ATTRIBUTES | LIKE_TO_MANY_REFERENCES -->
> <property name="toOneReferences" value="LIKE_ATTRIBUTES"/>
>
> <!-- MODEL | STRATEGY -->
> <property name="mappingPrecedence" value="MODEL"/>
>
This one is obsolete.

> </mappingStrategy>
>
> <dbAdapter name="derby-embedded"/>
> <dataSource class="org.apache.derby.jdbc.EmbeddedDataSource"
> databaseName="/tmp/cdodb1"
> createDatabase="create"/>
>
> </store>
> </repository>
>
>
>
>
> What do you think about that ? What throughput do you think I should
> expect ?
As I said that's very difficult to judge.
I believe it's more appropriate to compare the actual values with values
from other measures (other setups, properties, etc. or even other
products) instead of random expectations.
Apologies that I didn't have the time so far to provide some reference
measures myself. Maybe this gets better now that the team starts to grow
slowly ;-)

> What are the main reasons fur such figures ?
>
Hmm, I *suspect* you don't like the figures? I can find no evidence your
assessment ;-)

Ahh,one more note:
Tracing can also have a significant impact on performance and end-to-end
throughput.
Turn it off by all means. And please make sure that it's really turned
off, not only invisible (i.e. console is not a target).

Cheers
/Eike



--------------090102080203020800020703
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Cédric,<br>
<br>
That sounds interesting!<br>
Comments inline...<br>
<br>
<br>
Cédric Brun schrieb:
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex one -
250 EClassifiers/300 features and a simple one - 1 Classifier/3 Features)

The test is a "quick and dirty" one so please do not take these figures too
seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.
</pre>
</blockquote>
Looks like a sign that the NIO socket channels are efficient ,-)<br>
You used localhost for TCP transport I guess?<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">

* First case : building a small model (10000 elements) with the simple MM,
only adding instances in an empty database :
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.
</pre>
</blockquote>
Seems to be consistent with the fact that only for commits there is
remote traffic.<br>
And the backend transaction management is another overhead.<br>
So the impact of the number objects/commit starts with bigger
transactions.<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">
browsing the just commited elements opening a new transaction and getting a
new Resource instance, I get around 30 000 objets/sec rate just iterating
thanks to Resource.getAllContents().
</pre>
</blockquote>
Hehe, this is a sign that client side caching works well:<br>
All views and transactions of a single session share (and cache) their
object state.<br>
Otherwise I'd expect much smaller values.<br>
With his ObjectivityStore implementation Simon (cc'ed) reached rates of
several thousands objects/sec.<br>
But the DBStore (which you used) is not that optimized.<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">
* Second case : I built a big model in the database using the complex
metamodel. It contains 700 000 elements. The creation went pretty nicely (I
did not measured it exactly as it was launched during a night, but it seems
I got around 100 objects/sec rate in writting as I was writting every 700
objects.
</pre>
</blockquote>
Sounds not too bad.<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">

Once the model is built the derby database is 222 MB (which - I heard - is
not so big for derby).
</pre>
</blockquote>
Hmm, I have no extensive experience with Derby.<br>
To be honest, I'm really no database expert (although I can write
simple SQL statements).<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">
Retrieving the distant resource and calling eAllContents() on it, I get a 50
Objects/sec rate at best, which mean my eAllContents() call would take 4
hours !
</pre>
</blockquote>
Currently I'm not sure what exactly eAllContents() does do (iteration
order and so).<br>
There are several CDO features that can have a major impact on load
rates!<br>
<br>
- Collections (xrefs and containment) can be loaded lazily. I don't
mean the target objects, which are lazy by default anyway, but even the
target IDs that are in the collections. We call that "reference
chunking" (but search for a better name is ongoing).<br>
<title></title>
<style type="text/css">
<!--code { font-family: Courier New, Courier; font-size: 10pt; margin: 0px; }-->
</style>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<!-- ======================================================== --><!-- = Java Sourcecode to HTML automatically converted code = --><!-- = Java2Html Converter 5.0 [2006-02-26] by Markus Gebhard markus@jave.de = -->
<!-- = Further information: http://www.java2html.de = -->
<div class="java" align="left">
<table bgcolor="#ffffff" border="0" cellpadding="3" cellspacing="0">
<tbody>
<tr>
<!-- start source code --> <td align="left" nowrap="nowrap"
valign="top"> <code><font color="#7f0055"><b>public interface </b></font><font
color="#000000">CDOSession </font><font color="#7f0055"><b>extends </b></font><font
color="#000000">CDOProtocolSession, IContainer&lt;CDOView&gt; </font><br>
<font color="#000000">{</font><br>
<font color="#ffffff">  </font><font color="#7f0055"><b>public </b></font><font
color="#7f0055"><b>int </b></font><font color="#000000">getReferenceChunkSize</font><font
color="#000000">()</font><font color="#000000">;</font><br>
<font color="#ffffff"></font><br>
<font color="#ffffff">  </font><font color="#7f0055"><b>public </b></font><font
color="#7f0055"><b>void </b></font><font color="#000000">setReferenceChunkSize</font><font
color="#000000">(</font><font color="#7f0055"><b>int </b></font><font
color="#000000">referenceChunkSize</font><font color="#000000">)</font><font
color="#000000">;</font></code> </td>
<!-- end source code --> </tr>
</tbody>
</table>
</div>
<!-- = END of automatically generated HTML code = --><br>
<!-- ======================================================== -->
- Loading collection elements (target objects) can be batched (the
current name is even worse)<br>
<title></title>
<style type="text/css">
<!--code { font-family: Courier New, Courier; font-size: 10pt; margin: 0px; }-->
</style>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<!-- ======================================================== -->
<!-- = Java Sourcecode to HTML automatically converted code = --><!-- = Java2Html Converter 5.0 [2006-02-26] by Markus Gebhard markus@jave.de = -->
<!-- = Further information: http://www.java2html.de = -->
<div class="java" align="left">
<table bgcolor="#ffffff" border="0" cellpadding="3" cellspacing="0">
<tbody>
<tr>
<!-- start source code --> <td align="left" nowrap="nowrap"
valign="top"> <code><font color="#7f0055"><b>public interface </b></font><font
color="#000000">CDOView </font><font color="#7f0055"><b>extends </b></font><font
color="#000000">CDOProtocolView, INotifier</font><br>
<font color="#000000">{</font><br>
<font color="#ffffff"></font><font color="#ffffff">  </font><font
color="#7f0055"><b>public </b></font><font color="#7f0055"><b>int </b></font><font
color="#000000">getLoadRevisionCollectionChunkSize</font> <font
color="#000000">()</font><font color="#000000">;</font><br>
<font color="#ffffff"></font><br>
<font color="#ffffff">  </font><font color="#7f0055"><b>public </b></font><font
color="#7f0055"><b>void </b></font><font color="#000000">setLoadRevisionCollectionChunkSize</font> <font
color="#000000">(</font><font color="#7f0055"><b>int </b></font><font
color="#000000">loadRevisionCollectionChunkSize</font><font
color="#000000">)</font><font color="#000000">;</font></code> </td>
<!-- end source code --> </tr>
</tbody>
</table>
</div>
<!-- = END of automatically generated HTML code = -->
<!-- ======================================================== --><br>
- One of the most sophisticated feature is automatic model usage
analyzation. The result are fetch rules that are sent to the server.
When later objects are loaded from the server, it can answer with
additional objects that are likely to be needed by the client in the
near future.<br>
<title></title>
<style type="text/css">
<!--code { font-family: Courier New, Courier; font-size: 10pt; margin: 0px; }-->
</style>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<!-- ======================================================== -->
<!-- = Java Sourcecode to HTML automatically converted code = --><!-- = Java2Html Converter 5.0 [2006-02-26] by Markus Gebhard markus@jave.de = -->
<!-- = Further information: http://www.java2html.de = -->
<div class="java" align="left">
<table bgcolor="#ffffff" border="0" cellpadding="3" cellspacing="0">
<tbody>
<tr>
<!-- start source code --> <td align="left" nowrap="nowrap"
valign="top"> <code><font color="#7f0055"><b>public class </b></font><font
color="#000000">CDOViewImpl </font><font color="#7f0055"><b>extends </b></font><font
color="#000000">org.eclipse.net4j.internal.util.event.Notifier  </font><font
color="#7f0055"><b>implements </b></font><font color="#000000">CDOView, CDOIDProvider,</font>
<font color="#000000">Adapter.Internal</font><br>
<font color="#000000">{</font><br>
<font color="#ffffff"></font><font color="#ffffff">  </font><font
color="#7f0055"><b>public </b></font><font color="#000000">CDOFeatureAnalyzer getFeatureAnalyzer</font ><font
color="#000000">()</font><br>
<font color="#ffffff">  </font><font color="#000000">{</font><br>
<font color="#ffffff">    </font><font color="#7f0055"><b>return </b></font><font
color="#000000">featureAnalyzer;</font><br>
<font color="#ffffff">  </font><font color="#000000">}</font><br>
<font color="#ffffff"></font><br>
<font color="#ffffff">  </font><font color="#7f0055"><b>public </b></font><font
color="#7f0055"><b>void </b></font><font color="#000000">setFeatureAnalyzer</font><font
color="#000000">(</font><font color="#000000">CDOFeatureAnalyzer featureAnalyzer</font ><font
color="#000000">)</font><br>
<font color="#ffffff">  </font><font color="#000000">{</font><br>
<font color="#ffffff">    </font><font color="#7f0055"><b>this</b></font><font
color="#000000">.featureAnalyzer = featureAnalyzer ==  </font><font
color="#7f0055"><b>null </b></font><font color="#000000">? CDOFeatureAnalyzer.NOOP : featureAnalyzer; </font><br>
<font color="#ffffff">  </font><font color="#000000">}</font></code>
</td>
<!-- end source code --> </tr>
</tbody>
</table>
</div>
<!-- = END of automatically generated HTML code = -->
<!-- ======================================================== --><br>
- There's a SmartReadAheadThread (if you're curious, please ask Simon).<br>
- Caches can be configured<br>
- ...<br>
<br>
<br>
Known issues of the DBStore:<br>
- When in auditing mode updates are always written as complete inserts
(by design)<br>
- When configuring toManyReferences to a value different from
LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.<br>
- No (few) prepared statements are used (see
<a class="moz-txt-link-freetext" href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487">https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487</a>)<br>
- Maybe others...<br>
<br>
Please note that the DBStore was not written by me to compete with
Hibernate in terms of performance or mapping capabilities.<br>
Now that Martin and I, well in the end mostly Martin, provide a
HibernateStore and this HibernateStore gets more and more features I
expect that there will be a trend away from the DBStore. <br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">
This is probably linked to the way the mapping to db is done, here's the
config I used (came from CDO-server example if I recall correctly).


&lt;repository name="repo1"&gt;
&lt;property name="overrideUUID"
value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/&gt;
&lt;property name="supportingAudits" value="true"/&gt;
&lt;property name="verifyingRevisions" value="false"/&gt;
&lt;property name="currentLRUCapacity" value="10000"/&gt;
</pre>
</blockquote>
I guess this one could be larger.<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap=""> &lt;property name="revisedLRUCapacity" value="100"/&gt;

&lt;store type="db"&gt;
&lt;!-- type: horizontal | vertical | &lt;any user-contributed type&gt; --&gt;
&lt;mappingStrategy type="horizontal"&gt;
&lt;!-- ONE_TABLE_PER_REFERENCE | ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES --&gt;
&lt;property name="toManyReferences" value="ONE_TABLE_PER_REFERENCE"/&gt;

&lt;!-- LIKE_ATTRIBUTES | LIKE_TO_MANY_REFERENCES --&gt;
&lt;property name="toOneReferences" value="LIKE_ATTRIBUTES"/&gt;

&lt;!-- MODEL | STRATEGY --&gt;
&lt;property name="mappingPrecedence" value="MODEL"/&gt;
</pre>
</blockquote>
This one is obsolete.<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap=""> &lt;/mappingStrategy&gt;

&lt;dbAdapter name="derby-embedded"/&gt;
&lt;dataSource class="org.apache.derby.jdbc.EmbeddedDataSource"
databaseName="/tmp/cdodb1"
createDatabase="create"/&gt;

&lt;/store&gt;
&lt;/repository&gt;




What do you think about that ? What throughput do you think I should
expect ? </pre>
</blockquote>
As I said that's very difficult to judge.<br>
I believe it's more appropriate to compare the actual values with
values from other measures (other setups, properties, etc. or even
other products) instead of random expectations.<br>
Apologies that I didn't have the time so far to provide some reference
measures myself. Maybe this gets better now that the team starts to
grow slowly ;-)<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">What are the main reasons fur such figures ?
</pre>
</blockquote>
Hmm, I *suspect* you don't like the figures? I can find no evidence
your assessment ;-)<br>
<br>
Ahh,one more note:<br>
Tracing can also have a significant impact on performance and
end-to-end throughput.<br>
Turn it off by all means. And please make sure that it's really turned
off, not only invisible (i.e. console is not a target).<br>
<br>
Cheers<br>
/Eike<br>
<br>
<br>
</body>
</html>

--------------090102080203020800020703--


Re: [CDO] reading/writting throughput performances [message #619064 is a reply to message #123598] Thu, 29 May 2008 13:03 Go to previous message
Cedric Brun is currently offline Cedric BrunFriend
Messages: 431
Registered: July 2009
Senior Member
Eike,

thanks for your detailled answer, I'll digg further and try to profile the
Net4J/CDO stack so that I understand the "50 objects/sec rate" when
retrieving data (which I think is surprisingly low).

Concerning the logging, what can I change to be *sure* that no logging will
occurs ?

Cheers,

Cédric



Eike Stepper wrote:

> Hi Cédric,
>
> That sounds interesting!
> Comments inline...
>
>
> Cédric Brun schrieb:
>> Hi Eike,
>>
>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>> measuring read/write throughput when I use CDO (RC2).
>>
>> I'm using an embeded-derby database, two different metamodels (complex
>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>> Features)
>>
>> The test is a "quick and dirty" one so please do not take these figures
>> too seriously, I would just like to have your input about the "order of
>> magnitude" I should expect or settings affecting seriously these
>> performances. I tried both JVM and TCP acceptor but the performances are
>> quite similar though JVM is a bit faster.
>>
> Looks like a sign that the NIO socket channels are efficient ,-)
> You used localhost for TCP transport I guess?
>
>>
>> * First case : building a small model (10000 elements) with the simple
>> MM, only adding instances in an empty database :
>> commiting change on every add I get a 2 objects per second rate.
>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>
> Seems to be consistent with the fact that only for commits there is
> remote traffic.
> And the backend transaction management is another overhead.
> So the impact of the number objects/commit starts with bigger
> transactions.
>
>> browsing the just commited elements opening a new transaction and getting
>> a new Resource instance, I get around 30 000 objets/sec rate just
>> iterating thanks to Resource.getAllContents().
>>
> Hehe, this is a sign that client side caching works well:
> All views and transactions of a single session share (and cache) their
> object state.
> Otherwise I'd expect much smaller values.
> With his ObjectivityStore implementation Simon (cc'ed) reached rates of
> several thousands objects/sec.
> But the DBStore (which you used) is not that optimized.
>
>> * Second case : I built a big model in the database using the complex
>> metamodel. It contains 700 000 elements. The creation went pretty nicely
>> (I did not measured it exactly as it was launched during a night, but it
>> seems I got around 100 objects/sec rate in writting as I was writting
>> every 700 objects.
>>
> Sounds not too bad.
>
>>
>> Once the model is built the derby database is 222 MB (which - I heard -
>> is not so big for derby).
>>
> Hmm, I have no extensive experience with Derby.
> To be honest, I'm really no database expert (although I can write simple
> SQL statements).
>
>> Retrieving the distant resource and calling eAllContents() on it, I get a
>> 50 Objects/sec rate at best, which mean my eAllContents() call would take
>> 4 hours !
>>
> Currently I'm not sure what exactly eAllContents() does do (iteration
> order and so).
> There are several CDO features that can have a major impact on load rates!
>
> - Collections (xrefs and containment) can be loaded lazily. I don't mean
> the target objects, which are lazy by default anyway, but even the
> target IDs that are in the collections. We call that "reference
> chunking" (but search for a better name is ongoing).
> |*public interface *CDOSession *extends *CDOProtocolSession,
> |IContainer<CDOView>
> {
> *public **int *getReferenceChunkSize();
>
> *public **void *setReferenceChunkSize(*int *referenceChunkSize);|
>
>
> - Loading collection elements (target objects) can be batched (the
> current name is even worse)
> |*public interface *CDOView *extends *CDOProtocolView, INotifier
> {
> *public **int *getLoadRevisionCollectionChunkSize();
>
> *public **void *setLoadRevisionCollectionChunkSize(*int
> *loadRevisionCollectionChunkSize);|
>
>
>
> - One of the most sophisticated feature is automatic model usage
> analyzation. The result are fetch rules that are sent to the server.
> When later objects are loaded from the server, it can answer with
> additional objects that are likely to be needed by the client in the
> near future.
> |*public class *CDOViewImpl *extends
> |*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
> |CDOIDProvider,
> Adapter.Internal
> {
> *public *CDOFeatureAnalyzer getFeatureAnalyzer()
> {
> *return *featureAnalyzer;
> }
>
> *public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
> {
> *this*.featureAnalyzer = featureAnalyzer == *null *?
> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
> }|
>
>
> - There's a SmartReadAheadThread (if you're curious, please ask Simon).
> - Caches can be configured
> - ...
>
>
> Known issues of the DBStore:
> - When in auditing mode updates are always written as complete inserts
> (by design)
> - When configuring toManyReferences to a value different from
> LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
> - No (few) prepared statements are used (see
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
> - Maybe others...
>
> Please note that the DBStore was not written by me to compete with
> Hibernate in terms of performance or mapping capabilities.
> Now that Martin and I, well in the end mostly Martin, provide a
> HibernateStore and this HibernateStore gets more and more features I
> expect that there will be a trend away from the DBStore.
>
>> This is probably linked to the way the mapping to db is done, here's the
>> config I used (came from CDO-server example if I recall correctly).
>>
>>
>> <repository name="repo1">
>> <property name="overrideUUID"
>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>> <property name="supportingAudits" value="true"/>
>> <property name="verifyingRevisions" value="false"/>
>> <property name="currentLRUCapacity" value="10000"/>
>>
> I guess this one could be larger.
>
>> <property name="revisedLRUCapacity" value="100"/>
>>
>> <store type="db">
>> <!-- type: horizontal | vertical | <any
>> user-contributed type> --> <mappingStrategy
>> type="horizontal">
>> <!-- ONE_TABLE_PER_REFERENCE |
>> ONE_TABLE_PER_CLASS |
>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>> <property name="toManyReferences"
>> value="ONE_TABLE_PER_REFERENCE"/>
>>
>> <!-- LIKE_ATTRIBUTES |
>> LIKE_TO_MANY_REFERENCES --> <property
>> name="toOneReferences"
>> value="LIKE_ATTRIBUTES"/>
>>
>> <!-- MODEL | STRATEGY -->
>> <property name="mappingPrecedence"
>> value="MODEL"/>
>>
> This one is obsolete.
>
>> </mappingStrategy>
>>
>> <dbAdapter name="derby-embedded"/>
>> <dataSource
>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>> databaseName="/tmp/cdodb1"
>> createDatabase="create"/>
>>
>> </store>
>> </repository>
>>
>>
>>
>>
>> What do you think about that ? What throughput do you think I should
>> expect ?
> As I said that's very difficult to judge.
> I believe it's more appropriate to compare the actual values with values
> from other measures (other setups, properties, etc. or even other
> products) instead of random expectations.
> Apologies that I didn't have the time so far to provide some reference
> measures myself. Maybe this gets better now that the team starts to grow
> slowly ;-)
>
>> What are the main reasons fur such figures ?
>>
> Hmm, I *suspect* you don't like the figures? I can find no evidence your
> assessment ;-)
>
> Ahh,one more note:
> Tracing can also have a significant impact on performance and end-to-end
> throughput.
> Turn it off by all means. And please make sure that it's really turned
> off, not only invisible (i.e. console is not a target).
>
> Cheers
> /Eike


http://cedric.brun.io news and articles on eclipse and eclipse modeling.
Re: [CDO] reading/writting throughput performances [message #619065 is a reply to message #123598] Thu, 29 May 2008 13:46 Go to previous message
Cedric Brun is currently offline Cedric BrunFriend
Messages: 431
Registered: July 2009
Senior Member
Eike,

As I don't know (yet ;) ) all the internals of CDO I'm not sure this is
relevant, but in the testcase I'm working on (with the complex metamodel,
big model in database and 50 o/sec throughput) my profiler says that most
of the time (like 99.9%) is spent in a readString call from
CDOClassRefImpl :

public CDOClassRefImpl(ExtendedDataInput in, String defaultURI) throws
IOException
{
// TODO Optimize transfer of URIs
packageURI = in.readString();
if (packageURI == null)
{
packageURI = defaultURI;
}

classifierID = in.readInt();
}

The TODO let me think this seems relevant but I'm not sure, what do you
think of it ?

Cédric


Eike Stepper wrote:

> Hi Cédric,
>
> That sounds interesting!
> Comments inline...
>
>
> Cédric Brun schrieb:
>> Hi Eike,
>>
>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>> measuring read/write throughput when I use CDO (RC2).
>>
>> I'm using an embeded-derby database, two different metamodels (complex
>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>> Features)
>>
>> The test is a "quick and dirty" one so please do not take these figures
>> too seriously, I would just like to have your input about the "order of
>> magnitude" I should expect or settings affecting seriously these
>> performances. I tried both JVM and TCP acceptor but the performances are
>> quite similar though JVM is a bit faster.
>>
> Looks like a sign that the NIO socket channels are efficient ,-)
> You used localhost for TCP transport I guess?
>
>>
>> * First case : building a small model (10000 elements) with the simple
>> MM, only adding instances in an empty database :
>> commiting change on every add I get a 2 objects per second rate.
>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>
> Seems to be consistent with the fact that only for commits there is
> remote traffic.
> And the backend transaction management is another overhead.
> So the impact of the number objects/commit starts with bigger
> transactions.
>
>> browsing the just commited elements opening a new transaction and getting
>> a new Resource instance, I get around 30 000 objets/sec rate just
>> iterating thanks to Resource.getAllContents().
>>
> Hehe, this is a sign that client side caching works well:
> All views and transactions of a single session share (and cache) their
> object state.
> Otherwise I'd expect much smaller values.
> With his ObjectivityStore implementation Simon (cc'ed) reached rates of
> several thousands objects/sec.
> But the DBStore (which you used) is not that optimized.
>
>> * Second case : I built a big model in the database using the complex
>> metamodel. It contains 700 000 elements. The creation went pretty nicely
>> (I did not measured it exactly as it was launched during a night, but it
>> seems I got around 100 objects/sec rate in writting as I was writting
>> every 700 objects.
>>
> Sounds not too bad.
>
>>
>> Once the model is built the derby database is 222 MB (which - I heard -
>> is not so big for derby).
>>
> Hmm, I have no extensive experience with Derby.
> To be honest, I'm really no database expert (although I can write simple
> SQL statements).
>
>> Retrieving the distant resource and calling eAllContents() on it, I get a
>> 50 Objects/sec rate at best, which mean my eAllContents() call would take
>> 4 hours !
>>
> Currently I'm not sure what exactly eAllContents() does do (iteration
> order and so).
> There are several CDO features that can have a major impact on load rates!
>
> - Collections (xrefs and containment) can be loaded lazily. I don't mean
> the target objects, which are lazy by default anyway, but even the
> target IDs that are in the collections. We call that "reference
> chunking" (but search for a better name is ongoing).
> |*public interface *CDOSession *extends *CDOProtocolSession,
> |IContainer<CDOView>
> {
> *public **int *getReferenceChunkSize();
>
> *public **void *setReferenceChunkSize(*int *referenceChunkSize);|
>
>
> - Loading collection elements (target objects) can be batched (the
> current name is even worse)
> |*public interface *CDOView *extends *CDOProtocolView, INotifier
> {
> *public **int *getLoadRevisionCollectionChunkSize();
>
> *public **void *setLoadRevisionCollectionChunkSize(*int
> *loadRevisionCollectionChunkSize);|
>
>
>
> - One of the most sophisticated feature is automatic model usage
> analyzation. The result are fetch rules that are sent to the server.
> When later objects are loaded from the server, it can answer with
> additional objects that are likely to be needed by the client in the
> near future.
> |*public class *CDOViewImpl *extends
> |*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
> |CDOIDProvider,
> Adapter.Internal
> {
> *public *CDOFeatureAnalyzer getFeatureAnalyzer()
> {
> *return *featureAnalyzer;
> }
>
> *public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
> {
> *this*.featureAnalyzer = featureAnalyzer == *null *?
> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
> }|
>
>
> - There's a SmartReadAheadThread (if you're curious, please ask Simon).
> - Caches can be configured
> - ...
>
>
> Known issues of the DBStore:
> - When in auditing mode updates are always written as complete inserts
> (by design)
> - When configuring toManyReferences to a value different from
> LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
> - No (few) prepared statements are used (see
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
> - Maybe others...
>
> Please note that the DBStore was not written by me to compete with
> Hibernate in terms of performance or mapping capabilities.
> Now that Martin and I, well in the end mostly Martin, provide a
> HibernateStore and this HibernateStore gets more and more features I
> expect that there will be a trend away from the DBStore.
>
>> This is probably linked to the way the mapping to db is done, here's the
>> config I used (came from CDO-server example if I recall correctly).
>>
>>
>> <repository name="repo1">
>> <property name="overrideUUID"
>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>> <property name="supportingAudits" value="true"/>
>> <property name="verifyingRevisions" value="false"/>
>> <property name="currentLRUCapacity" value="10000"/>
>>
> I guess this one could be larger.
>
>> <property name="revisedLRUCapacity" value="100"/>
>>
>> <store type="db">
>> <!-- type: horizontal | vertical | <any
>> user-contributed type> --> <mappingStrategy
>> type="horizontal">
>> <!-- ONE_TABLE_PER_REFERENCE |
>> ONE_TABLE_PER_CLASS |
>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>> <property name="toManyReferences"
>> value="ONE_TABLE_PER_REFERENCE"/>
>>
>> <!-- LIKE_ATTRIBUTES |
>> LIKE_TO_MANY_REFERENCES --> <property
>> name="toOneReferences"
>> value="LIKE_ATTRIBUTES"/>
>>
>> <!-- MODEL | STRATEGY -->
>> <property name="mappingPrecedence"
>> value="MODEL"/>
>>
> This one is obsolete.
>
>> </mappingStrategy>
>>
>> <dbAdapter name="derby-embedded"/>
>> <dataSource
>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>> databaseName="/tmp/cdodb1"
>> createDatabase="create"/>
>>
>> </store>
>> </repository>
>>
>>
>>
>>
>> What do you think about that ? What throughput do you think I should
>> expect ?
> As I said that's very difficult to judge.
> I believe it's more appropriate to compare the actual values with values
> from other measures (other setups, properties, etc. or even other
> products) instead of random expectations.
> Apologies that I didn't have the time so far to provide some reference
> measures myself. Maybe this gets better now that the team starts to grow
> slowly ;-)
>
>> What are the main reasons fur such figures ?
>>
> Hmm, I *suspect* you don't like the figures? I can find no evidence your
> assessment ;-)
>
> Ahh,one more note:
> Tracing can also have a significant impact on performance and end-to-end
> throughput.
> Turn it off by all means. And please make sure that it's really turned
> off, not only invisible (i.e. console is not a target).
>
> Cheers
> /Eike


http://cedric.brun.io news and articles on eclipse and eclipse modeling.
Re: [CDO] reading/writting throughput performances [message #619067 is a reply to message #123598] Thu, 29 May 2008 16:13 Go to previous message
Simon Mc Duff is currently offline Simon Mc DuffFriend
Messages: 596
Registered: July 2009
Senior Member
This is a multi-part message in MIME format.

------=_NextPart_000_04AC_01C8C185.678D3E20
Content-Type: text/plain;
charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

We noticed a performance degradation in the following case:
=20
When we fetch a collection of objects that contains collections that we =
don't want to use.=20
Ex:
=20
Model:
=20
ClassB
{
int foo1;
List childrens1;
List childrens2;
List childrens3;
};
=20
Code:

Resource.contents contains objects of type ClassB.
=20
By fetching instances of classB, we will also fetched some items in the =
list(childrens1, childrens2, childrens3) The problem we had.. is we do =
not know if we want to use it. SO better not to download them.

So In our production(we used our own IStore), we never download any =
objects from collection.=20
I'm working at the moment to put in place all feature we used in the =
current CDO.
That one will be part of=20
https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214477 To begin =
with,CDORevisionImpl needs to supports list that are 0 even if =
referenceChunkSize =3D=3D 0.
=20
It is true that we obtain good performance.
Worst case :
1500 objects /sec (Real fetch from disk. Page that contains the objects, =
wasn't in the cache of the filesystem.)=20
=20
Best case=20
12 000 objects/Sec (Fetch objects that are already in the case of the =
filesystem)
=20
(These statistics always fetch objects from server to client. The cache =
client isn't used)
=20
We have another optimization one with SmartReadAhead but for now only =
work for UI client. (for now) This is something we will integrate as =
well in CDO. But for now we concentrate on the other things.
=20
I used an Object database. Based on my preliminary test without CDO.
I compare
=20
Hibernate vs Objectivity/EMF :=20
Objectivity/EMF is around 3 to 50 times faster than Hibernate.
=20
The worst.. you will get 30 to 240 objects/sec or
the best... you will get 500 to 4000 objects/sec
=20
Based on your comments... you are getting the worst (with Derby)!! :-)
=20
=20
Another comments about insertion.
=20
We found another problem when we insert data(DOn't know if it is still =
true). Resource.Contents is a unique list. Basically each time we insert =
one element, it needs to fetch all the collection to see if we already =
have this element. Basically.. it is time consuming. In our case we are =
not going through this stage to persist our objects... but the objects =
are not accessible through that list.
=20


If speed is an issue for you!! We (You included :-)) can optimize it. =
What I like is Eike did a really good job doing that framwork and =
optimizations are easy to add.
Also you can change your back-end without affecting you client side. =
(Maybe going to an Object database!!)
THis was one of the purpose of CDO!! :-)


Simon


"Eike Stepper" <stepper@sympedia.de> a =C3=A9crit dans le message de =
news: 483E9332.6050800@sympedia.de...
Hi C=C3=A9dric,

That sounds interesting!
Comments inline...


C=C3=A9dric Brun schrieb:=20
Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex =
one -
250 EClassifiers/300 features and a simple one - 1 Classifier/3 =
Features)

The test is a "quick and dirty" one so please do not take these figures =
too
seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.
Looks like a sign that the NIO socket channels are efficient ,-)
You used localhost for TCP transport I guess?



* First case : building a small model (10000 elements) with the simple =
MM,
only adding instances in an empty database :=20
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.
Seems to be consistent with the fact that only for commits there is =
remote traffic.
And the backend transaction management is another overhead.
So the impact of the number objects/commit starts with bigger =
transactions.


browsing the just commited elements opening a new transaction and =
getting a
new Resource instance, I get around 30 000 objets/sec rate just =
iterating
thanks to Resource.getAllContents().
Hehe, this is a sign that client side caching works well:
All views and transactions of a single session share (and cache) their =
object state.
Otherwise I'd expect much smaller values.
With his ObjectivityStore implementation Simon (cc'ed) reached rates =
of several thousands objects/sec.
But the DBStore (which you used) is not that optimized.


* Second case : I built a big model in the database using the complex
metamodel. It contains 700 000 elements. The creation went pretty nicely =
(I
did not measured it exactly as it was launched during a night, but it =
seems
I got around 100 objects/sec rate in writting as I was writting every =
700
objects.
Sounds not too bad.



Once the model is built the derby database is 222 MB (which - I heard - =
is
not so big for derby).
Hmm, I have no extensive experience with Derby.
To be honest, I'm really no database expert (although I can write =
simple SQL statements).


Retrieving the distant resource and calling eAllContents() on it, I get =
a 50
Objects/sec rate at best, which mean my eAllContents() call would take 4
hours !
Currently I'm not sure what exactly eAllContents() does do (iteration =
order and so).
There are several CDO features that can have a major impact on load =
rates!

- Collections (xrefs and containment) can be loaded lazily. I don't =
mean the target objects, which are lazy by default anyway, but even the =
target IDs that are in the collections. We call that "reference =
chunking" (but search for a better name is ongoing).

public interface CDOSession extends CDOProtocolSession, =
IContainer<CDOView>
{
public int getReferenceChunkSize();

public void setReferenceChunkSize(int referenceChunkSize); =20


- Loading collection elements (target objects) can be batched (the =
current name is even worse)

public interface CDOView extends CDOProtocolView, INotifier
{
public int getLoadRevisionCollectionChunkSize();

public void setLoadRevisionCollectionChunkSize(int =
loadRevisionCollectionChunkSize); =20


- One of the most sophisticated feature is automatic model usage =
analyzation. The result are fetch rules that are sent to the server. =
When later objects are loaded from the server, it can answer with =
additional objects that are likely to be needed by the client in the =
near future.

public class CDOViewImpl extends =
org.eclipse.net4j.internal.util.event.Notifier implements CDOView, =
CDOIDProvider, Adapter.Internal
{
public CDOFeatureAnalyzer getFeatureAnalyzer()
{
return featureAnalyzer;
}

public void setFeatureAnalyzer(CDOFeatureAnalyzer =
featureAnalyzer)
{
this.featureAnalyzer =3D featureAnalyzer =3D=3D null ? =
CDOFeatureAnalyzer.NOOP : featureAnalyzer;
} =20


- There's a SmartReadAheadThread (if you're curious, please ask =
Simon).
- Caches can be configured
- ...


Known issues of the DBStore:
- When in auditing mode updates are always written as complete inserts =
(by design)
- When configuring toManyReferences to a value different from =
LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
- No (few) prepared statements are used (see =
https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214487)
- Maybe others...

Please note that the DBStore was not written by me to compete with =
Hibernate in terms of performance or mapping capabilities.
Now that Martin and I, well in the end mostly Martin, provide a =
HibernateStore and this HibernateStore gets more and more features I =
expect that there will be a trend away from the DBStore.=20


This is probably linked to the way the mapping to db is done, here's the
config I used (came from CDO-server example if I recall correctly).


<repository name=3D"repo1">
<property name=3D"overrideUUID"
value=3D"1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
<property name=3D"supportingAudits" value=3D"true"/>
<property name=3D"verifyingRevisions" value=3D"false"/>
<property name=3D"currentLRUCapacity" value=3D"10000"/>
I guess this one could be larger.


<property name=3D"revisedLRUCapacity" value=3D"100"/>
=20
<store type=3D"db">
<!-- type: horizontal | vertical | <any =
user-contributed type> -->
<mappingStrategy type=3D"horizontal">
<!-- ONE_TABLE_PER_REFERENCE | =
ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
<property name=3D"toManyReferences" =
value=3D"ONE_TABLE_PER_REFERENCE"/>

<!-- LIKE_ATTRIBUTES | =
LIKE_TO_MANY_REFERENCES -->
<property name=3D"toOneReferences" =
value=3D"LIKE_ATTRIBUTES"/>
=20
<!-- MODEL | STRATEGY -->
<property name=3D"mappingPrecedence" =
value=3D"MODEL"/>
This one is obsolete.


</mappingStrategy>
=20
<dbAdapter name=3D"derby-embedded"/>
<dataSource =
class=3D"org.apache.derby.jdbc.EmbeddedDataSource"=20
databaseName=3D"/tmp/cdodb1"
createDatabase=3D"create"/>

</store>
</repository>




What do you think about that ? What throughput do you think I should
expect ? As I said that's very difficult to judge.
I believe it's more appropriate to compare the actual values with =
values from other measures (other setups, properties, etc. or even other =
products) instead of random expectations.
Apologies that I didn't have the time so far to provide some reference =
measures myself. Maybe this gets better now that the team starts to grow =
slowly ;-)


What are the main reasons fur such figures ?
Hmm, I *suspect* you don't like the figures? I can find no evidence =
your assessment ;-)

Ahh,one more note:
Tracing can also have a significant impact on performance and =
end-to-end throughput.
Turn it off by all means. And please make sure that it's really turned =
off, not only invisible (i.e. console is not a target).

Cheers
/Eike



------=_NextPart_000_04AC_01C8C185.678D3E20
Content-Type: text/html;
charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

=EF=BB=BF<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD><TITLE></TITLE>
<META http-equiv=3DContent-Type content=3Dtext/html;charset=3DUTF-8>
<META content=3D"MSHTML 6.00.6000.16608" name=3DGENERATOR></HEAD>
<BODY text=3D#000000 bgColor=3D#ffffff>
<DIV>We noticed a performance degradation in the following=20
case:<BR>&nbsp;<BR>When we fetch a collection of objects that contains=20
collections that we don't want to use.=20
<BR>Ex:<BR>&nbsp;<BR>Model:<BR>&nbsp;<BR>ClassB <BR>{<BR>&nbsp; int=20
foo1;<BR>&nbsp; List childrens1;<BR>&nbsp; List childrens2;<BR>&nbsp; =
List=20
childrens3;<BR>};<BR>&nbsp;<BR>Code:</DIV>
<DIV><BR>Resource.contents contains objects of type =
ClassB.<BR>&nbsp;<BR>By=20
fetching instances of classB, we will also fetched some items in the=20
list(childrens1, childrens2, childrens3)&nbsp;<FONT face=3D"Courier =
New"><FONT=20
color=3D#000000>The problem we had.. is we do not know if we want to use =
it. SO=20
better not to download them.</FONT></FONT></DIV><FONT face=3D"Courier =
New"></FONT>
<DIV><BR><FONT face=3D"Courier New" color=3D#000000>So In our =
production(we used our=20
own IStore), we never download any objects from collection. =
</FONT><BR><FONT=20
face=3D"Courier New" color=3D#000000>I'm working at the moment to put in =
place all=20
feature we used in the current CDO.</FONT><BR><FONT face=3D"Courier New" =

color=3D#000000>That one will be part of </FONT><BR><A=20
href=3D"https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214477" =
target=3D_blank><FONT=20
color=3D#0068cf>https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214477</F=
ONT></A>.=20
To begin with,CDORevisionImpl needs to supports list that are 0 even if =
<FONT=20
face=3D"Courier New" color=3D#000000>referenceChunkSize =3D=3D =
0.</FONT><BR><FONT=20
face=3D"Courier New" color=3D#000000></FONT>&nbsp;<BR><FONT =
face=3D"Courier New"=20
color=3D#000000>It is true that we obtain good =
performance.</FONT><BR><FONT=20
face=3D"Courier New" color=3D#000000>Worst case :</FONT><BR><FONT =
face=3D"Courier New"=20
color=3D#000000>1500 objects /sec (Real fetch from&nbsp;disk. Page that =
contains=20
the objects, wasn't in the cache of the =
filesystem.)&nbsp;</FONT><BR><FONT=20
face=3D"Courier New" color=3D#000000></FONT>&nbsp;<BR><FONT =
face=3D"Courier New"=20
color=3D#000000>Best case </FONT><BR><FONT face=3D"Courier New" =
color=3D#000000>12 000=20
objects/Sec (Fetch objects that are already in the case of the=20
filesystem)</FONT><BR><FONT face=3D"Courier New"=20
color=3D#000000></FONT>&nbsp;<BR>(These statistics always fetch objects =
from=20
server to client. The cache client isn't used)<BR>&nbsp;<BR><FONT=20
face=3D"Courier New" color=3D#000000>We have another optimization one =
with=20
SmartReadAhead but for now only work for UI client. (for now)&nbsp; This =
is=20
something we will integrate as well in CDO. But for now we concentrate =
on the=20
other things.</FONT><BR><FONT face=3D"Courier New"=20
color=3D#000000></FONT>&nbsp;<BR><FONT face=3D"Courier New" =
color=3D#000000>I used an=20
Object database. Based on my preliminary test without =
CDO.</FONT><BR><FONT=20
face=3D"Courier New" color=3D#000000>I compare</FONT><BR><FONT =
face=3D"Courier New"=20
color=3D#000000></FONT>&nbsp;<BR><FONT face=3D"Courier New" =
color=3D#000000>Hibernate=20
vs </FONT><FONT face=3D"Courier New" color=3D#000000>Objectivity/EMF=20
:&nbsp;</FONT><BR><FONT face=3D"Courier New" =
color=3D#000000>Objectivity/EMF is=20
around 3 to 50 times faster than Hibernate.</FONT><BR><FONT =
face=3D"Courier New"=20
color=3D#000000></FONT>&nbsp;<BR><FONT face=3D"Courier New" =
color=3D#000000>The=20
worst.. you will get 30 to 240 objects/sec or</FONT><BR><FONT =
face=3D"Courier New"=20
color=3D#000000>the best... you will get 500 to 4000 =
objects/sec</FONT><BR><FONT=20
face=3D"Courier New" color=3D#000000></FONT>&nbsp;<BR><FONT =
face=3D"Courier New"=20
color=3D#000000>Based on your comments... you are getting the worst =
(with Derby)!!=20
:-)</FONT><BR>&nbsp;<BR>&nbsp;<BR>Another comments about=20
insertion.<BR>&nbsp;<BR>We found another problem when we insert =
data(DOn't know=20
if it is still true). Resource.Contents is&nbsp;a unique list. Basically =
each=20
time we insert one element, it needs to fetch all the collection to see =
if we=20
already have this element. Basically.. it is time consuming. In our case =
we are=20
not going through this stage to persist our objects... but the objects =
are not=20
accessible through that list.<BR>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>If speed is an issue for you!! We (You =
included=20
:-)) can optimize it. What I like is Eike did a really good job doing =
that=20
framwork and optimizations are easy to add.</FONT></DIV>
<DIV><FONT face=3DArial size=3D2>Also you can change your back-end =
without affecting=20
you client side. (Maybe going to an Object database!!)</FONT></DIV>
<DIV><FONT face=3DArial size=3D2>THis was one of the purpose of CDO!!=20
:-)</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><BR>Simon<BR><BR></DIV>
<BLOCKQUOTE=20
style=3D"PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; =
BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">
<DIV>"Eike Stepper" &lt;<A=20
href=3D"mailto:stepper@sympedia.de">stepper@sympedia.de</A>&gt; a =
=C3=A9crit dans le=20
message de news: <A=20
=
href=3D"mailto:483E9332.6050800@sympedia.de">483E9332.6050800@sympedia.de=
</A>...</DIV>Hi=20
C=C3=A9dric,<BR><BR>That sounds interesting!<BR>Comments=20
inline...<BR><BR><BR>C=C3=A9dric Brun schrieb:=20
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex =
one -
250 EClassifiers/300 features and a simple one - 1 Classifier/3 =
Features)

The test is a "quick and dirty" one so please do not take these figures =
too
seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.
</PRE></BLOCKQUOTE>Looks like a sign that the NIO socket channels are=20
efficient ,-)<BR>You used localhost for TCP transport I guess?<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">
* First case : building a small model (10000 elements) with the simple =
MM,
only adding instances in an empty database :=20
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.
</PRE></BLOCKQUOTE>Seems to be consistent with the fact that only for=20
commits there is remote traffic.<BR>And the backend transaction =
management is=20
another overhead.<BR>So the impact of the number objects/commit starts =
with=20
bigger transactions.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">browsing the just commited elements opening =
a new transaction and getting a
new Resource instance, I get around 30 000 objets/sec rate just =
iterating
thanks to Resource.getAllContents().
</PRE></BLOCKQUOTE>Hehe, this is a sign that client side caching works =

well:<BR>All views and transactions of a single session share (and =
cache)=20
their object state.<BR>Otherwise I'd expect much smaller =
values.<BR>With his=20
ObjectivityStore implementation Simon (cc'ed) reached rates of several =

thousands objects/sec.<BR>But the DBStore (which you used) is not that =

optimized.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">* Second case : I built a big model in the =
database using the complex
metamodel. It contains 700 000 elements. The creation went pretty nicely =
(I
did not measured it exactly as it was launched during a night, but it =
seems
I got around 100 objects/sec rate in writting as I was writting every =
700
objects.
</PRE></BLOCKQUOTE>Sounds not too bad.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">
Once the model is built the derby database is 222 MB (which - I heard - =
is
not so big for derby).
</PRE></BLOCKQUOTE>Hmm, I have no extensive experience with =
Derby.<BR>To be=20
honest, I'm really no database expert (although I can write simple SQL =

statements).<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">Retrieving the distant resource and calling =
eAllContents() on it, I get a 50
Objects/sec rate at best, which mean my eAllContents() call would take 4
hours !
</PRE></BLOCKQUOTE>Currently I'm not sure what exactly eAllContents() =
does=20
do (iteration order and so).<BR>There are several CDO features that =
can have a=20
major impact on load rates!<BR><BR>- Collections (xrefs and =
containment) can=20
be loaded lazily. I don't mean the target objects, which are lazy by =
default=20
anyway, but even the target IDs that are in the collections. We call =
that=20
"reference chunking" (but search for a better name is ongoing).<BR>
<STYLE type=3Dtext/css>CODE {
FONT-SIZE: 10pt; MARGIN: 0px; FONT-FAMILY: Courier New, Courier
}
</STYLE>
<!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><!-- =3D Java Sourcecode to HTML automatically =
converted code =3D --><!-- =3D Java2Html Converter 5.0 [2006-02-26] by =
Markus Gebhard markus@jave.de =3D --><!-- =3D Further =
information: http://www.java2html.de =3D -->
<DIV class=3Djava align=3Dleft>
<TABLE cellSpacing=3D0 cellPadding=3D3 bgColor=3D#ffffff border=3D0>
<TBODY>
<TR><!-- start source code -->
<TD vAlign=3Dtop noWrap align=3Dleft><CODE><FONT=20
color=3D#7f0055><B>public&nbsp;interface&nbsp;</B ></FONT><FONT=20
color=3D#000000>CDOSession&nbsp;</FONT><FONT=20
color=3D#7f0055><B>extends&nbsp;</B></FONT><FONT=20
=
color=3D#000000> CDOProtocolSession,&nbsp;IContainer&lt;CDOView&g t; </FONT>=
<BR><FONT=20
color=3D#000000>{</FONT><BR><FONT =
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>int&nbsp;</B></FONT><FONT=20
color=3D#000000>getReferenceChunkSize</FONT><FONT=20
color=3D#000000>()</FONT><FONT color=3D#000000>;</FONT><BR><FONT =

color=3D#ffffff></FONT><BR><FONT =
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>void&nbsp;</B></FONT><FONT=20
color=3D#000000>setReferenceChunkSize</FONT><FONT=20
color=3D#000000>(</FONT><FONT =
color=3D#7f0055><B>int&nbsp;</B></FONT><FONT=20
color=3D#000000>referenceChunkSize</FONT><FONT =
color=3D#000000>)</FONT><FONT=20
color=3D#000000>;</FONT></CODE>=20
</TD><!-- end source code --></TR></TBODY></TABLE></DIV><!-- =3D =
END of automatically generated HTML code =3D --><BR><!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D -->-=20
Loading collection elements (target objects) can be batched (the =
current name=20
is even worse)<BR>
<STYLE type=3Dtext/css>
<!--code { font-family: Courier New, Courier; font-size: 10pt; =
margin: 0px; }-->
</STYLE>
<!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><!-- =3D Java Sourcecode to HTML automatically =
converted code =3D --><!-- =3D Java2Html Converter 5.0 [2006-02-26] by =
Markus Gebhard markus@jave.de =3D --><!-- =3D Further =
information: http://www.java2html.de =3D -->
<DIV class=3Djava align=3Dleft>
<TABLE cellSpacing=3D0 cellPadding=3D3 bgColor=3D#ffffff border=3D0>
<TBODY>
<TR><!-- start source code -->
<TD vAlign=3Dtop noWrap align=3Dleft><CODE><FONT=20
color=3D#7f0055><B>public&nbsp;interface&nbsp;</B ></FONT><FONT=20
color=3D#000000>CDOView&nbsp;</FONT><FONT=20
color=3D#7f0055><B>extends&nbsp;</B></FONT><FONT=20
color=3D#000000>CDOProtocolView,&nbsp;INotifier</FONT> <BR><FONT=20
color=3D#000000>{</FONT><BR><FONT color=3D#ffffff></FONT><FONT=20
color=3D#ffffff>&nbsp; </FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>int&nbsp;</B></FONT><FONT=20
color=3D#000000>getLoadRevisionCollectionChunkSize</FONT><FONT=20
color=3D#000000>()</FONT><FONT color=3D#000000>;</FONT><BR><FONT =

color=3D#ffffff></FONT><BR><FONT =
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>void&nbsp;</B></FONT><FONT=20
color=3D#000000>setLoadRevisionCollectionChunkSize</FONT><FONT=20
color=3D#000000>(</FONT><FONT =
color=3D#7f0055><B>int&nbsp;</B></FONT><FONT=20
color=3D#000000>loadRevisionCollectionChunkSize</FONT><FONT=20
color=3D#000000>)</FONT><FONT color=3D#000000>;</FONT></CODE> =
</TD><!-- end source code --></TR></TBODY></TABLE></DIV><!-- =3D =
END of automatically generated HTML code =3D --><!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><BR>-=20
One of the most sophisticated feature is automatic model usage =
analyzation.=20
The result are fetch rules that are sent to the server. When later =
objects are=20
loaded from the server, it can answer with additional objects that are =
likely=20
to be needed by the client in the near future.<BR>
<STYLE type=3Dtext/css>
<!--code { font-family: Courier New, Courier; font-size: 10pt; =
margin: 0px; }-->
</STYLE>
<!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><!-- =3D Java Sourcecode to HTML automatically =
converted code =3D --><!-- =3D Java2Html Converter 5.0 [2006-02-26] by =
Markus Gebhard markus@jave.de =3D --><!-- =3D Further =
information: http://www.java2html.de =3D -->
<DIV class=3Djava align=3Dleft>
<TABLE cellSpacing=3D0 cellPadding=3D3 bgColor=3D#ffffff border=3D0>
<TBODY>
<TR><!-- start source code -->
<TD vAlign=3Dtop noWrap align=3Dleft><CODE><FONT=20
color=3D#7f0055><B>public&nbsp;class&nbsp;</B></FONT ><FONT=20
color=3D#000000>CDOViewImpl&nbsp;</FONT><FONT=20
color=3D#7f0055><B>extends&nbsp;</B></FONT><FONT=20
=
color=3D#000000>org.eclipse.net4j.internal.util.event.Notifier&nbsp; </FON=
T><FONT=20
color=3D#7f0055><B>implements&nbsp;</B></FONT><FONT=20
color=3D#000000>CDOView,&nbsp;CDOIDProvider,</FONT> <FONT=20
color=3D#000000>Adapter.Internal</FONT><BR><FONT=20
color=3D#000000>{</FONT><BR><FONT color=3D#ffffff></FONT><FONT=20
color=3D#ffffff>&nbsp; </FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
=
color=3D#000000>CDOFeatureAnalyzer&nbsp;getFeatureAnalyzer </FONT><FONT=20
color=3D#000000>()</FONT><BR><FONT =
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#000000>{</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;&nbsp;&nbsp;</FONT ><FONT=20
color=3D#7f0055><B>return&nbsp;</B></FONT><FONT=20
color=3D#000000>featureAnalyzer;</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT =
color=3D#000000>}</FONT><BR><FONT=20
color=3D#ffffff></FONT><BR><FONT =
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>void&nbsp;</B></FONT><FONT=20
color=3D#000000>setFeatureAnalyzer</FONT><FONT =
color=3D#000000>(</FONT><FONT=20
=
color=3D#000000>CDOFeatureAnalyzer&nbsp;featureAnalyzer </FONT><FONT=20
color=3D#000000>)</FONT><BR><FONT =
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#000000>{</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;&nbsp;&nbsp;</FONT ><FONT=20
color=3D#7f0055><B>this</B></FONT><FONT=20
=
color=3D#000000> .featureAnalyzer&nbsp;=3D&nbsp;featureAnalyzer&n bsp;=3D=3D=
&nbsp;</FONT><FONT=20
color=3D#7f0055><B>null&nbsp;</B></FONT><FONT=20
=
color=3D#000000> ?&nbsp;CDOFeatureAnalyzer.NOOP&nbsp;:&nbsp;featu reAnalyze=
r;</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT =
color=3D#000000>}</FONT></CODE> </TD><!-- end source code =
--></TR></TBODY></TABLE></DIV><!-- =3D END of automatically =
generated HTML code =3D --><!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><BR>-=20
There's a SmartReadAheadThread (if you're curious, please ask =
Simon).<BR>-=20
Caches can be configured<BR>- ...<BR><BR><BR>Known issues of the =
DBStore:<BR>-=20
When in auditing mode updates are always written as complete inserts =
(by=20
design)<BR>- When configuring toManyReferences to a value different =
from=20
LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.<BR>- =
No (few)=20
prepared statements are used (see <A class=3Dmoz-txt-link-freetext=20
=
href=3D"https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214487">https://b=
ugs.eclipse.org/bugs/show_bug.cgi?id=3D214487</A>)<BR>-=20
Maybe others...<BR><BR>Please note that the DBStore was not written by =
me to=20
compete with Hibernate in terms of performance or mapping =
capabilities.<BR>Now=20
that Martin and I, well in the end mostly Martin, provide a =
HibernateStore and=20
this HibernateStore gets more and more features I expect that there =
will be a=20
trend away from the DBStore. <BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">This is probably linked to the way the =
mapping to db is done, here's the
config I used (came from CDO-server example if I recall correctly).


&lt;repository name=3D"repo1"&gt;
&lt;property name=3D"overrideUUID"
value=3D"1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/&gt;
&lt;property name=3D"supportingAudits" =
value=3D"true"/&gt;
&lt;property name=3D"verifyingRevisions" =
value=3D"false"/&gt;
&lt;property name=3D"currentLRUCapacity" =
value=3D"10000"/&gt;
</PRE></BLOCKQUOTE>I guess this one could be larger.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D""> &lt;property =
name=3D"revisedLRUCapacity" value=3D"100"/&gt;
=20
&lt;store type=3D"db"&gt;
&lt;!-- type: horizontal | vertical | &lt;any =
user-contributed type&gt; --&gt;
&lt;mappingStrategy type=3D"horizontal"&gt;
&lt;!-- ONE_TABLE_PER_REFERENCE | =
ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES =
--&gt;
&lt;property name=3D"toManyReferences" =
value=3D"ONE_TABLE_PER_REFERENCE"/&gt;

&lt;!-- LIKE_ATTRIBUTES | =
LIKE_TO_MANY_REFERENCES --&gt;
&lt;property name=3D"toOneReferences" =
value=3D"LIKE_ATTRIBUTES"/&gt;
=20
&lt;!-- MODEL | STRATEGY --&gt;
&lt;property name=3D"mappingPrecedence" =
value=3D"MODEL"/&gt;
</PRE></BLOCKQUOTE>This one is obsolete.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D""> =
&lt;/mappingStrategy&gt;
=20
&lt;dbAdapter name=3D"derby-embedded"/&gt;
&lt;dataSource =
class=3D"org.apache.derby.jdbc.EmbeddedDataSource"=20
databaseName=3D"/tmp/cdodb1"
createDatabase=3D"create"/&gt;

&lt;/store&gt;
&lt;/repository&gt;




What do you think about that ? What throughput do you think I should
expect ? </PRE></BLOCKQUOTE>As I said that's very difficult to =
judge.<BR>I=20
believe it's more appropriate to compare the actual values with values =
from=20
other measures (other setups, properties, etc. or even other products) =
instead=20
of random expectations.<BR>Apologies that I didn't have the time so =
far to=20
provide some reference measures myself. Maybe this gets better now =
that the=20
team starts to grow slowly ;-)<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">What are the main reasons fur such figures =
?
</PRE></BLOCKQUOTE>Hmm, I *suspect* you don't like the figures? I can =
find=20
no evidence your assessment ;-)<BR><BR>Ahh,one more note:<BR>Tracing =
can also=20
have a significant impact on performance and end-to-end =
throughput.<BR>Turn it=20
off by all means. And please make sure that it's really turned off, =
not only=20
invisible (i.e. console is not a=20
target).<BR><BR>Cheers<BR>/Eike<BR><BR><BR></BLOCKQUOTE ></BODY></HTML>

------=_NextPart_000_04AC_01C8C185.678D3E20--
Re: [CDO] reading/writting throughput performances [message #619068 is a reply to message #123623] Thu, 29 May 2008 16:23 Go to previous message
Simon Mc Duff is currently offline Simon Mc DuffFriend
Messages: 596
Registered: July 2009
Senior Member
Did you disable the following :
CDOSessionConfigurationImpl.setLegacySupportEnabled(true);

before opening your session ?

This could cause what you are observing ?

If your classes are generating with CDOObjectImpl as the super class, you
can disableLegacySupport by putting it at true.

Simon


"C
Re: [CDO] reading/writting throughput performances [message #619069 is a reply to message #123664] Thu, 29 May 2008 16:28 Go to previous message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
This is a multi-part message in MIME format.
--------------090803020509090708070909
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 8bit



Simon McDuff schrieb:
> Did you disable the following :
> CDOSessionConfigurationImpl.setLegacySupportEnabled(true);
>
> before opening your session ?
>
> This could cause what you are observing ?
>
> If your classes are generating with CDOObjectImpl as the super class, you
> can disableLegacySupport by putting it at true.
>
Good point in general!

But I added some logic some time ago that ensures that *session legacy
support* can only be enabled if the *legacy system* is available in general:

| *public **void *setDisableLegacyObjects(*boolean *disableLegacyObjects)
{
checkInactive();
*if *(!disableLegacyObjects && !FSMUtil.isLegacySystemAvailable())
{
*throw new *LegacySystemNotAvailableException();
}

*this*.disableLegacyObjects = disableLegacyObjects;
}|


Notice that the whole legacy system is currently (since the work on the
HibernateStore) broken.
The needed CDOWeaver is currently not part of the distro anymore (also
due to issues with the build system).

This altogether seems to exclude the legacy system as root cause for
performance degradation.

Cheers
/Eike



> Simon
>
>
> "C


Re: [CDO] reading/writting throughput performances [message #619070 is a reply to message #123681] Thu, 29 May 2008 16:41 Go to previous message
Simon Mc Duff is currently offline Simon Mc DuffFriend
Messages: 596
Registered: July 2009
Senior Member
This is a multi-part message in MIME format.

------=_NextPart_000_04D9_01C8C189.41CE2B50
Content-Type: text/plain;
charset="ISO-8859-15"
Content-Transfer-Encoding: quoted-printable

Good point Eike,

Cedric,

How do you create your CDOSessionIMpl ?
Are you using your own mechanism ?

Because it could still occured in the case we doesn`t go through =
CDOSessionConfigurationImpl and create directly the CDOSession

Ex:

CDOSession session =3D CDOSessionImpl();

By default disablyLegacySystem is false.

In this case Eike, Do we have something else to avoid that ? Maybe I`m =
not aware of other mechanism.

Simon

"Eike Stepper" <stepper@sympedia.de> a =E9crit dans le message de news: =
483ED9A4.7010302@sympedia.de...


Simon McDuff schrieb:=20
Did you disable the following :
CDOSessionConfigurationImpl.setLegacySupportEnabled(true);

before opening your session ?

This could cause what you are observing ?

If your classes are generating with CDOObjectImpl as the super class, =
you=20
can disableLegacySupport by putting it at true.
Good point in general!

But I added some logic some time ago that ensures that *session legacy =
support* can only be enabled if the *legacy system* is available in =
general:


public void setDisableLegacyObjects(boolean =
disableLegacyObjects)
{
checkInactive();
if (!disableLegacyObjects && =
!FSMUtil.isLegacySystemAvailable())
{
throw new LegacySystemNotAvailableException();
}

this.disableLegacyObjects =3D disableLegacyObjects;
} =20


Notice that the whole legacy system is currently (since the work on =
the HibernateStore) broken.
The needed CDOWeaver is currently not part of the distro anymore (also =
due to issues with the build system).

This altogether seems to exclude the legacy system as root cause for =
performance degradation.

Cheers
/Eike




Simon


"C=E9dric Brun" <cedric.brun@obeo.fr> a =E9crit dans le message de news: =

g1mbhe$2cl$1@build.eclipse.org...
Eike,

As I don't know (yet ;) ) all the internals of CDO I'm not sure this is
relevant, but in the testcase I'm working on (with the complex =
metamodel,
big model in database and 50 o/sec throughput) my profiler says that =
most
of the time (like 99.9%) is spent in a readString call from
CDOClassRefImpl :

public CDOClassRefImpl(ExtendedDataInput in, String defaultURI) throws
IOException
{
// TODO Optimize transfer of URIs
packageURI =3D in.readString();
if (packageURI =3D=3D null)
{
packageURI =3D defaultURI;
}

classifierID =3D in.readInt();
}

The TODO let me think this seems relevant but I'm not sure, what do you
think of it ?

C=E9dric


Eike Stepper wrote:

Hi C=E9dric,

That sounds interesting!
Comments inline...


C=E9dric Brun schrieb:
Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex
one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
Features)

The test is a "quick and dirty" one so please do not take these figures
too seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.

Looks like a sign that the NIO socket channels are efficient ,-)
You used localhost for TCP transport I guess?

* First case : building a small model (10000 elements) with the =
simple
MM, only adding instances in an empty database :
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.

Seems to be consistent with the fact that only for commits there =
is
remote traffic.
And the backend transaction management is another overhead.
So the impact of the number objects/commit starts with bigger
transactions.

browsing the just commited elements opening a new transaction and=20
getting
a new Resource instance, I get around 30 000 objets/sec rate just
iterating thanks to Resource.getAllContents().

Hehe, this is a sign that client side caching works well:
All views and transactions of a single session share (and cache) their
object state.
Otherwise I'd expect much smaller values.
With his ObjectivityStore implementation Simon (cc'ed) reached rates of
several thousands objects/sec.
But the DBStore (which you used) is not that optimized.

* Second case : I built a big model in the database using the =
complex
metamodel. It contains 700 000 elements. The creation went pretty nicely
(I did not measured it exactly as it was launched during a night, but it
seems I got around 100 objects/sec rate in writting as I was writting
every 700 objects.

Sounds not too bad.

Once the model is built the derby database is 222 MB (which - I =
heard -
is not so big for derby).

Hmm, I have no extensive experience with Derby.
To be honest, I'm really no database expert (although I can write simple
SQL statements).

Retrieving the distant resource and calling eAllContents() on it, =
I get=20
a
50 Objects/sec rate at best, which mean my eAllContents() call would=20
take
4 hours !

Currently I'm not sure what exactly eAllContents() does do =
(iteration
order and so).
There are several CDO features that can have a major impact on load=20
rates!

- Collections (xrefs and containment) can be loaded lazily. I don't mean
the target objects, which are lazy by default anyway, but even the
target IDs that are in the collections. We call that "reference
chunking" (but search for a better name is ongoing).
|*public interface *CDOSession *extends *CDOProtocolSession,
|IContainer<CDOView>
{
*public **int *getReferenceChunkSize();

*public **void *setReferenceChunkSize(*int *referenceChunkSize);|


- Loading collection elements (target objects) can be batched (the
current name is even worse)
|*public interface *CDOView *extends *CDOProtocolView, INotifier
{
*public **int *getLoadRevisionCollectionChunkSize();

*public **void *setLoadRevisionCollectionChunkSize(*int
*loadRevisionCollectionChunkSize);|



- One of the most sophisticated feature is automatic model usage
analyzation. The result are fetch rules that are sent to the server.
When later objects are loaded from the server, it can answer with
additional objects that are likely to be needed by the client in the
near future.
|*public class *CDOViewImpl *extends
|*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
|CDOIDProvider,
Adapter.Internal
{
*public *CDOFeatureAnalyzer getFeatureAnalyzer()
{
*return *featureAnalyzer;
}

*public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
{
*this*.featureAnalyzer =3D featureAnalyzer =3D=3D *null *?
CDOFeatureAnalyzer.NOOP : featureAnalyzer;
}|


- There's a SmartReadAheadThread (if you're curious, please ask Simon).
- Caches can be configured
- ...


Known issues of the DBStore:
- When in auditing mode updates are always written as complete inserts
(by design)
- When configuring toManyReferences to a value different from
LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
- No (few) prepared statements are used (see
https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214487)
- Maybe others...

Please note that the DBStore was not written by me to compete with
Hibernate in terms of performance or mapping capabilities.
Now that Martin and I, well in the end mostly Martin, provide a
HibernateStore and this HibernateStore gets more and more features I
expect that there will be a trend away from the DBStore.

This is probably linked to the way the mapping to db is done, =
here's the
config I used (came from CDO-server example if I recall correctly).


<repository name=3D"repo1">
<property name=3D"overrideUUID"
value=3D"1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
<property name=3D"supportingAudits" value=3D"true"/>
<property name=3D"verifyingRevisions" value=3D"false"/>
<property name=3D"currentLRUCapacity" value=3D"10000"/>

I guess this one could be larger.

<property name=3D"revisedLRUCapacity" =
value=3D"100"/>

<store type=3D"db">
<!-- type: horizontal | vertical | <any
user-contributed type> --> <mappingStrategy
type=3D"horizontal">
<!-- ONE_TABLE_PER_REFERENCE |
ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
<property name=3D"toManyReferences"
value=3D"ONE_TABLE_PER_REFERENCE"/>

<!-- LIKE_ATTRIBUTES |
LIKE_TO_MANY_REFERENCES --> <property
name=3D"toOneReferences"
value=3D"LIKE_ATTRIBUTES"/>

<!-- MODEL | STRATEGY -->
<property name=3D"mappingPrecedence"
value=3D"MODEL"/>

This one is obsolete.

</mappingStrategy>

<dbAdapter name=3D"derby-embedded"/>
<dataSource
=
class=3D"org.apache.derby.jdbc.EmbeddedDataSource"
databaseName=3D"/tmp/cdodb1"
createDatabase=3D"create"/>

</store>
</repository>




What do you think about that ? What throughput do you think I should
expect ?
As I said that's very difficult to judge.
I believe it's more appropriate to compare the actual values with values
from other measures (other setups, properties, etc. or even other
products) instead of random expectations.
Apologies that I didn't have the time so far to provide some reference
measures myself. Maybe this gets better now that the team starts to grow
slowly ;-)

What are the main reasons fur such figures ?

Hmm, I *suspect* you don't like the figures? I can find no =
evidence your
assessment ;-)

Ahh,one more note:
Tracing can also have a significant impact on performance and end-to-end
throughput.
Turn it off by all means. And please make sure that it's really turned
off, not only invisible (i.e. console is not a target).

Cheers
/Eike
=20


------=_NextPart_000_04D9_01C8C189.41CE2B50
Content-Type: text/html;
charset="ISO-8859-15"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD><TITLE></TITLE>
<META http-equiv=3DContent-Type =
content=3Dtext/html;charset=3DISO-8859-15>
<META content=3D"MSHTML 6.00.6000.16608" name=3DGENERATOR></HEAD>
<BODY text=3D#000000 bgColor=3D#ffffff>
<DIV><FONT face=3DArial size=3D2>Good point Eike,</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>Cedric,</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>How do you create your CDOSessionIMpl=20
?</FONT></DIV>
<DIV><FONT face=3DArial size=3D2>Are you using your own mechanism =
?</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>Because it could still occured in the =
case&nbsp;we=20
doesn`t go through <FONT size=3D3>CDOSessionConfigurationImpl =
and</FONT><FONT=20
size=3D2> create directly the CDOSession</FONT></FONT></DIV>
<DIV><FONT face=3DArial size=3D2><FONT =
size=3D2></FONT></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>Ex:</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2><FONT size=3D2>CDOSession session =3D=20
CDOSessionImpl();</FONT></FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>By default disablyLegacySystem is=20
false.</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>In this case Eike, Do we have something =
else to=20
avoid that ? Maybe I`m not aware of other mechanism.</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>Simon</FONT></DIV>
<DIV><FONT face=3DArial size=3D2></FONT>&nbsp;</DIV>
<DIV>"Eike Stepper" &lt;<A=20
href=3D"mailto:stepper@sympedia.de">stepper@sympedia.de</A>&gt; a =
=E9crit dans le=20
message de news: <A=20
href=3D"mailto:483ED9A4.7010302@sympedia.de">483ED9A4.7010302@sympedia.de=
</A>...</DIV>
<BLOCKQUOTE dir=3Dltr=20
style=3D"PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; =
BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px"><BR><BR>Simon=20
McDuff schrieb:=20
<BLOCKQUOTE cite=3Dmid:g1ml62$a15$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">Did you disable the following :
CDOSessionConfigurationImpl.setLegacySupportEnabled(true);

before opening your session ?

This could cause what you are observing ?

If your classes are generating with CDOObjectImpl as the super class, =
you=20
can disableLegacySupport by putting it at true.
</PRE></BLOCKQUOTE>Good point in general!<BR><BR>But I added some =
logic some=20
time ago that ensures that *session legacy support* can only be =
enabled if the=20
*legacy system* is available in general:<BR><BR>
<STYLE type=3Dtext/css>CODE {
FONT-SIZE: 10pt; MARGIN: 0px; FONT-FAMILY: Courier New, Courier
}
</STYLE>
<!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><!-- =3D Java Sourcecode to HTML automatically =
converted code =3D --><!-- =3D Java2Html Converter 5.0 [2006-02-26] by =
Markus Gebhard markus@jave.de =3D --><!-- =3D Further =
information: http://www.java2html.de =3D -->
<DIV class=3Djava align=3Dleft>
<TABLE cellSpacing=3D0 cellPadding=3D3 bgColor=3D#ffffff border=3D0>
<TBODY>
<TR><!-- start source code -->
<TD vAlign=3Dtop noWrap align=3Dleft><CODE><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>void&nbsp;</B></FONT><FONT=20
color=3D#000000>setDisableLegacyObjects</FONT><FONT=20
color=3D#000000>(</FONT><FONT=20
color=3D#7f0055><B>boolean&nbsp;</B></FONT><FONT=20
color=3D#000000>disableLegacyObjects</FONT><FONT=20
color=3D#000000>)</FONT><BR><FONT =
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#000000>{</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;&nbsp;&nbsp;</FONT ><FONT=20
color=3D#000000>checkInactive</FONT><FONT =
color=3D#000000>()</FONT><FONT=20
color=3D#000000>;</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;&nbsp;&nbsp;</FONT ><FONT=20
color=3D#7f0055><B>if&nbsp;</B></FONT><FONT =
color=3D#000000>(</FONT><FONT=20
=
color=3D#000000> !disableLegacyObjects&nbsp;&amp;&amp;&nbsp;! FSMUtil.isLeg=
acySystemAvailable</FONT><FONT=20
color=3D#000000>())</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;&nbsp;&nbsp;</FONT ><FONT=20
color=3D#000000>{</FONT><BR><FONT=20
color=3D#ffffff> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; </FONT><FONT =

color=3D#7f0055><B>throw&nbsp;new&nbsp;</B></FONT ><FONT=20
color=3D#000000>LegacySystemNotAvailableException</FONT><FONT=20
color=3D#000000>()</FONT><FONT color=3D#000000>;</FONT><BR><FONT =

color=3D#ffffff>&nbsp;&nbsp;&nbsp;&nbsp;</FONT ><FONT=20
color=3D#000000>}</FONT><BR><FONT =
color=3D#ffffff></FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;&nbsp;&nbsp;</FONT ><FONT=20
color=3D#7f0055><B>this</B></FONT><FONT=20
=
color=3D#000000> .disableLegacyObjects&nbsp;=3D&nbsp;disableLegacyObj ects;=
</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT =
color=3D#000000>}</FONT></CODE> </TD><!-- end source code =
--></TR></TBODY></TABLE></DIV><!-- =3D END of automatically =
generated HTML code =3D --><!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><BR>Notice=20
that the whole legacy system is currently (since the work on the=20
HibernateStore) broken.<BR>The needed CDOWeaver is currently not part =
of the=20
distro anymore (also due to issues with the build system).<BR><BR>This =

altogether seems to exclude the legacy system as root cause for =
performance=20
degradation.<BR><BR>Cheers<BR>/Eike<BR><BR><BR><BR>
<BLOCKQUOTE cite=3Dmid:g1ml62$a15$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">Simon


"C=E9dric Brun" <A class=3Dmoz-txt-link-rfc2396E =
href=3D"mailto:cedric.brun@obeo.fr">&lt;cedric.brun@obeo.fr&gt;</A> a =
=E9crit dans le message de news:=20
<A class=3Dmoz-txt-link-abbreviated =
href=3D"mailto:g1mbhe$2cl$1@build.eclipse.org">g1mbhe$2cl$1@build.eclipse=
..org</A>...
</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">Eike,

As I don't know (yet ;) ) all the internals of CDO I'm not sure this is
relevant, but in the testcase I'm working on (with the complex =
metamodel,
big model in database and 50 o/sec throughput) my profiler says that =
most
of the time (like 99.9%) is spent in a readString call from
CDOClassRefImpl :

public CDOClassRefImpl(ExtendedDataInput in, String defaultURI) throws
IOException
{
// TODO Optimize transfer of URIs
packageURI =3D in.readString();
if (packageURI =3D=3D null)
{
packageURI =3D defaultURI;
}

classifierID =3D in.readInt();
}

The TODO let me think this seems relevant but I'm not sure, what do you
think of it ?

C=E9dric


Eike Stepper wrote:

</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">Hi C=E9dric,

That sounds interesting!
Comments inline...


C=E9dric Brun schrieb:
</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex
one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
Features)

The test is a "quick and dirty" one so please do not take these figures
too seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.

</PRE></BLOCKQUOTE><PRE wrap=3D"">Looks like a sign that the NIO =
socket channels are efficient ,-)
You used localhost for TCP transport I guess?

</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">* First case : building =
a small model (10000 elements) with the simple
MM, only adding instances in an empty database :
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.

</PRE></BLOCKQUOTE><PRE wrap=3D"">Seems to be consistent with =
the fact that only for commits there is
remote traffic.
And the backend transaction management is another overhead.
So the impact of the number objects/commit starts with bigger
transactions.

</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">browsing the just =
commited elements opening a new transaction and=20
getting
a new Resource instance, I get around 30 000 objets/sec rate just
iterating thanks to Resource.getAllContents().

</PRE></BLOCKQUOTE><PRE wrap=3D"">Hehe, this is a sign that =
client side caching works well:
All views and transactions of a single session share (and cache) their
object state.
Otherwise I'd expect much smaller values.
With his ObjectivityStore implementation Simon (cc'ed) reached rates of
several thousands objects/sec.
But the DBStore (which you used) is not that optimized.

</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">* Second case : I built =
a big model in the database using the complex
metamodel. It contains 700 000 elements. The creation went pretty nicely
(I did not measured it exactly as it was launched during a night, but it
seems I got around 100 objects/sec rate in writting as I was writting
every 700 objects.

</PRE></BLOCKQUOTE><PRE wrap=3D"">Sounds not too bad.

</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">Once the model is built =
the derby database is 222 MB (which - I heard -
is not so big for derby).

</PRE></BLOCKQUOTE><PRE wrap=3D"">Hmm, I have no extensive =
experience with Derby.
To be honest, I'm really no database expert (although I can write simple
SQL statements).

</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">Retrieving the distant =
resource and calling eAllContents() on it, I get=20
a
50 Objects/sec rate at best, which mean my eAllContents() call would=20
take
4 hours !

</PRE></BLOCKQUOTE><PRE wrap=3D"">Currently I'm not sure what =
exactly eAllContents() does do (iteration
order and so).
There are several CDO features that can have a major impact on load=20
rates!

- Collections (xrefs and containment) can be loaded lazily. I don't mean
the target objects, which are lazy by default anyway, but even the
target IDs that are in the collections. We call that "reference
chunking" (but search for a better name is ongoing).
|*public interface *CDOSession *extends *CDOProtocolSession,
|IContainer&lt;CDOView&gt;
{
*public **int *getReferenceChunkSize();

*public **void *setReferenceChunkSize(*int *referenceChunkSize);|


- Loading collection elements (target objects) can be batched (the
current name is even worse)
|*public interface *CDOView *extends *CDOProtocolView, INotifier
{
*public **int *getLoadRevisionCollectionChunkSize();

*public **void *setLoadRevisionCollectionChunkSize(*int
*loadRevisionCollectionChunkSize);|



- One of the most sophisticated feature is automatic model usage
analyzation. The result are fetch rules that are sent to the server.
When later objects are loaded from the server, it can answer with
additional objects that are likely to be needed by the client in the
near future.
|*public class *CDOViewImpl *extends
|*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
|CDOIDProvider,
Adapter.Internal
{
*public *CDOFeatureAnalyzer getFeatureAnalyzer()
{
*return *featureAnalyzer;
}

*public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
{
*this*.featureAnalyzer =3D featureAnalyzer =3D=3D *null *?
CDOFeatureAnalyzer.NOOP : featureAnalyzer;
}|


- There's a SmartReadAheadThread (if you're curious, please ask Simon).
- Caches can be configured
- ...


Known issues of the DBStore:
- When in auditing mode updates are always written as complete inserts
(by design)
- When configuring toManyReferences to a value different from
LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
- No (few) prepared statements are used (see
<A class=3Dmoz-txt-link-freetext =
href=3D"https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214487">https://b=
ugs.eclipse.org/bugs/show_bug.cgi?id=3D214487</A>)
- Maybe others...

Please note that the DBStore was not written by me to compete with
Hibernate in terms of performance or mapping capabilities.
Now that Martin and I, well in the end mostly Martin, provide a
HibernateStore and this HibernateStore gets more and more features I
expect that there will be a trend away from the DBStore.

</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">This is probably linked =
to the way the mapping to db is done, here's the
config I used (came from CDO-server example if I recall correctly).


&lt;repository name=3D"repo1"&gt;
&lt;property name=3D"overrideUUID"
value=3D"1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/&gt;
&lt;property name=3D"supportingAudits" =
value=3D"true"/&gt;
&lt;property name=3D"verifyingRevisions" =
value=3D"false"/&gt;
&lt;property name=3D"currentLRUCapacity" =
value=3D"10000"/&gt;

</PRE></BLOCKQUOTE><PRE wrap=3D"">I guess this one could be =
larger.

</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D""> =
&lt;property name=3D"revisedLRUCapacity" value=3D"100"/&gt;

&lt;store type=3D"db"&gt;
&lt;!-- type: horizontal | vertical | &lt;any
user-contributed type&gt; --&gt; =
&lt;mappingStrategy
type=3D"horizontal"&gt;
&lt;!-- ONE_TABLE_PER_REFERENCE |
ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES =
--&gt;
&lt;property name=3D"toManyReferences"
value=3D"ONE_TABLE_PER_REFERENCE"/&gt;

&lt;!-- LIKE_ATTRIBUTES |
LIKE_TO_MANY_REFERENCES --&gt; =
&lt;property
name=3D"toOneReferences"
value=3D"LIKE_ATTRIBUTES"/&gt;

&lt;!-- MODEL | STRATEGY --&gt;
&lt;property name=3D"mappingPrecedence"
value=3D"MODEL"/&gt;

</PRE></BLOCKQUOTE><PRE wrap=3D"">This one is obsolete.

</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D""> =
&lt;/mappingStrategy&gt;

&lt;dbAdapter name=3D"derby-embedded"/&gt;
&lt;dataSource
=
class=3D"org.apache.derby.jdbc.EmbeddedDataSource"
databaseName=3D"/tmp/cdodb1"
createDatabase=3D"create"/&gt;

&lt;/store&gt;
&lt;/repository&gt;




What do you think about that ? What throughput do you think I should
expect ?
</PRE></BLOCKQUOTE><PRE wrap=3D"">As I said that's very =
difficult to judge.
I believe it's more appropriate to compare the actual values with values
from other measures (other setups, properties, etc. or even other
products) instead of random expectations.
Apologies that I didn't have the time so far to provide some reference
measures myself. Maybe this gets better now that the team starts to grow
slowly ;-)

</PRE>
<BLOCKQUOTE type=3D"cite"><PRE wrap=3D"">What are the main =
reasons fur such figures ?

</PRE></BLOCKQUOTE><PRE wrap=3D"">Hmm, I *suspect* you don't =
like the figures? I can find no evidence your
assessment ;-)

Ahh,one more note:
Tracing can also have a significant impact on performance and end-to-end
throughput.
Turn it off by all means. And please make sure that it's really turned
off, not only invisible (i.e. console is not a target).

Cheers
/Eike
</PRE></BLOCKQUOTE></BLOCKQUOTE><PRE wrap=3D""><!---->

</PRE></BLOCKQUOTE></BLOCKQUOTE></BODY></HTML>

------=_NextPart_000_04D9_01C8C189.41CE2B50--
Re: [CDO] reading/writting throughput performances [message #619072 is a reply to message #123696] Thu, 29 May 2008 16:50 Go to previous message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
This is a multi-part message in MIME format.
--------------020208050506060501030003
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 8bit

Simon McDuff schrieb:
> Good point Eike,
>
> Cedric,
>
> How do you create your CDOSessionIMpl ?
> Are you using your own mechanism ?
>
> Because it could still occured in the case we doesn`t go through
> CDOSessionConfigurationImpl and create directly the CDOSession
You could be right, too. Basically I hate myself because once I
advocated for default legacy support!!! You remember these discussions?
Since several months now I'd really like to change that (and the
checkbox in the OpenSessionDialog is early evidence). I didn't dare to
do it because formerly disableLegacyObjects was a ctor arg and a
semantic change would have broken all clients without compiler warning.
Now that there are no ctor args anymore IO really consider making this
change a late one for Ganymede. Otherwise we'd have to carry this odd
issue for another year and maybe maintain two different versions of the
semantic. What do you think?

>
> Ex:
>
> CDOSession session = CDOSessionImpl();
>
> By default disablyLegacySystem is false.
>
> In this case Eike, Do we have something else to avoid that ? Maybe I`m
> not aware of other mechanism.
I can add the same check to doBeforeActivate(), too.

Cheers
/Eike

>
> Simon
>
> "Eike Stepper" <stepper@sympedia.de <mailto:stepper@sympedia.de>> a
>


Re: [CDO] reading/writting throughput performances [message #619074 is a reply to message #123729] Thu, 29 May 2008 17:11 Go to previous message
Mark Geib is currently offline Mark GeibFriend
Messages: 432
Registered: July 2009
Senior Member
I would vote to make the change. You will be sorry if you miss this
opportunity. The users can handle the change I think.

Just my thoughts,
Mark.



Eike Stepper wrote:
> Simon McDuff schrieb:
>> Good point Eike,
>>
>> Cedric,
>>
>> How do you create your CDOSessionIMpl ?
>> Are you using your own mechanism ?
>>
>> Because it could still occured in the case we doesn`t go through
>> CDOSessionConfigurationImpl and create directly the CDOSession
> You could be right, too. Basically I hate myself because once I
> advocated for default legacy support!!! You remember these discussions?
> Since several months now I'd really like to change that (and the
> checkbox in the OpenSessionDialog is early evidence). I didn't dare to
> do it because formerly disableLegacyObjects was a ctor arg and a
> semantic change would have broken all clients without compiler warning.
> Now that there are no ctor args anymore IO really consider making this
> change a late one for Ganymede. Otherwise we'd have to carry this odd
> issue for another year and maybe maintain two different versions of the
> semantic. What do you think?
>
>>
>> Ex:
>>
>> CDOSession session = CDOSessionImpl();
>>
>> By default disablyLegacySystem is false.
>>
>> In this case Eike, Do we have something else to avoid that ? Maybe I`m
>> not aware of other mechanism.
> I can add the same check to doBeforeActivate(), too.
>
> Cheers
> /Eike
>
>>
>> Simon
>>
>> "Eike Stepper" <stepper@sympedia.de <mailto:stepper@sympedia.de>> a
>> écrit dans le message de news: 483ED9A4.7010302@sympedia.de
>> <mailto:483ED9A4.7010302@sympedia.de>...
>>
>>
>>
>> Simon McDuff schrieb:
>>> Did you disable the following :
>>> CDOSessionConfigurationImpl.setLegacySupportEnabled(true);
>>>
>>> before opening your session ?
>>>
>>> This could cause what you are observing ?
>>>
>>> If your classes are generating with CDOObjectImpl as the super class, you
>>> can disableLegacySupport by putting it at true.
>>>
>> Good point in general!
>>
>> But I added some logic some time ago that ensures that *session
>> legacy support* can only be enabled if the *legacy system* is
>> available in general:
>>
>> | *public **void *setDisableLegacyObjects(*boolean *disableLegacyObjects)
>> {
>> checkInactive();
>> *if *(!disableLegacyObjects && !FSMUtil.isLegacySystemAvailable())
>> {
>> *throw new *LegacySystemNotAvailableException();
>> }
>>
>> *this*.disableLegacyObjects = disableLegacyObjects;
>> }|
>>
>>
>> Notice that the whole legacy system is currently (since the work
>> on the HibernateStore) broken.
>> The needed CDOWeaver is currently not part of the distro anymore
>> (also due to issues with the build system).
>>
>> This altogether seems to exclude the legacy system as root cause
>> for performance degradation.
>>
>> Cheers
>> /Eike
>>
>>
>>
>>> Simon
>>>
>>>
>>> "Cédric Brun" <cedric.brun@obeo.fr> a écrit dans le message de news:
>>> g1mbhe$2cl$1@build.eclipse..org...
>>>
>>>> Eike,
>>>>
>>>> As I don't know (yet ;) ) all the internals of CDO I'm not sure this is
>>>> relevant, but in the testcase I'm working on (with the complex metamodel,
>>>> big model in database and 50 o/sec throughput) my profiler says that most
>>>> of the time (like 99.9%) is spent in a readString call from
>>>> CDOClassRefImpl :
>>>>
>>>> public CDOClassRefImpl(ExtendedDataInput in, String defaultURI) throws
>>>> IOException
>>>> {
>>>> // TODO Optimize transfer of URIs
>>>> packageURI = in.readString();
>>>> if (packageURI == null)
>>>> {
>>>> packageURI = defaultURI;
>>>> }
>>>>
>>>> classifierID = in.readInt();
>>>> }
>>>>
>>>> The TODO let me think this seems relevant but I'm not sure, what do you
>>>> think of it ?
>>>>
>>>> Cédric
>>>>
>>>>
>>>> Eike Stepper wrote:
>>>>
>>>>
>>>>> Hi Cédric,
>>>>>
>>>>> That sounds interesting!
>>>>> Comments inline...
>>>>>
>>>>>
>>>>> Cédric Brun schrieb:
>>>>>
>>>>>> Hi Eike,
>>>>>>
>>>>>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>>>>>> measuring read/write throughput when I use CDO (RC2).
>>>>>>
>>>>>> I'm using an embeded-derby database, two different metamodels (complex
>>>>>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>>>>>> Features)
>>>>>>
>>>>>> The test is a "quick and dirty" one so please do not take these figures
>>>>>> too seriously, I would just like to have your input about the "order of
>>>>>> magnitude" I should expect or settings affecting seriously these
>>>>>> performances. I tried both JVM and TCP acceptor but the performances are
>>>>>> quite similar though JVM is a bit faster.
>>>>>>
>>>>>>
>>>>> Looks like a sign that the NIO socket channels are efficient ,-)
>>>>> You used localhost for TCP transport I guess?
>>>>>
>>>>>
>>>>>> * First case : building a small model (10000 elements) with the simple
>>>>>> MM, only adding instances in an empty database :
>>>>>> commiting change on every add I get a 2 objects per second rate.
>>>>>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>>>>>
>>>>>>
>>>>> Seems to be consistent with the fact that only for commits there is
>>>>> remote traffic.
>>>>> And the backend transaction management is another overhead.
>>>>> So the impact of the number objects/commit starts with bigger
>>>>> transactions.
>>>>>
>>>>>
>>>>>> browsing the just commited elements opening a new transaction and
>>>>>> getting
>>>>>> a new Resource instance, I get around 30 000 objets/sec rate just
>>>>>> iterating thanks to Resource.getAllContents().
>>>>>>
>>>>>>
>>>>> Hehe, this is a sign that client side caching works well:
>>>>> All views and transactions of a single session share (and cache) their
>>>>> object state.
>>>>> Otherwise I'd expect much smaller values.
>>>>> With his ObjectivityStore implementation Simon (cc'ed) reached rates of
>>>>> several thousands objects/sec.
>>>>> But the DBStore (which you used) is not that optimized.
>>>>>
>>>>>
>>>>>> * Second case : I built a big model in the database using the complex
>>>>>> metamodel. It contains 700 000 elements. The creation went pretty nicely
>>>>>> (I did not measured it exactly as it was launched during a night, but it
>>>>>> seems I got around 100 objects/sec rate in writting as I was writting
>>>>>> every 700 objects.
>>>>>>
>>>>>>
>>>>> Sounds not too bad.
>>>>>
>>>>>
>>>>>> Once the model is built the derby database is 222 MB (which - I heard -
>>>>>> is not so big for derby).
>>>>>>
>>>>>>
>>>>> Hmm, I have no extensive experience with Derby.
>>>>> To be honest, I'm really no database expert (although I can write simple
>>>>> SQL statements).
>>>>>
>>>>>
>>>>>> Retrieving the distant resource and calling eAllContents() on it, I get
>>>>>> a
>>>>>> 50 Objects/sec rate at best, which mean my eAllContents() call would
>>>>>> take
>>>>>> 4 hours !
>>>>>>
>>>>>>
>>>>> Currently I'm not sure what exactly eAllContents() does do (iteration
>>>>> order and so).
>>>>> There are several CDO features that can have a major impact on load
>>>>> rates!
>>>>>
>>>>> - Collections (xrefs and containment) can be loaded lazily. I don't mean
>>>>> the target objects, which are lazy by default anyway, but even the
>>>>> target IDs that are in the collections. We call that "reference
>>>>> chunking" (but search for a better name is ongoing).
>>>>> |*public interface *CDOSession *extends *CDOProtocolSession,
>>>>> |IContainer<CDOView>
>>>>> {
>>>>> *public **int *getReferenceChunkSize();
>>>>>
>>>>> *public **void *setReferenceChunkSize(*int *referenceChunkSize);|
>>>>>
>>>>>
>>>>> - Loading collection elements (target objects) can be batched (the
>>>>> current name is even worse)
>>>>> |*public interface *CDOView *extends *CDOProtocolView, INotifier
>>>>> {
>>>>> *public **int *getLoadRevisionCollectionChunkSize();
>>>>>
>>>>> *public **void *setLoadRevisionCollectionChunkSize(*int
>>>>> *loadRevisionCollectionChunkSize);|
>>>>>
>>>>>
>>>>>
>>>>> - One of the most sophisticated feature is automatic model usage
>>>>> analyzation. The result are fetch rules that are sent to the server.
>>>>> When later objects are loaded from the server, it can answer with
>>>>> additional objects that are likely to be needed by the client in the
>>>>> near future.
>>>>> |*public class *CDOViewImpl *extends
>>>>> |*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
>>>>> |CDOIDProvider,
>>>>> Adapter.Internal
>>>>> {
>>>>> *public *CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>> {
>>>>> *return *featureAnalyzer;
>>>>> }
>>>>>
>>>>> *public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
>>>>> {
>>>>> *this*.featureAnalyzer = featureAnalyzer == *null *?
>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>> }|
>>>>>
>>>>>
>>>>> - There's a SmartReadAheadThread (if you're curious, please ask Simon).
>>>>> - Caches can be configured
>>>>> - ...
>>>>>
>>>>>
>>>>> Known issues of the DBStore:
>>>>> - When in auditing mode updates are always written as complete inserts
>>>>> (by design)
>>>>> - When configuring toManyReferences to a value different from
>>>>> LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
>>>>> - No (few) prepared statements are used (see
>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
>>>>> - Maybe others...
>>>>>
>>>>> Please note that the DBStore was not written by me to compete with
>>>>> Hibernate in terms of performance or mapping capabilities.
>>>>> Now that Martin and I, well in the end mostly Martin, provide a
>>>>> HibernateStore and this HibernateStore gets more and more features I
>>>>> expect that there will be a trend away from the DBStore.
>>>>>
>>>>>
>>>>>> This is probably linked to the way the mapping to db is done, here's the
>>>>>> config I used (came from CDO-server example if I recall correctly).
>>>>>>
>>>>>>
>>>>>> <repository name="repo1">
>>>>>> <property name="overrideUUID"
>>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>>> <property name="supportingAudits" value="true"/>
>>>>>> <property name="verifyingRevisions" value="false"/>
>>>>>> <property name="currentLRUCapacity" value="10000"/>
>>>>>>
>>>>>>
>>>>> I guess this one could be larger.
>>>>>
>>>>>
>>>>>> <property name="revisedLRUCapacity" value="100"/>
>>>>>>
>>>>>> <store type="db">
>>>>>> <!-- type: horizontal | vertical | <any
>>>>>> user-contributed type> --> <mappingStrategy
>>>>>> type="horizontal">
>>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>>> ONE_TABLE_PER_CLASS |
>>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>>>>>> <property name="toManyReferences"
>>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>>
>>>>>> <!-- LIKE_ATTRIBUTES |
>>>>>> LIKE_TO_MANY_REFERENCES --> <property
>>>>>> name="toOneReferences"
>>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>>
>>>>>> <!-- MODEL | STRATEGY -->
>>>>>> <property name="mappingPrecedence"
>>>>>> value="MODEL"/>
>>>>>>
>>>>>>
>>>>> This one is obsolete.
>>>>>
>>>>>
>>>>>> </mappingStrategy>
>>>>>>
>>>>>> <dbAdapter name="derby-embedded"/>
>>>>>> <dataSource
>>>>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>>>> databaseName="/tmp/cdodb1"
>>>>>> createDatabase="create"/>
>>>>>>
>>>>>> </store>
>>>>>> </repository>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> What do you think about that ? What throughput do you think I should
>>>>>> expect ?
>>>>>>
>>>>> As I said that's very difficult to judge.
>>>>> I believe it's more appropriate to compare the actual values with values
>>>>> from other measures (other setups, properties, etc. or even other
>>>>> products) instead of random expectations.
>>>>> Apologies that I didn't have the time so far to provide some reference
>>>>> measures myself. Maybe this gets better now that the team starts to grow
>>>>> slowly ;-)
>>>>>
>>>>>
>>>>>> What are the main reasons fur such figures ?
>>>>>>
>>>>>>
>>>>> Hmm, I *suspect* you don't like the figures? I can find no evidence your
>>>>> assessment ;-)
>>>>>
>>>>> Ahh,one more note:
>>>>> Tracing can also have a significant impact on performance and end-to-end
>>>>> throughput.
>>>>> Turn it off by all means. And please make sure that it's really turned
>>>>> off, not only invisible (i.e. console is not a target).
>>>>>
>>>>> Cheers
>>>>> /Eike
>>>>>
>>>
>>>
>>>
>>
Re: [CDO] reading/writting throughput performances [message #619075 is a reply to message #123755] Thu, 29 May 2008 17:13 Go to previous message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Mark,

Thank you to encourage me ;-)
RC3 will have the change.

Cheers
/Eike



Mark Geib schrieb:
> I would vote to make the change. You will be sorry if you miss this
> opportunity. The users can handle the change I think.
>
> Just my thoughts,
> Mark.
>
>
>
> Eike Stepper wrote:
>> Simon McDuff schrieb:
>>> Good point Eike,
>>>
>>> Cedric,
>>>
>>> How do you create your CDOSessionIMpl ?
>>> Are you using your own mechanism ?
>>>
>>> Because it could still occured in the case we doesn`t go through
>>> CDOSessionConfigurationImpl and create directly the CDOSession
>> You could be right, too. Basically I hate myself because once I
>> advocated for default legacy support!!! You remember these
>> discussions? Since several months now I'd really like to change that
>> (and the checkbox in the OpenSessionDialog is early evidence). I
>> didn't dare to do it because formerly disableLegacyObjects was a ctor
>> arg and a semantic change would have broken all clients without
>> compiler warning. Now that there are no ctor args anymore IO really
>> consider making this change a late one for Ganymede. Otherwise we'd
>> have to carry this odd issue for another year and maybe maintain two
>> different versions of the semantic. What do you think?
>>
>>>
>>> Ex:
>>>
>>> CDOSession session = CDOSessionImpl();
>>>
>>> By default disablyLegacySystem is false.
>>>
>>> In this case Eike, Do we have something else to avoid that ? Maybe
>>> I`m not aware of other mechanism.
>> I can add the same check to doBeforeActivate(), too.
>>
>> Cheers
>> /Eike
>>
>>>
>>> Simon
>>>
>>> "Eike Stepper" <stepper@sympedia.de <mailto:stepper@sympedia.de>> a
>>> écrit dans le message de news: 483ED9A4.7010302@sympedia.de
>>> <mailto:483ED9A4.7010302@sympedia.de>...
>>>
>>>
>>>
>>> Simon McDuff schrieb:
>>>> Did you disable the following :
>>>> CDOSessionConfigurationImpl.setLegacySupportEnabled(true);
>>>>
>>>> before opening your session ?
>>>>
>>>> This could cause what you are observing ?
>>>>
>>>> If your classes are generating with CDOObjectImpl as the super
>>>> class, you can disableLegacySupport by putting it at true.
>>>>
>>> Good point in general!
>>>
>>> But I added some logic some time ago that ensures that *session
>>> legacy support* can only be enabled if the *legacy system* is
>>> available in general:
>>>
>>> | *public **void *setDisableLegacyObjects(*boolean
>>> *disableLegacyObjects)
>>> {
>>> checkInactive();
>>> *if *(!disableLegacyObjects &&
>>> !FSMUtil.isLegacySystemAvailable())
>>> {
>>> *throw new *LegacySystemNotAvailableException();
>>> }
>>>
>>> *this*.disableLegacyObjects = disableLegacyObjects;
>>> }|
>>>
>>>
>>> Notice that the whole legacy system is currently (since the work
>>> on the HibernateStore) broken.
>>> The needed CDOWeaver is currently not part of the distro anymore
>>> (also due to issues with the build system).
>>>
>>> This altogether seems to exclude the legacy system as root cause
>>> for performance degradation.
>>>
>>> Cheers
>>> /Eike
>>>
>>>
>>>
>>>> Simon
>>>>
>>>>
>>>> "Cédric Brun" <cedric.brun@obeo.fr> a écrit dans le message de
>>>> news: g1mbhe$2cl$1@build.eclipse..org...
>>>>
>>>>> Eike,
>>>>>
>>>>> As I don't know (yet ;) ) all the internals of CDO I'm not
>>>>> sure this is
>>>>> relevant, but in the testcase I'm working on (with the complex
>>>>> metamodel,
>>>>> big model in database and 50 o/sec throughput) my profiler
>>>>> says that most
>>>>> of the time (like 99.9%) is spent in a readString call from
>>>>> CDOClassRefImpl :
>>>>>
>>>>> public CDOClassRefImpl(ExtendedDataInput in, String
>>>>> defaultURI) throws
>>>>> IOException
>>>>> {
>>>>> // TODO Optimize transfer of URIs
>>>>> packageURI = in.readString();
>>>>> if (packageURI == null)
>>>>> {
>>>>> packageURI = defaultURI;
>>>>> }
>>>>>
>>>>> classifierID = in.readInt();
>>>>> }
>>>>>
>>>>> The TODO let me think this seems relevant but I'm not sure,
>>>>> what do you
>>>>> think of it ?
>>>>>
>>>>> Cédric
>>>>>
>>>>>
>>>>> Eike Stepper wrote:
>>>>>
>>>>>
>>>>>> Hi Cédric,
>>>>>>
>>>>>> That sounds interesting!
>>>>>> Comments inline...
>>>>>>
>>>>>>
>>>>>> Cédric Brun schrieb:
>>>>>>
>>>>>>> Hi Eike,
>>>>>>>
>>>>>>> I've been messing around CDO lately and I hacked a few ugly
>>>>>>> unit-tests
>>>>>>> measuring read/write throughput when I use CDO (RC2).
>>>>>>>
>>>>>>> I'm using an embeded-derby database, two different
>>>>>>> metamodels (complex
>>>>>>> one - 250 EClassifiers/300 features and a simple one - 1
>>>>>>> Classifier/3
>>>>>>> Features)
>>>>>>>
>>>>>>> The test is a "quick and dirty" one so please do not take
>>>>>>> these figures
>>>>>>> too seriously, I would just like to have your input about
>>>>>>> the "order of
>>>>>>> magnitude" I should expect or settings affecting seriously
>>>>>>> these
>>>>>>> performances. I tried both JVM and TCP acceptor but the
>>>>>>> performances are
>>>>>>> quite similar though JVM is a bit faster.
>>>>>>>
>>>>>>>
>>>>>> Looks like a sign that the NIO socket channels are efficient ,-)
>>>>>> You used localhost for TCP transport I guess?
>>>>>>
>>>>>>
>>>>>>> * First case : building a small model (10000 elements) with
>>>>>>> the simple
>>>>>>> MM, only adding instances in an empty database :
>>>>>>> commiting change on every add I get a 2 objects per second
>>>>>>> rate.
>>>>>>> commiting changes every 10 adds - I get a 30 objects per
>>>>>>> second rate.
>>>>>>>
>>>>>>>
>>>>>> Seems to be consistent with the fact that only for commits
>>>>>> there is
>>>>>> remote traffic.
>>>>>> And the backend transaction management is another overhead.
>>>>>> So the impact of the number objects/commit starts with bigger
>>>>>> transactions.
>>>>>>
>>>>>>
>>>>>>> browsing the just commited elements opening a new
>>>>>>> transaction and getting
>>>>>>> a new Resource instance, I get around 30 000 objets/sec rate
>>>>>>> just
>>>>>>> iterating thanks to Resource.getAllContents().
>>>>>>>
>>>>>>>
>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>> All views and transactions of a single session share (and
>>>>>> cache) their
>>>>>> object state.
>>>>>> Otherwise I'd expect much smaller values.
>>>>>> With his ObjectivityStore implementation Simon (cc'ed)
>>>>>> reached rates of
>>>>>> several thousands objects/sec.
>>>>>> But the DBStore (which you used) is not that optimized.
>>>>>>
>>>>>>
>>>>>>> * Second case : I built a big model in the database using
>>>>>>> the complex
>>>>>>> metamodel. It contains 700 000 elements. The creation went
>>>>>>> pretty nicely
>>>>>>> (I did not measured it exactly as it was launched during a
>>>>>>> night, but it
>>>>>>> seems I got around 100 objects/sec rate in writting as I was
>>>>>>> writting
>>>>>>> every 700 objects.
>>>>>>>
>>>>>>>
>>>>>> Sounds not too bad.
>>>>>>
>>>>>>
>>>>>>> Once the model is built the derby database is 222 MB (which
>>>>>>> - I heard -
>>>>>>> is not so big for derby).
>>>>>>>
>>>>>>>
>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>> To be honest, I'm really no database expert (although I can
>>>>>> write simple
>>>>>> SQL statements).
>>>>>>
>>>>>>
>>>>>>> Retrieving the distant resource and calling eAllContents()
>>>>>>> on it, I get a
>>>>>>> 50 Objects/sec rate at best, which mean my eAllContents()
>>>>>>> call would take
>>>>>>> 4 hours !
>>>>>>>
>>>>>>>
>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>> (iteration
>>>>>> order and so).
>>>>>> There are several CDO features that can have a major impact
>>>>>> on load rates!
>>>>>>
>>>>>> - Collections (xrefs and containment) can be loaded lazily. I
>>>>>> don't mean
>>>>>> the target objects, which are lazy by default anyway, but
>>>>>> even the
>>>>>> target IDs that are in the collections. We call that "reference
>>>>>> chunking" (but search for a better name is ongoing).
>>>>>> |*public interface *CDOSession *extends *CDOProtocolSession,
>>>>>> |IContainer<CDOView>
>>>>>> {
>>>>>> *public **int *getReferenceChunkSize();
>>>>>>
>>>>>> *public **void *setReferenceChunkSize(*int
>>>>>> *referenceChunkSize);|
>>>>>>
>>>>>>
>>>>>> - Loading collection elements (target objects) can be batched
>>>>>> (the
>>>>>> current name is even worse)
>>>>>> |*public interface *CDOView *extends *CDOProtocolView, INotifier
>>>>>> {
>>>>>> *public **int *getLoadRevisionCollectionChunkSize();
>>>>>>
>>>>>> *public **void *setLoadRevisionCollectionChunkSize(*int
>>>>>> *loadRevisionCollectionChunkSize);|
>>>>>>
>>>>>>
>>>>>>
>>>>>> - One of the most sophisticated feature is automatic model usage
>>>>>> analyzation. The result are fetch rules that are sent to the
>>>>>> server.
>>>>>> When later objects are loaded from the server, it can answer
>>>>>> with
>>>>>> additional objects that are likely to be needed by the client
>>>>>> in the
>>>>>> near future.
>>>>>> |*public class *CDOViewImpl *extends
>>>>>> |*org.eclipse.net4j.internal.util.event.Notifier *implements
>>>>>> *CDOView,
>>>>>> |CDOIDProvider,
>>>>>> Adapter.Internal
>>>>>> {
>>>>>> *public *CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>>> {
>>>>>> *return *featureAnalyzer;
>>>>>> }
>>>>>>
>>>>>> *public **void *setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>>> featureAnalyzer)
>>>>>> {
>>>>>> *this*.featureAnalyzer = featureAnalyzer == *null *?
>>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>>> }|
>>>>>>
>>>>>>
>>>>>> - There's a SmartReadAheadThread (if you're curious, please
>>>>>> ask Simon).
>>>>>> - Caches can be configured
>>>>>> - ...
>>>>>>
>>>>>>
>>>>>> Known issues of the DBStore:
>>>>>> - When in auditing mode updates are always written as
>>>>>> complete inserts
>>>>>> (by design)
>>>>>> - When configuring toManyReferences to a value different from
>>>>>> LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
>>>>>> - No (few) prepared statements are used (see
>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
>>>>>> - Maybe others...
>>>>>>
>>>>>> Please note that the DBStore was not written by me to compete
>>>>>> with
>>>>>> Hibernate in terms of performance or mapping capabilities.
>>>>>> Now that Martin and I, well in the end mostly Martin, provide a
>>>>>> HibernateStore and this HibernateStore gets more and more
>>>>>> features I
>>>>>> expect that there will be a trend away from the DBStore.
>>>>>>
>>>>>>
>>>>>>> This is probably linked to the way the mapping to db is
>>>>>>> done, here's the
>>>>>>> config I used (came from CDO-server example if I recall
>>>>>>> correctly).
>>>>>>>
>>>>>>>
>>>>>>> <repository name="repo1">
>>>>>>> <property name="overrideUUID"
>>>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>>>> <property name="supportingAudits"
>>>>>>> value="true"/>
>>>>>>> <property name="verifyingRevisions"
>>>>>>> value="false"/>
>>>>>>> <property name="currentLRUCapacity"
>>>>>>> value="10000"/>
>>>>>>>
>>>>>>>
>>>>>> I guess this one could be larger.
>>>>>>
>>>>>>
>>>>>>> <property name="revisedLRUCapacity"
>>>>>>> value="100"/>
>>>>>>>
>>>>>>> <store type="db">
>>>>>>> <!-- type: horizontal | vertical | <any
>>>>>>> user-contributed type> -->
>>>>>>> <mappingStrategy
>>>>>>> type="horizontal">
>>>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>>>> ONE_TABLE_PER_CLASS |
>>>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY |
>>>>>>> LIKE_ATTRIBUTES -->
>>>>>>> <property
>>>>>>> name="toManyReferences"
>>>>>>>
>>>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>>>
>>>>>>> <!-- LIKE_ATTRIBUTES |
>>>>>>> LIKE_TO_MANY_REFERENCES -->
>>>>>>> <property
>>>>>>> name="toOneReferences"
>>>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>>>
>>>>>>> <!-- MODEL | STRATEGY -->
>>>>>>> <property
>>>>>>> name="mappingPrecedence"
>>>>>>> value="MODEL"/>
>>>>>>>
>>>>>>>
>>>>>> This one is obsolete.
>>>>>>
>>>>>>
>>>>>>> </mappingStrategy>
>>>>>>>
>>>>>>> <dbAdapter name="derby-embedded"/>
>>>>>>> <dataSource
>>>>>>>
>>>>>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>>>>> databaseName="/tmp/cdodb1"
>>>>>>> createDatabase="create"/>
>>>>>>>
>>>>>>> </store>
>>>>>>> </repository>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> What do you think about that ? What throughput do you think
>>>>>>> I should
>>>>>>> expect ?
>>>>>>>
>>>>>> As I said that's very difficult to judge.
>>>>>> I believe it's more appropriate to compare the actual values
>>>>>> with values
>>>>>> from other measures (other setups, properties, etc. or even
>>>>>> other
>>>>>> products) instead of random expectations.
>>>>>> Apologies that I didn't have the time so far to provide some
>>>>>> reference
>>>>>> measures myself. Maybe this gets better now that the team
>>>>>> starts to grow
>>>>>> slowly ;-)
>>>>>>
>>>>>>
>>>>>>> What are the main reasons fur such figures ?
>>>>>>>
>>>>>>>
>>>>>> Hmm, I *suspect* you don't like the figures? I can find no
>>>>>> evidence your
>>>>>> assessment ;-)
>>>>>>
>>>>>> Ahh,one more note:
>>>>>> Tracing can also have a significant impact on performance and
>>>>>> end-to-end
>>>>>> throughput.
>>>>>> Turn it off by all means. And please make sure that it's
>>>>>> really turned
>>>>>> off, not only invisible (i.e. console is not a target).
>>>>>>
>>>>>> Cheers
>>>>>> /Eike
>>>>>>
>>>>
>>>>
>>>>
>>>


Re: [CDO] reading/writting throughput performances [message #619076 is a reply to message #123651] Thu, 29 May 2008 17:27 Go to previous message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
This is a multi-part message in MIME format.
--------------050305010008090907040306
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Guys,

Comments inline...


Simon McDuff schrieb:
> We noticed a performance degradation in the following case:
>
> When we fetch a collection of objects that contains collections that
> we don't want to use.
> Ex:
>
> Model:
>
> ClassB
> {
> int foo1;
> List childrens1;
> List childrens2;
> List childrens3;
> };
>
> Code:
>
> Resource.contents contains objects of type ClassB.
>
> By fetching instances of classB, we will also fetched some items in
> the list(childrens1, childrens2, childrens3) The problem we had.. is
> we do not know if we want to use it. SO better not to download them.
>
> So In our production(we used our own IStore), we never download any
> objects from collection.
> I'm working at the moment to put in place all feature we used in the
> current CDO.
> That one will be part of
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477 To begin
> with,CDORevisionImpl needs to supports list that are 0 even if
> referenceChunkSize == 0.
>
> It is true that we obtain good performance.
> Worst case :
> 1500 objects /sec (Real fetch from disk. Page that contains the
> objects, wasn't in the cache of the filesystem.)
>
> Best case
> 12 000 objects/Sec (Fetch objects that are already in the case of the
> filesystem)
>
> (These statistics always fetch objects from server to client. The
> cache client isn't used)
>
> We have another optimization one with SmartReadAhead but for now only
> work for UI client. (for now) This is something we will integrate as
> well in CDO. But for now we concentrate on the other things.
>
> I used an Object database. Based on my preliminary test without CDO.
> I compare
>
> Hibernate vs Objectivity/EMF :
> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>
> The worst.. you will get 30 to 240 objects/sec or
> the best... you will get 500 to 4000 objects/sec
>
> Based on your comments... you are getting the worst (with Derby)!! :-)
And with the DBStore!!
I'd really like to encourage you to use the HibernateStore.
It is much younger than the DBStore but Hibernate itself should do a
much better job on mapping than the DBStore.

That said, the DBStore will, for multiple reasons, never cease to exist
in CDO.

>
>
> Another comments about insertion.
>
> We found another problem when we insert data(DOn't know if it is still
> true). Resource.Contents is a unique list. Basically each time we
> insert one element, it needs to fetch all the collection to see if we
> already have this element. Basically.. it is time consuming. In our
> case we are not going through this stage to persist our objects... but
> the objects are not accessible through that list.
I was not aware of this. Sounds as if we should offer an alternative
implementation for Resource.Contents.
Could you please open a Bugzilla to discuss/track this?

>
>
>
> If speed is an issue for you!! We (You included :-)) can optimize it.
> What I like is Eike did a really good job doing that framwork and
> optimizations are easy to add.
> Also you can change your back-end without affecting you client side.
> (Maybe going to an Object database!!)
> THis was one of the purpose of CDO!! :-)
Thank you ;-)
Indeed, CDO was intended to focus of the protocol aspects, the
repository and the client side addition to the EMF concepts.
I feel that this will become clearer the more IStore implementations we
get that depend on mature 3rd party code for the mapping part.

Cheers
/Eike


>
>
> Simon
>
> "Eike Stepper" <stepper@sympedia.de <mailto:stepper@sympedia.de>>
> a écrit dans le message de news: 483E9332.6050800@sympedia.de
> <mailto:483E9332.6050800@sympedia.de>...
> Hi Cédric,
>
> That sounds interesting!
> Comments inline...
>
>
> Cédric Brun schrieb:
>> Hi Eike,
>>
>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>> measuring read/write throughput when I use CDO (RC2).
>>
>> I'm using an embeded-derby database, two different metamodels (complex one -
>> 250 EClassifiers/300 features and a simple one - 1 Classifier/3 Features)
>>
>> The test is a "quick and dirty" one so please do not take these figures too
>> seriously, I would just like to have your input about the "order of
>> magnitude" I should expect or settings affecting seriously these
>> performances. I tried both JVM and TCP acceptor but the performances are
>> quite similar though JVM is a bit faster.
>>
> Looks like a sign that the NIO socket channels are efficient ,-)
> You used localhost for TCP transport I guess?
>
>> * First case : building a small model (10000 elements) with the simple MM,
>> only adding instances in an empty database :
>> commiting change on every add I get a 2 objects per second rate.
>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>
> Seems to be consistent with the fact that only for commits there
> is remote traffic.
> And the backend transaction management is another overhead.
> So the impact of the number objects/commit starts with bigger
> transactions.
>
>> browsing the just commited elements opening a new transaction and getting a
>> new Resource instance, I get around 30 000 objets/sec rate just iterating
>> thanks to Resource.getAllContents().
>>
> Hehe, this is a sign that client side caching works well:
> All views and transactions of a single session share (and cache)
> their object state.
> Otherwise I'd expect much smaller values.
> With his ObjectivityStore implementation Simon (cc'ed) reached
> rates of several thousands objects/sec.
> But the DBStore (which you used) is not that optimized.
>
>> * Second case : I built a big model in the database using the complex
>> metamodel. It contains 700 000 elements. The creation went pretty nicely (I
>> did not measured it exactly as it was launched during a night, but it seems
>> I got around 100 objects/sec rate in writting as I was writting every 700
>> objects.
>>
> Sounds not too bad.
>
>> Once the model is built the derby database is 222 MB (which - I heard - is
>> not so big for derby).
>>
> Hmm, I have no extensive experience with Derby.
> To be honest, I'm really no database expert (although I can write
> simple SQL statements).
>
>> Retrieving the distant resource and calling eAllContents() on it, I get a 50
>> Objects/sec rate at best, which mean my eAllContents() call would take 4
>> hours !
>>
> Currently I'm not sure what exactly eAllContents() does do
> (iteration order and so).
> There are several CDO features that can have a major impact on
> load rates!
>
> - Collections (xrefs and containment) can be loaded lazily. I
> don't mean the target objects, which are lazy by default anyway,
> but even the target IDs that are in the collections. We call that
> "reference chunking" (but search for a better name is ongoing).
> |*public interface *CDOSession *extends *CDOProtocolSession, IContainer<CDOView>
> {
> *public **int *getReferenceChunkSize();
>
> *public **void *setReferenceChunkSize(*int *referenceChunkSize);|
>
>
> - Loading collection elements (target objects) can be batched (the
> current name is even worse)
> |*public interface *CDOView *extends *CDOProtocolView, INotifier
> {
> *public **int *getLoadRevisionCollectionChunkSize();
>
> *public **void *setLoadRevisionCollectionChunkSize(*int *loadRevisionCollectionChunkSize);|
>
>
>
> - One of the most sophisticated feature is automatic model usage
> analyzation. The result are fetch rules that are sent to the
> server. When later objects are loaded from the server, it can
> answer with additional objects that are likely to be needed by the
> client in the near future.
> |*public class *CDOViewImpl *extends *org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView, CDOIDProvider,
> Adapter.Internal
> {
> *public *CDOFeatureAnalyzer getFeatureAnalyzer()
> {
> *return *featureAnalyzer;
> }
>
> *public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
> {
> *this*.featureAnalyzer = featureAnalyzer == *null *? CDOFeatureAnalyzer.NOOP : featureAnalyzer;
> }|
>
>
> - There's a SmartReadAheadThread (if you're curious, please ask
> Simon).
> - Caches can be configured
> - ...
>
>
> Known issues of the DBStore:
> - When in auditing mode updates are always written as complete
> inserts (by design)
> - When configuring toManyReferences to a value different from
> LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
> - No (few) prepared statements are used (see
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
> - Maybe others...
>
> Please note that the DBStore was not written by me to compete with
> Hibernate in terms of performance or mapping capabilities.
> Now that Martin and I, well in the end mostly Martin, provide a
> HibernateStore and this HibernateStore gets more and more features
> I expect that there will be a trend away from the DBStore.
>
>> This is probably linked to the way the mapping to db is done, here's the
>> config I used (came from CDO-server example if I recall correctly).
>>
>>
>> <repository name="repo1">
>> <property name="overrideUUID"
>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>> <property name="supportingAudits" value="true"/>
>> <property name="verifyingRevisions" value="false"/>
>> <property name="currentLRUCapacity" value="10000"/>
>>
> I guess this one could be larger.
>
>> <property name="revisedLRUCapacity" value="100"/>
>>
>> <store type="db">
>> <!-- type: horizontal | vertical | <any user-contributed type> -->
>> <mappingStrategy type="horizontal">
>> <!-- ONE_TABLE_PER_REFERENCE | ONE_TABLE_PER_CLASS |
>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>> <property name="toManyReferences" value="ONE_TABLE_PER_REFERENCE"/>
>>
>> <!-- LIKE_ATTRIBUTES | LIKE_TO_MANY_REFERENCES -->
>> <property name="toOneReferences" value="LIKE_ATTRIBUTES"/>
>>
>> <!-- MODEL | STRATEGY -->
>> <property name="mappingPrecedence" value="MODEL"/>
>>
> This one is obsolete.
>
>> </mappingStrategy>
>>
>> <dbAdapter name="derby-embedded"/>
>> <dataSource class="org.apache.derby.jdbc.EmbeddedDataSource"
>> databaseName="/tmp/cdodb1"
>> createDatabase="create"/>
>>
>> </store>
>> </repository>
>>
>>
>>
>>
>> What do you think about that ? What throughput do you think I should
>> expect ?
> As I said that's very difficult to judge.
> I believe it's more appropriate to compare the actual values with
> values from other measures (other setups, properties, etc. or even
> other products) instead of random expectations.
> Apologies that I didn't have the time so far to provide some
> reference measures myself. Maybe this gets better now that the
> team starts to grow slowly ;-)
>
>> What are the main reasons fur such figures ?
>>
> Hmm, I *suspect* you don't like the figures? I can find no
> evidence your assessment ;-)
>
> Ahh,one more note:
> Tracing can also have a significant impact on performance and
> end-to-end throughput.
> Turn it off by all means. And please make sure that it's really
> turned off, not only invisible (i.e. console is not a target).
>
> Cheers
> /Eike
>
>

--------------050305010008090907040306
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Guys,<br>
<br>
Comments inline...<br>
<br>
<br>
Simon McDuff schrieb:
<blockquote cite="mid:g1mkji$h5d$1@build.eclipse.org" type="cite">
<title></title>
<meta http-equiv="Content-Type" content="text/html;charset=UTF-8">
<meta content="MSHTML 6.00.6000.16608" name="GENERATOR">
<div>We noticed a performance degradation in the following case:<br>
 <br>
When we fetch a collection of objects that contains collections that we
don't want to use. <br>
Ex:<br>
 <br>
Model:<br>
 <br>
ClassB<br>
{<br>
  int foo1;<br>
  List childrens1;<br>
  List childrens2;<br>
  List childrens3;<br>
};<br>
 <br>
Code:</div>
<div><br>
Resource.contents contains objects of type ClassB.<br>
 <br>
By fetching instances of classB, we will also fetched some items in the
list(childrens1, childrens2, childrens3) <font face="Courier New"><font
color="#000000">The problem we had.. is we do not know if we want to
use it. SO better not to download them.</font></font></div>
<div><br>
<font color="#000000" face="Courier New">So In our production(we used
our own IStore), we never download any objects from collection. </font><br>
<font color="#000000" face="Courier New">I'm working at the moment to
put in place all feature we used in the current CDO.</font><br>
<font color="#000000" face="Courier New">That one will be part of </font><br>
<a moz-do-not-send="true"
href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477"
target="_blank"><font color="#0068cf">https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477</font></a>.
To begin with,CDORevisionImpl needs to supports list that are 0 even if
<font color="#000000" face="Courier New">referenceChunkSize == 0.</font><br>
 <br>
<font color="#000000" face="Courier New">It is true that we obtain
good performance.</font><br>
<font color="#000000" face="Courier New">Worst case :</font><br>
<font color="#000000" face="Courier New">1500 objects /sec (Real
fetch from disk. Page that contains the objects, wasn't in the cache of
the filesystem.) </font><br>
 <br>
<font color="#000000" face="Courier New">Best case </font><br>
<font color="#000000" face="Courier New">12 000 objects/Sec (Fetch
objects that are already in the case of the filesystem)</font><br>
 <br>
(These statistics always fetch objects from server to client. The cache
client isn't used)<br>
 <br>
<font color="#000000" face="Courier New">We have another optimization
one with SmartReadAhead but for now only work for UI client. (for now) 
This is something we will integrate as well in CDO. But for now we
concentrate on the other things.</font><br>
 <br>
<font color="#000000" face="Courier New">I used an Object database.
Based on my preliminary test without CDO.</font><br>
<font color="#000000" face="Courier New">I compare</font><br>
 <br>
<font color="#000000" face="Courier New">Hibernate vs </font><font
color="#000000" face="Courier New">Objectivity/EMF : </font><br>
<font color="#000000" face="Courier New">Objectivity/EMF is around 3
to 50 times faster than Hibernate.</font><br>
 <br>
<font color="#000000" face="Courier New">The worst.. you will get 30
to 240 objects/sec or</font><br>
<font color="#000000" face="Courier New">the best... you will get 500
to 4000 objects/sec</font><br>
 <br>
<font color="#000000" face="Courier New">Based on your comments...
you are getting the worst (with Derby)!! :-)</font><br>
</div>
</blockquote>
And with the DBStore!!<br>
I'd really like to encourage you to use the HibernateStore. <br>
It is much younger than the DBStore but Hibernate itself should do a
much better job on mapping than the DBStore.<br>
<br>
That said, the DBStore will, for multiple reasons, never cease to exist
in CDO.<br>
<br>
<blockquote cite="mid:g1mkji$h5d$1@build.eclipse.org" type="cite">
<div> <br>
 <br>
Another comments about insertion.<br>
 <br>
We found another problem when we insert data(DOn't know if it is still
true). Resource.Contents is a unique list. Basically each time we
insert one element, it needs to fetch all the collection to see if we
already have this element. Basically.. it is time consuming. In our
case we are not going through this stage to persist our objects... but
the objects are not accessible through that list.<br>
</div>
</blockquote>
I was not aware of this. Sounds as if we should offer an alternative
implementation for Resource.Contents.<br>
Could you please open a Bugzilla to discuss/track this?<br>
<br>
<blockquote cite="mid:g1mkji$h5d$1@build.eclipse.org" type="cite">
<div> </div>
<div> </div>
<div> </div>
<div><font face="Arial" size="2">If speed is an issue for you!! We
(You included :-)) can optimize it. What I like is Eike did a really
good job doing that framwork and optimizations are easy to add.</font></div>
<div><font face="Arial" size="2">Also you can change your back-end
without affecting you client side. (Maybe going to an Object database!!)</font></div>
<div><font face="Arial" size="2">THis was one of the purpose of CDO!!
:-)</font></div>
</blockquote>
Thank you ;-)<br>
Indeed, CDO was intended to focus of the protocol aspects, the
repository and the client side addition to the EMF concepts.<br>
I feel that this will become clearer the more IStore implementations we
get that depend on mature 3rd party code for the mapping part.<br>
<br>
Cheers<br>
/Eike<br>
<br>
<br>
<blockquote cite="mid:g1mkji$h5d$1@build.eclipse.org" type="cite">
<div> </div>
<div><br>
Simon<br>
<br>
</div>
<blockquote
style="border-left: 2px solid rgb(0, 0, 0); padding-right: 0px; padding-left: 5px; margin-left: 5px; margin-right: 0px;">
<div>"Eike Stepper" &lt;<a moz-do-not-send="true"
href="mailto:stepper@sympedia.de">stepper@sympedia.de</a>&gt; a écrit
dans le message de news: <a moz-do-not-send="true"
href="mailto:483E9332.6050800@sympedia.de">483E9332.6050800@sympedia.de</a>...</div>
Hi Cédric,<br>
<br>
That sounds interesting!<br>
Comments inline...<br>
<br>
<br>
Cédric Brun schrieb:
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex one -
250 EClassifiers/300 features and a simple one - 1 Classifier/3 Features)

The test is a "quick and dirty" one so please do not take these figures too
seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.
</pre>
</blockquote>
Looks like a sign that the NIO socket channels are efficient ,-)<br>
You used localhost for TCP transport I guess?<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">* First case : building a small model (10000 elements) with the simple MM,
only adding instances in an empty database :
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.
</pre>
</blockquote>
Seems to be consistent with the fact that only for commits there is
remote traffic.<br>
And the backend transaction management is another overhead.<br>
So the impact of the number objects/commit starts with bigger
transactions.<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">browsing the just commited elements opening a new transaction and getting a
new Resource instance, I get around 30 000 objets/sec rate just iterating
thanks to Resource.getAllContents().
</pre>
</blockquote>
Hehe, this is a sign that client side caching works well:<br>
All views and transactions of a single session share (and cache) their
object state.<br>
Otherwise I'd expect much smaller values.<br>
With his ObjectivityStore implementation Simon (cc'ed) reached rates of
several thousands objects/sec.<br>
But the DBStore (which you used) is not that optimized.<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">* Second case : I built a big model in the database using the complex
metamodel. It contains 700 000 elements. The creation went pretty nicely (I
did not measured it exactly as it was launched during a night, but it seems
I got around 100 objects/sec rate in writting as I was writting every 700
objects.
</pre>
</blockquote>
Sounds not too bad.<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">Once the model is built the derby database is 222 MB (which - I heard - is
not so big for derby).
</pre>
</blockquote>
Hmm, I have no extensive experience with Derby.<br>
To be honest, I'm really no database expert (although I can write
simple SQL statements).<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">Retrieving the distant resource and calling eAllContents() on it, I get a 50
Objects/sec rate at best, which mean my eAllContents() call would take 4
hours !
</pre>
</blockquote>
Currently I'm not sure what exactly eAllContents() does do (iteration
order and so).<br>
There are several CDO features that can have a major impact on load
rates!<br>
<br>
- Collections (xrefs and containment) can be loaded lazily. I don't
mean the target objects, which are lazy by default anyway, but even the
target IDs that are in the collections. We call that "reference
chunking" (but search for a better name is ongoing).<br>
<style type="text/css">CODE {
FONT-SIZE: 10pt; MARGIN: 0px; FONT-FAMILY: Courier New, Courier
}
</style><!-- ======================================================== --><!-- = Java Sourcecode to HTML automatically converted code = --><!-- = Java2Html Converter 5.0 [2006-02-26] by Markus Gebhard markus@jave.de = --><!-- = Further information: http://www.java2html.de = -->
<div class="java" align="left">
<table bgcolor="#ffffff" border="0" cellpadding="3" cellspacing="0">
<tbody>
<tr>
<!-- start source code --> <td align="left" nowrap="nowrap"
valign="top"><code><font color="#7f0055"><b>public interface </b></font><font
color="#000000">CDOSession </font><font color="#7f0055"><b>extends </b></font><font
color="#000000">CDOProtocolSession, IContainer&lt;CDOView&gt; </font><br>
<font color="#000000">{</font><br>
<font color="#ffffff">  </font><font color="#7f0055"><b>public </b></font><font
color="#7f0055"><b>int </b></font><font color="#000000">getReferenceChunkSize</font><font
color="#000000">()</font><font color="#000000">;</font><br>
<br>
<font color="#ffffff">  </font><font color="#7f0055"><b>public </b></font><font
color="#7f0055"><b>void </b></font><font color="#000000">setReferenceChunkSize</font><font
color="#000000">(</font><font color="#7f0055"><b>int </b></font><font
color="#000000">referenceChunkSize</font><font color="#000000">)</font><font
color="#000000">;</font></code> </td>
<!-- end source code --></tr>
</tbody>
</table>
</div>
<!-- = END of automatically generated HTML code = --><br>
<!-- ======================================================== -->-
Loading collection elements (target objects) can be batched (the
current name is even worse)<br>
<style type="text/css">
<!--code { font-family: Courier New, Courier; font-size: 10pt; margin: 0px; }-->
</style><!-- ======================================================== --><!-- = Java Sourcecode to HTML automatically converted code = --><!-- = Java2Html Converter 5.0 [2006-02-26] by Markus Gebhard markus@jave.de = --><!-- = Further information: http://www.java2html.de = -->
<div class="java" align="left">
<table bgcolor="#ffffff" border="0" cellpadding="3" cellspacing="0">
<tbody>
<tr>
<!-- start source code --> <td align="left" nowrap="nowrap"
valign="top"><code><font color="#7f0055"><b>public interface </b></font><font
color="#000000">CDOView </font><font color="#7f0055"><b>extends </b></font><font
color="#000000">CDOProtocolView, INotifier</font><br>
<font color="#000000">{</font><br>
<font color="#ffffff">  </font><font color="#7f0055"><b>public </b></font><font
color="#7f0055"><b>int </b></font><font color="#000000">getLoadRevisionCollectionChunkSize</font> <font
color="#000000">()</font><font color="#000000">;</font><br>
<br>
<font color="#ffffff">  </font><font color="#7f0055"><b>public </b></font><font
color="#7f0055"><b>void </b></font><font color="#000000">setLoadRevisionCollectionChunkSize</font> <font
color="#000000">(</font><font color="#7f0055"><b>int </b></font><font
color="#000000">loadRevisionCollectionChunkSize</font><font
color="#000000">)</font><font color="#000000">;</font></code> </td>
<!-- end source code --></tr>
</tbody>
</table>
</div>
<!-- = END of automatically generated HTML code = --><!-- ======================================================== --><br>
- One of the most sophisticated feature is automatic model usage
analyzation. The result are fetch rules that are sent to the server.
When later objects are loaded from the server, it can answer with
additional objects that are likely to be needed by the client in the
near future.<br>
<style type="text/css">
<!--code { font-family: Courier New, Courier; font-size: 10pt; margin: 0px; }-->
</style><!-- ======================================================== --><!-- = Java Sourcecode to HTML automatically converted code = --><!-- = Java2Html Converter 5.0 [2006-02-26] by Markus Gebhard markus@jave.de = --><!-- = Further information: http://www.java2html.de = -->
<div class="java" align="left">
<table bgcolor="#ffffff" border="0" cellpadding="3" cellspacing="0">
<tbody>
<tr>
<!-- start source code --> <td align="left" nowrap="nowrap"
valign="top"><code><font color="#7f0055"><b>public class </b></font><font
color="#000000">CDOViewImpl </font><font color="#7f0055"><b>extends </b></font><font
color="#000000">org.eclipse.net4j.internal.util.event.Notifier  </font><font
color="#7f0055"><b>implements </b></font><font color="#000000">CDOView, CDOIDProvider,</font>
<font color="#000000">Adapter.Internal</font><br>
<font color="#000000">{</font><br>
<font color="#ffffff">  </font><font color="#7f0055"><b>public </b></font><font
color="#000000">CDOFeatureAnalyzer getFeatureAnalyzer</font ><font
color="#000000">()</font><br>
<font color="#ffffff">  </font><font color="#000000">{</font><br>
<font color="#ffffff">    </font><font color="#7f0055"><b>return </b></font><font
color="#000000">featureAnalyzer;</font><br>
<font color="#ffffff">  </font><font color="#000000">}</font><br>
<br>
<font color="#ffffff">  </font><font color="#7f0055"><b>public </b></font><font
color="#7f0055"><b>void </b></font><font color="#000000">setFeatureAnalyzer</font><font
color="#000000">(</font><font color="#000000">CDOFeatureAnalyzer featureAnalyzer</font ><font
color="#000000">)</font><br>
<font color="#ffffff">  </font><font color="#000000">{</font><br>
<font color="#ffffff">    </font><font color="#7f0055"><b>this</b></font><font
color="#000000">.featureAnalyzer = featureAnalyzer ==  </font><font
color="#7f0055"><b>null </b></font><font color="#000000">? CDOFeatureAnalyzer.NOOP : featureAnalyzer; </font><br>
<font color="#ffffff">  </font><font color="#000000">}</font></code>
</td>
<!-- end source code --></tr>
</tbody>
</table>
</div>
<!-- = END of automatically generated HTML code = --><!-- ======================================================== --><br>
- There's a SmartReadAheadThread (if you're curious, please ask Simon).<br>
- Caches can be configured<br>
- ...<br>
<br>
<br>
Known issues of the DBStore:<br>
- When in auditing mode updates are always written as complete inserts
(by design)<br>
- When configuring toManyReferences to a value different from
LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.<br>
- No (few) prepared statements are used (see <a moz-do-not-send="true"
class="moz-txt-link-freetext"
href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487">https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487</a>)<br>
- Maybe others...<br>
<br>
Please note that the DBStore was not written by me to compete with
Hibernate in terms of performance or mapping capabilities.<br>
Now that Martin and I, well in the end mostly Martin, provide a
HibernateStore and this HibernateStore gets more and more features I
expect that there will be a trend away from the DBStore. <br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">This is probably linked to the way the mapping to db is done, here's the
config I used (came from CDO-server example if I recall correctly).


&lt;repository name="repo1"&gt;
&lt;property name="overrideUUID"
value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/&gt;
&lt;property name="supportingAudits" value="true"/&gt;
&lt;property name="verifyingRevisions" value="false"/&gt;
&lt;property name="currentLRUCapacity" value="10000"/&gt;
</pre>
</blockquote>
I guess this one could be larger.<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap=""> &lt;property name="revisedLRUCapacity" value="100"/&gt;

&lt;store type="db"&gt;
&lt;!-- type: horizontal | vertical | &lt;any user-contributed type&gt; --&gt;
&lt;mappingStrategy type="horizontal"&gt;
&lt;!-- ONE_TABLE_PER_REFERENCE | ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES --&gt;
&lt;property name="toManyReferences" value="ONE_TABLE_PER_REFERENCE"/&gt;

&lt;!-- LIKE_ATTRIBUTES | LIKE_TO_MANY_REFERENCES --&gt;
&lt;property name="toOneReferences" value="LIKE_ATTRIBUTES"/&gt;

&lt;!-- MODEL | STRATEGY --&gt;
&lt;property name="mappingPrecedence" value="MODEL"/&gt;
</pre>
</blockquote>
This one is obsolete.<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap=""> &lt;/mappingStrategy&gt;

&lt;dbAdapter name="derby-embedded"/&gt;
&lt;dataSource class="org.apache.derby.jdbc.EmbeddedDataSource"
databaseName="/tmp/cdodb1"
createDatabase="create"/&gt;

&lt;/store&gt;
&lt;/repository&gt;




What do you think about that ? What throughput do you think I should
expect ? </pre>
</blockquote>
As I said that's very difficult to judge.<br>
I believe it's more appropriate to compare the actual values with
values from other measures (other setups, properties, etc. or even
other products) instead of random expectations.<br>
Apologies that I didn't have the time so far to provide some reference
measures myself. Maybe this gets better now that the team starts to
grow slowly ;-)<br>
<br>
<blockquote cite="mid:g1lvjh$jd7$1@build.eclipse.org" type="cite">
<pre wrap="">What are the main reasons fur such figures ?
</pre>
</blockquote>
Hmm, I *suspect* you don't like the figures? I can find no evidence
your assessment ;-)<br>
<br>
Ahh,one more note:<br>
Tracing can also have a significant impact on performance and
end-to-end throughput.<br>
Turn it off by all means. And please make sure that it's really turned
off, not only invisible (i.e. console is not a target).<br>
<br>
Cheers<br>
/Eike<br>
<br>
<br>
</blockquote>
</blockquote>
</body>
</html>

--------------050305010008090907040306--


Re: [CDO] reading/writting throughput performances [message #619078 is a reply to message #123623] Thu, 29 May 2008 17:34 Go to previous message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
This is a multi-part message in MIME format.
--------------010307040705010407070206
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi Cedric,

Without your measure I'd say this is unlikely because there are probably
dozens of readString() calls everywhere.
And that would have hit all other users as well.

But maybe it's simply the mass of CDOClassRefImpls that are to be
transfered in your scenario.
Can you please file an enhancement request?
Then I can work on an optimization that transfers single inegers instead
of URI Strings.

Cheers
/Eike



Cédric Brun schrieb:
> Eike,
>
> As I don't know (yet ;) ) all the internals of CDO I'm not sure this is
> relevant, but in the testcase I'm working on (with the complex metamodel,
> big model in database and 50 o/sec throughput) my profiler says that most
> of the time (like 99.9%) is spent in a readString call from
> CDOClassRefImpl :
>
> public CDOClassRefImpl(ExtendedDataInput in, String defaultURI) throws
> IOException
> {
> // TODO Optimize transfer of URIs
> packageURI = in.readString();
> if (packageURI == null)
> {
> packageURI = defaultURI;
> }
>
> classifierID = in.readInt();
> }
>
> The TODO let me think this seems relevant but I'm not sure, what do you
> think of it ?
>
> Cédric
>
>
> Eike Stepper wrote:
>
>
>> Hi Cédric,
>>
>> That sounds interesting!
>> Comments inline...
>>
>>
>> Cédric Brun schrieb:
>>
>>> Hi Eike,
>>>
>>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>>> measuring read/write throughput when I use CDO (RC2).
>>>
>>> I'm using an embeded-derby database, two different metamodels (complex
>>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>>> Features)
>>>
>>> The test is a "quick and dirty" one so please do not take these figures
>>> too seriously, I would just like to have your input about the "order of
>>> magnitude" I should expect or settings affecting seriously these
>>> performances. I tried both JVM and TCP acceptor but the performances are
>>> quite similar though JVM is a bit faster.
>>>
>>>
>> Looks like a sign that the NIO socket channels are efficient ,-)
>> You used localhost for TCP transport I guess?
>>
>>
>>> * First case : building a small model (10000 elements) with the simple
>>> MM, only adding instances in an empty database :
>>> commiting change on every add I get a 2 objects per second rate.
>>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>>
>>>
>> Seems to be consistent with the fact that only for commits there is
>> remote traffic.
>> And the backend transaction management is another overhead.
>> So the impact of the number objects/commit starts with bigger
>> transactions.
>>
>>
>>> browsing the just commited elements opening a new transaction and getting
>>> a new Resource instance, I get around 30 000 objets/sec rate just
>>> iterating thanks to Resource.getAllContents().
>>>
>>>
>> Hehe, this is a sign that client side caching works well:
>> All views and transactions of a single session share (and cache) their
>> object state.
>> Otherwise I'd expect much smaller values.
>> With his ObjectivityStore implementation Simon (cc'ed) reached rates of
>> several thousands objects/sec.
>> But the DBStore (which you used) is not that optimized.
>>
>>
>>> * Second case : I built a big model in the database using the complex
>>> metamodel. It contains 700 000 elements. The creation went pretty nicely
>>> (I did not measured it exactly as it was launched during a night, but it
>>> seems I got around 100 objects/sec rate in writting as I was writting
>>> every 700 objects.
>>>
>>>
>> Sounds not too bad.
>>
>>
>>> Once the model is built the derby database is 222 MB (which - I heard -
>>> is not so big for derby).
>>>
>>>
>> Hmm, I have no extensive experience with Derby.
>> To be honest, I'm really no database expert (although I can write simple
>> SQL statements).
>>
>>
>>> Retrieving the distant resource and calling eAllContents() on it, I get a
>>> 50 Objects/sec rate at best, which mean my eAllContents() call would take
>>> 4 hours !
>>>
>>>
>> Currently I'm not sure what exactly eAllContents() does do (iteration
>> order and so).
>> There are several CDO features that can have a major impact on load rates!
>>
>> - Collections (xrefs and containment) can be loaded lazily. I don't mean
>> the target objects, which are lazy by default anyway, but even the
>> target IDs that are in the collections. We call that "reference
>> chunking" (but search for a better name is ongoing).
>> |*public interface *CDOSession *extends *CDOProtocolSession,
>> |IContainer<CDOView>
>> {
>> *public **int *getReferenceChunkSize();
>>
>> *public **void *setReferenceChunkSize(*int *referenceChunkSize);|
>>
>>
>> - Loading collection elements (target objects) can be batched (the
>> current name is even worse)
>> |*public interface *CDOView *extends *CDOProtocolView, INotifier
>> {
>> *public **int *getLoadRevisionCollectionChunkSize();
>>
>> *public **void *setLoadRevisionCollectionChunkSize(*int
>> *loadRevisionCollectionChunkSize);|
>>
>>
>>
>> - One of the most sophisticated feature is automatic model usage
>> analyzation. The result are fetch rules that are sent to the server.
>> When later objects are loaded from the server, it can answer with
>> additional objects that are likely to be needed by the client in the
>> near future.
>> |*public class *CDOViewImpl *extends
>> |*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
>> |CDOIDProvider,
>> Adapter.Internal
>> {
>> *public *CDOFeatureAnalyzer getFeatureAnalyzer()
>> {
>> *return *featureAnalyzer;
>> }
>>
>> *public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
>> {
>> *this*.featureAnalyzer = featureAnalyzer == *null *?
>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>> }|
>>
>>
>> - There's a SmartReadAheadThread (if you're curious, please ask Simon).
>> - Caches can be configured
>> - ...
>>
>>
>> Known issues of the DBStore:
>> - When in auditing mode updates are always written as complete inserts
>> (by design)
>> - When configuring toManyReferences to a value different from
>> LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
>> - No (few) prepared statements are used (see
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
>> - Maybe others...
>>
>> Please note that the DBStore was not written by me to compete with
>> Hibernate in terms of performance or mapping capabilities.
>> Now that Martin and I, well in the end mostly Martin, provide a
>> HibernateStore and this HibernateStore gets more and more features I
>> expect that there will be a trend away from the DBStore.
>>
>>
>>> This is probably linked to the way the mapping to db is done, here's the
>>> config I used (came from CDO-server example if I recall correctly).
>>>
>>>
>>> <repository name="repo1">
>>> <property name="overrideUUID"
>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>> <property name="supportingAudits" value="true"/>
>>> <property name="verifyingRevisions" value="false"/>
>>> <property name="currentLRUCapacity" value="10000"/>
>>>
>>>
>> I guess this one could be larger.
>>
>>
>>> <property name="revisedLRUCapacity" value="100"/>
>>>
>>> <store type="db">
>>> <!-- type: horizontal | vertical | <any
>>> user-contributed type> --> <mappingStrategy
>>> type="horizontal">
>>> <!-- ONE_TABLE_PER_REFERENCE |
>>> ONE_TABLE_PER_CLASS |
>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>>> <property name="toManyReferences"
>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>
>>> <!-- LIKE_ATTRIBUTES |
>>> LIKE_TO_MANY_REFERENCES --> <property
>>> name="toOneReferences"
>>> value="LIKE_ATTRIBUTES"/>
>>>
>>> <!-- MODEL | STRATEGY -->
>>> <property name="mappingPrecedence"
>>> value="MODEL"/>
>>>
>>>
>> This one is obsolete.
>>
>>
>>> </mappingStrategy>
>>>
>>> <dbAdapter name="derby-embedded"/>
>>> <dataSource
>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>> databaseName="/tmp/cdodb1"
>>> createDatabase="create"/>
>>>
>>> </store>
>>> </repository>
>>>
>>>
>>>
>>>
>>> What do you think about that ? What throughput do you think I should
>>> expect ?
>>>
>> As I said that's very difficult to judge.
>> I believe it's more appropriate to compare the actual values with values
>> from other measures (other setups, properties, etc. or even other
>> products) instead of random expectations.
>> Apologies that I didn't have the time so far to provide some reference
>> measures myself. Maybe this gets better now that the team starts to grow
>> slowly ;-)
>>
>>
>>> What are the main reasons fur such figures ?
>>>
>>>
>> Hmm, I *suspect* you don't like the figures? I can find no evidence your
>> assessment ;-)
>>
>> Ahh,one more note:
>> Tracing can also have a significant impact on performance and end-to-end
>> throughput.
>> Turn it off by all means. And please make sure that it's really turned
>> off, not only invisible (i.e. console is not a target).
>>
>> Cheers
>> /Eike
>>
>
>

--------------010307040705010407070206
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Cedric,<br>
<br>
Without your measure I'd say this is unlikely because there are
probably dozens of readString() calls everywhere.<br>
And that would have hit all other users as well.<br>
<br>
But maybe it's simply the mass of CDOClassRefImpls  that are to be
transfered in your scenario.<br>
Can you please file an enhancement request? <br>
Then I can work on an optimization that transfers single inegers
instead of URI Strings.<br>
<br>
Cheers<br>
/Eike<br>
<br>
<br>
<br>
Cédric Brun schrieb:
<blockquote cite="mid:g1mbhe$2cl$1@build.eclipse.org" type="cite">
<pre wrap="">Eike,

As I don't know (yet ;) ) all the internals of CDO I'm not sure this is
relevant, but in the testcase I'm working on (with the complex metamodel,
big model in database and 50 o/sec throughput) my profiler says that most
of the time (like 99.9%) is spent in a readString call from
CDOClassRefImpl :

public CDOClassRefImpl(ExtendedDataInput in, String defaultURI) throws
IOException
{
// TODO Optimize transfer of URIs
packageURI = in.readString();
if (packageURI == null)
{
packageURI = defaultURI;
}

classifierID = in.readInt();
}

The TODO let me think this seems relevant but I'm not sure, what do you
think of it ?

Cédric


Eike Stepper wrote:

</pre>
<blockquote type="cite">
<pre wrap="">Hi Cédric,

That sounds interesting!
Comments inline...


Cédric Brun schrieb:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex
one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
Features)

The test is a "quick and dirty" one so please do not take these figures
too seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.

</pre>
</blockquote>
<pre wrap="">Looks like a sign that the NIO socket channels are efficient ,-)
You used localhost for TCP transport I guess?

</pre>
<blockquote type="cite">
<pre wrap="">* First case : building a small model (10000 elements) with the simple
MM, only adding instances in an empty database :
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.

</pre>
</blockquote>
<pre wrap="">Seems to be consistent with the fact that only for commits there is
remote traffic.
And the backend transaction management is another overhead.
So the impact of the number objects/commit starts with bigger
transactions.

</pre>
<blockquote type="cite">
<pre wrap="">browsing the just commited elements opening a new transaction and getting
a new Resource instance, I get around 30 000 objets/sec rate just
iterating thanks to Resource.getAllContents().

</pre>
</blockquote>
<pre wrap="">Hehe, this is a sign that client side caching works well:
All views and transactions of a single session share (and cache) their
object state.
Otherwise I'd expect much smaller values.
With his ObjectivityStore implementation Simon (cc'ed) reached rates of
several thousands objects/sec.
But the DBStore (which you used) is not that optimized.

</pre>
<blockquote type="cite">
<pre wrap="">* Second case : I built a big model in the database using the complex
metamodel. It contains 700 000 elements. The creation went pretty nicely
(I did not measured it exactly as it was launched during a night, but it
seems I got around 100 objects/sec rate in writting as I was writting
every 700 objects.

</pre>
</blockquote>
<pre wrap="">Sounds not too bad.

</pre>
<blockquote type="cite">
<pre wrap="">Once the model is built the derby database is 222 MB (which - I heard -
is not so big for derby).

</pre>
</blockquote>
<pre wrap="">Hmm, I have no extensive experience with Derby.
To be honest, I'm really no database expert (although I can write simple
SQL statements).

</pre>
<blockquote type="cite">
<pre wrap="">Retrieving the distant resource and calling eAllContents() on it, I get a
50 Objects/sec rate at best, which mean my eAllContents() call would take
4 hours !

</pre>
</blockquote>
<pre wrap="">Currently I'm not sure what exactly eAllContents() does do (iteration
order and so).
There are several CDO features that can have a major impact on load rates!

- Collections (xrefs and containment) can be loaded lazily. I don't mean
the target objects, which are lazy by default anyway, but even the
target IDs that are in the collections. We call that "reference
chunking" (but search for a better name is ongoing).
|*public interface *CDOSession *extends *CDOProtocolSession,
|IContainer&lt;CDOView&gt;
{
*public **int *getReferenceChunkSize();

*public **void *setReferenceChunkSize(*int *referenceChunkSize);|


- Loading collection elements (target objects) can be batched (the
current name is even worse)
|*public interface *CDOView *extends *CDOProtocolView, INotifier
{
*public **int *getLoadRevisionCollectionChunkSize();

*public **void *setLoadRevisionCollectionChunkSize(*int
*loadRevisionCollectionChunkSize);|



- One of the most sophisticated feature is automatic model usage
analyzation. The result are fetch rules that are sent to the server.
When later objects are loaded from the server, it can answer with
additional objects that are likely to be needed by the client in the
near future.
|*public class *CDOViewImpl *extends
|*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
|CDOIDProvider,
Adapter.Internal
{
*public *CDOFeatureAnalyzer getFeatureAnalyzer()
{
*return *featureAnalyzer;
}

*public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
{
*this*.featureAnalyzer = featureAnalyzer == *null *?
CDOFeatureAnalyzer.NOOP : featureAnalyzer;
}|


- There's a SmartReadAheadThread (if you're curious, please ask Simon).
- Caches can be configured
- ...


Known issues of the DBStore:
- When in auditing mode updates are always written as complete inserts
(by design)
- When configuring toManyReferences to a value different from
LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
- No (few) prepared statements are used (see
<a class="moz-txt-link-freetext" href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487">https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487</a>)
- Maybe others...

Please note that the DBStore was not written by me to compete with
Hibernate in terms of performance or mapping capabilities.
Now that Martin and I, well in the end mostly Martin, provide a
HibernateStore and this HibernateStore gets more and more features I
expect that there will be a trend away from the DBStore.

</pre>
<blockquote type="cite">
<pre wrap="">This is probably linked to the way the mapping to db is done, here's the
config I used (came from CDO-server example if I recall correctly).


&lt;repository name="repo1"&gt;
&lt;property name="overrideUUID"
value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/&gt;
&lt;property name="supportingAudits" value="true"/&gt;
&lt;property name="verifyingRevisions" value="false"/&gt;
&lt;property name="currentLRUCapacity" value="10000"/&gt;

</pre>
</blockquote>
<pre wrap="">I guess this one could be larger.

</pre>
<blockquote type="cite">
<pre wrap=""> &lt;property name="revisedLRUCapacity" value="100"/&gt;

&lt;store type="db"&gt;
&lt;!-- type: horizontal | vertical | &lt;any
user-contributed type&gt; --&gt; &lt;mappingStrategy
type="horizontal"&gt;
&lt;!-- ONE_TABLE_PER_REFERENCE |
ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES --&gt;
&lt;property name="toManyReferences"
value="ONE_TABLE_PER_REFERENCE"/&gt;

&lt;!-- LIKE_ATTRIBUTES |
LIKE_TO_MANY_REFERENCES --&gt; &lt;property
name="toOneReferences"
value="LIKE_ATTRIBUTES"/&gt;

&lt;!-- MODEL | STRATEGY --&gt;
&lt;property name="mappingPrecedence"
value="MODEL"/&gt;

</pre>
</blockquote>
<pre wrap="">This one is obsolete.

</pre>
<blockquote type="cite">
<pre wrap=""> &lt;/mappingStrategy&gt;

&lt;dbAdapter name="derby-embedded"/&gt;
&lt;dataSource
class="org.apache.derby.jdbc.EmbeddedDataSource"
databaseName="/tmp/cdodb1"
createDatabase="create"/&gt;

&lt;/store&gt;
&lt;/repository&gt;




What do you think about that ? What throughput do you think I should
expect ?
</pre>
</blockquote>
<pre wrap="">As I said that's very difficult to judge.
I believe it's more appropriate to compare the actual values with values
from other measures (other setups, properties, etc. or even other
products) instead of random expectations.
Apologies that I didn't have the time so far to provide some reference
measures myself. Maybe this gets better now that the team starts to grow
slowly ;-)

</pre>
<blockquote type="cite">
<pre wrap="">What are the main reasons fur such figures ?

</pre>
</blockquote>
<pre wrap="">Hmm, I *suspect* you don't like the figures? I can find no evidence your
assessment ;-)

Ahh,one more note:
Tracing can also have a significant impact on performance and end-to-end
throughput.
Turn it off by all means. And please make sure that it's really turned
off, not only invisible (i.e. console is not a target).

Cheers
/Eike
</pre>
</blockquote>
<pre wrap=""><!---->
</pre>
</blockquote>
</body>
</html>

--------------010307040705010407070206--


Re: [CDO] reading/writting throughput performances [message #619079 is a reply to message #123610] Thu, 29 May 2008 17:42 Go to previous message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
This is a multi-part message in MIME format.
--------------030500070800010205060903
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Cédric Brun schrieb:
> Eike,
>
> thanks for your detailled answer, I'll digg further and try to profile the
> Net4J/CDO stack so that I understand the "50 objects/sec rate" when
> retrieving data (which I think is surprisingly low).
>
Looking forward to new insights!

> Concerning the logging, what can I change to be *sure* that no logging will
> occurs ?
>
In Net4j/CDO there's a basic difference between logging and tracing.
Logging is for important production information (I always advise against
using the DEBUG loglevel!) and can't be switched off.

Tracing can be configured in several dimensions, per bundle, per tracer
hierachy, per trace context (usually == classname).
If you launch an OSGi Framework or Eclipse Application you can control
tracing via the Tracing page of the launch config.
In stand-alone mode call OMPlatform.INSTANCE.setDebugging(false).

Simply removing all ITraceHandlers is not enough.
If you set a breakpoint in
org.eclipse.net4j.internal.util.bundle.AbstractPlatform.trac e(OMTraceHandlerEvent)
you can be absolutely sure.

Cheers
/Eike


> Cheers,
>
> Cédric
>
>
>
> Eike Stepper wrote:
>
>
>> Hi Cédric,
>>
>> That sounds interesting!
>> Comments inline...
>>
>>
>> Cédric Brun schrieb:
>>
>>> Hi Eike,
>>>
>>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>>> measuring read/write throughput when I use CDO (RC2).
>>>
>>> I'm using an embeded-derby database, two different metamodels (complex
>>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>>> Features)
>>>
>>> The test is a "quick and dirty" one so please do not take these figures
>>> too seriously, I would just like to have your input about the "order of
>>> magnitude" I should expect or settings affecting seriously these
>>> performances. I tried both JVM and TCP acceptor but the performances are
>>> quite similar though JVM is a bit faster.
>>>
>>>
>> Looks like a sign that the NIO socket channels are efficient ,-)
>> You used localhost for TCP transport I guess?
>>
>>
>>> * First case : building a small model (10000 elements) with the simple
>>> MM, only adding instances in an empty database :
>>> commiting change on every add I get a 2 objects per second rate.
>>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>>
>>>
>> Seems to be consistent with the fact that only for commits there is
>> remote traffic.
>> And the backend transaction management is another overhead.
>> So the impact of the number objects/commit starts with bigger
>> transactions.
>>
>>
>>> browsing the just commited elements opening a new transaction and getting
>>> a new Resource instance, I get around 30 000 objets/sec rate just
>>> iterating thanks to Resource.getAllContents().
>>>
>>>
>> Hehe, this is a sign that client side caching works well:
>> All views and transactions of a single session share (and cache) their
>> object state.
>> Otherwise I'd expect much smaller values.
>> With his ObjectivityStore implementation Simon (cc'ed) reached rates of
>> several thousands objects/sec.
>> But the DBStore (which you used) is not that optimized.
>>
>>
>>> * Second case : I built a big model in the database using the complex
>>> metamodel. It contains 700 000 elements. The creation went pretty nicely
>>> (I did not measured it exactly as it was launched during a night, but it
>>> seems I got around 100 objects/sec rate in writting as I was writting
>>> every 700 objects.
>>>
>>>
>> Sounds not too bad.
>>
>>
>>> Once the model is built the derby database is 222 MB (which - I heard -
>>> is not so big for derby).
>>>
>>>
>> Hmm, I have no extensive experience with Derby.
>> To be honest, I'm really no database expert (although I can write simple
>> SQL statements).
>>
>>
>>> Retrieving the distant resource and calling eAllContents() on it, I get a
>>> 50 Objects/sec rate at best, which mean my eAllContents() call would take
>>> 4 hours !
>>>
>>>
>> Currently I'm not sure what exactly eAllContents() does do (iteration
>> order and so).
>> There are several CDO features that can have a major impact on load rates!
>>
>> - Collections (xrefs and containment) can be loaded lazily. I don't mean
>> the target objects, which are lazy by default anyway, but even the
>> target IDs that are in the collections. We call that "reference
>> chunking" (but search for a better name is ongoing).
>> |*public interface *CDOSession *extends *CDOProtocolSession,
>> |IContainer<CDOView>
>> {
>> *public **int *getReferenceChunkSize();
>>
>> *public **void *setReferenceChunkSize(*int *referenceChunkSize);|
>>
>>
>> - Loading collection elements (target objects) can be batched (the
>> current name is even worse)
>> |*public interface *CDOView *extends *CDOProtocolView, INotifier
>> {
>> *public **int *getLoadRevisionCollectionChunkSize();
>>
>> *public **void *setLoadRevisionCollectionChunkSize(*int
>> *loadRevisionCollectionChunkSize);|
>>
>>
>>
>> - One of the most sophisticated feature is automatic model usage
>> analyzation. The result are fetch rules that are sent to the server.
>> When later objects are loaded from the server, it can answer with
>> additional objects that are likely to be needed by the client in the
>> near future.
>> |*public class *CDOViewImpl *extends
>> |*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
>> |CDOIDProvider,
>> Adapter.Internal
>> {
>> *public *CDOFeatureAnalyzer getFeatureAnalyzer()
>> {
>> *return *featureAnalyzer;
>> }
>>
>> *public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
>> {
>> *this*.featureAnalyzer = featureAnalyzer == *null *?
>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>> }|
>>
>>
>> - There's a SmartReadAheadThread (if you're curious, please ask Simon).
>> - Caches can be configured
>> - ...
>>
>>
>> Known issues of the DBStore:
>> - When in auditing mode updates are always written as complete inserts
>> (by design)
>> - When configuring toManyReferences to a value different from
>> LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
>> - No (few) prepared statements are used (see
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
>> - Maybe others...
>>
>> Please note that the DBStore was not written by me to compete with
>> Hibernate in terms of performance or mapping capabilities.
>> Now that Martin and I, well in the end mostly Martin, provide a
>> HibernateStore and this HibernateStore gets more and more features I
>> expect that there will be a trend away from the DBStore.
>>
>>
>>> This is probably linked to the way the mapping to db is done, here's the
>>> config I used (came from CDO-server example if I recall correctly).
>>>
>>>
>>> <repository name="repo1">
>>> <property name="overrideUUID"
>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>> <property name="supportingAudits" value="true"/>
>>> <property name="verifyingRevisions" value="false"/>
>>> <property name="currentLRUCapacity" value="10000"/>
>>>
>>>
>> I guess this one could be larger.
>>
>>
>>> <property name="revisedLRUCapacity" value="100"/>
>>>
>>> <store type="db">
>>> <!-- type: horizontal | vertical | <any
>>> user-contributed type> --> <mappingStrategy
>>> type="horizontal">
>>> <!-- ONE_TABLE_PER_REFERENCE |
>>> ONE_TABLE_PER_CLASS |
>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>>> <property name="toManyReferences"
>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>
>>> <!-- LIKE_ATTRIBUTES |
>>> LIKE_TO_MANY_REFERENCES --> <property
>>> name="toOneReferences"
>>> value="LIKE_ATTRIBUTES"/>
>>>
>>> <!-- MODEL | STRATEGY -->
>>> <property name="mappingPrecedence"
>>> value="MODEL"/>
>>>
>>>
>> This one is obsolete.
>>
>>
>>> </mappingStrategy>
>>>
>>> <dbAdapter name="derby-embedded"/>
>>> <dataSource
>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>> databaseName="/tmp/cdodb1"
>>> createDatabase="create"/>
>>>
>>> </store>
>>> </repository>
>>>
>>>
>>>
>>>
>>> What do you think about that ? What throughput do you think I should
>>> expect ?
>>>
>> As I said that's very difficult to judge.
>> I believe it's more appropriate to compare the actual values with values
>> from other measures (other setups, properties, etc. or even other
>> products) instead of random expectations.
>> Apologies that I didn't have the time so far to provide some reference
>> measures myself. Maybe this gets better now that the team starts to grow
>> slowly ;-)
>>
>>
>>> What are the main reasons fur such figures ?
>>>
>>>
>> Hmm, I *suspect* you don't like the figures? I can find no evidence your
>> assessment ;-)
>>
>> Ahh,one more note:
>> Tracing can also have a significant impact on performance and end-to-end
>> throughput.
>> Turn it off by all means. And please make sure that it's really turned
>> off, not only invisible (i.e. console is not a target).
>>
>> Cheers
>> /Eike
>>
>
>

--------------030500070800010205060903
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Cédric Brun schrieb:
<blockquote cite="mid:g1m91c$i2q$1@build.eclipse.org" type="cite">
<pre wrap="">Eike,

thanks for your detailled answer, I'll digg further and try to profile the
Net4J/CDO stack so that I understand the "50 objects/sec rate" when
retrieving data (which I think is surprisingly low).
</pre>
</blockquote>
Looking forward to new insights!<br>
<br>
<blockquote cite="mid:g1m91c$i2q$1@build.eclipse.org" type="cite">
<pre wrap="">
Concerning the logging, what can I change to be *sure* that no logging will
occurs ?
</pre>
</blockquote>
In Net4j/CDO there's a basic difference between logging and tracing.<br>
Logging is for important production information (I always advise
against using the DEBUG loglevel!) and can't be switched off.<br>
<br>
Tracing can be configured in several dimensions, per bundle, per tracer
hierachy, per trace context (usually == classname).<br>
If you launch an OSGi Framework or Eclipse Application you can control
tracing via the Tracing page of the launch config.<br>
In stand-alone mode call OMPlatform.INSTANCE.setDebugging(false).<br>
<br>
Simply removing all ITraceHandlers is not enough.<br>
If you set a breakpoint in
org.eclipse.net4j.internal.util.bundle.AbstractPlatform.trac e(OMTraceHandlerEvent)
you can be absolutely sure.<br>
<br>
Cheers<br>
/Eike<br>
<br>
<br>
<blockquote cite="mid:g1m91c$i2q$1@build.eclipse.org" type="cite">
<pre wrap="">
Cheers,

Cédric



Eike Stepper wrote:

</pre>
<blockquote type="cite">
<pre wrap="">Hi Cédric,

That sounds interesting!
Comments inline...


Cédric Brun schrieb:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex
one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
Features)

The test is a "quick and dirty" one so please do not take these figures
too seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.

</pre>
</blockquote>
<pre wrap="">Looks like a sign that the NIO socket channels are efficient ,-)
You used localhost for TCP transport I guess?

</pre>
<blockquote type="cite">
<pre wrap="">* First case : building a small model (10000 elements) with the simple
MM, only adding instances in an empty database :
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.

</pre>
</blockquote>
<pre wrap="">Seems to be consistent with the fact that only for commits there is
remote traffic.
And the backend transaction management is another overhead.
So the impact of the number objects/commit starts with bigger
transactions.

</pre>
<blockquote type="cite">
<pre wrap="">browsing the just commited elements opening a new transaction and getting
a new Resource instance, I get around 30 000 objets/sec rate just
iterating thanks to Resource.getAllContents().

</pre>
</blockquote>
<pre wrap="">Hehe, this is a sign that client side caching works well:
All views and transactions of a single session share (and cache) their
object state.
Otherwise I'd expect much smaller values.
With his ObjectivityStore implementation Simon (cc'ed) reached rates of
several thousands objects/sec.
But the DBStore (which you used) is not that optimized.

</pre>
<blockquote type="cite">
<pre wrap="">* Second case : I built a big model in the database using the complex
metamodel. It contains 700 000 elements. The creation went pretty nicely
(I did not measured it exactly as it was launched during a night, but it
seems I got around 100 objects/sec rate in writting as I was writting
every 700 objects.

</pre>
</blockquote>
<pre wrap="">Sounds not too bad.

</pre>
<blockquote type="cite">
<pre wrap="">Once the model is built the derby database is 222 MB (which - I heard -
is not so big for derby).

</pre>
</blockquote>
<pre wrap="">Hmm, I have no extensive experience with Derby.
To be honest, I'm really no database expert (although I can write simple
SQL statements).

</pre>
<blockquote type="cite">
<pre wrap="">Retrieving the distant resource and calling eAllContents() on it, I get a
50 Objects/sec rate at best, which mean my eAllContents() call would take
4 hours !

</pre>
</blockquote>
<pre wrap="">Currently I'm not sure what exactly eAllContents() does do (iteration
order and so).
There are several CDO features that can have a major impact on load rates!

- Collections (xrefs and containment) can be loaded lazily. I don't mean
the target objects, which are lazy by default anyway, but even the
target IDs that are in the collections. We call that "reference
chunking" (but search for a better name is ongoing).
|*public interface *CDOSession *extends *CDOProtocolSession,
|IContainer&lt;CDOView&gt;
{
*public **int *getReferenceChunkSize();

*public **void *setReferenceChunkSize(*int *referenceChunkSize);|


- Loading collection elements (target objects) can be batched (the
current name is even worse)
|*public interface *CDOView *extends *CDOProtocolView, INotifier
{
*public **int *getLoadRevisionCollectionChunkSize();

*public **void *setLoadRevisionCollectionChunkSize(*int
*loadRevisionCollectionChunkSize);|



- One of the most sophisticated feature is automatic model usage
analyzation. The result are fetch rules that are sent to the server.
When later objects are loaded from the server, it can answer with
additional objects that are likely to be needed by the client in the
near future.
|*public class *CDOViewImpl *extends
|*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
|CDOIDProvider,
Adapter.Internal
{
*public *CDOFeatureAnalyzer getFeatureAnalyzer()
{
*return *featureAnalyzer;
}

*public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
{
*this*.featureAnalyzer = featureAnalyzer == *null *?
CDOFeatureAnalyzer.NOOP : featureAnalyzer;
}|


- There's a SmartReadAheadThread (if you're curious, please ask Simon).
- Caches can be configured
- ...


Known issues of the DBStore:
- When in auditing mode updates are always written as complete inserts
(by design)
- When configuring toManyReferences to a value different from
LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
- No (few) prepared statements are used (see
<a class="moz-txt-link-freetext" href="https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487">https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487</a>)
- Maybe others...

Please note that the DBStore was not written by me to compete with
Hibernate in terms of performance or mapping capabilities.
Now that Martin and I, well in the end mostly Martin, provide a
HibernateStore and this HibernateStore gets more and more features I
expect that there will be a trend away from the DBStore.

</pre>
<blockquote type="cite">
<pre wrap="">This is probably linked to the way the mapping to db is done, here's the
config I used (came from CDO-server example if I recall correctly).


&lt;repository name="repo1"&gt;
&lt;property name="overrideUUID"
value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/&gt;
&lt;property name="supportingAudits" value="true"/&gt;
&lt;property name="verifyingRevisions" value="false"/&gt;
&lt;property name="currentLRUCapacity" value="10000"/&gt;

</pre>
</blockquote>
<pre wrap="">I guess this one could be larger.

</pre>
<blockquote type="cite">
<pre wrap=""> &lt;property name="revisedLRUCapacity" value="100"/&gt;

&lt;store type="db"&gt;
&lt;!-- type: horizontal | vertical | &lt;any
user-contributed type&gt; --&gt; &lt;mappingStrategy
type="horizontal"&gt;
&lt;!-- ONE_TABLE_PER_REFERENCE |
ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES --&gt;
&lt;property name="toManyReferences"
value="ONE_TABLE_PER_REFERENCE"/&gt;

&lt;!-- LIKE_ATTRIBUTES |
LIKE_TO_MANY_REFERENCES --&gt; &lt;property
name="toOneReferences"
value="LIKE_ATTRIBUTES"/&gt;

&lt;!-- MODEL | STRATEGY --&gt;
&lt;property name="mappingPrecedence"
value="MODEL"/&gt;

</pre>
</blockquote>
<pre wrap="">This one is obsolete.

</pre>
<blockquote type="cite">
<pre wrap=""> &lt;/mappingStrategy&gt;

&lt;dbAdapter name="derby-embedded"/&gt;
&lt;dataSource
class="org.apache.derby.jdbc.EmbeddedDataSource"
databaseName="/tmp/cdodb1"
createDatabase="create"/&gt;

&lt;/store&gt;
&lt;/repository&gt;




What do you think about that ? What throughput do you think I should
expect ?
</pre>
</blockquote>
<pre wrap="">As I said that's very difficult to judge.
I believe it's more appropriate to compare the actual values with values
from other measures (other setups, properties, etc. or even other
products) instead of random expectations.
Apologies that I didn't have the time so far to provide some reference
measures myself. Maybe this gets better now that the team starts to grow
slowly ;-)

</pre>
<blockquote type="cite">
<pre wrap="">What are the main reasons fur such figures ?

</pre>
</blockquote>
<pre wrap="">Hmm, I *suspect* you don't like the figures? I can find no evidence your
assessment ;-)

Ahh,one more note:
Tracing can also have a significant impact on performance and end-to-end
throughput.
Turn it off by all means. And please make sure that it's really turned
off, not only invisible (i.e. console is not a target).

Cheers
/Eike
</pre>
</blockquote>
<pre wrap=""><!---->
</pre>
</blockquote>
</body>
</html>

--------------030500070800010205060903--


Re: [CDO] reading/writting throughput performances [message #619081 is a reply to message #123770] Thu, 29 May 2008 19:00 Go to previous message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Eike Stepper schrieb:
> Mark,
>
> Thank you to encourage me ;-)
> RC3 will have the change.
Please cc yourself if you want:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=234702

>
> Cheers
> /Eike
>
>
>
> Mark Geib schrieb:
>> I would vote to make the change. You will be sorry if you miss this
>> opportunity. The users can handle the change I think.
>>
>> Just my thoughts,
>> Mark.
>>
>>
>>
>> Eike Stepper wrote:
>>> Simon McDuff schrieb:
>>>> Good point Eike,
>>>>
>>>> Cedric,
>>>>
>>>> How do you create your CDOSessionIMpl ?
>>>> Are you using your own mechanism ?
>>>>
>>>> Because it could still occured in the case we doesn`t go through
>>>> CDOSessionConfigurationImpl and create directly the CDOSession
>>> You could be right, too. Basically I hate myself because once I
>>> advocated for default legacy support!!! You remember these
>>> discussions? Since several months now I'd really like to change that
>>> (and the checkbox in the OpenSessionDialog is early evidence). I
>>> didn't dare to do it because formerly disableLegacyObjects was a
>>> ctor arg and a semantic change would have broken all clients without
>>> compiler warning. Now that there are no ctor args anymore IO really
>>> consider making this change a late one for Ganymede. Otherwise we'd
>>> have to carry this odd issue for another year and maybe maintain two
>>> different versions of the semantic. What do you think?
>>>
>>>>
>>>> Ex:
>>>>
>>>> CDOSession session = CDOSessionImpl();
>>>>
>>>> By default disablyLegacySystem is false.
>>>>
>>>> In this case Eike, Do we have something else to avoid that ? Maybe
>>>> I`m not aware of other mechanism.
>>> I can add the same check to doBeforeActivate(), too.
>>>
>>> Cheers
>>> /Eike
>>>
>>>>
>>>> Simon
>>>>
>>>> "Eike Stepper" <stepper@sympedia.de <mailto:stepper@sympedia.de>> a
>>>> écrit dans le message de news: 483ED9A4.7010302@sympedia.de
>>>> <mailto:483ED9A4.7010302@sympedia.de>...
>>>>
>>>>
>>>>
>>>> Simon McDuff schrieb:
>>>>> Did you disable the following :
>>>>> CDOSessionConfigurationImpl.setLegacySupportEnabled(true);
>>>>>
>>>>> before opening your session ?
>>>>>
>>>>> This could cause what you are observing ?
>>>>>
>>>>> If your classes are generating with CDOObjectImpl as the super
>>>>> class, you can disableLegacySupport by putting it at true.
>>>>>
>>>> Good point in general!
>>>>
>>>> But I added some logic some time ago that ensures that *session
>>>> legacy support* can only be enabled if the *legacy system* is
>>>> available in general:
>>>>
>>>> | *public **void *setDisableLegacyObjects(*boolean
>>>> *disableLegacyObjects)
>>>> {
>>>> checkInactive();
>>>> *if *(!disableLegacyObjects &&
>>>> !FSMUtil.isLegacySystemAvailable())
>>>> {
>>>> *throw new *LegacySystemNotAvailableException();
>>>> }
>>>>
>>>> *this*.disableLegacyObjects = disableLegacyObjects;
>>>> }|
>>>>
>>>>
>>>> Notice that the whole legacy system is currently (since the work
>>>> on the HibernateStore) broken.
>>>> The needed CDOWeaver is currently not part of the distro anymore
>>>> (also due to issues with the build system).
>>>>
>>>> This altogether seems to exclude the legacy system as root cause
>>>> for performance degradation.
>>>>
>>>> Cheers
>>>> /Eike
>>>>
>>>>
>>>>
>>>>> Simon
>>>>>
>>>>>
>>>>> "Cédric Brun" <cedric.brun@obeo.fr> a écrit dans le message de
>>>>> news: g1mbhe$2cl$1@build.eclipse..org...
>>>>>
>>>>>> Eike,
>>>>>>
>>>>>> As I don't know (yet ;) ) all the internals of CDO I'm not
>>>>>> sure this is
>>>>>> relevant, but in the testcase I'm working on (with the
>>>>>> complex metamodel,
>>>>>> big model in database and 50 o/sec throughput) my profiler
>>>>>> says that most
>>>>>> of the time (like 99.9%) is spent in a readString call from
>>>>>> CDOClassRefImpl :
>>>>>>
>>>>>> public CDOClassRefImpl(ExtendedDataInput in, String
>>>>>> defaultURI) throws
>>>>>> IOException
>>>>>> {
>>>>>> // TODO Optimize transfer of URIs
>>>>>> packageURI = in.readString();
>>>>>> if (packageURI == null)
>>>>>> {
>>>>>> packageURI = defaultURI;
>>>>>> }
>>>>>>
>>>>>> classifierID = in.readInt();
>>>>>> }
>>>>>>
>>>>>> The TODO let me think this seems relevant but I'm not sure,
>>>>>> what do you
>>>>>> think of it ?
>>>>>>
>>>>>> Cédric
>>>>>>
>>>>>>
>>>>>> Eike Stepper wrote:
>>>>>>
>>>>>>
>>>>>>> Hi Cédric,
>>>>>>>
>>>>>>> That sounds interesting!
>>>>>>> Comments inline...
>>>>>>>
>>>>>>>
>>>>>>> Cédric Brun schrieb:
>>>>>>>
>>>>>>>> Hi Eike,
>>>>>>>>
>>>>>>>> I've been messing around CDO lately and I hacked a few ugly
>>>>>>>> unit-tests
>>>>>>>> measuring read/write throughput when I use CDO (RC2).
>>>>>>>>
>>>>>>>> I'm using an embeded-derby database, two different
>>>>>>>> metamodels (complex
>>>>>>>> one - 250 EClassifiers/300 features and a simple one - 1
>>>>>>>> Classifier/3
>>>>>>>> Features)
>>>>>>>>
>>>>>>>> The test is a "quick and dirty" one so please do not take
>>>>>>>> these figures
>>>>>>>> too seriously, I would just like to have your input about
>>>>>>>> the "order of
>>>>>>>> magnitude" I should expect or settings affecting seriously
>>>>>>>> these
>>>>>>>> performances. I tried both JVM and TCP acceptor but the
>>>>>>>> performances are
>>>>>>>> quite similar though JVM is a bit faster.
>>>>>>>>
>>>>>>>>
>>>>>>> Looks like a sign that the NIO socket channels are efficient
>>>>>>> ,-)
>>>>>>> You used localhost for TCP transport I guess?
>>>>>>>
>>>>>>>
>>>>>>>> * First case : building a small model (10000 elements) with
>>>>>>>> the simple
>>>>>>>> MM, only adding instances in an empty database :
>>>>>>>> commiting change on every add I get a 2 objects per second
>>>>>>>> rate.
>>>>>>>> commiting changes every 10 adds - I get a 30 objects per
>>>>>>>> second rate.
>>>>>>>>
>>>>>>>>
>>>>>>> Seems to be consistent with the fact that only for commits
>>>>>>> there is
>>>>>>> remote traffic.
>>>>>>> And the backend transaction management is another overhead.
>>>>>>> So the impact of the number objects/commit starts with bigger
>>>>>>> transactions.
>>>>>>>
>>>>>>>
>>>>>>>> browsing the just commited elements opening a new
>>>>>>>> transaction and getting
>>>>>>>> a new Resource instance, I get around 30 000 objets/sec
>>>>>>>> rate just
>>>>>>>> iterating thanks to Resource.getAllContents().
>>>>>>>>
>>>>>>>>
>>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>>> All views and transactions of a single session share (and
>>>>>>> cache) their
>>>>>>> object state.
>>>>>>> Otherwise I'd expect much smaller values.
>>>>>>> With his ObjectivityStore implementation Simon (cc'ed)
>>>>>>> reached rates of
>>>>>>> several thousands objects/sec.
>>>>>>> But the DBStore (which you used) is not that optimized.
>>>>>>>
>>>>>>>
>>>>>>>> * Second case : I built a big model in the database using
>>>>>>>> the complex
>>>>>>>> metamodel. It contains 700 000 elements. The creation went
>>>>>>>> pretty nicely
>>>>>>>> (I did not measured it exactly as it was launched during a
>>>>>>>> night, but it
>>>>>>>> seems I got around 100 objects/sec rate in writting as I
>>>>>>>> was writting
>>>>>>>> every 700 objects.
>>>>>>>>
>>>>>>>>
>>>>>>> Sounds not too bad.
>>>>>>>
>>>>>>>
>>>>>>>> Once the model is built the derby database is 222 MB (which
>>>>>>>> - I heard -
>>>>>>>> is not so big for derby).
>>>>>>>>
>>>>>>>>
>>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>>> To be honest, I'm really no database expert (although I can
>>>>>>> write simple
>>>>>>> SQL statements).
>>>>>>>
>>>>>>>
>>>>>>>> Retrieving the distant resource and calling eAllContents()
>>>>>>>> on it, I get a
>>>>>>>> 50 Objects/sec rate at best, which mean my eAllContents()
>>>>>>>> call would take
>>>>>>>> 4 hours !
>>>>>>>>
>>>>>>>>
>>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>>> (iteration
>>>>>>> order and so).
>>>>>>> There are several CDO features that can have a major impact
>>>>>>> on load rates!
>>>>>>>
>>>>>>> - Collections (xrefs and containment) can be loaded lazily.
>>>>>>> I don't mean
>>>>>>> the target objects, which are lazy by default anyway, but
>>>>>>> even the
>>>>>>> target IDs that are in the collections. We call that "reference
>>>>>>> chunking" (but search for a better name is ongoing).
>>>>>>> |*public interface *CDOSession *extends *CDOProtocolSession,
>>>>>>> |IContainer<CDOView>
>>>>>>> {
>>>>>>> *public **int *getReferenceChunkSize();
>>>>>>>
>>>>>>> *public **void *setReferenceChunkSize(*int
>>>>>>> *referenceChunkSize);|
>>>>>>>
>>>>>>>
>>>>>>> - Loading collection elements (target objects) can be
>>>>>>> batched (the
>>>>>>> current name is even worse)
>>>>>>> |*public interface *CDOView *extends *CDOProtocolView,
>>>>>>> INotifier
>>>>>>> {
>>>>>>> *public **int *getLoadRevisionCollectionChunkSize();
>>>>>>>
>>>>>>> *public **void *setLoadRevisionCollectionChunkSize(*int
>>>>>>> *loadRevisionCollectionChunkSize);|
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> - One of the most sophisticated feature is automatic model
>>>>>>> usage
>>>>>>> analyzation. The result are fetch rules that are sent to the
>>>>>>> server.
>>>>>>> When later objects are loaded from the server, it can answer
>>>>>>> with
>>>>>>> additional objects that are likely to be needed by the
>>>>>>> client in the
>>>>>>> near future.
>>>>>>> |*public class *CDOViewImpl *extends
>>>>>>> |*org.eclipse.net4j.internal.util.event.Notifier *implements
>>>>>>> *CDOView,
>>>>>>> |CDOIDProvider,
>>>>>>> Adapter.Internal
>>>>>>> {
>>>>>>> *public *CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>>>> {
>>>>>>> *return *featureAnalyzer;
>>>>>>> }
>>>>>>>
>>>>>>> *public **void *setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>>>> featureAnalyzer)
>>>>>>> {
>>>>>>> *this*.featureAnalyzer = featureAnalyzer == *null *?
>>>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>>>> }|
>>>>>>>
>>>>>>>
>>>>>>> - There's a SmartReadAheadThread (if you're curious, please
>>>>>>> ask Simon).
>>>>>>> - Caches can be configured
>>>>>>> - ...
>>>>>>>
>>>>>>>
>>>>>>> Known issues of the DBStore:
>>>>>>> - When in auditing mode updates are always written as
>>>>>>> complete inserts
>>>>>>> (by design)
>>>>>>> - When configuring toManyReferences to a value different from
>>>>>>> LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
>>>>>>> - No (few) prepared statements are used (see
>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
>>>>>>> - Maybe others...
>>>>>>>
>>>>>>> Please note that the DBStore was not written by me to
>>>>>>> compete with
>>>>>>> Hibernate in terms of performance or mapping capabilities.
>>>>>>> Now that Martin and I, well in the end mostly Martin, provide a
>>>>>>> HibernateStore and this HibernateStore gets more and more
>>>>>>> features I
>>>>>>> expect that there will be a trend away from the DBStore.
>>>>>>>
>>>>>>>
>>>>>>>> This is probably linked to the way the mapping to db is
>>>>>>>> done, here's the
>>>>>>>> config I used (came from CDO-server example if I recall
>>>>>>>> correctly).
>>>>>>>>
>>>>>>>>
>>>>>>>> <repository name="repo1">
>>>>>>>> <property name="overrideUUID"
>>>>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>>>>> <property name="supportingAudits"
>>>>>>>> value="true"/>
>>>>>>>> <property name="verifyingRevisions"
>>>>>>>> value="false"/>
>>>>>>>> <property name="currentLRUCapacity"
>>>>>>>> value="10000"/>
>>>>>>>>
>>>>>>>>
>>>>>>> I guess this one could be larger.
>>>>>>>
>>>>>>>
>>>>>>>> <property name="revisedLRUCapacity"
>>>>>>>> value="100"/>
>>>>>>>>
>>>>>>>> <store type="db">
>>>>>>>> <!-- type: horizontal | vertical |
>>>>>>>> <any
>>>>>>>> user-contributed type> -->
>>>>>>>> <mappingStrategy
>>>>>>>> type="horizontal">
>>>>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>>>>> ONE_TABLE_PER_CLASS |
>>>>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY |
>>>>>>>> LIKE_ATTRIBUTES -->
>>>>>>>> <property
>>>>>>>> name="toManyReferences"
>>>>>>>>
>>>>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>>>>
>>>>>>>> <!-- LIKE_ATTRIBUTES |
>>>>>>>> LIKE_TO_MANY_REFERENCES -->
>>>>>>>> <property
>>>>>>>> name="toOneReferences"
>>>>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>>>>
>>>>>>>> <!-- MODEL | STRATEGY -->
>>>>>>>> <property
>>>>>>>> name="mappingPrecedence"
>>>>>>>> value="MODEL"/>
>>>>>>>>
>>>>>>>>
>>>>>>> This one is obsolete.
>>>>>>>
>>>>>>>
>>>>>>>> </mappingStrategy>
>>>>>>>>
>>>>>>>> <dbAdapter name="derby-embedded"/>
>>>>>>>> <dataSource
>>>>>>>>
>>>>>>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>>>>>> databaseName="/tmp/cdodb1"
>>>>>>>> createDatabase="create"/>
>>>>>>>>
>>>>>>>> </store>
>>>>>>>> </repository>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> What do you think about that ? What throughput do you think
>>>>>>>> I should
>>>>>>>> expect ?
>>>>>>>>
>>>>>>> As I said that's very difficult to judge.
>>>>>>> I believe it's more appropriate to compare the actual values
>>>>>>> with values
>>>>>>> from other measures (other setups, properties, etc. or even
>>>>>>> other
>>>>>>> products) instead of random expectations.
>>>>>>> Apologies that I didn't have the time so far to provide some
>>>>>>> reference
>>>>>>> measures myself. Maybe this gets better now that the team
>>>>>>> starts to grow
>>>>>>> slowly ;-)
>>>>>>>
>>>>>>>
>>>>>>>> What are the main reasons fur such figures ?
>>>>>>>>
>>>>>>>>
>>>>>>> Hmm, I *suspect* you don't like the figures? I can find no
>>>>>>> evidence your
>>>>>>> assessment ;-)
>>>>>>>
>>>>>>> Ahh,one more note:
>>>>>>> Tracing can also have a significant impact on performance
>>>>>>> and end-to-end
>>>>>>> throughput.
>>>>>>> Turn it off by all means. And please make sure that it's
>>>>>>> really turned
>>>>>>> off, not only invisible (i.e. console is not a target).
>>>>>>>
>>>>>>> Cheers
>>>>>>> /Eike
>>>>>>>
>>>>>
>>>>>
>>>>>
>>>>


Re: [CDO] reading/writting throughput performances [message #619098 is a reply to message #123651] Fri, 30 May 2008 12:43 Go to previous message
Cedric Brun is currently offline Cedric BrunFriend
Messages: 431
Registered: July 2009
Senior Member
Hi Simon and Eike,

thanks for your feedback, now I know what I could expect in using optimized
implementation, I'll have a try on the hibernate store and see if it is
scaling in a better way.

You gave me many pointers to explain the performances issues but in fact it
seems that the SQL execution is taking so much time (not suprisingly in
fact ;) . My first profile was wrong.

So I'm installing the hibernate store retrieving all the source code from
CVS, and I'm trying to launch the unit tests. I added many jars and
everything gets compiled but most of the tests are failling with "

"
org.hibernate.MappingException: Could not determine type for:
org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType, at
table: cdo_features, for columns: [org.hibernate.mapping.Column(type)]
"

any hint ?

Cédric


Simon McDuff wrote:

> We noticed a performance degradation in the following case:
>
> When we fetch a collection of objects that contains collections that we
> don't want to use. Ex:
>
> Model:
>
> ClassB
> {
> int foo1;
> List childrens1;
> List childrens2;
> List childrens3;
> };
>
> Code:
>
> Resource.contents contains objects of type ClassB.
>
> By fetching instances of classB, we will also fetched some items in the
> list(childrens1, childrens2, childrens3) The problem we had.. is we do not
> know if we want to use it. SO better not to download them.
>
> So In our production(we used our own IStore), we never download any
> objects from collection. I'm working at the moment to put in place all
> feature we used in the current CDO. That one will be part of
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477 To begin
> with,CDORevisionImpl needs to supports list that are 0 even if
> referenceChunkSize == 0.
>
> It is true that we obtain good performance.
> Worst case :
> 1500 objects /sec (Real fetch from disk. Page that contains the objects,
> wasn't in the cache of the filesystem.)
>
> Best case
> 12 000 objects/Sec (Fetch objects that are already in the case of the
> filesystem)
>
> (These statistics always fetch objects from server to client. The cache
> client isn't used)
>
> We have another optimization one with SmartReadAhead but for now only work
> for UI client. (for now) This is something we will integrate as well in
> CDO. But for now we concentrate on the other things.
>
> I used an Object database. Based on my preliminary test without CDO.
> I compare
>
> Hibernate vs Objectivity/EMF :
> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>
> The worst.. you will get 30 to 240 objects/sec or
> the best... you will get 500 to 4000 objects/sec
>
> Based on your comments... you are getting the worst (with Derby)!! :-)
>
>
> Another comments about insertion.
>
> We found another problem when we insert data(DOn't know if it is still
> true). Resource.Contents is a unique list. Basically each time we insert
> one element, it needs to fetch all the collection to see if we already
> have this element. Basically.. it is time consuming. In our case we are
> not going through this stage to persist our objects... but the objects are
> not accessible through that list.
>
>
>
> If speed is an issue for you!! We (You included :-)) can optimize it. What
> I like is Eike did a really good job doing that framwork and optimizations
> are easy to add. Also you can change your back-end without affecting you
> client side. (Maybe going to an Object database!!) THis was one of the
> purpose of CDO!! :-)
>
>
> Simon
>
>
> "Eike Stepper" <stepper@sympedia.de> a écrit dans le message de news:
> 483E9332.6050800@sympedia.de... Hi Cédric,
>
> That sounds interesting!
> Comments inline...
>
>
> Cédric Brun schrieb:
> Hi Eike,
>
> I've been messing around CDO lately and I hacked a few ugly unit-tests
> measuring read/write throughput when I use CDO (RC2).
>
> I'm using an embeded-derby database, two different metamodels (complex one
> - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
> Features)
>
> The test is a "quick and dirty" one so please do not take these figures
> too seriously, I would just like to have your input about the "order of
> magnitude" I should expect or settings affecting seriously these
> performances. I tried both JVM and TCP acceptor but the performances are
> quite similar though JVM is a bit faster.
> Looks like a sign that the NIO socket channels are efficient ,-)
> You used localhost for TCP transport I guess?
>
>
>
> * First case : building a small model (10000 elements) with the simple MM,
> only adding instances in an empty database :
> commiting change on every add I get a 2 objects per second rate.
> commiting changes every 10 adds - I get a 30 objects per second rate.
> Seems to be consistent with the fact that only for commits there is
> remote traffic. And the backend transaction management is another
> overhead. So the impact of the number objects/commit starts with bigger
> transactions.
>
>
> browsing the just commited elements opening a new transaction and getting
> a new Resource instance, I get around 30 000 objets/sec rate just
> iterating thanks to Resource.getAllContents().
> Hehe, this is a sign that client side caching works well:
> All views and transactions of a single session share (and cache) their
> object state. Otherwise I'd expect much smaller values.
> With his ObjectivityStore implementation Simon (cc'ed) reached rates of
> several thousands objects/sec. But the DBStore (which you used) is not
> that optimized.
>
>
> * Second case : I built a big model in the database using the complex
> metamodel. It contains 700 000 elements. The creation went pretty nicely
> (I did not measured it exactly as it was launched during a night, but it
> seems I got around 100 objects/sec rate in writting as I was writting
> every 700 objects.
> Sounds not too bad.
>
>
>
> Once the model is built the derby database is 222 MB (which - I heard - is
> not so big for derby).
> Hmm, I have no extensive experience with Derby.
> To be honest, I'm really no database expert (although I can write simple
> SQL statements).
>
>
> Retrieving the distant resource and calling eAllContents() on it, I get a
> 50 Objects/sec rate at best, which mean my eAllContents() call would take
> 4 hours !
> Currently I'm not sure what exactly eAllContents() does do (iteration
> order and so). There are several CDO features that can have a major
> impact on load rates!
>
> - Collections (xrefs and containment) can be loaded lazily. I don't mean
> the target objects, which are lazy by default anyway, but even the
> target IDs that are in the collections. We call that "reference
> chunking" (but search for a better name is ongoing).
>
> public interface CDOSession extends CDOProtocolSession,
> IContainer<CDOView>
> {
> public int getReferenceChunkSize();
>
> public void setReferenceChunkSize(int referenceChunkSize);
>
>
> - Loading collection elements (target objects) can be batched (the
> current name is even worse)
>
> public interface CDOView extends CDOProtocolView, INotifier
> {
> public int getLoadRevisionCollectionChunkSize();
>
> public void setLoadRevisionCollectionChunkSize(int
> loadRevisionCollectionChunkSize);
>
>
> - One of the most sophisticated feature is automatic model usage
> analyzation. The result are fetch rules that are sent to the server.
> When later objects are loaded from the server, it can answer with
> additional objects that are likely to be needed by the client in the
> near future.
>
> public class CDOViewImpl extends
> org.eclipse.net4j.internal.util.event.Notifier implements CDOView,
> CDOIDProvider, Adapter.Internal
> {
> public CDOFeatureAnalyzer getFeatureAnalyzer()
> {
> return featureAnalyzer;
> }
>
> public void setFeatureAnalyzer(CDOFeatureAnalyzer
> featureAnalyzer)
> {
> this.featureAnalyzer = featureAnalyzer == null ?
> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
> }
>
>
> - There's a SmartReadAheadThread (if you're curious, please ask Simon).
> - Caches can be configured
> - ...
>
>
> Known issues of the DBStore:
> - When in auditing mode updates are always written as complete inserts
> (by design) - When configuring toManyReferences to a value different
> from LIKE_ATTRIBUTES (which is not implemented yet) it is very slow. -
> No (few) prepared statements are used (see
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe others...
>
> Please note that the DBStore was not written by me to compete with
> Hibernate in terms of performance or mapping capabilities. Now that
> Martin and I, well in the end mostly Martin, provide a HibernateStore
> and this HibernateStore gets more and more features I expect that there
> will be a trend away from the DBStore.
>
>
> This is probably linked to the way the mapping to db is done, here's the
> config I used (came from CDO-server example if I recall correctly).
>
>
> <repository name="repo1">
> <property name="overrideUUID"
> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
> <property name="supportingAudits" value="true"/>
> <property name="verifyingRevisions" value="false"/>
> <property name="currentLRUCapacity" value="10000"/>
> I guess this one could be larger.
>
>
> <property name="revisedLRUCapacity" value="100"/>
>
> <store type="db">
> <!-- type: horizontal | vertical | <any
> user-contributed type> --> <mappingStrategy
> type="horizontal">
> <!-- ONE_TABLE_PER_REFERENCE |
> ONE_TABLE_PER_CLASS |
> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
> <property name="toManyReferences"
> value="ONE_TABLE_PER_REFERENCE"/>
>
> <!-- LIKE_ATTRIBUTES |
> LIKE_TO_MANY_REFERENCES --> <property
> name="toOneReferences"
> value="LIKE_ATTRIBUTES"/>
>
> <!-- MODEL | STRATEGY -->
> <property name="mappingPrecedence"
> value="MODEL"/>
> This one is obsolete.
>
>
> </mappingStrategy>
>
> <dbAdapter name="derby-embedded"/>
> <dataSource
> class="org.apache.derby.jdbc.EmbeddedDataSource"
> databaseName="/tmp/cdodb1"
> createDatabase="create"/>
>
> </store>
> </repository>
>
>
>
>
> What do you think about that ? What throughput do you think I should
> expect ? As I said that's very difficult to judge.
> I believe it's more appropriate to compare the actual values with values
> from other measures (other setups, properties, etc. or even other
> products) instead of random expectations. Apologies that I didn't have
> the time so far to provide some reference measures myself. Maybe this
> gets better now that the team starts to grow slowly ;-)
>
>
> What are the main reasons fur such figures ?
> Hmm, I *suspect* you don't like the figures? I can find no evidence your
> assessment ;-)
>
> Ahh,one more note:
> Tracing can also have a significant impact on performance and end-to-end
> throughput. Turn it off by all means. And please make sure that it's
> really turned off, not only invisible (i.e. console is not a target).
>
> Cheers
> /Eike


http://cedric.brun.io news and articles on eclipse and eclipse modeling.
Re: [CDO] reading/writting throughput performances [message #619099 is a reply to message #124080] Fri, 30 May 2008 12:39 Go to previous message
Martin Taal is currently offline Martin TaalFriend
Messages: 5468
Registered: July 2009
Senior Member
Hi Cédric,
Most of the time this exception is thrown because hibernate can't find the class. A common cause is
that the hibernate jar files are located in a separate plugin which can not reach the classes in
other plugins. If you set the eclipse buddy policy of the hibernate plugin to dependent then this
can help.

gr. Martin

Cédric Brun wrote:
> Hi Simon and Eike,
>
> thanks for your feedback, now I know what I could expect in using optimized
> implementation, I'll have a try on the hibernate store and see if it is
> scaling in a better way.
>
> You gave me many pointers to explain the performances issues but in fact it
> seems that the SQL execution is taking so much time (not suprisingly in
> fact ;) . My first profile was wrong.
>
> So I'm installing the hibernate store retrieving all the source code from
> CVS, and I'm trying to launch the unit tests. I added many jars and
> everything gets compiled but most of the tests are failling with "
>
> "
> org.hibernate.MappingException: Could not determine type for:
> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType, at
> table: cdo_features, for columns: [org.hibernate.mapping.Column(type)]
> "
>
> any hint ?
>
> Cédric
>
>
> Simon McDuff wrote:
>
>> We noticed a performance degradation in the following case:
>>
>> When we fetch a collection of objects that contains collections that we
>> don't want to use. Ex:
>>
>> Model:
>>
>> ClassB
>> {
>> int foo1;
>> List childrens1;
>> List childrens2;
>> List childrens3;
>> };
>>
>> Code:
>>
>> Resource.contents contains objects of type ClassB.
>>
>> By fetching instances of classB, we will also fetched some items in the
>> list(childrens1, childrens2, childrens3) The problem we had.. is we do not
>> know if we want to use it. SO better not to download them.
>>
>> So In our production(we used our own IStore), we never download any
>> objects from collection. I'm working at the moment to put in place all
>> feature we used in the current CDO. That one will be part of
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477 To begin
>> with,CDORevisionImpl needs to supports list that are 0 even if
>> referenceChunkSize == 0.
>>
>> It is true that we obtain good performance.
>> Worst case :
>> 1500 objects /sec (Real fetch from disk. Page that contains the objects,
>> wasn't in the cache of the filesystem.)
>>
>> Best case
>> 12 000 objects/Sec (Fetch objects that are already in the case of the
>> filesystem)
>>
>> (These statistics always fetch objects from server to client. The cache
>> client isn't used)
>>
>> We have another optimization one with SmartReadAhead but for now only work
>> for UI client. (for now) This is something we will integrate as well in
>> CDO. But for now we concentrate on the other things.
>>
>> I used an Object database. Based on my preliminary test without CDO.
>> I compare
>>
>> Hibernate vs Objectivity/EMF :
>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>
>> The worst.. you will get 30 to 240 objects/sec or
>> the best... you will get 500 to 4000 objects/sec
>>
>> Based on your comments... you are getting the worst (with Derby)!! :-)
>>
>>
>> Another comments about insertion.
>>
>> We found another problem when we insert data(DOn't know if it is still
>> true). Resource.Contents is a unique list. Basically each time we insert
>> one element, it needs to fetch all the collection to see if we already
>> have this element. Basically.. it is time consuming. In our case we are
>> not going through this stage to persist our objects... but the objects are
>> not accessible through that list.
>>
>>
>>
>> If speed is an issue for you!! We (You included :-)) can optimize it. What
>> I like is Eike did a really good job doing that framwork and optimizations
>> are easy to add. Also you can change your back-end without affecting you
>> client side. (Maybe going to an Object database!!) THis was one of the
>> purpose of CDO!! :-)
>>
>>
>> Simon
>>
>>
>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le message de news:
>> 483E9332.6050800@sympedia.de... Hi Cédric,
>>
>> That sounds interesting!
>> Comments inline...
>>
>>
>> Cédric Brun schrieb:
>> Hi Eike,
>>
>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>> measuring read/write throughput when I use CDO (RC2).
>>
>> I'm using an embeded-derby database, two different metamodels (complex one
>> - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>> Features)
>>
>> The test is a "quick and dirty" one so please do not take these figures
>> too seriously, I would just like to have your input about the "order of
>> magnitude" I should expect or settings affecting seriously these
>> performances. I tried both JVM and TCP acceptor but the performances are
>> quite similar though JVM is a bit faster.
>> Looks like a sign that the NIO socket channels are efficient ,-)
>> You used localhost for TCP transport I guess?
>>
>>
>>
>> * First case : building a small model (10000 elements) with the simple MM,
>> only adding instances in an empty database :
>> commiting change on every add I get a 2 objects per second rate.
>> commiting changes every 10 adds - I get a 30 objects per second rate.
>> Seems to be consistent with the fact that only for commits there is
>> remote traffic. And the backend transaction management is another
>> overhead. So the impact of the number objects/commit starts with bigger
>> transactions.
>>
>>
>> browsing the just commited elements opening a new transaction and getting
>> a new Resource instance, I get around 30 000 objets/sec rate just
>> iterating thanks to Resource.getAllContents().
>> Hehe, this is a sign that client side caching works well:
>> All views and transactions of a single session share (and cache) their
>> object state. Otherwise I'd expect much smaller values.
>> With his ObjectivityStore implementation Simon (cc'ed) reached rates of
>> several thousands objects/sec. But the DBStore (which you used) is not
>> that optimized.
>>
>>
>> * Second case : I built a big model in the database using the complex
>> metamodel. It contains 700 000 elements. The creation went pretty nicely
>> (I did not measured it exactly as it was launched during a night, but it
>> seems I got around 100 objects/sec rate in writting as I was writting
>> every 700 objects.
>> Sounds not too bad.
>>
>>
>>
>> Once the model is built the derby database is 222 MB (which - I heard - is
>> not so big for derby).
>> Hmm, I have no extensive experience with Derby.
>> To be honest, I'm really no database expert (although I can write simple
>> SQL statements).
>>
>>
>> Retrieving the distant resource and calling eAllContents() on it, I get a
>> 50 Objects/sec rate at best, which mean my eAllContents() call would take
>> 4 hours !
>> Currently I'm not sure what exactly eAllContents() does do (iteration
>> order and so). There are several CDO features that can have a major
>> impact on load rates!
>>
>> - Collections (xrefs and containment) can be loaded lazily. I don't mean
>> the target objects, which are lazy by default anyway, but even the
>> target IDs that are in the collections. We call that "reference
>> chunking" (but search for a better name is ongoing).
>>
>> public interface CDOSession extends CDOProtocolSession,
>> IContainer<CDOView>
>> {
>> public int getReferenceChunkSize();
>>
>> public void setReferenceChunkSize(int referenceChunkSize);
>>
>>
>> - Loading collection elements (target objects) can be batched (the
>> current name is even worse)
>>
>> public interface CDOView extends CDOProtocolView, INotifier
>> {
>> public int getLoadRevisionCollectionChunkSize();
>>
>> public void setLoadRevisionCollectionChunkSize(int
>> loadRevisionCollectionChunkSize);
>>
>>
>> - One of the most sophisticated feature is automatic model usage
>> analyzation. The result are fetch rules that are sent to the server.
>> When later objects are loaded from the server, it can answer with
>> additional objects that are likely to be needed by the client in the
>> near future.
>>
>> public class CDOViewImpl extends
>> org.eclipse.net4j.internal.util.event.Notifier implements CDOView,
>> CDOIDProvider, Adapter.Internal
>> {
>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>> {
>> return featureAnalyzer;
>> }
>>
>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>> featureAnalyzer)
>> {
>> this.featureAnalyzer = featureAnalyzer == null ?
>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>> }
>>
>>
>> - There's a SmartReadAheadThread (if you're curious, please ask Simon).
>> - Caches can be configured
>> - ...
>>
>>
>> Known issues of the DBStore:
>> - When in auditing mode updates are always written as complete inserts
>> (by design) - When configuring toManyReferences to a value different
>> from LIKE_ATTRIBUTES (which is not implemented yet) it is very slow. -
>> No (few) prepared statements are used (see
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe others...
>>
>> Please note that the DBStore was not written by me to compete with
>> Hibernate in terms of performance or mapping capabilities. Now that
>> Martin and I, well in the end mostly Martin, provide a HibernateStore
>> and this HibernateStore gets more and more features I expect that there
>> will be a trend away from the DBStore.
>>
>>
>> This is probably linked to the way the mapping to db is done, here's the
>> config I used (came from CDO-server example if I recall correctly).
>>
>>
>> <repository name="repo1">
>> <property name="overrideUUID"
>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>> <property name="supportingAudits" value="true"/>
>> <property name="verifyingRevisions" value="false"/>
>> <property name="currentLRUCapacity" value="10000"/>
>> I guess this one could be larger.
>>
>>
>> <property name="revisedLRUCapacity" value="100"/>
>>
>> <store type="db">
>> <!-- type: horizontal | vertical | <any
>> user-contributed type> --> <mappingStrategy
>> type="horizontal">
>> <!-- ONE_TABLE_PER_REFERENCE |
>> ONE_TABLE_PER_CLASS |
>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>> <property name="toManyReferences"
>> value="ONE_TABLE_PER_REFERENCE"/>
>>
>> <!-- LIKE_ATTRIBUTES |
>> LIKE_TO_MANY_REFERENCES --> <property
>> name="toOneReferences"
>> value="LIKE_ATTRIBUTES"/>
>>
>> <!-- MODEL | STRATEGY -->
>> <property name="mappingPrecedence"
>> value="MODEL"/>
>> This one is obsolete.
>>
>>
>> </mappingStrategy>
>>
>> <dbAdapter name="derby-embedded"/>
>> <dataSource
>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>> databaseName="/tmp/cdodb1"
>> createDatabase="create"/>
>>
>> </store>
>> </repository>
>>
>>
>>
>>
>> What do you think about that ? What throughput do you think I should
>> expect ? As I said that's very difficult to judge.
>> I believe it's more appropriate to compare the actual values with values
>> from other measures (other setups, properties, etc. or even other
>> products) instead of random expectations. Apologies that I didn't have
>> the time so far to provide some reference measures myself. Maybe this
>> gets better now that the team starts to grow slowly ;-)
>>
>>
>> What are the main reasons fur such figures ?
>> Hmm, I *suspect* you don't like the figures? I can find no evidence your
>> assessment ;-)
>>
>> Ahh,one more note:
>> Tracing can also have a significant impact on performance and end-to-end
>> throughput. Turn it off by all means. And please make sure that it's
>> really turned off, not only invisible (i.e. console is not a target).
>>
>> Cheers
>> /Eike
>


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Cell: +31 (0)6 288 48 943
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: [CDO] reading/writting throughput performances [message #619101 is a reply to message #123787] Fri, 30 May 2008 13:08 Go to previous message
Simon Mc Duff is currently offline Simon Mc DuffFriend
Messages: 596
Registered: July 2009
Senior Member
This is a multi-part message in MIME format.

------=_NextPart_000_0540_01C8C234.B8DE3F40
Content-Type: text/plain;
charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Performance problem with resource.content
https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D234839
"Eike Stepper" <stepper@sympedia.de> a =C3=A9crit dans le message de =
news: g1mp23$pu1$4@build.eclipse.org...
Guys,

Comments inline...


Simon McDuff schrieb:=20
We noticed a performance degradation in the following case:
=20
When we fetch a collection of objects that contains collections that =
we don't want to use.=20
Ex:
=20
Model:
=20
ClassB
{
int foo1;
List childrens1;
List childrens2;
List childrens3;
};
=20
Code:

Resource.contents contains objects of type ClassB.
=20
By fetching instances of classB, we will also fetched some items in =
the list(childrens1, childrens2, childrens3) The problem we had.. is we =
do not know if we want to use it. SO better not to download them.

So In our production(we used our own IStore), we never download any =
objects from collection.=20
I'm working at the moment to put in place all feature we used in the =
current CDO.
That one will be part of=20
https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214477 To begin =
with,CDORevisionImpl needs to supports list that are 0 even if =
referenceChunkSize =3D=3D 0.
=20
It is true that we obtain good performance.
Worst case :
1500 objects /sec (Real fetch from disk. Page that contains the =
objects, wasn't in the cache of the filesystem.)=20
=20
Best case=20
12 000 objects/Sec (Fetch objects that are already in the case of =
the filesystem)
=20
(These statistics always fetch objects from server to client. The =
cache client isn't used)
=20
We have another optimization one with SmartReadAhead but for now =
only work for UI client. (for now) This is something we will integrate =
as well in CDO. But for now we concentrate on the other things.
=20
I used an Object database. Based on my preliminary test without CDO.
I compare
=20
Hibernate vs Objectivity/EMF :=20
Objectivity/EMF is around 3 to 50 times faster than Hibernate.
=20
The worst.. you will get 30 to 240 objects/sec or
the best... you will get 500 to 4000 objects/sec
=20
Based on your comments... you are getting the worst (with Derby)!! =
:-)

And with the DBStore!!
I'd really like to encourage you to use the HibernateStore.=20
It is much younger than the DBStore but Hibernate itself should do a =
much better job on mapping than the DBStore.

That said, the DBStore will, for multiple reasons, never cease to =
exist in CDO.



=20
Another comments about insertion.
=20
We found another problem when we insert data(DOn't know if it is =
still true). Resource.Contents is a unique list. Basically each time we =
insert one element, it needs to fetch all the collection to see if we =
already have this element. Basically.. it is time consuming. In our case =
we are not going through this stage to persist our objects... but the =
objects are not accessible through that list.

I was not aware of this. Sounds as if we should offer an alternative =
implementation for Resource.Contents.
Could you please open a Bugzilla to discuss/track this?





If speed is an issue for you!! We (You included :-)) can optimize =
it. What I like is Eike did a really good job doing that framwork and =
optimizations are easy to add.
Also you can change your back-end without affecting you client side. =
(Maybe going to an Object database!!)
THis was one of the purpose of CDO!! :-)
Thank you ;-)
Indeed, CDO was intended to focus of the protocol aspects, the =
repository and the client side addition to the EMF concepts.
I feel that this will become clearer the more IStore implementations =
we get that depend on mature 3rd party code for the mapping part.

Cheers
/Eike





Simon


"Eike Stepper" <stepper@sympedia.de> a =C3=A9crit dans le message =
de news: 483E9332.6050800@sympedia.de...
Hi C=C3=A9dric,

That sounds interesting!
Comments inline...


C=C3=A9dric Brun schrieb:=20
Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex =
one -
250 EClassifiers/300 features and a simple one - 1 Classifier/3 =
Features)

The test is a "quick and dirty" one so please do not take these figures =
too
seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.
Looks like a sign that the NIO socket channels are efficient ,-)
You used localhost for TCP transport I guess?


* First case : building a small model (10000 elements) with the simple =
MM,
only adding instances in an empty database :=20
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.
Seems to be consistent with the fact that only for commits there is =
remote traffic.
And the backend transaction management is another overhead.
So the impact of the number objects/commit starts with bigger =
transactions.


browsing the just commited elements opening a new transaction and =
getting a
new Resource instance, I get around 30 000 objets/sec rate just =
iterating
thanks to Resource.getAllContents().
Hehe, this is a sign that client side caching works well:
All views and transactions of a single session share (and cache) =
their object state.
Otherwise I'd expect much smaller values.
With his ObjectivityStore implementation Simon (cc'ed) reached =
rates of several thousands objects/sec.
But the DBStore (which you used) is not that optimized.


* Second case : I built a big model in the database using the complex
metamodel. It contains 700 000 elements. The creation went pretty nicely =
(I
did not measured it exactly as it was launched during a night, but it =
seems
I got around 100 objects/sec rate in writting as I was writting every =
700
objects.
Sounds not too bad.


Once the model is built the derby database is 222 MB (which - I heard - =
is
not so big for derby).
Hmm, I have no extensive experience with Derby.
To be honest, I'm really no database expert (although I can write =
simple SQL statements).


Retrieving the distant resource and calling eAllContents() on it, I get =
a 50
Objects/sec rate at best, which mean my eAllContents() call would take 4
hours !
Currently I'm not sure what exactly eAllContents() does do (iteration =
order and so).
There are several CDO features that can have a major impact on =
load rates!

- Collections (xrefs and containment) can be loaded lazily. I =
don't mean the target objects, which are lazy by default anyway, but =
even the target IDs that are in the collections. We call that "reference =
chunking" (but search for a better name is ongoing).

public interface CDOSession extends CDOProtocolSession, =
IContainer<CDOView>
{
public int getReferenceChunkSize();

public void setReferenceChunkSize(int referenceChunkSize); =
=20


- Loading collection elements (target objects) can be batched (the =
current name is even worse)

public interface CDOView extends CDOProtocolView, INotifier
{
public int getLoadRevisionCollectionChunkSize();

public void setLoadRevisionCollectionChunkSize(int =
loadRevisionCollectionChunkSize); =20


- One of the most sophisticated feature is automatic model usage =
analyzation. The result are fetch rules that are sent to the server. =
When later objects are loaded from the server, it can answer with =
additional objects that are likely to be needed by the client in the =
near future.

public class CDOViewImpl extends =
org.eclipse.net4j.internal.util.event.Notifier implements CDOView, =
CDOIDProvider, Adapter.Internal
{
public CDOFeatureAnalyzer getFeatureAnalyzer()
{
return featureAnalyzer;
}

public void setFeatureAnalyzer(CDOFeatureAnalyzer =
featureAnalyzer)
{
this.featureAnalyzer =3D featureAnalyzer =3D=3D null ? =
CDOFeatureAnalyzer.NOOP : featureAnalyzer;
} =20


- There's a SmartReadAheadThread (if you're curious, please ask =
Simon).
- Caches can be configured
- ...


Known issues of the DBStore:
- When in auditing mode updates are always written as complete =
inserts (by design)
- When configuring toManyReferences to a value different from =
LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
- No (few) prepared statements are used (see =
https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214487)
- Maybe others...

Please note that the DBStore was not written by me to compete with =
Hibernate in terms of performance or mapping capabilities.
Now that Martin and I, well in the end mostly Martin, provide a =
HibernateStore and this HibernateStore gets more and more features I =
expect that there will be a trend away from the DBStore.=20


This is probably linked to the way the mapping to db is done, here's the
config I used (came from CDO-server example if I recall correctly).


<repository name=3D"repo1">
<property name=3D"overrideUUID"
value=3D"1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
<property name=3D"supportingAudits" value=3D"true"/>
<property name=3D"verifyingRevisions" value=3D"false"/>
<property name=3D"currentLRUCapacity" value=3D"10000"/>
I guess this one could be larger.


<property name=3D"revisedLRUCapacity" value=3D"100"/>
=20
<store type=3D"db">
<!-- type: horizontal | vertical | <any =
user-contributed type> -->
<mappingStrategy type=3D"horizontal">
<!-- ONE_TABLE_PER_REFERENCE | =
ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
<property name=3D"toManyReferences" =
value=3D"ONE_TABLE_PER_REFERENCE"/>

<!-- LIKE_ATTRIBUTES | =
LIKE_TO_MANY_REFERENCES -->
<property name=3D"toOneReferences" =
value=3D"LIKE_ATTRIBUTES"/>
=20
<!-- MODEL | STRATEGY -->
<property name=3D"mappingPrecedence" =
value=3D"MODEL"/>
This one is obsolete.


</mappingStrategy>
=20
<dbAdapter name=3D"derby-embedded"/>
<dataSource =
class=3D"org.apache.derby.jdbc.EmbeddedDataSource"=20
databaseName=3D"/tmp/cdodb1"
createDatabase=3D"create"/>

</store>
</repository>




What do you think about that ? What throughput do you think I should
expect ? As I said that's very difficult to judge.
I believe it's more appropriate to compare the actual values with =
values from other measures (other setups, properties, etc. or even other =
products) instead of random expectations.
Apologies that I didn't have the time so far to provide some =
reference measures myself. Maybe this gets better now that the team =
starts to grow slowly ;-)


What are the main reasons fur such figures ?
Hmm, I *suspect* you don't like the figures? I can find no evidence =
your assessment ;-)

Ahh,one more note:
Tracing can also have a significant impact on performance and =
end-to-end throughput.
Turn it off by all means. And please make sure that it's really =
turned off, not only invisible (i.e. console is not a target).

Cheers
/Eike



------=_NextPart_000_0540_01C8C234.B8DE3F40
Content-Type: text/html;
charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

=EF=BB=BF<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD><TITLE></TITLE>
<META http-equiv=3DContent-Type content=3Dtext/html;charset=3DUTF-8>
<META content=3D"MSHTML 6.00.6000.16608" name=3DGENERATOR></HEAD>
<BODY text=3D#000000 bgColor=3D#ffffff>
<DIV><FONT face=3DArial size=3D2>Performance problem with=20
resource.content</FONT></DIV>
<DIV><FONT face=3DArial size=3D2><A=20
href=3D"https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D234839">https://b=
ugs.eclipse.org/bugs/show_bug.cgi?id=3D234839</A></FONT></DIV >
<BLOCKQUOTE dir=3Dltr=20
style=3D"PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; =
BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">
<DIV>"Eike Stepper" &lt;<A=20
href=3D"mailto:stepper@sympedia.de">stepper@sympedia.de</A>&gt; a =
=C3=A9crit dans le=20
message de news: <A=20
=
href=3D"mailto:g1mp23$pu1$4@build.eclipse.org">g1mp23$pu1$4@build.eclipse=
..org</A>...</DIV>Guys,<BR><BR>Comments=20
inline...<BR><BR><BR>Simon McDuff schrieb:=20
<BLOCKQUOTE cite=3Dmid:g1mkji$h5d$1@build.eclipse.org type=3D"cite">
<META content=3D"MSHTML 6.00.6000.16608" name=3DGENERATOR>
<DIV>We noticed a performance degradation in the following=20
case:<BR>&nbsp;<BR>When we fetch a collection of objects that =
contains=20
collections that we don't want to use.=20
<BR>Ex:<BR>&nbsp;<BR>Model:<BR>&nbsp;<BR>ClassB <BR>{<BR>&nbsp; int=20
foo1;<BR>&nbsp; List childrens1;<BR>&nbsp; List =
childrens2;<BR>&nbsp; List=20
childrens3;<BR>};<BR>&nbsp;<BR>Code:</DIV>
<DIV><BR>Resource.contents contains objects of type =
ClassB.<BR>&nbsp;<BR>By=20
fetching instances of classB, we will also fetched some items in the =

list(childrens1, childrens2, childrens3)&nbsp;<FONT face=3D"Courier =
New"><FONT=20
color=3D#000000>The problem we had.. is we do not know if we want to =
use it.=20
SO better not to download them.</FONT></FONT></DIV>
<DIV><BR><FONT face=3D"Courier New" color=3D#000000>So In our =
production(we used=20
our own IStore), we never download any objects from collection.=20
</FONT><BR><FONT face=3D"Courier New" color=3D#000000>I'm working at =
the moment=20
to put in place all feature we used in the current =
CDO.</FONT><BR><FONT=20
face=3D"Courier New" color=3D#000000>That one will be part of =
</FONT><BR><A=20
href=3D"https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214477" =
target=3D_blank=20
moz-do-not-send=3D"true"><FONT=20
=
color=3D#0068cf>https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214477</F=
ONT></A>.=20
To begin with,CDORevisionImpl needs to supports list that are 0 even =
if=20
<FONT face=3D"Courier New" color=3D#000000>referenceChunkSize =3D=3D =

0.</FONT><BR>&nbsp;<BR><FONT face=3D"Courier New" color=3D#000000>It =
is true=20
that we obtain good performance.</FONT><BR><FONT face=3D"Courier =
New"=20
color=3D#000000>Worst case :</FONT><BR><FONT face=3D"Courier New"=20
color=3D#000000>1500 objects /sec (Real fetch from&nbsp;disk. Page =
that=20
contains the objects, wasn't in the cache of the=20
filesystem.)&nbsp;</FONT><BR>&nbsp;<BR><FONT face=3D"Courier New"=20
color=3D#000000>Best case </FONT><BR><FONT face=3D"Courier New" =
color=3D#000000>12=20
000 objects/Sec (Fetch objects that are already in the case of the=20
filesystem)</FONT><BR>&nbsp;<BR>(These statistics always fetch =
objects from=20
server to client. The cache client isn't used)<BR>&nbsp;<BR><FONT=20
face=3D"Courier New" color=3D#000000>We have another optimization =
one with=20
SmartReadAhead but for now only work for UI client. (for now)&nbsp; =
This is=20
something we will integrate as well in CDO. But for now we =
concentrate on=20
the other things.</FONT><BR>&nbsp;<BR><FONT face=3D"Courier New"=20
color=3D#000000>I used an Object database. Based on my preliminary =
test=20
without CDO.</FONT><BR><FONT face=3D"Courier New" color=3D#000000>I=20
compare</FONT><BR>&nbsp;<BR><FONT face=3D"Courier New" =
color=3D#000000>Hibernate=20
vs </FONT><FONT face=3D"Courier New" color=3D#000000>Objectivity/EMF =

:&nbsp;</FONT><BR><FONT face=3D"Courier New" =
color=3D#000000>Objectivity/EMF is=20
around 3 to 50 times faster than =
Hibernate.</FONT><BR>&nbsp;<BR><FONT=20
face=3D"Courier New" color=3D#000000>The worst.. you will get 30 to =
240=20
objects/sec or</FONT><BR><FONT face=3D"Courier New" =
color=3D#000000>the best...=20
you will get 500 to 4000 objects/sec</FONT><BR>&nbsp;<BR><FONT=20
face=3D"Courier New" color=3D#000000>Based on your comments... you =
are getting=20
the worst (with Derby)!! :-)</FONT><BR></DIV></BLOCKQUOTE>And with =
the=20
DBStore!!<BR>I'd really like to encourage you to use the =
HibernateStore.=20
<BR>It is much younger than the DBStore but Hibernate itself should do =
a much=20
better job on mapping than the DBStore.<BR><BR>That said, the DBStore =
will,=20
for multiple reasons, never cease to exist in CDO.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1mkji$h5d$1@build.eclipse.org type=3D"cite">
<DIV><BR>&nbsp;<BR>Another comments about insertion.<BR>&nbsp;<BR>We =
found=20
another problem when we insert data(DOn't know if it is still true). =

Resource.Contents is&nbsp;a unique list. Basically each time we =
insert one=20
element, it needs to fetch all the collection to see if we already =
have this=20
element. Basically.. it is time consuming. In our case we are not =
going=20
through this stage to persist our objects... but the objects are not =

accessible through that list.<BR></DIV></BLOCKQUOTE>I was not aware =
of this.=20
Sounds as if we should offer an alternative implementation for=20
Resource.Contents.<BR>Could you please open a Bugzilla to =
discuss/track=20
this?<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1mkji$h5d$1@build.eclipse.org type=3D"cite">
<DIV>&nbsp;</DIV>
<DIV>&nbsp;</DIV>
<DIV>&nbsp;</DIV>
<DIV><FONT face=3DArial size=3D2>If speed is an issue for you!! We =
(You included=20
:-)) can optimize it. What I like is Eike did a really good job =
doing that=20
framwork and optimizations are easy to add.</FONT></DIV>
<DIV><FONT face=3DArial size=3D2>Also you can change your back-end =
without=20
affecting you client side. (Maybe going to an Object=20
database!!)</FONT></DIV>
<DIV><FONT face=3DArial size=3D2>THis was one of the purpose of =
CDO!!=20
:-)</FONT></DIV></BLOCKQUOTE>Thank you ;-)<BR>Indeed, CDO was =
intended to=20
focus of the protocol aspects, the repository and the client side =
addition to=20
the EMF concepts.<BR>I feel that this will become clearer the more =
IStore=20
implementations we get that depend on mature 3rd party code for the =
mapping=20
part.<BR><BR>Cheers<BR>/Eike<BR><BR><BR>
<BLOCKQUOTE cite=3Dmid:g1mkji$h5d$1@build.eclipse.org type=3D"cite">
<DIV>&nbsp;</DIV>
<DIV><BR>Simon<BR><BR></DIV>
<BLOCKQUOTE=20
style=3D"PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; =
BORDER-LEFT: rgb(0,0,0) 2px solid; MARGIN-RIGHT: 0px">
<DIV>"Eike Stepper" &lt;<A href=3D"mailto:stepper@sympedia.de"=20
moz-do-not-send=3D"true">stepper@sympedia.de</A>&gt; a =C3=A9crit =
dans le message=20
de news: <A href=3D"mailto:483E9332.6050800@sympedia.de"=20
=
moz-do-not-send=3D"true">483E9332.6050800@sympedia.de</A>...</DIV>Hi=20
C=C3=A9dric,<BR><BR>That sounds interesting!<BR>Comments=20
inline...<BR><BR><BR>C=C3=A9dric Brun schrieb:=20
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">Hi Eike,

I've been messing around CDO lately and I hacked a few ugly unit-tests
measuring read/write throughput when I use CDO (RC2).

I'm using an embeded-derby database, two different metamodels (complex =
one -
250 EClassifiers/300 features and a simple one - 1 Classifier/3 =
Features)

The test is a "quick and dirty" one so please do not take these figures =
too
seriously, I would just like to have your input about the "order of
magnitude" I should expect or settings affecting seriously these
performances. I tried both JVM and TCP acceptor but the performances are
quite similar though JVM is a bit faster.
</PRE></BLOCKQUOTE>Looks like a sign that the NIO socket channels are=20
efficient ,-)<BR>You used localhost for TCP transport I =
guess?<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">* First case : building a small model =
(10000 elements) with the simple MM,
only adding instances in an empty database :=20
commiting change on every add I get a 2 objects per second rate.
commiting changes every 10 adds - I get a 30 objects per second rate.
</PRE></BLOCKQUOTE>Seems to be consistent with the fact that only for=20
commits there is remote traffic.<BR>And the backend transaction =
management=20
is another overhead.<BR>So the impact of the number objects/commit =
starts=20
with bigger transactions.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">browsing the just commited elements opening =
a new transaction and getting a
new Resource instance, I get around 30 000 objets/sec rate just =
iterating
thanks to Resource.getAllContents().
</PRE></BLOCKQUOTE>Hehe, this is a sign that client side caching works =

well:<BR>All views and transactions of a single session share (and =
cache)=20
their object state.<BR>Otherwise I'd expect much smaller =
values.<BR>With=20
his ObjectivityStore implementation Simon (cc'ed) reached rates of =
several=20
thousands objects/sec.<BR>But the DBStore (which you used) is not =
that=20
optimized.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">* Second case : I built a big model in the =
database using the complex
metamodel. It contains 700 000 elements. The creation went pretty nicely =
(I
did not measured it exactly as it was launched during a night, but it =
seems
I got around 100 objects/sec rate in writting as I was writting every =
700
objects.
</PRE></BLOCKQUOTE>Sounds not too bad.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">Once the model is built the derby database =
is 222 MB (which - I heard - is
not so big for derby).
</PRE></BLOCKQUOTE>Hmm, I have no extensive experience with =
Derby.<BR>To=20
be honest, I'm really no database expert (although I can write =
simple SQL=20
statements).<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">Retrieving the distant resource and calling =
eAllContents() on it, I get a 50
Objects/sec rate at best, which mean my eAllContents() call would take 4
hours !
</PRE></BLOCKQUOTE>Currently I'm not sure what exactly eAllContents()=20
does do (iteration order and so).<BR>There are several CDO =
features that=20
can have a major impact on load rates!<BR><BR>- Collections (xrefs =
and=20
containment) can be loaded lazily. I don't mean the target =
objects, which=20
are lazy by default anyway, but even the target IDs that are in =
the=20
collections. We call that "reference chunking" (but search for a =
better=20
name is ongoing).<BR>
<STYLE type=3Dtext/css>CODE {
FONT-SIZE: 10pt; MARGIN: 0px; FONT-FAMILY: Courier New, Courier
}
</STYLE>
<!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><!-- =3D Java Sourcecode to HTML automatically =
converted code =3D --><!-- =3D Java2Html Converter 5.0 [2006-02-26] by =
Markus Gebhard markus@jave.de =3D --><!-- =3D Further =
information: http://www.java2html.de =3D -->
<DIV class=3Djava align=3Dleft>
<TABLE cellSpacing=3D0 cellPadding=3D3 bgColor=3D#ffffff =
border=3D0>
<TBODY>
<TR><!-- start source code -->
<TD vAlign=3Dtop noWrap align=3Dleft><CODE><FONT=20
=
color=3D#7f0055><B>public&nbsp;interface&nbsp;</B ></FONT><FONT=20
color=3D#000000>CDOSession&nbsp;</FONT><FONT=20
color=3D#7f0055><B>extends&nbsp;</B></FONT><FONT=20
=
color=3D#000000> CDOProtocolSession,&nbsp;IContainer&lt;CDOView&g t; </FONT>=
<BR><FONT=20
color=3D#000000>{</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>int&nbsp;</B></FONT><FONT=20
color=3D#000000>getReferenceChunkSize</FONT><FONT=20
color=3D#000000>()</FONT><FONT =
color=3D#000000>;</FONT><BR><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>void&nbsp;</B></FONT><FONT=20
color=3D#000000>setReferenceChunkSize</FONT><FONT=20
color=3D#000000>(</FONT><FONT=20
color=3D#7f0055><B>int&nbsp;</B></FONT><FONT=20
color=3D#000000>referenceChunkSize</FONT><FONT=20
color=3D#000000>)</FONT><FONT =
color=3D#000000>;</FONT></CODE> </TD><!-- end source code =
--></TR></TBODY></TABLE></DIV><!-- =3D END of automatically =
generated HTML code =3D --><BR><!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D -->-=20
Loading collection elements (target objects) can be batched (the =
current=20
name is even worse)<BR>
<STYLE type=3Dtext/css>
<!--code { font-family: Courier New, Courier; font-size: 10pt; =
margin: 0px; }-->
</STYLE>
<!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><!-- =3D Java Sourcecode to HTML automatically =
converted code =3D --><!-- =3D Java2Html Converter 5.0 [2006-02-26] by =
Markus Gebhard markus@jave.de =3D --><!-- =3D Further =
information: http://www.java2html.de =3D -->
<DIV class=3Djava align=3Dleft>
<TABLE cellSpacing=3D0 cellPadding=3D3 bgColor=3D#ffffff =
border=3D0>
<TBODY>
<TR><!-- start source code -->
<TD vAlign=3Dtop noWrap align=3Dleft><CODE><FONT=20
=
color=3D#7f0055><B>public&nbsp;interface&nbsp;</B ></FONT><FONT=20
color=3D#000000>CDOView&nbsp;</FONT><FONT=20
color=3D#7f0055><B>extends&nbsp;</B></FONT><FONT=20
=
color=3D#000000>CDOProtocolView,&nbsp;INotifier</FONT> <BR><FONT=20
color=3D#000000>{</FONT><BR><FONT color=3D#ffffff>&nbsp; =
</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>int&nbsp;</B></FONT><FONT=20
=
color=3D#000000>getLoadRevisionCollectionChunkSize</FONT><FONT=20
color=3D#000000>()</FONT><FONT =
color=3D#000000>;</FONT><BR><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>void&nbsp;</B></FONT><FONT=20
=
color=3D#000000>setLoadRevisionCollectionChunkSize</FONT><FONT=20
color=3D#000000>(</FONT><FONT=20
color=3D#7f0055><B>int&nbsp;</B></FONT><FONT=20
color=3D#000000>loadRevisionCollectionChunkSize</FONT><FONT=20
color=3D#000000>)</FONT><FONT =
color=3D#000000>;</FONT></CODE> </TD><!-- end source code =
--></TR></TBODY></TABLE></DIV><!-- =3D END of automatically =
generated HTML code =3D --><!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><BR>-=20
One of the most sophisticated feature is automatic model usage=20
analyzation. The result are fetch rules that are sent to the =
server. When=20
later objects are loaded from the server, it can answer with =
additional=20
objects that are likely to be needed by the client in the near =
future.<BR>
<STYLE type=3Dtext/css>
<!--code { font-family: Courier New, Courier; font-size: 10pt; =
margin: 0px; }-->
</STYLE>
<!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><!-- =3D Java Sourcecode to HTML automatically =
converted code =3D --><!-- =3D Java2Html Converter 5.0 [2006-02-26] by =
Markus Gebhard markus@jave.de =3D --><!-- =3D Further =
information: http://www.java2html.de =3D -->
<DIV class=3Djava align=3Dleft>
<TABLE cellSpacing=3D0 cellPadding=3D3 bgColor=3D#ffffff =
border=3D0>
<TBODY>
<TR><!-- start source code -->
<TD vAlign=3Dtop noWrap align=3Dleft><CODE><FONT=20
color=3D#7f0055><B>public&nbsp;class&nbsp;</B></FONT ><FONT=20
color=3D#000000>CDOViewImpl&nbsp;</FONT><FONT=20
color=3D#7f0055><B>extends&nbsp;</B></FONT><FONT=20
=
color=3D#000000>org.eclipse.net4j.internal.util.event.Notifier&nbsp; </FON=
T><FONT=20
color=3D#7f0055><B>implements&nbsp;</B></FONT><FONT=20
color=3D#000000>CDOView,&nbsp;CDOIDProvider,</FONT> <FONT=20
color=3D#000000>Adapter.Internal</FONT><BR><FONT=20
color=3D#000000>{</FONT><BR><FONT color=3D#ffffff>&nbsp; =
</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
=
color=3D#000000>CDOFeatureAnalyzer&nbsp;getFeatureAnalyzer </FONT><FONT=20
color=3D#000000>()</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#000000>{</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;&nbsp;&nbsp;</FONT ><FONT=20
color=3D#7f0055><B>return&nbsp;</B></FONT><FONT=20
color=3D#000000>featureAnalyzer;</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#000000>}</FONT><BR><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#7f0055><B>public&nbsp;</B></FONT><FONT=20
color=3D#7f0055><B>void&nbsp;</B></FONT><FONT=20
color=3D#000000>setFeatureAnalyzer</FONT><FONT=20
color=3D#000000>(</FONT><FONT=20
=
color=3D#000000>CDOFeatureAnalyzer&nbsp;featureAnalyzer </FONT><FONT=20
color=3D#000000>)</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT=20
color=3D#000000>{</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;&nbsp;&nbsp;</FONT ><FONT=20
color=3D#7f0055><B>this</B></FONT><FONT=20
=
color=3D#000000> .featureAnalyzer&nbsp;=3D&nbsp;featureAnalyzer&n bsp;=3D=3D=
&nbsp;</FONT><FONT=20
color=3D#7f0055><B>null&nbsp;</B></FONT><FONT=20
=
color=3D#000000> ?&nbsp;CDOFeatureAnalyzer.NOOP&nbsp;:&nbsp;featu reAnalyze=
r;</FONT><BR><FONT=20
color=3D#ffffff>&nbsp;&nbsp;</FONT><FONT =
color=3D#000000>}</FONT></CODE>=20
</TD><!-- end source code --></TR></TBODY></TABLE></DIV><!-- =
=3D END of automatically generated HTML code =3D --><!-- =
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D --><BR>-=20
There's a SmartReadAheadThread (if you're curious, please ask =
Simon).<BR>-=20
Caches can be configured<BR>- ...<BR><BR><BR>Known issues of the=20
DBStore:<BR>- When in auditing mode updates are always written as =
complete=20
inserts (by design)<BR>- When configuring toManyReferences to a =
value=20
different from LIKE_ATTRIBUTES (which is not implemented yet) it =
is very=20
slow.<BR>- No (few) prepared statements are used (see <A=20
class=3Dmoz-txt-link-freetext=20
href=3D"https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D214487"=20
=
moz-do-not-send=3D"true">https://bugs.eclipse.org/bugs/show_bug.cgi?id=3D=
214487</A>)<BR>-=20
Maybe others...<BR><BR>Please note that the DBStore was not =
written by me=20
to compete with Hibernate in terms of performance or mapping=20
capabilities.<BR>Now that Martin and I, well in the end mostly =
Martin,=20
provide a HibernateStore and this HibernateStore gets more and =
more=20
features I expect that there will be a trend away from the =
DBStore.=20
<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">This is probably linked to the way the =
mapping to db is done, here's the
config I used (came from CDO-server example if I recall correctly).


&lt;repository name=3D"repo1"&gt;
&lt;property name=3D"overrideUUID"
value=3D"1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/&gt;
&lt;property name=3D"supportingAudits" =
value=3D"true"/&gt;
&lt;property name=3D"verifyingRevisions" =
value=3D"false"/&gt;
&lt;property name=3D"currentLRUCapacity" =
value=3D"10000"/&gt;
</PRE></BLOCKQUOTE>I guess this one could be larger.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D""> &lt;property =
name=3D"revisedLRUCapacity" value=3D"100"/&gt;
=20
&lt;store type=3D"db"&gt;
&lt;!-- type: horizontal | vertical | &lt;any =
user-contributed type&gt; --&gt;
&lt;mappingStrategy type=3D"horizontal"&gt;
&lt;!-- ONE_TABLE_PER_REFERENCE | =
ONE_TABLE_PER_CLASS |
ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES =
--&gt;
&lt;property name=3D"toManyReferences" =
value=3D"ONE_TABLE_PER_REFERENCE"/&gt;

&lt;!-- LIKE_ATTRIBUTES | =
LIKE_TO_MANY_REFERENCES --&gt;
&lt;property name=3D"toOneReferences" =
value=3D"LIKE_ATTRIBUTES"/&gt;
=20
&lt;!-- MODEL | STRATEGY --&gt;
&lt;property name=3D"mappingPrecedence" =
value=3D"MODEL"/&gt;
</PRE></BLOCKQUOTE>This one is obsolete.<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D""> =
&lt;/mappingStrategy&gt;
=20
&lt;dbAdapter name=3D"derby-embedded"/&gt;
&lt;dataSource =
class=3D"org.apache.derby.jdbc.EmbeddedDataSource"=20
databaseName=3D"/tmp/cdodb1"
createDatabase=3D"create"/&gt;

&lt;/store&gt;
&lt;/repository&gt;




What do you think about that ? What throughput do you think I should
expect ? </PRE></BLOCKQUOTE>As I said that's very difficult to=20
judge.<BR>I believe it's more appropriate to compare the actual =
values=20
with values from other measures (other setups, properties, etc. or =
even=20
other products) instead of random expectations.<BR>Apologies that =
I didn't=20
have the time so far to provide some reference measures myself. =
Maybe this=20
gets better now that the team starts to grow slowly ;-)<BR><BR>
<BLOCKQUOTE cite=3Dmid:g1lvjh$jd7$1@build.eclipse.org =
type=3D"cite"><PRE wrap=3D"">What are the main reasons fur such figures =
?
</PRE></BLOCKQUOTE>Hmm, I *suspect* you don't like the figures? I can=20
find no evidence your assessment ;-)<BR><BR>Ahh,one more =
note:<BR>Tracing=20
can also have a significant impact on performance and end-to-end=20
throughput.<BR>Turn it off by all means. And please make sure that =
it's=20
really turned off, not only invisible (i.e. console is not a=20
=
target).<BR><BR>Cheers<BR>/Eike<BR><BR><BR></BLOCKQUOTE ></BLOCKQUOTE></BL=
OCKQUOTE></BODY></HTML>

------=_NextPart_000_0540_01C8C234.B8DE3F40--
Re: [CDO] reading/writting throughput performances [message #619103 is a reply to message #124092] Fri, 30 May 2008 13:19 Go to previous message
Cedric Brun is currently offline Cedric BrunFriend
Messages: 431
Registered: July 2009
Senior Member
Hi Martin,

and thanks for your quick answer ! what I did is the following :

I used the org.eclipse.emf.cdo.server.hibernate.libraries bundle and added
that :

Bundle-ClassPath: lib/hibernate3.jar,
lib/cglib-2.1.3.jar,
lib/commons-collections-2.1.1.jar,
lib/dom4j.jar,
lib/jta.jar
lib/asm-3.0.jar

and this directive :

Eclipse-BuddyPolicy: dependent

as stated in
http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries

Then I don't get the same error, now when I start my own plugin (JUnit
plugin test) depending on org.eclipse.emf.cdo.server.hibernate.libraries I
get :

java.lang.NoClassDefFoundError: org/objectweb/asm/Type
at net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
at net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
at net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
at
org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)

These jars are such a headache ! Do you know - by chance - what's going
on ?


Cédric


Martin Taal wrote:

> Hi Cédric,
> Most of the time this exception is thrown because hibernate can't find the
> class. A common cause is that the hibernate jar files are located in a
> separate plugin which can not reach the classes in other plugins. If you
> set the eclipse buddy policy of the hibernate plugin to dependent then
> this can help.
>
> gr. Martin
>
> Cédric Brun wrote:
>> Hi Simon and Eike,
>>
>> thanks for your feedback, now I know what I could expect in using
>> optimized implementation, I'll have a try on the hibernate store and see
>> if it is scaling in a better way.
>>
>> You gave me many pointers to explain the performances issues but in fact
>> it seems that the SQL execution is taking so much time (not suprisingly
>> in fact ;) . My first profile was wrong.
>>
>> So I'm installing the hibernate store retrieving all the source code from
>> CVS, and I'm trying to launch the unit tests. I added many jars and
>> everything gets compiled but most of the tests are failling with "
>>
>> "
>> org.hibernate.MappingException: Could not determine type for:
>> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>> at table: cdo_features, for columns: [org.hibernate.mapping.Column(type)]
>> "
>>
>> any hint ?
>>
>> Cédric
>>
>>
>> Simon McDuff wrote:
>>
>>> We noticed a performance degradation in the following case:
>>>
>>> When we fetch a collection of objects that contains collections that we
>>> don't want to use. Ex:
>>>
>>> Model:
>>>
>>> ClassB
>>> {
>>> int foo1;
>>> List childrens1;
>>> List childrens2;
>>> List childrens3;
>>> };
>>>
>>> Code:
>>>
>>> Resource.contents contains objects of type ClassB.
>>>
>>> By fetching instances of classB, we will also fetched some items in the
>>> list(childrens1, childrens2, childrens3) The problem we had.. is we do
>>> not know if we want to use it. SO better not to download them.
>>>
>>> So In our production(we used our own IStore), we never download any
>>> objects from collection. I'm working at the moment to put in place all
>>> feature we used in the current CDO. That one will be part of
>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477 To begin
>>> with,CDORevisionImpl needs to supports list that are 0 even if
>>> referenceChunkSize == 0.
>>>
>>> It is true that we obtain good performance.
>>> Worst case :
>>> 1500 objects /sec (Real fetch from disk. Page that contains the objects,
>>> wasn't in the cache of the filesystem.)
>>>
>>> Best case
>>> 12 000 objects/Sec (Fetch objects that are already in the case of the
>>> filesystem)
>>>
>>> (These statistics always fetch objects from server to client. The cache
>>> client isn't used)
>>>
>>> We have another optimization one with SmartReadAhead but for now only
>>> work
>>> for UI client. (for now) This is something we will integrate as well in
>>> CDO. But for now we concentrate on the other things.
>>>
>>> I used an Object database. Based on my preliminary test without CDO.
>>> I compare
>>>
>>> Hibernate vs Objectivity/EMF :
>>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>>
>>> The worst.. you will get 30 to 240 objects/sec or
>>> the best... you will get 500 to 4000 objects/sec
>>>
>>> Based on your comments... you are getting the worst (with Derby)!! :-)
>>>
>>>
>>> Another comments about insertion.
>>>
>>> We found another problem when we insert data(DOn't know if it is still
>>> true). Resource.Contents is a unique list. Basically each time we insert
>>> one element, it needs to fetch all the collection to see if we already
>>> have this element. Basically.. it is time consuming. In our case we are
>>> not going through this stage to persist our objects... but the objects
>>> are not accessible through that list.
>>>
>>>
>>>
>>> If speed is an issue for you!! We (You included :-)) can optimize it.
>>> What I like is Eike did a really good job doing that framwork and
>>> optimizations are easy to add. Also you can change your back-end without
>>> affecting you client side. (Maybe going to an Object database!!) THis
>>> was one of the purpose of CDO!! :-)
>>>
>>>
>>> Simon
>>>
>>>
>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le message de news:
>>> 483E9332.6050800@sympedia.de... Hi Cédric,
>>>
>>> That sounds interesting!
>>> Comments inline...
>>>
>>>
>>> Cédric Brun schrieb:
>>> Hi Eike,
>>>
>>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>>> measuring read/write throughput when I use CDO (RC2).
>>>
>>> I'm using an embeded-derby database, two different metamodels (complex
>>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>>> Features)
>>>
>>> The test is a "quick and dirty" one so please do not take these figures
>>> too seriously, I would just like to have your input about the "order of
>>> magnitude" I should expect or settings affecting seriously these
>>> performances. I tried both JVM and TCP acceptor but the performances are
>>> quite similar though JVM is a bit faster.
>>> Looks like a sign that the NIO socket channels are efficient ,-)
>>> You used localhost for TCP transport I guess?
>>>
>>>
>>>
>>> * First case : building a small model (10000 elements) with the simple
>>> MM, only adding instances in an empty database :
>>> commiting change on every add I get a 2 objects per second rate.
>>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>> Seems to be consistent with the fact that only for commits there is
>>> remote traffic. And the backend transaction management is another
>>> overhead. So the impact of the number objects/commit starts with
>>> bigger transactions.
>>>
>>>
>>> browsing the just commited elements opening a new transaction and
>>> getting a new Resource instance, I get around 30 000 objets/sec rate
>>> just iterating thanks to Resource.getAllContents().
>>> Hehe, this is a sign that client side caching works well:
>>> All views and transactions of a single session share (and cache) their
>>> object state. Otherwise I'd expect much smaller values.
>>> With his ObjectivityStore implementation Simon (cc'ed) reached rates
>>> of several thousands objects/sec. But the DBStore (which you used) is
>>> not that optimized.
>>>
>>>
>>> * Second case : I built a big model in the database using the complex
>>> metamodel. It contains 700 000 elements. The creation went pretty nicely
>>> (I did not measured it exactly as it was launched during a night, but it
>>> seems I got around 100 objects/sec rate in writting as I was writting
>>> every 700 objects.
>>> Sounds not too bad.
>>>
>>>
>>>
>>> Once the model is built the derby database is 222 MB (which - I heard -
>>> is not so big for derby).
>>> Hmm, I have no extensive experience with Derby.
>>> To be honest, I'm really no database expert (although I can write
>>> simple SQL statements).
>>>
>>>
>>> Retrieving the distant resource and calling eAllContents() on it, I get
>>> a 50 Objects/sec rate at best, which mean my eAllContents() call would
>>> take 4 hours !
>>> Currently I'm not sure what exactly eAllContents() does do (iteration
>>> order and so). There are several CDO features that can have a major
>>> impact on load rates!
>>>
>>> - Collections (xrefs and containment) can be loaded lazily. I don't
>>> mean the target objects, which are lazy by default anyway, but even
>>> the target IDs that are in the collections. We call that "reference
>>> chunking" (but search for a better name is ongoing).
>>>
>>> public interface CDOSession extends CDOProtocolSession,
>>> IContainer<CDOView>
>>> {
>>> public int getReferenceChunkSize();
>>>
>>> public void setReferenceChunkSize(int referenceChunkSize);
>>>
>>>
>>> - Loading collection elements (target objects) can be batched (the
>>> current name is even worse)
>>>
>>> public interface CDOView extends CDOProtocolView, INotifier
>>> {
>>> public int getLoadRevisionCollectionChunkSize();
>>>
>>> public void setLoadRevisionCollectionChunkSize(int
>>> loadRevisionCollectionChunkSize);
>>>
>>>
>>> - One of the most sophisticated feature is automatic model usage
>>> analyzation. The result are fetch rules that are sent to the server.
>>> When later objects are loaded from the server, it can answer with
>>> additional objects that are likely to be needed by the client in the
>>> near future.
>>>
>>> public class CDOViewImpl extends
>>> org.eclipse.net4j.internal.util.event.Notifier implements
>>> CDOView, CDOIDProvider, Adapter.Internal
>>> {
>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>> {
>>> return featureAnalyzer;
>>> }
>>>
>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>> featureAnalyzer)
>>> {
>>> this.featureAnalyzer = featureAnalyzer == null ?
>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>> }
>>>
>>>
>>> - There's a SmartReadAheadThread (if you're curious, please ask
>>> Simon). - Caches can be configured
>>> - ...
>>>
>>>
>>> Known issues of the DBStore:
>>> - When in auditing mode updates are always written as complete inserts
>>> (by design) - When configuring toManyReferences to a value different
>>> from LIKE_ATTRIBUTES (which is not implemented yet) it is very slow. -
>>> No (few) prepared statements are used (see
>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe
>>> others...
>>>
>>> Please note that the DBStore was not written by me to compete with
>>> Hibernate in terms of performance or mapping capabilities. Now that
>>> Martin and I, well in the end mostly Martin, provide a HibernateStore
>>> and this HibernateStore gets more and more features I expect that
>>> there will be a trend away from the DBStore.
>>>
>>>
>>> This is probably linked to the way the mapping to db is done, here's the
>>> config I used (came from CDO-server example if I recall correctly).
>>>
>>>
>>> <repository name="repo1">
>>> <property name="overrideUUID"
>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>> <property name="supportingAudits" value="true"/>
>>> <property name="verifyingRevisions" value="false"/>
>>> <property name="currentLRUCapacity" value="10000"/>
>>> I guess this one could be larger.
>>>
>>>
>>> <property name="revisedLRUCapacity" value="100"/>
>>>
>>> <store type="db">
>>> <!-- type: horizontal | vertical | <any
>>> user-contributed type> --> <mappingStrategy
>>> type="horizontal">
>>> <!-- ONE_TABLE_PER_REFERENCE |
>>> ONE_TABLE_PER_CLASS |
>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>>> <property name="toManyReferences"
>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>
>>> <!-- LIKE_ATTRIBUTES |
>>> LIKE_TO_MANY_REFERENCES --> <property
>>> name="toOneReferences"
>>> value="LIKE_ATTRIBUTES"/>
>>>
>>> <!-- MODEL | STRATEGY -->
>>> <property name="mappingPrecedence"
>>> value="MODEL"/>
>>> This one is obsolete.
>>>
>>>
>>> </mappingStrategy>
>>>
>>> <dbAdapter name="derby-embedded"/>
>>> <dataSource
>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>> databaseName="/tmp/cdodb1"
>>> createDatabase="create"/>
>>>
>>> </store>
>>> </repository>
>>>
>>>
>>>
>>>
>>> What do you think about that ? What throughput do you think I should
>>> expect ? As I said that's very difficult to judge.
>>> I believe it's more appropriate to compare the actual values with
>>> values from other measures (other setups, properties, etc. or even
>>> other products) instead of random expectations. Apologies that I
>>> didn't have the time so far to provide some reference measures myself.
>>> Maybe this gets better now that the team starts to grow slowly ;-)
>>>
>>>
>>> What are the main reasons fur such figures ?
>>> Hmm, I *suspect* you don't like the figures? I can find no evidence
>>> your assessment ;-)
>>>
>>> Ahh,one more note:
>>> Tracing can also have a significant impact on performance and
>>> end-to-end throughput. Turn it off by all means. And please make sure
>>> that it's really turned off, not only invisible (i.e. console is not a
>>> target).
>>>
>>> Cheers
>>> /Eike
>>
>
>


http://cedric.brun.io news and articles on eclipse and eclipse modeling.
Re: [CDO] reading/writting throughput performances [message #619104 is a reply to message #124150] Fri, 30 May 2008 13:21 Go to previous message
Martin Taal is currently offline Martin TaalFriend
Messages: 5468
Registered: July 2009
Senior Member
Hmm, I can understand your headache. Classpath issues are the most annoying errors...

In your junit test source file, what happens when you enter the org.objectweb.asm.Type, can you
source file see this class?
Do you have cglib in some other plugin which does not have asm?

What I also find slightly strange is that cglib is used in this case because normally cdo objects
are not proxied using cglib. Can you post the generated mapping?
How does your cdo initialization code look like?

gr. Martin

Cédric Brun wrote:
> Hi Martin,
>
> and thanks for your quick answer ! what I did is the following :
>
> I used the org.eclipse.emf.cdo.server.hibernate.libraries bundle and added
> that :
>
> Bundle-ClassPath: lib/hibernate3.jar,
> lib/cglib-2.1.3.jar,
> lib/commons-collections-2.1.1.jar,
> lib/dom4j.jar,
> lib/jta.jar
> lib/asm-3.0.jar
>
> and this directive :
>
> Eclipse-BuddyPolicy: dependent
>
> as stated in
> http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>
> Then I don't get the same error, now when I start my own plugin (JUnit
> plugin test) depending on org.eclipse.emf.cdo.server.hibernate.libraries I
> get :
>
> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
> at net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
> at net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
> at net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
> at
> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>
> These jars are such a headache ! Do you know - by chance - what's going
> on ?
>
>
> Cédric
>
>
> Martin Taal wrote:
>
>> Hi Cédric,
>> Most of the time this exception is thrown because hibernate can't find the
>> class. A common cause is that the hibernate jar files are located in a
>> separate plugin which can not reach the classes in other plugins. If you
>> set the eclipse buddy policy of the hibernate plugin to dependent then
>> this can help.
>>
>> gr. Martin
>>
>> Cédric Brun wrote:
>>> Hi Simon and Eike,
>>>
>>> thanks for your feedback, now I know what I could expect in using
>>> optimized implementation, I'll have a try on the hibernate store and see
>>> if it is scaling in a better way.
>>>
>>> You gave me many pointers to explain the performances issues but in fact
>>> it seems that the SQL execution is taking so much time (not suprisingly
>>> in fact ;) . My first profile was wrong.
>>>
>>> So I'm installing the hibernate store retrieving all the source code from
>>> CVS, and I'm trying to launch the unit tests. I added many jars and
>>> everything gets compiled but most of the tests are failling with "
>>>
>>> "
>>> org.hibernate.MappingException: Could not determine type for:
>>> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>> at table: cdo_features, for columns: [org.hibernate.mapping.Column(type)]
>>> "
>>>
>>> any hint ?
>>>
>>> Cédric
>>>
>>>
>>> Simon McDuff wrote:
>>>
>>>> We noticed a performance degradation in the following case:
>>>>
>>>> When we fetch a collection of objects that contains collections that we
>>>> don't want to use. Ex:
>>>>
>>>> Model:
>>>>
>>>> ClassB
>>>> {
>>>> int foo1;
>>>> List childrens1;
>>>> List childrens2;
>>>> List childrens3;
>>>> };
>>>>
>>>> Code:
>>>>
>>>> Resource.contents contains objects of type ClassB.
>>>>
>>>> By fetching instances of classB, we will also fetched some items in the
>>>> list(childrens1, childrens2, childrens3) The problem we had.. is we do
>>>> not know if we want to use it. SO better not to download them.
>>>>
>>>> So In our production(we used our own IStore), we never download any
>>>> objects from collection. I'm working at the moment to put in place all
>>>> feature we used in the current CDO. That one will be part of
>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477 To begin
>>>> with,CDORevisionImpl needs to supports list that are 0 even if
>>>> referenceChunkSize == 0.
>>>>
>>>> It is true that we obtain good performance.
>>>> Worst case :
>>>> 1500 objects /sec (Real fetch from disk. Page that contains the objects,
>>>> wasn't in the cache of the filesystem.)
>>>>
>>>> Best case
>>>> 12 000 objects/Sec (Fetch objects that are already in the case of the
>>>> filesystem)
>>>>
>>>> (These statistics always fetch objects from server to client. The cache
>>>> client isn't used)
>>>>
>>>> We have another optimization one with SmartReadAhead but for now only
>>>> work
>>>> for UI client. (for now) This is something we will integrate as well in
>>>> CDO. But for now we concentrate on the other things.
>>>>
>>>> I used an Object database. Based on my preliminary test without CDO.
>>>> I compare
>>>>
>>>> Hibernate vs Objectivity/EMF :
>>>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>>>
>>>> The worst.. you will get 30 to 240 objects/sec or
>>>> the best... you will get 500 to 4000 objects/sec
>>>>
>>>> Based on your comments... you are getting the worst (with Derby)!! :-)
>>>>
>>>>
>>>> Another comments about insertion.
>>>>
>>>> We found another problem when we insert data(DOn't know if it is still
>>>> true). Resource.Contents is a unique list. Basically each time we insert
>>>> one element, it needs to fetch all the collection to see if we already
>>>> have this element. Basically.. it is time consuming. In our case we are
>>>> not going through this stage to persist our objects... but the objects
>>>> are not accessible through that list.
>>>>
>>>>
>>>>
>>>> If speed is an issue for you!! We (You included :-)) can optimize it.
>>>> What I like is Eike did a really good job doing that framwork and
>>>> optimizations are easy to add. Also you can change your back-end without
>>>> affecting you client side. (Maybe going to an Object database!!) THis
>>>> was one of the purpose of CDO!! :-)
>>>>
>>>>
>>>> Simon
>>>>
>>>>
>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le message de news:
>>>> 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>
>>>> That sounds interesting!
>>>> Comments inline...
>>>>
>>>>
>>>> Cédric Brun schrieb:
>>>> Hi Eike,
>>>>
>>>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>>>> measuring read/write throughput when I use CDO (RC2).
>>>>
>>>> I'm using an embeded-derby database, two different metamodels (complex
>>>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>>>> Features)
>>>>
>>>> The test is a "quick and dirty" one so please do not take these figures
>>>> too seriously, I would just like to have your input about the "order of
>>>> magnitude" I should expect or settings affecting seriously these
>>>> performances. I tried both JVM and TCP acceptor but the performances are
>>>> quite similar though JVM is a bit faster.
>>>> Looks like a sign that the NIO socket channels are efficient ,-)
>>>> You used localhost for TCP transport I guess?
>>>>
>>>>
>>>>
>>>> * First case : building a small model (10000 elements) with the simple
>>>> MM, only adding instances in an empty database :
>>>> commiting change on every add I get a 2 objects per second rate.
>>>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>>> Seems to be consistent with the fact that only for commits there is
>>>> remote traffic. And the backend transaction management is another
>>>> overhead. So the impact of the number objects/commit starts with
>>>> bigger transactions.
>>>>
>>>>
>>>> browsing the just commited elements opening a new transaction and
>>>> getting a new Resource instance, I get around 30 000 objets/sec rate
>>>> just iterating thanks to Resource.getAllContents().
>>>> Hehe, this is a sign that client side caching works well:
>>>> All views and transactions of a single session share (and cache) their
>>>> object state. Otherwise I'd expect much smaller values.
>>>> With his ObjectivityStore implementation Simon (cc'ed) reached rates
>>>> of several thousands objects/sec. But the DBStore (which you used) is
>>>> not that optimized.
>>>>
>>>>
>>>> * Second case : I built a big model in the database using the complex
>>>> metamodel. It contains 700 000 elements. The creation went pretty nicely
>>>> (I did not measured it exactly as it was launched during a night, but it
>>>> seems I got around 100 objects/sec rate in writting as I was writting
>>>> every 700 objects.
>>>> Sounds not too bad.
>>>>
>>>>
>>>>
>>>> Once the model is built the derby database is 222 MB (which - I heard -
>>>> is not so big for derby).
>>>> Hmm, I have no extensive experience with Derby.
>>>> To be honest, I'm really no database expert (although I can write
>>>> simple SQL statements).
>>>>
>>>>
>>>> Retrieving the distant resource and calling eAllContents() on it, I get
>>>> a 50 Objects/sec rate at best, which mean my eAllContents() call would
>>>> take 4 hours !
>>>> Currently I'm not sure what exactly eAllContents() does do (iteration
>>>> order and so). There are several CDO features that can have a major
>>>> impact on load rates!
>>>>
>>>> - Collections (xrefs and containment) can be loaded lazily. I don't
>>>> mean the target objects, which are lazy by default anyway, but even
>>>> the target IDs that are in the collections. We call that "reference
>>>> chunking" (but search for a better name is ongoing).
>>>>
>>>> public interface CDOSession extends CDOProtocolSession,
>>>> IContainer<CDOView>
>>>> {
>>>> public int getReferenceChunkSize();
>>>>
>>>> public void setReferenceChunkSize(int referenceChunkSize);
>>>>
>>>>
>>>> - Loading collection elements (target objects) can be batched (the
>>>> current name is even worse)
>>>>
>>>> public interface CDOView extends CDOProtocolView, INotifier
>>>> {
>>>> public int getLoadRevisionCollectionChunkSize();
>>>>
>>>> public void setLoadRevisionCollectionChunkSize(int
>>>> loadRevisionCollectionChunkSize);
>>>>
>>>>
>>>> - One of the most sophisticated feature is automatic model usage
>>>> analyzation. The result are fetch rules that are sent to the server.
>>>> When later objects are loaded from the server, it can answer with
>>>> additional objects that are likely to be needed by the client in the
>>>> near future.
>>>>
>>>> public class CDOViewImpl extends
>>>> org.eclipse.net4j.internal.util.event.Notifier implements
>>>> CDOView, CDOIDProvider, Adapter.Internal
>>>> {
>>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>>> {
>>>> return featureAnalyzer;
>>>> }
>>>>
>>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>>> featureAnalyzer)
>>>> {
>>>> this.featureAnalyzer = featureAnalyzer == null ?
>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>> }
>>>>
>>>>
>>>> - There's a SmartReadAheadThread (if you're curious, please ask
>>>> Simon). - Caches can be configured
>>>> - ...
>>>>
>>>>
>>>> Known issues of the DBStore:
>>>> - When in auditing mode updates are always written as complete inserts
>>>> (by design) - When configuring toManyReferences to a value different
>>>> from LIKE_ATTRIBUTES (which is not implemented yet) it is very slow. -
>>>> No (few) prepared statements are used (see
>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe
>>>> others...
>>>>
>>>> Please note that the DBStore was not written by me to compete with
>>>> Hibernate in terms of performance or mapping capabilities. Now that
>>>> Martin and I, well in the end mostly Martin, provide a HibernateStore
>>>> and this HibernateStore gets more and more features I expect that
>>>> there will be a trend away from the DBStore.
>>>>
>>>>
>>>> This is probably linked to the way the mapping to db is done, here's the
>>>> config I used (came from CDO-server example if I recall correctly).
>>>>
>>>>
>>>> <repository name="repo1">
>>>> <property name="overrideUUID"
>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>> <property name="supportingAudits" value="true"/>
>>>> <property name="verifyingRevisions" value="false"/>
>>>> <property name="currentLRUCapacity" value="10000"/>
>>>> I guess this one could be larger.
>>>>
>>>>
>>>> <property name="revisedLRUCapacity" value="100"/>
>>>>
>>>> <store type="db">
>>>> <!-- type: horizontal | vertical | <any
>>>> user-contributed type> --> <mappingStrategy
>>>> type="horizontal">
>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>> ONE_TABLE_PER_CLASS |
>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>>>> <property name="toManyReferences"
>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>
>>>> <!-- LIKE_ATTRIBUTES |
>>>> LIKE_TO_MANY_REFERENCES --> <property
>>>> name="toOneReferences"
>>>> value="LIKE_ATTRIBUTES"/>
>>>>
>>>> <!-- MODEL | STRATEGY -->
>>>> <property name="mappingPrecedence"
>>>> value="MODEL"/>
>>>> This one is obsolete.
>>>>
>>>>
>>>> </mappingStrategy>
>>>>
>>>> <dbAdapter name="derby-embedded"/>
>>>> <dataSource
>>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>> databaseName="/tmp/cdodb1"
>>>> createDatabase="create"/>
>>>>
>>>> </store>
>>>> </repository>
>>>>
>>>>
>>>>
>>>>
>>>> What do you think about that ? What throughput do you think I should
>>>> expect ? As I said that's very difficult to judge.
>>>> I believe it's more appropriate to compare the actual values with
>>>> values from other measures (other setups, properties, etc. or even
>>>> other products) instead of random expectations. Apologies that I
>>>> didn't have the time so far to provide some reference measures myself.
>>>> Maybe this gets better now that the team starts to grow slowly ;-)
>>>>
>>>>
>>>> What are the main reasons fur such figures ?
>>>> Hmm, I *suspect* you don't like the figures? I can find no evidence
>>>> your assessment ;-)
>>>>
>>>> Ahh,one more note:
>>>> Tracing can also have a significant impact on performance and
>>>> end-to-end throughput. Turn it off by all means. And please make sure
>>>> that it's really turned off, not only invisible (i.e. console is not a
>>>> target).
>>>>
>>>> Cheers
>>>> /Eike
>>
>


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Cell: +31 (0)6 288 48 943
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: [CDO] reading/writting throughput performances [message #619109 is a reply to message #124164] Fri, 30 May 2008 14:34 Go to previous message
Cedric Brun is currently offline Cedric BrunFriend
Messages: 431
Registered: July 2009
Senior Member
I tried to do the same things as in the CDO/Hibernate unit tests.

Here's a zipped project trying to launch an HibernateServer with a unit test
adding elements.
( http://cedric.brun.free.fr/Tortoose/org.eclipse.cdo-teneo-te st.zip) The
libs I found are included in the same project, but I still have no luck and
failling at :

at
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
... 62 more
Caused by: java.lang.NoClassDefFoundError:
org/hibernate/proxy/HibernateProxy
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
... 68 more

please note that the test project I provide really is a "quick and ugly" one
^^


Cédric



Martin Taal wrote:

> Hmm, I can understand your headache. Classpath issues are the most
> annoying errors...
>
> In your junit test source file, what happens when you enter the
> org.objectweb.asm.Type, can you source file see this class?
> Do you have cglib in some other plugin which does not have asm?
>
> What I also find slightly strange is that cglib is used in this case
> because normally cdo objects are not proxied using cglib. Can you post the
> generated mapping? How does your cdo initialization code look like?
>
> gr. Martin
>
> Cédric Brun wrote:
>> Hi Martin,
>>
>> and thanks for your quick answer ! what I did is the following :
>>
>> I used the org.eclipse.emf.cdo.server.hibernate.libraries bundle and
>> added that :
>>
>> Bundle-ClassPath: lib/hibernate3.jar,
>> lib/cglib-2.1.3.jar,
>> lib/commons-collections-2.1.1.jar,
>> lib/dom4j.jar,
>> lib/jta.jar
>> lib/asm-3.0.jar
>>
>> and this directive :
>>
>> Eclipse-BuddyPolicy: dependent
>>
>> as stated in
>> http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>>
>> Then I don't get the same error, now when I start my own plugin (JUnit
>> plugin test) depending on org.eclipse.emf.cdo.server.hibernate.libraries
>> I get :
>>
>> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
>> at net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
>> at net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
>> at net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
>> at
>>
org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>>
>> These jars are such a headache ! Do you know - by chance - what's going
>> on ?
>>
>>
>> Cédric
>>
>>
>> Martin Taal wrote:
>>
>>> Hi Cédric,
>>> Most of the time this exception is thrown because hibernate can't find
>>> the class. A common cause is that the hibernate jar files are located in
>>> a separate plugin which can not reach the classes in other plugins. If
>>> you set the eclipse buddy policy of the hibernate plugin to dependent
>>> then this can help.
>>>
>>> gr. Martin
>>>
>>> Cédric Brun wrote:
>>>> Hi Simon and Eike,
>>>>
>>>> thanks for your feedback, now I know what I could expect in using
>>>> optimized implementation, I'll have a try on the hibernate store and
>>>> see if it is scaling in a better way.
>>>>
>>>> You gave me many pointers to explain the performances issues but in
>>>> fact it seems that the SQL execution is taking so much time (not
>>>> suprisingly in fact ;) . My first profile was wrong.
>>>>
>>>> So I'm installing the hibernate store retrieving all the source code
>>>> from CVS, and I'm trying to launch the unit tests. I added many jars
>>>> and everything gets compiled but most of the tests are failling with "
>>>>
>>>> "
>>>> org.hibernate.MappingException: Could not determine type for:
>>>> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>>> at table: cdo_features, for columns:
>>>> [org.hibernate.mapping.Column(type)] "
>>>>
>>>> any hint ?
>>>>
>>>> Cédric
>>>>
>>>>
>>>> Simon McDuff wrote:
>>>>
>>>>> We noticed a performance degradation in the following case:
>>>>>
>>>>> When we fetch a collection of objects that contains collections that
>>>>> we don't want to use. Ex:
>>>>>
>>>>> Model:
>>>>>
>>>>> ClassB
>>>>> {
>>>>> int foo1;
>>>>> List childrens1;
>>>>> List childrens2;
>>>>> List childrens3;
>>>>> };
>>>>>
>>>>> Code:
>>>>>
>>>>> Resource.contents contains objects of type ClassB.
>>>>>
>>>>> By fetching instances of classB, we will also fetched some items in
>>>>> the list(childrens1, childrens2, childrens3) The problem we had.. is
>>>>> we do not know if we want to use it. SO better not to download them.
>>>>>
>>>>> So In our production(we used our own IStore), we never download any
>>>>> objects from collection. I'm working at the moment to put in place all
>>>>> feature we used in the current CDO. That one will be part of
>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477 To begin
>>>>> with,CDORevisionImpl needs to supports list that are 0 even if
>>>>> referenceChunkSize == 0.
>>>>>
>>>>> It is true that we obtain good performance.
>>>>> Worst case :
>>>>> 1500 objects /sec (Real fetch from disk. Page that contains the
>>>>> objects, wasn't in the cache of the filesystem.)
>>>>>
>>>>> Best case
>>>>> 12 000 objects/Sec (Fetch objects that are already in the case of the
>>>>> filesystem)
>>>>>
>>>>> (These statistics always fetch objects from server to client. The
>>>>> cache client isn't used)
>>>>>
>>>>> We have another optimization one with SmartReadAhead but for now only
>>>>> work
>>>>> for UI client. (for now) This is something we will integrate as well
>>>>> in CDO. But for now we concentrate on the other things.
>>>>>
>>>>> I used an Object database. Based on my preliminary test without CDO.
>>>>> I compare
>>>>>
>>>>> Hibernate vs Objectivity/EMF :
>>>>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>>>>
>>>>> The worst.. you will get 30 to 240 objects/sec or
>>>>> the best... you will get 500 to 4000 objects/sec
>>>>>
>>>>> Based on your comments... you are getting the worst (with Derby)!! :-)
>>>>>
>>>>>
>>>>> Another comments about insertion.
>>>>>
>>>>> We found another problem when we insert data(DOn't know if it is still
>>>>> true). Resource.Contents is a unique list. Basically each time we
>>>>> insert one element, it needs to fetch all the collection to see if we
>>>>> already have this element. Basically.. it is time consuming. In our
>>>>> case we are not going through this stage to persist our objects... but
>>>>> the objects are not accessible through that list.
>>>>>
>>>>>
>>>>>
>>>>> If speed is an issue for you!! We (You included :-)) can optimize it.
>>>>> What I like is Eike did a really good job doing that framwork and
>>>>> optimizations are easy to add. Also you can change your back-end
>>>>> without affecting you client side. (Maybe going to an Object
>>>>> database!!) THis was one of the purpose of CDO!! :-)
>>>>>
>>>>>
>>>>> Simon
>>>>>
>>>>>
>>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le message de
>>>>> news: 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>>
>>>>> That sounds interesting!
>>>>> Comments inline...
>>>>>
>>>>>
>>>>> Cédric Brun schrieb:
>>>>> Hi Eike,
>>>>>
>>>>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>>>>> measuring read/write throughput when I use CDO (RC2).
>>>>>
>>>>> I'm using an embeded-derby database, two different metamodels (complex
>>>>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>>>>> Features)
>>>>>
>>>>> The test is a "quick and dirty" one so please do not take these
>>>>> figures too seriously, I would just like to have your input about the
>>>>> "order of magnitude" I should expect or settings affecting seriously
>>>>> these performances. I tried both JVM and TCP acceptor but the
>>>>> performances are quite similar though JVM is a bit faster.
>>>>> Looks like a sign that the NIO socket channels are efficient ,-)
>>>>> You used localhost for TCP transport I guess?
>>>>>
>>>>>
>>>>>
>>>>> * First case : building a small model (10000 elements) with the simple
>>>>> MM, only adding instances in an empty database :
>>>>> commiting change on every add I get a 2 objects per second rate.
>>>>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>>>> Seems to be consistent with the fact that only for commits there is
>>>>> remote traffic. And the backend transaction management is another
>>>>> overhead. So the impact of the number objects/commit starts with
>>>>> bigger transactions.
>>>>>
>>>>>
>>>>> browsing the just commited elements opening a new transaction and
>>>>> getting a new Resource instance, I get around 30 000 objets/sec rate
>>>>> just iterating thanks to Resource.getAllContents().
>>>>> Hehe, this is a sign that client side caching works well:
>>>>> All views and transactions of a single session share (and cache)
>>>>> their object state. Otherwise I'd expect much smaller values.
>>>>> With his ObjectivityStore implementation Simon (cc'ed) reached rates
>>>>> of several thousands objects/sec. But the DBStore (which you used)
>>>>> is not that optimized.
>>>>>
>>>>>
>>>>> * Second case : I built a big model in the database using the complex
>>>>> metamodel. It contains 700 000 elements. The creation went pretty
>>>>> nicely (I did not measured it exactly as it was launched during a
>>>>> night, but it seems I got around 100 objects/sec rate in writting as I
>>>>> was writting every 700 objects.
>>>>> Sounds not too bad.
>>>>>
>>>>>
>>>>>
>>>>> Once the model is built the derby database is 222 MB (which - I heard
>>>>> - is not so big for derby).
>>>>> Hmm, I have no extensive experience with Derby.
>>>>> To be honest, I'm really no database expert (although I can write
>>>>> simple SQL statements).
>>>>>
>>>>>
>>>>> Retrieving the distant resource and calling eAllContents() on it, I
>>>>> get a 50 Objects/sec rate at best, which mean my eAllContents() call
>>>>> would take 4 hours !
>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>> (iteration order and so). There are several CDO features that can
>>>>> have a major impact on load rates!
>>>>>
>>>>> - Collections (xrefs and containment) can be loaded lazily. I don't
>>>>> mean the target objects, which are lazy by default anyway, but even
>>>>> the target IDs that are in the collections. We call that "reference
>>>>> chunking" (but search for a better name is ongoing).
>>>>>
>>>>> public interface CDOSession extends CDOProtocolSession,
>>>>> IContainer<CDOView>
>>>>> {
>>>>> public int getReferenceChunkSize();
>>>>>
>>>>> public void setReferenceChunkSize(int referenceChunkSize);
>>>>>
>>>>>
>>>>> - Loading collection elements (target objects) can be batched (the
>>>>> current name is even worse)
>>>>>
>>>>> public interface CDOView extends CDOProtocolView, INotifier
>>>>> {
>>>>> public int getLoadRevisionCollectionChunkSize();
>>>>>
>>>>> public void setLoadRevisionCollectionChunkSize(int
>>>>> loadRevisionCollectionChunkSize);
>>>>>
>>>>>
>>>>> - One of the most sophisticated feature is automatic model usage
>>>>> analyzation. The result are fetch rules that are sent to the server.
>>>>> When later objects are loaded from the server, it can answer with
>>>>> additional objects that are likely to be needed by the client in the
>>>>> near future.
>>>>>
>>>>> public class CDOViewImpl extends
>>>>> org.eclipse.net4j.internal.util.event.Notifier implements
>>>>> CDOView, CDOIDProvider, Adapter.Internal
>>>>> {
>>>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>> {
>>>>> return featureAnalyzer;
>>>>> }
>>>>>
>>>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>> featureAnalyzer)
>>>>> {
>>>>> this.featureAnalyzer = featureAnalyzer == null ?
>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>> }
>>>>>
>>>>>
>>>>> - There's a SmartReadAheadThread (if you're curious, please ask
>>>>> Simon). - Caches can be configured
>>>>> - ...
>>>>>
>>>>>
>>>>> Known issues of the DBStore:
>>>>> - When in auditing mode updates are always written as complete
>>>>> inserts (by design) - When configuring toManyReferences to a value
>>>>> different from LIKE_ATTRIBUTES (which is not implemented yet) it is
>>>>> very slow. - No (few) prepared statements are used (see
>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe
>>>>> others...
>>>>>
>>>>> Please note that the DBStore was not written by me to compete with
>>>>> Hibernate in terms of performance or mapping capabilities. Now that
>>>>> Martin and I, well in the end mostly Martin, provide a
>>>>> HibernateStore and this HibernateStore gets more and more features I
>>>>> expect that there will be a trend away from the DBStore.
>>>>>
>>>>>
>>>>> This is probably linked to the way the mapping to db is done, here's
>>>>> the config I used (came from CDO-server example if I recall
>>>>> correctly).
>>>>>
>>>>>
>>>>> <repository name="repo1">
>>>>> <property name="overrideUUID"
>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>> <property name="supportingAudits" value="true"/>
>>>>> <property name="verifyingRevisions" value="false"/>
>>>>> <property name="currentLRUCapacity" value="10000"/>
>>>>> I guess this one could be larger.
>>>>>
>>>>>
>>>>> <property name="revisedLRUCapacity" value="100"/>
>>>>>
>>>>> <store type="db">
>>>>> <!-- type: horizontal | vertical | <any
>>>>> user-contributed type> --> <mappingStrategy
>>>>> type="horizontal">
>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>> ONE_TABLE_PER_CLASS |
>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>>>>> <property name="toManyReferences"
>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>
>>>>> <!-- LIKE_ATTRIBUTES |
>>>>> LIKE_TO_MANY_REFERENCES --> <property
>>>>> name="toOneReferences"
>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>
>>>>> <!-- MODEL | STRATEGY -->
>>>>> <property name="mappingPrecedence"
>>>>> value="MODEL"/>
>>>>> This one is obsolete.
>>>>>
>>>>>
>>>>> </mappingStrategy>
>>>>>
>>>>> <dbAdapter name="derby-embedded"/>
>>>>> <dataSource
>>>>>
class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>>> databaseName="/tmp/cdodb1"
>>>>> createDatabase="create"/>
>>>>>
>>>>> </store>
>>>>> </repository>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> What do you think about that ? What throughput do you think I should
>>>>> expect ? As I said that's very difficult to judge.
>>>>> I believe it's more appropriate to compare the actual values with
>>>>> values from other measures (other setups, properties, etc. or even
>>>>> other products) instead of random expectations. Apologies that I
>>>>> didn't have the time so far to provide some reference measures
>>>>> myself. Maybe this gets better now that the team starts to grow
>>>>> slowly ;-)
>>>>>
>>>>>
>>>>> What are the main reasons fur such figures ?
>>>>> Hmm, I *suspect* you don't like the figures? I can find no evidence
>>>>> your assessment ;-)
>>>>>
>>>>> Ahh,one more note:
>>>>> Tracing can also have a significant impact on performance and
>>>>> end-to-end throughput. Turn it off by all means. And please make
>>>>> sure that it's really turned off, not only invisible (i.e. console
>>>>> is not a target).
>>>>>
>>>>> Cheers
>>>>> /Eike
>>>
>>
>
>


http://cedric.brun.io news and articles on eclipse and eclipse modeling.
Re: [CDO] reading/writting throughput performances [message #619110 is a reply to message #124231] Fri, 30 May 2008 14:30 Go to previous message
Martin Taal is currently offline Martin TaalFriend
Messages: 5468
Registered: July 2009
Senior Member
I am downloading your zip-file. I don't have time to look at it right away but I will do my best
later this evening.

gr. Martin

Cédric Brun wrote:
> I tried to do the same things as in the CDO/Hibernate unit tests.
>
> Here's a zipped project trying to launch an HibernateServer with a unit test
> adding elements.
> ( http://cedric.brun.free.fr/Tortoose/org.eclipse.cdo-teneo-te st.zip) The
> libs I found are included in the same project, but I still have no luck and
> failling at :
>
> at
> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
> ... 62 more
> Caused by: java.lang.NoClassDefFoundError:
> org/hibernate/proxy/HibernateProxy
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
> ... 68 more
>
> please note that the test project I provide really is a "quick and ugly" one
> ^^
>
>
> Cédric
>
>
>
> Martin Taal wrote:
>
>> Hmm, I can understand your headache. Classpath issues are the most
>> annoying errors...
>>
>> In your junit test source file, what happens when you enter the
>> org.objectweb.asm.Type, can you source file see this class?
>> Do you have cglib in some other plugin which does not have asm?
>>
>> What I also find slightly strange is that cglib is used in this case
>> because normally cdo objects are not proxied using cglib. Can you post the
>> generated mapping? How does your cdo initialization code look like?
>>
>> gr. Martin
>>
>> Cédric Brun wrote:
>>> Hi Martin,
>>>
>>> and thanks for your quick answer ! what I did is the following :
>>>
>>> I used the org.eclipse.emf.cdo.server.hibernate.libraries bundle and
>>> added that :
>>>
>>> Bundle-ClassPath: lib/hibernate3.jar,
>>> lib/cglib-2.1.3.jar,
>>> lib/commons-collections-2.1.1.jar,
>>> lib/dom4j.jar,
>>> lib/jta.jar
>>> lib/asm-3.0.jar
>>>
>>> and this directive :
>>>
>>> Eclipse-BuddyPolicy: dependent
>>>
>>> as stated in
>>> http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>>>
>>> Then I don't get the same error, now when I start my own plugin (JUnit
>>> plugin test) depending on org.eclipse.emf.cdo.server.hibernate.libraries
>>> I get :
>>>
>>> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
>>> at net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
>>> at net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
>>> at net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
>>> at
>>>
> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>>> These jars are such a headache ! Do you know - by chance - what's going
>>> on ?
>>>
>>>
>>> Cédric
>>>
>>>
>>> Martin Taal wrote:
>>>
>>>> Hi Cédric,
>>>> Most of the time this exception is thrown because hibernate can't find
>>>> the class. A common cause is that the hibernate jar files are located in
>>>> a separate plugin which can not reach the classes in other plugins. If
>>>> you set the eclipse buddy policy of the hibernate plugin to dependent
>>>> then this can help.
>>>>
>>>> gr. Martin
>>>>
>>>> Cédric Brun wrote:
>>>>> Hi Simon and Eike,
>>>>>
>>>>> thanks for your feedback, now I know what I could expect in using
>>>>> optimized implementation, I'll have a try on the hibernate store and
>>>>> see if it is scaling in a better way.
>>>>>
>>>>> You gave me many pointers to explain the performances issues but in
>>>>> fact it seems that the SQL execution is taking so much time (not
>>>>> suprisingly in fact ;) . My first profile was wrong.
>>>>>
>>>>> So I'm installing the hibernate store retrieving all the source code
>>>>> from CVS, and I'm trying to launch the unit tests. I added many jars
>>>>> and everything gets compiled but most of the tests are failling with "
>>>>>
>>>>> "
>>>>> org.hibernate.MappingException: Could not determine type for:
>>>>> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>>>> at table: cdo_features, for columns:
>>>>> [org.hibernate.mapping.Column(type)] "
>>>>>
>>>>> any hint ?
>>>>>
>>>>> Cédric
>>>>>
>>>>>
>>>>> Simon McDuff wrote:
>>>>>
>>>>>> We noticed a performance degradation in the following case:
>>>>>>
>>>>>> When we fetch a collection of objects that contains collections that
>>>>>> we don't want to use. Ex:
>>>>>>
>>>>>> Model:
>>>>>>
>>>>>> ClassB
>>>>>> {
>>>>>> int foo1;
>>>>>> List childrens1;
>>>>>> List childrens2;
>>>>>> List childrens3;
>>>>>> };
>>>>>>
>>>>>> Code:
>>>>>>
>>>>>> Resource.contents contains objects of type ClassB.
>>>>>>
>>>>>> By fetching instances of classB, we will also fetched some items in
>>>>>> the list(childrens1, childrens2, childrens3) The problem we had.. is
>>>>>> we do not know if we want to use it. SO better not to download them.
>>>>>>
>>>>>> So In our production(we used our own IStore), we never download any
>>>>>> objects from collection. I'm working at the moment to put in place all
>>>>>> feature we used in the current CDO. That one will be part of
>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477 To begin
>>>>>> with,CDORevisionImpl needs to supports list that are 0 even if
>>>>>> referenceChunkSize == 0.
>>>>>>
>>>>>> It is true that we obtain good performance.
>>>>>> Worst case :
>>>>>> 1500 objects /sec (Real fetch from disk. Page that contains the
>>>>>> objects, wasn't in the cache of the filesystem.)
>>>>>>
>>>>>> Best case
>>>>>> 12 000 objects/Sec (Fetch objects that are already in the case of the
>>>>>> filesystem)
>>>>>>
>>>>>> (These statistics always fetch objects from server to client. The
>>>>>> cache client isn't used)
>>>>>>
>>>>>> We have another optimization one with SmartReadAhead but for now only
>>>>>> work
>>>>>> for UI client. (for now) This is something we will integrate as well
>>>>>> in CDO. But for now we concentrate on the other things.
>>>>>>
>>>>>> I used an Object database. Based on my preliminary test without CDO.
>>>>>> I compare
>>>>>>
>>>>>> Hibernate vs Objectivity/EMF :
>>>>>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>>>>>
>>>>>> The worst.. you will get 30 to 240 objects/sec or
>>>>>> the best... you will get 500 to 4000 objects/sec
>>>>>>
>>>>>> Based on your comments... you are getting the worst (with Derby)!! :-)
>>>>>>
>>>>>>
>>>>>> Another comments about insertion.
>>>>>>
>>>>>> We found another problem when we insert data(DOn't know if it is still
>>>>>> true). Resource.Contents is a unique list. Basically each time we
>>>>>> insert one element, it needs to fetch all the collection to see if we
>>>>>> already have this element. Basically.. it is time consuming. In our
>>>>>> case we are not going through this stage to persist our objects... but
>>>>>> the objects are not accessible through that list.
>>>>>>
>>>>>>
>>>>>>
>>>>>> If speed is an issue for you!! We (You included :-)) can optimize it.
>>>>>> What I like is Eike did a really good job doing that framwork and
>>>>>> optimizations are easy to add. Also you can change your back-end
>>>>>> without affecting you client side. (Maybe going to an Object
>>>>>> database!!) THis was one of the purpose of CDO!! :-)
>>>>>>
>>>>>>
>>>>>> Simon
>>>>>>
>>>>>>
>>>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le message de
>>>>>> news: 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>>>
>>>>>> That sounds interesting!
>>>>>> Comments inline...
>>>>>>
>>>>>>
>>>>>> Cédric Brun schrieb:
>>>>>> Hi Eike,
>>>>>>
>>>>>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>>>>>> measuring read/write throughput when I use CDO (RC2).
>>>>>>
>>>>>> I'm using an embeded-derby database, two different metamodels (complex
>>>>>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>>>>>> Features)
>>>>>>
>>>>>> The test is a "quick and dirty" one so please do not take these
>>>>>> figures too seriously, I would just like to have your input about the
>>>>>> "order of magnitude" I should expect or settings affecting seriously
>>>>>> these performances. I tried both JVM and TCP acceptor but the
>>>>>> performances are quite similar though JVM is a bit faster.
>>>>>> Looks like a sign that the NIO socket channels are efficient ,-)
>>>>>> You used localhost for TCP transport I guess?
>>>>>>
>>>>>>
>>>>>>
>>>>>> * First case : building a small model (10000 elements) with the simple
>>>>>> MM, only adding instances in an empty database :
>>>>>> commiting change on every add I get a 2 objects per second rate.
>>>>>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>>>>> Seems to be consistent with the fact that only for commits there is
>>>>>> remote traffic. And the backend transaction management is another
>>>>>> overhead. So the impact of the number objects/commit starts with
>>>>>> bigger transactions.
>>>>>>
>>>>>>
>>>>>> browsing the just commited elements opening a new transaction and
>>>>>> getting a new Resource instance, I get around 30 000 objets/sec rate
>>>>>> just iterating thanks to Resource.getAllContents().
>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>> All views and transactions of a single session share (and cache)
>>>>>> their object state. Otherwise I'd expect much smaller values.
>>>>>> With his ObjectivityStore implementation Simon (cc'ed) reached rates
>>>>>> of several thousands objects/sec. But the DBStore (which you used)
>>>>>> is not that optimized.
>>>>>>
>>>>>>
>>>>>> * Second case : I built a big model in the database using the complex
>>>>>> metamodel. It contains 700 000 elements. The creation went pretty
>>>>>> nicely (I did not measured it exactly as it was launched during a
>>>>>> night, but it seems I got around 100 objects/sec rate in writting as I
>>>>>> was writting every 700 objects.
>>>>>> Sounds not too bad.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Once the model is built the derby database is 222 MB (which - I heard
>>>>>> - is not so big for derby).
>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>> To be honest, I'm really no database expert (although I can write
>>>>>> simple SQL statements).
>>>>>>
>>>>>>
>>>>>> Retrieving the distant resource and calling eAllContents() on it, I
>>>>>> get a 50 Objects/sec rate at best, which mean my eAllContents() call
>>>>>> would take 4 hours !
>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>> (iteration order and so). There are several CDO features that can
>>>>>> have a major impact on load rates!
>>>>>>
>>>>>> - Collections (xrefs and containment) can be loaded lazily. I don't
>>>>>> mean the target objects, which are lazy by default anyway, but even
>>>>>> the target IDs that are in the collections. We call that "reference
>>>>>> chunking" (but search for a better name is ongoing).
>>>>>>
>>>>>> public interface CDOSession extends CDOProtocolSession,
>>>>>> IContainer<CDOView>
>>>>>> {
>>>>>> public int getReferenceChunkSize();
>>>>>>
>>>>>> public void setReferenceChunkSize(int referenceChunkSize);
>>>>>>
>>>>>>
>>>>>> - Loading collection elements (target objects) can be batched (the
>>>>>> current name is even worse)
>>>>>>
>>>>>> public interface CDOView extends CDOProtocolView, INotifier
>>>>>> {
>>>>>> public int getLoadRevisionCollectionChunkSize();
>>>>>>
>>>>>> public void setLoadRevisionCollectionChunkSize(int
>>>>>> loadRevisionCollectionChunkSize);
>>>>>>
>>>>>>
>>>>>> - One of the most sophisticated feature is automatic model usage
>>>>>> analyzation. The result are fetch rules that are sent to the server.
>>>>>> When later objects are loaded from the server, it can answer with
>>>>>> additional objects that are likely to be needed by the client in the
>>>>>> near future.
>>>>>>
>>>>>> public class CDOViewImpl extends
>>>>>> org.eclipse.net4j.internal.util.event.Notifier implements
>>>>>> CDOView, CDOIDProvider, Adapter.Internal
>>>>>> {
>>>>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>>> {
>>>>>> return featureAnalyzer;
>>>>>> }
>>>>>>
>>>>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>>> featureAnalyzer)
>>>>>> {
>>>>>> this.featureAnalyzer = featureAnalyzer == null ?
>>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>>> }
>>>>>>
>>>>>>
>>>>>> - There's a SmartReadAheadThread (if you're curious, please ask
>>>>>> Simon). - Caches can be configured
>>>>>> - ...
>>>>>>
>>>>>>
>>>>>> Known issues of the DBStore:
>>>>>> - When in auditing mode updates are always written as complete
>>>>>> inserts (by design) - When configuring toManyReferences to a value
>>>>>> different from LIKE_ATTRIBUTES (which is not implemented yet) it is
>>>>>> very slow. - No (few) prepared statements are used (see
>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe
>>>>>> others...
>>>>>>
>>>>>> Please note that the DBStore was not written by me to compete with
>>>>>> Hibernate in terms of performance or mapping capabilities. Now that
>>>>>> Martin and I, well in the end mostly Martin, provide a
>>>>>> HibernateStore and this HibernateStore gets more and more features I
>>>>>> expect that there will be a trend away from the DBStore.
>>>>>>
>>>>>>
>>>>>> This is probably linked to the way the mapping to db is done, here's
>>>>>> the config I used (came from CDO-server example if I recall
>>>>>> correctly).
>>>>>>
>>>>>>
>>>>>> <repository name="repo1">
>>>>>> <property name="overrideUUID"
>>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>>> <property name="supportingAudits" value="true"/>
>>>>>> <property name="verifyingRevisions" value="false"/>
>>>>>> <property name="currentLRUCapacity" value="10000"/>
>>>>>> I guess this one could be larger.
>>>>>>
>>>>>>
>>>>>> <property name="revisedLRUCapacity" value="100"/>
>>>>>>
>>>>>> <store type="db">
>>>>>> <!-- type: horizontal | vertical | <any
>>>>>> user-contributed type> --> <mappingStrategy
>>>>>> type="horizontal">
>>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>>> ONE_TABLE_PER_CLASS |
>>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>>>>>> <property name="toManyReferences"
>>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>>
>>>>>> <!-- LIKE_ATTRIBUTES |
>>>>>> LIKE_TO_MANY_REFERENCES --> <property
>>>>>> name="toOneReferences"
>>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>>
>>>>>> <!-- MODEL | STRATEGY -->
>>>>>> <property name="mappingPrecedence"
>>>>>> value="MODEL"/>
>>>>>> This one is obsolete.
>>>>>>
>>>>>>
>>>>>> </mappingStrategy>
>>>>>>
>>>>>> <dbAdapter name="derby-embedded"/>
>>>>>> <dataSource
>>>>>>
> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>>>> databaseName="/tmp/cdodb1"
>>>>>> createDatabase="create"/>
>>>>>>
>>>>>> </store>
>>>>>> </repository>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> What do you think about that ? What throughput do you think I should
>>>>>> expect ? As I said that's very difficult to judge.
>>>>>> I believe it's more appropriate to compare the actual values with
>>>>>> values from other measures (other setups, properties, etc. or even
>>>>>> other products) instead of random expectations. Apologies that I
>>>>>> didn't have the time so far to provide some reference measures
>>>>>> myself. Maybe this gets better now that the team starts to grow
>>>>>> slowly ;-)
>>>>>>
>>>>>>
>>>>>> What are the main reasons fur such figures ?
>>>>>> Hmm, I *suspect* you don't like the figures? I can find no evidence
>>>>>> your assessment ;-)
>>>>>>
>>>>>> Ahh,one more note:
>>>>>> Tracing can also have a significant impact on performance and
>>>>>> end-to-end throughput. Turn it off by all means. And please make
>>>>>> sure that it's really turned off, not only invisible (i.e. console
>>>>>> is not a target).
>>>>>>
>>>>>> Cheers
>>>>>> /Eike
>>
>


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Cell: +31 (0)6 288 48 943
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: [CDO] reading/writting throughput performances [message #619111 is a reply to message #124246] Fri, 30 May 2008 14:52 Go to previous message
Cedric Brun is currently offline Cedric BrunFriend
Messages: 431
Registered: July 2009
Senior Member
Thanks a lot ! I'm looking forward to your advices !

Cédric

Martin Taal wrote:

> I am downloading your zip-file. I don't have time to look at it right away
> but I will do my best later this evening.
>
> gr. Martin
>
> Cédric Brun wrote:
>> I tried to do the same things as in the CDO/Hibernate unit tests.
>>
>> Here's a zipped project trying to launch an HibernateServer with a unit
>> test adding elements.
>> ( http://cedric.brun.free.fr/Tortoose/org.eclipse.cdo-teneo-te st.zip) The
>> libs I found are included in the same project, but I still have no luck
>> and failling at :
>>
>> at
>>
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>> ... 62 more
>> Caused by: java.lang.NoClassDefFoundError:
>> org/hibernate/proxy/HibernateProxy
>> at java.lang.ClassLoader.defineClass1(Native Method)
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>> ... 68 more
>>
>> please note that the test project I provide really is a "quick and ugly"
>> one ^^
>>
>>
>> Cédric
>>
>>
>>
>> Martin Taal wrote:
>>
>>> Hmm, I can understand your headache. Classpath issues are the most
>>> annoying errors...
>>>
>>> In your junit test source file, what happens when you enter the
>>> org.objectweb.asm.Type, can you source file see this class?
>>> Do you have cglib in some other plugin which does not have asm?
>>>
>>> What I also find slightly strange is that cglib is used in this case
>>> because normally cdo objects are not proxied using cglib. Can you post
>>> the generated mapping? How does your cdo initialization code look like?
>>>
>>> gr. Martin
>>>
>>> Cédric Brun wrote:
>>>> Hi Martin,
>>>>
>>>> and thanks for your quick answer ! what I did is the following :
>>>>
>>>> I used the org.eclipse.emf.cdo.server.hibernate.libraries bundle and
>>>> added that :
>>>>
>>>> Bundle-ClassPath: lib/hibernate3.jar,
>>>> lib/cglib-2.1.3.jar,
>>>> lib/commons-collections-2.1.1.jar,
>>>> lib/dom4j.jar,
>>>> lib/jta.jar
>>>> lib/asm-3.0.jar
>>>>
>>>> and this directive :
>>>>
>>>> Eclipse-BuddyPolicy: dependent
>>>>
>>>> as stated in
>>>> http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>>>>
>>>> Then I don't get the same error, now when I start my own plugin (JUnit
>>>> plugin test) depending on
>>>> org.eclipse.emf.cdo.server.hibernate.libraries I get :
>>>>
>>>> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
>>>> at net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
>>>> at net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
>>>> at net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
>>>> at
>>>>
>>
org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>>>> These jars are such a headache ! Do you know - by chance - what's
>>>> going on ?
>>>>
>>>>
>>>> Cédric
>>>>
>>>>
>>>> Martin Taal wrote:
>>>>
>>>>> Hi Cédric,
>>>>> Most of the time this exception is thrown because hibernate can't find
>>>>> the class. A common cause is that the hibernate jar files are located
>>>>> in a separate plugin which can not reach the classes in other plugins.
>>>>> If you set the eclipse buddy policy of the hibernate plugin to
>>>>> dependent then this can help.
>>>>>
>>>>> gr. Martin
>>>>>
>>>>> Cédric Brun wrote:
>>>>>> Hi Simon and Eike,
>>>>>>
>>>>>> thanks for your feedback, now I know what I could expect in using
>>>>>> optimized implementation, I'll have a try on the hibernate store and
>>>>>> see if it is scaling in a better way.
>>>>>>
>>>>>> You gave me many pointers to explain the performances issues but in
>>>>>> fact it seems that the SQL execution is taking so much time (not
>>>>>> suprisingly in fact ;) . My first profile was wrong.
>>>>>>
>>>>>> So I'm installing the hibernate store retrieving all the source code
>>>>>> from CVS, and I'm trying to launch the unit tests. I added many jars
>>>>>> and everything gets compiled but most of the tests are failling with
>>>>>> "
>>>>>>
>>>>>> "
>>>>>> org.hibernate.MappingException: Could not determine type for:
>>>>>>
org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>>>>> at table: cdo_features, for columns:
>>>>>> [org.hibernate.mapping.Column(type)] "
>>>>>>
>>>>>> any hint ?
>>>>>>
>>>>>> Cédric
>>>>>>
>>>>>>
>>>>>> Simon McDuff wrote:
>>>>>>
>>>>>>> We noticed a performance degradation in the following case:
>>>>>>>
>>>>>>> When we fetch a collection of objects that contains collections that
>>>>>>> we don't want to use. Ex:
>>>>>>>
>>>>>>> Model:
>>>>>>>
>>>>>>> ClassB
>>>>>>> {
>>>>>>> int foo1;
>>>>>>> List childrens1;
>>>>>>> List childrens2;
>>>>>>> List childrens3;
>>>>>>> };
>>>>>>>
>>>>>>> Code:
>>>>>>>
>>>>>>> Resource.contents contains objects of type ClassB.
>>>>>>>
>>>>>>> By fetching instances of classB, we will also fetched some items in
>>>>>>> the list(childrens1, childrens2, childrens3) The problem we had.. is
>>>>>>> we do not know if we want to use it. SO better not to download them.
>>>>>>>
>>>>>>> So In our production(we used our own IStore), we never download any
>>>>>>> objects from collection. I'm working at the moment to put in place
>>>>>>> all feature we used in the current CDO. That one will be part of
>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477 To begin
>>>>>>> with,CDORevisionImpl needs to supports list that are 0 even if
>>>>>>> referenceChunkSize == 0.
>>>>>>>
>>>>>>> It is true that we obtain good performance.
>>>>>>> Worst case :
>>>>>>> 1500 objects /sec (Real fetch from disk. Page that contains the
>>>>>>> objects, wasn't in the cache of the filesystem.)
>>>>>>>
>>>>>>> Best case
>>>>>>> 12 000 objects/Sec (Fetch objects that are already in the case of
>>>>>>> the filesystem)
>>>>>>>
>>>>>>> (These statistics always fetch objects from server to client. The
>>>>>>> cache client isn't used)
>>>>>>>
>>>>>>> We have another optimization one with SmartReadAhead but for now
>>>>>>> only work
>>>>>>> for UI client. (for now) This is something we will integrate as
>>>>>>> well in CDO. But for now we concentrate on the other things.
>>>>>>>
>>>>>>> I used an Object database. Based on my preliminary test without CDO.
>>>>>>> I compare
>>>>>>>
>>>>>>> Hibernate vs Objectivity/EMF :
>>>>>>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>>>>>>
>>>>>>> The worst.. you will get 30 to 240 objects/sec or
>>>>>>> the best... you will get 500 to 4000 objects/sec
>>>>>>>
>>>>>>> Based on your comments... you are getting the worst (with Derby)!!
>>>>>>> :-)
>>>>>>>
>>>>>>>
>>>>>>> Another comments about insertion.
>>>>>>>
>>>>>>> We found another problem when we insert data(DOn't know if it is
>>>>>>> still true). Resource.Contents is a unique list. Basically each time
>>>>>>> we insert one element, it needs to fetch all the collection to see
>>>>>>> if we already have this element. Basically.. it is time consuming.
>>>>>>> In our case we are not going through this stage to persist our
>>>>>>> objects... but the objects are not accessible through that list.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> If speed is an issue for you!! We (You included :-)) can optimize
>>>>>>> it. What I like is Eike did a really good job doing that framwork
>>>>>>> and optimizations are easy to add. Also you can change your back-end
>>>>>>> without affecting you client side. (Maybe going to an Object
>>>>>>> database!!) THis was one of the purpose of CDO!! :-)
>>>>>>>
>>>>>>>
>>>>>>> Simon
>>>>>>>
>>>>>>>
>>>>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le message de
>>>>>>> news: 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>>>>
>>>>>>> That sounds interesting!
>>>>>>> Comments inline...
>>>>>>>
>>>>>>>
>>>>>>> Cédric Brun schrieb:
>>>>>>> Hi Eike,
>>>>>>>
>>>>>>> I've been messing around CDO lately and I hacked a few ugly
>>>>>>> unit-tests measuring read/write throughput when I use CDO (RC2).
>>>>>>>
>>>>>>> I'm using an embeded-derby database, two different metamodels
>>>>>>> (complex one - 250 EClassifiers/300 features and a simple one - 1
>>>>>>> Classifier/3 Features)
>>>>>>>
>>>>>>> The test is a "quick and dirty" one so please do not take these
>>>>>>> figures too seriously, I would just like to have your input about
>>>>>>> the "order of magnitude" I should expect or settings affecting
>>>>>>> seriously these performances. I tried both JVM and TCP acceptor but
>>>>>>> the performances are quite similar though JVM is a bit faster.
>>>>>>> Looks like a sign that the NIO socket channels are efficient ,-)
>>>>>>> You used localhost for TCP transport I guess?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> * First case : building a small model (10000 elements) with the
>>>>>>> simple MM, only adding instances in an empty database :
>>>>>>> commiting change on every add I get a 2 objects per second rate.
>>>>>>> commiting changes every 10 adds - I get a 30 objects per second
>>>>>>> rate.
>>>>>>> Seems to be consistent with the fact that only for commits there
>>>>>>> is remote traffic. And the backend transaction management is
>>>>>>> another overhead. So the impact of the number objects/commit
>>>>>>> starts with bigger transactions.
>>>>>>>
>>>>>>>
>>>>>>> browsing the just commited elements opening a new transaction and
>>>>>>> getting a new Resource instance, I get around 30 000 objets/sec rate
>>>>>>> just iterating thanks to Resource.getAllContents().
>>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>>> All views and transactions of a single session share (and cache)
>>>>>>> their object state. Otherwise I'd expect much smaller values.
>>>>>>> With his ObjectivityStore implementation Simon (cc'ed) reached
>>>>>>> rates of several thousands objects/sec. But the DBStore (which you
>>>>>>> used) is not that optimized.
>>>>>>>
>>>>>>>
>>>>>>> * Second case : I built a big model in the database using the
>>>>>>> complex metamodel. It contains 700 000 elements. The creation went
>>>>>>> pretty nicely (I did not measured it exactly as it was launched
>>>>>>> during a night, but it seems I got around 100 objects/sec rate in
>>>>>>> writting as I was writting every 700 objects.
>>>>>>> Sounds not too bad.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Once the model is built the derby database is 222 MB (which - I
>>>>>>> heard - is not so big for derby).
>>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>>> To be honest, I'm really no database expert (although I can write
>>>>>>> simple SQL statements).
>>>>>>>
>>>>>>>
>>>>>>> Retrieving the distant resource and calling eAllContents() on it, I
>>>>>>> get a 50 Objects/sec rate at best, which mean my eAllContents() call
>>>>>>> would take 4 hours !
>>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>>> (iteration order and so). There are several CDO features that can
>>>>>>> have a major impact on load rates!
>>>>>>>
>>>>>>> - Collections (xrefs and containment) can be loaded lazily. I
>>>>>>> don't mean the target objects, which are lazy by default anyway,
>>>>>>> but even the target IDs that are in the collections. We call that
>>>>>>> "reference chunking" (but search for a better name is ongoing).
>>>>>>>
>>>>>>> public interface CDOSession extends CDOProtocolSession,
>>>>>>> IContainer<CDOView>
>>>>>>> {
>>>>>>> public int getReferenceChunkSize();
>>>>>>>
>>>>>>> public void setReferenceChunkSize(int referenceChunkSize);
>>>>>>>
>>>>>>>
>>>>>>> - Loading collection elements (target objects) can be batched (the
>>>>>>> current name is even worse)
>>>>>>>
>>>>>>> public interface CDOView extends CDOProtocolView, INotifier
>>>>>>> {
>>>>>>> public int getLoadRevisionCollectionChunkSize();
>>>>>>>
>>>>>>> public void setLoadRevisionCollectionChunkSize(int
>>>>>>> loadRevisionCollectionChunkSize);
>>>>>>>
>>>>>>>
>>>>>>> - One of the most sophisticated feature is automatic model usage
>>>>>>> analyzation. The result are fetch rules that are sent to the
>>>>>>> server. When later objects are loaded from the server, it can
>>>>>>> answer with additional objects that are likely to be needed by the
>>>>>>> client in the near future.
>>>>>>>
>>>>>>> public class CDOViewImpl extends
>>>>>>> org.eclipse.net4j.internal.util.event.Notifier implements
>>>>>>> CDOView, CDOIDProvider, Adapter.Internal
>>>>>>> {
>>>>>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>>>> {
>>>>>>> return featureAnalyzer;
>>>>>>> }
>>>>>>>
>>>>>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>>>> featureAnalyzer)
>>>>>>> {
>>>>>>> this.featureAnalyzer = featureAnalyzer == null ?
>>>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> - There's a SmartReadAheadThread (if you're curious, please ask
>>>>>>> Simon). - Caches can be configured
>>>>>>> - ...
>>>>>>>
>>>>>>>
>>>>>>> Known issues of the DBStore:
>>>>>>> - When in auditing mode updates are always written as complete
>>>>>>> inserts (by design) - When configuring toManyReferences to a value
>>>>>>> different from LIKE_ATTRIBUTES (which is not implemented yet) it
>>>>>>> is very slow. - No (few) prepared statements are used (see
>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe
>>>>>>> others...
>>>>>>>
>>>>>>> Please note that the DBStore was not written by me to compete with
>>>>>>> Hibernate in terms of performance or mapping capabilities. Now
>>>>>>> that Martin and I, well in the end mostly Martin, provide a
>>>>>>> HibernateStore and this HibernateStore gets more and more features
>>>>>>> I expect that there will be a trend away from the DBStore.
>>>>>>>
>>>>>>>
>>>>>>> This is probably linked to the way the mapping to db is done, here's
>>>>>>> the config I used (came from CDO-server example if I recall
>>>>>>> correctly).
>>>>>>>
>>>>>>>
>>>>>>> <repository name="repo1">
>>>>>>> <property name="overrideUUID"
>>>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>>>> <property name="supportingAudits" value="true"/>
>>>>>>> <property name="verifyingRevisions" value="false"/>
>>>>>>> <property name="currentLRUCapacity" value="10000"/>
>>>>>>> I guess this one could be larger.
>>>>>>>
>>>>>>>
>>>>>>> <property name="revisedLRUCapacity" value="100"/>
>>>>>>>
>>>>>>> <store type="db">
>>>>>>> <!-- type: horizontal | vertical | <any
>>>>>>> user-contributed type> --> <mappingStrategy
>>>>>>> type="horizontal">
>>>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>>>> ONE_TABLE_PER_CLASS |
>>>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES
>>>>>>> -->
>>>>>>> <property name="toManyReferences"
>>>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>>>
>>>>>>> <!-- LIKE_ATTRIBUTES |
>>>>>>> LIKE_TO_MANY_REFERENCES -->
>>>>>>> <property name="toOneReferences"
>>>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>>>
>>>>>>> <!-- MODEL | STRATEGY -->
>>>>>>> <property name="mappingPrecedence"
>>>>>>> value="MODEL"/>
>>>>>>> This one is obsolete.
>>>>>>>
>>>>>>>
>>>>>>> </mappingStrategy>
>>>>>>>
>>>>>>> <dbAdapter name="derby-embedded"/>
>>>>>>> <dataSource
>>>>>>>
>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>>>>> databaseName="/tmp/cdodb1"
>>>>>>> createDatabase="create"/>
>>>>>>>
>>>>>>> </store>
>>>>>>> </repository>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> What do you think about that ? What throughput do you think I should
>>>>>>> expect ? As I said that's very difficult to judge.
>>>>>>> I believe it's more appropriate to compare the actual values with
>>>>>>> values from other measures (other setups, properties, etc. or even
>>>>>>> other products) instead of random expectations. Apologies that I
>>>>>>> didn't have the time so far to provide some reference measures
>>>>>>> myself. Maybe this gets better now that the team starts to grow
>>>>>>> slowly ;-)
>>>>>>>
>>>>>>>
>>>>>>> What are the main reasons fur such figures ?
>>>>>>> Hmm, I *suspect* you don't like the figures? I can find no
>>>>>>> evidence your assessment ;-)
>>>>>>>
>>>>>>> Ahh,one more note:
>>>>>>> Tracing can also have a significant impact on performance and
>>>>>>> end-to-end throughput. Turn it off by all means. And please make
>>>>>>> sure that it's really turned off, not only invisible (i.e. console
>>>>>>> is not a target).
>>>>>>>
>>>>>>> Cheers
>>>>>>> /Eike
>>>
>>
>
>


http://cedric.brun.io news and articles on eclipse and eclipse modeling.
Re: [CDO] reading/writting throughput performances [message #619126 is a reply to message #124260] Sat, 31 May 2008 23:02 Go to previous message
Martin Taal is currently offline Martin TaalFriend
Messages: 5468
Registered: July 2009
Senior Member
Hi Cédric,
I looked at it for many hours trying different things (using your testcase) but I am kind of stuck
with this (getting the same error as you). The difference I can see with the way I run the
cdo.hibernate testcases (as a junit test) and how your testcase is run (junit plugin test). This for
sure means difference in class loader behavior.

Eike, do you have an idea? I am not sure how classloading works when you start a cdo server in a
plugin. Which plugins are then available in the cdo server? The testcases can be downloaded from the
link below.

gr. Martin

Cédric Brun wrote:
> Thanks a lot ! I'm looking forward to your advices !
>
> Cédric
>
> Martin Taal wrote:
>
>> I am downloading your zip-file. I don't have time to look at it right away
>> but I will do my best later this evening.
>>
>> gr. Martin
>>
>> Cédric Brun wrote:
>>> I tried to do the same things as in the CDO/Hibernate unit tests.
>>>
>>> Here's a zipped project trying to launch an HibernateServer with a unit
>>> test adding elements.
>>> ( http://cedric.brun.free.fr/Tortoose/org.eclipse.cdo-teneo-te st.zip) The
>>> libs I found are included in the same project, but I still have no luck
>>> and failling at :
>>>
>>> at
>>>
> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>> ... 62 more
>>> Caused by: java.lang.NoClassDefFoundError:
>>> org/hibernate/proxy/HibernateProxy
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>>> ... 68 more
>>>
>>> please note that the test project I provide really is a "quick and ugly"
>>> one ^^
>>>
>>>
>>> Cédric
>>>
>>>
>>>
>>> Martin Taal wrote:
>>>
>>>> Hmm, I can understand your headache. Classpath issues are the most
>>>> annoying errors...
>>>>
>>>> In your junit test source file, what happens when you enter the
>>>> org.objectweb.asm.Type, can you source file see this class?
>>>> Do you have cglib in some other plugin which does not have asm?
>>>>
>>>> What I also find slightly strange is that cglib is used in this case
>>>> because normally cdo objects are not proxied using cglib. Can you post
>>>> the generated mapping? How does your cdo initialization code look like?
>>>>
>>>> gr. Martin
>>>>
>>>> Cédric Brun wrote:
>>>>> Hi Martin,
>>>>>
>>>>> and thanks for your quick answer ! what I did is the following :
>>>>>
>>>>> I used the org.eclipse.emf.cdo.server.hibernate.libraries bundle and
>>>>> added that :
>>>>>
>>>>> Bundle-ClassPath: lib/hibernate3.jar,
>>>>> lib/cglib-2.1.3.jar,
>>>>> lib/commons-collections-2.1.1.jar,
>>>>> lib/dom4j.jar,
>>>>> lib/jta.jar
>>>>> lib/asm-3.0.jar
>>>>>
>>>>> and this directive :
>>>>>
>>>>> Eclipse-BuddyPolicy: dependent
>>>>>
>>>>> as stated in
>>>>> http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>>>>>
>>>>> Then I don't get the same error, now when I start my own plugin (JUnit
>>>>> plugin test) depending on
>>>>> org.eclipse.emf.cdo.server.hibernate.libraries I get :
>>>>>
>>>>> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
>>>>> at net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
>>>>> at net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
>>>>> at net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
>>>>> at
>>>>>
> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>>>>> These jars are such a headache ! Do you know - by chance - what's
>>>>> going on ?
>>>>>
>>>>>
>>>>> Cédric
>>>>>
>>>>>
>>>>> Martin Taal wrote:
>>>>>
>>>>>> Hi Cédric,
>>>>>> Most of the time this exception is thrown because hibernate can't find
>>>>>> the class. A common cause is that the hibernate jar files are located
>>>>>> in a separate plugin which can not reach the classes in other plugins.
>>>>>> If you set the eclipse buddy policy of the hibernate plugin to
>>>>>> dependent then this can help.
>>>>>>
>>>>>> gr. Martin
>>>>>>
>>>>>> Cédric Brun wrote:
>>>>>>> Hi Simon and Eike,
>>>>>>>
>>>>>>> thanks for your feedback, now I know what I could expect in using
>>>>>>> optimized implementation, I'll have a try on the hibernate store and
>>>>>>> see if it is scaling in a better way.
>>>>>>>
>>>>>>> You gave me many pointers to explain the performances issues but in
>>>>>>> fact it seems that the SQL execution is taking so much time (not
>>>>>>> suprisingly in fact ;) . My first profile was wrong.
>>>>>>>
>>>>>>> So I'm installing the hibernate store retrieving all the source code
>>>>>>> from CVS, and I'm trying to launch the unit tests. I added many jars
>>>>>>> and everything gets compiled but most of the tests are failling with
>>>>>>> "
>>>>>>>
>>>>>>> "
>>>>>>> org.hibernate.MappingException: Could not determine type for:
>>>>>>>
> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>>>>>> at table: cdo_features, for columns:
>>>>>>> [org.hibernate.mapping.Column(type)] "
>>>>>>>
>>>>>>> any hint ?
>>>>>>>
>>>>>>> Cédric
>>>>>>>
>>>>>>>
>>>>>>> Simon McDuff wrote:
>>>>>>>
>>>>>>>> We noticed a performance degradation in the following case:
>>>>>>>>
>>>>>>>> When we fetch a collection of objects that contains collections that
>>>>>>>> we don't want to use. Ex:
>>>>>>>>
>>>>>>>> Model:
>>>>>>>>
>>>>>>>> ClassB
>>>>>>>> {
>>>>>>>> int foo1;
>>>>>>>> List childrens1;
>>>>>>>> List childrens2;
>>>>>>>> List childrens3;
>>>>>>>> };
>>>>>>>>
>>>>>>>> Code:
>>>>>>>>
>>>>>>>> Resource.contents contains objects of type ClassB.
>>>>>>>>
>>>>>>>> By fetching instances of classB, we will also fetched some items in
>>>>>>>> the list(childrens1, childrens2, childrens3) The problem we had.. is
>>>>>>>> we do not know if we want to use it. SO better not to download them.
>>>>>>>>
>>>>>>>> So In our production(we used our own IStore), we never download any
>>>>>>>> objects from collection. I'm working at the moment to put in place
>>>>>>>> all feature we used in the current CDO. That one will be part of
>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477 To begin
>>>>>>>> with,CDORevisionImpl needs to supports list that are 0 even if
>>>>>>>> referenceChunkSize == 0.
>>>>>>>>
>>>>>>>> It is true that we obtain good performance.
>>>>>>>> Worst case :
>>>>>>>> 1500 objects /sec (Real fetch from disk. Page that contains the
>>>>>>>> objects, wasn't in the cache of the filesystem.)
>>>>>>>>
>>>>>>>> Best case
>>>>>>>> 12 000 objects/Sec (Fetch objects that are already in the case of
>>>>>>>> the filesystem)
>>>>>>>>
>>>>>>>> (These statistics always fetch objects from server to client. The
>>>>>>>> cache client isn't used)
>>>>>>>>
>>>>>>>> We have another optimization one with SmartReadAhead but for now
>>>>>>>> only work
>>>>>>>> for UI client. (for now) This is something we will integrate as
>>>>>>>> well in CDO. But for now we concentrate on the other things.
>>>>>>>>
>>>>>>>> I used an Object database. Based on my preliminary test without CDO.
>>>>>>>> I compare
>>>>>>>>
>>>>>>>> Hibernate vs Objectivity/EMF :
>>>>>>>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>>>>>>>
>>>>>>>> The worst.. you will get 30 to 240 objects/sec or
>>>>>>>> the best... you will get 500 to 4000 objects/sec
>>>>>>>>
>>>>>>>> Based on your comments... you are getting the worst (with Derby)!!
>>>>>>>> :-)
>>>>>>>>
>>>>>>>>
>>>>>>>> Another comments about insertion.
>>>>>>>>
>>>>>>>> We found another problem when we insert data(DOn't know if it is
>>>>>>>> still true). Resource.Contents is a unique list. Basically each time
>>>>>>>> we insert one element, it needs to fetch all the collection to see
>>>>>>>> if we already have this element. Basically.. it is time consuming.
>>>>>>>> In our case we are not going through this stage to persist our
>>>>>>>> objects... but the objects are not accessible through that list.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> If speed is an issue for you!! We (You included :-)) can optimize
>>>>>>>> it. What I like is Eike did a really good job doing that framwork
>>>>>>>> and optimizations are easy to add. Also you can change your back-end
>>>>>>>> without affecting you client side. (Maybe going to an Object
>>>>>>>> database!!) THis was one of the purpose of CDO!! :-)
>>>>>>>>
>>>>>>>>
>>>>>>>> Simon
>>>>>>>>
>>>>>>>>
>>>>>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le message de
>>>>>>>> news: 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>>>>>
>>>>>>>> That sounds interesting!
>>>>>>>> Comments inline...
>>>>>>>>
>>>>>>>>
>>>>>>>> Cédric Brun schrieb:
>>>>>>>> Hi Eike,
>>>>>>>>
>>>>>>>> I've been messing around CDO lately and I hacked a few ugly
>>>>>>>> unit-tests measuring read/write throughput when I use CDO (RC2).
>>>>>>>>
>>>>>>>> I'm using an embeded-derby database, two different metamodels
>>>>>>>> (complex one - 250 EClassifiers/300 features and a simple one - 1
>>>>>>>> Classifier/3 Features)
>>>>>>>>
>>>>>>>> The test is a "quick and dirty" one so please do not take these
>>>>>>>> figures too seriously, I would just like to have your input about
>>>>>>>> the "order of magnitude" I should expect or settings affecting
>>>>>>>> seriously these performances. I tried both JVM and TCP acceptor but
>>>>>>>> the performances are quite similar though JVM is a bit faster.
>>>>>>>> Looks like a sign that the NIO socket channels are efficient ,-)
>>>>>>>> You used localhost for TCP transport I guess?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> * First case : building a small model (10000 elements) with the
>>>>>>>> simple MM, only adding instances in an empty database :
>>>>>>>> commiting change on every add I get a 2 objects per second rate.
>>>>>>>> commiting changes every 10 adds - I get a 30 objects per second
>>>>>>>> rate.
>>>>>>>> Seems to be consistent with the fact that only for commits there
>>>>>>>> is remote traffic. And the backend transaction management is
>>>>>>>> another overhead. So the impact of the number objects/commit
>>>>>>>> starts with bigger transactions.
>>>>>>>>
>>>>>>>>
>>>>>>>> browsing the just commited elements opening a new transaction and
>>>>>>>> getting a new Resource instance, I get around 30 000 objets/sec rate
>>>>>>>> just iterating thanks to Resource.getAllContents().
>>>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>>>> All views and transactions of a single session share (and cache)
>>>>>>>> their object state. Otherwise I'd expect much smaller values.
>>>>>>>> With his ObjectivityStore implementation Simon (cc'ed) reached
>>>>>>>> rates of several thousands objects/sec. But the DBStore (which you
>>>>>>>> used) is not that optimized.
>>>>>>>>
>>>>>>>>
>>>>>>>> * Second case : I built a big model in the database using the
>>>>>>>> complex metamodel. It contains 700 000 elements. The creation went
>>>>>>>> pretty nicely (I did not measured it exactly as it was launched
>>>>>>>> during a night, but it seems I got around 100 objects/sec rate in
>>>>>>>> writting as I was writting every 700 objects.
>>>>>>>> Sounds not too bad.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Once the model is built the derby database is 222 MB (which - I
>>>>>>>> heard - is not so big for derby).
>>>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>>>> To be honest, I'm really no database expert (although I can write
>>>>>>>> simple SQL statements).
>>>>>>>>
>>>>>>>>
>>>>>>>> Retrieving the distant resource and calling eAllContents() on it, I
>>>>>>>> get a 50 Objects/sec rate at best, which mean my eAllContents() call
>>>>>>>> would take 4 hours !
>>>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>>>> (iteration order and so). There are several CDO features that can
>>>>>>>> have a major impact on load rates!
>>>>>>>>
>>>>>>>> - Collections (xrefs and containment) can be loaded lazily. I
>>>>>>>> don't mean the target objects, which are lazy by default anyway,
>>>>>>>> but even the target IDs that are in the collections. We call that
>>>>>>>> "reference chunking" (but search for a better name is ongoing).
>>>>>>>>
>>>>>>>> public interface CDOSession extends CDOProtocolSession,
>>>>>>>> IContainer<CDOView>
>>>>>>>> {
>>>>>>>> public int getReferenceChunkSize();
>>>>>>>>
>>>>>>>> public void setReferenceChunkSize(int referenceChunkSize);
>>>>>>>>
>>>>>>>>
>>>>>>>> - Loading collection elements (target objects) can be batched (the
>>>>>>>> current name is even worse)
>>>>>>>>
>>>>>>>> public interface CDOView extends CDOProtocolView, INotifier
>>>>>>>> {
>>>>>>>> public int getLoadRevisionCollectionChunkSize();
>>>>>>>>
>>>>>>>> public void setLoadRevisionCollectionChunkSize(int
>>>>>>>> loadRevisionCollectionChunkSize);
>>>>>>>>
>>>>>>>>
>>>>>>>> - One of the most sophisticated feature is automatic model usage
>>>>>>>> analyzation. The result are fetch rules that are sent to the
>>>>>>>> server. When later objects are loaded from the server, it can
>>>>>>>> answer with additional objects that are likely to be needed by the
>>>>>>>> client in the near future.
>>>>>>>>
>>>>>>>> public class CDOViewImpl extends
>>>>>>>> org.eclipse.net4j.internal.util.event.Notifier implements
>>>>>>>> CDOView, CDOIDProvider, Adapter.Internal
>>>>>>>> {
>>>>>>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>>>>> {
>>>>>>>> return featureAnalyzer;
>>>>>>>> }
>>>>>>>>
>>>>>>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>>>>> featureAnalyzer)
>>>>>>>> {
>>>>>>>> this.featureAnalyzer = featureAnalyzer == null ?
>>>>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>>>>> }
>>>>>>>>
>>>>>>>>
>>>>>>>> - There's a SmartReadAheadThread (if you're curious, please ask
>>>>>>>> Simon). - Caches can be configured
>>>>>>>> - ...
>>>>>>>>
>>>>>>>>
>>>>>>>> Known issues of the DBStore:
>>>>>>>> - When in auditing mode updates are always written as complete
>>>>>>>> inserts (by design) - When configuring toManyReferences to a value
>>>>>>>> different from LIKE_ATTRIBUTES (which is not implemented yet) it
>>>>>>>> is very slow. - No (few) prepared statements are used (see
>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe
>>>>>>>> others...
>>>>>>>>
>>>>>>>> Please note that the DBStore was not written by me to compete with
>>>>>>>> Hibernate in terms of performance or mapping capabilities. Now
>>>>>>>> that Martin and I, well in the end mostly Martin, provide a
>>>>>>>> HibernateStore and this HibernateStore gets more and more features
>>>>>>>> I expect that there will be a trend away from the DBStore.
>>>>>>>>
>>>>>>>>
>>>>>>>> This is probably linked to the way the mapping to db is done, here's
>>>>>>>> the config I used (came from CDO-server example if I recall
>>>>>>>> correctly).
>>>>>>>>
>>>>>>>>
>>>>>>>> <repository name="repo1">
>>>>>>>> <property name="overrideUUID"
>>>>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>>>>> <property name="supportingAudits" value="true"/>
>>>>>>>> <property name="verifyingRevisions" value="false"/>
>>>>>>>> <property name="currentLRUCapacity" value="10000"/>
>>>>>>>> I guess this one could be larger.
>>>>>>>>
>>>>>>>>
>>>>>>>> <property name="revisedLRUCapacity" value="100"/>
>>>>>>>>
>>>>>>>> <store type="db">
>>>>>>>> <!-- type: horizontal | vertical | <any
>>>>>>>> user-contributed type> --> <mappingStrategy
>>>>>>>> type="horizontal">
>>>>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>>>>> ONE_TABLE_PER_CLASS |
>>>>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES
>>>>>>>> -->
>>>>>>>> <property name="toManyReferences"
>>>>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>>>>
>>>>>>>> <!-- LIKE_ATTRIBUTES |
>>>>>>>> LIKE_TO_MANY_REFERENCES -->
>>>>>>>> <property name="toOneReferences"
>>>>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>>>>
>>>>>>>> <!-- MODEL | STRATEGY -->
>>>>>>>> <property name="mappingPrecedence"
>>>>>>>> value="MODEL"/>
>>>>>>>> This one is obsolete.
>>>>>>>>
>>>>>>>>
>>>>>>>> </mappingStrategy>
>>>>>>>>
>>>>>>>> <dbAdapter name="derby-embedded"/>
>>>>>>>> <dataSource
>>>>>>>>
>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>>>>>> databaseName="/tmp/cdodb1"
>>>>>>>> createDatabase="create"/>
>>>>>>>>
>>>>>>>> </store>
>>>>>>>> </repository>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> What do you think about that ? What throughput do you think I should
>>>>>>>> expect ? As I said that's very difficult to judge.
>>>>>>>> I believe it's more appropriate to compare the actual values with
>>>>>>>> values from other measures (other setups, properties, etc. or even
>>>>>>>> other products) instead of random expectations. Apologies that I
>>>>>>>> didn't have the time so far to provide some reference measures
>>>>>>>> myself. Maybe this gets better now that the team starts to grow
>>>>>>>> slowly ;-)
>>>>>>>>
>>>>>>>>
>>>>>>>> What are the main reasons fur such figures ?
>>>>>>>> Hmm, I *suspect* you don't like the figures? I can find no
>>>>>>>> evidence your assessment ;-)
>>>>>>>>
>>>>>>>> Ahh,one more note:
>>>>>>>> Tracing can also have a significant impact on performance and
>>>>>>>> end-to-end throughput. Turn it off by all means. And please make
>>>>>>>> sure that it's really turned off, not only invisible (i.e. console
>>>>>>>> is not a target).
>>>>>>>>
>>>>>>>> Cheers
>>>>>>>> /Eike
>>
>


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Cell: +31 (0)6 288 48 943
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: [CDO] reading/writting throughput performances [message #619127 is a reply to message #124481] Sun, 01 June 2008 04:41 Go to previous message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
This is a multi-part message in MIME format.
--------------050306020307050804040207
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi guys,

I can confirm, making these 3rd party libraries with their own
classloading run in OSGi is a nightmare! I took me hours teach where the
hsqldb driver is. On the way I came over different kinds of classloading
related errors.

First I removed all the jars from the test plugin (the one Cedric
provided). They're already in the cdo.server.hibernate.libraries plugin
which, in fact, needs the buddy policy set to dependent. I think this
last change needs to be committed to CVS! Btw. I'd prefer to use
*registered* buddies instead of making all dependent bundles visible.

The test plugin then needs to provide the database driver. Therefore I
added a dependency on the org.eclipse.net4j.db.hsqldb plugin.

With this setup I can start the server without CNFE, NCDFE, CCI and the
like. At this point I get the following exception, which seems the same
as the one Cedric observed in the end:

net.sf.cglib.core.CodeGenerationException:
java.lang.reflect.InvocationTargetException-->null
at
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:237)
at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
at net.sf.cglib.proxy.Enhancer.createClass(Enhancer.java:317)
at
org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:127)
at
org.hibernate.proxy.pojo.cglib.CGLIBProxyFactory.postInstant iate(CGLIBProxyFactory.java:43)
at
org.hibernate.tuple.entity.PojoEntityTuplizer.buildProxyFact ory(PojoEntityTuplizer.java:162)
at
org.hibernate.tuple.entity.AbstractEntityTuplizer.<init>(AbstractEntityTuplizer.java:135)
at
org.hibernate.tuple.entity.PojoEntityTuplizer.<init>(PojoEntityTuplizer.java:55)
at
org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping . <init>(EntityEntityModeToTuplizerMapping.java:56)
at
org.hibernate.tuple.entity.EntityMetamodel.<init>(EntityMetamodel.java:295)
at
org.hibernate.persister.entity.AbstractEntityPersister.<init >(AbstractEntityPersister.java:434)
at
org.hibernate.persister.entity.JoinedSubclassEntityPersister . <init>(JoinedSubclassEntityPersister.java:91)
at
org.hibernate.persister.PersisterFactory.createClassPersiste r(PersisterFactory.java:58)
at
org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:226)
at
org.hibernate.cfg.Configuration.buildSessionFactory(Configur ation.java:1294)
at
org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getSessionFactory(HibernatePackageHandler.java:335 )
at
org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.readPackageInfos(HibernatePackageHandler.java:293)
at
org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getCDOPackageInfos(HibernatePackageHandler.java:21 5)
at
org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Reader.readPackageInfos(HibernateStoreReader.java:87)
at
org.eclipse.emf.cdo.internal.server.PackageManager.doActivat e(PackageManager.java:72)
at
org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
at
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
at
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
at
org.eclipse.emf.cdo.internal.server.Repository.activateRepos itory(Repository.java:264)
at
org.eclipse.emf.cdo.internal.server.Repository.doActivate(Re pository.java:251)
at
org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
at
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
at
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
at
org.eclipse.emf.cdo.server.CDOServerUtil.addRepository(CDOSe rverUtil.java:67)
at org.eclipse.cdo.bug.LaunchServerTest.setUp(LaunchServerTest. java:54)
at junit.framework.TestCase.runBare(TestCase.java:128)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:120)
at junit.framework.TestSuite.runTest(TestSuite.java:230)
at junit.framework.TestSuite.run(TestSuite.java:225)
at
org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
at
org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
at
org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hodWithException(EclipseAppContainer.java:574)
at
org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
at
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:110)
at
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:79)
at
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:379)
at
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 549)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
at org.eclipse.equinox.launcher.Main.main(Main.java:1212)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at net.sf.cglib.core.ReflectUtils.defineClass(ReflectUtils.java :384)
at
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
... 61 more
Caused by: java.lang.NoClassDefFoundError:
org/hibernate/proxy/HibernateProxy
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
... 67 more

I must admit that I'm still not sure about te fundamental interwork
between Hibernate and the user defined meta model. It seems as if asm
and cglib are somewhere in between but I have no idea why at all. I
thought we have implemented a bunch of static Hibernate interfaces to
make the CDORevision instances known to Hibernate. Please somebody
explain why we need runtime byte code engineering? Is there some
configuration missing to tell Hibernate to use our special tuplizers?

I have changed several things in the code of the test plugin and since
it's quite small now that the jars are removed, I attach my version of
the zip here (including a JUnit Plugin launch config). And don't forget
to update the cdo.server.hibernate.libraries plugin from CVS to get the
new buddy policy!

Cheers
/Eike

Martin Taal schrieb:
> Hi Cédric,
> I looked at it for many hours trying different things (using your
> testcase) but I am kind of stuck with this (getting the same error as
> you). The difference I can see with the way I run the cdo.hibernate
> testcases (as a junit test) and how your testcase is run (junit plugin
> test). This for sure means difference in class loader behavior.
>
> Eike, do you have an idea? I am not sure how classloading works when
> you start a cdo server in a plugin. Which plugins are then available
> in the cdo server? The testcases can be downloaded from the link below.
>
> gr. Martin
>
> Cédric Brun wrote:
>> Thanks a lot ! I'm looking forward to your advices !
>>
>> Cédric
>>
>> Martin Taal wrote:
>>
>>> I am downloading your zip-file. I don't have time to look at it
>>> right away
>>> but I will do my best later this evening.
>>>
>>> gr. Martin
>>>
>>> Cédric Brun wrote:
>>>> I tried to do the same things as in the CDO/Hibernate unit tests.
>>>>
>>>> Here's a zipped project trying to launch an HibernateServer with a
>>>> unit
>>>> test adding elements.
>>>> ( http://cedric.brun.free.fr/Tortoose/org.eclipse.cdo-teneo-te st.zip)
>>>> The
>>>> libs I found are included in the same project, but I still have no
>>>> luck
>>>> and failling at :
>>>>
>>>> at
>>>>
>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>
>>>> ... 62 more
>>>> Caused by: java.lang.NoClassDefFoundError:
>>>> org/hibernate/proxy/HibernateProxy
>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>>>> ... 68 more
>>>>
>>>> please note that the test project I provide really is a "quick and
>>>> ugly"
>>>> one ^^
>>>>
>>>>
>>>> Cédric
>>>>
>>>>
>>>>
>>>> Martin Taal wrote:
>>>>
>>>>> Hmm, I can understand your headache. Classpath issues are the most
>>>>> annoying errors...
>>>>>
>>>>> In your junit test source file, what happens when you enter the
>>>>> org.objectweb.asm.Type, can you source file see this class?
>>>>> Do you have cglib in some other plugin which does not have asm?
>>>>>
>>>>> What I also find slightly strange is that cglib is used in this case
>>>>> because normally cdo objects are not proxied using cglib. Can you
>>>>> post
>>>>> the generated mapping? How does your cdo initialization code look
>>>>> like?
>>>>>
>>>>> gr. Martin
>>>>>
>>>>> Cédric Brun wrote:
>>>>>> Hi Martin,
>>>>>>
>>>>>> and thanks for your quick answer ! what I did is the following :
>>>>>>
>>>>>> I used the org.eclipse.emf.cdo.server.hibernate.libraries bundle and
>>>>>> added that :
>>>>>>
>>>>>> Bundle-ClassPath: lib/hibernate3.jar,
>>>>>> lib/cglib-2.1.3.jar,
>>>>>> lib/commons-collections-2.1.1.jar,
>>>>>> lib/dom4j.jar,
>>>>>> lib/jta.jar
>>>>>> lib/asm-3.0.jar
>>>>>>
>>>>>> and this directive :
>>>>>>
>>>>>> Eclipse-BuddyPolicy: dependent
>>>>>>
>>>>>> as stated in
>>>>>> http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>>>>>>
>>>>>> Then I don't get the same error, now when I start my own plugin
>>>>>> (JUnit
>>>>>> plugin test) depending on
>>>>>> org.eclipse.emf.cdo.server.hibernate.libraries I get :
>>>>>>
>>>>>> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
>>>>>> at net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
>>>>>> at net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
>>>>>> at net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
>>>>>> at
>>>>>>
>> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>>
>>>>>> These jars are such a headache ! Do you know - by chance - what's
>>>>>> going on ?
>>>>>>
>>>>>>
>>>>>> Cédric
>>>>>>
>>>>>>
>>>>>> Martin Taal wrote:
>>>>>>
>>>>>>> Hi Cédric,
>>>>>>> Most of the time this exception is thrown because hibernate
>>>>>>> can't find
>>>>>>> the class. A common cause is that the hibernate jar files are
>>>>>>> located
>>>>>>> in a separate plugin which can not reach the classes in other
>>>>>>> plugins.
>>>>>>> If you set the eclipse buddy policy of the hibernate plugin to
>>>>>>> dependent then this can help.
>>>>>>>
>>>>>>> gr. Martin
>>>>>>>
>>>>>>> Cédric Brun wrote:
>>>>>>>> Hi Simon and Eike,
>>>>>>>>
>>>>>>>> thanks for your feedback, now I know what I could expect in using
>>>>>>>> optimized implementation, I'll have a try on the hibernate
>>>>>>>> store and
>>>>>>>> see if it is scaling in a better way.
>>>>>>>>
>>>>>>>> You gave me many pointers to explain the performances issues
>>>>>>>> but in
>>>>>>>> fact it seems that the SQL execution is taking so much time (not
>>>>>>>> suprisingly in fact ;) . My first profile was wrong.
>>>>>>>>
>>>>>>>> So I'm installing the hibernate store retrieving all the source
>>>>>>>> code
>>>>>>>> from CVS, and I'm trying to launch the unit tests. I added many
>>>>>>>> jars
>>>>>>>> and everything gets compiled but most of the tests are failling
>>>>>>>> with
>>>>>>>> "
>>>>>>>>
>>>>>>>> "
>>>>>>>> org.hibernate.MappingException: Could not determine type for:
>>>>>>>>
>> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>>>>>>> at table: cdo_features, for columns:
>>>>>>>> [org.hibernate.mapping.Column(type)] "
>>>>>>>>
>>>>>>>> any hint ?
>>>>>>>>
>>>>>>>> Cédric
>>>>>>>>
>>>>>>>>
>>>>>>>> Simon McDuff wrote:
>>>>>>>>
>>>>>>>>> We noticed a performance degradation in the following case:
>>>>>>>>>
>>>>>>>>> When we fetch a collection of objects that contains
>>>>>>>>> collections that
>>>>>>>>> we don't want to use. Ex:
>>>>>>>>>
>>>>>>>>> Model:
>>>>>>>>>
>>>>>>>>> ClassB
>>>>>>>>> {
>>>>>>>>> int foo1;
>>>>>>>>> List childrens1;
>>>>>>>>> List childrens2;
>>>>>>>>> List childrens3;
>>>>>>>>> };
>>>>>>>>>
>>>>>>>>> Code:
>>>>>>>>>
>>>>>>>>> Resource.contents contains objects of type ClassB.
>>>>>>>>>
>>>>>>>>> By fetching instances of classB, we will also fetched some
>>>>>>>>> items in
>>>>>>>>> the list(childrens1, childrens2, childrens3) The problem we
>>>>>>>>> had.. is
>>>>>>>>> we do not know if we want to use it. SO better not to download
>>>>>>>>> them.
>>>>>>>>>
>>>>>>>>> So In our production(we used our own IStore), we never
>>>>>>>>> download any
>>>>>>>>> objects from collection. I'm working at the moment to put in
>>>>>>>>> place
>>>>>>>>> all feature we used in the current CDO. That one will be part of
>>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477 To begin
>>>>>>>>> with,CDORevisionImpl needs to supports list that are 0 even if
>>>>>>>>> referenceChunkSize == 0.
>>>>>>>>>
>>>>>>>>> It is true that we obtain good performance.
>>>>>>>>> Worst case :
>>>>>>>>> 1500 objects /sec (Real fetch from disk. Page that contains the
>>>>>>>>> objects, wasn't in the cache of the filesystem.)
>>>>>>>>>
>>>>>>>>> Best case
>>>>>>>>> 12 000 objects/Sec (Fetch objects that are already in the case of
>>>>>>>>> the filesystem)
>>>>>>>>>
>>>>>>>>> (These statistics always fetch objects from server to client. The
>>>>>>>>> cache client isn't used)
>>>>>>>>>
>>>>>>>>> We have another optimization one with SmartReadAhead but for now
>>>>>>>>> only work
>>>>>>>>> for UI client. (for now) This is something we will integrate as
>>>>>>>>> well in CDO. But for now we concentrate on the other things.
>>>>>>>>>
>>>>>>>>> I used an Object database. Based on my preliminary test
>>>>>>>>> without CDO.
>>>>>>>>> I compare
>>>>>>>>>
>>>>>>>>> Hibernate vs Objectivity/EMF :
>>>>>>>>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>>>>>>>>
>>>>>>>>> The worst.. you will get 30 to 240 objects/sec or
>>>>>>>>> the best... you will get 500 to 4000 objects/sec
>>>>>>>>>
>>>>>>>>> Based on your comments... you are getting the worst (with
>>>>>>>>> Derby)!!
>>>>>>>>> :-)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Another comments about insertion.
>>>>>>>>>
>>>>>>>>> We found another problem when we insert data(DOn't know if it is
>>>>>>>>> still true). Resource.Contents is a unique list. Basically
>>>>>>>>> each time
>>>>>>>>> we insert one element, it needs to fetch all the collection to
>>>>>>>>> see
>>>>>>>>> if we already have this element. Basically.. it is time
>>>>>>>>> consuming.
>>>>>>>>> In our case we are not going through this stage to persist our
>>>>>>>>> objects... but the objects are not accessible through that list.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> If speed is an issue for you!! We (You included :-)) can optimize
>>>>>>>>> it. What I like is Eike did a really good job doing that framwork
>>>>>>>>> and optimizations are easy to add. Also you can change your
>>>>>>>>> back-end
>>>>>>>>> without affecting you client side. (Maybe going to an Object
>>>>>>>>> database!!) THis was one of the purpose of CDO!! :-)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Simon
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le message de
>>>>>>>>> news: 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>>>>>>
>>>>>>>>> That sounds interesting!
>>>>>>>>> Comments inline...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cédric Brun schrieb:
>>>>>>>>> Hi Eike,
>>>>>>>>>
>>>>>>>>> I've been messing around CDO lately and I hacked a few ugly
>>>>>>>>> unit-tests measuring read/write throughput when I use CDO (RC2).
>>>>>>>>>
>>>>>>>>> I'm using an embeded-derby database, two different metamodels
>>>>>>>>> (complex one - 250 EClassifiers/300 features and a simple one - 1
>>>>>>>>> Classifier/3 Features)
>>>>>>>>>
>>>>>>>>> The test is a "quick and dirty" one so please do not take these
>>>>>>>>> figures too seriously, I would just like to have your input about
>>>>>>>>> the "order of magnitude" I should expect or settings affecting
>>>>>>>>> seriously these performances. I tried both JVM and TCP
>>>>>>>>> acceptor but
>>>>>>>>> the performances are quite similar though JVM is a bit faster.
>>>>>>>>> Looks like a sign that the NIO socket channels are efficient
>>>>>>>>> ,-)
>>>>>>>>> You used localhost for TCP transport I guess?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> * First case : building a small model (10000 elements) with the
>>>>>>>>> simple MM, only adding instances in an empty database :
>>>>>>>>> commiting change on every add I get a 2 objects per second rate.
>>>>>>>>> commiting changes every 10 adds - I get a 30 objects per second
>>>>>>>>> rate.
>>>>>>>>> Seems to be consistent with the fact that only for commits
>>>>>>>>> there
>>>>>>>>> is remote traffic. And the backend transaction management is
>>>>>>>>> another overhead. So the impact of the number objects/commit
>>>>>>>>> starts with bigger transactions.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> browsing the just commited elements opening a new transaction and
>>>>>>>>> getting a new Resource instance, I get around 30 000
>>>>>>>>> objets/sec rate
>>>>>>>>> just iterating thanks to Resource.getAllContents().
>>>>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>>>>> All views and transactions of a single session share (and
>>>>>>>>> cache)
>>>>>>>>> their object state. Otherwise I'd expect much smaller values.
>>>>>>>>> With his ObjectivityStore implementation Simon (cc'ed) reached
>>>>>>>>> rates of several thousands objects/sec. But the DBStore
>>>>>>>>> (which you
>>>>>>>>> used) is not that optimized.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> * Second case : I built a big model in the database using the
>>>>>>>>> complex metamodel. It contains 700 000 elements. The creation
>>>>>>>>> went
>>>>>>>>> pretty nicely (I did not measured it exactly as it was launched
>>>>>>>>> during a night, but it seems I got around 100 objects/sec rate in
>>>>>>>>> writting as I was writting every 700 objects.
>>>>>>>>> Sounds not too bad.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Once the model is built the derby database is 222 MB (which - I
>>>>>>>>> heard - is not so big for derby).
>>>>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>>>>> To be honest, I'm really no database expert (although I can
>>>>>>>>> write
>>>>>>>>> simple SQL statements).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Retrieving the distant resource and calling eAllContents() on
>>>>>>>>> it, I
>>>>>>>>> get a 50 Objects/sec rate at best, which mean my
>>>>>>>>> eAllContents() call
>>>>>>>>> would take 4 hours !
>>>>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>>>>> (iteration order and so). There are several CDO features
>>>>>>>>> that can
>>>>>>>>> have a major impact on load rates!
>>>>>>>>>
>>>>>>>>> - Collections (xrefs and containment) can be loaded lazily. I
>>>>>>>>> don't mean the target objects, which are lazy by default
>>>>>>>>> anyway,
>>>>>>>>> but even the target IDs that are in the collections. We call
>>>>>>>>> that
>>>>>>>>> "reference chunking" (but search for a better name is ongoing).
>>>>>>>>>
>>>>>>>>> public interface CDOSession extends CDOProtocolSession,
>>>>>>>>> IContainer<CDOView>
>>>>>>>>> {
>>>>>>>>> public int getReferenceChunkSize();
>>>>>>>>>
>>>>>>>>> public void setReferenceChunkSize(int
>>>>>>>>> referenceChunkSize);
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> - Loading collection elements (target objects) can be
>>>>>>>>> batched (the
>>>>>>>>> current name is even worse)
>>>>>>>>>
>>>>>>>>> public interface CDOView extends CDOProtocolView,
>>>>>>>>> INotifier
>>>>>>>>> {
>>>>>>>>> public int getLoadRevisionCollectionChunkSize();
>>>>>>>>>
>>>>>>>>> public void setLoadRevisionCollectionChunkSize(int
>>>>>>>>> loadRevisionCollectionChunkSize);
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> - One of the most sophisticated feature is automatic model
>>>>>>>>> usage
>>>>>>>>> analyzation. The result are fetch rules that are sent to the
>>>>>>>>> server. When later objects are loaded from the server, it can
>>>>>>>>> answer with additional objects that are likely to be needed
>>>>>>>>> by the
>>>>>>>>> client in the near future.
>>>>>>>>>
>>>>>>>>> public class CDOViewImpl extends
>>>>>>>>> org.eclipse.net4j.internal.util.event.Notifier implements
>>>>>>>>> CDOView, CDOIDProvider, Adapter.Internal
>>>>>>>>> {
>>>>>>>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>>>>>> {
>>>>>>>>> return featureAnalyzer;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>>>>>> featureAnalyzer)
>>>>>>>>> {
>>>>>>>>> this.featureAnalyzer = featureAnalyzer == null ?
>>>>>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> - There's a SmartReadAheadThread (if you're curious, please ask
>>>>>>>>> Simon). - Caches can be configured
>>>>>>>>> - ...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Known issues of the DBStore:
>>>>>>>>> - When in auditing mode updates are always written as complete
>>>>>>>>> inserts (by design) - When configuring toManyReferences to a
>>>>>>>>> value
>>>>>>>>> different from LIKE_ATTRIBUTES (which is not implemented
>>>>>>>>> yet) it
>>>>>>>>> is very slow. - No (few) prepared statements are used (see
>>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe
>>>>>>>>> others...
>>>>>>>>>
>>>>>>>>> Please note that the DBStore was not written by me to
>>>>>>>>> compete with
>>>>>>>>> Hibernate in terms of performance or mapping capabilities. Now
>>>>>>>>> that Martin and I, well in the end mostly Martin, provide a
>>>>>>>>> HibernateStore and this HibernateStore gets more and more
>>>>>>>>> features
>>>>>>>>> I expect that there will be a trend away from the DBStore.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This is probably linked to the way the mapping to db is done,
>>>>>>>>> here's
>>>>>>>>> the config I used (came from CDO-server example if I recall
>>>>>>>>> correctly).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <repository name="repo1">
>>>>>>>>> <property name="overrideUUID"
>>>>>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>>>>>> <property name="supportingAudits" value="true"/>
>>>>>>>>> <property name="verifyingRevisions"
>>>>>>>>> value="false"/>
>>>>>>>>> <property name="currentLRUCapacity"
>>>>>>>>> value="10000"/>
>>>>>>>>> I guess this one could be larger.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> <property name="revisedLRUCapacity" value="100"/>
>>>>>>>>> <store type="db">
>>>>>>>>> <!-- type: horizontal | vertical | <any
>>>>>>>>> user-contributed type> -->
>>>>>>>>> <mappingStrategy
>>>>>>>>> type="horizontal">
>>>>>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>>>>>> ONE_TABLE_PER_CLASS |
>>>>>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY |
>>>>>>>>> LIKE_ATTRIBUTES
>>>>>>>>> -->
>>>>>>>>> <property name="toManyReferences"
>>>>>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>>>>>
>>>>>>>>> <!-- LIKE_ATTRIBUTES |
>>>>>>>>> LIKE_TO_MANY_REFERENCES -->
>>>>>>>>> <property name="toOneReferences"
>>>>>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>>>>>
>>>>>>>>> <!-- MODEL | STRATEGY -->
>>>>>>>>> <property
>>>>>>>>> name="mappingPrecedence"
>>>>>>>>> value="MODEL"/>
>>>>>>>>> This one is obsolete.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> </mappingStrategy>
>>>>>>>>> <dbAdapter
>>>>>>>>> name="derby-embedded"/>
>>>>>>>>> <dataSource
>>>>>>>>>
>>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>>>>>>> databaseName="/tmp/cdodb1"
>>>>>>>>> createDatabase="create"/>
>>>>>>>>>
>>>>>>>>> </store>
>>>>>>>>> </repository>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> What do you think about that ? What throughput do you think I
>>>>>>>>> should
>>>>>>>>> expect ? As I said that's very difficult to judge.
>>>>>>>>> I believe it's more appropriate to compare the actual values
>>>>>>>>> with
>>>>>>>>> values from other measures (other setups, properties, etc.
>>>>>>>>> or even
>>>>>>>>> other products) instead of random expectations. Apologies
>>>>>>>>> that I
>>>>>>>>> didn't have the time so far to provide some reference measures
>>>>>>>>> myself. Maybe this gets better now that the team starts to grow
>>>>>>>>> slowly ;-)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> What are the main reasons fur such figures ?
>>>>>>>>> Hmm, I *suspect* you don't like the figures? I can find no
>>>>>>>>> evidence your assessment ;-)
>>>>>>>>>
>>>>>>>>> Ahh,one more note:
>>>>>>>>> Tracing can also have a significant impact on performance and
>>>>>>>>> end-to-end throughput. Turn it off by all means. And please
>>>>>>>>> make
>>>>>>>>> sure that it's really turned off, not only invisible (i.e.
>>>>>>>>> console
>>>>>>>>> is not a target).
>>>>>>>>>
>>>>>>>>> Cheers
>>>>>>>>> /Eike
>>>
>>
>
>

--------------050306020307050804040207
Content-Type: application/x-zip-compressed;
name="org.eclipse.cdo.bug.zip"
Content-Transfer-Encoding: base64
Content-Disposition: inline;
filename="org.eclipse.cdo.bug.zip"

UEsDBBQAAAAIAAgpwTjj4/0u0gAAAKQBAAAeAAAAb3JnLmVjbGlwc2UuY2Rv LmJ1Zy8uY2xh
c3NwYXRolZDBSgMxEIbPCr7DkvvOWkHwsGsRWcGCVdq1V0mTIR2NkzhJxL69 K1oUD0Vv8w/f
fAx/O3179tUrSqLAnZrAsaqQTbDErlP3w1V9pqbnR4et8TqlqPNmDAffCTnL tnoitp0ygVX1
sexUEAdoPMWE8GgzeF3YbEYnzBb9w+XtfLi4nveL5jdHnFFYe7C4Lg4KfV2i wDJrtlrs6mbY
RmxmJ8u+nsCpav79ULQIJgiC4EshQXvniyNO+1RJzE41jrVD/ju9jwwlx5J3 8Jo+tW3zs+53
UEsDBBQAAAAIAMaDvjhMuH3Q+wAAAEoDAAAcAAAAb3JnLmVjbGlwc2UuY2Rv LmJ1Zy8ucHJv
amVjdL2TzW4CIRSF1/UpzOyFducCx/Qn3dU0sX0AhFuKgQvhx9S3L8yMplNr YtLE3TmXc/ku
Adjyy5rpDkLUDhfNHbltpoDCSY1q0by/Pc/mzbKdMB/cFkR6giiC9qmE28kN Q26hdUEREEb7
CERIRzZZMdqtlIRw1gKmltGDKsVhs1g1/Wk2WRu59iCKGdxjaeMoa+EP3lYm Ilwogu94l4dw
hJcGHlSu1NhbOvKM/iZcgvQSyAtH/QExPVyNuBafYPn1eNxrkpwzsap75GYf dfw3figMN1zI
KQfoo70+mePVZKVx1S1Wbhc6kx89Bryo5exRT4gHWaY9vtnRb/gGUEsDBAoA AAAAAMo0wTgA
AAAAAAAAAAAAAAAeAAAAb3JnLmVjbGlwc2UuY2RvLmJ1Zy8uc2V0dGluZ3Mv UEsDBBQAAAAI
AMaDvjjku112lgAAAEQBAAA4AAAAb3JnLmVjbGlwc2UuY2RvLmJ1Zy8uc2V0 dGluZ3Mvb3Jn
LmVjbGlwc2UuamR0LmNvcmUucHJlZnOVjr0KgzAAhHefItA9qFQQwal06CAU 6gukekpK/rjE
Qt++Gdq5djmO4z7uDuMGMaiXqEtRHbum7OpWnM63MQdlW2AyOkTIQCwg3IQo n2DU3vVV4bnK
b+MxJzl5IosN2oDZzFjhZFJcka5GpcXT9pVsfoI2GK3y2I5yoL8bWKliBNNl hkt60WAP0nMn
DLfZP9HoN34OvgFQSwMEFAAAAAgAuzDBOB0MEFSNAAAA+AAAACQAAABvcmcu ZWNsaXBzZS5j
ZG8uYnVnL2J1aWxkLnByb3BlcnRpZXN1jzkOAjEMRXufIhJTMskFAIkCpCmG ipIGEisEJWPL
SSS4PYGK1d173/ICM7WwxDcJ/lxW0Mi8YhPd4DqAU5h0mGysDrNaKj0/gHqv RA6j+fbjZr/u
h932R8Sx+jb3muLfjIUYpQTMcDlK1pYSh4iaxKE8DoFMVSxq3SCL7T1OH5ua NUC1cC3PrvaK
gTtQSwMEFAAAAAgAxoO+OAyLrPzUAAAATgEAAB0AAABvcmcuZWNsaXBzZS5j ZG8uYnVnL2Rl
cmJ5LmxvZ6WOTU7DMBCF9z7F7AoLu+M4KUkkFkBFN7CiFxj/0EQCO3ImoN4e R1VOwNvMSO/N
N0/K/0lUiK3ERlYdYNc3XY+otK7g9H7uBTynxGO8wDFke4WfkOcxRTgPAZ4m cmV8pE/+pRzg
NS3RE6+23MzblQSNyiit6rLeNQf90NX3PYxxZoougENtWkQvUWuSptZe2saW UqsOlozBVhRs
oZOlOYAfc3Cc8hX2/D3techpuQw8LQwgxHGLvXzRPMNbIh8ylF+Zgy8N/FpK bTDl1tREPDzu
duIPUEsDBBQAAAAIAEAvwThXhVk+yAUAAJUYAAArAAAAb3JnLmVjbGlwc2Uu Y2RvLmJ1Zy9M
YXVuY2hTZXJ2ZXJUZXN0LmxhdW5jaKVZbW/bNhD+PmD/wTD6qVjol7RdBjQr jMTr0jaO4TgF
9smgRFqmQ5EqSfkFRf97SVm2ZelEqxmMAPYd7+7hvfCOzPsPm5i3VlRpJsV1 u4e67RYVoSRM
RNftp+k/F1ftljZYEMyloNdtIdsf/v79t/ccpyJc3EgxZ1GqsLHiLbNN7Aqp IkRDzhJNUUIo
Shn6lApmvhQk2k5FICWnWAyMUSxIDW090+11GycJFQRhFel2a4V5alUaldJ2 x8lou1ZEVRHO
wgzDQaSMYukgIJUKw2KKQqmooXZfBcmOB5N+Di1ZHbTPMdfUK5EaGVu94YCQ F0h9xZwRbGhT
D1h1xjJwchDwLA4XNHymBHQVjecoJPIPgIYSJY0MJQeZmiqbRR4WWrCAKmE3 1WgRYuRknaDm
zbJKQctVDFBNmADU1DDuc38W4zxBq473yKyhVPWu5zKC0gKOV4boiyxl+Kvv a6medYJDOuMy
/NFBMTXYZg3uoISnERO6U64Cl/cd++UiKwcfTkLnOOWmMUhCNeU0NJTMjrBy GGCm7UKyMCZB
rhI1FNtYe3mYxEyc8NxqML0yNWdScLcmxiKSiBJmaqwCmV6AlDCA43YJkElQ u70gJYRRiJN5
LJRxLAW4C7rBccJLoqUqI4F3b2hJgtDnIkkov/T4CKpI/0Z950fBaK/OqCtt 22lAJ9crPZcG
IJ/IEALAzmG/9Oz+vHS/SVojy6YC8g8M2ibgQn/jYDqECwyXUiUCewbo/v0O weaRo6BBGsHo
4q2Fd9Y19ZDWFNdl1Y6F8JLAojGL7FgjYeF9kSGduLMQBk+oCrYAC6zQo9Nh P5b7HFQcgFh2
WnhqK/dCyny9gImQp4Q+JK7/YN50KOHnOxbqZI3o4kDfqeNMm5KyIvgsYbJe hu4H4/HwdjYZ
Pj48TW6Gs/Fg+u9je69jKIza7q2fNEO3d6uko1Xo6J2c3rH0jqPvhtXHzHFT 62K0xCu8A9c5
Qff/4E7/Gw9r4PbqrIG+LtqzGZ2PuzcPo+ngbjSclMZCOMywjs/D4XjyNBrd jT7OBtPppPFE
AKubDh+no8H9sMmgWq9h9vludAuOFcd1XGLiepn7cfkLVnY3G7vORutulIUI tJSnECqnyotM
jScPHyeD+9lg8vHpfjiaPh5MXkjdevXd2PsQNUjqH62LdYGwdgSswsWR5H5Z ouBHkuA/Xorq
0/Bmehp3wAcvUl6o2Jk19PXudggbyS+R7owIqLtCcqztYWEWYyVXzMbYY97J 5rfbg+5L5EsH
26hIGhoQSWmM2i+tV7aficfZIOyZg49XE6+e2S6g3sE6O2gU1TJVoe1QayYu +2hz9e70/P+W
MiE30CC5oPx0WC36PmNge1IvaKG/ZTX2+hJdoT5a9bvdq+5l/8+yjhWja2Di qgLINjBntkNu
taFxYQfuEN5k3ufUvO6jt6ib23vTe9vv9d+c2kxINhMd3ihKzdzty6k+Ieq1 8YGp8g6errJW
WDEcQLM4zfibuDr2ZZyqKpslJ0SpI5Y5gpUtL+e2jdqxYAPsYymDEpY8DRSN bJtRp9OK9Y+t
rgpCQwWtTsYZtTCJxu4dRxXTZRdpbc/lyOZu9LqH3CdPl+673l+9LhS+0jSU 7cOG1Noz7r2p
hBiIz+7JpxzkPKv81aFpmCpmtl5mjY5SGdrd2yOLBYyX9VVCdchbVd4dmktV ul0ZiuPq0mNE
sBDSZGMYkKB0k1iEusLMcqgOE1A0u5xzTw8BE6Q8EOchi+2VigAgvGLHB7uC ++B03ddJIR5W
CrHA/oXpyVIsDOCzPK7WJXOq7GFXKixXcqCAzfSmwHFqFk2O+WZPKHn3PTHu vfRWHtmqtSzA
97O9AvAW57wp87vBrhWAiOqvmq75efBCrGbb4yxQ9gwGr1rgBa1W09nXyDpJ 38CtF3I9O8Rc
Cr5t/lRsFA5tAjV+d0w1vd096f3iA2dZcKAobg7TSo9LA1VBpgP8F8HSfwJQ SwMECgAAAAAA
yjTBOAAAAAAAAAAAAAAAAB0AAABvcmcuZWNsaXBzZS5jZG8uYnVnL01FVEEt SU5GL1BLAwQK
AAAAAADGg7443Tz9/zoAAAA6AAAAIwAAAG9yZy5lY2xpcHNlLmNkby5idWcv TUVUQS1JTkYv
Q0RPLk1GVGhpcyBpcyBhIG1hcmtlciBmaWxlIGZvciBidW5kbGVzIHdpdGgg Q0RPIG5hdGl2
ZSBtb2RlbHMuClBLAwQUAAAACABbMsE4uA/lHVIBAADkAgAAKAAAAG9yZy5l Y2xpcHNlLmNk
by5idWcvTUVUQS1JTkYvTUFOSUZFU1QuTUaVUstOwkAU3fcrGhJ39gaIJqaE hWA3RrSRxP20
vZSL05kyj4by9fYJWNHEVTs5j3vnnFkxQRvUxvtApUkK353A2FlYkXD0Vh14 wqY98soy9N2b
nNuURH3ogXWZRZJT3BKkSgFjTrlGiBMJkU1nmkTK0VR2c6PsSXg5/7zBkjOt Q2a2vgtnpkik
qqcrWVCC6nL+i4wZpyMzjVe7X4+9496SwiQ4YGxrQiAKUlJkKIzvPk/XgTeB e6ejea1qcAmp
EJQVhjK8db5BmG2q/wqfFaQpIk6m9OcK8ZBLZa6Qq0D+QQWNqkA1i9q7FG1c 89EYHmA86jQ7
K+hvPWwpQiWYQTAoUP7wa+IfDTwEmrsdJBFs9Z4n0S+iPufH2FDRNBDWT6H0 Xc6O5QCtG7zy
PGD59LawadgWFzSBeCGLP1laVSGqpnW1W/MFynJ+OlSF8sHWnaXzBVBLAwQK AAAAAADKNME4
AAAAAAAAAAAAAAAAGgAAAG9yZy5lY2xpcHNlLmNkby5idWcvbW9kZWwvUEsD BBQAAAAIAMaD
vjhyMBIWWAEAABkDAAAlAAAAb3JnLmVjbGlwc2UuY2RvLmJ1Zy9tb2RlbC9u YW1lcy5lY29y
ZaVSXWvCMBR991eE7Fmj7mWUVtmcgrCBqIO9xvS2hjVJycfq/v1uW3UWFDaW hzZtzjn3nHsT
Tw+qIJ9gnTQ6oaPBkBLQwqRS5wl92y76D3Q66cUgjIVovuLig+dADkpGZ9IY ST2CC6W0i/As
oXvvy4ixqqoGRuUDY3P2/rqkJ4jrQqr7BjEeDkcIe9mIPSjel9p5rgVcijc+ OlwQhSwdNAKg
slpkzOY1jBLNFYLrp2tFtHtbL2/SlUmhwOBMpIbtQs5aJrJWFjJ5OElNUCuG WcGdk5nENhAM
FPmvEosdG9UcngzMjPZcarANs+ZuvA3CB8uLBXB8wxWJR++t3AXfyYHT2V6C nrnn9Q/yq47c
MTbH0hiRsr9YWUMGFupRHK14vivwI5Ql2CcTdJrQ/qhtcb2OHrHcT3Qi2q0C 7VHABvinB2GK
oDS9Wuzs5EbRmF2OD+83617wSe8bUEsDBBQAAAAIAMaDvjiE2blUBAIAABAF AAAoAAAAb3Jn
LmVjbGlwc2UuY2RvLmJ1Zy9tb2RlbC9uYW1lcy5nZW5tb2RlbMVUXY+bMBB8 z6+w3OeL00iV
KgScqnxUkXp3UdVWfXXMQtwaG9kmyf37rg2kcJdTH8sDAnZmdnbHIr2/1Iqc wDppdEbfzxeU
gBamkLrK6Pdv27uP9D6fpRXo2hSgks+gH8IDudQyufKWyJsRvFBNuwRrGT16 3ySMnc/nuamr
ubEV+/mwoz0EhLEwAYFQsnEQgVCXbLlYLNkmwMbSg5F/UwenlETCWloQ3tjn jDLEXTmiMPND
WzFnxR2Kd70iY6/aSurdOqM38L3qI69xinB3lFhj/ObiQRdupz3YkguYktFc FFitn54Ov9BP
125EXCnu3GuSDHqaqyl7Vzc4noVS4Ys8wRoUVNzHSLxt+8XJujEW6S8nibp9 LehSUgL3rZ3I
fL2KUyIM9pNcC/gCpxDBhyF2YZpnK6uj30pQBfovuXJAcyymJSYoqy6LXIdN zWP4KZtUAnS0
9h/cSn5Q4HKcNruxw5TdRAcZjHHPxW9egSMNbkderhEV0jXGBejempMswG55 fyriwki01rMz
OrL7jsVxOvmYEqrHSh/ZBMtWRnsuNdie1fG23YIdERafYHWUqhiW1Yn1iIzG t2TzyXsrD60H
8oY+C98pu9mlsaYB63G4R6NDgKGfheF0TF2M5n9pAg8BIE28bcKHlf53F8Ko ttaDjZT9TSqe
CzY6GPhPY69+avnsD1BLAwQUAAAACADGg744oyIn2E4AAABgAAAAJQAAAG9y Zy5lY2xpcHNl
LmNkby5idWcvcGx1Z2luLnByb3BlcnRpZXPjUlawSc4vqCzKTM8oseMC8vSR uUABFc8UFS6u
gpzS9Mw8v8TcVAVbBRBVrOCbn5Kaw1VQlF+WmZJaBJUqLy/XS61IzC3ISdXL L0rnAgBQSwME
FAAAAAgAxoO+ONU+MTvjAAAAdgEAAB4AAABvcmcuZWNsaXBzZS5jZG8uYnVn L3BsdWdpbi54
bWxVj0FPxCAQhe/8CpzslaLxYgy0NxMPGg96NhVGliwFAtSu/17o1tXlQHjD m2/mieE4OfqF
KdvgJdx010DRq6CtNxLeXh/YHQw9EQMqZ2PGP+tttdYfIq4YI1SoEL+TNfvS V8H/KUJ3j3pH
GGve6GZjfStSKvBY0DcYjcH6IiEk021zOpw+6zsk7Ax6TGNB/R5HdRgNQt/a G2ErbPJ05mQl
7EuJ95wvy3IGVjifgkZXk3GlA/+YDffjhBku+pUbc5aw/nTP7X7Zxl7Y6lZP jSZhhZ5IbddV
Al93FPycsaXnv/F/AFBLAwQKAAAAAADKNME4AAAAAAAAAAAAAAAAGAAAAG9y Zy5lY2xpcHNl
LmNkby5idWcvc3JjL1BLAwQUAAAACAC0McE4Lk88qGcBAAChAwAAJgAAAG9y Zy5lY2xpcHNl
LmNkby5idWcvc3JjL2FwcC5wcm9wZXJ0aWVznZJdb4IwFIbv+RVNvLZuM9kF CRcodZIIOKnJ
vDKlbQZLoa5Fjfv1K/g9QZP1qj3nfU/OOU/TLOGqICWHaZK/MCYgWZfSoYqb WJcpubLSk0Sn
crvU38Ip1ZpfxKksCk7LTBZwJaVY6uyHO89PltUB3f8f4/bQyJ1PMIgRxn74 FoNRNAN+CAIU
RLMFGMfvE28APBe7AzdG0OpcNMUyIkxTjlSf8CYKa+v+3jwIU9mGqyUVROt9 DTM4S+AXS6hX
55p9ayWcSmPv9XbOc5syedK2uHQVy7mjScteTRtbqZjTnK6gUZnnWXnDhtCU w5WSm4xdz/NH
MiY6LUki+LB6Tg8Gw/AxqXvHUEQfbjCdoCPFGmKwMADO5MA8HEZBgEIM3NAD yPMxwBGY11CB
1XmMyEwP853ZeU0IHhA1G0+MaoPd6wlJiUilLu1+/+m1d8WrpcQRmNG2SM7I LvL3f2WwM1vx
i0J6g+Pn/AVQSwMEFAAAAAgAxoO+OAFLmkPZAQAA7wQAACgAAABvcmcuZWNs aXBzZS5jZG8u
YnVnL3NyYy9sb2c0ai5wcm9wZXJ0aWVzjZRBb9swDIXv/hUEAt8WDS26Yw5Z imIoUjRFdhgw
7CDbrK1UER1KTup/P8pxjTZt4x7jx/eRepQygTUGYKIAlsoSGSzu0UIgQGZi 0K4AEzyQsy3o
ukZXSJHI8wuViOVqo6J7oQOWxO2sc30TNenVvFfU7er+z+z+5iZJJiKD8eCl tZAyBA0Lcp4s
zvsOL+yXjmp+MSMu5bfOK1RH8cTTgxuPHlY6BGS31C014QOYsp3ynvnG+Kkv tt4je0OuN8zS
6RXD3zT8g3T6o4Y0h/QZppBuUyeDHUHHiFVsWplMXJLNMbFktGDy5qOqGUNo hTyZxM3Br4cl
7BrkFmrNXlak82D2Rko+J6tqZ5X2Qc3Xv/suA2/TeFlOhbB+WJ5DiHxqvb3+ uYDMyMWRUfQW
JR4PIp7DhLYeDvq1sqGdl+VtNeBzTRy+N3UhhaPtiKyqsu1lUdhTXkyyyxAC I/qR/E7debxL
Q/hjc3TVp4jA2vlIIPelLb6q/3AVjJ4azuNUu8Z4EwvP8TZFlr8GodOZxe4y PJK1dDCuBGsc
gnkEeQ1w0K57yDJI/gQFHRzk5BwejxAj6OZB/aRLeZuHCp280ki5ZiPP6E47 EXgxeFZMexP/
Z8biGxxqhNSf5z9QSwMECgAAAAAAyjTBOAAAAAAAAAAAAAAAABwAAABvcmcu ZWNsaXBzZS5j
ZG8uYnVnL3NyYy9vcmcvUEsDBAoAAAAAAMo0wTgAAAAAAAAAAAAAAAAkAAAA b3JnLmVjbGlw
c2UuY2RvLmJ1Zy9zcmMvb3JnL2VjbGlwc2UvUEsDBAoAAAAAAMo0wTgAAAAA AAAAAAAAAAAo
AAAAb3JnLmVjbGlwc2UuY2RvLmJ1Zy9zcmMvb3JnL2VjbGlwc2UvY2RvL1BL AwQKAAAAAADK
NME4AAAAAAAAAAAAAAAALAAAAG9yZy5lY2xpcHNlLmNkby5idWcvc3JjL29y Zy9lY2xpcHNl
L2Nkby9idWcvUEsDBBQAAAAIAMaDvjh5kB2zlAEAAOIDAAA9AAAAb3JnLmVj bGlwc2UuY2Rv
LmJ1Zy9zcmMvb3JnL2VjbGlwc2UvY2RvL2J1Zy9DRE9CdWdQbHVnaW4uamF2 Ya2S3WobMRCF
r7VPMSQ3tiG7D7AUQuISHEoT+nNdFO14LSJLYjRyUkrevbPW2vX2h9DSq2Wl c47mm5mozaPu
EQL1NRpnY8LadKF+yH1bVXYbA/H0MhDWlD3bLdb3LvfWt6e6kHpbr0lv8SnQ Y32VfefwOnjG
Z5bEZrGoYAGfNgjasN1pDgTG6ZTAiIiCS8ByGSX5wnpwdo1gvhqHYmuqmB+c NaPhenl3lftS
A0g8+i7B+PutqlTT7N85RK2WlRr9iTXLZ229dvCRyfoe7t99vlm9/7Jawhs4 +007ztofkWmj
CTuwXoK8QcmlAQUPwZPK4tgkJXahVyO+4CambKQBw1lzLO7UPJsLiXoZXi7O mQ/+4lbvdBfM
fH9ymRBfm1A6l8KIZ68NaD6tZBdsB8U5ke1HNchlVBSeErx9NhjZhqHvSqUc keriOygFX5VG
SHt5Y1P7H7CEKsR/hBLj3zAda/fZufYEUnJOGAtSYfqAnMmXdf5lYaBoLmkv +pOm+XljJ4vV
Iy9xrbPjsiZqDDsunFTzUn0HUEsDBBQAAAAIAIMswThuaWhuKAQAAC8MAABK AAAAb3JnLmVj
bGlwc2UuY2RvLmJ1Zy9zcmMvb3JnL2VjbGlwc2UvY2RvL2J1Zy9IYlN0b3Jl UmVwb3NpdG9y
eVByb3ZpZGVyLmphdmGtVttu4kgQfecrSjyZEbTJbjJhkt3VIGAmlpKAuMxq nlBjF7gT39Td
QNCKf99q22AugSXasRDgdp1T1VWnqm1/+nVXCT5BK05WUsx8DZZbgd/q9Wuo mZ8GdMQrwkBj
kqCswneUIY9WVeCRB7H2USpm8M0ggBSvQKJCuUCPwdAXChIZzyQPUwDZA3fd OEyIQ0QzCLlG
KXigDAeXSAsemSy4CPgkQJhHHsoURnahgnia3nTcQCQKoTefBMKFR+FiRLeL K1Y3REtfuP7W
kUBFIIrEE0pLMZlrEUfZBmix8MW1wfpaJ3e2vVwuGWZeWCxndoAzHtiYBLXF VZ35OgyMcZa5
KGONpbozC3Tt5ozyKCKhaZPQ7DmZ2zAJMMRIcxMKYX7ZZZcS7r7yGQIFvd2A 68VsMp/dl0rk
OZZ67yGG09QgLZpkrXZ3kP4baRHcXwBw+pjEStD2VxeZD8gSP0jMejJO1CUg X0xQRqQq5jxs
/j7xJCGxEcdCkJw+RqMxwpgNzff/IRSRNthgh3lLl2fkPZYI9fULI8UG7G9J XtHrvLmYGNkU
gBfScGbzwJVPwd0fP3l/1aQVpaYOITY7mwRf+Vz7sdzT8O76EydABEPOTQuQ 4LIedAOuFDxM
0s0UldvkCPCNMukpOPG89E8JIKdSpi9cmArKF9lLMyiemr2e8/x9/M157MCf UA6zEkxFgGWK
nbBSLCiXG/DpQEREJpGLxBLh8rShVcl492I6TTtD7eTMVoVwZj9As1DPZbT1 eU+L62PWRSw8
UDsE/x194eKkLdvZ6VEAX7ukSklmRTA7/QauREpmsWDlZYh4iFWSQPJHtlDN 6/MXzWelCzkV
4b1nm5hu3uQ/k+yhjcm+gaemLJlrKx0BrNfv9sbdH51+32l3xqOR065Cedq4 atDntvb5xsPa
tdvgtQbe3tSw7uLNl+svjd9vefkc4WDU63X7QyOw5qjtDAeGlE4lvBTV7/xw Bk73edzuPA6b
l8FpE863n7voy3CtUb/feR6OH/ujcavZa7ac4U8CXtXpOgtM3XTaZ4Hc87Zz KR16RU2tlDTr
CgDbhjiXECyF9vNDWunUd4bYD4VeEawDleRceZfsHT3sSIKZ9rLlVPJWpQqb mDZ9lU+BtKPO
7eWkLAvlarlKf7M7yOdRwQF6nzMX9I6TPKdwaMmCmHsWTYyWGZpWhdHfPqp4 Ll1sKgoHeWiV
bRpxrEhmubKlm9IYtrqTF3Q1vOIK7o4c0OoAtVWp5IjNHnaVkfd0hWyrRwQU UX632jUsYliX
im+Xa3rfsrZHE2BlL3Pal/ESDg8wtqQFC3PG9bnJlJZ7v/TbMp085yE8uM/q c/Yo35SM1J2L
Mp1Reyd11gfVQ/oC+dGISPeOeT0weYklFT43UzZ58uauZv4kZG9hsGnS06G9 F9O6tC79C1BL
AwQUAAAACAAgL8E4jF8EPboDAABoDAAAQQAAAG9yZy5lY2xpcHNlLmNkby5i dWcvc3JjL29y
Zy9lY2xpcHNlL2Nkby9idWcvTGF1bmNoU2VydmVyVGVzdC5qYXZhzVbLbus2 EN37KwhtKgMG
gwRd1U3bQPG9MZBYhq10VSCgqbHDXIlUScrxRZF/LynqQdnxA0VRdGPT5Jx5 nDkcuiD0G9kA
EnKDgWasUIBpKvCq3IwHA5YXQmrESQ4KR4JrwjjIcX9/Zj+/EKqF/N5hfIeQ ryun0X28BKWY
4OMzZokkXBmX50wLKbSgIrOYeb02eSpNuFYnkQrkFqRLya6eNcsuAUwXUAjF ThfLuAbJSbZX
9UmSOOgf3zpkafLBIseZ2OC5NNuPYvNAeJp5/F8A15JQcA4SuzzvokLSptd4 Os/KDeOHzT+C
ZAJP46NkeoYmufhpnhG9FjI/TiVQIQFLUKKUppJFvegAb2RLbNSqxqWWQPJx 76yK9kDU6xMp
PjmpdtvtkjON19JI+l3IbzgBpSOibLiiXGWMIpoRpdAjKTl9dcqxNgh2Gniq UAMY/DVAqJBs
SzQgI0dtoGtmOoNWQmRAOEoWd9HkJYlfoni2jB8n6BZpWdpIlwK/TCvUmmTq U5jHCKp00LBj
TH+LTeaSpVDBhAaqIUVbwVKkQD8X4RDpVyneFZrsKBT2IhpLWxRCqiyMMGqz cbXVdRJPZ8vk
bhZNMEnTTrPhnoZxXXWNZ2sU9uoaVtsuHvKzNwVzePdrCwMncr3TQe3uaEL+ HQhbP71dL9aw
dvcxOEyyyb+X5yVRDyLuUVEHM7L82WTB+GaE3PcvtlGFqgmoBb1v0zTEG1NI dstb9LBamhV0
x2Zsbo0OJN6AnlaTk0I4xNQQoD2zMLBuroORy6IO0xuftlQPsD85WkpGXkbD sSv3qLFNa5JB
DlyHweEoIdSqU0hlEgvetrn9SmFNyqwWw8cZtWsg8l6881OCdwMN00wo6Olj 7N2HzlEbtrmP
3QuAHK31r3DYhnDts8mlJdVfpSgL06yDpwPPF/H9c5S8fF3Ez/OxD9XfC3CQ g4fQopI4ih9f
ZndPkx4qBUUlq8o1YMvgT1dXNYG/dm2yb/ytE4CDS9Cl5CjsMhxe1EK/wFGV 88jPwePOjdu6
R0rPzISNRFbm3OZyqlse26r+vt3nvVWv90cD6Z0xrCFYFMC9wwbSvD+GAHsT 9a69KG4/DK7M
yXXQ6LplA9kn9dpA/L9LGFqOnJvWvIlXoeysraoOqp/NlOv7vvk3fN8EvTPT uISsMjsPzN0O
K4thIwBlj61T01flWVw3tVtyRJ4z3QRsuHUX6XivnTZ/J1n5f2/0zX/S6NNc H1D9mT9LK/qz
NKMP/RGgraU2+CfYH85hP+3xx+BvUEsDBBQAAAAIAJcrwTg9kHQ/kgIAAOcG AABIAAAAb3Jn
LmVjbGlwc2UuY2RvLmJ1Zy9zcmMvb3JnL2VjbGlwc2UvY2RvL2J1Zy9TdG9y ZVJlcG9zaXRv
cnlQcm92aWRlci5qYXZhrVTfb9owEH7PX3HqE1TU0GkPU9mmIYo6pFIifkza EzLJNfHqxJbt
gKqp//sucUSglLaa6pckl+/77vPd2d3zj1sBnMNQ6UcjktRBK2rDp17vM1yU jy8wEg8Ic4da
o+nADZqM548d4HkMyqVoLCv5Aymh4lswaNFsMGawSIUFbVRieFYRCA88ilSm SUPkCWTcoRFc
2lKDG6RATJANF5KvJUKRx2gqGuEyC+q++hhFUmiLEBZrKSK4FRHm9Lm5ZL1S aJuKKN0lEmiJ
RE5iYZ0R68IJlfsNULDJxV3JTZ3TV93udrtl6LMwZZKuxITLLmp5sbnssdRl sgT7yuVeVRl7
VQZo7deM6ihy4WiTMAjHPm2mJWaYO15aIc6HrW6gefTAEwQyvdtAFCu2LpJ+ EFBmZdzBT8zu
K0DVNMOG19N59bZ0QvbfQRjPUCsraPuP74LPCYnvQVbA/3DBQqO0bTb7h1rM qOmS/eQ2nXDd
P/5TRYOuPws/eOFSZQ66uB+fcONEDgvOyyGgkvspjCS3FirXjRnyshE0w8Hf AOgoiA0NPNiy
8dEpKI0LAfII4RvkuD0Fa7XJMWn65G9IJujGtWqrTazSDdBJdYXJd/n6FHw6 1twoEYPdE3jL
d5PgBJLt7fA4uVEOI4cx+FGByCAVrXo/9r4bEuZhk9GkRr4k2FipVZsA7cqU N1LOM+xQi/VX
H+iAf36nG8i6crSQuo+2cfISVpcjWPevnrrnGO8QPJTpwrWquWXhbBqu5ssw nM4W47ub1Wz0
azwfT+9W16PbxWDegTNnCjx7jT1czmaju8XqdrZcDQfhYDhe/CbeZY/Wq8Qq 1+j6JPGQSXd+
61lRakjdnIO7hB1V3Jf6oL0dr1437yn4B1BLAwQKAAAAAADKNME4AAAAAAAA AAAAAAAAHAAA
AG9yZy5lY2xpcHNlLmNkby5idWcvc3JjLWdlbi9QSwMECgAAAAAAyjTBOAAA AAAAAAAAAAAA
ACIAAABvcmcuZWNsaXBzZS5jZG8uYnVnL3NyYy1nZW4vbmFtZXMvUEsDBBQA AAAIAMaDvjiN
QfxktAIAAO4KAAAwAAAAb3JnLmVjbGlwc2UuY2RvLmJ1Zy9zcmMtZ2VuL25h bWVzL0NvbnRh
aW5lci5qYXZh1VZdT9swFH1ufsUdILVFa/IOWdWJ7QFpGtPgfXKS2+Dh2JHt wBDiv89fSdou
XWnhYXuqY997fH3O8XWT09MITiHNRf0oaXmr5+4zWfu2UyeXxYn5TaKa5Hek ROCkQnUeRbSq
hdQgZBljzmitMMZqGeeFiC8+XV1lPzHX51ujRFUJHjeasvjzF6pMZJSEkt7N ZpBhSfmsUShn
hchhNnPlfQSJtUSFXBNNBQexBH2LUIkCGQi3JYxTrOZpNr8QJopylGmSzdPE TI7jDh95sYnu
1mq3z43BXArGxAPlJSyR6MbsCkQiqKa2B8LizMU3zCUApIzOnxaM8jtPUNxt f1yi/mqmwNZl
B66W5zQxGS/JvSEZ88lutGf2hWBNxV26H27kp0k4QprULQsLhUHm2Narvnnl PVzAnkxdpGPe
jfCXNqQq6LR3syWaWGLo8hZqMkZzoFyjXJIcoYODP9LhKRpZR4wMzHc0AnDl tL4nrMFW+FZr
T2srMxCtJc0ajbFL3+aoUVDc/l4GJyHhVvNV/E6zFWCgio815AyJfO8ATLix h0TC2COoW9Gw
wuxprCldtQQKVLmktfOtjY3jUF0SShg2pl1ZSEfAlvMP1eehnZDHyvtvcm3W eDntV3ZI/MNl
hXgv9Ghd05EVdeRxIdh8MjVXuVPuGvWQbHvdlP303E5iTSSpQiG2Co4PB7DZ HXOYjHtBC1hn
3G+yRstOQ4er3jk69xxVpvMZjy2Nfbi5Psw0Tl+b7Vj2ywWaIN+sLOhjjTBM 9/Ob3I6+KW2r
8h+6KruL7Sl90RVxgGt3ZBX3w5GWDR4NO8U9e2kHNYe210/2s0rb1v/ulTfR euUF+Q/E3l3t
Rpf0CZNOkj1aZUg9yAj9I9i9169solse/cOt8dqmup8UKzQMM9Y22U3B+k77 DEnS/7uIfgNQ
SwMECgAAAAAAyjTBOAAAAAAAAAAAAAAAACcAAABvcmcuZWNsaXBzZS5jZG8u YnVnL3NyYy1n
ZW4vbmFtZXMvaW1wbC9QSwMEFAAAAAgAxoO+ODzLGO1dAgAAiAgAADkAAABv cmcuZWNsaXBz
ZS5jZG8uYnVnL3NyYy1nZW4vbmFtZXMvaW1wbC9Db250YWluZXJJbXBsLmph dmG9lF9v2jAQ
wJ+TT3HrKhXQkux5RAjG2IREYRpMe6wS5wgejh05Truq4rvPTkJCqLrRafAS 2xff3e98f7xe
z4Ye+ESkj5LGGzUojl7rbETX0+har56dBmQbxAg8SDBzaZKyvm3rRUhVycaC q4BylP22fG6+
X0v1RkfI2EXCaJqhi8naJSJJBHdzRZk7mdFM9V+6iURIdCdjFmTZy/YoVyh5 wFwSCXf8abEI
fyJR05Lbq8J/4zgQYky5k2conUgQcJziKUYcTIyYoA5KUcFBrEFtEBIRIQNR WIMbH5OBHw7q
0H0vHPieFt64tQPk0TPzflosK21wLRgTD5THsMZA5RIzCCQ23jH6UGjkrFAB 8BkdPA0Z5duD
ZDSvb0J8G6Myzw6Gz2wKpp3vadVXGVkFISutFLt/NTMWLE94YafcHhnyvSo6 30v3lTeMUVsI
dPxl/eUhowSIyTq0HAD+UvqNtfQwy80DHlyHJ9syubf+kPz637O8WW0qy2BZ qRRKu8SoDdXp
Gl9WlqcoO92+be3s/+56uLhHKWmEhxRlXwAuTdmS4lCxSNTVxeGwHd0Z1V0S MN28i/lqNJ1P
vl0QVbfonvNzWfpjkXPVxn1/HqCqnJZKms6r2qXtuVP+7OIXVJ2/vNrd3Xx0 O3kHSuZ4pmxX
xPeCRpBVvBU+xwdzLOlxeTruXvFMSV/maarnWfYjkFxzZp2rnJMNki1GV906 omLa+3X3DGA/
d47ScXzvxMSsRh9nF8lMM2XqiXcUQX3jRPTxYvb9dn7Jqqqwm1B0hZSy1xTX nrtRLuF34Hmt
KWn/BlBLAwQUAAAACADGg744RnFH6Z0CAAAiCAAAPAAAAG9yZy5lY2xpcHNl LmNkby5idWcv
c3JjLWdlbi9uYW1lcy9pbXBsL05hbWVzRmFjdG9yeUltcGwuamF2YbVUwU7j MBA9J18xVEik
RY3vFCqq0pV6KYjlB1xnmnrXtSPHoaAV/76246QJLeJCD1E8zmTmvTfPJqNR DCO4Zap41zzf
mqkPSS92W5fL7NK+SVxQ9pfmCJLusEz5rhCTOLYvpU3YGx02lM5TZIIXJaa4 29i10pgu5oKW
5eSbpMf1H2Tmu6ynGs53LR3OdPGLMqP0+7IH+vQPhahyLtOFC5782v5BgloX 4zGs0e6NqxL1
OFMMxmOv3EyCa4U7lIYariSoDZgtwk5lKOB2PQ0Ybsl6mrbFUGZHpe5zlKip wazWvVoLzoA5
6WDlhO6wAXwztkYJXYoHJP0f4F8cOSaRbTLXaDuUHmKGG1oJA5uQ1ieS+vyv qEdfMon6VCLH
JQpkSlea9cFxyU0ydBgjU2ONol6ChdqL7yDpxsPGEumzhVnaGuly9ftltpov 0hxNI1Ay2BpT
3BCy3+/b8VsrED8oLnPCMkXWVU68qQfDCTgkfAPJZwAXdyArIWrMUaTRVFp+ hjlx3z7i+mHU
sC0kizeGhTcJNqtQpGO7A3qh8uSQOAm1QjuJ+yNXJC7pIz6eNrUulVZ8ybDx Zxj6OaZ8DMuT
LKsC9RHEH2p9//iKWvMMWxThPgHmNUjqOwjQvwKgPfdjqfecWfyCbzjq5UMy DLNhtMSaU2O0
+ePqZbZcLZ5vIEwjCd2Gdbe5soeIy0A3isJJu/F+MVut9n5+SyEwp2Km88od u9YeyeDFTqg+
+VcDuIYDQofD6nkNgyvgJUhlgMIrFTyr8z34QWOWMwgd1G0ZwhFjr1ob+4uJ tdl3nnjva61R
ELLNPI9Nug4N04SgahMHAs1gu5+G7j5p034GYYaFRtZAPGHsh07CiWu0w+Ik gZ5xsblbauwf
QMjnwxr/B1BLAwQUAAAACADGg744I4VLk7AGAACPGAAAPAAAAG9yZy5lY2xp cHNlLmNkby5i
dWcvc3JjLWdlbi9uYW1lcy9pbXBsL05hbWVzUGFja2FnZUltcGwuamF2Yc1Y 33MaNxB+hr9i
43QmkIEj6WOcuCH4nN6MDS7gTN484k6AmuPEnIQdkvH/3l1J9wvOOK3jti82 klar3dW3366u
9/JlE17C21Cut6lYLPWJGfYqY5r6JYh+wf+95pqFX9iCQ8JWXHlitY6Pm038 J1Pt5gYy0Uwk
PD2uzg/p7xkLtUy3dUuXVnOhTqYLj4exWCvu8dUcf8uUe35f61TMNpofPyA4 iJlSDwnlpx4W
G/M5T3kSPmgeRSTXGtjw9FyUn3W7MOMLkXQ3iqfdSIbQ7ZqI9xOgjXzFMXZa yATkHPSSw0pG
PIa3sxOn8W1vduLlyngS7al6v+AYe6Z5ZO9rM4tFCCHFAspxJtuAf9WoQ0HZ 4MKS6gb43myQ
J40DrjTutaxRNa1BtjXWqbjBIdirgjBDjhu/g2QTUwDzcwcpR3kFDOOVKM3w Qg5FqgMp2qg0
Xl0Et0IvjZLv72ORfHkAEN7Y7Ey3sDdzB7OtOdLlglGa5cXVOIAbFm+4Z4Ox PhlKzd8YeTwg
5aGGW4b7JYTGmbImEApuBDNTioAQGiVzmzSw4nopo8z+5yIRGuhPq33Xgdul CJfAYiVhzdO5
TFfKbCYBwWLxrYIrd2AHo4Ax0ps0UWa+FK9MxGqZg0w4qkeboy3iBqWU9xgs KM5/9AqKDft0
8ZwPJ9cY9ELmuQ3JYcTtZkKrTfBuqA3GruVUdqBMWR4PhpNpfzjw28fNxl3z yZLBXjzMpIw5
4VwF6A/exzvEQax4TToUMMefLImKO+f2VqtZAQgOnBYWHyZv7DaaZ8kWJO5J FWzWiBcLKwRa
xNdEFR7AxBCE2WwnkRcFJWWKxqPtai7QXMyRkMWI1IU5KwMvqsTZymadQU1h FNEGDpGkbWgK
7sQfHN3Gc6clNREGi9cCnJwQCcaiRn0kKP/ibQclcDZFS7i1hcjB2plnhtCK x/M8jUfzyrqN
M1Jk5SyKQ4ccLM40ATRKluwGESw1bLlGT3lSzjZLKSItCMQGdC5SpR1VGKSY g4uNJi68OI72
oKKkhAHCA+hbicjia/XG3jrXrGspU87+xKAoe/toeZUhrEZ3fnY/eDPWFPxd OqcDShhGTvLt
L1TdUSskwJQqKtKgUYR3qgyTs8SAL496kNzIsMRcBQJuBdpK0WTzOZEqATe3 WS+ZNvG


Re: [CDO] reading/writting throughput performances [message #619128 is a reply to message #124496] Sun, 01 June 2008 08:17 Go to previous message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
This is a multi-part message in MIME format.
--------------000609090607080808010708
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi,

Here is a diagram of the dependencies of the relevant plugins (please
correct me if something is wrong).
The black arrows are ordinary downstream dependencies (Require-Bundle).
The red arrows are upstream dependencies (Eclipse-BuddyPolicy: dependent).



For research purposes I added code to the libraries plugin that tries to
reflectively load CDOFeatureImpl and that is not a problem!
It seems to me as if our problem is not class loading related because
the hibernate.jar should be able to see the needed classes.
I believe this is either a bug in Hibernate or our wrong usage of
Hibernate. I tend to believe the latter ;-)
I put a breakpoint in
org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping .EntityEntityModeToTuplizerMapping(PersistentClass,
EntityMetamodel)
to see how the tuplizers are created. I'm too unexperienced with
Hibernate to get a clue about what happens there...

Btw. I'd prefer to see the application plugin depend on net4j.db instead
of net4j.db.hsqldb.
But it seems as if the BuddyPolicy in net4j.db is not "reexported" so
that the application can see the hsqldb.jar then ;-(
I posted a respective question to equinox...

Cheers
/Eike


Eike Stepper schrieb:
> Hi guys,
>
> I can confirm, making these 3rd party libraries with their own
> classloading run in OSGi is a nightmare! I took me hours teach where
> the hsqldb driver is. On the way I came over different kinds of
> classloading related errors.
>
> First I removed all the jars from the test plugin (the one Cedric
> provided). They're already in the cdo.server.hibernate.libraries
> plugin which, in fact, needs the buddy policy set to dependent. I
> think this last change needs to be committed to CVS! Btw. I'd prefer
> to use *registered* buddies instead of making all dependent bundles
> visible.
>
> The test plugin then needs to provide the database driver. Therefore I
> added a dependency on the org.eclipse.net4j.db.hsqldb plugin.
>
> With this setup I can start the server without CNFE, NCDFE, CCI and
> the like. At this point I get the following exception, which seems the
> same as the one Cedric observed in the end:
>
> net.sf.cglib.core.CodeGenerationException:
> java.lang.reflect.InvocationTargetException-->null
> at
> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:237)
>
> at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
> at net.sf.cglib.proxy.Enhancer.createClass(Enhancer.java:317)
> at
> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:127)
>
> at
> org.hibernate.proxy.pojo.cglib.CGLIBProxyFactory.postInstant iate(CGLIBProxyFactory.java:43)
>
> at
> org.hibernate.tuple.entity.PojoEntityTuplizer.buildProxyFact ory(PojoEntityTuplizer.java:162)
>
> at
> org.hibernate.tuple.entity.AbstractEntityTuplizer.<init>(AbstractEntityTuplizer.java:135)
>
> at
> org.hibernate.tuple.entity.PojoEntityTuplizer.<init>(PojoEntityTuplizer.java:55)
>
> at
> org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping . <init>(EntityEntityModeToTuplizerMapping.java:56)
>
> at
> org.hibernate.tuple.entity.EntityMetamodel.<init>(EntityMetamodel.java:295)
>
> at
> org.hibernate.persister.entity.AbstractEntityPersister.<init >(AbstractEntityPersister.java:434)
>
> at
> org.hibernate.persister.entity.JoinedSubclassEntityPersister . <init>(JoinedSubclassEntityPersister.java:91)
>
> at
> org.hibernate.persister.PersisterFactory.createClassPersiste r(PersisterFactory.java:58)
>
> at
> org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:226)
> at
> org.hibernate.cfg.Configuration.buildSessionFactory(Configur ation.java:1294)
>
> at
> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getSessionFactory(HibernatePackageHandler.java:335 )
>
> at
> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.readPackageInfos(HibernatePackageHandler.java:293)
>
> at
> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getCDOPackageInfos(HibernatePackageHandler.java:21 5)
>
> at
> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Reader.readPackageInfos(HibernateStoreReader.java:87)
>
> at
> org.eclipse.emf.cdo.internal.server.PackageManager.doActivat e(PackageManager.java:72)
>
> at
> org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
>
> at
> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>
> at
> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>
> at
> org.eclipse.emf.cdo.internal.server.Repository.activateRepos itory(Repository.java:264)
>
> at
> org.eclipse.emf.cdo.internal.server.Repository.doActivate(Re pository.java:251)
>
> at
> org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
>
> at
> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>
> at
> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>
> at
> org.eclipse.emf.cdo.server.CDOServerUtil.addRepository(CDOSe rverUtil.java:67)
>
> at
> org.eclipse.cdo.bug.LaunchServerTest.setUp(LaunchServerTest. java:54)
> at junit.framework.TestCase.runBare(TestCase.java:128)
> at junit.framework.TestResult$1.protect(TestResult.java:106)
> at junit.framework.TestResult.runProtected(TestResult.java:124)
> at junit.framework.TestResult.run(TestResult.java:109)
> at junit.framework.TestCase.run(TestCase.java:120)
> at junit.framework.TestSuite.runTest(TestSuite.java:230)
> at junit.framework.TestSuite.run(TestSuite.java:225)
> at
> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
>
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
>
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
>
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
>
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
>
> at
> org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
>
> at
> org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:585)
> at
> org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hodWithException(EclipseAppContainer.java:574)
>
> at
> org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
>
> at
> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:110)
>
> at
> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:79)
>
> at
> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:379)
>
> at
> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:179)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 549)
> at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
> at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
> at org.eclipse.equinox.launcher.Main.main(Main.java:1212)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:585)
> at net.sf.cglib.core.ReflectUtils.defineClass(ReflectUtils.java :384)
> at
> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>
> ... 61 more
> Caused by: java.lang.NoClassDefFoundError:
> org/hibernate/proxy/HibernateProxy
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
> ... 67 more
>
> I must admit that I'm still not sure about te fundamental interwork
> between Hibernate and the user defined meta model. It seems as if asm
> and cglib are somewhere in between but I have no idea why at all. I
> thought we have implemented a bunch of static Hibernate interfaces to
> make the CDORevision instances known to Hibernate. Please somebody
> explain why we need runtime byte code engineering? Is there some
> configuration missing to tell Hibernate to use our special tuplizers?
>
> I have changed several things in the code of the test plugin and since
> it's quite small now that the jars are removed, I attach my version of
> the zip here (including a JUnit Plugin launch config). And don't
> forget to update the cdo.server.hibernate.libraries plugin from CVS to
> get the new buddy policy!
>
> Cheers
> /Eike
>
> Martin Taal schrieb:
>> Hi Cédric,
>> I looked at it for many hours trying different things (using your
>> testcase) but I am kind of stuck with this (getting the same error as
>> you). The difference I can see with the way I run the cdo.hibernate
>> testcases (as a junit test) and how your testcase is run (junit
>> plugin test). This for sure means difference in class loader behavior.
>>
>> Eike, do you have an idea? I am not sure how classloading works when
>> you start a cdo server in a plugin. Which plugins are then available
>> in the cdo server? The testcases can be downloaded from the link below.
>>
>> gr. Martin
>>
>> Cédric Brun wrote:
>>> Thanks a lot ! I'm looking forward to your advices !
>>>
>>> Cédric
>>>
>>> Martin Taal wrote:
>>>
>>>> I am downloading your zip-file. I don't have time to look at it
>>>> right away
>>>> but I will do my best later this evening.
>>>>
>>>> gr. Martin
>>>>
>>>> Cédric Brun wrote:
>>>>> I tried to do the same things as in the CDO/Hibernate unit tests.
>>>>>
>>>>> Here's a zipped project trying to launch an HibernateServer with a
>>>>> unit
>>>>> test adding elements.
>>>>> ( http://cedric.brun.free.fr/Tortoose/org.eclipse.cdo-teneo-te st.zip)
>>>>> The
>>>>> libs I found are included in the same project, but I still have no
>>>>> luck
>>>>> and failling at :
>>>>>
>>>>> at
>>>>>
>>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>>
>>>>> ... 62 more
>>>>> Caused by: java.lang.NoClassDefFoundError:
>>>>> org/hibernate/proxy/HibernateProxy
>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>>>>> ... 68 more
>>>>>
>>>>> please note that the test project I provide really is a "quick and
>>>>> ugly"
>>>>> one ^^
>>>>>
>>>>>
>>>>> Cédric
>>>>>
>>>>>
>>>>>
>>>>> Martin Taal wrote:
>>>>>
>>>>>> Hmm, I can understand your headache. Classpath issues are the most
>>>>>> annoying errors...
>>>>>>
>>>>>> In your junit test source file, what happens when you enter the
>>>>>> org.objectweb.asm.Type, can you source file see this class?
>>>>>> Do you have cglib in some other plugin which does not have asm?
>>>>>>
>>>>>> What I also find slightly strange is that cglib is used in this case
>>>>>> because normally cdo objects are not proxied using cglib. Can you
>>>>>> post
>>>>>> the generated mapping? How does your cdo initialization code look
>>>>>> like?
>>>>>>
>>>>>> gr. Martin
>>>>>>
>>>>>> Cédric Brun wrote:
>>>>>>> Hi Martin,
>>>>>>>
>>>>>>> and thanks for your quick answer ! what I did is the following :
>>>>>>>
>>>>>>> I used the org.eclipse.emf.cdo.server.hibernate.libraries bundle
>>>>>>> and
>>>>>>> added that :
>>>>>>>
>>>>>>> Bundle-ClassPath: lib/hibernate3.jar,
>>>>>>> lib/cglib-2.1.3.jar,
>>>>>>> lib/commons-collections-2.1.1.jar,
>>>>>>> lib/dom4j.jar,
>>>>>>> lib/jta.jar
>>>>>>> lib/asm-3.0.jar
>>>>>>>
>>>>>>> and this directive :
>>>>>>>
>>>>>>> Eclipse-BuddyPolicy: dependent
>>>>>>>
>>>>>>> as stated in
>>>>>>> http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>>>>>>>
>>>>>>>
>>>>>>> Then I don't get the same error, now when I start my own plugin
>>>>>>> (JUnit
>>>>>>> plugin test) depending on
>>>>>>> org.eclipse.emf.cdo.server.hibernate.libraries I get :
>>>>>>>
>>>>>>> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
>>>>>>> at
>>>>>>> net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
>>>>>>> at
>>>>>>> net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
>>>>>>> at net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
>>>>>>> at
>>>>>>>
>>> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>>>
>>>>>>> These jars are such a headache ! Do you know - by chance - what's
>>>>>>> going on ?
>>>>>>>
>>>>>>>
>>>>>>> Cédric
>>>>>>>
>>>>>>>
>>>>>>> Martin Taal wrote:
>>>>>>>
>>>>>>>> Hi Cédric,
>>>>>>>> Most of the time this exception is thrown because hibernate
>>>>>>>> can't find
>>>>>>>> the class. A common cause is that the hibernate jar files are
>>>>>>>> located
>>>>>>>> in a separate plugin which can not reach the classes in other
>>>>>>>> plugins.
>>>>>>>> If you set the eclipse buddy policy of the hibernate plugin to
>>>>>>>> dependent then this can help.
>>>>>>>>
>>>>>>>> gr. Martin
>>>>>>>>
>>>>>>>> Cédric Brun wrote:
>>>>>>>>> Hi Simon and Eike,
>>>>>>>>>
>>>>>>>>> thanks for your feedback, now I know what I could expect in using
>>>>>>>>> optimized implementation, I'll have a try on the hibernate
>>>>>>>>> store and
>>>>>>>>> see if it is scaling in a better way.
>>>>>>>>>
>>>>>>>>> You gave me many pointers to explain the performances issues
>>>>>>>>> but in
>>>>>>>>> fact it seems that the SQL execution is taking so much time (not
>>>>>>>>> suprisingly in fact ;) . My first profile was wrong.
>>>>>>>>>
>>>>>>>>> So I'm installing the hibernate store retrieving all the
>>>>>>>>> source code
>>>>>>>>> from CVS, and I'm trying to launch the unit tests. I added
>>>>>>>>> many jars
>>>>>>>>> and everything gets compiled but most of the tests are
>>>>>>>>> failling with
>>>>>>>>> "
>>>>>>>>>
>>>>>>>>> "
>>>>>>>>> org.hibernate.MappingException: Could not determine type for:
>>>>>>>>>
>>> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>>>>>>>> at table: cdo_features, for columns:
>>>>>>>>> [org.hibernate.mapping.Column(type)] "
>>>>>>>>>
>>>>>>>>> any hint ?
>>>>>>>>>
>>>>>>>>> Cédric
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Simon McDuff wrote:
>>>>>>>>>
>>>>>>>>>> We noticed a performance degradation in the following case:
>>>>>>>>>>
>>>>>>>>>> When we fetch a collection of objects that contains
>>>>>>>>>> collections that
>>>>>>>>>> we don't want to use. Ex:
>>>>>>>>>>
>>>>>>>>>> Model:
>>>>>>>>>>
>>>>>>>>>> ClassB
>>>>>>>>>> {
>>>>>>>>>> int foo1;
>>>>>>>>>> List childrens1;
>>>>>>>>>> List childrens2;
>>>>>>>>>> List childrens3;
>>>>>>>>>> };
>>>>>>>>>>
>>>>>>>>>> Code:
>>>>>>>>>>
>>>>>>>>>> Resource.contents contains objects of type ClassB.
>>>>>>>>>>
>>>>>>>>>> By fetching instances of classB, we will also fetched some
>>>>>>>>>> items in
>>>>>>>>>> the list(childrens1, childrens2, childrens3) The problem we
>>>>>>>>>> had.. is
>>>>>>>>>> we do not know if we want to use it. SO better not to
>>>>>>>>>> download them.
>>>>>>>>>>
>>>>>>>>>> So In our production(we used our own IStore), we never
>>>>>>>>>> download any
>>>>>>>>>> objects from collection. I'm working at the moment to put in
>>>>>>>>>> place
>>>>>>>>>> all feature we used in the current CDO. That one will be part of
>>>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477 To begin
>>>>>>>>>> with,CDORevisionImpl needs to supports list that are 0 even if
>>>>>>>>>> referenceChunkSize == 0.
>>>>>>>>>>
>>>>>>>>>> It is true that we obtain good performance.
>>>>>>>>>> Worst case :
>>>>>>>>>> 1500 objects /sec (Real fetch from disk. Page that contains the
>>>>>>>>>> objects, wasn't in the cache of the filesystem.)
>>>>>>>>>>
>>>>>>>>>> Best case
>>>>>>>>>> 12 000 objects/Sec (Fetch objects that are already in the
>>>>>>>>>> case of
>>>>>>>>>> the filesystem)
>>>>>>>>>>
>>>>>>>>>> (These statistics always fetch objects from server to client.
>>>>>>>>>> The
>>>>>>>>>> cache client isn't used)
>>>>>>>>>>
>>>>>>>>>> We have another optimization one with SmartReadAhead but for now
>>>>>>>>>> only work
>>>>>>>>>> for UI client. (for now) This is something we will integrate as
>>>>>>>>>> well in CDO. But for now we concentrate on the other things.
>>>>>>>>>>
>>>>>>>>>> I used an Object database. Based on my preliminary test
>>>>>>>>>> without CDO.
>>>>>>>>>> I compare
>>>>>>>>>>
>>>>>>>>>> Hibernate vs Objectivity/EMF :
>>>>>>>>>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>>>>>>>>>
>>>>>>>>>> The worst.. you will get 30 to 240 objects/sec or
>>>>>>>>>> the best... you will get 500 to 4000 objects/sec
>>>>>>>>>>
>>>>>>>>>> Based on your comments... you are getting the worst (with
>>>>>>>>>> Derby)!!
>>>>>>>>>> :-)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Another comments about insertion.
>>>>>>>>>>
>>>>>>>>>> We found another problem when we insert data(DOn't know if it is
>>>>>>>>>> still true). Resource.Contents is a unique list. Basically
>>>>>>>>>> each time
>>>>>>>>>> we insert one element, it needs to fetch all the collection
>>>>>>>>>> to see
>>>>>>>>>> if we already have this element. Basically.. it is time
>>>>>>>>>> consuming.
>>>>>>>>>> In our case we are not going through this stage to persist our
>>>>>>>>>> objects... but the objects are not accessible through that list.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> If speed is an issue for you!! We (You included :-)) can
>>>>>>>>>> optimize
>>>>>>>>>> it. What I like is Eike did a really good job doing that
>>>>>>>>>> framwork
>>>>>>>>>> and optimizations are easy to add. Also you can change your
>>>>>>>>>> back-end
>>>>>>>>>> without affecting you client side. (Maybe going to an Object
>>>>>>>>>> database!!) THis was one of the purpose of CDO!! :-)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Simon
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le
>>>>>>>>>> message de
>>>>>>>>>> news: 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>>>>>>>
>>>>>>>>>> That sounds interesting!
>>>>>>>>>> Comments inline...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Cédric Brun schrieb:
>>>>>>>>>> Hi Eike,
>>>>>>>>>>
>>>>>>>>>> I've been messing around CDO lately and I hacked a few ugly
>>>>>>>>>> unit-tests measuring read/write throughput when I use CDO (RC2).
>>>>>>>>>>
>>>>>>>>>> I'm using an embeded-derby database, two different metamodels
>>>>>>>>>> (complex one - 250 EClassifiers/300 features and a simple one
>>>>>>>>>> - 1
>>>>>>>>>> Classifier/3 Features)
>>>>>>>>>>
>>>>>>>>>> The test is a "quick and dirty" one so please do not take these
>>>>>>>>>> figures too seriously, I would just like to have your input
>>>>>>>>>> about
>>>>>>>>>> the "order of magnitude" I should expect or settings affecting
>>>>>>>>>> seriously these performances. I tried both JVM and TCP
>>>>>>>>>> acceptor but
>>>>>>>>>> the performances are quite similar though JVM is a bit faster.
>>>>>>>>>> Looks like a sign that the NIO socket channels are
>>>>>>>>>> efficient ,-)
>>>>>>>>>> You used localhost for TCP transport I guess?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> * First case : building a small model (10000 elements) with the
>>>>>>>>>> simple MM, only adding instances in an empty database :
>>>>>>>>>> commiting change on every add I get a 2 objects per second rate.
>>>>>>>>>> commiting changes every 10 adds - I get a 30 objects per second
>>>>>>>>>> rate.
>>>>>>>>>> Seems to be consistent with the fact that only for commits
>>>>>>>>>> there
>>>>>>>>>> is remote traffic. And the backend transaction management is
>>>>>>>>>> another overhead. So the impact of the number objects/commit
>>>>>>>>>> starts with bigger transactions.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> browsing the just commited elements opening a new transaction
>>>>>>>>>> and
>>>>>>>>>> getting a new Resource instance, I get around 30 000
>>>>>>>>>> objets/sec rate
>>>>>>>>>> just iterating thanks to Resource.getAllContents().
>>>>>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>>>>>> All views and transactions of a single session share (and
>>>>>>>>>> cache)
>>>>>>>>>> their object state. Otherwise I'd expect much smaller values.
>>>>>>>>>> With his ObjectivityStore implementation Simon (cc'ed) reached
>>>>>>>>>> rates of several thousands objects/sec. But the DBStore
>>>>>>>>>> (which you
>>>>>>>>>> used) is not that optimized.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> * Second case : I built a big model in the database using the
>>>>>>>>>> complex metamodel. It contains 700 000 elements. The creation
>>>>>>>>>> went
>>>>>>>>>> pretty nicely (I did not measured it exactly as it was launched
>>>>>>>>>> during a night, but it seems I got around 100 objects/sec
>>>>>>>>>> rate in
>>>>>>>>>> writting as I was writting every 700 objects.
>>>>>>>>>> Sounds not too bad.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Once the model is built the derby database is 222 MB (which - I
>>>>>>>>>> heard - is not so big for derby).
>>>>>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>>>>>> To be honest, I'm really no database expert (although I can
>>>>>>>>>> write
>>>>>>>>>> simple SQL statements).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Retrieving the distant resource and calling eAllContents() on
>>>>>>>>>> it, I
>>>>>>>>>> get a 50 Objects/sec rate at best, which mean my
>>>>>>>>>> eAllContents() call
>>>>>>>>>> would take 4 hours !
>>>>>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>>>>>> (iteration order and so). There are several CDO features
>>>>>>>>>> that can
>>>>>>>>>> have a major impact on load rates!
>>>>>>>>>>
>>>>>>>>>> - Collections (xrefs and containment) can be loaded lazily. I
>>>>>>>>>> don't mean the target objects, which are lazy by default
>>>>>>>>>> anyway,
>>>>>>>>>> but even the target IDs that are in the collections. We
>>>>>>>>>> call that
>>>>>>>>>> "reference chunking" (but search for a better name is
>>>>>>>>>> ongoing).
>>>>>>>>>>
>>>>>>>>>> public interface CDOSession extends CDOProtocolSession,
>>>>>>>>>> IContainer<CDOView>
>>>>>>>>>> {
>>>>>>>>>> public int getReferenceChunkSize();
>>>>>>>>>>
>>>>>>>>>> public void setReferenceChunkSize(int
>>>>>>>>>> referenceChunkSize);
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> - Loading collection elements (target objects) can be
>>>>>>>>>> batched (the
>>>>>>>>>> current name is even worse)
>>>>>>>>>>
>>>>>>>>>> public interface CDOView extends CDOProtocolView,
>>>>>>>>>> INotifier
>>>>>>>>>> {
>>>>>>>>>> public int getLoadRevisionCollectionChunkSize();
>>>>>>>>>>
>>>>>>>>>> public void setLoadRevisionCollectionChunkSize(int
>>>>>>>>>> loadRevisionCollectionChunkSize);
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> - One of the most sophisticated feature is automatic model
>>>>>>>>>> usage
>>>>>>>>>> analyzation. The result are fetch rules that are sent to the
>>>>>>>>>> server. When later objects are loaded from the server, it can
>>>>>>>>>> answer with additional objects that are likely to be needed
>>>>>>>>>> by the
>>>>>>>>>> client in the near future.
>>>>>>>>>>
>>>>>>>>>> public class CDOViewImpl extends
>>>>>>>>>> org.eclipse.net4j.internal.util.event.Notifier
>>>>>>>>>> implements
>>>>>>>>>> CDOView, CDOIDProvider, Adapter.Internal
>>>>>>>>>> {
>>>>>>>>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>>>>>>> {
>>>>>>>>>> return featureAnalyzer;
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>>>>>>> featureAnalyzer)
>>>>>>>>>> {
>>>>>>>>>> this.featureAnalyzer = featureAnalyzer == null ?
>>>>>>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> - There's a SmartReadAheadThread (if you're curious, please
>>>>>>>>>> ask
>>>>>>>>>> Simon). - Caches can be configured
>>>>>>>>>> - ...
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Known issues of the DBStore:
>>>>>>>>>> - When in auditing mode updates are always written as complete
>>>>>>>>>> inserts (by design) - When configuring toManyReferences to
>>>>>>>>>> a value
>>>>>>>>>> different from LIKE_ATTRIBUTES (which is not implemented
>>>>>>>>>> yet) it
>>>>>>>>>> is very slow. - No (few) prepared statements are used (see
>>>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe
>>>>>>>>>> others...
>>>>>>>>>>
>>>>>>>>>> Please note that the DBStore was not written by me to
>>>>>>>>>> compete with
>>>>>>>>>> Hibernate in terms of performance or mapping capabilities. Now
>>>>>>>>>> that Martin and I, well in the end mostly Martin, provide a
>>>>>>>>>> HibernateStore and this HibernateStore gets more and more
>>>>>>>>>> features
>>>>>>>>>> I expect that there will be a trend away from the DBStore.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This is probably linked to the way the mapping to db is done,
>>>>>>>>>> here's
>>>>>>>>>> the config I used (came from CDO-server example if I recall
>>>>>>>>>> correctly).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <repository name="repo1">
>>>>>>>>>> <property name="overrideUUID"
>>>>>>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>>>>>>> <property name="supportingAudits" value="true"/>
>>>>>>>>>> <property name="verifyingRevisions"
>>>>>>>>>> value="false"/>
>>>>>>>>>> <property name="currentLRUCapacity"
>>>>>>>>>> value="10000"/>
>>>>>>>>>> I guess this one could be larger.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> <property name="revisedLRUCapacity"
>>>>>>>>>> value="100"/>
>>>>>>>>>> <store type="db">
>>>>>>>>>> <!-- type: horizontal | vertical | <any
>>>>>>>>>> user-contributed type> -->
>>>>>>>>>> <mappingStrategy
>>>>>>>>>> type="horizontal">
>>>>>>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>>>>>>> ONE_TABLE_PER_CLASS |
>>>>>>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY |
>>>>>>>>>> LIKE_ATTRIBUTES
>>>>>>>>>> -->
>>>>>>>>>> <property
>>>>>>>>>> name="toManyReferences"
>>>>>>>>>>
>>>>>>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>>>>>>
>>>>>>>>>> <!-- LIKE_ATTRIBUTES |
>>>>>>>>>> LIKE_TO_MANY_REFERENCES -->
>>>>>>>>>> <property name="toOneReferences"
>>>>>>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>>>>>>
>>>>>>>>>> <!-- MODEL | STRATEGY -->
>>>>>>>>>> <property
>>>>>>>>>> name="mappingPrecedence"
>>>>>>>>>> value="MODEL"/>
>>>>>>>>>> This one is obsolete.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> </mappingStrategy>
>>>>>>>>>> <dbAdapter
>>>>>>>>>> name="derby-embedded"/>
>>>>>>>>>> <dataSource
>>>>>>>>>>
>>>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>>>>>>>> databaseName="/tmp/cdodb1"
>>>>>>>>>> createDatabase="create"/>
>>>>>>>>>>
>>>>>>>>>> </store>
>>>>>>>>>> </repository>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> What do you think about that ? What throughput do you think I
>>>>>>>>>> should
>>>>>>>>>> expect ? As I said that's very difficult to judge.
>>>>>>>>>> I believe it's more appropriate to compare the actual
>>>>>>>>>> values with
>>>>>>>>>> values from other measures (other setups, properties, etc.
>>>>>>>>>> or even
>>>>>>>>>> other products) instead of random expectations. Apologies
>>>>>>>>>> that I
>>>>>>>>>> didn't have the time so far to provide some reference measures
>>>>>>>>>> myself. Maybe this gets better now that the team starts to
>>>>>>>>>> grow
>>>>>>>>>> slowly ;-)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> What are the main reasons fur such figures ?
>>>>>>>>>> Hmm, I *suspect* you don't like the figures? I can find no
>>>>>>>>>> evidence your assessment ;-)
>>>>>>>>>>
>>>>>>>>>> Ahh,one more note:
>>>>>>>>>> Tracing can also have a significant impact on performance and
>>>>>>>>>> end-to-end throughput. Turn it off by all means. And please
>>>>>>>>>> make
>>>>>>>>>> sure that it's really turned off, not only invisible (i.e.
>>>>>>>>>> console
>>>>>>>>>> is not a target).
>>>>>>>>>>
>>>>>>>>>> Cheers
>>>>>>>>>> /Eike
>>>>
>>>
>>
>>

--------------000609090607080808010708
Content-Type: multipart/related;
boundary="------------040900020203060703010001"


--------------040900020203060703010001
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi,<br>
<br>
Here is a diagram of the dependencies of the relevant plugins (please
correct me if something is wrong).<br>
The black arrows are ordinary downstream dependencies (Require-Bundle).<br>
The red arrows are upstream dependencies (Eclipse-BuddyPolicy:
dependent).<br>
<br>
<img src="cid:part1.07020404.09020102@sympedia.de" alt=""><br>
<br>
For research purposes I added code to the libraries plugin that tries
to reflectively load CDOFeatureImpl and that is not a problem!<br>
It seems to me as if our problem is not class loading related because
the hibernate.jar should be able to see the needed classes.<br>
I believe this is either a bug in Hibernate or our wrong usage of
Hibernate. I tend to believe the latter ;-)<br>
I put a breakpoint in
org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping .EntityEntityModeToTuplizerMapping(PersistentClass,
EntityMetamodel) <br>
to see how the tuplizers are created. I'm too unexperienced with
Hibernate to get a clue about what happens there...<br>
<br>
Btw. I'd prefer to see the application plugin depend on net4j.db
instead of net4j.db.hsqldb.<br>
But it seems as if the BuddyPolicy in net4j.db is not "reexported" so
that the application can see the hsqldb.jar then ;-(<br>
I posted a respective question to equinox...<br>
<br>
Cheers<br>
/Eike<br>
<br>
<br>
Eike Stepper schrieb:
<blockquote cite="mid:g1t99s$tla$1@build.eclipse.org" type="cite">Hi
guys,
<br>
<br>
I can confirm, making these 3rd party libraries with their own
classloading run in OSGi is a nightmare! I took me hours teach where
the hsqldb driver is. On the way I came over different kinds of
classloading related errors.
<br>
<br>
First I removed all the jars from the test plugin (the one Cedric
provided). They're already in the cdo.server.hibernate.libraries plugin
which, in fact, needs the buddy policy set to dependent. I think this
last change needs to be committed to CVS! Btw. I'd prefer to use
*registered* buddies instead of making all dependent bundles visible.
<br>
<br>
The test plugin then needs to provide the database driver. Therefore I
added a dependency on the org.eclipse.net4j.db.hsqldb plugin.
<br>
<br>
With this setup I can start the server without CNFE, NCDFE, CCI and the
like. At this point I get the following exception, which seems the same
as the one Cedric observed in the end:
<br>
<br>
net.sf.cglib.core.CodeGenerationException:
java.lang.reflect.InvocationTargetException--&gt;null
<br>
   at
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:237)
<br>
   at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
<br>
   at net.sf.cglib.proxy.Enhancer.createClass(Enhancer.java:317)
<br>
   at
org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:127)
<br>
   at
org.hibernate.proxy.pojo.cglib.CGLIBProxyFactory.postInstant iate(CGLIBProxyFactory.java:43)
<br>
   at
org.hibernate.tuple.entity.PojoEntityTuplizer.buildProxyFact ory(PojoEntityTuplizer.java:162)
<br>
   at
org.hibernate.tuple.entity.AbstractEntityTuplizer.&lt;in it&gt;(AbstractEntityTuplizer.java:135)
<br>
   at
org.hibernate.tuple.entity.PojoEntityTuplizer.&lt;init&a mp;gt;(PojoEntityTuplizer.java:55)
<br>
   at
org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping .&lt;init&gt;(EntityEntityModeToTuplizerMapping.java :56)
<br>
   at
org.hibernate.tuple.entity.EntityMetamodel.&lt;init& gt;(EntityMetamodel.java:295)
<br>
   at
org.hibernate.persister.entity.AbstractEntityPersister.& lt;init&gt;(AbstractEntityPersister.java:434)
<br>
   at
org.hibernate.persister.entity.JoinedSubclassEntityPersister .&lt;init&gt;(JoinedSubclassEntityPersister.java:91)
<br>
   at
org.hibernate.persister.PersisterFactory.createClassPersiste r(PersisterFactory.java:58)
<br>
   at
org.hibernate.impl.SessionFactoryImpl.&lt;init&gt;(S essionFactoryImpl.java:226)
<br>
   at
org.hibernate.cfg.Configuration.buildSessionFactory(Configur ation.java:1294)
<br>
   at
org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getSessionFactory(HibernatePackageHandler.java:335 )
<br>
   at
org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.readPackageInfos(HibernatePackageHandler.java:293)
<br>
   at
org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getCDOPackageInfos(HibernatePackageHandler.java:21 5)
<br>
   at
org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Reader.readPackageInfos(HibernateStoreReader.java:87)
<br>
   at
org.eclipse.emf.cdo.internal.server.PackageManager.doActivat e(PackageManager.java:72)
<br>
   at
org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
<br>
   at
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
<br>
   at
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
<br>
   at
org.eclipse.emf.cdo.internal.server.Repository.activateRepos itory(Repository.java:264)
<br>
   at
org.eclipse.emf.cdo.internal.server.Repository.doActivate(Re pository.java:251)
<br>
   at
org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
<br>
   at
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
<br>
   at
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
<br>
   at
org.eclipse.emf.cdo.server.CDOServerUtil.addRepository(CDOSe rverUtil.java:67)
<br>
   at
org.eclipse.cdo.bug.LaunchServerTest.setUp(LaunchServerTest. java:54)
<br>
   at junit.framework.TestCase.runBare(TestCase.java:128)
<br>
   at junit.framework.TestResult$1.protect(TestResult.java:106)
<br>
   at junit.framework.TestResult.runProtected(TestResult.java:124)
<br>
   at junit.framework.TestResult.run(TestResult.java:109)
<br>
   at junit.framework.TestCase.run(TestCase.java:120)
<br>
   at junit.framework.TestSuite.runTest(TestSuite.java:230)
<br>
   at junit.framework.TestSuite.run(TestSuite.java:225)
<br>
   at
org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
<br>
   at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
<br>
   at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
<br>
   at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
<br>
   at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
<br>
   at
org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
<br>
   at
org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
<br>
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
<br>
   at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
<br>
   at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
<br>
   at java.lang.reflect.Method.invoke(Method.java:585)
<br>
   at
org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hodWithException(EclipseAppContainer.java:574)
<br>
   at
org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
<br>
   at
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:110)
<br>
   at
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:79)
<br>
   at
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:379)
<br>
   at
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:179)
<br>
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
<br>
   at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
<br>
   at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
<br>
   at java.lang.reflect.Method.invoke(Method.java:585)
<br>
   at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 549)
<br>
   at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
<br>
   at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
<br>
   at org.eclipse.equinox.launcher.Main.main(Main.java:1212)
<br>
Caused by: java.lang.reflect.InvocationTargetException
<br>
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
<br>
   at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
<br>
   at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
<br>
   at java.lang.reflect.Method.invoke(Method.java:585)
<br>
   at net.sf.cglib.core.ReflectUtils.defineClass(ReflectUtils.java :384)
<br>
   at
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
<br>
   ... 61 more
<br>
Caused by: java.lang.NoClassDefFoundError:
org/hibernate/proxy/HibernateProxy
<br>
   at java.lang.ClassLoader.defineClass1(Native Method)
<br>
   at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
<br>
   ... 67 more
<br>
<br>
I must admit that I'm still not sure about te fundamental interwork
between Hibernate and the user defined meta model. It seems as if asm
and cglib are somewhere in between but I have no idea why at all. I
thought we have implemented a bunch of static Hibernate interfaces to
make the CDORevision instances known to Hibernate. Please somebody
explain why we need runtime byte code engineering? Is there some
configuration missing to tell Hibernate to use our special tuplizers?
<br>
<br>
I have changed several things in the code of the test plugin and since
it's quite small now that the jars are removed, I attach my version of
the zip here (including a JUnit Plugin launch config). And don't forget
to update the cdo.server.hibernate.libraries plugin from CVS to get the
new buddy policy!
<br>
<br>
Cheers
<br>
/Eike
<br>
<br>
Martin Taal schrieb:
<br>
<blockquote type="cite">Hi Cédric,
<br>
I looked at it for many hours trying different things (using your
testcase) but I am kind of stuck with this (getting the same error as
you). The difference I can see with the way I run the cdo.hibernate
testcases (as a junit test) and how your testcase is run (junit plugin
test). This for sure means difference in class loader behavior.
<br>
<br>
Eike, do you have an idea? I am not sure how classloading works when
you start a cdo ser


Re: [CDO] reading/writting throughput performances [message #619129 is a reply to message #124519] Sun, 01 June 2008 10:06 Go to previous message
Martin Taal is currently offline Martin TaalFriend
Messages: 5468
Registered: July 2009
Senior Member
Hi Eike, Cedric,
I can't say to much about net4j.db versus net4j.db.hsqldb.

You are correct that the hibernate.libraries plugin should have Eclipse-BuddyPolicy: dependent (or
another way to use the classpath of the dependent plugins). Btw, I agree that registered is better
as afaik dependent gets slow which larger plugin environments.

The reason that cglib is used is because one of the class mappings in
org.eclipse.emf.cdo.server.hibernate/mappings/meta.hbm/xml did not have the lazy="false" directive.
I added this and now I get another exception (see below). So this 'solves' the problem. If you do a
get latest of this file then I think this problem is 'overcome'.

Overall I still do not see why it went wrong. As far as I can see the cglib library could not find
the hibernate class, which is strange as they are in the same plugin??
The cdo testcases don't have this problem and neither does Teneo.

As it does not occur anymore we can ignore it for now (not very satisfactory but there are other
things to do)... Unless someone else has an idea how to handle this ofcourse.

gr. Martin

java.lang.IllegalStateException: acceptor == null
at org.eclipse.net4j.internal.jvm.JVMClientConnector.doBeforeAc tivate(JVMClientConnector.java:57)
at org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:56)
at org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
at org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
at org.eclipse.net4j.internal.util.container.ManagedContainer.g etElement(ManagedContainer.java:260)
at org.eclipse.emf.internal.cdo.util.ChannelInjector.getConnect or(ChannelInjector.java:74)
at org.eclipse.emf.internal.cdo.util.ChannelInjector.process(Ch annelInjector.java:41)
at
org.eclipse.net4j.internal.util.container.ManagedContainer.p ostProcessElement(ManagedContainer.java:407)
at org.eclipse.net4j.internal.util.container.ManagedContainer.g etElement(ManagedContainer.java:259)
at org.eclipse.cdo.bug.hibernate.HibernateServer.createSession( HibernateServer.java:93)
at org.eclipse.cdo.bug.unit.LaunchServerTest.testStringValue(La unchServerTest.java:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at junit.framework.TestCase.runTest(TestCase.java:164)
at junit.framework.TestCase.runBare(TestCase.java:130)
at junit.framework.TestResult$1.protect(TestResult.java:106)
at junit.framework.TestResult.runProtected(TestResult.java:124)
at junit.framework.TestResult.run(TestResult.java:109)
at junit.framework.TestCase.run(TestCase.java:120)
at junit.framework.TestSuite.runTest(TestSuite.java:230)
at junit.framework.TestSuite.run(TestSuite.java:225)
at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
at org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
at org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hod(EclipseAppContainer.java:547)
at org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
at
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:106)
at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:76)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:362)
at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:175)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 564)
at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
at org.eclipse.equinox.launcher.Main.run(Main.java:1251)
at org.eclipse.equinox.launcher.Main.main(Main.java:1227)



Eike Stepper wrote:
> Hi,
>
> Here is a diagram of the dependencies of the relevant plugins (please
> correct me if something is wrong).
> The black arrows are ordinary downstream dependencies (Require-Bundle).
> The red arrows are upstream dependencies (Eclipse-BuddyPolicy: dependent).
>
>
>
> For research purposes I added code to the libraries plugin that tries to
> reflectively load CDOFeatureImpl and that is not a problem!
> It seems to me as if our problem is not class loading related because
> the hibernate.jar should be able to see the needed classes.
> I believe this is either a bug in Hibernate or our wrong usage of
> Hibernate. I tend to believe the latter ;-)
> I put a breakpoint in
> org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping .EntityEntityModeToTuplizerMapping(PersistentClass,
> EntityMetamodel)
> to see how the tuplizers are created. I'm too unexperienced with
> Hibernate to get a clue about what happens there...
>
> Btw. I'd prefer to see the application plugin depend on net4j.db instead
> of net4j.db.hsqldb.
> But it seems as if the BuddyPolicy in net4j.db is not "reexported" so
> that the application can see the hsqldb.jar then ;-(
> I posted a respective question to equinox...
>
> Cheers
> /Eike
>
>
> Eike Stepper schrieb:
>> Hi guys,
>>
>> I can confirm, making these 3rd party libraries with their own
>> classloading run in OSGi is a nightmare! I took me hours teach where
>> the hsqldb driver is. On the way I came over different kinds of
>> classloading related errors.
>>
>> First I removed all the jars from the test plugin (the one Cedric
>> provided). They're already in the cdo.server.hibernate.libraries
>> plugin which, in fact, needs the buddy policy set to dependent. I
>> think this last change needs to be committed to CVS! Btw. I'd prefer
>> to use *registered* buddies instead of making all dependent bundles
>> visible.
>>
>> The test plugin then needs to provide the database driver. Therefore I
>> added a dependency on the org.eclipse.net4j.db.hsqldb plugin.
>>
>> With this setup I can start the server without CNFE, NCDFE, CCI and
>> the like. At this point I get the following exception, which seems the
>> same as the one Cedric observed in the end:
>>
>> net.sf.cglib.core.CodeGenerationException:
>> java.lang.reflect.InvocationTargetException-->null
>> at
>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:237)
>>
>> at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
>> at net.sf.cglib.proxy.Enhancer.createClass(Enhancer.java:317)
>> at
>> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:127)
>>
>> at
>> org.hibernate.proxy.pojo.cglib.CGLIBProxyFactory.postInstant iate(CGLIBProxyFactory.java:43)
>>
>> at
>> org.hibernate.tuple.entity.PojoEntityTuplizer.buildProxyFact ory(PojoEntityTuplizer.java:162)
>>
>> at
>> org.hibernate.tuple.entity.AbstractEntityTuplizer.<init>(AbstractEntityTuplizer.java:135)
>>
>> at
>> org.hibernate.tuple.entity.PojoEntityTuplizer.<init>(PojoEntityTuplizer.java:55)
>>
>> at
>> org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping . <init>(EntityEntityModeToTuplizerMapping.java:56)
>>
>> at
>> org.hibernate.tuple.entity.EntityMetamodel.<init>(EntityMetamodel.java:295)
>>
>> at
>> org.hibernate.persister.entity.AbstractEntityPersister.<init >(AbstractEntityPersister.java:434)
>>
>> at
>> org.hibernate.persister.entity.JoinedSubclassEntityPersister . <init>(JoinedSubclassEntityPersister.java:91)
>>
>> at
>> org.hibernate.persister.PersisterFactory.createClassPersiste r(PersisterFactory.java:58)
>>
>> at
>> org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:226)
>> at
>> org.hibernate.cfg.Configuration.buildSessionFactory(Configur ation.java:1294)
>>
>> at
>> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getSessionFactory(HibernatePackageHandler.java:335 )
>>
>> at
>> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.readPackageInfos(HibernatePackageHandler.java:293)
>>
>> at
>> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getCDOPackageInfos(HibernatePackageHandler.java:21 5)
>>
>> at
>> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Reader.readPackageInfos(HibernateStoreReader.java:87)
>>
>> at
>> org.eclipse.emf.cdo.internal.server.PackageManager.doActivat e(PackageManager.java:72)
>>
>> at
>> org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
>>
>> at
>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>>
>> at
>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>>
>> at
>> org.eclipse.emf.cdo.internal.server.Repository.activateRepos itory(Repository.java:264)
>>
>> at
>> org.eclipse.emf.cdo.internal.server.Repository.doActivate(Re pository.java:251)
>>
>> at
>> org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
>>
>> at
>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>>
>> at
>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>>
>> at
>> org.eclipse.emf.cdo.server.CDOServerUtil.addRepository(CDOSe rverUtil.java:67)
>>
>> at
>> org.eclipse.cdo.bug.LaunchServerTest.setUp(LaunchServerTest. java:54)
>> at junit.framework.TestCase.runBare(TestCase.java:128)
>> at junit.framework.TestResult$1.protect(TestResult.java:106)
>> at junit.framework.TestResult.runProtected(TestResult.java:124)
>> at junit.framework.TestResult.run(TestResult.java:109)
>> at junit.framework.TestCase.run(TestCase.java:120)
>> at junit.framework.TestSuite.runTest(TestSuite.java:230)
>> at junit.framework.TestSuite.run(TestSuite.java:225)
>> at
>> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
>>
>> at
>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
>>
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
>>
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
>>
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
>>
>> at
>> org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
>>
>> at
>> org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>
>> at java.lang.reflect.Method.invoke(Method.java:585)
>> at
>> org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hodWithException(EclipseAppContainer.java:574)
>>
>> at
>> org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
>>
>> at
>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:110)
>>
>> at
>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:79)
>>
>> at
>> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:379)
>>
>> at
>> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:179)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>
>> at java.lang.reflect.Method.invoke(Method.java:585)
>> at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 549)
>> at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
>> at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
>> at org.eclipse.equinox.launcher.Main.main(Main.java:1212)
>> Caused by: java.lang.reflect.InvocationTargetException
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>
>> at java.lang.reflect.Method.invoke(Method.java:585)
>> at net.sf.cglib.core.ReflectUtils.defineClass(ReflectUtils.java :384)
>> at
>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>
>> ... 61 more
>> Caused by: java.lang.NoClassDefFoundError:
>> org/hibernate/proxy/HibernateProxy
>> at java.lang.ClassLoader.defineClass1(Native Method)
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>> ... 67 more
>>
>> I must admit that I'm still not sure about te fundamental interwork
>> between Hibernate and the user defined meta model. It seems as if asm
>> and cglib are somewhere in between but I have no idea why at all. I
>> thought we have implemented a bunch of static Hibernate interfaces to
>> make the CDORevision instances known to Hibernate. Please somebody
>> explain why we need runtime byte code engineering? Is there some
>> configuration missing to tell Hibernate to use our special tuplizers?
>>
>> I have changed several things in the code of the test plugin and since
>> it's quite small now that the jars are removed, I attach my version of
>> the zip here (including a JUnit Plugin launch config). And don't
>> forget to update the cdo.server.hibernate.libraries plugin from CVS to
>> get the new buddy policy!
>>
>> Cheers
>> /Eike
>>
>> Martin Taal schrieb:
>>> Hi Cédric,
>>> I looked at it for many hours trying different things (using your
>>> testcase) but I am kind of stuck with this (getting the same error as
>>> you). The difference I can see with the way I run the cdo.hibernate
>>> testcases (as a junit test) and how your testcase is run (junit
>>> plugin test). This for sure means difference in class loader behavior.
>>>
>>> Eike, do you have an idea? I am not sure how classloading works when
>>> you start a cdo server in a plugin. Which plugins are then available
>>> in the cdo server? The testcases can be downloaded from the link below.
>>>
>>> gr. Martin
>>>
>>> Cédric Brun wrote:
>>>> Thanks a lot ! I'm looking forward to your advices !
>>>>
>>>> Cédric
>>>>
>>>> Martin Taal wrote:
>>>>
>>>>> I am downloading your zip-file. I don't have time to look at it
>>>>> right away
>>>>> but I will do my best later this evening.
>>>>>
>>>>> gr. Martin
>>>>>
>>>>> Cédric Brun wrote:
>>>>>> I tried to do the same things as in the CDO/Hibernate unit tests.
>>>>>>
>>>>>> Here's a zipped project trying to launch an HibernateServer with a
>>>>>> unit
>>>>>> test adding elements.
>>>>>> ( http://cedric.brun.free.fr/Tortoose/org.eclipse.cdo-teneo-te st.zip)
>>>>>> The
>>>>>> libs I found are included in the same project, but I still have no
>>>>>> luck
>>>>>> and failling at :
>>>>>>
>>>>>> at
>>>>>>
>>>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>>>
>>>>>> ... 62 more
>>>>>> Caused by: java.lang.NoClassDefFoundError:
>>>>>> org/hibernate/proxy/HibernateProxy
>>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>>>>>> ... 68 more
>>>>>>
>>>>>> please note that the test project I provide really is a "quick and
>>>>>> ugly"
>>>>>> one ^^
>>>>>>
>>>>>>
>>>>>> Cédric
>>>>>>
>>>>>>
>>>>>>
>>>>>> Martin Taal wrote:
>>>>>>
>>>>>>> Hmm, I can understand your headache. Classpath issues are the most
>>>>>>> annoying errors...
>>>>>>>
>>>>>>> In your junit test source file, what happens when you enter the
>>>>>>> org.objectweb.asm.Type, can you source file see this class?
>>>>>>> Do you have cglib in some other plugin which does not have asm?
>>>>>>>
>>>>>>> What I also find slightly strange is that cglib is used in this case
>>>>>>> because normally cdo objects are not proxied using cglib. Can you
>>>>>>> post
>>>>>>> the generated mapping? How does your cdo initialization code look
>>>>>>> like?
>>>>>>>
>>>>>>> gr. Martin
>>>>>>>
>>>>>>> Cédric Brun wrote:
>>>>>>>> Hi Martin,
>>>>>>>>
>>>>>>>> and thanks for your quick answer ! what I did is the following :
>>>>>>>>
>>>>>>>> I used the org.eclipse.emf.cdo.server.hibernate.libraries bundle
>>>>>>>> and
>>>>>>>> added that :
>>>>>>>>
>>>>>>>> Bundle-ClassPath: lib/hibernate3.jar,
>>>>>>>> lib/cglib-2.1.3.jar,
>>>>>>>> lib/commons-collections-2.1.1.jar,
>>>>>>>> lib/dom4j.jar,
>>>>>>>> lib/jta.jar
>>>>>>>> lib/asm-3.0.jar
>>>>>>>>
>>>>>>>> and this directive :
>>>>>>>>
>>>>>>>> Eclipse-BuddyPolicy: dependent
>>>>>>>>
>>>>>>>> as stated in
>>>>>>>> http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>>>>>>>>
>>>>>>>>
>>>>>>>> Then I don't get the same error, now when I start my own plugin
>>>>>>>> (JUnit
>>>>>>>> plugin test) depending on
>>>>>>>> org.eclipse.emf.cdo.server.hibernate.libraries I get :
>>>>>>>>
>>>>>>>> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
>>>>>>>> at
>>>>>>>> net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
>>>>>>>> at
>>>>>>>> net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
>>>>>>>> at net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
>>>>>>>> at
>>>>>>>>
>>>> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>>>>
>>>>>>>> These jars are such a headache ! Do you know - by chance - what's
>>>>>>>> going on ?
>>>>>>>>
>>>>>>>>
>>>>>>>> Cédric
>>>>>>>>
>>>>>>>>
>>>>>>>> Martin Taal wrote:
>>>>>>>>
>>>>>>>>> Hi Cédric,
>>>>>>>>> Most of the time this exception is thrown because hibernate
>>>>>>>>> can't find
>>>>>>>>> the class. A common cause is that the hibernate jar files are
>>>>>>>>> located
>>>>>>>>> in a separate plugin which can not reach the classes in other
>>>>>>>>> plugins.
>>>>>>>>> If you set the eclipse buddy policy of the hibernate plugin to
>>>>>>>>> dependent then this can help.
>>>>>>>>>
>>>>>>>>> gr. Martin
>>>>>>>>>
>>>>>>>>> Cédric Brun wrote:
>>>>>>>>>> Hi Simon and Eike,
>>>>>>>>>>
>>>>>>>>>> thanks for your feedback, now I know what I could expect in using
>>>>>>>>>> optimized implementation, I'll have a try on the hibernate
>>>>>>>>>> store and
>>>>>>>>>> see if it is scaling in a better way.
>>>>>>>>>>
>>>>>>>>>> You gave me many pointers to explain the performances issues
>>>>>>>>>> but in
>>>>>>>>>> fact it seems that the SQL execution is taking so much time (not
>>>>>>>>>> suprisingly in fact ;) . My first profile was wrong.
>>>>>>>>>>
>>>>>>>>>> So I'm installing the hibernate store retrieving all the
>>>>>>>>>> source code
>>>>>>>>>> from CVS, and I'm trying to launch the unit tests. I added
>>>>>>>>>> many jars
>>>>>>>>>> and everything gets compiled but most of the tests are
>>>>>>>>>> failling with
>>>>>>>>>> "
>>>>>>>>>>
>>>>>>>>>> "
>>>>>>>>>> org.hibernate.MappingException: Could not determine type for:
>>>>>>>>>>
>>>> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>>>>>>>>> at table: cdo_features, for columns:
>>>>>>>>>> [org.hibernate.mapping.Column(type)] "
>>>>>>>>>>
>>>>>>>>>> any hint ?
>>>>>>>>>>
>>>>>>>>>> Cédric
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Simon McDuff wrote:
>>>>>>>>>>
>>>>>>>>>>> We noticed a performance degradation in the following case:
>>>>>>>>>>>
>>>>>>>>>>> When we fetch a collection of objects that contains
>>>>>>>>>>> collections that
>>>>>>>>>>> we don't want to use. Ex:
>>>>>>>>>>>
>>>>>>>>>>> Model:
>>>>>>>>>>>
>>>>>>>>>>> ClassB
>>>>>>>>>>> {
>>>>>>>>>>> int foo1;
>>>>>>>>>>> List childrens1;
>>>>>>>>>>> List childrens2;
>>>>>>>>>>> List childrens3;
>>>>>>>>>>> };
>>>>>>>>>>>
>>>>>>>>>>> Code:
>>>>>>>>>>>
>>>>>>>>>>> Resource.contents contains objects of type ClassB.
>>>>>>>>>>>
>>>>>>>>>>> By fetching instances of classB, we will also fetched some
>>>>>>>>>>> items in
>>>>>>>>>>> the list(childrens1, childrens2, childrens3) The problem we
>>>>>>>>>>> had.. is
>>>>>>>>>>> we do not know if we want to use it. SO better not to
>>>>>>>>>>> download them.
>>>>>>>>>>>
>>>>>>>>>>> So In our production(we used our own IStore), we never
>>>>>>>>>>> download any
>>>>>>>>>>> objects from collection. I'm working at the moment to put in
>>>>>>>>>>> place
>>>>>>>>>>> all feature we used in the current CDO. That one will be part of
>>>>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477 To begin
>>>>>>>>>>> with,CDORevisionImpl needs to supports list that are 0 even if
>>>>>>>>>>> referenceChunkSize == 0.
>>>>>>>>>>>
>>>>>>>>>>> It is true that we obtain good performance.
>>>>>>>>>>> Worst case :
>>>>>>>>>>> 1500 objects /sec (Real fetch from disk. Page that contains the
>>>>>>>>>>> objects, wasn't in the cache of the filesystem.)
>>>>>>>>>>>
>>>>>>>>>>> Best case
>>>>>>>>>>> 12 000 objects/Sec (Fetch objects that are already in the
>>>>>>>>>>> case of
>>>>>>>>>>> the filesystem)
>>>>>>>>>>>
>>>>>>>>>>> (These statistics always fetch objects from server to client.
>>>>>>>>>>> The
>>>>>>>>>>> cache client isn't used)
>>>>>>>>>>>
>>>>>>>>>>> We have another optimization one with SmartReadAhead but for now
>>>>>>>>>>> only work
>>>>>>>>>>> for UI client. (for now) This is something we will integrate as
>>>>>>>>>>> well in CDO. But for now we concentrate on the other things.
>>>>>>>>>>>
>>>>>>>>>>> I used an Object database. Based on my preliminary test
>>>>>>>>>>> without CDO.
>>>>>>>>>>> I compare
>>>>>>>>>>>
>>>>>>>>>>> Hibernate vs Objectivity/EMF :
>>>>>>>>>>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>>>>>>>>>>
>>>>>>>>>>> The worst.. you will get 30 to 240 objects/sec or
>>>>>>>>>>> the best... you will get 500 to 4000 objects/sec
>>>>>>>>>>>
>>>>>>>>>>> Based on your comments... you are getting the worst (with
>>>>>>>>>>> Derby)!!
>>>>>>>>>>> :-)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Another comments about insertion.
>>>>>>>>>>>
>>>>>>>>>>> We found another problem when we insert data(DOn't know if it is
>>>>>>>>>>> still true). Resource.Contents is a unique list. Basically
>>>>>>>>>>> each time
>>>>>>>>>>> we insert one element, it needs to fetch all the collection
>>>>>>>>>>> to see
>>>>>>>>>>> if we already have this element. Basically.. it is time
>>>>>>>>>>> consuming.
>>>>>>>>>>> In our case we are not going through this stage to persist our
>>>>>>>>>>> objects... but the objects are not accessible through that list.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> If speed is an issue for you!! We (You included :-)) can
>>>>>>>>>>> optimize
>>>>>>>>>>> it. What I like is Eike did a really good job doing that
>>>>>>>>>>> framwork
>>>>>>>>>>> and optimizations are easy to add. Also you can change your
>>>>>>>>>>> back-end
>>>>>>>>>>> without affecting you client side. (Maybe going to an Object
>>>>>>>>>>> database!!) THis was one of the purpose of CDO!! :-)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Simon
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le
>>>>>>>>>>> message de
>>>>>>>>>>> news: 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>>>>>>>>
>>>>>>>>>>> That sounds interesting!
>>>>>>>>>>> Comments inline...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Cédric Brun schrieb:
>>>>>>>>>>> Hi Eike,
>>>>>>>>>>>
>>>>>>>>>>> I've been messing around CDO lately and I hacked a few ugly
>>>>>>>>>>> unit-tests measuring read/write throughput when I use CDO (RC2).
>>>>>>>>>>>
>>>>>>>>>>> I'm using an embeded-derby database, two different metamodels
>>>>>>>>>>> (complex one - 250 EClassifiers/300 features and a simple one
>>>>>>>>>>> - 1
>>>>>>>>>>> Classifier/3 Features)
>>>>>>>>>>>
>>>>>>>>>>> The test is a "quick and dirty" one so please do not take these
>>>>>>>>>>> figures too seriously, I would just like to have your input
>>>>>>>>>>> about
>>>>>>>>>>> the "order of magnitude" I should expect or settings affecting
>>>>>>>>>>> seriously these performances. I tried both JVM and TCP
>>>>>>>>>>> acceptor but
>>>>>>>>>>> the performances are quite similar though JVM is a bit faster.
>>>>>>>>>>> Looks like a sign that the NIO socket channels are
>>>>>>>>>>> efficient ,-)
>>>>>>>>>>> You used localhost for TCP transport I guess?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> * First case : building a small model (10000 elements) with the
>>>>>>>>>>> simple MM, only adding instances in an empty database :
>>>>>>>>>>> commiting change on every add I get a 2 objects per second rate.
>>>>>>>>>>> commiting changes every 10 adds - I get a 30 objects per second
>>>>>>>>>>> rate.
>>>>>>>>>>> Seems to be consistent with the fact that only for commits
>>>>>>>>>>> there
>>>>>>>>>>> is remote traffic. And the backend transaction management is
>>>>>>>>>>> another overhead. So the impact of the number objects/commit
>>>>>>>>>>> starts with bigger transactions.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> browsing the just commited elements opening a new transaction
>>>>>>>>>>> and
>>>>>>>>>>> getting a new Resource instance, I get around 30 000
>>>>>>>>>>> objets/sec rate
>>>>>>>>>>> just iterating thanks to Resource.getAllContents().
>>>>>>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>>>>>>> All views and transactions of a single session share (and
>>>>>>>>>>> cache)
>>>>>>>>>>> their object state. Otherwise I'd expect much smaller values.
>>>>>>>>>>> With his ObjectivityStore implementation Simon (cc'ed) reached
>>>>>>>>>>> rates of several thousands objects/sec. But the DBStore
>>>>>>>>>>> (which you
>>>>>>>>>>> used) is not that optimized.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> * Second case : I built a big model in the database using the
>>>>>>>>>>> complex metamodel. It contains 700 000 elements. The creation
>>>>>>>>>>> went
>>>>>>>>>>> pretty nicely (I did not measured it exactly as it was launched
>>>>>>>>>>> during a night, but it seems I got around 100 objects/sec
>>>>>>>>>>> rate in
>>>>>>>>>>> writting as I was writting every 700 objects.
>>>>>>>>>>> Sounds not too bad.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Once the model is built the derby database is 222 MB (which - I
>>>>>>>>>>> heard - is not so big for derby).
>>>>>>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>>>>>>> To be honest, I'm really no database expert (although I can
>>>>>>>>>>> write
>>>>>>>>>>> simple SQL statements).
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Retrieving the distant resource and calling eAllContents() on
>>>>>>>>>>> it, I
>>>>>>>>>>> get a 50 Objects/sec rate at best, which mean my
>>>>>>>>>>> eAllContents() call
>>>>>>>>>>> would take 4 hours !
>>>>>>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>>>>>>> (iteration order and so). There are several CDO features
>>>>>>>>>>> that can
>>>>>>>>>>> have a major impact on load rates!
>>>>>>>>>>>
>>>>>>>>>>> - Collections (xrefs and containment) can be loaded lazily. I
>>>>>>>>>>> don't mean the target objects, which are lazy by default
>>>>>>>>>>> anyway,
>>>>>>>>>>> but even the target IDs that are in the collections. We
>>>>>>>>>>> call that
>>>>>>>>>>> "reference chunking" (but search for a better name is
>>>>>>>>>>> ongoing).
>>>>>>>>>>>
>>>>>>>>>>> public interface CDOSession extends CDOProtocolSession,
>>>>>>>>>>> IContainer<CDOView>
>>>>>>>>>>> {
>>>>>>>>>>> public int getReferenceChunkSize();
>>>>>>>>>>>
>>>>>>>>>>> public void setReferenceChunkSize(int
>>>>>>>>>>> referenceChunkSize);
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> - Loading collection elements (target objects) can be
>>>>>>>>>>> batched (the
>>>>>>>>>>> current name is even worse)
>>>>>>>>>>>
>>>>>>>>>>> public interface CDOView extends CDOProtocolView,
>>>>>>>>>>> INotifier
>>>>>>>>>>> {
>>>>>>>>>>> public int getLoadRevisionCollectionChunkSize();
>>>>>>>>>>>
>>>>>>>>>>> public void setLoadRevisionCollectionChunkSize(int
>>>>>>>>>>> loadRevisionCollectionChunkSize);
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> - One of the most sophisticated feature is automatic model
>>>>>>>>>>> usage
>>>>>>>>>>> analyzation. The result are fetch rules that are sent to the
>>>>>>>>>>> server. When later objects are loaded from the server, it can
>>>>>>>>>>> answer with additional objects that are likely to be needed
>>>>>>>>>>> by the
>>>>>>>>>>> client in the near future.
>>>>>>>>>>>
>>>>>>>>>>> public class CDOViewImpl extends
>>>>>>>>>>> org.eclipse.net4j.internal.util.event.Notifier
>>>>>>>>>>> implements
>>>>>>>>>>> CDOView, CDOIDProvider, Adapter.Internal
>>>>>>>>>>> {
>>>>>>>>>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>>>>>>>> {
>>>>>>>>>>> return featureAnalyzer;
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>>>>>>>> featureAnalyzer)
>>>>>>>>>>> {
>>>>>>>>>>> this.featureAnalyzer = featureAnalyzer == null ?
>>>>>>>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> - There's a SmartReadAheadThread (if you're curious, please
>>>>>>>>>>> ask
>>>>>>>>>>> Simon). - Caches can be configured
>>>>>>>>>>> - ...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Known issues of the DBStore:
>>>>>>>>>>> - When in auditing mode updates are always written as complete
>>>>>>>>>>> inserts (by design) - When configuring toManyReferences to
>>>>>>>>>>> a value
>>>>>>>>>>> different from LIKE_ATTRIBUTES (which is not implemented
>>>>>>>>>>> yet) it
>>>>>>>>>>> is very slow. - No (few) prepared statements are used (see
>>>>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) - Maybe
>>>>>>>>>>> others...
>>>>>>>>>>>
>>>>>>>>>>> Please note that the DBStore was not written by me to
>>>>>>>>>>> compete with
>>>>>>>>>>> Hibernate in terms of performance or mapping capabilities. Now
>>>>>>>>>>> that Martin and I, well in the end mostly Martin, provide a
>>>>>>>>>>> HibernateStore and this HibernateStore gets more and more
>>>>>>>>>>> features
>>>>>>>>>>> I expect that there will be a trend away from the DBStore.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This is probably linked to the way the mapping to db is done,
>>>>>>>>>>> here's
>>>>>>>>>>> the config I used (came from CDO-server example if I recall
>>>>>>>>>>> correctly).
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <repository name="repo1">
>>>>>>>>>>> <property name="overrideUUID"
>>>>>>>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>>>>>>>> <property name="supportingAudits" value="true"/>
>>>>>>>>>>> <property name="verifyingRevisions"
>>>>>>>>>>> value="false"/>
>>>>>>>>>>> <property name="currentLRUCapacity"
>>>>>>>>>>> value="10000"/>
>>>>>>>>>>> I guess this one could be larger.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> <property name="revisedLRUCapacity"
>>>>>>>>>>> value="100"/>
>>>>>>>>>>> <store type="db">
>>>>>>>>>>> <!-- type: horizontal | vertical | <any
>>>>>>>>>>> user-contributed type> -->
>>>>>>>>>>> <mappingStrategy
>>>>>>>>>>> type="horizontal">
>>>>>>>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>>>>>>>> ONE_TABLE_PER_CLASS |
>>>>>>>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY |
>>>>>>>>>>> LIKE_ATTRIBUTES
>>>>>>>>>>> -->
>>>>>>>>>>> <property
>>>>>>>>>>> name="toManyReferences"
>>>>>>>>>>>
>>>>>>>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>>>>>>>
>>>>>>>>>>> <!-- LIKE_ATTRIBUTES |
>>>>>>>>>>> LIKE_TO_MANY_REFERENCES -->
>>>>>>>>>>> <property name="toOneReferences"
>>>>>>>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>>>>>>>
>>>>>>>>>>> <!-- MODEL | STRATEGY -->
>>>>>>>>>>> <property
>>>>>>>>>>> name="mappingPrecedence"
>>>>>>>>>>> value="MODEL"/>
>>>>>>>>>>> This one is obsolete.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> </mappingStrategy>
>>>>>>>>>>> <dbAdapter
>>>>>>>>>>> name="derby-embedded"/>
>>>>>>>>>>> <dataSource
>>>>>>>>>>>
>>>>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>>>>>>>>>> databaseName="/tmp/cdodb1"
>>>>>>>>>>> createDatabase="create"/>
>>>>>>>>>>>
>>>>>>>>>>> </store>
>>>>>>>>>>> </repository>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> What do you think about that ? What throughput do you think I
>>>>>>>>>>> should
>>>>>>>>>>> expect ? As I said that's very difficult to judge.
>>>>>>>>>>> I believe it's more appropriate to compare the actual
>>>>>>>>>>> values with
>>>>>>>>>>> values from other measures (other setups, properties, etc.
>>>>>>>>>>> or even
>>>>>>>>>>> other products) instead of random expectations. Apologies
>>>>>>>>>>> that I
>>>>>>>>>>> didn't have the time so far to provide some reference measures
>>>>>>>>>>> myself. Maybe this gets better now that the team starts to
>>>>>>>>>>> grow
>>>>>>>>>>> slowly ;-)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> What are the main reasons fur such figures ?
>>>>>>>>>>> Hmm, I *suspect* you don't like the figures? I can find no
>>>>>>>>>>> evidence your assessment ;-)
>>>>>>>>>>>
>>>>>>>>>>> Ahh,one more note:
>>>>>>>>>>> Tracing can also have a significant impact on performance and
>>>>>>>>>>> end-to-end throughput. Turn it off by all means. And please
>>>>>>>>>>> make
>>>>>>>>>>> sure that it's really turned off, not only invisible (i.e.
>>>>>>>>>>> console
>>>>>>>>>>> is not a target).
>>>>>>>>>>>
>>>>>>>>>>> Cheers
>>>>>>>>>>> /Eike
>>>>>
>>>>
>>>
>>>


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Cell: +31 (0)6 288 48 943
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: [CDO] reading/writting throughput performances [message #619130 is a reply to message #124535] Sun, 01 June 2008 12:40 Go to previous message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
This is a multi-part message in MIME format.
--------------070303090402040107080201
Content-Type: multipart/alternative;
boundary="------------040308010500040802070703"


--------------040308010500040802070703
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Hi Martin,

Ah, now with the new mapping I get a big step further!
As I said, Cedric's test code contained some bugs, too. IIRC a TCP
acceptor was created but the client uses a JVMConnector, and stuff like
this.
You could try to overcome the exception below by adding the following to
the server code somewhere:

| IPluginContainer.INSTANCE.getElement("org.eclipse.net4j.acceptors ", "jvm", "default");|



Better just use the attached zip because there may be other changes that
I don't remember in detail anymore.
Now I run into another exception:

Thread-4 [debug.signal] ================ Indicating
CommitTransactionIndication
Thread-4 [debug] Created
org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Writer for
repository repo1
Thread-4 [debug] Created
org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Writer for
repository repo1
Thread-4 [debug] Created
org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Writer for
repository repo1
Thread-4 [debug.protocol] Reading 0 new packages
Thread-4 [debug.protocol] Reading 2 new objects
Thread-4 [debug.model] Reading CDOID of type 3 (TEMP_OBJECT)
Thread-4 [debug.model] Reading CDOID of type 3 (TEMP_OBJECT)
Thread-4 [debug.model] Reading CDOID of type 0 (NULL)
Thread-4 [debug.revision] Reading revision: ID=oid1,
classRef=CDOClassRef(http://www.eclipse.org/emf/CDO/resource/1.0.0, 0),
className=CDOResource, version=1, created=0, revised=0, resource=oid1,
container=NULL, feature=0
Thread-4 [debug.revision] Read feature CDOFeature(ID=9, name=path,
type=STRING, referenceType=null): /res2
Thread-4 [debug.revision] Read feature CDOFeature(ID=2, name=contents,
type=OBJECT, referenceType=CDOClass(ID=0, name=CDOObject)): size=1
Thread-4 [debug.model] Reading CDOID of type 3 (TEMP_OBJECT)
Thread-4 [debug.revision] oid2
[ERROR] ClassRef unresolveable:
CDOClassRef(http://www.eclipse.org/modeling/cdo/bug/names, 0)
java.lang.IllegalStateException: ClassRef unresolveable:
CDOClassRef(http://www.eclipse.org/modeling/cdo/bug/names, 0)
at
org.eclipse.emf.cdo.internal.protocol.revision.CDORevisionIm pl. <init>(CDORevisionImpl.java:116)
at
org.eclipse.emf.cdo.protocol.revision.CDORevisionUtil.read(C DORevisionUtil.java:48)
at
org.eclipse.emf.cdo.internal.server.protocol.CommitTransacti onIndication.indicating(CommitTransactionIndication.java:95)
at
org.eclipse.net4j.signal.IndicationWithResponse.execute(Indi cationWithResponse.java:46)
at org.eclipse.net4j.signal.Signal.runSync(Signal.java:143)
at org.eclipse.net4j.signal.Signal.run(Signal.java:124)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Threa dPoolExecutor.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo lExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)

Cheers
/Eike



Martin Taal schrieb:
> Hi Eike, Cedric,
> I can't say to much about net4j.db versus net4j.db.hsqldb.
>
> You are correct that the hibernate.libraries plugin should have
> Eclipse-BuddyPolicy: dependent (or another way to use the classpath of
> the dependent plugins). Btw, I agree that registered is better as
> afaik dependent gets slow which larger plugin environments.
>
> The reason that cglib is used is because one of the class mappings in
> org.eclipse.emf.cdo.server.hibernate/mappings/meta.hbm/xml did not
> have the lazy="false" directive. I added this and now I get another
> exception (see below). So this 'solves' the problem. If you do a get
> latest of this file then I think this problem is 'overcome'.
>
> Overall I still do not see why it went wrong. As far as I can see the
> cglib library could not find the hibernate class, which is strange as
> they are in the same plugin??
> The cdo testcases don't have this problem and neither does Teneo.
>
> As it does not occur anymore we can ignore it for now (not very
> satisfactory but there are other things to do)... Unless someone else
> has an idea how to handle this ofcourse.
>
> gr. Martin
>
> java.lang.IllegalStateException: acceptor == null
> at
> org.eclipse.net4j.internal.jvm.JVMClientConnector.doBeforeAc tivate(JVMClientConnector.java:57)
>
> at
> org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:56)
>
> at
> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>
> at
> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>
> at
> org.eclipse.net4j.internal.util.container.ManagedContainer.g etElement(ManagedContainer.java:260)
>
> at
> org.eclipse.emf.internal.cdo.util.ChannelInjector.getConnect or(ChannelInjector.java:74)
>
> at
> org.eclipse.emf.internal.cdo.util.ChannelInjector.process(Ch annelInjector.java:41)
>
> at
> org.eclipse.net4j.internal.util.container.ManagedContainer.p ostProcessElement(ManagedContainer.java:407)
>
> at
> org.eclipse.net4j.internal.util.container.ManagedContainer.g etElement(ManagedContainer.java:259)
>
> at
> org.eclipse.cdo.bug.hibernate.HibernateServer.createSession( HibernateServer.java:93)
>
> at
> org.eclipse.cdo.bug.unit.LaunchServerTest.testStringValue(La unchServerTest.java:43)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:585)
> at junit.framework.TestCase.runTest(TestCase.java:164)
> at junit.framework.TestCase.runBare(TestCase.java:130)
> at junit.framework.TestResult$1.protect(TestResult.java:106)
> at junit.framework.TestResult.runProtected(TestResult.java:124)
> at junit.framework.TestResult.run(TestResult.java:109)
> at junit.framework.TestCase.run(TestCase.java:120)
> at junit.framework.TestSuite.runTest(TestSuite.java:230)
> at junit.framework.TestSuite.run(TestSuite.java:225)
> at
> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
>
> at
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
>
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
>
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
>
> at
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
>
> at
> org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
>
> at
> org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:585)
> at
> org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hod(EclipseAppContainer.java:547)
>
> at
> org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
>
> at
> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:106)
>
> at
> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:76)
>
> at
> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:362)
>
> at
> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:175)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 564)
> at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
> at org.eclipse.equinox.launcher.Main.run(Main.java:1251)
> at org.eclipse.equinox.launcher.Main.main(Main.java:1227)
>
>
>
> Eike Stepper wrote:
>> Hi,
>>
>> Here is a diagram of the dependencies of the relevant plugins (please
>> correct me if something is wrong).
>> The black arrows are ordinary downstream dependencies (Require-Bundle).
>> The red arrows are upstream dependencies (Eclipse-BuddyPolicy:
>> dependent).
>>
>>
>>
>> For research purposes I added code to the libraries plugin that tries
>> to reflectively load CDOFeatureImpl and that is not a problem!
>> It seems to me as if our problem is not class loading related because
>> the hibernate.jar should be able to see the needed classes.
>> I believe this is either a bug in Hibernate or our wrong usage of
>> Hibernate. I tend to believe the latter ;-)
>> I put a breakpoint in
>> org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping .EntityEntityModeToTuplizerMapping(PersistentClass,
>> EntityMetamodel)
>> to see how the tuplizers are created. I'm too unexperienced with
>> Hibernate to get a clue about what happens there...
>>
>> Btw. I'd prefer to see the application plugin depend on net4j.db
>> instead of net4j.db.hsqldb.
>> But it seems as if the BuddyPolicy in net4j.db is not "reexported" so
>> that the application can see the hsqldb.jar then ;-(
>> I posted a respective question to equinox...
>>
>> Cheers
>> /Eike
>>
>>
>> Eike Stepper schrieb:
>>> Hi guys,
>>>
>>> I can confirm, making these 3rd party libraries with their own
>>> classloading run in OSGi is a nightmare! I took me hours teach where
>>> the hsqldb driver is. On the way I came over different kinds of
>>> classloading related errors.
>>>
>>> First I removed all the jars from the test plugin (the one Cedric
>>> provided). They're already in the cdo.server.hibernate.libraries
>>> plugin which, in fact, needs the buddy policy set to dependent. I
>>> think this last change needs to be committed to CVS! Btw. I'd prefer
>>> to use *registered* buddies instead of making all dependent bundles
>>> visible.
>>>
>>> The test plugin then needs to provide the database driver. Therefore
>>> I added a dependency on the org.eclipse.net4j.db.hsqldb plugin.
>>>
>>> With this setup I can start the server without CNFE, NCDFE, CCI and
>>> the like. At this point I get the following exception, which seems
>>> the same as the one Cedric observed in the end:
>>>
>>> net.sf.cglib.core.CodeGenerationException:
>>> java.lang.reflect.InvocationTargetException-->null
>>> at
>>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:237)
>>>
>>> at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
>>> at net.sf.cglib.proxy.Enhancer.createClass(Enhancer.java:317)
>>> at
>>> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:127)
>>>
>>> at
>>> org.hibernate.proxy.pojo.cglib.CGLIBProxyFactory.postInstant iate(CGLIBProxyFactory.java:43)
>>>
>>> at
>>> org.hibernate.tuple.entity.PojoEntityTuplizer.buildProxyFact ory(PojoEntityTuplizer.java:162)
>>>
>>> at
>>> org.hibernate.tuple.entity.AbstractEntityTuplizer.<init>(AbstractEntityTuplizer.java:135)
>>>
>>> at
>>> org.hibernate.tuple.entity.PojoEntityTuplizer.<init>(PojoEntityTuplizer.java:55)
>>>
>>> at
>>> org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping . <init>(EntityEntityModeToTuplizerMapping.java:56)
>>>
>>> at
>>> org.hibernate.tuple.entity.EntityMetamodel.<init>(EntityMetamodel.java:295)
>>>
>>> at
>>> org.hibernate.persister.entity.AbstractEntityPersister.<init >(AbstractEntityPersister.java:434)
>>>
>>> at
>>> org.hibernate.persister.entity.JoinedSubclassEntityPersister . <init>(JoinedSubclassEntityPersister.java:91)
>>>
>>> at
>>> org.hibernate.persister.PersisterFactory.createClassPersiste r(PersisterFactory.java:58)
>>>
>>> at
>>> org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:226)
>>>
>>> at
>>> org.hibernate.cfg.Configuration.buildSessionFactory(Configur ation.java:1294)
>>>
>>> at
>>> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getSessionFactory(HibernatePackageHandler.java:335 )
>>>
>>> at
>>> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.readPackageInfos(HibernatePackageHandler.java:293)
>>>
>>> at
>>> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getCDOPackageInfos(HibernatePackageHandler.java:21 5)
>>>
>>> at
>>> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Reader.readPackageInfos(HibernateStoreReader.java:87)
>>>
>>> at
>>> org.eclipse.emf.cdo.internal.server.PackageManager.doActivat e(PackageManager.java:72)
>>>
>>> at
>>> org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
>>>
>>> at
>>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>>>
>>> at
>>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>>>
>>> at
>>> org.eclipse.emf.cdo.internal.server.Repository.activateRepos itory(Repository.java:264)
>>>
>>> at
>>> org.eclipse.emf.cdo.internal.server.Repository.doActivate(Re pository.java:251)
>>>
>>> at
>>> org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
>>>
>>> at
>>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>>>
>>> at
>>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>>>
>>> at
>>> org.eclipse.emf.cdo.server.CDOServerUtil.addRepository(CDOSe rverUtil.java:67)
>>>
>>> at
>>> org.eclipse.cdo.bug.LaunchServerTest.setUp(LaunchServerTest. java:54)
>>> at junit.framework.TestCase.runBare(TestCase.java:128)
>>> at junit.framework.TestResult$1.protect(TestResult.java:106)
>>> at junit.framework.TestResult.runProtected(TestResult.java:124)
>>> at junit.framework.TestResult.run(TestResult.java:109)
>>> at junit.framework.TestCase.run(TestCase.java:120)
>>> at junit.framework.TestSuite.runTest(TestSuite.java:230)
>>> at junit.framework.TestSuite.run(TestSuite.java:225)
>>> at
>>> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
>>>
>>> at
>>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
>>>
>>> at
>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
>>>
>>> at
>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
>>>
>>> at
>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
>>>
>>> at
>>> org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
>>>
>>> at
>>> org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>> at
>>> org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hodWithException(EclipseAppContainer.java:574)
>>>
>>> at
>>> org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
>>>
>>> at
>>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:110)
>>>
>>> at
>>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:79)
>>>
>>> at
>>> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:379)
>>>
>>> at
>>> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:179)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>> at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 549)
>>> at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
>>> at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
>>> at org.eclipse.equinox.launcher.Main.main(Main.java:1212)
>>> Caused by: java.lang.reflect.InvocationTargetException
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>> at net.sf.cglib.core.ReflectUtils.defineClass(ReflectUtils.java :384)
>>> at
>>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>>
>>> ... 61 more
>>> Caused by: java.lang.NoClassDefFoundError:
>>> org/hibernate/proxy/HibernateProxy
>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>>> ... 67 more
>>>
>>> I must admit that I'm still not sure about te fundamental interwork
>>> between Hibernate and the user defined meta model. It seems as if
>>> asm and cglib are somewhere in between but I have no idea why at
>>> all. I thought we have implemented a bunch of static Hibernate
>>> interfaces to make the CDORevision instances known to Hibernate.
>>> Please somebody explain why we need runtime byte code engineering?
>>> Is there some configuration missing to tell Hibernate to use our
>>> special tuplizers?
>>>
>>> I have changed several things in the code of the test plugin and
>>> since it's quite small now that the jars are removed, I attach my
>>> version of the zip here (including a JUnit Plugin launch config).
>>> And don't forget to update the cdo.server.hibernate.libraries plugin
>>> from CVS to get the new buddy policy!
>>>
>>> Cheers
>>> /Eike
>>>
>>> Martin Taal schrieb:
>>>> Hi Cédric,
>>>> I looked at it for many hours trying different things (using your
>>>> testcase) but I am kind of stuck with this (getting the same error
>>>> as you). The difference I can see with the way I run the
>>>> cdo.hibernate testcases (as a junit test) and how your testcase is
>>>> run (junit plugin test). This for sure means difference in class
>>>> loader behavior.
>>>>
>>>> Eike, do you have an idea? I am not sure how classloading works
>>>> when you start a cdo server in a plugin. Which plugins are then
>>>> available in the cdo server? The testcases can be downloaded from
>>>> the link below.
>>>>
>>>> gr. Martin
>>>>
>>>> Cédric Brun wrote:
>>>>> Thanks a lot ! I'm looking forward to your advices !
>>>>>
>>>>> Cédric
>>>>>
>>>>> Martin Taal wrote:
>>>>>
>>>>>> I am downloading your zip-file. I don't have time to look at it
>>>>>> right away
>>>>>> but I will do my best later this evening.
>>>>>>
>>>>>> gr. Martin
>>>>>>
>>>>>> Cédric Brun wrote:
>>>>>>> I tried to do the same things as in the CDO/Hibernate unit tests.
>>>>>>>
>>>>>>> Here's a zipped project trying to launch an HibernateServer with
>>>>>>> a unit
>>>>>>> test adding elements.
>>>>>>> ( http://cedric.brun.free.fr/Tortoose/org.eclipse.cdo-teneo-te st.zip)
>>>>>>> The
>>>>>>> libs I found are included in the same project, but I still have
>>>>>>> no luck
>>>>>>> and failling at :
>>>>>>>
>>>>>>> at
>>>>>>>
>>>>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>>>>
>>>>>>> ... 62 more
>>>>>>> Caused by: java.lang.NoClassDefFoundError:
>>>>>>> org/hibernate/proxy/HibernateProxy
>>>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>>>>>>> ... 68 more
>>>>>>>
>>>>>>> please note that the test project I provide really is a "quick
>>>>>>> and ugly"
>>>>>>> one ^^
>>>>>>>
>>>>>>>
>>>>>>> Cédric
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Martin Taal wrote:
>>>>>>>
>>>>>>>> Hmm, I can understand your headache. Classpath issues are the most
>>>>>>>> annoying errors...
>>>>>>>>
>>>>>>>> In your junit test source file, what happens when you enter the
>>>>>>>> org.objectweb.asm.Type, can you source file see this class?
>>>>>>>> Do you have cglib in some other plugin which does not have asm?
>>>>>>>>
>>>>>>>> What I also find slightly strange is that cglib is used in this
>>>>>>>> case
>>>>>>>> because normally cdo objects are not proxied using cglib. Can
>>>>>>>> you post
>>>>>>>> the generated mapping? How does your cdo initialization code
>>>>>>>> look like?
>>>>>>>>
>>>>>>>> gr. Martin
>>>>>>>>
>>>>>>>> Cédric Brun wrote:
>>>>>>>>> Hi Martin,
>>>>>>>>>
>>>>>>>>> and thanks for your quick answer ! what I did is the following :
>>>>>>>>>
>>>>>>>>> I used the org.eclipse.emf.cdo.server.hibernate.libraries
>>>>>>>>> bundle and
>>>>>>>>> added that :
>>>>>>>>>
>>>>>>>>> Bundle-ClassPath: lib/hibernate3.jar,
>>>>>>>>> lib/cglib-2.1.3.jar,
>>>>>>>>> lib/commons-collections-2.1.1.jar,
>>>>>>>>> lib/dom4j.jar,
>>>>>>>>> lib/jta.jar
>>>>>>>>> lib/asm-3.0.jar
>>>>>>>>>
>>>>>>>>> and this directive :
>>>>>>>>>
>>>>>>>>> Eclipse-BuddyPolicy: dependent
>>>>>>>>>
>>>>>>>>> as stated in
>>>>>>>>> http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Then I don't get the same error, now when I start my own
>>>>>>>>> plugin (JUnit
>>>>>>>>> plugin test) depending on
>>>>>>>>> org.eclipse.emf.cdo.server.hibernate.libraries I get :
>>>>>>>>>
>>>>>>>>> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
>>>>>>>>> at
>>>>>>>>> net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
>>>>>>>>> at
>>>>>>>>> net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
>>>>>>>>> at net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
>>>>>>>>> at
>>>>>>>>>
>>>>> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>>>>>
>>>>>>>>> These jars are such a headache ! Do you know - by chance -
>>>>>>>>> what's
>>>>>>>>> going on ?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cédric
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Martin Taal wrote:
>>>>>>>>>
>>>>>>>>>> Hi Cédric,
>>>>>>>>>> Most of the time this exception is thrown because hibernate
>>>>>>>>>> can't find
>>>>>>>>>> the class. A common cause is that the hibernate jar files are
>>>>>>>>>> located
>>>>>>>>>> in a separate plugin which can not reach the classes in other
>>>>>>>>>> plugins.
>>>>>>>>>> If you set the eclipse buddy policy of the hibernate plugin to
>>>>>>>>>> dependent then this can help.
>>>>>>>>>>
>>>>>>>>>> gr. Martin
>>>>>>>>>>
>>>>>>>>>> Cédric Brun wrote:
>>>>>>>>>>> Hi Simon and Eike,
>>>>>>>>>>>
>>>>>>>>>>> thanks for your feedback, now I know what I could expect in
>>>>>>>>>>> using
>>>>>>>>>>> optimized implementation, I'll have a try on the hibernate
>>>>>>>>>>> store and
>>>>>>>>>>> see if it is scaling in a better way.
>>>>>>>>>>>
>>>>>>>>>>> You gave me many pointers to explain the performances issues
>>>>>>>>>>> but in
>>>>>>>>>>> fact it seems that the SQL execution is taking so much time
>>>>>>>>>>> (not
>>>>>>>>>>> suprisingly in fact ;) . My first profile was wrong.
>>>>>>>>>>>
>>>>>>>>>>> So I'm installing the hibernate store retrieving all the
>>>>>>>>>>> source code
>>>>>>>>>>> from CVS, and I'm trying to launch the unit tests. I added
>>>>>>>>>>> many jars
>>>>>>>>>>> and everything gets compiled but most of the tests are
>>>>>>>>>>> failling with
>>>>>>>>>>> "
>>>>>>>>>>>
>>>>>>>>>>> "
>>>>>>>>>>> org.hibernate.MappingException: Could not determine type for:
>>>>>>>>>>>
>>>>> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>>>>
>>>>>>>>>>> at table: cdo_features, for columns:
>>>>>>>>>>> [org.hibernate.mapping.Column(type)] "
>>>>>>>>>>>
>>>>>>>>>>> any hint ?
>>>>>>>>>>>
>>>>>>>>>>> Cédric
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Simon McDuff wrote:
>>>>>>>>>>>
>>>>>>>>>>>> We noticed a performance degradation in the following case:
>>>>>>>>>>>>
>>>>>>>>>>>> When we fetch a collection of objects that contains
>>>>>>>>>>>> collections that
>>>>>>>>>>>> we don't want to use. Ex:
>>>>>>>>>>>>
>>>>>>>>>>>> Model:
>>>>>>>>>>>>
>>>>>>>>>>>> ClassB
>>>>>>>>>>>> {
>>>>>>>>>>>> int foo1;
>>>>>>>>>>>> List childrens1;
>>>>>>>>>>>> List childrens2;
>>>>>>>>>>>> List childrens3;
>>>>>>>>>>>> };
>>>>>>>>>>>>
>>>>>>>>>>>> Code:
>>>>>>>>>>>>
>>>>>>>>>>>> Resource.contents contains objects of type ClassB.
>>>>>>>>>>>>
>>>>>>>>>>>> By fetching instances of classB, we will also fetched some
>>>>>>>>>>>> items in
>>>>>>>>>>>> the list(childrens1, childrens2, childrens3) The problem we
>>>>>>>>>>>> had.. is
>>>>>>>>>>>> we do not know if we want to use it. SO better not to
>>>>>>>>>>>> download them.
>>>>>>>>>>>>
>>>>>>>>>>>> So In our production(we used our own IStore), we never
>>>>>>>>>>>> download any
>>>>>>>>>>>> objects from collection. I'm working at the moment to put
>>>>>>>>>>>> in place
>>>>>>>>>>>> all feature we used in the current CDO. That one will be
>>>>>>>>>>>> part of
>>>>>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477 To begin
>>>>>>>>>>>> with,CDORevisionImpl needs to supports list that are 0 even if
>>>>>>>>>>>> referenceChunkSize == 0.
>>>>>>>>>>>>
>>>>>>>>>>>> It is true that we obtain good performance.
>>>>>>>>>>>> Worst case :
>>>>>>>>>>>> 1500 objects /sec (Real fetch from disk. Page that contains
>>>>>>>>>>>> the
>>>>>>>>>>>> objects, wasn't in the cache of the filesystem.)
>>>>>>>>>>>>
>>>>>>>>>>>> Best case
>>>>>>>>>>>> 12 000 objects/Sec (Fetch objects that are already in the
>>>>>>>>>>>> case of
>>>>>>>>>>>> the filesystem)
>>>>>>>>>>>>
>>>>>>>>>>>> (These statistics always fetch objects from server to
>>>>>>>>>>>> client. The
>>>>>>>>>>>> cache client isn't used)
>>>>>>>>>>>>
>>>>>>>>>>>> We have another optimization one with SmartReadAhead but
>>>>>>>>>>>> for now
>>>>>>>>>>>> only work
>>>>>>>>>>>> for UI client. (for now) This is something we will
>>>>>>>>>>>> integrate as
>>>>>>>>>>>> well in CDO. But for now we concentrate on the other things.
>>>>>>>>>>>>
>>>>>>>>>>>> I used an Object database. Based on my preliminary test
>>>>>>>>>>>> without CDO.
>>>>>>>>>>>> I compare
>>>>>>>>>>>>
>>>>>>>>>>>> Hibernate vs Objectivity/EMF :
>>>>>>>>>>>> Objectivity/EMF is around 3 to 50 times faster than Hibernate.
>>>>>>>>>>>>
>>>>>>>>>>>> The worst.. you will get 30 to 240 objects/sec or
>>>>>>>>>>>> the best... you will get 500 to 4000 objects/sec
>>>>>>>>>>>>
>>>>>>>>>>>> Based on your comments... you are getting the worst (with
>>>>>>>>>>>> Derby)!!
>>>>>>>>>>>> :-)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Another comments about insertion.
>>>>>>>>>>>>
>>>>>>>>>>>> We found another problem when we insert data(DOn't know if
>>>>>>>>>>>> it is
>>>>>>>>>>>> still true). Resource.Contents is a unique list. Basically
>>>>>>>>>>>> each time
>>>>>>>>>>>> we insert one element, it needs to fetch all the collection
>>>>>>>>>>>> to see
>>>>>>>>>>>> if we already have this element. Basically.. it is time
>>>>>>>>>>>> consuming.
>>>>>>>>>>>> In our case we are not going through this stage to persist our
>>>>>>>>>>>> objects... but the objects are not accessible through that
>>>>>>>>>>>> list.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> If speed is an issue for you!! We (You included :-)) can
>>>>>>>>>>>> optimize
>>>>>>>>>>>> it. What I like is Eike did a really good job doing that
>>>>>>>>>>>> framwork
>>>>>>>>>>>> and optimizations are easy to add. Also you can change your
>>>>>>>>>>>> back-end
>>>>>>>>>>>> without affecting you client side. (Maybe going to an Object
>>>>>>>>>>>> database!!) THis was one of the purpose of CDO!! :-)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Simon
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le
>>>>>>>>>>>> message de
>>>>>>>>>>>> news: 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>>>>>>>>>
>>>>>>>>>>>> That sounds interesting!
>>>>>>>>>>>> Comments inline...
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Cédric Brun schrieb:
>>>>>>>>>>>> Hi Eike,
>>>>>>>>>>>>
>>>>>>>>>>>> I've been messing around CDO lately and I hacked a few ugly
>>>>>>>>>>>> unit-tests measuring read/write throughput when I use CDO
>>>>>>>>>>>> (RC2).
>>>>>>>>>>>>
>>>>>>>>>>>> I'm using an embeded-derby database, two different metamodels
>>>>>>>>>>>> (complex one - 250 EClassifiers/300 features and a simple
>>>>>>>>>>>> one - 1
>>>>>>>>>>>> Classifier/3 Features)
>>>>>>>>>>>>
>>>>>>>>>>>> The test is a "quick and dirty" one so please do not take
>>>>>>>>>>>> these
>>>>>>>>>>>> figures too seriously, I would just like to have your input
>>>>>>>>>>>> about
>>>>>>>>>>>> the "order of magnitude" I should expect or settings affecting
>>>>>>>>>>>> seriously these performances. I tried both JVM and TCP
>>>>>>>>>>>> acceptor but
>>>>>>>>>>>> the performances are quite similar though JVM is a bit faster.
>>>>>>>>>>>> Looks like a sign that the NIO socket channels are
>>>>>>>>>>>> efficient ,-)
>>>>>>>>>>>> You used localhost for TCP transport I guess?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> * First case : building a small model (10000 elements) with
>>>>>>>>>>>> the
>>>>>>>>>>>> simple MM, only adding instances in an empty database :
>>>>>>>>>>>> commiting change on every add I get a 2 objects per second
>>>>>>>>>>>> rate.
>>>>>>>>>>>> commiting changes every 10 adds - I get a 30 objects per
>>>>>>>>>>>> second
>>>>>>>>>>>> rate.
>>>>>>>>>>>> Seems to be consistent with the fact that only for
>>>>>>>>>>>> commits there
>>>>>>>>>>>> is remote traffic. And the backend transaction management is
>>>>>>>>>>>> another overhead. So the impact of the number objects/commit
>>>>>>>>>>>> starts with bigger transactions.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> browsing the just commited elements opening a new
>>>>>>>>>>>> transaction and
>>>>>>>>>>>> getting a new Resource instance, I get around 30 000
>>>>>>>>>>>> objets/sec rate
>>>>>>>>>>>> just iterating thanks to Resource.getAllContents().
>>>>>>>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>>>>>>>> All views and transactions of a single session share (and
>>>>>>>>>>>> cache)
>>>>>>>>>>>> their object state. Otherwise I'd expect much smaller
>>>>>>>>>>>> values.
>>>>>>>>>>>> With his ObjectivityStore implementation Simon (cc'ed)
>>>>>>>>>>>> reached
>>>>>>>>>>>> rates of several thousands objects/sec. But the DBStore
>>>>>>>>>>>> (which you
>>>>>>>>>>>> used) is not that optimized.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> * Second case : I built a big model in the database using the
>>>>>>>>>>>> complex metamodel. It contains 700 000 elements. The
>>>>>>>>>>>> creation went
>>>>>>>>>>>> pretty nicely (I did not measured it exactly as it was
>>>>>>>>>>>> launched
>>>>>>>>>>>> during a night, but it seems I got around 100 objects/sec
>>>>>>>>>>>> rate in
>>>>>>>>>>>> writting as I was writting every 700 objects.
>>>>>>>>>>>> Sounds not too bad.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Once the model is built the derby database is 222 MB (which
>>>>>>>>>>>> - I
>>>>>>>>>>>> heard - is not so big for derby).
>>>>>>>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>>>>>>>> To be honest, I'm really no database expert (although I
>>>>>>>>>>>> can write
>>>>>>>>>>>> simple SQL statements).
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Retrieving the distant resource and calling eAllContents()
>>>>>>>>>>>> on it, I
>>>>>>>>>>>> get a 50 Objects/sec rate at best, which mean my
>>>>>>>>>>>> eAllContents() call
>>>>>>>>>>>> would take 4 hours !
>>>>>>>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>>>>>>>> (iteration order and so). There are several CDO features
>>>>>>>>>>>> that can
>>>>>>>>>>>> have a major impact on load rates!
>>>>>>>>>>>>
>>>>>>>>>>>> - Collections (xrefs and containment) can be loaded
>>>>>>>>>>>> lazily. I
>>>>>>>>>>>> don't mean the target objects, which are lazy by default
>>>>>>>>>>>> anyway,
>>>>>>>>>>>> but even the target IDs that are in the collections. We
>>>>>>>>>>>> call that
>>>>>>>>>>>> "reference chunking" (but search for a better name is
>>>>>>>>>>>> ongoing).
>>>>>>>>>>>>
>>>>>>>>>>>> public interface CDOSession extends
>>>>>>>>>>>> CDOProtocolSession,
>>>>>>>>>>>> IContainer<CDOView>
>>>>>>>>>>>> {
>>>>>>>>>>>> public int getReferenceChunkSize();
>>>>>>>>>>>>
>>>>>>>>>>>> public void setReferenceChunkSize(int
>>>>>>>>>>>> referenceChunkSize);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> - Loading collection elements (target objects) can be
>>>>>>>>>>>> batched (the
>>>>>>>>>>>> current name is even worse)
>>>>>>>>>>>>
>>>>>>>>>>>> public interface CDOView extends CDOProtocolView,
>>>>>>>>>>>> INotifier
>>>>>>>>>>>> {
>>>>>>>>>>>> public int getLoadRevisionCollectionChunkSize();
>>>>>>>>>>>>
>>>>>>>>>>>> public void setLoadRevisionCollectionChunkSize(int
>>>>>>>>>>>> loadRevisionCollectionChunkSize);
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> - One of the most sophisticated feature is automatic
>>>>>>>>>>>> model usage
>>>>>>>>>>>> analyzation. The result are fetch rules that are sent to the
>>>>>>>>>>>> server. When later objects are loaded from the server, it
>>>>>>>>>>>> can
>>>>>>>>>>>> answer with additional objects that are likely to be
>>>>>>>>>>>> needed by the
>>>>>>>>>>>> client in the near future.
>>>>>>>>>>>>
>>>>>>>>>>>> public class CDOViewImpl extends
>>>>>>>>>>>> org.eclipse.net4j.internal.util.event.Notifier
>>>>>>>>>>>> implements
>>>>>>>>>>>> CDOView, CDOIDProvider, Adapter.Internal
>>>>>>>>>>>> {
>>>>>>>>>>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>>>>>>>>> {
>>>>>>>>>>>> return featureAnalyzer;
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>>>>>>>>> featureAnalyzer)
>>>>>>>>>>>> {
>>>>>>>>>>>> this.featureAnalyzer = featureAnalyzer == null ?
>>>>>>>>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>>>>>>>>> }
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> - There's a SmartReadAheadThread (if you're curious,
>>>>>>>>>>>> please ask
>>>>>>>>>>>> Simon). - Caches can be configured
>>>>>>>>>>>> - ...
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Known issues of the DBStore:
>>>>>>>>>>>> - When in auditing mode updates are always written as
>>>>>>>>>>>> complete
>>>>>>>>>>>> inserts (by design) - When configuring toManyReferences
>>>>>>>>>>>> to a value
>>>>>>>>>>>> different from LIKE_ATTRIBUTES (which is not implemented
>>>>>>>>>>>> yet) it
>>>>>>>>>>>> is very slow. - No (few) prepared statements are used (see
>>>>>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487) -
>>>>>>>>>>>> Maybe
>>>>>>>>>>>> others...
>>>>>>>>>>>>
>>>>>>>>>>>> Please note that the DBStore was not written by me to
>>>>>>>>>>>> compete with
>>>>>>>>>>>> Hibernate in terms of performance or mapping
>>>>>>>>>>>> capabilities. Now
>>>>>>>>>>>> that Martin and I, well in the end mostly Martin, provide a
>>>>>>>>>>>> HibernateStore and this HibernateStore gets more and more
>>>>>>>>>>>> features
>>>>>>>>>>>> I expect that there will be a trend away from the DBStore.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This is probably linked to the way the mapping to db is
>>>>>>>>>>>> done, here's
>>>>>>>>>>>> the config I used (came from CDO-server example if I recall
>>>>>>>>>>>> correctly).
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> <repository name="repo1">
>>>>>>>>>>>> <property name="overrideUUID"
>>>>>>>>>>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>>>>>>>>>>> <property name="supportingAudits"
>>>>>>>>>>>> value="true"/>
>>>>>>>>>>>> <property name="verifyingRevisions"
>>>>>>>>>>>> value="false"/>
>>>>>>>>>>>> <property name="currentLRUCapacity"
>>>>>>>>>>>> value="10000"/>
>>>>>>>>>>>> I guess this one could be larger.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> <property name="revisedLRUCapacity"
>>>>>>>>>>>> value="100"/>
>>>>>>>>>>>> <store type="db">
>>>>>>>>>>>> <!-- type: horizontal | vertical |
>>>>>>>>>>>> <any
>>>>>>>>>>>> user-contributed type> -->
>>>>>>>>>>>> <mappingStrategy
>>>>>>>>>>>> type="horizontal">
>>>>>>>>>>>> <!-- ONE_TABLE_PER_REFERENCE |
>>>>>>>>>>>> ONE_TABLE_PER_CLASS |
>>>>>>>>>>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY |
>>>>>>>>>>>> LIKE_ATTRIBUTES
>>>>>>>>>>>> -->
>>>>>>>>>>>> <property
>>>>>>>>>>>> name="toManyReferences"
>>>>>>>>>>>>
>>>>>>>>>>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>>>>>>>>>>
>>>>>>>>>>>> <!-- LIKE_ATTRIBUTES |
>>>>>>>>>>>> LIKE_TO_MANY_REFERENCES -->
>>>>>>>>>>>> <property
>>>>>>>>>>>> name="toOneReferences"
>>>>>>>>>>>> value="LIKE_ATTRIBUTES"/>
>>>>>>>>>>>>
>>>>>>>>>>>> <!-- MODEL | STRATEGY -->
>>>>>>>>>>>> <property
>>>>>>>>>>>> name="mappingPrecedence"
>>>>>


Re: [CDO] reading/writting throughput performances (SUCCESS) [message #619131 is a reply to message #124561] Sun, 01 June 2008 13:05 Go to previous message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
This is a multi-part message in MIME format.
--------------080000030302050701060709
Content-Type: multipart/alternative;
boundary="------------070501090100080408030009"


--------------070501090100080408030009
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit

Martin,

In the console log snippet below, please notice that it is the first
commit transaction signal and that there are no new packages contained.
This is caused by the client (test case) that didn't register the used
package with the client session before using it. The following snippet
uses the new automatic registry:

| *private *CDOSession createSession()
{
IConnector connector = (IConnector)IPluginContainer.INSTANCE.getElement("org.eclipse.net4j.connectors ", "jvm", "default");
System.out.println(connector);

CDOSessionConfiguration configuration = CDOUtil.createSessionConfiguration();
configuration.setConnector(connector);
configuration.setRepositoryName(REPO_NAME);
configuration.setDemandPopulatingPackageRegistry();
*return *configuration.openSession();
}|


Now the test cases pass properly!!!
Please see the attached log file.

Cheers
/Eike



Eike Stepper schrieb:
> Hi Martin,
>
> Ah, now with the new mapping I get a big step further!
> As I said, Cedric's test code contained some bugs, too. IIRC a TCP
> acceptor was created but the client uses a JVMConnector, and stuff
> like this.
> You could try to overcome the exception below by adding the following
> to the server code somewhere:
>
> | IPluginContainer.INSTANCE.getElement("org.eclipse.net4j.acceptors ", "jvm", "default");|
>
>
>
> Better just use the attached zip because there may be other changes
> that I don't remember in detail anymore.
> Now I run into another exception:
>
> Thread-4 [debug.signal] ================ Indicating
> CommitTransactionIndication
> Thread-4 [debug] Created
> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Writer for
> repository repo1
> Thread-4 [debug] Created
> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Writer for
> repository repo1
> Thread-4 [debug] Created
> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Writer for
> repository repo1
> Thread-4 [debug.protocol] Reading 0 new packages
> Thread-4 [debug.protocol] Reading 2 new objects
> Thread-4 [debug.model] Reading CDOID of type 3 (TEMP_OBJECT)
> Thread-4 [debug.model] Reading CDOID of type 3 (TEMP_OBJECT)
> Thread-4 [debug.model] Reading CDOID of type 0 (NULL)
> Thread-4 [debug.revision] Reading revision: ID=oid1,
> classRef=CDOClassRef(http://www.eclipse.org/emf/CDO/resource/1.0.0,
> 0), className=CDOResource, version=1, created=0, revised=0,
> resource=oid1, container=NULL, feature=0
> Thread-4 [debug.revision] Read feature CDOFeature(ID=9, name=path,
> type=STRING, referenceType=null): /res2
> Thread-4 [debug.revision] Read feature CDOFeature(ID=2, name=contents,
> type=OBJECT, referenceType=CDOClass(ID=0, name=CDOObject)): size=1
> Thread-4 [debug.model] Reading CDOID of type 3 (TEMP_OBJECT)
> Thread-4 [debug.revision] oid2
> [ERROR] ClassRef unresolveable:
> CDOClassRef(http://www.eclipse.org/modeling/cdo/bug/names, 0)
> java.lang.IllegalStateException: ClassRef unresolveable:
> CDOClassRef(http://www.eclipse.org/modeling/cdo/bug/names, 0)
> at
> org.eclipse.emf.cdo.internal.protocol.revision.CDORevisionIm pl. <init>(CDORevisionImpl.java:116)
> at
> org.eclipse.emf.cdo.protocol.revision.CDORevisionUtil.read(C DORevisionUtil.java:48)
> at
> org.eclipse.emf.cdo.internal.server.protocol.CommitTransacti onIndication.indicating(CommitTransactionIndication.java:95)
> at
> org.eclipse.net4j.signal.IndicationWithResponse.execute(Indi cationWithResponse.java:46)
> at org.eclipse.net4j.signal.Signal.runSync(Signal.java:143)
> at org.eclipse.net4j.signal.Signal.run(Signal.java:124)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Threa dPoolExecutor.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo lExecutor.java:675)
> at java.lang.Thread.run(Thread.java:595)
>
> Cheers
> /Eike
>
>
>
> Martin Taal schrieb:
>> Hi Eike, Cedric,
>> I can't say to much about net4j.db versus net4j.db.hsqldb.
>>
>> You are correct that the hibernate.libraries plugin should have
>> Eclipse-BuddyPolicy: dependent (or another way to use the classpath
>> of the dependent plugins). Btw, I agree that registered is better as
>> afaik dependent gets slow which larger plugin environments.
>>
>> The reason that cglib is used is because one of the class mappings in
>> org.eclipse.emf.cdo.server.hibernate/mappings/meta.hbm/xml did not
>> have the lazy="false" directive. I added this and now I get another
>> exception (see below). So this 'solves' the problem. If you do a get
>> latest of this file then I think this problem is 'overcome'.
>>
>> Overall I still do not see why it went wrong. As far as I can see the
>> cglib library could not find the hibernate class, which is strange as
>> they are in the same plugin??
>> The cdo testcases don't have this problem and neither does Teneo.
>>
>> As it does not occur anymore we can ignore it for now (not very
>> satisfactory but there are other things to do)... Unless someone else
>> has an idea how to handle this ofcourse.
>>
>> gr. Martin
>>
>> java.lang.IllegalStateException: acceptor == null
>> at
>> org.eclipse.net4j.internal.jvm.JVMClientConnector.doBeforeAc tivate(JVMClientConnector.java:57)
>>
>> at
>> org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:56)
>>
>> at
>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>>
>> at
>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>>
>> at
>> org.eclipse.net4j.internal.util.container.ManagedContainer.g etElement(ManagedContainer.java:260)
>>
>> at
>> org.eclipse.emf.internal.cdo.util.ChannelInjector.getConnect or(ChannelInjector.java:74)
>>
>> at
>> org.eclipse.emf.internal.cdo.util.ChannelInjector.process(Ch annelInjector.java:41)
>>
>> at
>> org.eclipse.net4j.internal.util.container.ManagedContainer.p ostProcessElement(ManagedContainer.java:407)
>>
>> at
>> org.eclipse.net4j.internal.util.container.ManagedContainer.g etElement(ManagedContainer.java:259)
>>
>> at
>> org.eclipse.cdo.bug.hibernate.HibernateServer.createSession( HibernateServer.java:93)
>>
>> at
>> org.eclipse.cdo.bug.unit.LaunchServerTest.testStringValue(La unchServerTest.java:43)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>
>> at java.lang.reflect.Method.invoke(Method.java:585)
>> at junit.framework.TestCase.runTest(TestCase.java:164)
>> at junit.framework.TestCase.runBare(TestCase.java:130)
>> at junit.framework.TestResult$1.protect(TestResult.java:106)
>> at junit.framework.TestResult.runProtected(TestResult.java:124)
>> at junit.framework.TestResult.run(TestResult.java:109)
>> at junit.framework.TestCase.run(TestCase.java:120)
>> at junit.framework.TestSuite.runTest(TestSuite.java:230)
>> at junit.framework.TestSuite.run(TestSuite.java:225)
>> at
>> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
>>
>> at
>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
>>
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
>>
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
>>
>> at
>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
>>
>> at
>> org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
>>
>> at
>> org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>
>> at java.lang.reflect.Method.invoke(Method.java:585)
>> at
>> org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hod(EclipseAppContainer.java:547)
>>
>> at
>> org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
>>
>> at
>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:106)
>>
>> at
>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:76)
>>
>> at
>> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:362)
>>
>> at
>> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:175)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>
>> at java.lang.reflect.Method.invoke(Method.java:585)
>> at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 564)
>> at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
>> at org.eclipse.equinox.launcher.Main.run(Main.java:1251)
>> at org.eclipse.equinox.launcher.Main.main(Main.java:1227)
>>
>>
>>
>> Eike Stepper wrote:
>>> Hi,
>>>
>>> Here is a diagram of the dependencies of the relevant plugins
>>> (please correct me if something is wrong).
>>> The black arrows are ordinary downstream dependencies (Require-Bundle).
>>> The red arrows are upstream dependencies (Eclipse-BuddyPolicy:
>>> dependent).
>>>
>>>
>>>
>>> For research purposes I added code to the libraries plugin that
>>> tries to reflectively load CDOFeatureImpl and that is not a problem!
>>> It seems to me as if our problem is not class loading related
>>> because the hibernate.jar should be able to see the needed classes.
>>> I believe this is either a bug in Hibernate or our wrong usage of
>>> Hibernate. I tend to believe the latter ;-)
>>> I put a breakpoint in
>>> org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping .EntityEntityModeToTuplizerMapping(PersistentClass,
>>> EntityMetamodel)
>>> to see how the tuplizers are created. I'm too unexperienced with
>>> Hibernate to get a clue about what happens there...
>>>
>>> Btw. I'd prefer to see the application plugin depend on net4j.db
>>> instead of net4j.db.hsqldb.
>>> But it seems as if the BuddyPolicy in net4j.db is not "reexported"
>>> so that the application can see the hsqldb.jar then ;-(
>>> I posted a respective question to equinox...
>>>
>>> Cheers
>>> /Eike
>>>
>>>
>>> Eike Stepper schrieb:
>>>> Hi guys,
>>>>
>>>> I can confirm, making these 3rd party libraries with their own
>>>> classloading run in OSGi is a nightmare! I took me hours teach
>>>> where the hsqldb driver is. On the way I came over different kinds
>>>> of classloading related errors.
>>>>
>>>> First I removed all the jars from the test plugin (the one Cedric
>>>> provided). They're already in the cdo.server.hibernate.libraries
>>>> plugin which, in fact, needs the buddy policy set to dependent. I
>>>> think this last change needs to be committed to CVS! Btw. I'd
>>>> prefer to use *registered* buddies instead of making all dependent
>>>> bundles visible.
>>>>
>>>> The test plugin then needs to provide the database driver.
>>>> Therefore I added a dependency on the org.eclipse.net4j.db.hsqldb
>>>> plugin.
>>>>
>>>> With this setup I can start the server without CNFE, NCDFE, CCI and
>>>> the like. At this point I get the following exception, which seems
>>>> the same as the one Cedric observed in the end:
>>>>
>>>> net.sf.cglib.core.CodeGenerationException:
>>>> java.lang.reflect.InvocationTargetException-->null
>>>> at
>>>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:237)
>>>>
>>>> at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
>>>> at net.sf.cglib.proxy.Enhancer.createClass(Enhancer.java:317)
>>>> at
>>>> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:127)
>>>>
>>>> at
>>>> org.hibernate.proxy.pojo.cglib.CGLIBProxyFactory.postInstant iate(CGLIBProxyFactory.java:43)
>>>>
>>>> at
>>>> org.hibernate.tuple.entity.PojoEntityTuplizer.buildProxyFact ory(PojoEntityTuplizer.java:162)
>>>>
>>>> at
>>>> org.hibernate.tuple.entity.AbstractEntityTuplizer.<init>(AbstractEntityTuplizer.java:135)
>>>>
>>>> at
>>>> org.hibernate.tuple.entity.PojoEntityTuplizer.<init>(PojoEntityTuplizer.java:55)
>>>>
>>>> at
>>>> org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping . <init>(EntityEntityModeToTuplizerMapping.java:56)
>>>>
>>>> at
>>>> org.hibernate.tuple.entity.EntityMetamodel.<init>(EntityMetamodel.java:295)
>>>>
>>>> at
>>>> org.hibernate.persister.entity.AbstractEntityPersister.<init >(AbstractEntityPersister.java:434)
>>>>
>>>> at
>>>> org.hibernate.persister.entity.JoinedSubclassEntityPersister . <init>(JoinedSubclassEntityPersister.java:91)
>>>>
>>>> at
>>>> org.hibernate.persister.PersisterFactory.createClassPersiste r(PersisterFactory.java:58)
>>>>
>>>> at
>>>> org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:226)
>>>>
>>>> at
>>>> org.hibernate.cfg.Configuration.buildSessionFactory(Configur ation.java:1294)
>>>>
>>>> at
>>>> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getSessionFactory(HibernatePackageHandler.java:335 )
>>>>
>>>> at
>>>> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.readPackageInfos(HibernatePackageHandler.java:293)
>>>>
>>>> at
>>>> org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getCDOPackageInfos(HibernatePackageHandler.java:21 5)
>>>>
>>>> at
>>>> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Reader.readPackageInfos(HibernateStoreReader.java:87)
>>>>
>>>> at
>>>> org.eclipse.emf.cdo.internal.server.PackageManager.doActivat e(PackageManager.java:72)
>>>>
>>>> at
>>>> org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
>>>>
>>>> at
>>>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>>>>
>>>> at
>>>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>>>>
>>>> at
>>>> org.eclipse.emf.cdo.internal.server.Repository.activateRepos itory(Repository.java:264)
>>>>
>>>> at
>>>> org.eclipse.emf.cdo.internal.server.Repository.doActivate(Re pository.java:251)
>>>>
>>>> at
>>>> org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
>>>>
>>>> at
>>>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>>>>
>>>> at
>>>> org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>>>>
>>>> at
>>>> org.eclipse.emf.cdo.server.CDOServerUtil.addRepository(CDOSe rverUtil.java:67)
>>>>
>>>> at
>>>> org.eclipse.cdo.bug.LaunchServerTest.setUp(LaunchServerTest. java:54)
>>>> at junit.framework.TestCase.runBare(TestCase.java:128)
>>>> at junit.framework.TestResult$1.protect(TestResult.java:106)
>>>> at junit.framework.TestResult.runProtected(TestResult.java:124)
>>>> at junit.framework.TestResult.run(TestResult.java:109)
>>>> at junit.framework.TestCase.run(TestCase.java:120)
>>>> at junit.framework.TestSuite.runTest(TestSuite.java:230)
>>>> at junit.framework.TestSuite.run(TestSuite.java:225)
>>>> at
>>>> org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
>>>>
>>>> at
>>>> org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
>>>>
>>>> at
>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
>>>>
>>>> at
>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
>>>>
>>>> at
>>>> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
>>>>
>>>> at
>>>> org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
>>>>
>>>> at
>>>> org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
>>>>
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>>
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>> at
>>>> org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hodWithException(EclipseAppContainer.java:574)
>>>>
>>>> at
>>>> org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
>>>>
>>>> at
>>>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:110)
>>>>
>>>> at
>>>> org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:79)
>>>>
>>>> at
>>>> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:379)
>>>>
>>>> at
>>>> org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:179)
>>>>
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>>
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>> at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 549)
>>>> at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
>>>> at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
>>>> at org.eclipse.equinox.launcher.Main.main(Main.java:1212)
>>>> Caused by: java.lang.reflect.InvocationTargetException
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>>
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>>
>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>> at
>>>> net.sf.cglib.core.ReflectUtils.defineClass(ReflectUtils.java :384)
>>>> at
>>>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>>>
>>>> ... 61 more
>>>> Caused by: java.lang.NoClassDefFoundError:
>>>> org/hibernate/proxy/HibernateProxy
>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>>>> ... 67 more
>>>>
>>>> I must admit that I'm still not sure about te fundamental interwork
>>>> between Hibernate and the user defined meta model. It seems as if
>>>> asm and cglib are somewhere in between but I have no idea why at
>>>> all. I thought we have implemented a bunch of static Hibernate
>>>> interfaces to make the CDORevision instances known to Hibernate.
>>>> Please somebody explain why we need runtime byte code engineering?
>>>> Is there some configuration missing to tell Hibernate to use our
>>>> special tuplizers?
>>>>
>>>> I have changed several things in the code of the test plugin and
>>>> since it's quite small now that the jars are removed, I attach my
>>>> version of the zip here (including a JUnit Plugin launch config).
>>>> And don't forget to update the cdo.server.hibernate.libraries
>>>> plugin from CVS to get the new buddy policy!
>>>>
>>>> Cheers
>>>> /Eike
>>>>
>>>> Martin Taal schrieb:
>>>>> Hi Cédric,
>>>>> I looked at it for many hours trying different things (using your
>>>>> testcase) but I am kind of stuck with this (getting the same error
>>>>> as you). The difference I can see with the way I run the
>>>>> cdo.hibernate testcases (as a junit test) and how your testcase is
>>>>> run (junit plugin test). This for sure means difference in class
>>>>> loader behavior.
>>>>>
>>>>> Eike, do you have an idea? I am not sure how classloading works
>>>>> when you start a cdo server in a plugin. Which plugins are then
>>>>> available in the cdo server? The testcases can be downloaded from
>>>>> the link below.
>>>>>
>>>>> gr. Martin
>>>>>
>>>>> Cédric Brun wrote:
>>>>>> Thanks a lot ! I'm looking forward to your advices !
>>>>>>
>>>>>> Cédric
>>>>>>
>>>>>> Martin Taal wrote:
>>>>>>
>>>>>>> I am downloading your zip-file. I don't have time to look at it
>>>>>>> right away
>>>>>>> but I will do my best later this evening.
>>>>>>>
>>>>>>> gr. Martin
>>>>>>>
>>>>>>> Cédric Brun wrote:
>>>>>>>> I tried to do the same things as in the CDO/Hibernate unit tests.
>>>>>>>>
>>>>>>>> Here's a zipped project trying to launch an HibernateServer
>>>>>>>> with a unit
>>>>>>>> test adding elements.
>>>>>>>> ( http://cedric.brun.free.fr/Tortoose/org.eclipse.cdo-teneo-te st.zip)
>>>>>>>> The
>>>>>>>> libs I found are included in the same project, but I still have
>>>>>>>> no luck
>>>>>>>> and failling at :
>>>>>>>>
>>>>>>>> at
>>>>>>>>
>>>>>> net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>>>>>
>>>>>>>> ... 62 more
>>>>>>>> Caused by: java.lang.NoClassDefFoundError:
>>>>>>>> org/hibernate/proxy/HibernateProxy
>>>>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>>>>>>>> ... 68 more
>>>>>>>>
>>>>>>>> please note that the test project I provide really is a "quick
>>>>>>>> and ugly"
>>>>>>>> one ^^
>>>>>>>>
>>>>>>>>
>>>>>>>> Cédric
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Martin Taal wrote:
>>>>>>>>
>>>>>>>>> Hmm, I can understand your headache. Classpath issues are the
>>>>>>>>> most
>>>>>>>>> annoying errors...
>>>>>>>>>
>>>>>>>>> In your junit test source file, what happens when you enter the
>>>>>>>>> org.objectweb.asm.Type, can you source file see this class?
>>>>>>>>> Do you have cglib in some other plugin which does not have asm?
>>>>>>>>>
>>>>>>>>> What I also find slightly strange is that cglib is used in
>>>>>>>>> this case
>>>>>>>>> because normally cdo objects are not proxied using cglib. Can
>>>>>>>>> you post
>>>>>>>>> the generated mapping? How does your cdo initialization code
>>>>>>>>> look like?
>>>>>>>>>
>>>>>>>>> gr. Martin
>>>>>>>>>
>>>>>>>>> Cédric Brun wrote:
>>>>>>>>>> Hi Martin,
>>>>>>>>>>
>>>>>>>>>> and thanks for your quick answer ! what I did is the
>>>>>>>>>> following :
>>>>>>>>>>
>>>>>>>>>> I used the org.eclipse.emf.cdo.server.hibernate.libraries
>>>>>>>>>> bundle and
>>>>>>>>>> added that :
>>>>>>>>>>
>>>>>>>>>> Bundle-ClassPath: lib/hibernate3.jar,
>>>>>>>>>> lib/cglib-2.1.3.jar,
>>>>>>>>>> lib/commons-collections-2.1.1.jar,
>>>>>>>>>> lib/dom4j.jar,
>>>>>>>>>> lib/jta.jar
>>>>>>>>>> lib/asm-3.0.jar
>>>>>>>>>>
>>>>>>>>>> and this directive :
>>>>>>>>>>
>>>>>>>>>> Eclipse-BuddyPolicy: dependent
>>>>>>>>>>
>>>>>>>>>> as stated in
>>>>>>>>>> http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Then I don't get the same error, now when I start my own
>>>>>>>>>> plugin (JUnit
>>>>>>>>>> plugin test) depending on
>>>>>>>>>> org.eclipse.emf.cdo.server.hibernate.libraries I get :
>>>>>>>>>>
>>>>>>>>>> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
>>>>>>>>>> at
>>>>>>>>>> net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
>>>>>>>>>> at
>>>>>>>>>> net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
>>>>>>>>>> at
>>>>>>>>>> net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>> org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>>>>>>
>>>>>>>>>> These jars are such a headache ! Do you know - by chance -
>>>>>>>>>> what's
>>>>>>>>>> going on ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Cédric
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Martin Taal wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Cédric,
>>>>>>>>>>> Most of the time this exception is thrown because hibernate
>>>>>>>>>>> can't find
>>>>>>>>>>> the class. A common cause is that the hibernate jar files
>>>>>>>>>>> are located
>>>>>>>>>>> in a separate plugin which can not reach the classes in
>>>>>>>>>>> other plugins.
>>>>>>>>>>> If you set the eclipse buddy policy of the hibernate plugin to
>>>>>>>>>>> dependent then this can help.
>>>>>>>>>>>
>>>>>>>>>>> gr. Martin
>>>>>>>>>>>
>>>>>>>>>>> Cédric Brun wrote:
>>>>>>>>>>>> Hi Simon and Eike,
>>>>>>>>>>>>
>>>>>>>>>>>> thanks for your feedback, now I know what I could expect in
>>>>>>>>>>>> using
>>>>>>>>>>>> optimized implementation, I'll have a try on the hibernate
>>>>>>>>>>>> store and
>>>>>>>>>>>> see if it is scaling in a better way.
>>>>>>>>>>>>
>>>>>>>>>>>> You gave me many pointers to explain the performances
>>>>>>>>>>>> issues but in
>>>>>>>>>>>> fact it seems that the SQL execution is taking so much time
>>>>>>>>>>>> (not
>>>>>>>>>>>> suprisingly in fact ;) . My first profile was wrong.
>>>>>>>>>>>>
>>>>>>>>>>>> So I'm installing the hibernate store retrieving all the
>>>>>>>>>>>> source code
>>>>>>>>>>>> from CVS, and I'm trying to launch the unit tests. I added
>>>>>>>>>>>> many jars
>>>>>>>>>>>> and everything gets compiled but most of the tests are
>>>>>>>>>>>> failling with
>>>>>>>>>>>> "
>>>>>>>>>>>>
>>>>>>>>>>>> "
>>>>>>>>>>>> org.hibernate.MappingException: Could not determine type for:
>>>>>>>>>>>>
>>>>>> org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>>>>>
>>>>>>>>>>>> at table: cdo_features, for columns:
>>>>>>>>>>>> [org.hibernate.mapping.Column(type)] "
>>>>>>>>>>>>
>>>>>>>>>>>> any hint ?
>>>>>>>>>>>>
>>>>>>>>>>>> Cédric
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Simon McDuff wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> We noticed a performance degradation in the following case:
>>>>>>>>>>>>>
>>>>>>>>>>>>> When we fetch a collection of objects that contains
>>>>>>>>>>>>> collections that
>>>>>>>>>>>>> we don't want to use. Ex:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Model:
>>>>>>>>>>>>>
>>>>>>>>>>>>> ClassB
>>>>>>>>>>>>> {
>>>>>>>>>>>>> int foo1;
>>>>>>>>>>>>> List childrens1;
>>>>>>>>>>>>> List childrens2;
>>>>>>>>>>>>> List childrens3;
>>>>>>>>>>>>> };
>>>>>>>>>>>>>
>>>>>>>>>>>>> Code:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Resource.contents contains objects of type ClassB.
>>>>>>>>>>>>>
>>>>>>>>>>>>> By fetching instances of classB, we will also fetched some
>>>>>>>>>>>>> items in
>>>>>>>>>>>>> the list(childrens1, childrens2, childrens3) The problem
>>>>>>>>>>>>> we had.. is
>>>>>>>>>>>>> we do not know if we want to use it. SO better not to
>>>>>>>>>>>>> download them.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So In our production(we used our own IStore), we never
>>>>>>>>>>>>> download any
>>>>>>>>>>>>> objects from collection. I'm working at the moment to put
>>>>>>>>>>>>> in place
>>>>>>>>>>>>> all feature we used in the current CDO. That one will be
>>>>>>>>>>>>> part of
>>>>>>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477 To
>>>>>>>>>>>>> begin
>>>>>>>>>>>>> with,CDORevisionImpl needs to supports list that are 0
>>>>>>>>>>>>> even if
>>>>>>>>>>>>> referenceChunkSize == 0.
>>>>>>>>>>>>>
>>>>>>>>>>>>> It is true that we obtain good performance.
>>>>>>>>>>>>> Worst case :
>>>>>>>>>>>>> 1500 objects /sec (Real fetch from disk. Page that
>>>>>>>>>>>>> contains the
>>>>>>>>>>>>> objects, wasn't in the cache of the filesystem.)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best case
>>>>>>>>>>>>> 12 000 objects/Sec (Fetch objects that are already in the
>>>>>>>>>>>>> case of
>>>>>>>>>>>>> the filesystem)
>>>>>>>>>>>>>
>>>>>>>>>>>>> (These statistics always fetch objects from server to
>>>>>>>>>>>>> client. The
>>>>>>>>>>>>> cache client isn't used)
>>>>>>>>>>>>>
>>>>>>>>>>>>> We have another optimization one with SmartReadAhead but
>>>>>>>>>>>>> for now
>>>>>>>>>>>>> only work
>>>>>>>>>>>>> for UI client. (for now) This is something we will
>>>>>>>>>>>>> integrate as
>>>>>>>>>>>>> well in CDO. But for now we concentrate on the other things.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I used an Object database. Based on my preliminary test
>>>>>>>>>>>>> without CDO.
>>>>>>>>>>>>> I compare
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hibernate vs Objectivity/EMF :
>>>>>>>>>>>>> Objectivity/EMF is around 3 to 50 times faster than
>>>>>>>>>>>>> Hibernate.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The worst.. you will get 30 to 240 objects/sec or
>>>>>>>>>>>>> the best... you will get 500 to 4000 objects/sec
>>>>>>>>>>>>>
>>>>>>>>>>>>> Based on your comments... you are getting the worst (with
>>>>>>>>>>>>> Derby)!!
>>>>>>>>>>>>> :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Another comments about insertion.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We found another problem when we insert data(DOn't know if
>>>>>>>>>>>>> it is
>>>>>>>>>>>>> still true). Resource.Contents is a unique list. Basically
>>>>>>>>>>>>> each time
>>>>>>>>>>>>> we insert one element, it needs to fetch all the
>>>>>>>>>>>>> collection to see
>>>>>>>>>>>>> if we already have this element. Basically.. it is time
>>>>>>>>>>>>> consuming.
>>>>>>>>>>>>> In our case we are not going through this stage to persist
>>>>>>>>>>>>> our
>>>>>>>>>>>>> objects... but the objects are not accessible through that
>>>>>>>>>>>>> list.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> If speed is an issue for you!! We (You included :-)) can
>>>>>>>>>>>>> optimize
>>>>>>>>>>>>> it. What I like is Eike did a really good job doing that
>>>>>>>>>>>>> framwork
>>>>>>>>>>>>> and optimizations are easy to add. Also you can change
>>>>>>>>>>>>> your back-end
>>>>>>>>>>>>> without affecting you client side. (Maybe going to an Object
>>>>>>>>>>>>> database!!) THis was one of the purpose of CDO!! :-)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le
>>>>>>>>>>>>> message de
>>>>>>>>>>>>> news: 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>>>>>>>>>>
>>>>>>>>>>>>> That sounds interesting!
>>>>>>>>>>>>> Comments inline...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cédric Brun schrieb:
>>>>>>>>>>>>> Hi Eike,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've been messing around CDO lately and I hacked a few ugly
>>>>>>>>>>>>> unit-tests measuring read/write throughput when I use CDO
>>>>>>>>>>>>> (RC2).
>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm using an embeded-derby database, two different metamodels
>>>>>>>>>>>>> (complex one - 250 EClassifiers/300 features and a simple
>>>>>>>>>>>>> one - 1
>>>>>>>>>>>>> Classifier/3 Features)
>>>>>>>>>>>>>
>>>>>>>>>>>>> The test is a "quick and dirty" one so please do not take
>>>>>>>>>>>>> these
>>>>>>>>>>>>> figures too seriously, I would just like to have your
>>>>>>>>>>>>> input about
>>>>>>>>>>>>> the "order of magnitude" I should expect or settings
>>>>>>>>>>>>> affecting
>>>>>>>>>>>>> seriously these performances. I tried both JVM and TCP
>>>>>>>>>>>>> acceptor but
>>>>>>>>>>>>> the performances are quite similar though JVM is a bit
>>>>>>>>>>>>> faster.
>>>>>>>>>>>>> Looks like a sign that the NIO socket channels are
>>>>>>>>>>>>> efficient ,-)
>>>>>>>>>>>>> You used localhost for TCP transport I guess?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> * First case : building a small model (10000 elements)
>>>>>>>>>>>>> with the
>>>>>>>>>>>>> simple MM, only adding instances in an empty database :
>>>>>>>>>>>>> commiting change on every add I get a 2 objects per second
>>>>>>>>>>>>> rate.
>>>>>>>>>>>>> commiting changes every 10 adds - I get a 30 objects per
>>>>>>>>>>>>> second
>>>>>>>>>>>>> rate.
>>>>>>>>>>>>> Seems to be consistent with the fact that only for
>>>>>>>>>>>>> commits there
>>>>>>>>>>>>> is remote traffic. And the backend transaction
>>>>>>>>>>>>> management is
>>>>>>>>>>>>> another overhead. So the impact of the number
>>>>>>>>>>>>> objects/commit
>>>>>>>>>>>>> starts with bigger transactions.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> browsing the just commited elements opening a new
>>>>>>>>>>>>> transaction and
>>>>>>>>>>>>> getting a new Resource instance, I get around 30 000
>>>>>>>>>>>>> objets/sec rate
>>>>>>>>>>>>> just iterating thanks to Resource.getAllContents().
>>>>>>>>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>>>>>>>>> All views and transactions of a single session share
>>>>>>>>>>>>> (and cache)
>>>>>>>>>>>>> their object state. Otherwise I'd expect much smaller
>>>>>>>>>>>>> values.
>>>>>>>>>>>>> With his ObjectivityStore implementation Simon (cc'ed)
>>>>>>>>>>>>> reached
>>>>>>>>>>>>> rates of several thousands objects/sec. But the DBStore
>>>>>>>>>>>>> (which you
>>>>>>>>>>>>> used) is not that optimized.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> * Second case : I built a big model in the database using the
>>>>>>>>>>>>> complex metamodel. It contains 700 000 elements. The
>>>>>>>>>>>>> creation went
>>>>>>>>>>>>> pretty nicely (I did not measured it exactly as it was
>>>>>>>>>>>>> launched
>>>>>>>>>>>>> during a night, but it seems I got around 100 objects/sec
>>>>>>>>>>>>> rate in
>>>>>>>>>>>>> writting as I was writting every 700 objects.
>>>>>>>>>>>>> Sounds not too bad.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Once the model is built the derby database is 222 MB
>>>>>>>>>>>>> (which - I
>>>>>>>>>>>>> heard - is not so big for derby).
>>>>>>>>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>>>>>>>>> To be honest, I'm really no database expert (although I
>>>>>>>>>>>>> can write
>>>>>>>>>>>>> simple SQL statements).
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Retrieving the distant resource and calling eAllContents()
>>>>>>>>>>>>> on it, I
>>>>>>>>>>>>> get a 50 Objects/sec rate at best, which mean my
>>>>>>>>>>>>> eAllContents() call
>>>>>>>>>>>>> would take 4 hours !
>>>>>>>>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>>>>>>>>> (iteration order and so). There are several CDO features
>>>>>>>>>>>>> that can
>>>>>>>>>>>>> have a major impact on load rates!
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Collections (xrefs and containment) can be loaded
>>>>>>>>>>>>> lazily. I
>>>>>>>>>>>>> don't mean the target objects, which are lazy by default
>>>>>>>>>>>>> anyway,
>>>>>>>>>>>>> but even the target IDs that are in the collections. We
>>>>>>>>>>>>> call that
>>>>>>>>>>>>> "reference chunking" (but search for a better name is
>>>>>>>>>>>>> ongoing).
>>>>>>>>>>>>>
>>>>>>>>>>>>> public interface CDOSession extends
>>>>>>>>>>>>> CDOProtocolSession,
>>>>>>>>>>>>> IContainer<CDOView>
>>>>>>>>>>>>> {
>>>>>>>>>>>>> public int getReferenceChunkSize();
>>>>>>>>>>>>>
>>>>>>>>>>>>> public void setReferenceChunkSize(int
>>>>>>>>>>>>> referenceChunkSize);
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> - Loading collection elements (target objects) can be
>>>>>>>>>>>>> batched (the
>>>>>>>>>>>>> current name is even worse)
>>>>>>>>>>>>>
>>>>>>>>>>>>> public interface CDOView extends CDOProtocolView,
>>>>>>>>>>>>> INotifier
>>>>>>>>>>>>> {
>>>>>>>>>>>>> public int getLoadRevisionCollectionChunkSize();
>>>>>>>>>>>>>
>>>>>>>>>>>>> public void setLoadRevisionCollectionChunkSize(int
>>>>>>>>>>>>> loadRevisionCollectionChunkSize);
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> - One of the most sophisticated feature is automatic
>>>>>>>>>>>>> model usage
>>>>>>>>>>>>> analyzation. The result are fetch rules that are sent to
>>>>>>>>>>>>> the
>>>>>>>>>>>>> server. When later objects are loaded from the server,
>>>>>>>>>>>>> it can
>>>>>>>>>>>>> answer with additional objects that are likely to be
>>>>>>>>>>>>> needed by the
>>>>>>>>>>>>> client in the near future.
>>>>>>>>>>>>>
>>>>>>>>>>>>> public class CDOViewImpl extends
>>>>>>>>>>>>> org.eclipse.net4j.internal.util.event.Notifier
>>>>>>>>>>>>> implements
>>>>>>>>>>>>> CDOView, CDOIDProvider, Adapter.Internal
>>>>>>>>>>>>> {
>>>>>>>>>>>>> public CDOFeatureAnalyzer getFeatureAnalyzer()
>>>>>>>>>>>>> {
>>>>>>>>>>>>> return featureAnalyzer;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>> public void setFeatureAnalyzer(CDOFeatureAnalyzer
>>>>>>>>>>>>> featureAnalyzer)
>>>>>>>>>>>>> {
>>>>>>>>>>>>> this.featureAnalyzer = featureAnalyzer == null ?
>>>>>>>>>>>>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>>>>>>>>>>>>> }
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> - There's a SmartReadAheadThread (if you're curious,
>>>>>>>>>>>>> please ask
>>>>>>>>>>>>> Simon). - Caches can be configured
>>>>>>>>>>>>> - ...
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Known issues of the DBStore:
>>>>>>>>>>>>> - When in auditing mode updates are always written as
>>>>>>>>>>>>> complete
>>>>>>>>>>>>> inserts (by design) - When configuring toManyReferences
>>>>>>>>>>>>> to a value
>>>>>>>>>>>>> different from LIKE_ATTRIBUTES (which is not implemented
>>>>>>>>>>>>> yet) it
>>>>>>>>>>>>> is very slow. - No (few) prepared statements are used (see
>>>>>>>>


Re: [CDO] reading/writting throughput performances (SUCCESS) [message #619144 is a reply to message #124736] Mon, 02 June 2008 12:26 Go to previous message
Cedric Brun is currently offline Cedric BrunFriend
Messages: 431
Registered: July 2009
Senior Member
Hi Eike and Martin,

Thanks a lot for your help, I don't think I would have managed to get
through this "jar hell" without it !

I'm now going to benchmark the hibernate store with the same test case than
the dbstore (complex metamodel - big model - eAllContents throughput) and
gives you the figures, it might be of an interest even if it's a stupid
test for "interaction based" software.

Cédric

Eike Stepper wrote:

> Martin,
>
> In the console log snippet below, please notice that it is the first
> commit transaction signal and that there are no new packages contained.
> This is caused by the client (test case) that didn't register the used
> package with the client session before using it. The following snippet
> uses the new automatic registry:
>
> | *private *CDOSession createSession()
> {
> IConnector connector =
>
(IConnector)IPluginContainer.INSTANCE.getElement("org.eclipse.net4j.connectors ",
> "jvm", "default"); System.out.println(connector);
>
> CDOSessionConfiguration configuration =
> CDOUtil.createSessionConfiguration();
> configuration.setConnector(connector);
> configuration.setRepositoryName(REPO_NAME);
> configuration.setDemandPopulatingPackageRegistry(); *return
> *configuration.openSession();
> }|
>
>
> Now the test cases pass properly!!!
> Please see the attached log file.
>
> Cheers
> /Eike
>
>
>
> Eike Stepper schrieb:
>> Hi Martin,
>>
>> Ah, now with the new mapping I get a big step further!
>> As I said, Cedric's test code contained some bugs, too. IIRC a TCP
>> acceptor was created but the client uses a JVMConnector, and stuff
>> like this.
>> You could try to overcome the exception below by adding the following
>> to the server code somewhere:
>>
>> | IPluginContainer.INSTANCE.getElement("org.eclipse.net4j.acceptors ",
>> | "jvm", "default");|
>>
>>
>>
>> Better just use the attached zip because there may be other changes
>> that I don't remember in detail anymore.
>> Now I run into another exception:
>>
>> Thread-4 [debug.signal] ================ Indicating
>> CommitTransactionIndication
>> Thread-4 [debug] Created
>> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Writer for
>> repository repo1
>> Thread-4 [debug] Created
>> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Writer for
>> repository repo1
>> Thread-4 [debug] Created
>> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Writer for
>> repository repo1
>> Thread-4 [debug.protocol] Reading 0 new packages
>> Thread-4 [debug.protocol] Reading 2 new objects
>> Thread-4 [debug.model] Reading CDOID of type 3 (TEMP_OBJECT)
>> Thread-4 [debug.model] Reading CDOID of type 3 (TEMP_OBJECT)
>> Thread-4 [debug.model] Reading CDOID of type 0 (NULL)
>> Thread-4 [debug.revision] Reading revision: ID=oid1,
>> classRef=CDOClassRef(http://www.eclipse.org/emf/CDO/resource/1.0.0,
>> 0), className=CDOResource, version=1, created=0, revised=0,
>> resource=oid1, container=NULL, feature=0
>> Thread-4 [debug.revision] Read feature CDOFeature(ID=9, name=path,
>> type=STRING, referenceType=null): /res2
>> Thread-4 [debug.revision] Read feature CDOFeature(ID=2, name=contents,
>> type=OBJECT, referenceType=CDOClass(ID=0, name=CDOObject)): size=1
>> Thread-4 [debug.model] Reading CDOID of type 3 (TEMP_OBJECT)
>> Thread-4 [debug.revision] oid2
>> [ERROR] ClassRef unresolveable:
>> CDOClassRef(http://www.eclipse.org/modeling/cdo/bug/names, 0)
>> java.lang.IllegalStateException: ClassRef unresolveable:
>> CDOClassRef(http://www.eclipse.org/modeling/cdo/bug/names, 0)
>> at
>>
org.eclipse.emf.cdo.internal.protocol.revision.CDORevisionIm pl. <init>(CDORevisionImpl.java:116)
>> at
>>
org.eclipse.emf.cdo.protocol.revision.CDORevisionUtil.read(C DORevisionUtil.java:48)
>> at
>>
org.eclipse.emf.cdo.internal.server.protocol.CommitTransacti onIndication.indicating(CommitTransactionIndication.java:95)
>> at
>>
org.eclipse.net4j.signal.IndicationWithResponse.execute(Indi cationWithResponse.java:46)
>> at org.eclipse.net4j.signal.Signal.runSync(Signal.java:143)
>> at org.eclipse.net4j.signal.Signal.run(Signal.java:124)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Threa dPoolExecutor.java:650)
>> at
>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo lExecutor.java:675)
>> at java.lang.Thread.run(Thread.java:595)
>>
>> Cheers
>> /Eike
>>
>>
>>
>> Martin Taal schrieb:
>>> Hi Eike, Cedric,
>>> I can't say to much about net4j.db versus net4j.db.hsqldb.
>>>
>>> You are correct that the hibernate.libraries plugin should have
>>> Eclipse-BuddyPolicy: dependent (or another way to use the classpath
>>> of the dependent plugins). Btw, I agree that registered is better as
>>> afaik dependent gets slow which larger plugin environments.
>>>
>>> The reason that cglib is used is because one of the class mappings in
>>> org.eclipse.emf.cdo.server.hibernate/mappings/meta.hbm/xml did not
>>> have the lazy="false" directive. I added this and now I get another
>>> exception (see below). So this 'solves' the problem. If you do a get
>>> latest of this file then I think this problem is 'overcome'.
>>>
>>> Overall I still do not see why it went wrong. As far as I can see the
>>> cglib library could not find the hibernate class, which is strange as
>>> they are in the same plugin??
>>> The cdo testcases don't have this problem and neither does Teneo.
>>>
>>> As it does not occur anymore we can ignore it for now (not very
>>> satisfactory but there are other things to do)... Unless someone else
>>> has an idea how to handle this ofcourse.
>>>
>>> gr. Martin
>>>
>>> java.lang.IllegalStateException: acceptor == null
>>> at
>>>
org.eclipse.net4j.internal.jvm.JVMClientConnector.doBeforeAc tivate(JVMClientConnector.java:57)
>>>
>>> at
>>>
org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:56)
>>>
>>> at
>>>
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>>>
>>> at
>>>
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>>>
>>> at
>>>
org.eclipse.net4j.internal.util.container.ManagedContainer.g etElement(ManagedContainer.java:260)
>>>
>>> at
>>>
org.eclipse.emf.internal.cdo.util.ChannelInjector.getConnect or(ChannelInjector.java:74)
>>>
>>> at
>>>
org.eclipse.emf.internal.cdo.util.ChannelInjector.process(Ch annelInjector.java:41)
>>>
>>> at
>>>
org.eclipse.net4j.internal.util.container.ManagedContainer.p ostProcessElement(ManagedContainer.java:407)
>>>
>>> at
>>>
org.eclipse.net4j.internal.util.container.ManagedContainer.g etElement(ManagedContainer.java:259)
>>>
>>> at
>>>
org.eclipse.cdo.bug.hibernate.HibernateServer.createSession( HibernateServer.java:93)
>>>
>>> at
>>>
org.eclipse.cdo.bug.unit.LaunchServerTest.testStringValue(La unchServerTest.java:43)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>
>>> at
>>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>> at junit.framework.TestCase.runTest(TestCase.java:164)
>>> at junit.framework.TestCase.runBare(TestCase.java:130)
>>> at junit.framework.TestResult$1.protect(TestResult.java:106)
>>> at junit.framework.TestResult.runProtected(TestResult.java:124)
>>> at junit.framework.TestResult.run(TestResult.java:109)
>>> at junit.framework.TestCase.run(TestCase.java:120)
>>> at junit.framework.TestSuite.runTest(TestSuite.java:230)
>>> at junit.framework.TestSuite.run(TestSuite.java:225)
>>> at
>>>
org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
>>>
>>> at
>>>
org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
>>>
>>> at
>>>
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
>>>
>>> at
>>>
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
>>>
>>> at
>>>
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
>>>
>>> at
>>>
org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
>>>
>>> at
>>>
org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>
>>> at
>>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>> at
>>>
org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hod(EclipseAppContainer.java:547)
>>>
>>> at
>>>
org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
>>>
>>> at
>>>
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:106)
>>>
>>> at
>>>
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:76)
>>>
>>> at
>>>
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:362)
>>>
>>> at
>>>
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:175)
>>>
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>
>>> at
>>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>> at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 564)
>>> at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
>>> at org.eclipse.equinox.launcher.Main.run(Main.java:1251)
>>> at org.eclipse.equinox.launcher.Main.main(Main.java:1227)
>>>
>>>
>>>
>>> Eike Stepper wrote:
>>>> Hi,
>>>>
>>>> Here is a diagram of the dependencies of the relevant plugins
>>>> (please correct me if something is wrong).
>>>> The black arrows are ordinary downstream dependencies (Require-Bundle).
>>>> The red arrows are upstream dependencies (Eclipse-BuddyPolicy:
>>>> dependent).
>>>>
>>>>
>>>>
>>>> For research purposes I added code to the libraries plugin that
>>>> tries to reflectively load CDOFeatureImpl and that is not a problem!
>>>> It seems to me as if our problem is not class loading related
>>>> because the hibernate.jar should be able to see the needed classes.
>>>> I believe this is either a bug in Hibernate or our wrong usage of
>>>> Hibernate. I tend to believe the latter ;-)
>>>> I put a breakpoint in
>>>>
org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping .EntityEntityModeToTuplizerMapping(PersistentClass,
>>>> EntityMetamodel)
>>>> to see how the tuplizers are created. I'm too unexperienced with
>>>> Hibernate to get a clue about what happens there...
>>>>
>>>> Btw. I'd prefer to see the application plugin depend on net4j.db
>>>> instead of net4j.db.hsqldb.
>>>> But it seems as if the BuddyPolicy in net4j.db is not "reexported"
>>>> so that the application can see the hsqldb.jar then ;-(
>>>> I posted a respective question to equinox...
>>>>
>>>> Cheers
>>>> /Eike
>>>>
>>>>
>>>> Eike Stepper schrieb:
>>>>> Hi guys,
>>>>>
>>>>> I can confirm, making these 3rd party libraries with their own
>>>>> classloading run in OSGi is a nightmare! I took me hours teach
>>>>> where the hsqldb driver is. On the way I came over different kinds
>>>>> of classloading related errors.
>>>>>
>>>>> First I removed all the jars from the test plugin (the one Cedric
>>>>> provided). They're already in the cdo.server.hibernate.libraries
>>>>> plugin which, in fact, needs the buddy policy set to dependent. I
>>>>> think this last change needs to be committed to CVS! Btw. I'd
>>>>> prefer to use *registered* buddies instead of making all dependent
>>>>> bundles visible.
>>>>>
>>>>> The test plugin then needs to provide the database driver.
>>>>> Therefore I added a dependency on the org.eclipse.net4j.db.hsqldb
>>>>> plugin.
>>>>>
>>>>> With this setup I can start the server without CNFE, NCDFE, CCI and
>>>>> the like. At this point I get the following exception, which seems
>>>>> the same as the one Cedric observed in the end:
>>>>>
>>>>> net.sf.cglib.core.CodeGenerationException:
>>>>> java.lang.reflect.InvocationTargetException-->null
>>>>> at
>>>>>
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:237)
>>>>>
>>>>> at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
>>>>> at net.sf.cglib.proxy.Enhancer.createClass(Enhancer.java:317)
>>>>> at
>>>>>
org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:127)
>>>>>
>>>>> at
>>>>>
org.hibernate.proxy.pojo.cglib.CGLIBProxyFactory.postInstant iate(CGLIBProxyFactory.java:43)
>>>>>
>>>>> at
>>>>>
org.hibernate.tuple.entity.PojoEntityTuplizer.buildProxyFact ory(PojoEntityTuplizer.java:162)
>>>>>
>>>>> at
>>>>>
org.hibernate.tuple.entity.AbstractEntityTuplizer.<init>(AbstractEntityTuplizer.java:135)
>>>>>
>>>>> at
>>>>>
org.hibernate.tuple.entity.PojoEntityTuplizer.<init>(PojoEntityTuplizer.java:55)
>>>>>
>>>>> at
>>>>>
org.hibernate.tuple.entity.EntityEntityModeToTuplizerMapping . <init>(EntityEntityModeToTuplizerMapping.java:56)
>>>>>
>>>>> at
>>>>>
org.hibernate.tuple.entity.EntityMetamodel.<init>(EntityMetamodel.java:295)
>>>>>
>>>>> at
>>>>>
org.hibernate.persister.entity.AbstractEntityPersister.<init >(AbstractEntityPersister.java:434)
>>>>>
>>>>> at
>>>>>
org.hibernate.persister.entity.JoinedSubclassEntityPersister . <init>(JoinedSubclassEntityPersister.java:91)
>>>>>
>>>>> at
>>>>>
org.hibernate.persister.PersisterFactory.createClassPersiste r(PersisterFactory.java:58)
>>>>>
>>>>> at
>>>>>
org.hibernate.impl.SessionFactoryImpl.<init>(SessionFactoryImpl.java:226)
>>>>>
>>>>> at
>>>>>
org.hibernate.cfg.Configuration.buildSessionFactory(Configur ation.java:1294)
>>>>>
>>>>> at
>>>>>
org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getSessionFactory(HibernatePackageHandler.java:335 )
>>>>>
>>>>> at
>>>>>
org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.readPackageInfos(HibernatePackageHandler.java:293)
>>>>>
>>>>> at
>>>>>
org.eclipse.emf.cdo.server.internal.hibernate.HibernatePacka geHandler.getCDOPackageInfos(HibernatePackageHandler.java:21 5)
>>>>>
>>>>> at
>>>>>
org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Reader.readPackageInfos(HibernateStoreReader.java:87)
>>>>>
>>>>> at
>>>>>
org.eclipse.emf.cdo.internal.server.PackageManager.doActivat e(PackageManager.java:72)
>>>>>
>>>>> at
>>>>>
org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
>>>>>
>>>>> at
>>>>>
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>>>>>
>>>>> at
>>>>>
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>>>>>
>>>>> at
>>>>>
org.eclipse.emf.cdo.internal.server.Repository.activateRepos itory(Repository.java:264)
>>>>>
>>>>> at
>>>>>
org.eclipse.emf.cdo.internal.server.Repository.doActivate(Re pository.java:251)
>>>>>
>>>>> at
>>>>>
org.eclipse.net4j.internal.util.lifecycle.Lifecycle.activate (Lifecycle.java:57)
>>>>>
>>>>> at
>>>>>
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:71)
>>>>>
>>>>> at
>>>>>
org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(Life cycleUtil.java:61)
>>>>>
>>>>> at
>>>>>
org.eclipse.emf.cdo.server.CDOServerUtil.addRepository(CDOSe rverUtil.java:67)
>>>>>
>>>>> at
>>>>> org.eclipse.cdo.bug.LaunchServerTest.setUp(LaunchServerTest. java:54)
>>>>> at junit.framework.TestCase.runBare(TestCase.java:128)
>>>>> at junit.framework.TestResult$1.protect(TestResult.java:106)
>>>>> at junit.framework.TestResult.runProtected(TestResult.java:124)
>>>>> at junit.framework.TestResult.run(TestResult.java:109)
>>>>> at junit.framework.TestCase.run(TestCase.java:120)
>>>>> at junit.framework.TestSuite.runTest(TestSuite.java:230)
>>>>> at junit.framework.TestSuite.run(TestSuite.java:225)
>>>>> at
>>>>>
org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestRefer ence.run(JUnit3TestReference.java:130)
>>>>>
>>>>> at
>>>>>
org.eclipse.jdt.internal.junit.runner.TestExecution.run(Test Execution.java:38)
>>>>>
>>>>> at
>>>>>
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:460)
>>>>>
>>>>> at
>>>>>
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTe sts(RemoteTestRunner.java:673)
>>>>>
>>>>> at
>>>>>
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(R emoteTestRunner.java:386)
>>>>>
>>>>> at
>>>>>
org.eclipse.pde.internal.junit.runtime.RemotePluginTestRunne r.main(RemotePluginTestRunner.java:62)
>>>>>
>>>>> at
>>>>>
org.eclipse.pde.internal.junit.runtime.CoreTestApplication.r un(CoreTestApplication.java:23)
>>>>>
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>>>
>>>>> at
>>>>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>>>
>>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>>> at
>>>>>
org.eclipse.equinox.internal.app.EclipseAppContainer.callMet hodWithException(EclipseAppContainer.java:574)
>>>>>
>>>>> at
>>>>>
org.eclipse.equinox.internal.app.EclipseAppHandle.run(Eclips eAppHandle.java:195)
>>>>>
>>>>> at
>>>>>
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .runApplication(EclipseAppLauncher.java:110)
>>>>>
>>>>> at
>>>>>
org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher .start(EclipseAppLauncher.java:79)
>>>>>
>>>>> at
>>>>>
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:379)
>>>>>
>>>>> at
>>>>>
org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseS tarter.java:179)
>>>>>
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>>>
>>>>> at
>>>>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>>>
>>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>>> at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java: 549)
>>>>> at org.eclipse.equinox.launcher.Main.basicRun(Main.java:504)
>>>>> at org.eclipse.equinox.launcher.Main.run(Main.java:1236)
>>>>> at org.eclipse.equinox.launcher.Main.main(Main.java:1212)
>>>>> Caused by: java.lang.reflect.InvocationTargetException
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>>
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce ssorImpl.java:39)
>>>>>
>>>>> at
>>>>>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe thodAccessorImpl.java:25)
>>>>>
>>>>> at java.lang.reflect.Method.invoke(Method.java:585)
>>>>> at
>>>>> net.sf.cglib.core.ReflectUtils.defineClass(ReflectUtils.java :384)
>>>>> at
>>>>>
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>>>>
>>>>> ... 61 more
>>>>> Caused by: java.lang.NoClassDefFoundError:
>>>>> org/hibernate/proxy/HibernateProxy
>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>>>>> ... 67 more
>>>>>
>>>>> I must admit that I'm still not sure about te fundamental interwork
>>>>> between Hibernate and the user defined meta model. It seems as if
>>>>> asm and cglib are somewhere in between but I have no idea why at
>>>>> all. I thought we have implemented a bunch of static Hibernate
>>>>> interfaces to make the CDORevision instances known to Hibernate.
>>>>> Please somebody explain why we need runtime byte code engineering?
>>>>> Is there some configuration missing to tell Hibernate to use our
>>>>> special tuplizers?
>>>>>
>>>>> I have changed several things in the code of the test plugin and
>>>>> since it's quite small now that the jars are removed, I attach my
>>>>> version of the zip here (including a JUnit Plugin launch config).
>>>>> And don't forget to update the cdo.server.hibernate.libraries
>>>>> plugin from CVS to get the new buddy policy!
>>>>>
>>>>> Cheers
>>>>> /Eike
>>>>>
>>>>> Martin Taal schrieb:
>>>>>> Hi Cédric,
>>>>>> I looked at it for many hours trying different things (using your
>>>>>> testcase) but I am kind of stuck with this (getting the same error
>>>>>> as you). The difference I can see with the way I run the
>>>>>> cdo.hibernate testcases (as a junit test) and how your testcase is
>>>>>> run (junit plugin test). This for sure means difference in class
>>>>>> loader behavior.
>>>>>>
>>>>>> Eike, do you have an idea? I am not sure how classloading works
>>>>>> when you start a cdo server in a plugin. Which plugins are then
>>>>>> available in the cdo server? The testcases can be downloaded from
>>>>>> the link below.
>>>>>>
>>>>>> gr. Martin
>>>>>>
>>>>>> Cédric Brun wrote:
>>>>>>> Thanks a lot ! I'm looking forward to your advices !
>>>>>>>
>>>>>>> Cédric
>>>>>>>
>>>>>>> Martin Taal wrote:
>>>>>>>
>>>>>>>> I am downloading your zip-file. I don't have time to look at it
>>>>>>>> right away
>>>>>>>> but I will do my best later this evening.
>>>>>>>>
>>>>>>>> gr. Martin
>>>>>>>>
>>>>>>>> Cédric Brun wrote:
>>>>>>>>> I tried to do the same things as in the CDO/Hibernate unit tests.
>>>>>>>>>
>>>>>>>>> Here's a zipped project trying to launch an HibernateServer
>>>>>>>>> with a unit
>>>>>>>>> test adding elements.
>>>>>>>>>
( http://cedric.brun.free.fr/Tortoose/org.eclipse.cdo-teneo-te st.zip)
>>>>>>>>> The
>>>>>>>>> libs I found are included in the same project, but I still have
>>>>>>>>> no luck
>>>>>>>>> and failling at :
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>>
>>>>>>>
net.sf.cglib.core.AbstractClassGenerator.create(AbstractClas sGenerator.java:219)
>>>>>>>
>>>>>>>>> ... 62 more
>>>>>>>>> Caused by: java.lang.NoClassDefFoundError:
>>>>>>>>> org/hibernate/proxy/HibernateProxy
>>>>>>>>> at java.lang.ClassLoader.defineClass1(Native Method)
>>>>>>>>> at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
>>>>>>>>> ... 68 more
>>>>>>>>>
>>>>>>>>> please note that the test project I provide really is a "quick
>>>>>>>>> and ugly"
>>>>>>>>> one ^^
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cédric
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Martin Taal wrote:
>>>>>>>>>
>>>>>>>>>> Hmm, I can understand your headache. Classpath issues are the
>>>>>>>>>> most
>>>>>>>>>> annoying errors...
>>>>>>>>>>
>>>>>>>>>> In your junit test source file, what happens when you enter the
>>>>>>>>>> org.objectweb.asm.Type, can you source file see this class?
>>>>>>>>>> Do you have cglib in some other plugin which does not have asm?
>>>>>>>>>>
>>>>>>>>>> What I also find slightly strange is that cglib is used in
>>>>>>>>>> this case
>>>>>>>>>> because normally cdo objects are not proxied using cglib. Can
>>>>>>>>>> you post
>>>>>>>>>> the generated mapping? How does your cdo initialization code
>>>>>>>>>> look like?
>>>>>>>>>>
>>>>>>>>>> gr. Martin
>>>>>>>>>>
>>>>>>>>>> Cédric Brun wrote:
>>>>>>>>>>> Hi Martin,
>>>>>>>>>>>
>>>>>>>>>>> and thanks for your quick answer ! what I did is the
>>>>>>>>>>> following :
>>>>>>>>>>>
>>>>>>>>>>> I used the org.eclipse.emf.cdo.server.hibernate.libraries
>>>>>>>>>>> bundle and
>>>>>>>>>>> added that :
>>>>>>>>>>>
>>>>>>>>>>> Bundle-ClassPath: lib/hibernate3.jar,
>>>>>>>>>>> lib/cglib-2.1.3.jar,
>>>>>>>>>>> lib/commons-collections-2.1.1.jar,
>>>>>>>>>>> lib/dom4j.jar,
>>>>>>>>>>> lib/jta.jar
>>>>>>>>>>> lib/asm-3.0.jar
>>>>>>>>>>>
>>>>>>>>>>> and this directive :
>>>>>>>>>>>
>>>>>>>>>>> Eclipse-BuddyPolicy: dependent
>>>>>>>>>>>
>>>>>>>>>>> as stated in
>>>>>>>>>>>
http://www.elver.org/hibernate/installation.html#Hibernate+L ibraries
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Then I don't get the same error, now when I start my own
>>>>>>>>>>> plugin (JUnit
>>>>>>>>>>> plugin test) depending on
>>>>>>>>>>> org.eclipse.emf.cdo.server.hibernate.libraries I get :
>>>>>>>>>>>
>>>>>>>>>>> java.lang.NoClassDefFoundError: org/objectweb/asm/Type
>>>>>>>>>>> at
>>>>>>>>>>> net.sf.cglib.core.TypeUtils.parseType(TypeUtils.java:180)
>>>>>>>>>>> at
>>>>>>>>>>> net.sf.cglib.core.KeyFactory.<clinit>(KeyFactory.java:66)
>>>>>>>>>>> at
>>>>>>>>>>> net.sf.cglib.proxy.Enhancer.<clinit>(Enhancer.java:69)
>>>>>>>>>>> at
>>>>>>>>>>>
>>>>>>>
org.hibernate.proxy.pojo.cglib.CGLIBLazyInitializer.getProxy Factory(CGLIBLazyInitializer.java:117)
>>>>>>>
>>>>>>>>>>> These jars are such a headache ! Do you know - by chance -
>>>>>>>>>>> what's
>>>>>>>>>>> going on ?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Cédric
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Martin Taal wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Cédric,
>>>>>>>>>>>> Most of the time this exception is thrown because hibernate
>>>>>>>>>>>> can't find
>>>>>>>>>>>> the class. A common cause is that the hibernate jar files
>>>>>>>>>>>> are located
>>>>>>>>>>>> in a separate plugin which can not reach the classes in
>>>>>>>>>>>> other plugins.
>>>>>>>>>>>> If you set the eclipse buddy policy of the hibernate plugin to
>>>>>>>>>>>> dependent then this can help.
>>>>>>>>>>>>
>>>>>>>>>>>> gr. Martin
>>>>>>>>>>>>
>>>>>>>>>>>> Cédric Brun wrote:
>>>>>>>>>>>>> Hi Simon and Eike,
>>>>>>>>>>>>>
>>>>>>>>>>>>> thanks for your feedback, now I know what I could expect in
>>>>>>>>>>>>> using
>>>>>>>>>>>>> optimized implementation, I'll have a try on the hibernate
>>>>>>>>>>>>> store and
>>>>>>>>>>>>> see if it is scaling in a better way.
>>>>>>>>>>>>>
>>>>>>>>>>>>> You gave me many pointers to explain the performances
>>>>>>>>>>>>> issues but in
>>>>>>>>>>>>> fact it seems that the SQL execution is taking so much time
>>>>>>>>>>>>> (not
>>>>>>>>>>>>> suprisingly in fact ;) . My first profile was wrong.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So I'm installing the hibernate store retrieving all the
>>>>>>>>>>>>> source code
>>>>>>>>>>>>> from CVS, and I'm trying to launch the unit tests. I added
>>>>>>>>>>>>> many jars
>>>>>>>>>>>>> and everything gets compiled but most of the tests are
>>>>>>>>>>>>> failling with
>>>>>>>>>>>>> "
>>>>>>>>>>>>>
>>>>>>>>>>>>> "
>>>>>>>>>>>>> org.hibernate.MappingException: Could not determine type for:
>>>>>>>>>>>>>
>>>>>>>
org.eclipse.emf.cdo.server.internal.hibernate.tuplizer.CDOTy peUserType,
>>>>>>>
>>>>>>>>>>>>> at table: cdo_features, for columns:
>>>>>>>>>>>>> [org.hibernate.mapping.Column(type)] "
>>>>>>>>>>>>>
>>>>>>>>>>>>> any hint ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Cédric
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Simon McDuff wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> We noticed a performance degradation in the following case:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> When we fetch a collection of objects that contains
>>>>>>>>>>>>>> collections that
>>>>>>>>>>>>>> we don't want to use. Ex:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Model:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ClassB
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>> int foo1;
>>>>>>>>>>>>>> List childrens1;
>>>>>>>>>>>>>> List childrens2;
>>>>>>>>>>>>>> List childrens3;
>>>>>>>>>>>>>> };
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Code:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Resource.contents contains objects of type ClassB.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> By fetching instances of classB, we will also fetched some
>>>>>>>>>>>>>> items in
>>>>>>>>>>>>>> the list(childrens1, childrens2, childrens3) The problem
>>>>>>>>>>>>>> we had.. is
>>>>>>>>>>>>>> we do not know if we want to use it. SO better not to
>>>>>>>>>>>>>> download them.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So In our production(we used our own IStore), we never
>>>>>>>>>>>>>> download any
>>>>>>>>>>>>>> objects from collection. I'm working at the moment to put
>>>>>>>>>>>>>> in place
>>>>>>>>>>>>>> all feature we used in the current CDO. That one will be
>>>>>>>>>>>>>> part of
>>>>>>>>>>>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214477 To
>>>>>>>>>>>>>> begin
>>>>>>>>>>>>>> with,CDORevisionImpl needs to supports list that are 0
>>>>>>>>>>>>>> even if
>>>>>>>>>>>>>> referenceChunkSize == 0.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It is true that we obtain good performance.
>>>>>>>>>>>>>> Worst case :
>>>>>>>>>>>>>> 1500 objects /sec (Real fetch from disk. Page that
>>>>>>>>>>>>>> contains the
>>>>>>>>>>>>>> objects, wasn't in the cache of the filesystem.)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best case
>>>>>>>>>>>>>> 12 000 objects/Sec (Fetch objects that are already in the
>>>>>>>>>>>>>> case of
>>>>>>>>>>>>>> the filesystem)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (These statistics always fetch objects from server to
>>>>>>>>>>>>>> client. The
>>>>>>>>>>>>>> cache client isn't used)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We have another optimization one with SmartReadAhead but
>>>>>>>>>>>>>> for now
>>>>>>>>>>>>>> only work
>>>>>>>>>>>>>> for UI client. (for now) This is something we will
>>>>>>>>>>>>>> integrate as
>>>>>>>>>>>>>> well in CDO. But for now we concentrate on the other things.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I used an Object database. Based on my preliminary test
>>>>>>>>>>>>>> without CDO.
>>>>>>>>>>>>>> I compare
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hibernate vs Objectivity/EMF :
>>>>>>>>>>>>>> Objectivity/EMF is around 3 to 50 times faster than
>>>>>>>>>>>>>> Hibernate.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The worst.. you will get 30 to 240 objects/sec or
>>>>>>>>>>>>>> the best... you will get 500 to 4000 objects/sec
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Based on your comments... you are getting the worst (with
>>>>>>>>>>>>>> Derby)!!
>>>>>>>>>>>>>> :-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Another comments about insertion.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We found another problem when we insert data(DOn't know if
>>>>>>>>>>>>>> it is
>>>>>>>>>>>>>> still true). Resource.Contents is a unique list. Basically
>>>>>>>>>>>>>> each time
>>>>>>>>>>>>>> we insert one element, it needs to fetch all the
>>>>>>>>>>>>>> collection to see
>>>>>>>>>>>>>> if we already have this element. Basically.. it is time
>>>>>>>>>>>>>> consuming.
>>>>>>>>>>>>>> In our case we are not going through this stage to persist
>>>>>>>>>>>>>> our
>>>>>>>>>>>>>> objects... but the objects are not accessible through that
>>>>>>>>>>>>>> list.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If speed is an issue for you!! We (You included :-)) can
>>>>>>>>>>>>>> optimize
>>>>>>>>>>>>>> it. What I like is Eike did a really good job doing that
>>>>>>>>>>>>>> framwork
>>>>>>>>>>>>>> and optimizations are easy to add. Also you can change
>>>>>>>>>>>>>> your back-end
>>>>>>>>>>>>>> without affecting you client side. (Maybe going to an Object
>>>>>>>>>>>>>> database!!) THis was one of the purpose of CDO!! :-)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Simon
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "Eike Stepper" <stepper@sympedia.de> a écrit dans le
>>>>>>>>>>>>>> message de
>>>>>>>>>>>>>> news: 483E9332.6050800@sympedia.de... Hi Cédric,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> That sounds interesting!
>>>>>>>>>>>>>> Comments inline...
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cédric Brun schrieb:
>>>>>>>>>>>>>> Hi Eike,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I've been messing around CDO lately and I hacked a few ugly
>>>>>>>>>>>>>> unit-tests measuring read/write throughput when I use CDO
>>>>>>>>>>>>>> (RC2).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm using an embeded-derby database, two different metamodels
>>>>>>>>>>>>>> (complex one - 250 EClassifiers/300 features and a simple
>>>>>>>>>>>>>> one - 1
>>>>>>>>>>>>>> Classifier/3 Features)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The test is a "quick and dirty" one so please do not take
>>>>>>>>>>>>>> these
>>>>>>>>>>>>>> figures too seriously, I would just like to have your
>>>>>>>>>>>>>> input about
>>>>>>>>>>>>>> the "order of magnitude" I should expect or settings
>>>>>>>>>>>>>> affecting
>>>>>>>>>>>>>> seriously these performances. I tried both JVM and TCP
>>>>>>>>>>>>>> acceptor but
>>>>>>>>>>>>>> the performances are quite similar though JVM is a bit
>>>>>>>>>>>>>> faster.
>>>>>>>>>>>>>> Looks like a sign that the NIO socket channels are
>>>>>>>>>>>>>> efficient ,-)
>>>>>>>>>>>>>> You used localhost for TCP transport I guess?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * First case : building a small model (10000 elements)
>>>>>>>>>>>>>> with the
>>>>>>>>>>>>>> simple MM, only adding instances in an empty database :
>>>>>>>>>>>>>> commiting change on every add I get a 2 objects per second
>>>>>>>>>>>>>> rate.
>>>>>>>>>>>>>> commiting changes every 10 adds - I get a 30 objects per
>>>>>>>>>>>>>> second
>>>>>>>>>>>>>> rate.
>>>>>>>>>>>>>> Seems to be consistent with the fact that only for
>>>>>>>>>>>>>> commits there
>>>>>>>>>>>>>> is remote traffic. And the backend transaction
>>>>>>>>>>>>>> management is
>>>>>>>>>>>>>> another overhead. So the impact of the number
>>>>>>>>>>>>>> objects/commit
>>>>>>>>>>>>>> starts with bigger transactions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> browsing the just commited elements opening a new
>>>>>>>>>>>>>> transaction and
>>>>>>>>>>>>>> getting a new Resource instance, I get around 30 000
>>>>>>>>>>>>>> objets/sec rate
>>>>>>>>>>>>>> just iterating thanks to Resource.getAllContents().
>>>>>>>>>>>>>> Hehe, this is a sign that client side caching works well:
>>>>>>>>>>>>>> All views and transactions of a single session share
>>>>>>>>>>>>>> (and cache)
>>>>>>>>>>>>>> their object state. Otherwise I'd expect much smaller
>>>>>>>>>>>>>> values.
>>>>>>>>>>>>>> With his ObjectivityStore implementation Simon (cc'ed)
>>>>>>>>>>>>>> reached
>>>>>>>>>>>>>> rates of several thousands objects/sec. But the DBStore
>>>>>>>>>>>>>> (which you
>>>>>>>>>>>>>> used) is not that optimized.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> * Second case : I built a big model in the database using the
>>>>>>>>>>>>>> complex metamodel. It contains 700 000 elements. The
>>>>>>>>>>>>>> creation went
>>>>>>>>>>>>>> pretty nicely (I did not measured it exactly as it was
>>>>>>>>>>>>>> launched
>>>>>>>>>>>>>> during a night, but it seems I got around 100 objects/sec
>>>>>>>>>>>>>> rate in
>>>>>>>>>>>>>> writting as I was writting every 700 objects.
>>>>>>>>>>>>>> Sounds not too bad.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Once the model is built the derby database is 222 MB
>>>>>>>>>>>>>> (which - I
>>>>>>>>>>>>>> heard - is not so big for derby).
>>>>>>>>>>>>>> Hmm, I have no extensive experience with Derby.
>>>>>>>>>>>>>> To be honest, I'm really no database expert (although I
>>>>>>>>>>>>>> can write
>>>>>>>>>>>>>> simple SQL statements).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Retrieving the distant resource and calling eAllContents()
>>>>>>>>>>>>>> on it, I
>>>>>>>>>>>>>> get a 50 Objects/sec rate at best, which mean my
>>>>>>>>>>>>>> eAllContents() call
>>>>>>>>>>>>>> would take 4 hours !
>>>>>>>>>>>>>> Currently I'm not sure what exactly eAllContents() does do
>>>>>>>>>>>>>> (iteration order and so). There are several CDO features
>>>>>>>>>>>>>> that can
>>>>>>>>>>>>>> have a major impact on load rates!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Collections (xrefs and containment) can be loaded
>>>>>>>>>>>>>> lazily. I
>>>>>>>>>>>>>> don't mean the target objects, which are lazy by default
>>>>>>>>>>>>>> anyway,
>>>>>>>>>>>>>> but even the target IDs that are in the collections. We
>>>>>>>>>>>>>> call that
>>>>>>>>>>>>>> "reference chunking" (but search for a better name is
>>>>>>>>>>>>>> ongoing).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> public interface CDOSession extends
>>>>>>>>>>>>>> CDOProtocolSession,
>>>>>>>>>>>>>> IContainer<CDOView>
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>> public int getReferenceChunkSize();
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> public void setReferenceChunkSize(int
>>>>>>>>>>>>>> referenceChunkSize);
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> - Loading collection elements (target objects) can be
>>>>>>>>>>>>>> batched (the
>>>>>>>>>>>>>> current name is even worse)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> public interface CDOView extends CDOProtocolView,
>>>>>>>>>>>>>> INotifier
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>> public int getLoadRevisionCollectionChunkSize();
>>>>>>>&g


http://cedric.brun.io news and articles on eclipse and eclipse modeling.
Re: [CDO] reading/writting throughput performances (hibernate mapper) [message #619154 is a reply to message #123598] Tue, 03 June 2008 08:45 Go to previous message
Cedric Brun is currently offline Cedric BrunFriend
Messages: 431
Registered: July 2009
Senior Member
Hi,

here comes a quick report on my measures.

using cdo with the teneo backend (thanks again for your help setting this
one up !) mysql as a database (localhost one) and jvm CDO acceptor.

writing the objects on the database : the process starts at 150 objects/sec
but quickly drop to around 30 objects/sec and fail before the end when 40
000 instances are inserted in the database with:
[ERROR] FATAL: Memory infrastructure corrupted after successful commit
operation of the store
and a TimeOutException on the client.

I can only see "load" issues related to this failure as I'm just copying the
same data over and over on the database so it can't come from the data.

I noticed that my metamodel composed of 250 EClasses is mapped to a big
unique table (for my abstract top level element).

Reading elements : as I did not managed to fill the database like I did not
managed to get the figures.

I also often ends up with:

[ERROR] CommitContext not set
java.lang.IllegalStateException: CommitContext not set
at
org.eclipse.emf.cdo.server.internal.hibernate.HibernateThrea dContext.getCommitContext(HibernateThreadContext.java:69)
at
org.eclipse.emf.cdo.server.internal.hibernate.HibernateUtil. getCDORevision(HibernateUtil.java:116)
at
org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Reader.readRevision(HibernateStoreReader.java:147)
at
org.eclipse.emf.cdo.internal.server.RevisionManager.loadRevi sion(RevisionManager.java:210)
at
org.eclipse.emf.cdo.internal.protocol.revision.CDORevisionRe solverImpl.getRevision(CDORevisionResolverImpl.java:282)
at
org.eclipse.emf.cdo.internal.protocol.revision.CDORevisionRe solverImpl.getRevision(CDORevisionResolverImpl.java:144)
at
org.eclipse.emf.cdo.internal.server.protocol.LoadRevisionInd ication.getRevision(LoadRevisionIndication.java:171)
at
org.eclipse.emf.cdo.internal.server.protocol.LoadRevisionInd ication.responding(LoadRevisionIndication.java:147)
at
org.eclipse.net4j.signal.IndicationWithResponse.execute(Indi cationWithResponse.java:54)
at
org.eclipse.emf.cdo.internal.server.protocol.CDOReadIndicati on.execute(CDOReadIndication.java:38)
at org.eclipse.net4j.signal.Signal.runSync(Signal.java:143)

But I'm not able to identify why and under which circomstances.


I did not managed to use derby-embedded with teneo/hibernate as I only get
an exception just after the initialization process :

org.eclipse.net4j.util.transaction.TransactionException: could not insert:
[org.eclipse.emf.cdo.internal.protocol.model.CDOPackageImpl]
at
org.eclipse.emf.internal.cdo.CDOTransactionImpl.commit(CDOTr ansactionImpl.java:241)
at


It's a shame as I guess I would not have the overhead of the network and
concurrency stack. Derby-embedded fits better with my use cases.
I used the following app.properties:

hibernate.dialect=org.hibernate.dialect.DerbyDialect
hibernate.connection.driver_class=org.apache.derby.jdbc.Embe ddedDriver
hibernate.connection.url=jdbc:derby:/tmp/cdohibernatederby;c reate=true
hibernate.connection.autocommit=true
hibernate.cache.provider_class=org.hibernate.cache.Hashtable CacheProvider



Any hint about this error ?


Cédric


Eike Stepper wrote:

> Hi Cédric,
>
> That sounds interesting!
> Comments inline...
>
>
> Cédric Brun schrieb:
>> Hi Eike,
>>
>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>> measuring read/write throughput when I use CDO (RC2).
>>
>> I'm using an embeded-derby database, two different metamodels (complex
>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>> Features)
>>
>> The test is a "quick and dirty" one so please do not take these figures
>> too seriously, I would just like to have your input about the "order of
>> magnitude" I should expect or settings affecting seriously these
>> performances. I tried both JVM and TCP acceptor but the performances are
>> quite similar though JVM is a bit faster.
>>
> Looks like a sign that the NIO socket channels are efficient ,-)
> You used localhost for TCP transport I guess?
>
>>
>> * First case : building a small model (10000 elements) with the simple
>> MM, only adding instances in an empty database :
>> commiting change on every add I get a 2 objects per second rate.
>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>
> Seems to be consistent with the fact that only for commits there is
> remote traffic.
> And the backend transaction management is another overhead.
> So the impact of the number objects/commit starts with bigger
> transactions.
>
>> browsing the just commited elements opening a new transaction and getting
>> a new Resource instance, I get around 30 000 objets/sec rate just
>> iterating thanks to Resource.getAllContents().
>>
> Hehe, this is a sign that client side caching works well:
> All views and transactions of a single session share (and cache) their
> object state.
> Otherwise I'd expect much smaller values.
> With his ObjectivityStore implementation Simon (cc'ed) reached rates of
> several thousands objects/sec.
> But the DBStore (which you used) is not that optimized.
>
>> * Second case : I built a big model in the database using the complex
>> metamodel. It contains 700 000 elements. The creation went pretty nicely
>> (I did not measured it exactly as it was launched during a night, but it
>> seems I got around 100 objects/sec rate in writting as I was writting
>> every 700 objects.
>>
> Sounds not too bad.
>
>>
>> Once the model is built the derby database is 222 MB (which - I heard -
>> is not so big for derby).
>>
> Hmm, I have no extensive experience with Derby.
> To be honest, I'm really no database expert (although I can write simple
> SQL statements).
>
>> Retrieving the distant resource and calling eAllContents() on it, I get a
>> 50 Objects/sec rate at best, which mean my eAllContents() call would take
>> 4 hours !
>>
> Currently I'm not sure what exactly eAllContents() does do (iteration
> order and so).
> There are several CDO features that can have a major impact on load rates!
>
> - Collections (xrefs and containment) can be loaded lazily. I don't mean
> the target objects, which are lazy by default anyway, but even the
> target IDs that are in the collections. We call that "reference
> chunking" (but search for a better name is ongoing).
> |*public interface *CDOSession *extends *CDOProtocolSession,
> |IContainer<CDOView>
> {
> *public **int *getReferenceChunkSize();
>
> *public **void *setReferenceChunkSize(*int *referenceChunkSize);|
>
>
> - Loading collection elements (target objects) can be batched (the
> current name is even worse)
> |*public interface *CDOView *extends *CDOProtocolView, INotifier
> {
> *public **int *getLoadRevisionCollectionChunkSize();
>
> *public **void *setLoadRevisionCollectionChunkSize(*int
> *loadRevisionCollectionChunkSize);|
>
>
>
> - One of the most sophisticated feature is automatic model usage
> analyzation. The result are fetch rules that are sent to the server.
> When later objects are loaded from the server, it can answer with
> additional objects that are likely to be needed by the client in the
> near future.
> |*public class *CDOViewImpl *extends
> |*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
> |CDOIDProvider,
> Adapter.Internal
> {
> *public *CDOFeatureAnalyzer getFeatureAnalyzer()
> {
> *return *featureAnalyzer;
> }
>
> *public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
> {
> *this*.featureAnalyzer = featureAnalyzer == *null *?
> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
> }|
>
>
> - There's a SmartReadAheadThread (if you're curious, please ask Simon).
> - Caches can be configured
> - ...
>
>
> Known issues of the DBStore:
> - When in auditing mode updates are always written as complete inserts
> (by design)
> - When configuring toManyReferences to a value different from
> LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
> - No (few) prepared statements are used (see
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
> - Maybe others...
>
> Please note that the DBStore was not written by me to compete with
> Hibernate in terms of performance or mapping capabilities.
> Now that Martin and I, well in the end mostly Martin, provide a
> HibernateStore and this HibernateStore gets more and more features I
> expect that there will be a trend away from the DBStore.
>
>> This is probably linked to the way the mapping to db is done, here's the
>> config I used (came from CDO-server example if I recall correctly).
>>
>>
>> <repository name="repo1">
>> <property name="overrideUUID"
>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>> <property name="supportingAudits" value="true"/>
>> <property name="verifyingRevisions" value="false"/>
>> <property name="currentLRUCapacity" value="10000"/>
>>
> I guess this one could be larger.
>
>> <property name="revisedLRUCapacity" value="100"/>
>>
>> <store type="db">
>> <!-- type: horizontal | vertical | <any
>> user-contributed type> --> <mappingStrategy
>> type="horizontal">
>> <!-- ONE_TABLE_PER_REFERENCE |
>> ONE_TABLE_PER_CLASS |
>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>> <property name="toManyReferences"
>> value="ONE_TABLE_PER_REFERENCE"/>
>>
>> <!-- LIKE_ATTRIBUTES |
>> LIKE_TO_MANY_REFERENCES --> <property
>> name="toOneReferences"
>> value="LIKE_ATTRIBUTES"/>
>>
>> <!-- MODEL | STRATEGY -->
>> <property name="mappingPrecedence"
>> value="MODEL"/>
>>
> This one is obsolete.
>
>> </mappingStrategy>
>>
>> <dbAdapter name="derby-embedded"/>
>> <dataSource
>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>> databaseName="/tmp/cdodb1"
>> createDatabase="create"/>
>>
>> </store>
>> </repository>
>>
>>
>>
>>
>> What do you think about that ? What throughput do you think I should
>> expect ?
> As I said that's very difficult to judge.
> I believe it's more appropriate to compare the actual values with values
> from other measures (other setups, properties, etc. or even other
> products) instead of random expectations.
> Apologies that I didn't have the time so far to provide some reference
> measures myself. Maybe this gets better now that the team starts to grow
> slowly ;-)
>
>> What are the main reasons fur such figures ?
>>
> Hmm, I *suspect* you don't like the figures? I can find no evidence your
> assessment ;-)
>
> Ahh,one more note:
> Tracing can also have a significant impact on performance and end-to-end
> throughput.
> Turn it off by all means. And please make sure that it's really turned
> off, not only invisible (i.e. console is not a target).
>
> Cheers
> /Eike


http://cedric.brun.io news and articles on eclipse and eclipse modeling.
Re: [CDO] reading/writting throughput performances (hibernate mapper) [message #619155 is a reply to message #125026] Tue, 03 June 2008 09:43 Go to previous message
Martin Taal is currently offline Martin TaalFriend
Messages: 5468
Registered: July 2009
Senior Member
Hi Cédric,
I can't answer all questions but see my comments inline.

gr. Martin

Cédric Brun wrote:
> Hi,
>
> here comes a quick report on my measures.
>
> using cdo with the teneo backend (thanks again for your help setting this
> one up !) mysql as a database (localhost one) and jvm CDO acceptor.
>
> writing the objects on the database : the process starts at 150 objects/sec
> but quickly drop to around 30 objects/sec and fail before the end when 40
> 000 instances are inserted in the database with:
> [ERROR] FATAL: Memory infrastructure corrupted after successful commit
> operation of the store
> and a TimeOutException on the client.
MT>> Can you post the complete stacktrace for this?

>
> I can only see "load" issues related to this failure as I'm just copying the
> same data over and over on the database so it can't come from the data.
>
> I noticed that my metamodel composed of 250 EClasses is mapped to a big
> unique table (for my abstract top level element).
MT>> This is single table inheritance mapping strategy. You can also choose join-table inheritance
strategy then the database model will become more descriptive. See here:
http://www.elver.org/hibernate/inheritance.html#Supported+in heritance+mapping+strategies
To do join-table, you can do pass the property:
org.eclipse.emf.teneo.mapper.PersistenceMappingOptions.INHER ITANCEMAPPING
with the string value JOIN
You can set this property in the same properties object as the database connection info.

>
> Reading elements : as I did not managed to fill the database like I did not
> managed to get the figures.
>
> I also often ends up with:
>
> [ERROR] CommitContext not set
> java.lang.IllegalStateException: CommitContext not set
> at
> org.eclipse.emf.cdo.server.internal.hibernate.HibernateThrea dContext.getCommitContext(HibernateThreadContext.java:69)
> at
> org.eclipse.emf.cdo.server.internal.hibernate.HibernateUtil. getCDORevision(HibernateUtil.java:116)
> at
> org.eclipse.emf.cdo.server.internal.hibernate.HibernateStore Reader.readRevision(HibernateStoreReader.java:147)
> at
> org.eclipse.emf.cdo.internal.server.RevisionManager.loadRevi sion(RevisionManager.java:210)
> at
> org.eclipse.emf.cdo.internal.protocol.revision.CDORevisionRe solverImpl.getRevision(CDORevisionResolverImpl.java:282)
> at
> org.eclipse.emf.cdo.internal.protocol.revision.CDORevisionRe solverImpl.getRevision(CDORevisionResolverImpl.java:144)
> at
> org.eclipse.emf.cdo.internal.server.protocol.LoadRevisionInd ication.getRevision(LoadRevisionIndication.java:171)
> at
> org.eclipse.emf.cdo.internal.server.protocol.LoadRevisionInd ication.responding(LoadRevisionIndication.java:147)
> at
> org.eclipse.net4j.signal.IndicationWithResponse.execute(Indi cationWithResponse.java:54)
> at
> org.eclipse.emf.cdo.internal.server.protocol.CDOReadIndicati on.execute(CDOReadIndication.java:38)
> at org.eclipse.net4j.signal.Signal.runSync(Signal.java:143)
>
> But I'm not able to identify why and under which circomstances.
>
>
> I did not managed to use derby-embedded with teneo/hibernate as I only get
> an exception just after the initialization process :
>
> org.eclipse.net4j.util.transaction.TransactionException: could not insert:
> [org.eclipse.emf.cdo.internal.protocol.model.CDOPackageImpl]
> at
> org.eclipse.emf.internal.cdo.CDOTransactionImpl.commit(CDOTr ansactionImpl.java:241)
> at
MT>> Can you post the complete stacktrace? I am not sure if this is the stacktrace visible on the
server or if this is the stacktrace on the client? The one on the server gives the most information.

>
>
> It's a shame as I guess I would not have the overhead of the network and
> concurrency stack. Derby-embedded fits better with my use cases.
> I used the following app.properties:
>
> hibernate.dialect=org.hibernate.dialect.DerbyDialect
> hibernate.connection.driver_class=org.apache.derby.jdbc.Embe ddedDriver
> hibernate.connection.url=jdbc:derby:/tmp/cdohibernatederby;c reate=true
> hibernate.connection.autocommit=true
> hibernate.cache.provider_class=org.hibernate.cache.Hashtable CacheProvider
>
>
>
> Any hint about this error ?
>
>
> Cédric
>
>
> Eike Stepper wrote:
>
>> Hi Cédric,
>>
>> That sounds interesting!
>> Comments inline...
>>
>>
>> Cédric Brun schrieb:
>>> Hi Eike,
>>>
>>> I've been messing around CDO lately and I hacked a few ugly unit-tests
>>> measuring read/write throughput when I use CDO (RC2).
>>>
>>> I'm using an embeded-derby database, two different metamodels (complex
>>> one - 250 EClassifiers/300 features and a simple one - 1 Classifier/3
>>> Features)
>>>
>>> The test is a "quick and dirty" one so please do not take these figures
>>> too seriously, I would just like to have your input about the "order of
>>> magnitude" I should expect or settings affecting seriously these
>>> performances. I tried both JVM and TCP acceptor but the performances are
>>> quite similar though JVM is a bit faster.
>>>
>> Looks like a sign that the NIO socket channels are efficient ,-)
>> You used localhost for TCP transport I guess?
>>
>>> * First case : building a small model (10000 elements) with the simple
>>> MM, only adding instances in an empty database :
>>> commiting change on every add I get a 2 objects per second rate.
>>> commiting changes every 10 adds - I get a 30 objects per second rate.
>>>
>> Seems to be consistent with the fact that only for commits there is
>> remote traffic.
>> And the backend transaction management is another overhead.
>> So the impact of the number objects/commit starts with bigger
>> transactions.
>>
>>> browsing the just commited elements opening a new transaction and getting
>>> a new Resource instance, I get around 30 000 objets/sec rate just
>>> iterating thanks to Resource.getAllContents().
>>>
>> Hehe, this is a sign that client side caching works well:
>> All views and transactions of a single session share (and cache) their
>> object state.
>> Otherwise I'd expect much smaller values.
>> With his ObjectivityStore implementation Simon (cc'ed) reached rates of
>> several thousands objects/sec.
>> But the DBStore (which you used) is not that optimized.
>>
>>> * Second case : I built a big model in the database using the complex
>>> metamodel. It contains 700 000 elements. The creation went pretty nicely
>>> (I did not measured it exactly as it was launched during a night, but it
>>> seems I got around 100 objects/sec rate in writting as I was writting
>>> every 700 objects.
>>>
>> Sounds not too bad.
>>
>>> Once the model is built the derby database is 222 MB (which - I heard -
>>> is not so big for derby).
>>>
>> Hmm, I have no extensive experience with Derby.
>> To be honest, I'm really no database expert (although I can write simple
>> SQL statements).
>>
>>> Retrieving the distant resource and calling eAllContents() on it, I get a
>>> 50 Objects/sec rate at best, which mean my eAllContents() call would take
>>> 4 hours !
>>>
>> Currently I'm not sure what exactly eAllContents() does do (iteration
>> order and so).
>> There are several CDO features that can have a major impact on load rates!
>>
>> - Collections (xrefs and containment) can be loaded lazily. I don't mean
>> the target objects, which are lazy by default anyway, but even the
>> target IDs that are in the collections. We call that "reference
>> chunking" (but search for a better name is ongoing).
>> |*public interface *CDOSession *extends *CDOProtocolSession,
>> |IContainer<CDOView>
>> {
>> *public **int *getReferenceChunkSize();
>>
>> *public **void *setReferenceChunkSize(*int *referenceChunkSize);|
>>
>>
>> - Loading collection elements (target objects) can be batched (the
>> current name is even worse)
>> |*public interface *CDOView *extends *CDOProtocolView, INotifier
>> {
>> *public **int *getLoadRevisionCollectionChunkSize();
>>
>> *public **void *setLoadRevisionCollectionChunkSize(*int
>> *loadRevisionCollectionChunkSize);|
>>
>>
>>
>> - One of the most sophisticated feature is automatic model usage
>> analyzation. The result are fetch rules that are sent to the server.
>> When later objects are loaded from the server, it can answer with
>> additional objects that are likely to be needed by the client in the
>> near future.
>> |*public class *CDOViewImpl *extends
>> |*org.eclipse.net4j.internal.util.event.Notifier *implements *CDOView,
>> |CDOIDProvider,
>> Adapter.Internal
>> {
>> *public *CDOFeatureAnalyzer getFeatureAnalyzer()
>> {
>> *return *featureAnalyzer;
>> }
>>
>> *public **void *setFeatureAnalyzer(CDOFeatureAnalyzer featureAnalyzer)
>> {
>> *this*.featureAnalyzer = featureAnalyzer == *null *?
>> CDOFeatureAnalyzer.NOOP : featureAnalyzer;
>> }|
>>
>>
>> - There's a SmartReadAheadThread (if you're curious, please ask Simon).
>> - Caches can be configured
>> - ...
>>
>>
>> Known issues of the DBStore:
>> - When in auditing mode updates are always written as complete inserts
>> (by design)
>> - When configuring toManyReferences to a value different from
>> LIKE_ATTRIBUTES (which is not implemented yet) it is very slow.
>> - No (few) prepared statements are used (see
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=214487)
>> - Maybe others...
>>
>> Please note that the DBStore was not written by me to compete with
>> Hibernate in terms of performance or mapping capabilities.
>> Now that Martin and I, well in the end mostly Martin, provide a
>> HibernateStore and this HibernateStore gets more and more features I
>> expect that there will be a trend away from the DBStore.
>>
>>> This is probably linked to the way the mapping to db is done, here's the
>>> config I used (came from CDO-server example if I recall correctly).
>>>
>>>
>>> <repository name="repo1">
>>> <property name="overrideUUID"
>>> value="1ff5d226-b1f0-40fb-aba2-0c31b38c764f"/>
>>> <property name="supportingAudits" value="true"/>
>>> <property name="verifyingRevisions" value="false"/>
>>> <property name="currentLRUCapacity" value="10000"/>
>>>
>> I guess this one could be larger.
>>
>>> <property name="revisedLRUCapacity" value="100"/>
>>>
>>> <store type="db">
>>> <!-- type: horizontal | vertical | <any
>>> user-contributed type> --> <mappingStrategy
>>> type="horizontal">
>>> <!-- ONE_TABLE_PER_REFERENCE |
>>> ONE_TABLE_PER_CLASS |
>>> ONE_TABLE_PER_PACKAGE | ONE_TABLE_PER_REPOSITORY | LIKE_ATTRIBUTES -->
>>> <property name="toManyReferences"
>>> value="ONE_TABLE_PER_REFERENCE"/>
>>>
>>> <!-- LIKE_ATTRIBUTES |
>>> LIKE_TO_MANY_REFERENCES --> <property
>>> name="toOneReferences"
>>> value="LIKE_ATTRIBUTES"/>
>>>
>>> <!-- MODEL | STRATEGY -->
>>> <property name="mappingPrecedence"
>>> value="MODEL"/>
>>>
>> This one is obsolete.
>>
>>> </mappingStrategy>
>>>
>>> <dbAdapter name="derby-embedded"/>
>>> <dataSource
>>> class="org.apache.derby.jdbc.EmbeddedDataSource"
>>> databaseName="/tmp/cdodb1"
>>> createDatabase="create"/>
>>>
>>> </store>
>>> </repository>
>>>
>>>
>>>
>>>
>>> What do you think about that ? What throughput do you think I should
>>> expect ?
>> As I said that's very difficult to judge.
>> I believe it's more appropriate to compare the actual values with values
>> from other measures (other setups, properties, etc. or even other
>> products) instead of random expectations.
>> Apologies that I didn't have the time so far to provide some reference
>> measures myself. Maybe this gets better now that the team starts to grow
>> slowly ;-)
>>
>>> What are the main reasons fur such figures ?
>>>
>> Hmm, I *suspect* you don't like the figures? I can find no evidence your
>> assessment ;-)
>>
>> Ahh,one more note:
>> Tracing can also have a significant impact on performance and end-to-end
>> throughput.
>> Turn it off by all means. And please make sure that it's really turned
>> off, not only invisible (i.e. console is not a target).
>>
>> Cheers
>> /Eike
>


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Cell: +31 (0)6 288 48 943
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Previous Topic:[Teneo] ClassCastException from hibernate trying to persist a new EMap entry with containement=true.
Next Topic:[EMF Compare] Does EMF Compare viewer subscribe to workbench's SelectionProvider?
Goto Forum:
  


Current Time: Fri Mar 29 15:13:28 GMT 2024

Powered by FUDForum. Page generated in 0.09985 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top