Home » Modeling » EMF "Technology" (Ecore Tools, EMFatic, etc) » [Teneo] Model persistence performance
|
Re: [Teneo] Model persistence performance [message #92328 is a reply to message #92312] |
Wed, 01 August 2007 17:17 |
Eclipse User |
|
|
|
Originally posted by: merks.ca.ibm.com
Alain,
I know I've had folks complain about performance and have it turn out to
be a single println statement that the person thought was quite
innocuous so it would definitely be good to run your scenario without
spewing out almost 100 times more debug/log data than you'd get just for
the XMI serialization of it. I suppose it would also be good to know if
you are using Hibernate verses JPOX...
Alain Picard wrote:
> Now that I have my model persistence configured and running with small
> test cases, I decided to use it with a "standard" size case.
>
> Well, it took me 90 mins to persist the model to the local MySQL
> database (with debugging on, 600 Megs log file). The model is made up
> of 9 tables and this particular model instance contains about 90k
> records altogether. The equivalent XMI is about 9.5Megs and compresses
> to 229k with zip.
>
> For the most part, all of the records are quite small, mostly integer
> and bigint values, with a few varchar values.
>
> I understand that running without a debug file would surely help
> somewhat, but anything over 30 seconds here surely won't make the grade.
>
> What are other peoples experiences with the performance of persisting
> large EMF models ?
>
> For us this is a small/medium model instance. Many will be 5-10 times
> the size of this one.
>
> Thanks for the feedback.
>
> Alain
|
|
|
Re: [Teneo] Model persistence performance [message #92359 is a reply to message #92328] |
Wed, 01 August 2007 17:27 |
Alain Picard Messages: 266 Registered: July 2009 |
Senior Member |
|
|
Ed,
I am using Hibernate here. Let me try again without the debug log.
Alain
Ed Merks wrote:
> Alain,
>
> I know I've had folks complain about performance and have it turn out to
> be a single println statement that the person thought was quite
> innocuous so it would definitely be good to run your scenario without
> spewing out almost 100 times more debug/log data than you'd get just for
> the XMI serialization of it. I suppose it would also be good to know if
> you are using Hibernate verses JPOX...
>
>
> Alain Picard wrote:
>> Now that I have my model persistence configured and running with small
>> test cases, I decided to use it with a "standard" size case.
>>
>> Well, it took me 90 mins to persist the model to the local MySQL
>> database (with debugging on, 600 Megs log file). The model is made up
>> of 9 tables and this particular model instance contains about 90k
>> records altogether. The equivalent XMI is about 9.5Megs and compresses
>> to 229k with zip.
>>
>> For the most part, all of the records are quite small, mostly integer
>> and bigint values, with a few varchar values.
>>
>> I understand that running without a debug file would surely help
>> somewhat, but anything over 30 seconds here surely won't make the grade.
>>
>> What are other peoples experiences with the performance of persisting
>> large EMF models ?
>>
>> For us this is a small/medium model instance. Many will be 5-10 times
>> the size of this one.
>>
>> Thanks for the feedback.
>>
>> Alain
|
|
|
Re: [Teneo] Model persistence performance [message #92374 is a reply to message #92359] |
Wed, 01 August 2007 18:57 |
Eclipse User |
|
|
|
Originally posted by: benjamin.cabeNOSPAMM.anyware-tech.DOTcom
Hi Alain,
In my case -Hibernate too-, the ratio between log (with a "default"
log4j.properties) and no-log version is someting like 3-4x... Not
negligible :)
Benjamin.
Alain Picard a écrit :
> Ed,
>
> I am using Hibernate here. Let me try again without the debug log.
>
> Alain
|
|
| |
Re: [Teneo] Model persistence performance [message #92403 is a reply to message #92388] |
Wed, 01 August 2007 21:05 |
Alain Picard Messages: 266 Registered: July 2009 |
Senior Member |
|
|
Well, I have turned off the debugging and it did improve dramatically.
Here the new results:
Starting to save.
We're done now.
Completed in: 8m:22s:47ms
I even had a previous run, before tracking the time that was around 6 mins and this is always the same data (just changing the PK of
the root).
I will try your suggestions here, but I am still concerned. This is our first and almost simplest model. There will be 4-5 others as
well.
The issue is that for every project we handle we will have millions of records. Right now we are getting much better performance
with an XMI serializer / ZIP compressor / XMI de-serializer, but I feel that when we start to want to perform operations that cross
instance boundaries, that this is where that solution won't scale. We've also looked at other persistence approach (OODB like db4o,
XML storage with Db/2, etc). I'd like to see how others are addressing this performance challenge, which I understand is not really
caused by Teneo here.
I'll keep you all posted on the results we are getting by applying all of your suggestions, and keep them coming.
Cheers,
Alain
Martin Taal wrote:
> Hi Alain,
> Performance is a always a challenge. The main project in which I use
> Teneo/Hibernate has about 125000 records most of them consisting of
> about 10 fields. An insert of all these objects together with some
> conversion takes about 5 to 10 minutes.
> One setting you can set is the jdbc batch size.
>
> See here:
> http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml
>
> Also it is my experience that inserting/updating many objects in one
> transactions slows down the database as the database transaction log
> becomes very large. So a commit after every 10000 objects can result in
> a better performance (but you loose real transaction behavior). Or you
> can try autocommit is true.
>
> gr. Martin
>
> Benjamin CABE wrote:
>> Hi Alain,
>>
>> In my case -Hibernate too-, the ratio between log (with a "default"
>> log4j.properties) and no-log version is someting like 3-4x... Not
>> negligible :)
>>
>> Benjamin.
>>
>> Alain Picard a écrit :
>>> Ed,
>>>
>>> I am using Hibernate here. Let me try again without the debug log.
>>>
>>> Alain
>
>
|
|
|
Re: [Teneo] Model persistence performance [message #92419 is a reply to message #92403] |
Wed, 01 August 2007 21:45 |
Alain Picard Messages: 266 Registered: July 2009 |
Senior Member |
|
|
Well, here is for result #1 of increasing the hibernate batch size.
Starting to save.
We're done now.
Completed in: 8m:17s:875ms
There is essentially no improvement. I did find that this is more than likely related to the fact that we are using a joined strategy.
See http://opensource.atlassian.com/projects/hibernate/browse/HH H-2558
Excerpt:
<joined-subclass/> as well as entity's containing <join/> mappings currently cannot participating in JDBC batching. The reaons being
that the actions (i.e. Executables) perform the insert or update in an atomic fashion through the persisters; the persisters perform
the multiple statements.
Well, I guess I might need to look at implementing a different strategy and testing that.
Cheers,
Alain
Alain Picard wrote:
> Well, I have turned off the debugging and it did improve dramatically.
>
> Here the new results:
> Starting to save.
> We're done now.
> Completed in: 8m:22s:47ms
>
> I even had a previous run, before tracking the time that was around 6
> mins and this is always the same data (just changing the PK of the root).
>
> I will try your suggestions here, but I am still concerned. This is our
> first and almost simplest model. There will be 4-5 others as well.
>
> The issue is that for every project we handle we will have millions of
> records. Right now we are getting much better performance with an XMI
> serializer / ZIP compressor / XMI de-serializer, but I feel that when we
> start to want to perform operations that cross instance boundaries, that
> this is where that solution won't scale. We've also looked at other
> persistence approach (OODB like db4o, XML storage with Db/2, etc). I'd
> like to see how others are addressing this performance challenge, which
> I understand is not really caused by Teneo here.
>
> I'll keep you all posted on the results we are getting by applying all
> of your suggestions, and keep them coming.
>
> Cheers,
> Alain
>
>
> Martin Taal wrote:
>> Hi Alain,
>> Performance is a always a challenge. The main project in which I use
>> Teneo/Hibernate has about 125000 records most of them consisting of
>> about 10 fields. An insert of all these objects together with some
>> conversion takes about 5 to 10 minutes.
>> One setting you can set is the jdbc batch size.
>>
>> See here:
>> http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml
>>
>> Also it is my experience that inserting/updating many objects in one
>> transactions slows down the database as the database transaction log
>> becomes very large. So a commit after every 10000 objects can result
>> in a better performance (but you loose real transaction behavior). Or
>> you can try autocommit is true.
>>
>> gr. Martin
>>
>> Benjamin CABE wrote:
>>> Hi Alain,
>>>
>>> In my case -Hibernate too-, the ratio between log (with a "default"
>>> log4j.properties) and no-log version is someting like 3-4x... Not
>>> negligible :)
>>>
>>> Benjamin.
>>>
>>> Alain Picard a écrit :
>>>> Ed,
>>>>
>>>> I am using Hibernate here. Let me try again without the debug log.
>>>>
>>>> Alain
>>
>>
|
|
|
Re: [Teneo] Model persistence performance [message #92616 is a reply to message #92388] |
Sun, 05 August 2007 20:43 |
Alain Picard Messages: 266 Registered: July 2009 |
Senior Member |
|
|
Martin,
I have made more experiment by switching everything to JPOX and trying the same tests.
The results are essentially the same, from a few seconds less to a few seconds more. What is interesting though is that the CPU
usage pattern is very different. The Eclipse application used to take about 50% of CPU and MySql 2-3%. With JPOX, both together take
about 25-30% of CPU, with MySql actually taking a bit more than Eclipse. The other difference here is that the disk is constantly
being accessed, which was not the case with Hibernate.
I then realized that the Enums were saved as Medium blobs, which could account for a lot of the disk I/O and possibly some
performance degradation. I tried to switch to JPOX 2 with the Java5 plugin to try to fix this.
The issue is that I am getting the following exception:
org.jpox.exceptions.JPOXUserException: Level 1 Cache "org.eclipse.emf.teneo.jpox.cache.EMFWeakRefCache" is not registered. Please
check your CLASSPATH and specification.
Based on your reply on July 18th, regarding this issue I tried to disable the cache for now doing the following:
Properties props = jpoxDataStore.getProperties();
props.remove(PersistenceConfiguration.CACHE_LEVEL_1_TYPE_PRO PERTY);
but that still gives me the error. What is the correct way to cancel the cache and/or to specify the plugin.xml that JPOX requires?
I was also thinking of trying the db4o extension to see if that gives me better results? Have you tried to use it so far?
And what about using JDO outside of JPOX, should that work? I am thinking in terms of some other OODBMS, here.
Otherwise I am still open to other solutions or approaches, like using batch inserts or whatever.
Thanks
Alain
Martin Taal wrote:
> Hi Alain,
> Performance is a always a challenge. The main project in which I use
> Teneo/Hibernate has about 125000 records most of them consisting of
> about 10 fields. An insert of all these objects together with some
> conversion takes about 5 to 10 minutes.
> One setting you can set is the jdbc batch size.
>
> See here:
> http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml
>
> Also it is my experience that inserting/updating many objects in one
> transactions slows down the database as the database transaction log
> becomes very large. So a commit after every 10000 objects can result in
> a better performance (but you loose real transaction behavior). Or you
> can try autocommit is true.
>
> gr. Martin
>
> Benjamin CABE wrote:
>> Hi Alain,
>>
>> In my case -Hibernate too-, the ratio between log (with a "default"
>> log4j.properties) and no-log version is someting like 3-4x... Not
>> negligible :)
>>
>> Benjamin.
>>
>> Alain Picard a écrit :
>>> Ed,
>>>
>>> I am using Hibernate here. Let me try again without the debug log.
>>>
>>> Alain
>
>
|
|
|
Re: [Teneo] Model persistence performance [message #92637 is a reply to message #92616] |
Sun, 05 August 2007 21:05 |
Martin Taal Messages: 5468 Registered: July 2009 |
Senior Member |
|
|
Hi Alain,
Teneo only works with jpox 1.1.8. I have tried jpox 1.2 but could not get all the test cases
running. I will try again when jpox 1.2 final is releases. Also for jpox 1.1.8 enums should be saved
as a string/int. If this is not happening then I can look at that. Can you confirm?
Are you using a level 2 cache? If so then this can cost some performance.
I have also noticed when using hibernate that hibernate takes a lot of cpu compared to the db. I did
not have time to check out what was happening. If you have a testcase which I can try then I am
interested to try that out and see what happens (my case was not easily reproducable).
What is your current run-time for the insert of 90.000 records?
I have not tried db4o. I am not sure but afaiu this db does not support foreign key constraint. I
have not tried jdo other than jpox. Eventhough jdo is a standard spec Teneo uses jpox specific
extensions. I am almost 100% sure that Teneo will not work out of the box with another jdo supplier.
gr. Martin
Alain Picard wrote:
> Martin,
>
> I have made more experiment by switching everything to JPOX and trying
> the same tests.
>
> The results are essentially the same, from a few seconds less to a few
> seconds more. What is interesting though is that the CPU usage pattern
> is very different. The Eclipse application used to take about 50% of CPU
> and MySql 2-3%. With JPOX, both together take about 25-30% of CPU, with
> MySql actually taking a bit more than Eclipse. The other difference here
> is that the disk is constantly being accessed, which was not the case
> with Hibernate.
>
> I then realized that the Enums were saved as Medium blobs, which could
> account for a lot of the disk I/O and possibly some performance
> degradation. I tried to switch to JPOX 2 with the Java5 plugin to try to
> fix this.
>
> The issue is that I am getting the following exception:
> org.jpox.exceptions.JPOXUserException: Level 1 Cache
> "org.eclipse.emf.teneo.jpox.cache.EMFWeakRefCache" is not registered.
> Please check your CLASSPATH and specification.
>
> Based on your reply on July 18th, regarding this issue I tried to
> disable the cache for now doing the following:
> Properties props = jpoxDataStore.getProperties();
> props.remove(PersistenceConfiguration.CACHE_LEVEL_1_TYPE_PRO PERTY);
>
> but that still gives me the error. What is the correct way to cancel the
> cache and/or to specify the plugin.xml that JPOX requires?
>
> I was also thinking of trying the db4o extension to see if that gives me
> better results? Have you tried to use it so far?
>
> And what about using JDO outside of JPOX, should that work? I am
> thinking in terms of some other OODBMS, here.
>
> Otherwise I am still open to other solutions or approaches, like using
> batch inserts or whatever.
>
> Thanks
> Alain
>
> Martin Taal wrote:
>> Hi Alain,
>> Performance is a always a challenge. The main project in which I use
>> Teneo/Hibernate has about 125000 records most of them consisting of
>> about 10 fields. An insert of all these objects together with some
>> conversion takes about 5 to 10 minutes.
>> One setting you can set is the jdbc batch size.
>>
>> See here:
>> http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml
>>
>> Also it is my experience that inserting/updating many objects in one
>> transactions slows down the database as the database transaction log
>> becomes very large. So a commit after every 10000 objects can result
>> in a better performance (but you loose real transaction behavior). Or
>> you can try autocommit is true.
>>
>> gr. Martin
>>
>> Benjamin CABE wrote:
>>> Hi Alain,
>>>
>>> In my case -Hibernate too-, the ratio between log (with a "default"
>>> log4j.properties) and no-log version is someting like 3-4x... Not
>>> negligible :)
>>>
>>> Benjamin.
>>>
>>> Alain Picard a écrit :
>>>> Ed,
>>>>
>>>> I am using Hibernate here. Let me try again without the debug log.
>>>>
>>>> Alain
>>
>>
b
--
With Regards, Martin Taal
Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
|
|
|
Re: [Teneo] Model persistence performance [message #92652 is a reply to message #92637] |
Sun, 05 August 2007 21:43 |
Alain Picard Messages: 266 Registered: July 2009 |
Senior Member |
|
|
Martin,
I can confirm that I am getting eenum columns to show up as mediumblob type in MySql and here is the mapping for one of those:
<field name="formattingType" persistence-modifier="persistent" null-value="none" embedded="true"/>
I am not using any level 2 cache.
The current time is still around 8 mins. I tried to insert (with hibernate) the tree in section, but right now I am facing an issue
where it tells me that I am missing a required value, even if it is supplied. Not really sure why, but the whole exercise seems to
be a bit awkward also. I might have to relax the integrity rule in order to insert data "gradually".
Thanks
Alain
Martin Taal wrote:
> Hi Alain,
> Teneo only works with jpox 1.1.8. I have tried jpox 1.2 but could not
> get all the test cases running. I will try again when jpox 1.2 final is
> releases. Also for jpox 1.1.8 enums should be saved as a string/int. If
> this is not happening then I can look at that. Can you confirm?
>
> Are you using a level 2 cache? If so then this can cost some performance.
>
> I have also noticed when using hibernate that hibernate takes a lot of
> cpu compared to the db. I did not have time to check out what was
> happening. If you have a testcase which I can try then I am interested
> to try that out and see what happens (my case was not easily reproducable).
>
> What is your current run-time for the insert of 90.000 records?
>
> I have not tried db4o. I am not sure but afaiu this db does not support
> foreign key constraint. I have not tried jdo other than jpox. Eventhough
> jdo is a standard spec Teneo uses jpox specific extensions. I am almost
> 100% sure that Teneo will not work out of the box with another jdo
> supplier.
>
> gr. Martin
>
>
> Alain Picard wrote:
>> Martin,
>>
>> I have made more experiment by switching everything to JPOX and trying
>> the same tests.
>>
>> The results are essentially the same, from a few seconds less to a few
>> seconds more. What is interesting though is that the CPU usage pattern
>> is very different. The Eclipse application used to take about 50% of
>> CPU and MySql 2-3%. With JPOX, both together take about 25-30% of
>> CPU, with MySql actually taking a bit more than Eclipse. The other
>> difference here is that the disk is constantly being accessed, which
>> was not the case with Hibernate.
>>
>> I then realized that the Enums were saved as Medium blobs, which could
>> account for a lot of the disk I/O and possibly some performance
>> degradation. I tried to switch to JPOX 2 with the Java5 plugin to try
>> to fix this.
>>
>> The issue is that I am getting the following exception:
>> org.jpox.exceptions.JPOXUserException: Level 1 Cache
>> "org.eclipse.emf.teneo.jpox.cache.EMFWeakRefCache" is not registered.
>> Please check your CLASSPATH and specification.
>>
>> Based on your reply on July 18th, regarding this issue I tried to
>> disable the cache for now doing the following:
>> Properties props = jpoxDataStore.getProperties();
>>
>> props.remove(PersistenceConfiguration.CACHE_LEVEL_1_TYPE_PRO PERTY);
>>
>> but that still gives me the error. What is the correct way to cancel
>> the cache and/or to specify the plugin.xml that JPOX requires?
>>
>> I was also thinking of trying the db4o extension to see if that gives
>> me better results? Have you tried to use it so far?
>>
>> And what about using JDO outside of JPOX, should that work? I am
>> thinking in terms of some other OODBMS, here.
>>
>> Otherwise I am still open to other solutions or approaches, like using
>> batch inserts or whatever.
>>
>> Thanks
>> Alain
>>
>> Martin Taal wrote:
>>> Hi Alain,
>>> Performance is a always a challenge. The main project in which I use
>>> Teneo/Hibernate has about 125000 records most of them consisting of
>>> about 10 fields. An insert of all these objects together with some
>>> conversion takes about 5 to 10 minutes.
>>> One setting you can set is the jdbc batch size.
>>>
>>> See here:
>>> http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml
>>>
>>> Also it is my experience that inserting/updating many objects in one
>>> transactions slows down the database as the database transaction log
>>> becomes very large. So a commit after every 10000 objects can result
>>> in a better performance (but you loose real transaction behavior). Or
>>> you can try autocommit is true.
>>>
>>> gr. Martin
>>>
>>> Benjamin CABE wrote:
>>>> Hi Alain,
>>>>
>>>> In my case -Hibernate too-, the ratio between log (with a "default"
>>>> log4j.properties) and no-log version is someting like 3-4x... Not
>>>> negligible :)
>>>>
>>>> Benjamin.
>>>>
>>>> Alain Picard a écrit :
>>>>> Ed,
>>>>>
>>>>> I am using Hibernate here. Let me try again without the debug log.
>>>>>
>>>>> Alain
>>>
>>>
> b
>
|
|
| | |
Re: [Teneo] Model persistence performance [message #609406 is a reply to message #92312] |
Wed, 01 August 2007 17:17 |
Ed Merks Messages: 33216 Registered: July 2009 |
Senior Member |
|
|
Alain,
I know I've had folks complain about performance and have it turn out to
be a single println statement that the person thought was quite
innocuous so it would definitely be good to run your scenario without
spewing out almost 100 times more debug/log data than you'd get just for
the XMI serialization of it. I suppose it would also be good to know if
you are using Hibernate verses JPOX...
Alain Picard wrote:
> Now that I have my model persistence configured and running with small
> test cases, I decided to use it with a "standard" size case.
>
> Well, it took me 90 mins to persist the model to the local MySQL
> database (with debugging on, 600 Megs log file). The model is made up
> of 9 tables and this particular model instance contains about 90k
> records altogether. The equivalent XMI is about 9.5Megs and compresses
> to 229k with zip.
>
> For the most part, all of the records are quite small, mostly integer
> and bigint values, with a few varchar values.
>
> I understand that running without a debug file would surely help
> somewhat, but anything over 30 seconds here surely won't make the grade.
>
> What are other peoples experiences with the performance of persisting
> large EMF models ?
>
> For us this is a small/medium model instance. Many will be 5-10 times
> the size of this one.
>
> Thanks for the feedback.
>
> Alain
Ed Merks
Professional Support: https://www.macromodeling.com/
|
|
|
Re: [Teneo] Model persistence performance [message #609408 is a reply to message #92328] |
Wed, 01 August 2007 17:27 |
Alain Picard Messages: 266 Registered: July 2009 |
Senior Member |
|
|
Ed,
I am using Hibernate here. Let me try again without the debug log.
Alain
Ed Merks wrote:
> Alain,
>
> I know I've had folks complain about performance and have it turn out to
> be a single println statement that the person thought was quite
> innocuous so it would definitely be good to run your scenario without
> spewing out almost 100 times more debug/log data than you'd get just for
> the XMI serialization of it. I suppose it would also be good to know if
> you are using Hibernate verses JPOX...
>
>
> Alain Picard wrote:
>> Now that I have my model persistence configured and running with small
>> test cases, I decided to use it with a "standard" size case.
>>
>> Well, it took me 90 mins to persist the model to the local MySQL
>> database (with debugging on, 600 Megs log file). The model is made up
>> of 9 tables and this particular model instance contains about 90k
>> records altogether. The equivalent XMI is about 9.5Megs and compresses
>> to 229k with zip.
>>
>> For the most part, all of the records are quite small, mostly integer
>> and bigint values, with a few varchar values.
>>
>> I understand that running without a debug file would surely help
>> somewhat, but anything over 30 seconds here surely won't make the grade.
>>
>> What are other peoples experiences with the performance of persisting
>> large EMF models ?
>>
>> For us this is a small/medium model instance. Many will be 5-10 times
>> the size of this one.
>>
>> Thanks for the feedback.
>>
>> Alain
|
|
|
Re: [Teneo] Model persistence performance [message #609409 is a reply to message #92359] |
Wed, 01 August 2007 18:57 |
Eclipse User |
|
|
|
Originally posted by: benjamin.cabeNOSPAMM.anyware-tech.DOTcom
Hi Alain,
In my case -Hibernate too-, the ratio between log (with a "default"
log4j.properties) and no-log version is someting like 3-4x... Not
negligible :)
Benjamin.
Alain Picard a écrit :
> Ed,
>
> I am using Hibernate here. Let me try again without the debug log.
>
> Alain
|
|
| |
Re: [Teneo] Model persistence performance [message #609411 is a reply to message #92388] |
Wed, 01 August 2007 21:05 |
Alain Picard Messages: 266 Registered: July 2009 |
Senior Member |
|
|
Well, I have turned off the debugging and it did improve dramatically.
Here the new results:
Starting to save.
We're done now.
Completed in: 8m:22s:47ms
I even had a previous run, before tracking the time that was around 6 mins and this is always the same data (just changing the PK of
the root).
I will try your suggestions here, but I am still concerned. This is our first and almost simplest model. There will be 4-5 others as
well.
The issue is that for every project we handle we will have millions of records. Right now we are getting much better performance
with an XMI serializer / ZIP compressor / XMI de-serializer, but I feel that when we start to want to perform operations that cross
instance boundaries, that this is where that solution won't scale. We've also looked at other persistence approach (OODB like db4o,
XML storage with Db/2, etc). I'd like to see how others are addressing this performance challenge, which I understand is not really
caused by Teneo here.
I'll keep you all posted on the results we are getting by applying all of your suggestions, and keep them coming.
Cheers,
Alain
Martin Taal wrote:
> Hi Alain,
> Performance is a always a challenge. The main project in which I use
> Teneo/Hibernate has about 125000 records most of them consisting of
> about 10 fields. An insert of all these objects together with some
> conversion takes about 5 to 10 minutes.
> One setting you can set is the jdbc batch size.
>
> See here:
> http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml
>
> Also it is my experience that inserting/updating many objects in one
> transactions slows down the database as the database transaction log
> becomes very large. So a commit after every 10000 objects can result in
> a better performance (but you loose real transaction behavior). Or you
> can try autocommit is true.
>
> gr. Martin
>
> Benjamin CABE wrote:
>> Hi Alain,
>>
>> In my case -Hibernate too-, the ratio between log (with a "default"
>> log4j.properties) and no-log version is someting like 3-4x... Not
>> negligible :)
>>
>> Benjamin.
>>
>> Alain Picard a écrit :
>>> Ed,
>>>
>>> I am using Hibernate here. Let me try again without the debug log.
>>>
>>> Alain
>
>
|
|
|
Re: [Teneo] Model persistence performance [message #609412 is a reply to message #92403] |
Wed, 01 August 2007 21:45 |
Alain Picard Messages: 266 Registered: July 2009 |
Senior Member |
|
|
Well, here is for result #1 of increasing the hibernate batch size.
Starting to save.
We're done now.
Completed in: 8m:17s:875ms
There is essentially no improvement. I did find that this is more than likely related to the fact that we are using a joined strategy.
See http://opensource.atlassian.com/projects/hibernate/browse/HH H-2558
Excerpt:
<joined-subclass/> as well as entity's containing <join/> mappings currently cannot participating in JDBC batching. The reaons being
that the actions (i.e. Executables) perform the insert or update in an atomic fashion through the persisters; the persisters perform
the multiple statements.
Well, I guess I might need to look at implementing a different strategy and testing that.
Cheers,
Alain
Alain Picard wrote:
> Well, I have turned off the debugging and it did improve dramatically.
>
> Here the new results:
> Starting to save.
> We're done now.
> Completed in: 8m:22s:47ms
>
> I even had a previous run, before tracking the time that was around 6
> mins and this is always the same data (just changing the PK of the root).
>
> I will try your suggestions here, but I am still concerned. This is our
> first and almost simplest model. There will be 4-5 others as well.
>
> The issue is that for every project we handle we will have millions of
> records. Right now we are getting much better performance with an XMI
> serializer / ZIP compressor / XMI de-serializer, but I feel that when we
> start to want to perform operations that cross instance boundaries, that
> this is where that solution won't scale. We've also looked at other
> persistence approach (OODB like db4o, XML storage with Db/2, etc). I'd
> like to see how others are addressing this performance challenge, which
> I understand is not really caused by Teneo here.
>
> I'll keep you all posted on the results we are getting by applying all
> of your suggestions, and keep them coming.
>
> Cheers,
> Alain
>
>
> Martin Taal wrote:
>> Hi Alain,
>> Performance is a always a challenge. The main project in which I use
>> Teneo/Hibernate has about 125000 records most of them consisting of
>> about 10 fields. An insert of all these objects together with some
>> conversion takes about 5 to 10 minutes.
>> One setting you can set is the jdbc batch size.
>>
>> See here:
>> http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml
>>
>> Also it is my experience that inserting/updating many objects in one
>> transactions slows down the database as the database transaction log
>> becomes very large. So a commit after every 10000 objects can result
>> in a better performance (but you loose real transaction behavior). Or
>> you can try autocommit is true.
>>
>> gr. Martin
>>
>> Benjamin CABE wrote:
>>> Hi Alain,
>>>
>>> In my case -Hibernate too-, the ratio between log (with a "default"
>>> log4j.properties) and no-log version is someting like 3-4x... Not
>>> negligible :)
>>>
>>> Benjamin.
>>>
>>> Alain Picard a écrit :
>>>> Ed,
>>>>
>>>> I am using Hibernate here. Let me try again without the debug log.
>>>>
>>>> Alain
>>
>>
|
|
|
Re: [Teneo] Model persistence performance [message #609425 is a reply to message #92388] |
Sun, 05 August 2007 20:43 |
Alain Picard Messages: 266 Registered: July 2009 |
Senior Member |
|
|
Martin,
I have made more experiment by switching everything to JPOX and trying the same tests.
The results are essentially the same, from a few seconds less to a few seconds more. What is interesting though is that the CPU
usage pattern is very different. The Eclipse application used to take about 50% of CPU and MySql 2-3%. With JPOX, both together take
about 25-30% of CPU, with MySql actually taking a bit more than Eclipse. The other difference here is that the disk is constantly
being accessed, which was not the case with Hibernate.
I then realized that the Enums were saved as Medium blobs, which could account for a lot of the disk I/O and possibly some
performance degradation. I tried to switch to JPOX 2 with the Java5 plugin to try to fix this.
The issue is that I am getting the following exception:
org.jpox.exceptions.JPOXUserException: Level 1 Cache "org.eclipse.emf.teneo.jpox.cache.EMFWeakRefCache" is not registered. Please
check your CLASSPATH and specification.
Based on your reply on July 18th, regarding this issue I tried to disable the cache for now doing the following:
Properties props = jpoxDataStore.getProperties();
props.remove(PersistenceConfiguration.CACHE_LEVEL_1_TYPE_PRO PERTY);
but that still gives me the error. What is the correct way to cancel the cache and/or to specify the plugin.xml that JPOX requires?
I was also thinking of trying the db4o extension to see if that gives me better results? Have you tried to use it so far?
And what about using JDO outside of JPOX, should that work? I am thinking in terms of some other OODBMS, here.
Otherwise I am still open to other solutions or approaches, like using batch inserts or whatever.
Thanks
Alain
Martin Taal wrote:
> Hi Alain,
> Performance is a always a challenge. The main project in which I use
> Teneo/Hibernate has about 125000 records most of them consisting of
> about 10 fields. An insert of all these objects together with some
> conversion takes about 5 to 10 minutes.
> One setting you can set is the jdbc batch size.
>
> See here:
> http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml
>
> Also it is my experience that inserting/updating many objects in one
> transactions slows down the database as the database transaction log
> becomes very large. So a commit after every 10000 objects can result in
> a better performance (but you loose real transaction behavior). Or you
> can try autocommit is true.
>
> gr. Martin
>
> Benjamin CABE wrote:
>> Hi Alain,
>>
>> In my case -Hibernate too-, the ratio between log (with a "default"
>> log4j.properties) and no-log version is someting like 3-4x... Not
>> negligible :)
>>
>> Benjamin.
>>
>> Alain Picard a écrit :
>>> Ed,
>>>
>>> I am using Hibernate here. Let me try again without the debug log.
>>>
>>> Alain
>
>
|
|
|
Re: [Teneo] Model persistence performance [message #609426 is a reply to message #92616] |
Sun, 05 August 2007 21:05 |
Martin Taal Messages: 5468 Registered: July 2009 |
Senior Member |
|
|
Hi Alain,
Teneo only works with jpox 1.1.8. I have tried jpox 1.2 but could not get all the test cases
running. I will try again when jpox 1.2 final is releases. Also for jpox 1.1.8 enums should be saved
as a string/int. If this is not happening then I can look at that. Can you confirm?
Are you using a level 2 cache? If so then this can cost some performance.
I have also noticed when using hibernate that hibernate takes a lot of cpu compared to the db. I did
not have time to check out what was happening. If you have a testcase which I can try then I am
interested to try that out and see what happens (my case was not easily reproducable).
What is your current run-time for the insert of 90.000 records?
I have not tried db4o. I am not sure but afaiu this db does not support foreign key constraint. I
have not tried jdo other than jpox. Eventhough jdo is a standard spec Teneo uses jpox specific
extensions. I am almost 100% sure that Teneo will not work out of the box with another jdo supplier.
gr. Martin
Alain Picard wrote:
> Martin,
>
> I have made more experiment by switching everything to JPOX and trying
> the same tests.
>
> The results are essentially the same, from a few seconds less to a few
> seconds more. What is interesting though is that the CPU usage pattern
> is very different. The Eclipse application used to take about 50% of CPU
> and MySql 2-3%. With JPOX, both together take about 25-30% of CPU, with
> MySql actually taking a bit more than Eclipse. The other difference here
> is that the disk is constantly being accessed, which was not the case
> with Hibernate.
>
> I then realized that the Enums were saved as Medium blobs, which could
> account for a lot of the disk I/O and possibly some performance
> degradation. I tried to switch to JPOX 2 with the Java5 plugin to try to
> fix this.
>
> The issue is that I am getting the following exception:
> org.jpox.exceptions.JPOXUserException: Level 1 Cache
> "org.eclipse.emf.teneo.jpox.cache.EMFWeakRefCache" is not registered.
> Please check your CLASSPATH and specification.
>
> Based on your reply on July 18th, regarding this issue I tried to
> disable the cache for now doing the following:
> Properties props = jpoxDataStore.getProperties();
> props.remove(PersistenceConfiguration.CACHE_LEVEL_1_TYPE_PRO PERTY);
>
> but that still gives me the error. What is the correct way to cancel the
> cache and/or to specify the plugin.xml that JPOX requires?
>
> I was also thinking of trying the db4o extension to see if that gives me
> better results? Have you tried to use it so far?
>
> And what about using JDO outside of JPOX, should that work? I am
> thinking in terms of some other OODBMS, here.
>
> Otherwise I am still open to other solutions or approaches, like using
> batch inserts or whatever.
>
> Thanks
> Alain
>
> Martin Taal wrote:
>> Hi Alain,
>> Performance is a always a challenge. The main project in which I use
>> Teneo/Hibernate has about 125000 records most of them consisting of
>> about 10 fields. An insert of all these objects together with some
>> conversion takes about 5 to 10 minutes.
>> One setting you can set is the jdbc batch size.
>>
>> See here:
>> http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml
>>
>> Also it is my experience that inserting/updating many objects in one
>> transactions slows down the database as the database transaction log
>> becomes very large. So a commit after every 10000 objects can result
>> in a better performance (but you loose real transaction behavior). Or
>> you can try autocommit is true.
>>
>> gr. Martin
>>
>> Benjamin CABE wrote:
>>> Hi Alain,
>>>
>>> In my case -Hibernate too-, the ratio between log (with a "default"
>>> log4j.properties) and no-log version is someting like 3-4x... Not
>>> negligible :)
>>>
>>> Benjamin.
>>>
>>> Alain Picard a écrit :
>>>> Ed,
>>>>
>>>> I am using Hibernate here. Let me try again without the debug log.
>>>>
>>>> Alain
>>
>>
b
--
With Regards, Martin Taal
Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
|
|
|
Re: [Teneo] Model persistence performance [message #609427 is a reply to message #92637] |
Sun, 05 August 2007 21:43 |
Alain Picard Messages: 266 Registered: July 2009 |
Senior Member |
|
|
Martin,
I can confirm that I am getting eenum columns to show up as mediumblob type in MySql and here is the mapping for one of those:
<field name="formattingType" persistence-modifier="persistent" null-value="none" embedded="true"/>
I am not using any level 2 cache.
The current time is still around 8 mins. I tried to insert (with hibernate) the tree in section, but right now I am facing an issue
where it tells me that I am missing a required value, even if it is supplied. Not really sure why, but the whole exercise seems to
be a bit awkward also. I might have to relax the integrity rule in order to insert data "gradually".
Thanks
Alain
Martin Taal wrote:
> Hi Alain,
> Teneo only works with jpox 1.1.8. I have tried jpox 1.2 but could not
> get all the test cases running. I will try again when jpox 1.2 final is
> releases. Also for jpox 1.1.8 enums should be saved as a string/int. If
> this is not happening then I can look at that. Can you confirm?
>
> Are you using a level 2 cache? If so then this can cost some performance.
>
> I have also noticed when using hibernate that hibernate takes a lot of
> cpu compared to the db. I did not have time to check out what was
> happening. If you have a testcase which I can try then I am interested
> to try that out and see what happens (my case was not easily reproducable).
>
> What is your current run-time for the insert of 90.000 records?
>
> I have not tried db4o. I am not sure but afaiu this db does not support
> foreign key constraint. I have not tried jdo other than jpox. Eventhough
> jdo is a standard spec Teneo uses jpox specific extensions. I am almost
> 100% sure that Teneo will not work out of the box with another jdo
> supplier.
>
> gr. Martin
>
>
> Alain Picard wrote:
>> Martin,
>>
>> I have made more experiment by switching everything to JPOX and trying
>> the same tests.
>>
>> The results are essentially the same, from a few seconds less to a few
>> seconds more. What is interesting though is that the CPU usage pattern
>> is very different. The Eclipse application used to take about 50% of
>> CPU and MySql 2-3%. With JPOX, both together take about 25-30% of
>> CPU, with MySql actually taking a bit more than Eclipse. The other
>> difference here is that the disk is constantly being accessed, which
>> was not the case with Hibernate.
>>
>> I then realized that the Enums were saved as Medium blobs, which could
>> account for a lot of the disk I/O and possibly some performance
>> degradation. I tried to switch to JPOX 2 with the Java5 plugin to try
>> to fix this.
>>
>> The issue is that I am getting the following exception:
>> org.jpox.exceptions.JPOXUserException: Level 1 Cache
>> "org.eclipse.emf.teneo.jpox.cache.EMFWeakRefCache" is not registered.
>> Please check your CLASSPATH and specification.
>>
>> Based on your reply on July 18th, regarding this issue I tried to
>> disable the cache for now doing the following:
>> Properties props = jpoxDataStore.getProperties();
>>
>> props.remove(PersistenceConfiguration.CACHE_LEVEL_1_TYPE_PRO PERTY);
>>
>> but that still gives me the error. What is the correct way to cancel
>> the cache and/or to specify the plugin.xml that JPOX requires?
>>
>> I was also thinking of trying the db4o extension to see if that gives
>> me better results? Have you tried to use it so far?
>>
>> And what about using JDO outside of JPOX, should that work? I am
>> thinking in terms of some other OODBMS, here.
>>
>> Otherwise I am still open to other solutions or approaches, like using
>> batch inserts or whatever.
>>
>> Thanks
>> Alain
>>
>> Martin Taal wrote:
>>> Hi Alain,
>>> Performance is a always a challenge. The main project in which I use
>>> Teneo/Hibernate has about 125000 records most of them consisting of
>>> about 10 fields. An insert of all these objects together with some
>>> conversion takes about 5 to 10 minutes.
>>> One setting you can set is the jdbc batch size.
>>>
>>> See here:
>>> http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml
>>>
>>> Also it is my experience that inserting/updating many objects in one
>>> transactions slows down the database as the database transaction log
>>> becomes very large. So a commit after every 10000 objects can result
>>> in a better performance (but you loose real transaction behavior). Or
>>> you can try autocommit is true.
>>>
>>> gr. Martin
>>>
>>> Benjamin CABE wrote:
>>>> Hi Alain,
>>>>
>>>> In my case -Hibernate too-, the ratio between log (with a "default"
>>>> log4j.properties) and no-log version is someting like 3-4x... Not
>>>> negligible :)
>>>>
>>>> Benjamin.
>>>>
>>>> Alain Picard a écrit :
>>>>> Ed,
>>>>>
>>>>> I am using Hibernate here. Let me try again without the debug log.
>>>>>
>>>>> Alain
>>>
>>>
> b
>
|
|
| | |
Goto Forum:
Current Time: Fri Sep 20 15:31:41 GMT 2024
Powered by FUDForum. Page generated in 0.05006 seconds
|