Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF "Technology" (Ecore Tools, EMFatic, etc)  » [Teneo] Model persistence performance
[Teneo] Model persistence performance [message #92312] Wed, 01 August 2007 13:05 Go to next message
Alain Picard is currently offline Alain Picard
Messages: 176
Registered: July 2009
Senior Member
Now that I have my model persistence configured and running with small test cases, I decided to use it with a "standard" size case.

Well, it took me 90 mins to persist the model to the local MySQL database (with debugging on, 600 Megs log file). The model is made
up of 9 tables and this particular model instance contains about 90k records altogether. The equivalent XMI is about 9.5Megs and
compresses to 229k with zip.

For the most part, all of the records are quite small, mostly integer and bigint values, with a few varchar values.

I understand that running without a debug file would surely help somewhat, but anything over 30 seconds here surely won't make the
grade.

What are other peoples experiences with the performance of persisting large EMF models ?

For us this is a small/medium model instance. Many will be 5-10 times the size of this one.

Thanks for the feedback.

Alain
Re: [Teneo] Model persistence performance [message #92328 is a reply to message #92312] Wed, 01 August 2007 13:17 Go to previous messageGo to next message
Eclipse User
Originally posted by: merks.ca.ibm.com

Alain,

I know I've had folks complain about performance and have it turn out to
be a single println statement that the person thought was quite
innocuous so it would definitely be good to run your scenario without
spewing out almost 100 times more debug/log data than you'd get just for
the XMI serialization of it. I suppose it would also be good to know if
you are using Hibernate verses JPOX...


Alain Picard wrote:
> Now that I have my model persistence configured and running with small
> test cases, I decided to use it with a "standard" size case.
>
> Well, it took me 90 mins to persist the model to the local MySQL
> database (with debugging on, 600 Megs log file). The model is made up
> of 9 tables and this particular model instance contains about 90k
> records altogether. The equivalent XMI is about 9.5Megs and compresses
> to 229k with zip.
>
> For the most part, all of the records are quite small, mostly integer
> and bigint values, with a few varchar values.
>
> I understand that running without a debug file would surely help
> somewhat, but anything over 30 seconds here surely won't make the grade.
>
> What are other peoples experiences with the performance of persisting
> large EMF models ?
>
> For us this is a small/medium model instance. Many will be 5-10 times
> the size of this one.
>
> Thanks for the feedback.
>
> Alain
Re: [Teneo] Model persistence performance [message #92359 is a reply to message #92328] Wed, 01 August 2007 13:27 Go to previous messageGo to next message
Alain Picard is currently offline Alain Picard
Messages: 176
Registered: July 2009
Senior Member
Ed,

I am using Hibernate here. Let me try again without the debug log.

Alain

Ed Merks wrote:
> Alain,
>
> I know I've had folks complain about performance and have it turn out to
> be a single println statement that the person thought was quite
> innocuous so it would definitely be good to run your scenario without
> spewing out almost 100 times more debug/log data than you'd get just for
> the XMI serialization of it. I suppose it would also be good to know if
> you are using Hibernate verses JPOX...
>
>
> Alain Picard wrote:
>> Now that I have my model persistence configured and running with small
>> test cases, I decided to use it with a "standard" size case.
>>
>> Well, it took me 90 mins to persist the model to the local MySQL
>> database (with debugging on, 600 Megs log file). The model is made up
>> of 9 tables and this particular model instance contains about 90k
>> records altogether. The equivalent XMI is about 9.5Megs and compresses
>> to 229k with zip.
>>
>> For the most part, all of the records are quite small, mostly integer
>> and bigint values, with a few varchar values.
>>
>> I understand that running without a debug file would surely help
>> somewhat, but anything over 30 seconds here surely won't make the grade.
>>
>> What are other peoples experiences with the performance of persisting
>> large EMF models ?
>>
>> For us this is a small/medium model instance. Many will be 5-10 times
>> the size of this one.
>>
>> Thanks for the feedback.
>>
>> Alain
Re: [Teneo] Model persistence performance [message #92374 is a reply to message #92359] Wed, 01 August 2007 14:57 Go to previous messageGo to next message
Eclipse User
Originally posted by: benjamin.cabeNOSPAMM.anyware-tech.DOTcom

Hi Alain,

In my case -Hibernate too-, the ratio between log (with a "default"
log4j.properties) and no-log version is someting like 3-4x... Not
negligible :)

Benjamin.

Alain Picard a écrit :
> Ed,
>
> I am using Hibernate here. Let me try again without the debug log.
>
> Alain
Re: [Teneo] Model persistence performance [message #92388 is a reply to message #92374] Wed, 01 August 2007 16:10 Go to previous messageGo to next message
Martin Taal is currently offline Martin Taal
Messages: 5329
Registered: July 2009
Senior Member
Hi Alain,
Performance is a always a challenge. The main project in which I use Teneo/Hibernate has about
125000 records most of them consisting of about 10 fields. An insert of all these objects together
with some conversion takes about 5 to 10 minutes.
One setting you can set is the jdbc batch size.

See here:
http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml

Also it is my experience that inserting/updating many objects in one transactions slows down the
database as the database transaction log becomes very large. So a commit after every 10000 objects
can result in a better performance (but you loose real transaction behavior). Or you can try
autocommit is true.

gr. Martin

Benjamin CABE wrote:
> Hi Alain,
>
> In my case -Hibernate too-, the ratio between log (with a "default"
> log4j.properties) and no-log version is someting like 3-4x... Not
> negligible :)
>
> Benjamin.
>
> Alain Picard a écrit :
>> Ed,
>>
>> I am using Hibernate here. Let me try again without the debug log.
>>
>> Alain


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: [Teneo] Model persistence performance [message #92403 is a reply to message #92388] Wed, 01 August 2007 17:05 Go to previous messageGo to next message
Alain Picard is currently offline Alain Picard
Messages: 176
Registered: July 2009
Senior Member
Well, I have turned off the debugging and it did improve dramatically.

Here the new results:
Starting to save.
We're done now.
Completed in: 8m:22s:47ms

I even had a previous run, before tracking the time that was around 6 mins and this is always the same data (just changing the PK of
the root).

I will try your suggestions here, but I am still concerned. This is our first and almost simplest model. There will be 4-5 others as
well.

The issue is that for every project we handle we will have millions of records. Right now we are getting much better performance
with an XMI serializer / ZIP compressor / XMI de-serializer, but I feel that when we start to want to perform operations that cross
instance boundaries, that this is where that solution won't scale. We've also looked at other persistence approach (OODB like db4o,
XML storage with Db/2, etc). I'd like to see how others are addressing this performance challenge, which I understand is not really
caused by Teneo here.

I'll keep you all posted on the results we are getting by applying all of your suggestions, and keep them coming.

Cheers,
Alain


Martin Taal wrote:
> Hi Alain,
> Performance is a always a challenge. The main project in which I use
> Teneo/Hibernate has about 125000 records most of them consisting of
> about 10 fields. An insert of all these objects together with some
> conversion takes about 5 to 10 minutes.
> One setting you can set is the jdbc batch size.
>
> See here:
> http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml
>
> Also it is my experience that inserting/updating many objects in one
> transactions slows down the database as the database transaction log
> becomes very large. So a commit after every 10000 objects can result in
> a better performance (but you loose real transaction behavior). Or you
> can try autocommit is true.
>
> gr. Martin
>
> Benjamin CABE wrote:
>> Hi Alain,
>>
>> In my case -Hibernate too-, the ratio between log (with a "default"
>> log4j.properties) and no-log version is someting like 3-4x... Not
>> negligible :)
>>
>> Benjamin.
>>
>> Alain Picard a écrit :
>>> Ed,
>>>
>>> I am using Hibernate here. Let me try again without the debug log.
>>>
>>> Alain
>
>
Re: [Teneo] Model persistence performance [message #92419 is a reply to message #92403] Wed, 01 August 2007 17:45 Go to previous messageGo to next message
Alain Picard is currently offline Alain Picard
Messages: 176
Registered: July 2009
Senior Member
Well, here is for result #1 of increasing the hibernate batch size.

Starting to save.
We're done now.
Completed in: 8m:17s:875ms

There is essentially no improvement. I did find that this is more than likely related to the fact that we are using a joined strategy.


See http://opensource.atlassian.com/projects/hibernate/browse/HH H-2558

Excerpt:
<joined-subclass/> as well as entity's containing <join/> mappings currently cannot participating in JDBC batching. The reaons being
that the actions (i.e. Executables) perform the insert or update in an atomic fashion through the persisters; the persisters perform
the multiple statements.


Well, I guess I might need to look at implementing a different strategy and testing that.

Cheers,
Alain


Alain Picard wrote:
> Well, I have turned off the debugging and it did improve dramatically.
>
> Here the new results:
> Starting to save.
> We're done now.
> Completed in: 8m:22s:47ms
>
> I even had a previous run, before tracking the time that was around 6
> mins and this is always the same data (just changing the PK of the root).
>
> I will try your suggestions here, but I am still concerned. This is our
> first and almost simplest model. There will be 4-5 others as well.
>
> The issue is that for every project we handle we will have millions of
> records. Right now we are getting much better performance with an XMI
> serializer / ZIP compressor / XMI de-serializer, but I feel that when we
> start to want to perform operations that cross instance boundaries, that
> this is where that solution won't scale. We've also looked at other
> persistence approach (OODB like db4o, XML storage with Db/2, etc). I'd
> like to see how others are addressing this performance challenge, which
> I understand is not really caused by Teneo here.
>
> I'll keep you all posted on the results we are getting by applying all
> of your suggestions, and keep them coming.
>
> Cheers,
> Alain
>
>
> Martin Taal wrote:
>> Hi Alain,
>> Performance is a always a challenge. The main project in which I use
>> Teneo/Hibernate has about 125000 records most of them consisting of
>> about 10 fields. An insert of all these objects together with some
>> conversion takes about 5 to 10 minutes.
>> One setting you can set is the jdbc batch size.
>>
>> See here:
>> http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml
>>
>> Also it is my experience that inserting/updating many objects in one
>> transactions slows down the database as the database transaction log
>> becomes very large. So a commit after every 10000 objects can result
>> in a better performance (but you loose real transaction behavior). Or
>> you can try autocommit is true.
>>
>> gr. Martin
>>
>> Benjamin CABE wrote:
>>> Hi Alain,
>>>
>>> In my case -Hibernate too-, the ratio between log (with a "default"
>>> log4j.properties) and no-log version is someting like 3-4x... Not
>>> negligible :)
>>>
>>> Benjamin.
>>>
>>> Alain Picard a écrit :
>>>> Ed,
>>>>
>>>> I am using Hibernate here. Let me try again without the debug log.
>>>>
>>>> Alain
>>
>>
Re: [Teneo] Model persistence performance [message #92616 is a reply to message #92388] Sun, 05 August 2007 16:43 Go to previous messageGo to next message
Alain Picard is currently offline Alain Picard
Messages: 176
Registered: July 2009
Senior Member
Martin,

I have made more experiment by switching everything to JPOX and trying the same tests.

The results are essentially the same, from a few seconds less to a few seconds more. What is interesting though is that the CPU
usage pattern is very different. The Eclipse application used to take about 50% of CPU and MySql 2-3%. With JPOX, both together take
about 25-30% of CPU, with MySql actually taking a bit more than Eclipse. The other difference here is that the disk is constantly
being accessed, which was not the case with Hibernate.

I then realized that the Enums were saved as Medium blobs, which could account for a lot of the disk I/O and possibly some
performance degradation. I tried to switch to JPOX 2 with the Java5 plugin to try to fix this.

The issue is that I am getting the following exception:
org.jpox.exceptions.JPOXUserException: Level 1 Cache "org.eclipse.emf.teneo.jpox.cache.EMFWeakRefCache" is not registered. Please
check your CLASSPATH and specification.

Based on your reply on July 18th, regarding this issue I tried to disable the cache for now doing the following:
Properties props = jpoxDataStore.getProperties();
props.remove(PersistenceConfiguration.CACHE_LEVEL_1_TYPE_PRO PERTY);

but that still gives me the error. What is the correct way to cancel the cache and/or to specify the plugin.xml that JPOX requires?

I was also thinking of trying the db4o extension to see if that gives me better results? Have you tried to use it so far?

And what about using JDO outside of JPOX, should that work? I am thinking in terms of some other OODBMS, here.

Otherwise I am still open to other solutions or approaches, like using batch inserts or whatever.

Thanks
Alain

Martin Taal wrote:
> Hi Alain,
> Performance is a always a challenge. The main project in which I use
> Teneo/Hibernate has about 125000 records most of them consisting of
> about 10 fields. An insert of all these objects together with some
> conversion takes about 5 to 10 minutes.
> One setting you can set is the jdbc batch size.
>
> See here:
> http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml
>
> Also it is my experience that inserting/updating many objects in one
> transactions slows down the database as the database transaction log
> becomes very large. So a commit after every 10000 objects can result in
> a better performance (but you loose real transaction behavior). Or you
> can try autocommit is true.
>
> gr. Martin
>
> Benjamin CABE wrote:
>> Hi Alain,
>>
>> In my case -Hibernate too-, the ratio between log (with a "default"
>> log4j.properties) and no-log version is someting like 3-4x... Not
>> negligible :)
>>
>> Benjamin.
>>
>> Alain Picard a écrit :
>>> Ed,
>>>
>>> I am using Hibernate here. Let me try again without the debug log.
>>>
>>> Alain
>
>
Re: [Teneo] Model persistence performance [message #92637 is a reply to message #92616] Sun, 05 August 2007 17:05 Go to previous messageGo to next message
Martin Taal is currently offline Martin Taal
Messages: 5329
Registered: July 2009
Senior Member
Hi Alain,
Teneo only works with jpox 1.1.8. I have tried jpox 1.2 but could not get all the test cases
running. I will try again when jpox 1.2 final is releases. Also for jpox 1.1.8 enums should be saved
as a string/int. If this is not happening then I can look at that. Can you confirm?

Are you using a level 2 cache? If so then this can cost some performance.

I have also noticed when using hibernate that hibernate takes a lot of cpu compared to the db. I did
not have time to check out what was happening. If you have a testcase which I can try then I am
interested to try that out and see what happens (my case was not easily reproducable).

What is your current run-time for the insert of 90.000 records?

I have not tried db4o. I am not sure but afaiu this db does not support foreign key constraint. I
have not tried jdo other than jpox. Eventhough jdo is a standard spec Teneo uses jpox specific
extensions. I am almost 100% sure that Teneo will not work out of the box with another jdo supplier.

gr. Martin


Alain Picard wrote:
> Martin,
>
> I have made more experiment by switching everything to JPOX and trying
> the same tests.
>
> The results are essentially the same, from a few seconds less to a few
> seconds more. What is interesting though is that the CPU usage pattern
> is very different. The Eclipse application used to take about 50% of CPU
> and MySql 2-3%. With JPOX, both together take about 25-30% of CPU, with
> MySql actually taking a bit more than Eclipse. The other difference here
> is that the disk is constantly being accessed, which was not the case
> with Hibernate.
>
> I then realized that the Enums were saved as Medium blobs, which could
> account for a lot of the disk I/O and possibly some performance
> degradation. I tried to switch to JPOX 2 with the Java5 plugin to try to
> fix this.
>
> The issue is that I am getting the following exception:
> org.jpox.exceptions.JPOXUserException: Level 1 Cache
> "org.eclipse.emf.teneo.jpox.cache.EMFWeakRefCache" is not registered.
> Please check your CLASSPATH and specification.
>
> Based on your reply on July 18th, regarding this issue I tried to
> disable the cache for now doing the following:
> Properties props = jpoxDataStore.getProperties();
> props.remove(PersistenceConfiguration.CACHE_LEVEL_1_TYPE_PRO PERTY);
>
> but that still gives me the error. What is the correct way to cancel the
> cache and/or to specify the plugin.xml that JPOX requires?
>
> I was also thinking of trying the db4o extension to see if that gives me
> better results? Have you tried to use it so far?
>
> And what about using JDO outside of JPOX, should that work? I am
> thinking in terms of some other OODBMS, here.
>
> Otherwise I am still open to other solutions or approaches, like using
> batch inserts or whatever.
>
> Thanks
> Alain
>
> Martin Taal wrote:
>> Hi Alain,
>> Performance is a always a challenge. The main project in which I use
>> Teneo/Hibernate has about 125000 records most of them consisting of
>> about 10 fields. An insert of all these objects together with some
>> conversion takes about 5 to 10 minutes.
>> One setting you can set is the jdbc batch size.
>>
>> See here:
>> http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml
>>
>> Also it is my experience that inserting/updating many objects in one
>> transactions slows down the database as the database transaction log
>> becomes very large. So a commit after every 10000 objects can result
>> in a better performance (but you loose real transaction behavior). Or
>> you can try autocommit is true.
>>
>> gr. Martin
>>
>> Benjamin CABE wrote:
>>> Hi Alain,
>>>
>>> In my case -Hibernate too-, the ratio between log (with a "default"
>>> log4j.properties) and no-log version is someting like 3-4x... Not
>>> negligible :)
>>>
>>> Benjamin.
>>>
>>> Alain Picard a écrit :
>>>> Ed,
>>>>
>>>> I am using Hibernate here. Let me try again without the debug log.
>>>>
>>>> Alain
>>
>>
b

--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: [Teneo] Model persistence performance [message #92652 is a reply to message #92637] Sun, 05 August 2007 17:43 Go to previous messageGo to next message
Alain Picard is currently offline Alain Picard
Messages: 176
Registered: July 2009
Senior Member
Martin,

I can confirm that I am getting eenum columns to show up as mediumblob type in MySql and here is the mapping for one of those:
<field name="formattingType" persistence-modifier="persistent" null-value="none" embedded="true"/>

I am not using any level 2 cache.

The current time is still around 8 mins. I tried to insert (with hibernate) the tree in section, but right now I am facing an issue
where it tells me that I am missing a required value, even if it is supplied. Not really sure why, but the whole exercise seems to
be a bit awkward also. I might have to relax the integrity rule in order to insert data "gradually".

Thanks
Alain



Martin Taal wrote:
> Hi Alain,
> Teneo only works with jpox 1.1.8. I have tried jpox 1.2 but could not
> get all the test cases running. I will try again when jpox 1.2 final is
> releases. Also for jpox 1.1.8 enums should be saved as a string/int. If
> this is not happening then I can look at that. Can you confirm?
>
> Are you using a level 2 cache? If so then this can cost some performance.
>
> I have also noticed when using hibernate that hibernate takes a lot of
> cpu compared to the db. I did not have time to check out what was
> happening. If you have a testcase which I can try then I am interested
> to try that out and see what happens (my case was not easily reproducable).
>
> What is your current run-time for the insert of 90.000 records?
>
> I have not tried db4o. I am not sure but afaiu this db does not support
> foreign key constraint. I have not tried jdo other than jpox. Eventhough
> jdo is a standard spec Teneo uses jpox specific extensions. I am almost
> 100% sure that Teneo will not work out of the box with another jdo
> supplier.
>
> gr. Martin
>
>
> Alain Picard wrote:
>> Martin,
>>
>> I have made more experiment by switching everything to JPOX and trying
>> the same tests.
>>
>> The results are essentially the same, from a few seconds less to a few
>> seconds more. What is interesting though is that the CPU usage pattern
>> is very different. The Eclipse application used to take about 50% of
>> CPU and MySql 2-3%. With JPOX, both together take about 25-30% of
>> CPU, with MySql actually taking a bit more than Eclipse. The other
>> difference here is that the disk is constantly being accessed, which
>> was not the case with Hibernate.
>>
>> I then realized that the Enums were saved as Medium blobs, which could
>> account for a lot of the disk I/O and possibly some performance
>> degradation. I tried to switch to JPOX 2 with the Java5 plugin to try
>> to fix this.
>>
>> The issue is that I am getting the following exception:
>> org.jpox.exceptions.JPOXUserException: Level 1 Cache
>> "org.eclipse.emf.teneo.jpox.cache.EMFWeakRefCache" is not registered.
>> Please check your CLASSPATH and specification.
>>
>> Based on your reply on July 18th, regarding this issue I tried to
>> disable the cache for now doing the following:
>> Properties props = jpoxDataStore.getProperties();
>>
>> props.remove(PersistenceConfiguration.CACHE_LEVEL_1_TYPE_PRO PERTY);
>>
>> but that still gives me the error. What is the correct way to cancel
>> the cache and/or to specify the plugin.xml that JPOX requires?
>>
>> I was also thinking of trying the db4o extension to see if that gives
>> me better results? Have you tried to use it so far?
>>
>> And what about using JDO outside of JPOX, should that work? I am
>> thinking in terms of some other OODBMS, here.
>>
>> Otherwise I am still open to other solutions or approaches, like using
>> batch inserts or whatever.
>>
>> Thanks
>> Alain
>>
>> Martin Taal wrote:
>>> Hi Alain,
>>> Performance is a always a challenge. The main project in which I use
>>> Teneo/Hibernate has about 125000 records most of them consisting of
>>> about 10 fields. An insert of all these objects together with some
>>> conversion takes about 5 to 10 minutes.
>>> One setting you can set is the jdbc batch size.
>>>
>>> See here:
>>> http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml
>>>
>>> Also it is my experience that inserting/updating many objects in one
>>> transactions slows down the database as the database transaction log
>>> becomes very large. So a commit after every 10000 objects can result
>>> in a better performance (but you loose real transaction behavior). Or
>>> you can try autocommit is true.
>>>
>>> gr. Martin
>>>
>>> Benjamin CABE wrote:
>>>> Hi Alain,
>>>>
>>>> In my case -Hibernate too-, the ratio between log (with a "default"
>>>> log4j.properties) and no-log version is someting like 3-4x... Not
>>>> negligible :)
>>>>
>>>> Benjamin.
>>>>
>>>> Alain Picard a écrit :
>>>>> Ed,
>>>>>
>>>>> I am using Hibernate here. Let me try again without the debug log.
>>>>>
>>>>> Alain
>>>
>>>
> b
>
Re: [Teneo] Model persistence performance [message #92668 is a reply to message #92637] Sun, 05 August 2007 21:52 Go to previous messageGo to next message
Alain Picard is currently offline Alain Picard
Messages: 176
Registered: July 2009
Senior Member
Martin,

I fixed the issue with the WeakCache but I then realized that the current version doesn't even run against 1.2, many methods are
just not found.

I grabbed the code from CVS (teneo jpox) in order to see if it was limited to a few renames as they state on the JPOX site. I have
been able to reduce the number of errors by making some naming corrections, but these folks have no regards for backward
compatibility. Do you yourself at least have a version that compiles with 1.2?

o/w, i'll try to take a bit more time and try to at least got a copy that compiles.

Thanks
Alain
Re: [Teneo] Model persistence performance [message #92683 is a reply to message #92668] Mon, 06 August 2007 14:34 Go to previous message
Martin Taal is currently offline Martin Taal
Messages: 5329
Registered: July 2009
Senior Member
Alain,
Teneo partially uses the internal api of jpox which is less relevant for more standard jpox users so
therefore Teneo is more sensitive to jpox changes. I had a version which compiled with 1.2 but
rolled back the changes because I could not get all test cases to pass. I will look at 1.2 again
when I have time (somewhere in the autumn).

gr. Martin

Alain Picard wrote:
> Martin,
>
> I fixed the issue with the WeakCache but I then realized that the
> current version doesn't even run against 1.2, many methods are just not
> found.
>
> I grabbed the code from CVS (teneo jpox) in order to see if it was
> limited to a few renames as they state on the JPOX site. I have been
> able to reduce the number of errors by making some naming corrections,
> but these folks have no regards for backward compatibility. Do you
> yourself at least have a version that compiles with 1.2?
>
> o/w, i'll try to take a bit more time and try to at least got a copy
> that compiles.
>
> Thanks
> Alain
>


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: [Teneo] Model persistence performance [message #609406 is a reply to message #92312] Wed, 01 August 2007 13:17 Go to previous message
Ed Merks is currently offline Ed Merks
Messages: 26000
Registered: July 2009
Senior Member
Alain,

I know I've had folks complain about performance and have it turn out to
be a single println statement that the person thought was quite
innocuous so it would definitely be good to run your scenario without
spewing out almost 100 times more debug/log data than you'd get just for
the XMI serialization of it. I suppose it would also be good to know if
you are using Hibernate verses JPOX...


Alain Picard wrote:
> Now that I have my model persistence configured and running with small
> test cases, I decided to use it with a "standard" size case.
>
> Well, it took me 90 mins to persist the model to the local MySQL
> database (with debugging on, 600 Megs log file). The model is made up
> of 9 tables and this particular model instance contains about 90k
> records altogether. The equivalent XMI is about 9.5Megs and compresses
> to 229k with zip.
>
> For the most part, all of the records are quite small, mostly integer
> and bigint values, with a few varchar values.
>
> I understand that running without a debug file would surely help
> somewhat, but anything over 30 seconds here surely won't make the grade.
>
> What are other peoples experiences with the performance of persisting
> large EMF models ?
>
> For us this is a small/medium model instance. Many will be 5-10 times
> the size of this one.
>
> Thanks for the feedback.
>
> Alain
Re: [Teneo] Model persistence performance [message #609408 is a reply to message #92328] Wed, 01 August 2007 13:27 Go to previous message
Alain Picard is currently offline Alain Picard
Messages: 176
Registered: July 2009
Senior Member
Ed,

I am using Hibernate here. Let me try again without the debug log.

Alain

Ed Merks wrote:
> Alain,
>
> I know I've had folks complain about performance and have it turn out to
> be a single println statement that the person thought was quite
> innocuous so it would definitely be good to run your scenario without
> spewing out almost 100 times more debug/log data than you'd get just for
> the XMI serialization of it. I suppose it would also be good to know if
> you are using Hibernate verses JPOX...
>
>
> Alain Picard wrote:
>> Now that I have my model persistence configured and running with small
>> test cases, I decided to use it with a "standard" size case.
>>
>> Well, it took me 90 mins to persist the model to the local MySQL
>> database (with debugging on, 600 Megs log file). The model is made up
>> of 9 tables and this particular model instance contains about 90k
>> records altogether. The equivalent XMI is about 9.5Megs and compresses
>> to 229k with zip.
>>
>> For the most part, all of the records are quite small, mostly integer
>> and bigint values, with a few varchar values.
>>
>> I understand that running without a debug file would surely help
>> somewhat, but anything over 30 seconds here surely won't make the grade.
>>
>> What are other peoples experiences with the performance of persisting
>> large EMF models ?
>>
>> For us this is a small/medium model instance. Many will be 5-10 times
>> the size of this one.
>>
>> Thanks for the feedback.
>>
>> Alain
Re: [Teneo] Model persistence performance [message #609409 is a reply to message #92359] Wed, 01 August 2007 14:57 Go to previous message
Eclipse User
Originally posted by: benjamin.cabeNOSPAMM.anyware-tech.DOTcom

Hi Alain,

In my case -Hibernate too-, the ratio between log (with a "default"
log4j.properties) and no-log version is someting like 3-4x... Not
negligible :)

Benjamin.

Alain Picard a écrit :
> Ed,
>
> I am using Hibernate here. Let me try again without the debug log.
>
> Alain
Re: [Teneo] Model persistence performance [message #609410 is a reply to message #92374] Wed, 01 August 2007 16:10 Go to previous message
Martin Taal is currently offline Martin Taal
Messages: 5329
Registered: July 2009
Senior Member
Hi Alain,
Performance is a always a challenge. The main project in which I use Teneo/Hibernate has about
125000 records most of them consisting of about 10 fields. An insert of all these objects together
with some conversion takes about 5 to 10 minutes.
One setting you can set is the jdbc batch size.

See here:
http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml

Also it is my experience that inserting/updating many objects in one transactions slows down the
database as the database transaction log becomes very large. So a commit after every 10000 objects
can result in a better performance (but you loose real transaction behavior). Or you can try
autocommit is true.

gr. Martin

Benjamin CABE wrote:
> Hi Alain,
>
> In my case -Hibernate too-, the ratio between log (with a "default"
> log4j.properties) and no-log version is someting like 3-4x... Not
> negligible :)
>
> Benjamin.
>
> Alain Picard a écrit :
>> Ed,
>>
>> I am using Hibernate here. Let me try again without the debug log.
>>
>> Alain


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: [Teneo] Model persistence performance [message #609411 is a reply to message #92388] Wed, 01 August 2007 17:05 Go to previous message
Alain Picard is currently offline Alain Picard
Messages: 176
Registered: July 2009
Senior Member
Well, I have turned off the debugging and it did improve dramatically.

Here the new results:
Starting to save.
We're done now.
Completed in: 8m:22s:47ms

I even had a previous run, before tracking the time that was around 6 mins and this is always the same data (just changing the PK of
the root).

I will try your suggestions here, but I am still concerned. This is our first and almost simplest model. There will be 4-5 others as
well.

The issue is that for every project we handle we will have millions of records. Right now we are getting much better performance
with an XMI serializer / ZIP compressor / XMI de-serializer, but I feel that when we start to want to perform operations that cross
instance boundaries, that this is where that solution won't scale. We've also looked at other persistence approach (OODB like db4o,
XML storage with Db/2, etc). I'd like to see how others are addressing this performance challenge, which I understand is not really
caused by Teneo here.

I'll keep you all posted on the results we are getting by applying all of your suggestions, and keep them coming.

Cheers,
Alain


Martin Taal wrote:
> Hi Alain,
> Performance is a always a challenge. The main project in which I use
> Teneo/Hibernate has about 125000 records most of them consisting of
> about 10 fields. An insert of all these objects together with some
> conversion takes about 5 to 10 minutes.
> One setting you can set is the jdbc batch size.
>
> See here:
> http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml
>
> Also it is my experience that inserting/updating many objects in one
> transactions slows down the database as the database transaction log
> becomes very large. So a commit after every 10000 objects can result in
> a better performance (but you loose real transaction behavior). Or you
> can try autocommit is true.
>
> gr. Martin
>
> Benjamin CABE wrote:
>> Hi Alain,
>>
>> In my case -Hibernate too-, the ratio between log (with a "default"
>> log4j.properties) and no-log version is someting like 3-4x... Not
>> negligible :)
>>
>> Benjamin.
>>
>> Alain Picard a écrit :
>>> Ed,
>>>
>>> I am using Hibernate here. Let me try again without the debug log.
>>>
>>> Alain
>
>
Re: [Teneo] Model persistence performance [message #609412 is a reply to message #92403] Wed, 01 August 2007 17:45 Go to previous message
Alain Picard is currently offline Alain Picard
Messages: 176
Registered: July 2009
Senior Member
Well, here is for result #1 of increasing the hibernate batch size.

Starting to save.
We're done now.
Completed in: 8m:17s:875ms

There is essentially no improvement. I did find that this is more than likely related to the fact that we are using a joined strategy.


See http://opensource.atlassian.com/projects/hibernate/browse/HH H-2558

Excerpt:
<joined-subclass/> as well as entity's containing <join/> mappings currently cannot participating in JDBC batching. The reaons being
that the actions (i.e. Executables) perform the insert or update in an atomic fashion through the persisters; the persisters perform
the multiple statements.


Well, I guess I might need to look at implementing a different strategy and testing that.

Cheers,
Alain


Alain Picard wrote:
> Well, I have turned off the debugging and it did improve dramatically.
>
> Here the new results:
> Starting to save.
> We're done now.
> Completed in: 8m:22s:47ms
>
> I even had a previous run, before tracking the time that was around 6
> mins and this is always the same data (just changing the PK of the root).
>
> I will try your suggestions here, but I am still concerned. This is our
> first and almost simplest model. There will be 4-5 others as well.
>
> The issue is that for every project we handle we will have millions of
> records. Right now we are getting much better performance with an XMI
> serializer / ZIP compressor / XMI de-serializer, but I feel that when we
> start to want to perform operations that cross instance boundaries, that
> this is where that solution won't scale. We've also looked at other
> persistence approach (OODB like db4o, XML storage with Db/2, etc). I'd
> like to see how others are addressing this performance challenge, which
> I understand is not really caused by Teneo here.
>
> I'll keep you all posted on the results we are getting by applying all
> of your suggestions, and keep them coming.
>
> Cheers,
> Alain
>
>
> Martin Taal wrote:
>> Hi Alain,
>> Performance is a always a challenge. The main project in which I use
>> Teneo/Hibernate has about 125000 records most of them consisting of
>> about 10 fields. An insert of all these objects together with some
>> conversion takes about 5 to 10 minutes.
>> One setting you can set is the jdbc batch size.
>>
>> See here:
>> http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml
>>
>> Also it is my experience that inserting/updating many objects in one
>> transactions slows down the database as the database transaction log
>> becomes very large. So a commit after every 10000 objects can result
>> in a better performance (but you loose real transaction behavior). Or
>> you can try autocommit is true.
>>
>> gr. Martin
>>
>> Benjamin CABE wrote:
>>> Hi Alain,
>>>
>>> In my case -Hibernate too-, the ratio between log (with a "default"
>>> log4j.properties) and no-log version is someting like 3-4x... Not
>>> negligible :)
>>>
>>> Benjamin.
>>>
>>> Alain Picard a écrit :
>>>> Ed,
>>>>
>>>> I am using Hibernate here. Let me try again without the debug log.
>>>>
>>>> Alain
>>
>>
Re: [Teneo] Model persistence performance [message #609425 is a reply to message #92388] Sun, 05 August 2007 16:43 Go to previous message
Alain Picard is currently offline Alain Picard
Messages: 176
Registered: July 2009
Senior Member
Martin,

I have made more experiment by switching everything to JPOX and trying the same tests.

The results are essentially the same, from a few seconds less to a few seconds more. What is interesting though is that the CPU
usage pattern is very different. The Eclipse application used to take about 50% of CPU and MySql 2-3%. With JPOX, both together take
about 25-30% of CPU, with MySql actually taking a bit more than Eclipse. The other difference here is that the disk is constantly
being accessed, which was not the case with Hibernate.

I then realized that the Enums were saved as Medium blobs, which could account for a lot of the disk I/O and possibly some
performance degradation. I tried to switch to JPOX 2 with the Java5 plugin to try to fix this.

The issue is that I am getting the following exception:
org.jpox.exceptions.JPOXUserException: Level 1 Cache "org.eclipse.emf.teneo.jpox.cache.EMFWeakRefCache" is not registered. Please
check your CLASSPATH and specification.

Based on your reply on July 18th, regarding this issue I tried to disable the cache for now doing the following:
Properties props = jpoxDataStore.getProperties();
props.remove(PersistenceConfiguration.CACHE_LEVEL_1_TYPE_PRO PERTY);

but that still gives me the error. What is the correct way to cancel the cache and/or to specify the plugin.xml that JPOX requires?

I was also thinking of trying the db4o extension to see if that gives me better results? Have you tried to use it so far?

And what about using JDO outside of JPOX, should that work? I am thinking in terms of some other OODBMS, here.

Otherwise I am still open to other solutions or approaches, like using batch inserts or whatever.

Thanks
Alain

Martin Taal wrote:
> Hi Alain,
> Performance is a always a challenge. The main project in which I use
> Teneo/Hibernate has about 125000 records most of them consisting of
> about 10 fields. An insert of all these objects together with some
> conversion takes about 5 to 10 minutes.
> One setting you can set is the jdbc batch size.
>
> See here:
> http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml
>
> Also it is my experience that inserting/updating many objects in one
> transactions slows down the database as the database transaction log
> becomes very large. So a commit after every 10000 objects can result in
> a better performance (but you loose real transaction behavior). Or you
> can try autocommit is true.
>
> gr. Martin
>
> Benjamin CABE wrote:
>> Hi Alain,
>>
>> In my case -Hibernate too-, the ratio between log (with a "default"
>> log4j.properties) and no-log version is someting like 3-4x... Not
>> negligible :)
>>
>> Benjamin.
>>
>> Alain Picard a écrit :
>>> Ed,
>>>
>>> I am using Hibernate here. Let me try again without the debug log.
>>>
>>> Alain
>
>
Re: [Teneo] Model persistence performance [message #609426 is a reply to message #92616] Sun, 05 August 2007 17:05 Go to previous message
Martin Taal is currently offline Martin Taal
Messages: 5329
Registered: July 2009
Senior Member
Hi Alain,
Teneo only works with jpox 1.1.8. I have tried jpox 1.2 but could not get all the test cases
running. I will try again when jpox 1.2 final is releases. Also for jpox 1.1.8 enums should be saved
as a string/int. If this is not happening then I can look at that. Can you confirm?

Are you using a level 2 cache? If so then this can cost some performance.

I have also noticed when using hibernate that hibernate takes a lot of cpu compared to the db. I did
not have time to check out what was happening. If you have a testcase which I can try then I am
interested to try that out and see what happens (my case was not easily reproducable).

What is your current run-time for the insert of 90.000 records?

I have not tried db4o. I am not sure but afaiu this db does not support foreign key constraint. I
have not tried jdo other than jpox. Eventhough jdo is a standard spec Teneo uses jpox specific
extensions. I am almost 100% sure that Teneo will not work out of the box with another jdo supplier.

gr. Martin


Alain Picard wrote:
> Martin,
>
> I have made more experiment by switching everything to JPOX and trying
> the same tests.
>
> The results are essentially the same, from a few seconds less to a few
> seconds more. What is interesting though is that the CPU usage pattern
> is very different. The Eclipse application used to take about 50% of CPU
> and MySql 2-3%. With JPOX, both together take about 25-30% of CPU, with
> MySql actually taking a bit more than Eclipse. The other difference here
> is that the disk is constantly being accessed, which was not the case
> with Hibernate.
>
> I then realized that the Enums were saved as Medium blobs, which could
> account for a lot of the disk I/O and possibly some performance
> degradation. I tried to switch to JPOX 2 with the Java5 plugin to try to
> fix this.
>
> The issue is that I am getting the following exception:
> org.jpox.exceptions.JPOXUserException: Level 1 Cache
> "org.eclipse.emf.teneo.jpox.cache.EMFWeakRefCache" is not registered.
> Please check your CLASSPATH and specification.
>
> Based on your reply on July 18th, regarding this issue I tried to
> disable the cache for now doing the following:
> Properties props = jpoxDataStore.getProperties();
> props.remove(PersistenceConfiguration.CACHE_LEVEL_1_TYPE_PRO PERTY);
>
> but that still gives me the error. What is the correct way to cancel the
> cache and/or to specify the plugin.xml that JPOX requires?
>
> I was also thinking of trying the db4o extension to see if that gives me
> better results? Have you tried to use it so far?
>
> And what about using JDO outside of JPOX, should that work? I am
> thinking in terms of some other OODBMS, here.
>
> Otherwise I am still open to other solutions or approaches, like using
> batch inserts or whatever.
>
> Thanks
> Alain
>
> Martin Taal wrote:
>> Hi Alain,
>> Performance is a always a challenge. The main project in which I use
>> Teneo/Hibernate has about 125000 records most of them consisting of
>> about 10 fields. An insert of all these objects together with some
>> conversion takes about 5 to 10 minutes.
>> One setting you can set is the jdbc batch size.
>>
>> See here:
>> http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml
>>
>> Also it is my experience that inserting/updating many objects in one
>> transactions slows down the database as the database transaction log
>> becomes very large. So a commit after every 10000 objects can result
>> in a better performance (but you loose real transaction behavior). Or
>> you can try autocommit is true.
>>
>> gr. Martin
>>
>> Benjamin CABE wrote:
>>> Hi Alain,
>>>
>>> In my case -Hibernate too-, the ratio between log (with a "default"
>>> log4j.properties) and no-log version is someting like 3-4x... Not
>>> negligible :)
>>>
>>> Benjamin.
>>>
>>> Alain Picard a écrit :
>>>> Ed,
>>>>
>>>> I am using Hibernate here. Let me try again without the debug log.
>>>>
>>>> Alain
>>
>>
b

--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: [Teneo] Model persistence performance [message #609427 is a reply to message #92637] Sun, 05 August 2007 17:43 Go to previous message
Alain Picard is currently offline Alain Picard
Messages: 176
Registered: July 2009
Senior Member
Martin,

I can confirm that I am getting eenum columns to show up as mediumblob type in MySql and here is the mapping for one of those:
<field name="formattingType" persistence-modifier="persistent" null-value="none" embedded="true"/>

I am not using any level 2 cache.

The current time is still around 8 mins. I tried to insert (with hibernate) the tree in section, but right now I am facing an issue
where it tells me that I am missing a required value, even if it is supplied. Not really sure why, but the whole exercise seems to
be a bit awkward also. I might have to relax the integrity rule in order to insert data "gradually".

Thanks
Alain



Martin Taal wrote:
> Hi Alain,
> Teneo only works with jpox 1.1.8. I have tried jpox 1.2 but could not
> get all the test cases running. I will try again when jpox 1.2 final is
> releases. Also for jpox 1.1.8 enums should be saved as a string/int. If
> this is not happening then I can look at that. Can you confirm?
>
> Are you using a level 2 cache? If so then this can cost some performance.
>
> I have also noticed when using hibernate that hibernate takes a lot of
> cpu compared to the db. I did not have time to check out what was
> happening. If you have a testcase which I can try then I am interested
> to try that out and see what happens (my case was not easily reproducable).
>
> What is your current run-time for the insert of 90.000 records?
>
> I have not tried db4o. I am not sure but afaiu this db does not support
> foreign key constraint. I have not tried jdo other than jpox. Eventhough
> jdo is a standard spec Teneo uses jpox specific extensions. I am almost
> 100% sure that Teneo will not work out of the box with another jdo
> supplier.
>
> gr. Martin
>
>
> Alain Picard wrote:
>> Martin,
>>
>> I have made more experiment by switching everything to JPOX and trying
>> the same tests.
>>
>> The results are essentially the same, from a few seconds less to a few
>> seconds more. What is interesting though is that the CPU usage pattern
>> is very different. The Eclipse application used to take about 50% of
>> CPU and MySql 2-3%. With JPOX, both together take about 25-30% of
>> CPU, with MySql actually taking a bit more than Eclipse. The other
>> difference here is that the disk is constantly being accessed, which
>> was not the case with Hibernate.
>>
>> I then realized that the Enums were saved as Medium blobs, which could
>> account for a lot of the disk I/O and possibly some performance
>> degradation. I tried to switch to JPOX 2 with the Java5 plugin to try
>> to fix this.
>>
>> The issue is that I am getting the following exception:
>> org.jpox.exceptions.JPOXUserException: Level 1 Cache
>> "org.eclipse.emf.teneo.jpox.cache.EMFWeakRefCache" is not registered.
>> Please check your CLASSPATH and specification.
>>
>> Based on your reply on July 18th, regarding this issue I tried to
>> disable the cache for now doing the following:
>> Properties props = jpoxDataStore.getProperties();
>>
>> props.remove(PersistenceConfiguration.CACHE_LEVEL_1_TYPE_PRO PERTY);
>>
>> but that still gives me the error. What is the correct way to cancel
>> the cache and/or to specify the plugin.xml that JPOX requires?
>>
>> I was also thinking of trying the db4o extension to see if that gives
>> me better results? Have you tried to use it so far?
>>
>> And what about using JDO outside of JPOX, should that work? I am
>> thinking in terms of some other OODBMS, here.
>>
>> Otherwise I am still open to other solutions or approaches, like using
>> batch inserts or whatever.
>>
>> Thanks
>> Alain
>>
>> Martin Taal wrote:
>>> Hi Alain,
>>> Performance is a always a challenge. The main project in which I use
>>> Teneo/Hibernate has about 125000 records most of them consisting of
>>> about 10 fields. An insert of all these objects together with some
>>> conversion takes about 5 to 10 minutes.
>>> One setting you can set is the jdbc batch size.
>>>
>>> See here:
>>> http://www.hibernate.org/hib_docs/reference/en/html/batch.ht ml
>>>
>>> Also it is my experience that inserting/updating many objects in one
>>> transactions slows down the database as the database transaction log
>>> becomes very large. So a commit after every 10000 objects can result
>>> in a better performance (but you loose real transaction behavior). Or
>>> you can try autocommit is true.
>>>
>>> gr. Martin
>>>
>>> Benjamin CABE wrote:
>>>> Hi Alain,
>>>>
>>>> In my case -Hibernate too-, the ratio between log (with a "default"
>>>> log4j.properties) and no-log version is someting like 3-4x... Not
>>>> negligible :)
>>>>
>>>> Benjamin.
>>>>
>>>> Alain Picard a écrit :
>>>>> Ed,
>>>>>
>>>>> I am using Hibernate here. Let me try again without the debug log.
>>>>>
>>>>> Alain
>>>
>>>
> b
>
Re: [Teneo] Model persistence performance [message #609428 is a reply to message #92637] Sun, 05 August 2007 21:52 Go to previous message
Alain Picard is currently offline Alain Picard
Messages: 176
Registered: July 2009
Senior Member
Martin,

I fixed the issue with the WeakCache but I then realized that the current version doesn't even run against 1.2, many methods are
just not found.

I grabbed the code from CVS (teneo jpox) in order to see if it was limited to a few renames as they state on the JPOX site. I have
been able to reduce the number of errors by making some naming corrections, but these folks have no regards for backward
compatibility. Do you yourself at least have a version that compiles with 1.2?

o/w, i'll try to take a bit more time and try to at least got a copy that compiles.

Thanks
Alain
Re: [Teneo] Model persistence performance [message #609429 is a reply to message #92668] Mon, 06 August 2007 14:34 Go to previous message
Martin Taal is currently offline Martin Taal
Messages: 5329
Registered: July 2009
Senior Member
Alain,
Teneo partially uses the internal api of jpox which is less relevant for more standard jpox users so
therefore Teneo is more sensitive to jpox changes. I had a version which compiled with 1.2 but
rolled back the changes because I could not get all test cases to pass. I will look at 1.2 again
when I have time (somewhere in the autumn).

gr. Martin

Alain Picard wrote:
> Martin,
>
> I fixed the issue with the WeakCache but I then realized that the
> current version doesn't even run against 1.2, many methods are just not
> found.
>
> I grabbed the code from CVS (teneo jpox) in order to see if it was
> limited to a few renames as they state on the JPOX site. I have been
> able to reduce the number of errors by making some naming corrections,
> but these folks have no regards for backward compatibility. Do you
> yourself at least have a version that compiles with 1.2?
>
> o/w, i'll try to take a bit more time and try to at least got a copy
> that compiles.
>
> Thanks
> Alain
>


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Previous Topic:JPA Annotation for Composite Foregin Key
Next Topic:discriminator type
Goto Forum:
  


Current Time: Fri Aug 22 17:37:47 EDT 2014

Powered by FUDForum. Page generated in 0.09985 seconds