Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF "Technology" (Ecore Tools, EMFatic, etc)  » Performance Problem with Teneo 200610261350 / Jpox 1.1.3
Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #60282] Wed, 08 November 2006 12:58 Go to next message
Steffen Neumann is currently offline Steffen NeumannFriend
Messages: 41
Registered: July 2009
Member
Good news upfront:

with 200610261350 and Jpox 1.1.3 I am able to persist
my model to postgres again. I'll try Oracle this week.

However, the performance dropped to unacceptable levels.
My 53MB XML/EMF file took around 90secs with JPOX 1.1.0rc1. to persist,
and now I am well above 25 minutes with the same hardware setup.

Turning on debugging reveals numerous calls to
EContainerRepairControl(). Is this the reason
for my performance problem ? Is there a way around that ?

Any more debugging information I should supply ?

Yours,
Steffen

------------------------------------------------------------ ---

Trying to persist /vol/tools/E77L_ColHalle_1 1.mzData
22762 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
Repairing container relations of children of:
de.ipb.msbi.Mzdata.impl.DataTypeImpl
22766 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
Repairing container relations of children of:
de.ipb.msbi.Mzdata.impl.PeakListBinaryTypeImpl
22767 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
Repairing container relations of children of:
de.ipb.msbi.Mzdata.impl.PeakListBinaryTypeImpl
22767 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
Repairing reference data to child de.ipb.msbi.Mzdata.DataType
23627 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
Repairing container relations of children of:
de.ipb.msbi.Mzdata.impl.DataTypeImpl
23628 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
Repairing container relations of children of:
de.ipb.msbi.Mzdata.impl.PeakListBinaryTypeImpl
23628 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
Repairing container relations of children of:
de.ipb.msbi.Mzdata.impl.PeakListBinaryTypeImpl
23628 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
Repairing reference data to child de.ipb.msbi.Mzdata.DataType
24354 [main] DEBUG
org.eclipse.emf.teneo.mapping.elist.PersistableFeatureMap - Created
persistable featuremap FeatureMap of member group owned by
de.ipb.msbi.Mzdata.impl.AcquisitionTypeImpl with delegate list
java.util.ArrayList
24359 [main] DEBUG org.eclipse.emf.teneo.jpox.elist.FeatureMapWrapper -
Creating feature map wrapper for
org.eclipse.emf.teneo.jpox.elist.GenericFeatureMapEntry
24362 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
Repairing container relations of children of:
de.ipb.msbi.Mzdata.impl.AcquisitionTypeImpl
24362 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
Repairing container relations of children of:
de.ipb.msbi.Mzdata.impl.AcquisitionTypeImpl
24362 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
Repairing reference cvParam to child de.ipb.msbi.Mzdata.CvParamType
24362 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
Repairing reference userParam to child de.ipb.msbi.Mzdata.UserParamType
24362 [main] DEBUG org.eclipse.emf.teneo.mapping.elist.PersistableEList
- Created persistable list EList of type:
org.eclipse.emf.teneo.jpox.elist.EListWrapper of member acquisition
owned by de.ipb.msbi.Mzdata.impl.AcqSpecificationTypeImpl with delegate
list java.util.ArrayList
24363 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
Repairing container relations of children of:
de.ipb.msbi.Mzdata.impl.AcqSpecificationTypeImpl
24363 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
Repairing container relations of children of:
de.ipb.msbi.Mzdata.impl.AcqSpecificationTypeImpl
24363 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
Repairing reference acquisition to child de.ipb.msbi.Mzdata.AcquisitionType


------------------------------------------------------------ ---


Old, known good configuration:
Trying to get PersistenceManager...
0 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering a
jpox resource factory for all uri's with jpox as the protocol/extension
3 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering a
jpox resource factory for all uri's with ejdo as the protocol/extension
3 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering a
jpox resource dao factory for all uri's with jpoxdao as the
protocol/extension
8 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
EListMapping, EListWrapper at the jpox manager for handling elists
8 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
FeatureMapMapping, FeatureMapWrapper at the jpox manager for handling
FeatureMap
8 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
EObjectMapping at the jpox manager for handling EObjects/AnyType
8 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
XMLCalendarMapping at the jpox manager for handling EObjects/AnyType
9 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
XMLDurationMapping at the jpox manager for handling EObjects/AnyType
1156 [main] INFO JPOX.JDO - PersistenceManagerFactory - Vendor: JPOX
Version: 1.1.0-rc-1
Re: Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #60310 is a reply to message #60282] Wed, 08 November 2006 13:33 Go to previous messageGo to next message
Martin Taal is currently offline Martin TaalFriend
Messages: 5468
Registered: July 2009
Senior Member
The repaircontrol is done for each item added to the internal cache of jpox and when a resource is
loaded from the database. The repair is called for each object (and its children).

Does the cpu show very much database activity? Or is java the main process?

To prevent the calls to repaircontainer:
As you are persisting (saving) the calls to repaircontrol are probably from the cache. Teneo uses
its own cache implementation to do the repaircontrol. See the constructor in the JpoxDataStore:
properties.setProperty(PMFConfiguration.CACHE_LEVEL_1_TYPE_P ROPERTY,
"org.eclipse.emf.teneo.jpox.cache.EMFWeakRefCache");

What you can try is set the standard jpox weakrefcache (org.jpox.cache.WeakRefCache) instead of the
teneo. You can set this in the properties of the jpoxdatastore,

If this helps then the repair control is the reason for the performance degrade.

gr. Martin

Steffen Neumann wrote:
> Good news upfront:
>
> with 200610261350 and Jpox 1.1.3 I am able to persist
> my model to postgres again. I'll try Oracle this week.
>
> However, the performance dropped to unacceptable levels.
> My 53MB XML/EMF file took around 90secs with JPOX 1.1.0rc1. to persist,
> and now I am well above 25 minutes with the same hardware setup.
>
> Turning on debugging reveals numerous calls to
> EContainerRepairControl(). Is this the reason
> for my performance problem ? Is there a way around that ?
>
> Any more debugging information I should supply ?
>
> Yours,
> Steffen
>
> ------------------------------------------------------------ ---
>
> Trying to persist /vol/tools/E77L_ColHalle_1 1.mzData
> 22762 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.DataTypeImpl
> 22766 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.PeakListBinaryTypeImpl
> 22767 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.PeakListBinaryTypeImpl
> 22767 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing reference data to child de.ipb.msbi.Mzdata.DataType
> 23627 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.DataTypeImpl
> 23628 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.PeakListBinaryTypeImpl
> 23628 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.PeakListBinaryTypeImpl
> 23628 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing reference data to child de.ipb.msbi.Mzdata.DataType
> 24354 [main] DEBUG
> org.eclipse.emf.teneo.mapping.elist.PersistableFeatureMap - Created
> persistable featuremap FeatureMap of member group owned by
> de.ipb.msbi.Mzdata.impl.AcquisitionTypeImpl with delegate list
> java.util.ArrayList
> 24359 [main] DEBUG org.eclipse.emf.teneo.jpox.elist.FeatureMapWrapper -
> Creating feature map wrapper for
> org.eclipse.emf.teneo.jpox.elist.GenericFeatureMapEntry
> 24362 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.AcquisitionTypeImpl
> 24362 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.AcquisitionTypeImpl
> 24362 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing reference cvParam to child de.ipb.msbi.Mzdata.CvParamType
> 24362 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing reference userParam to child de.ipb.msbi.Mzdata.UserParamType
> 24362 [main] DEBUG org.eclipse.emf.teneo.mapping.elist.PersistableEList
> - Created persistable list EList of type:
> org.eclipse.emf.teneo.jpox.elist.EListWrapper of member acquisition
> owned by de.ipb.msbi.Mzdata.impl.AcqSpecificationTypeImpl with delegate
> list java.util.ArrayList
> 24363 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.AcqSpecificationTypeImpl
> 24363 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.AcqSpecificationTypeImpl
> 24363 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing reference acquisition to child de.ipb.msbi.Mzdata.AcquisitionType
>
>
> ------------------------------------------------------------ ---
>
>
> Old, known good configuration:
> Trying to get PersistenceManager...
> 0 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering a
> jpox resource factory for all uri's with jpox as the protocol/extension
> 3 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering a
> jpox resource factory for all uri's with ejdo as the protocol/extension
> 3 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering a
> jpox resource dao factory for all uri's with jpoxdao as the
> protocol/extension
> 8 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
> EListMapping, EListWrapper at the jpox manager for handling elists
> 8 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
> FeatureMapMapping, FeatureMapWrapper at the jpox manager for handling
> FeatureMap
> 8 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
> EObjectMapping at the jpox manager for handling EObjects/AnyType
> 8 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
> XMLCalendarMapping at the jpox manager for handling EObjects/AnyType
> 9 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
> XMLDurationMapping at the jpox manager for handling EObjects/AnyType
> 1156 [main] INFO JPOX.JDO - PersistenceManagerFactory - Vendor: JPOX
> Version: 1.1.0-rc-1
>
>


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #60357 is a reply to message #60310] Wed, 08 November 2006 16:47 Go to previous messageGo to next message
Steffen Neumann is currently offline Steffen NeumannFriend
Messages: 41
Registered: July 2009
Member
Martin Taal wrote:
> Does the cpu show very much database activity? Or is java the main process?
The CPU load seems to be distributed 30:60 and 50:50 between java/postmaster.

> To prevent the calls to repaircontainer:
....
> properties.setProperty(PMFConfiguration.CACHE_LEVEL_1_TYPE_P ROPERTY,
I found this in the log:
1771 [main] WARN JPOX.JDO - Property PMFConfiguration.CACHE_LEVEL_1_TYPE_PROPERTY unknown - will be ignored

> "org.eclipse.emf.teneo.jpox.cache.EMFWeakRefCache");
Since 1.1.3 they are named differently:
Up to JPOX 1.1.2 org.jpox.cache.WeakRefCache | org.jpox.cache.SoftRefCache | org.jpox.cache.HardRefCache | your class name
From JPOX 1.1.3 weak | soft | hard | {your-plugin-name}

So now I am using
org.jpox.cache.level1.type=weak

This got rid of the repair calls, but runtime is still inacceptable
(still running since 10 mins) I still get for the elements added a large number of (this time)

47933 [main] DEBUG org.eclipse.emf.teneo.mapping.elist.PersistableEList - Created persistable list EList of type: org.eclipse.emf.teneo.jpox.elist.EListWrapper of member comments owned by de.ipb.msbi.Mzdata.impl.SpectrumDescTypeImpl with delegate list java.util.ArrayList
49102 [main] DEBUG org.eclipse.emf.teneo.mapping.elist.PersistableEList - Created persistable list EList of type: org.eclipse.emf.teneo.jpox.elist.EListWrapper of member supDesc owned by de.ipb.msbi.Mzdata.impl.SpectrumType1Impl with delegate list java.util.ArrayList
49103 [main] DEBUG org.eclipse.emf.teneo.mapping.elist.PersistableFeatureMap - Created persistable featuremap FeatureMap of member group owned by de.ipb.msbi.Mzdata.impl.SpectrumType1Impl with delegate list java.util.ArrayList
49103 [main] DEBUG org.eclipse.emf.teneo.jpox.elist.FeatureMapWrapper - Creating feature map wrapper for org.eclipse.emf.teneo.jpox.elist.GenericFeatureMapEntry
49863 [main] DEBUG org.eclipse.emf.teneo.mapping.elist.PersistableFeatureMap - Created persistable featuremap FeatureMap of member group owned by de.ipb.msbi.Mzdata.impl.AcquisitionTypeImpl with delegate list java.util.ArrayList
49863 [main] DEBUG org.eclipse.emf.teneo.jpox.elist.FeatureMapWrapper - Creating feature map wrapper for org.eclipse.emf.teneo.jpox.elist.GenericFeatureMapEntry
49864 [main] DEBUG org.eclipse.emf.teneo.mapping.elist.PersistableEList - Created persistable list EList of type: org.eclipse.emf.teneo.jpox.elist.EListWrapper of member acquisition owned by de.ipb.msbi.Mzdata.impl.AcqSpecificationTypeImpl with delegate list java.util.ArrayList
50250 [main] DEBUG org.eclipse.emf.teneo.mapping.elist.PersistableFeatureMap - Created persistable featuremap FeatureMap of member group owned by de.ipb.msbi.Mzdata.impl.SpectrumInstrumentTypeImpl with delegate list java.util.ArrayList

So I in the mapping strategy between elver and teneo something changed.
Any ideas ?

Yours,
Steffen
Re: Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #60380 is a reply to message #60357] Wed, 08 November 2006 19:51 Go to previous messageGo to next message
Martin Taal is currently offline Martin TaalFriend
Messages: 5468
Registered: July 2009
Senior Member
Hi Steffen,
Were do you write the log (debug) messages to? To a file?

gr. Martin

Steffen Neumann wrote:
> Martin Taal wrote:
>> Does the cpu show very much database activity? Or is java the main
>> process?
> The CPU load seems to be distributed 30:60 and 50:50 between
> java/postmaster.
>
>> To prevent the calls to repaircontainer:
> ....
>> properties.setProperty(PMFConfiguration.CACHE_LEVEL_1_TYPE_P ROPERTY,
> I found this in the log:
> 1771 [main] WARN JPOX.JDO - Property
> PMFConfiguration.CACHE_LEVEL_1_TYPE_PROPERTY unknown - will be ignored
>
>> "org.eclipse.emf.teneo.jpox.cache.EMFWeakRefCache");
> Since 1.1.3 they are named differently:
> Up to JPOX 1.1.2 org.jpox.cache.WeakRefCache |
> org.jpox.cache.SoftRefCache | org.jpox.cache.HardRefCache | your class name
> From JPOX 1.1.3 weak | soft | hard | {your-plugin-name}
>
> So now I am using
> org.jpox.cache.level1.type=weak
>
> This got rid of the repair calls, but runtime is still inacceptable
> (still running since 10 mins) I still get for the elements added a large
> number of (this time)
>
> 47933 [main] DEBUG
> org.eclipse.emf.teneo.mapping.elist.PersistableEList - Created
> persistable list EList of type:
> org.eclipse.emf.teneo.jpox.elist.EListWrapper of member comments owned
> by de.ipb.msbi.Mzdata.impl.SpectrumDescTypeImpl with delegate list
> java.util.ArrayList
> 49102 [main] DEBUG
> org.eclipse.emf.teneo.mapping.elist.PersistableEList - Created
> persistable list EList of type:
> org.eclipse.emf.teneo.jpox.elist.EListWrapper of member supDesc owned by
> de.ipb.msbi.Mzdata.impl.SpectrumType1Impl with delegate list
> java.util.ArrayList
> 49103 [main] DEBUG
> org.eclipse.emf.teneo.mapping.elist.PersistableFeatureMap - Created
> persistable featuremap FeatureMap of member group owned by
> de.ipb.msbi.Mzdata.impl.SpectrumType1Impl with delegate list
> java.util.ArrayList
> 49103 [main] DEBUG
> org.eclipse.emf.teneo.jpox.elist.FeatureMapWrapper - Creating feature
> map wrapper for org.eclipse.emf.teneo.jpox.elist.GenericFeatureMapEntry
> 49863 [main] DEBUG
> org.eclipse.emf.teneo.mapping.elist.PersistableFeatureMap - Created
> persistable featuremap FeatureMap of member group owned by
> de.ipb.msbi.Mzdata.impl.AcquisitionTypeImpl with delegate list
> java.util.ArrayList
> 49863 [main] DEBUG
> org.eclipse.emf.teneo.jpox.elist.FeatureMapWrapper - Creating feature
> map wrapper for org.eclipse.emf.teneo.jpox.elist.GenericFeatureMapEntry
> 49864 [main] DEBUG
> org.eclipse.emf.teneo.mapping.elist.PersistableEList - Created
> persistable list EList of type:
> org.eclipse.emf.teneo.jpox.elist.EListWrapper of member acquisition
> owned by de.ipb.msbi.Mzdata.impl.AcqSpecificationTypeImpl with delegate
> list java.util.ArrayList
> 50250 [main] DEBUG
> org.eclipse.emf.teneo.mapping.elist.PersistableFeatureMap - Created
> persistable featuremap FeatureMap of member group owned by
> de.ipb.msbi.Mzdata.impl.SpectrumInstrumentTypeImpl with delegate list
> java.util.ArrayList
>
> So I in the mapping strategy between elver and teneo something changed.
> Any ideas ?
>
> Yours,
> Steffen


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #60403 is a reply to message #60380] Thu, 09 November 2006 09:54 Go to previous messageGo to next message
Steffen Neumann is currently offline Steffen NeumannFriend
Messages: 41
Registered: July 2009
Member
Martin Taal wrote:
> Hi Steffen,
> Were do you write the log (debug) messages to? To a file?

No, it goes to the Eclipse Console. But things remain slow
even if I set the Debuglevel to Info only.

I also tried dropping & recreating the DB
and recreating package.jdo as , to no avail.

More profiling ideas ?

Yours,
Steffen
Re: Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #60426 is a reply to message #60403] Thu, 09 November 2006 13:31 Go to previous messageGo to next message
Martin Taal is currently offline Martin TaalFriend
Messages: 5468
Registered: July 2009
Senior Member
Hi Steffen,
Performance issues are always tricky. It can be db locking, transaction log size, java heap size....

Can you try mysql?

What does the import logic look like? Do you use the JpoxDataStore import method?

Can you package so that I can easily test it myself?

gr. Martin

Steffen Neumann wrote:
> Martin Taal wrote:
>> Hi Steffen,
>> Were do you write the log (debug) messages to? To a file?
>
> No, it goes to the Eclipse Console. But things remain slow
> even if I set the Debuglevel to Info only.
>
> I also tried dropping & recreating the DB
> and recreating package.jdo as , to no avail.
>
> More profiling ideas ?
>
> Yours,
> Steffen
>


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #60448 is a reply to message #60426] Thu, 09 November 2006 16:10 Go to previous messageGo to next message
Steffen Neumann is currently offline Steffen NeumannFriend
Messages: 41
Registered: July 2009
Member
Martin Taal wrote:
> Can you try mysql?
Yup, same result. Still not finished after 15 mins.

> What does the import logic look like? Do you use the JpoxDataStore
> import method?
No, plain PersistenceManager()

docroot = (DocumentRoot) object;
mzdata = docroot.getMzData();
tx.begin();
pm.makePersistent(mzdata);
tx.commit();

> Can you package so that I can easily test it myself?
Hmpf. That'd expose some bad coding style.
OK, I cleaned up a bit, and send you an URL via PM.

Yours,
Steffen
Re: Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #61424 is a reply to message #60426] Wed, 15 November 2006 09:25 Go to previous message
Steffen Neumann is currently offline Steffen NeumannFriend
Messages: 41
Registered: July 2009
Member
Martin Taal wrote:
> Hi Steffen,
> Performance issues are always tricky. It can be db locking, transaction
> log size, java heap size....

So after some conversation off-list, we're back (and in the archives).

On Di, 2006-11-14 at 18:00 +0100, Martin Taal wrote:
Hi Steffen,
> I found it (I think/hope). The DataType has a member Value which is a
> byte array. As a default Teneo will persist an array in a join table. As
> this makes it possible to query using that information. In case of the
> byte array this resulted in a separate table (DATATYPEIMPL_VALUE) with a
> record for each byte in each instance of DataType.value.
A lot, given around 40+x MB of bytes.

> So solve this:
> You can force the system to persist everything in the DataType table in
> one field by setting the @Lob annotation on the Value EAttribute of
> DataType.
>
> To test it quickly you can also manually change the package.jdo, you
> need to look for a tag:
> <field name="value" persistence-modifier="persistent">
> <array/>
> <join/>
> <element/>
> </field>
>
> This needs to be changed to (just delete join and element):
> <field name="value" persistence-modifier="persistent">
> <array/>
> </field>
>
> The database needs to be recreated for this change to get effect.
> In addition you can set this (just when importing):
> org.jpox.persistenceByReachabilityAtCommit=false
>
> and in log4j.properties:
> log4j.category.JPOX=OFF

Done, and it helps. I am down to ~2 Minutes to persist the large 53MB
with both postgres and MySQL.

So the short answer is that by adding annotation to eCore,
I can trade performance for features (here: individually
adressing bytes in a base64 blob).

I still have a no-go with oracle, but this seems to be
a known deficiency of JPOX, related to versions of client/server;

I'll keep you posted...
69128 [main] INFO JPOX.RDBMS - ================ DatabaseAdapter ==================
69129 [main] INFO JPOX.RDBMS - Adapter : org.jpox.store.rdbms.adapter.OracleAdapter
69129 [main] INFO JPOX.RDBMS - Datastore : name="Oracle" version="Oracle9i Release 9.0.1.1.1 - Production
JServer Release 9.0.1.1.1 - Production" (major=9, minor=0, revision=1)
69129 [main] INFO JPOX.RDBMS - Driver : name="Oracle JDBC driver" version="9.2.0.3.0" (major=9, minor=2)
...
72679 [main] INFO JPOX.RDBMS - Creating table SEQUENCE_TABLE
javax.jdo.JDOFatalInternalException: Somehow org.jpox.store.rdbms.mapping.oracle.OracleBlobRDBMSMapping.s etObject() was called, which should have been impossible!

Thanks for the help,
yours,
Steffen
Re: Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #595267 is a reply to message #60282] Wed, 08 November 2006 13:33 Go to previous message
Martin Taal is currently offline Martin TaalFriend
Messages: 5468
Registered: July 2009
Senior Member
The repaircontrol is done for each item added to the internal cache of jpox and when a resource is
loaded from the database. The repair is called for each object (and its children).

Does the cpu show very much database activity? Or is java the main process?

To prevent the calls to repaircontainer:
As you are persisting (saving) the calls to repaircontrol are probably from the cache. Teneo uses
its own cache implementation to do the repaircontrol. See the constructor in the JpoxDataStore:
properties.setProperty(PMFConfiguration.CACHE_LEVEL_1_TYPE_P ROPERTY,
"org.eclipse.emf.teneo.jpox.cache.EMFWeakRefCache");

What you can try is set the standard jpox weakrefcache (org.jpox.cache.WeakRefCache) instead of the
teneo. You can set this in the properties of the jpoxdatastore,

If this helps then the repair control is the reason for the performance degrade.

gr. Martin

Steffen Neumann wrote:
> Good news upfront:
>
> with 200610261350 and Jpox 1.1.3 I am able to persist
> my model to postgres again. I'll try Oracle this week.
>
> However, the performance dropped to unacceptable levels.
> My 53MB XML/EMF file took around 90secs with JPOX 1.1.0rc1. to persist,
> and now I am well above 25 minutes with the same hardware setup.
>
> Turning on debugging reveals numerous calls to
> EContainerRepairControl(). Is this the reason
> for my performance problem ? Is there a way around that ?
>
> Any more debugging information I should supply ?
>
> Yours,
> Steffen
>
> ------------------------------------------------------------ ---
>
> Trying to persist /vol/tools/E77L_ColHalle_1 1.mzData
> 22762 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.DataTypeImpl
> 22766 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.PeakListBinaryTypeImpl
> 22767 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.PeakListBinaryTypeImpl
> 22767 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing reference data to child de.ipb.msbi.Mzdata.DataType
> 23627 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.DataTypeImpl
> 23628 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.PeakListBinaryTypeImpl
> 23628 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.PeakListBinaryTypeImpl
> 23628 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing reference data to child de.ipb.msbi.Mzdata.DataType
> 24354 [main] DEBUG
> org.eclipse.emf.teneo.mapping.elist.PersistableFeatureMap - Created
> persistable featuremap FeatureMap of member group owned by
> de.ipb.msbi.Mzdata.impl.AcquisitionTypeImpl with delegate list
> java.util.ArrayList
> 24359 [main] DEBUG org.eclipse.emf.teneo.jpox.elist.FeatureMapWrapper -
> Creating feature map wrapper for
> org.eclipse.emf.teneo.jpox.elist.GenericFeatureMapEntry
> 24362 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.AcquisitionTypeImpl
> 24362 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.AcquisitionTypeImpl
> 24362 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing reference cvParam to child de.ipb.msbi.Mzdata.CvParamType
> 24362 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing reference userParam to child de.ipb.msbi.Mzdata.UserParamType
> 24362 [main] DEBUG org.eclipse.emf.teneo.mapping.elist.PersistableEList
> - Created persistable list EList of type:
> org.eclipse.emf.teneo.jpox.elist.EListWrapper of member acquisition
> owned by de.ipb.msbi.Mzdata.impl.AcqSpecificationTypeImpl with delegate
> list java.util.ArrayList
> 24363 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.AcqSpecificationTypeImpl
> 24363 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.AcqSpecificationTypeImpl
> 24363 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing reference acquisition to child de.ipb.msbi.Mzdata.AcquisitionType
>
>
> ------------------------------------------------------------ ---
>
>
> Old, known good configuration:
> Trying to get PersistenceManager...
> 0 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering a
> jpox resource factory for all uri's with jpox as the protocol/extension
> 3 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering a
> jpox resource factory for all uri's with ejdo as the protocol/extension
> 3 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering a
> jpox resource dao factory for all uri's with jpoxdao as the
> protocol/extension
> 8 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
> EListMapping, EListWrapper at the jpox manager for handling elists
> 8 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
> FeatureMapMapping, FeatureMapWrapper at the jpox manager for handling
> FeatureMap
> 8 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
> EObjectMapping at the jpox manager for handling EObjects/AnyType
> 8 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
> XMLCalendarMapping at the jpox manager for handling EObjects/AnyType
> 9 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
> XMLDurationMapping at the jpox manager for handling EObjects/AnyType
> 1156 [main] INFO JPOX.JDO - PersistenceManagerFactory - Vendor: JPOX
> Version: 1.1.0-rc-1
>
>


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #595292 is a reply to message #60310] Wed, 08 November 2006 16:47 Go to previous message
Steffen Neumann is currently offline Steffen NeumannFriend
Messages: 41
Registered: July 2009
Member
Martin Taal wrote:
> Does the cpu show very much database activity? Or is java the main process?
The CPU load seems to be distributed 30:60 and 50:50 between java/postmaster.

> To prevent the calls to repaircontainer:
....
> properties.setProperty(PMFConfiguration.CACHE_LEVEL_1_TYPE_P ROPERTY,
I found this in the log:
1771 [main] WARN JPOX.JDO - Property PMFConfiguration.CACHE_LEVEL_1_TYPE_PROPERTY unknown - will be ignored

> "org.eclipse.emf.teneo.jpox.cache.EMFWeakRefCache");
Since 1.1.3 they are named differently:
Up to JPOX 1.1.2 org.jpox.cache.WeakRefCache | org.jpox.cache.SoftRefCache | org.jpox.cache.HardRefCache | your class name
From JPOX 1.1.3 weak | soft | hard | {your-plugin-name}

So now I am using
org.jpox.cache.level1.type=weak

This got rid of the repair calls, but runtime is still inacceptable
(still running since 10 mins) I still get for the elements added a large number of (this time)

47933 [main] DEBUG org.eclipse.emf.teneo.mapping.elist.PersistableEList - Created persistable list EList of type: org.eclipse.emf.teneo.jpox.elist.EListWrapper of member comments owned by de.ipb.msbi.Mzdata.impl.SpectrumDescTypeImpl with delegate list java.util.ArrayList
49102 [main] DEBUG org.eclipse.emf.teneo.mapping.elist.PersistableEList - Created persistable list EList of type: org.eclipse.emf.teneo.jpox.elist.EListWrapper of member supDesc owned by de.ipb.msbi.Mzdata.impl.SpectrumType1Impl with delegate list java.util.ArrayList
49103 [main] DEBUG org.eclipse.emf.teneo.mapping.elist.PersistableFeatureMap - Created persistable featuremap FeatureMap of member group owned by de.ipb.msbi.Mzdata.impl.SpectrumType1Impl with delegate list java.util.ArrayList
49103 [main] DEBUG org.eclipse.emf.teneo.jpox.elist.FeatureMapWrapper - Creating feature map wrapper for org.eclipse.emf.teneo.jpox.elist.GenericFeatureMapEntry
49863 [main] DEBUG org.eclipse.emf.teneo.mapping.elist.PersistableFeatureMap - Created persistable featuremap FeatureMap of member group owned by de.ipb.msbi.Mzdata.impl.AcquisitionTypeImpl with delegate list java.util.ArrayList
49863 [main] DEBUG org.eclipse.emf.teneo.jpox.elist.FeatureMapWrapper - Creating feature map wrapper for org.eclipse.emf.teneo.jpox.elist.GenericFeatureMapEntry
49864 [main] DEBUG org.eclipse.emf.teneo.mapping.elist.PersistableEList - Created persistable list EList of type: org.eclipse.emf.teneo.jpox.elist.EListWrapper of member acquisition owned by de.ipb.msbi.Mzdata.impl.AcqSpecificationTypeImpl with delegate list java.util.ArrayList
50250 [main] DEBUG org.eclipse.emf.teneo.mapping.elist.PersistableFeatureMap - Created persistable featuremap FeatureMap of member group owned by de.ipb.msbi.Mzdata.impl.SpectrumInstrumentTypeImpl with delegate list java.util.ArrayList

So I in the mapping strategy between elver and teneo something changed.
Any ideas ?

Yours,
Steffen
Re: Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #595297 is a reply to message #60357] Wed, 08 November 2006 19:51 Go to previous message
Martin Taal is currently offline Martin TaalFriend
Messages: 5468
Registered: July 2009
Senior Member
Hi Steffen,
Were do you write the log (debug) messages to? To a file?

gr. Martin

Steffen Neumann wrote:
> Martin Taal wrote:
>> Does the cpu show very much database activity? Or is java the main
>> process?
> The CPU load seems to be distributed 30:60 and 50:50 between
> java/postmaster.
>
>> To prevent the calls to repaircontainer:
> ....
>> properties.setProperty(PMFConfiguration.CACHE_LEVEL_1_TYPE_P ROPERTY,
> I found this in the log:
> 1771 [main] WARN JPOX.JDO - Property
> PMFConfiguration.CACHE_LEVEL_1_TYPE_PROPERTY unknown - will be ignored
>
>> "org.eclipse.emf.teneo.jpox.cache.EMFWeakRefCache");
> Since 1.1.3 they are named differently:
> Up to JPOX 1.1.2 org.jpox.cache.WeakRefCache |
> org.jpox.cache.SoftRefCache | org.jpox.cache.HardRefCache | your class name
> From JPOX 1.1.3 weak | soft | hard | {your-plugin-name}
>
> So now I am using
> org.jpox.cache.level1.type=weak
>
> This got rid of the repair calls, but runtime is still inacceptable
> (still running since 10 mins) I still get for the elements added a large
> number of (this time)
>
> 47933 [main] DEBUG
> org.eclipse.emf.teneo.mapping.elist.PersistableEList - Created
> persistable list EList of type:
> org.eclipse.emf.teneo.jpox.elist.EListWrapper of member comments owned
> by de.ipb.msbi.Mzdata.impl.SpectrumDescTypeImpl with delegate list
> java.util.ArrayList
> 49102 [main] DEBUG
> org.eclipse.emf.teneo.mapping.elist.PersistableEList - Created
> persistable list EList of type:
> org.eclipse.emf.teneo.jpox.elist.EListWrapper of member supDesc owned by
> de.ipb.msbi.Mzdata.impl.SpectrumType1Impl with delegate list
> java.util.ArrayList
> 49103 [main] DEBUG
> org.eclipse.emf.teneo.mapping.elist.PersistableFeatureMap - Created
> persistable featuremap FeatureMap of member group owned by
> de.ipb.msbi.Mzdata.impl.SpectrumType1Impl with delegate list
> java.util.ArrayList
> 49103 [main] DEBUG
> org.eclipse.emf.teneo.jpox.elist.FeatureMapWrapper - Creating feature
> map wrapper for org.eclipse.emf.teneo.jpox.elist.GenericFeatureMapEntry
> 49863 [main] DEBUG
> org.eclipse.emf.teneo.mapping.elist.PersistableFeatureMap - Created
> persistable featuremap FeatureMap of member group owned by
> de.ipb.msbi.Mzdata.impl.AcquisitionTypeImpl with delegate list
> java.util.ArrayList
> 49863 [main] DEBUG
> org.eclipse.emf.teneo.jpox.elist.FeatureMapWrapper - Creating feature
> map wrapper for org.eclipse.emf.teneo.jpox.elist.GenericFeatureMapEntry
> 49864 [main] DEBUG
> org.eclipse.emf.teneo.mapping.elist.PersistableEList - Created
> persistable list EList of type:
> org.eclipse.emf.teneo.jpox.elist.EListWrapper of member acquisition
> owned by de.ipb.msbi.Mzdata.impl.AcqSpecificationTypeImpl with delegate
> list java.util.ArrayList
> 50250 [main] DEBUG
> org.eclipse.emf.teneo.mapping.elist.PersistableFeatureMap - Created
> persistable featuremap FeatureMap of member group owned by
> de.ipb.msbi.Mzdata.impl.SpectrumInstrumentTypeImpl with delegate list
> java.util.ArrayList
>
> So I in the mapping strategy between elver and teneo something changed.
> Any ideas ?
>
> Yours,
> Steffen


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #595309 is a reply to message #60380] Thu, 09 November 2006 09:54 Go to previous message
Steffen Neumann is currently offline Steffen NeumannFriend
Messages: 41
Registered: July 2009
Member
Martin Taal wrote:
> Hi Steffen,
> Were do you write the log (debug) messages to? To a file?

No, it goes to the Eclipse Console. But things remain slow
even if I set the Debuglevel to Info only.

I also tried dropping & recreating the DB
and recreating package.jdo as , to no avail.

More profiling ideas ?

Yours,
Steffen
Re: Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #595319 is a reply to message #60403] Thu, 09 November 2006 13:31 Go to previous message
Martin Taal is currently offline Martin TaalFriend
Messages: 5468
Registered: July 2009
Senior Member
Hi Steffen,
Performance issues are always tricky. It can be db locking, transaction log size, java heap size....

Can you try mysql?

What does the import logic look like? Do you use the JpoxDataStore import method?

Can you package so that I can easily test it myself?

gr. Martin

Steffen Neumann wrote:
> Martin Taal wrote:
>> Hi Steffen,
>> Were do you write the log (debug) messages to? To a file?
>
> No, it goes to the Eclipse Console. But things remain slow
> even if I set the Debuglevel to Info only.
>
> I also tried dropping & recreating the DB
> and recreating package.jdo as , to no avail.
>
> More profiling ideas ?
>
> Yours,
> Steffen
>


--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
Re: Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #595330 is a reply to message #60426] Thu, 09 November 2006 16:10 Go to previous message
Steffen Neumann is currently offline Steffen NeumannFriend
Messages: 41
Registered: July 2009
Member
Martin Taal wrote:
> Can you try mysql?
Yup, same result. Still not finished after 15 mins.

> What does the import logic look like? Do you use the JpoxDataStore
> import method?
No, plain PersistenceManager()

docroot = (DocumentRoot) object;
mzdata = docroot.getMzData();
tx.begin();
pm.makePersistent(mzdata);
tx.commit();

> Can you package so that I can easily test it myself?
Hmpf. That'd expose some bad coding style.
OK, I cleaned up a bit, and send you an URL via PM.

Yours,
Steffen
Re: Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #595661 is a reply to message #60426] Wed, 15 November 2006 09:25 Go to previous message
Steffen Neumann is currently offline Steffen NeumannFriend
Messages: 41
Registered: July 2009
Member
Martin Taal wrote:
> Hi Steffen,
> Performance issues are always tricky. It can be db locking, transaction
> log size, java heap size....

So after some conversation off-list, we're back (and in the archives).

On Di, 2006-11-14 at 18:00 +0100, Martin Taal wrote:
Hi Steffen,
> I found it (I think/hope). The DataType has a member Value which is a
> byte array. As a default Teneo will persist an array in a join table. As
> this makes it possible to query using that information. In case of the
> byte array this resulted in a separate table (DATATYPEIMPL_VALUE) with a
> record for each byte in each instance of DataType.value.
A lot, given around 40+x MB of bytes.

> So solve this:
> You can force the system to persist everything in the DataType table in
> one field by setting the @Lob annotation on the Value EAttribute of
> DataType.
>
> To test it quickly you can also manually change the package.jdo, you
> need to look for a tag:
> <field name="value" persistence-modifier="persistent">
> <array/>
> <join/>
> <element/>
> </field>
>
> This needs to be changed to (just delete join and element):
> <field name="value" persistence-modifier="persistent">
> <array/>
> </field>
>
> The database needs to be recreated for this change to get effect.
> In addition you can set this (just when importing):
> org.jpox.persistenceByReachabilityAtCommit=false
>
> and in log4j.properties:
> log4j.category.JPOX=OFF

Done, and it helps. I am down to ~2 Minutes to persist the large 53MB
with both postgres and MySQL.

So the short answer is that by adding annotation to eCore,
I can trade performance for features (here: individually
adressing bytes in a base64 blob).

I still have a no-go with oracle, but this seems to be
a known deficiency of JPOX, related to versions of client/server;

I'll keep you posted...
69128 [main] INFO JPOX.RDBMS - ================ DatabaseAdapter ==================
69129 [main] INFO JPOX.RDBMS - Adapter : org.jpox.store.rdbms.adapter.OracleAdapter
69129 [main] INFO JPOX.RDBMS - Datastore : name="Oracle" version="Oracle9i Release 9.0.1.1.1 - Production
JServer Release 9.0.1.1.1 - Production" (major=9, minor=0, revision=1)
69129 [main] INFO JPOX.RDBMS - Driver : name="Oracle JDBC driver" version="9.2.0.3.0" (major=9, minor=2)
...
72679 [main] INFO JPOX.RDBMS - Creating table SEQUENCE_TABLE
javax.jdo.JDOFatalInternalException: Somehow org.jpox.store.rdbms.mapping.oracle.OracleBlobRDBMSMapping.s etObject() was called, which should have been impossible!

Thanks for the help,
yours,
Steffen
Previous Topic:Client/Server App with EMF and Teneo
Next Topic:How to declare type of empty collection in OCL?
Goto Forum:
  


Current Time: Fri Feb 26 11:06:49 GMT 2021

Powered by FUDForum. Page generated in 0.02415 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top