Home » Modeling » EMF "Technology" (Ecore Tools, EMFatic, etc) » Performance Problem with Teneo 200610261350 / Jpox 1.1.3
|
Re: Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #60310 is a reply to message #60282] |
Wed, 08 November 2006 13:33 |
Martin Taal Messages: 5468 Registered: July 2009 |
Senior Member |
|
|
The repaircontrol is done for each item added to the internal cache of jpox and when a resource is
loaded from the database. The repair is called for each object (and its children).
Does the cpu show very much database activity? Or is java the main process?
To prevent the calls to repaircontainer:
As you are persisting (saving) the calls to repaircontrol are probably from the cache. Teneo uses
its own cache implementation to do the repaircontrol. See the constructor in the JpoxDataStore:
properties.setProperty(PMFConfiguration.CACHE_LEVEL_1_TYPE_P ROPERTY,
"org.eclipse.emf.teneo.jpox.cache.EMFWeakRefCache");
What you can try is set the standard jpox weakrefcache (org.jpox.cache.WeakRefCache) instead of the
teneo. You can set this in the properties of the jpoxdatastore,
If this helps then the repair control is the reason for the performance degrade.
gr. Martin
Steffen Neumann wrote:
> Good news upfront:
>
> with 200610261350 and Jpox 1.1.3 I am able to persist
> my model to postgres again. I'll try Oracle this week.
>
> However, the performance dropped to unacceptable levels.
> My 53MB XML/EMF file took around 90secs with JPOX 1.1.0rc1. to persist,
> and now I am well above 25 minutes with the same hardware setup.
>
> Turning on debugging reveals numerous calls to
> EContainerRepairControl(). Is this the reason
> for my performance problem ? Is there a way around that ?
>
> Any more debugging information I should supply ?
>
> Yours,
> Steffen
>
> ------------------------------------------------------------ ---
>
> Trying to persist /vol/tools/E77L_ColHalle_1 1.mzData
> 22762 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.DataTypeImpl
> 22766 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.PeakListBinaryTypeImpl
> 22767 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.PeakListBinaryTypeImpl
> 22767 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing reference data to child de.ipb.msbi.Mzdata.DataType
> 23627 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.DataTypeImpl
> 23628 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.PeakListBinaryTypeImpl
> 23628 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.PeakListBinaryTypeImpl
> 23628 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing reference data to child de.ipb.msbi.Mzdata.DataType
> 24354 [main] DEBUG
> org.eclipse.emf.teneo.mapping.elist.PersistableFeatureMap - Created
> persistable featuremap FeatureMap of member group owned by
> de.ipb.msbi.Mzdata.impl.AcquisitionTypeImpl with delegate list
> java.util.ArrayList
> 24359 [main] DEBUG org.eclipse.emf.teneo.jpox.elist.FeatureMapWrapper -
> Creating feature map wrapper for
> org.eclipse.emf.teneo.jpox.elist.GenericFeatureMapEntry
> 24362 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.AcquisitionTypeImpl
> 24362 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.AcquisitionTypeImpl
> 24362 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing reference cvParam to child de.ipb.msbi.Mzdata.CvParamType
> 24362 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing reference userParam to child de.ipb.msbi.Mzdata.UserParamType
> 24362 [main] DEBUG org.eclipse.emf.teneo.mapping.elist.PersistableEList
> - Created persistable list EList of type:
> org.eclipse.emf.teneo.jpox.elist.EListWrapper of member acquisition
> owned by de.ipb.msbi.Mzdata.impl.AcqSpecificationTypeImpl with delegate
> list java.util.ArrayList
> 24363 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.AcqSpecificationTypeImpl
> 24363 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.AcqSpecificationTypeImpl
> 24363 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing reference acquisition to child de.ipb.msbi.Mzdata.AcquisitionType
>
>
> ------------------------------------------------------------ ---
>
>
> Old, known good configuration:
> Trying to get PersistenceManager...
> 0 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering a
> jpox resource factory for all uri's with jpox as the protocol/extension
> 3 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering a
> jpox resource factory for all uri's with ejdo as the protocol/extension
> 3 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering a
> jpox resource dao factory for all uri's with jpoxdao as the
> protocol/extension
> 8 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
> EListMapping, EListWrapper at the jpox manager for handling elists
> 8 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
> FeatureMapMapping, FeatureMapWrapper at the jpox manager for handling
> FeatureMap
> 8 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
> EObjectMapping at the jpox manager for handling EObjects/AnyType
> 8 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
> XMLCalendarMapping at the jpox manager for handling EObjects/AnyType
> 9 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
> XMLDurationMapping at the jpox manager for handling EObjects/AnyType
> 1156 [main] INFO JPOX.JDO - PersistenceManagerFactory - Vendor: JPOX
> Version: 1.1.0-rc-1
>
>
--
With Regards, Martin Taal
Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
|
|
| | | |
Re: Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #60426 is a reply to message #60403] |
Thu, 09 November 2006 13:31 |
Martin Taal Messages: 5468 Registered: July 2009 |
Senior Member |
|
|
Hi Steffen,
Performance issues are always tricky. It can be db locking, transaction log size, java heap size....
Can you try mysql?
What does the import logic look like? Do you use the JpoxDataStore import method?
Can you package so that I can easily test it myself?
gr. Martin
Steffen Neumann wrote:
> Martin Taal wrote:
>> Hi Steffen,
>> Were do you write the log (debug) messages to? To a file?
>
> No, it goes to the Eclipse Console. But things remain slow
> even if I set the Debuglevel to Info only.
>
> I also tried dropping & recreating the DB
> and recreating package.jdo as , to no avail.
>
> More profiling ideas ?
>
> Yours,
> Steffen
>
--
With Regards, Martin Taal
Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
|
|
| |
Re: Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #61424 is a reply to message #60426] |
Wed, 15 November 2006 09:25 |
Steffen Neumann Messages: 41 Registered: July 2009 |
Member |
|
|
Martin Taal wrote:
> Hi Steffen,
> Performance issues are always tricky. It can be db locking, transaction
> log size, java heap size....
So after some conversation off-list, we're back (and in the archives).
On Di, 2006-11-14 at 18:00 +0100, Martin Taal wrote:
Hi Steffen,
> I found it (I think/hope). The DataType has a member Value which is a
> byte array. As a default Teneo will persist an array in a join table. As
> this makes it possible to query using that information. In case of the
> byte array this resulted in a separate table (DATATYPEIMPL_VALUE) with a
> record for each byte in each instance of DataType.value.
A lot, given around 40+x MB of bytes.
> So solve this:
> You can force the system to persist everything in the DataType table in
> one field by setting the @Lob annotation on the Value EAttribute of
> DataType.
>
> To test it quickly you can also manually change the package.jdo, you
> need to look for a tag:
> <field name="value" persistence-modifier="persistent">
> <array/>
> <join/>
> <element/>
> </field>
>
> This needs to be changed to (just delete join and element):
> <field name="value" persistence-modifier="persistent">
> <array/>
> </field>
>
> The database needs to be recreated for this change to get effect.
> In addition you can set this (just when importing):
> org.jpox.persistenceByReachabilityAtCommit=false
>
> and in log4j.properties:
> log4j.category.JPOX=OFF
Done, and it helps. I am down to ~2 Minutes to persist the large 53MB
with both postgres and MySQL.
So the short answer is that by adding annotation to eCore,
I can trade performance for features (here: individually
adressing bytes in a base64 blob).
I still have a no-go with oracle, but this seems to be
a known deficiency of JPOX, related to versions of client/server;
I'll keep you posted...
69128 [main] INFO JPOX.RDBMS - ================ DatabaseAdapter ==================
69129 [main] INFO JPOX.RDBMS - Adapter : org.jpox.store.rdbms.adapter.OracleAdapter
69129 [main] INFO JPOX.RDBMS - Datastore : name="Oracle" version="Oracle9i Release 9.0.1.1.1 - Production
JServer Release 9.0.1.1.1 - Production" (major=9, minor=0, revision=1)
69129 [main] INFO JPOX.RDBMS - Driver : name="Oracle JDBC driver" version="9.2.0.3.0" (major=9, minor=2)
...
72679 [main] INFO JPOX.RDBMS - Creating table SEQUENCE_TABLE
javax.jdo.JDOFatalInternalException: Somehow org.jpox.store.rdbms.mapping.oracle.OracleBlobRDBMSMapping.s etObject() was called, which should have been impossible!
Thanks for the help,
yours,
Steffen
|
|
|
Re: Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #595267 is a reply to message #60282] |
Wed, 08 November 2006 13:33 |
Martin Taal Messages: 5468 Registered: July 2009 |
Senior Member |
|
|
The repaircontrol is done for each item added to the internal cache of jpox and when a resource is
loaded from the database. The repair is called for each object (and its children).
Does the cpu show very much database activity? Or is java the main process?
To prevent the calls to repaircontainer:
As you are persisting (saving) the calls to repaircontrol are probably from the cache. Teneo uses
its own cache implementation to do the repaircontrol. See the constructor in the JpoxDataStore:
properties.setProperty(PMFConfiguration.CACHE_LEVEL_1_TYPE_P ROPERTY,
"org.eclipse.emf.teneo.jpox.cache.EMFWeakRefCache");
What you can try is set the standard jpox weakrefcache (org.jpox.cache.WeakRefCache) instead of the
teneo. You can set this in the properties of the jpoxdatastore,
If this helps then the repair control is the reason for the performance degrade.
gr. Martin
Steffen Neumann wrote:
> Good news upfront:
>
> with 200610261350 and Jpox 1.1.3 I am able to persist
> my model to postgres again. I'll try Oracle this week.
>
> However, the performance dropped to unacceptable levels.
> My 53MB XML/EMF file took around 90secs with JPOX 1.1.0rc1. to persist,
> and now I am well above 25 minutes with the same hardware setup.
>
> Turning on debugging reveals numerous calls to
> EContainerRepairControl(). Is this the reason
> for my performance problem ? Is there a way around that ?
>
> Any more debugging information I should supply ?
>
> Yours,
> Steffen
>
> ------------------------------------------------------------ ---
>
> Trying to persist /vol/tools/E77L_ColHalle_1 1.mzData
> 22762 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.DataTypeImpl
> 22766 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.PeakListBinaryTypeImpl
> 22767 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.PeakListBinaryTypeImpl
> 22767 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing reference data to child de.ipb.msbi.Mzdata.DataType
> 23627 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.DataTypeImpl
> 23628 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.PeakListBinaryTypeImpl
> 23628 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.PeakListBinaryTypeImpl
> 23628 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing reference data to child de.ipb.msbi.Mzdata.DataType
> 24354 [main] DEBUG
> org.eclipse.emf.teneo.mapping.elist.PersistableFeatureMap - Created
> persistable featuremap FeatureMap of member group owned by
> de.ipb.msbi.Mzdata.impl.AcquisitionTypeImpl with delegate list
> java.util.ArrayList
> 24359 [main] DEBUG org.eclipse.emf.teneo.jpox.elist.FeatureMapWrapper -
> Creating feature map wrapper for
> org.eclipse.emf.teneo.jpox.elist.GenericFeatureMapEntry
> 24362 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.AcquisitionTypeImpl
> 24362 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.AcquisitionTypeImpl
> 24362 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing reference cvParam to child de.ipb.msbi.Mzdata.CvParamType
> 24362 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing reference userParam to child de.ipb.msbi.Mzdata.UserParamType
> 24362 [main] DEBUG org.eclipse.emf.teneo.mapping.elist.PersistableEList
> - Created persistable list EList of type:
> org.eclipse.emf.teneo.jpox.elist.EListWrapper of member acquisition
> owned by de.ipb.msbi.Mzdata.impl.AcqSpecificationTypeImpl with delegate
> list java.util.ArrayList
> 24363 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.AcqSpecificationTypeImpl
> 24363 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing container relations of children of:
> de.ipb.msbi.Mzdata.impl.AcqSpecificationTypeImpl
> 24363 [main] DEBUG org.eclipse.emf.teneo.EContainerRepairControl -
> Repairing reference acquisition to child de.ipb.msbi.Mzdata.AcquisitionType
>
>
> ------------------------------------------------------------ ---
>
>
> Old, known good configuration:
> Trying to get PersistenceManager...
> 0 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering a
> jpox resource factory for all uri's with jpox as the protocol/extension
> 3 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering a
> jpox resource factory for all uri's with ejdo as the protocol/extension
> 3 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering a
> jpox resource dao factory for all uri's with jpoxdao as the
> protocol/extension
> 8 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
> EListMapping, EListWrapper at the jpox manager for handling elists
> 8 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
> FeatureMapMapping, FeatureMapWrapper at the jpox manager for handling
> FeatureMap
> 8 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
> EObjectMapping at the jpox manager for handling EObjects/AnyType
> 8 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
> XMLCalendarMapping at the jpox manager for handling EObjects/AnyType
> 9 [main] INFO org.elver.store.jpox.emf.JpoxHelper - Registering
> XMLDurationMapping at the jpox manager for handling EObjects/AnyType
> 1156 [main] INFO JPOX.JDO - PersistenceManagerFactory - Vendor: JPOX
> Version: 1.1.0-rc-1
>
>
--
With Regards, Martin Taal
Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
|
|
| | | |
Re: Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #595319 is a reply to message #60403] |
Thu, 09 November 2006 13:31 |
Martin Taal Messages: 5468 Registered: July 2009 |
Senior Member |
|
|
Hi Steffen,
Performance issues are always tricky. It can be db locking, transaction log size, java heap size....
Can you try mysql?
What does the import logic look like? Do you use the JpoxDataStore import method?
Can you package so that I can easily test it myself?
gr. Martin
Steffen Neumann wrote:
> Martin Taal wrote:
>> Hi Steffen,
>> Were do you write the log (debug) messages to? To a file?
>
> No, it goes to the Eclipse Console. But things remain slow
> even if I set the Debuglevel to Info only.
>
> I also tried dropping & recreating the DB
> and recreating package.jdo as , to no avail.
>
> More profiling ideas ?
>
> Yours,
> Steffen
>
--
With Regards, Martin Taal
Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
|
|
| |
Re: Performance Problem with Teneo 200610261350 / Jpox 1.1.3 [message #595661 is a reply to message #60426] |
Wed, 15 November 2006 09:25 |
Steffen Neumann Messages: 41 Registered: July 2009 |
Member |
|
|
Martin Taal wrote:
> Hi Steffen,
> Performance issues are always tricky. It can be db locking, transaction
> log size, java heap size....
So after some conversation off-list, we're back (and in the archives).
On Di, 2006-11-14 at 18:00 +0100, Martin Taal wrote:
Hi Steffen,
> I found it (I think/hope). The DataType has a member Value which is a
> byte array. As a default Teneo will persist an array in a join table. As
> this makes it possible to query using that information. In case of the
> byte array this resulted in a separate table (DATATYPEIMPL_VALUE) with a
> record for each byte in each instance of DataType.value.
A lot, given around 40+x MB of bytes.
> So solve this:
> You can force the system to persist everything in the DataType table in
> one field by setting the @Lob annotation on the Value EAttribute of
> DataType.
>
> To test it quickly you can also manually change the package.jdo, you
> need to look for a tag:
> <field name="value" persistence-modifier="persistent">
> <array/>
> <join/>
> <element/>
> </field>
>
> This needs to be changed to (just delete join and element):
> <field name="value" persistence-modifier="persistent">
> <array/>
> </field>
>
> The database needs to be recreated for this change to get effect.
> In addition you can set this (just when importing):
> org.jpox.persistenceByReachabilityAtCommit=false
>
> and in log4j.properties:
> log4j.category.JPOX=OFF
Done, and it helps. I am down to ~2 Minutes to persist the large 53MB
with both postgres and MySQL.
So the short answer is that by adding annotation to eCore,
I can trade performance for features (here: individually
adressing bytes in a base64 blob).
I still have a no-go with oracle, but this seems to be
a known deficiency of JPOX, related to versions of client/server;
I'll keep you posted...
69128 [main] INFO JPOX.RDBMS - ================ DatabaseAdapter ==================
69129 [main] INFO JPOX.RDBMS - Adapter : org.jpox.store.rdbms.adapter.OracleAdapter
69129 [main] INFO JPOX.RDBMS - Datastore : name="Oracle" version="Oracle9i Release 9.0.1.1.1 - Production
JServer Release 9.0.1.1.1 - Production" (major=9, minor=0, revision=1)
69129 [main] INFO JPOX.RDBMS - Driver : name="Oracle JDBC driver" version="9.2.0.3.0" (major=9, minor=2)
...
72679 [main] INFO JPOX.RDBMS - Creating table SEQUENCE_TABLE
javax.jdo.JDOFatalInternalException: Somehow org.jpox.store.rdbms.mapping.oracle.OracleBlobRDBMSMapping.s etObject() was called, which should have been impossible!
Thanks for the help,
yours,
Steffen
|
|
|
Goto Forum:
Current Time: Fri Sep 20 06:03:12 GMT 2024
Powered by FUDForum. Page generated in 0.09869 seconds
|