Home » Modeling » EMF » [CDO] CDOServerExporter Out of memory exception
| |
| Re: [CDO] CDOServerExporter Out of memory exception [message #1386136 is a reply to message #1386128] |
Fri, 13 June 2014 14:45   |
 Per Sterner
Messages: 91
Registered: October 2011 |
Member |
|
|
[quote title=Christophe Bouhier wrote on Fri, 13 June 2014 15:45]On 13-06-14 14:39, Per Sterner wrote:
>> I am using the 'CDOServerExporter' and 'CDOServerImporter' classes for a
>> DB migration. First everything worked out.
>How do you migrate your exported data? Have you developped something for
this?
I have tried different approaches:
1. EMFCompare the ecore model and I implemented some DB-Changetasks.
- But this is incomplete and it's dangerous, especially the column 'cdo_feature'. If you add some Attributes in superclasses everything gets shiftet
2. Customized Export and Import (not really depending on CDO). It's not fast, especially the import with references and bidirectional references is customized.
3. CDOServerExport/CDOServerImport:
- I have not looked into xml transformation frameworks. My xml is 2gb in a simple test. The real size seems to be 200gb.
- With the simple test I sucessfully reimported the data, but I made some changes to the Exporter and to the Importer.
- The Importer uses the new Model in the runtime and applies the correct 'cdo_feature' value.
- Additionally I read the page 'http://www.postgresql.org/docs/9.3/static/populate.html' and applied some options to speedup the import
- For testing, I reduced the amount CDO-commits in ''org.eclipse.net4j.db.DBUtil'' on the import. With postgres I got 25% speedup with an internal h2 DB it didn't change much.
>>
>> Then I tried it with a DB with more items and I got an out of memory
>> exception. After searching a lot, I figured out, it is the SQL Statement
>> in 'HorizontalBranchingClassMapping.handleRevisions'.
>>
>> If the function 'setFetchSize' isn't set to anything it fetches the
>> whole data into the memory (At least with postgres).
>>
>Interresting, didn't know about this feature.
Me neither, but it works 
>> Perhaps the class 'DelegatingPreparedStatement' should/could set some
>> value in the constructor?
>>
>> After setting the value it works for 10 million rows (in one table).
>>
>How do you set it? Through the mapping definition in the cdo-server.xml?
Or programmatically?
For now I have manually modified the ''org.eclipse.net4j.db.jdbc'' for testing.
[Updated on: Fri, 13 June 2014 15:25] Report message to a moderator |
|
|
| Re: [CDO] CDOServerExporter Out of memory exception [message #1386404 is a reply to message #1386136] |
Tue, 17 June 2014 10:49 |
Christophe Bouhier
Messages: 937
Registered: July 2009 |
Senior Member |
|
|
Thanks for sharing this Per,
Just to let you know, that I have been investigating/trying CDO model
evolution for some time. I made a test with Edapt which works with EMF
Resources, so basically the migration would work resource-by-resource
and then load the result into a new repo. This however fails in the last
phase, because CDO will not accept the 'evolved' EPackage/nsURI.
without going into the details, it boils down to CDOobject maintaining
ClassInfo, and parent EPackage which will mismatch. Your welcome to have
a look at this here and run the test maybe.
http://git.eclipse.org/c/edapt/org.eclipse.emf.edapt.git/refs/
My 2nd attempt is to let Edapt generate liquibase DB-changetasks which
would be applied in parallel to the Metamodel migration. It's tricky as
the mapping strategy obviously directs the ORM part, which somehow the
liquibase task should know about. (Like the cdo-feature issue you
mention). I am putting a bit of love into this as we speak.
A more profound approach, would be to let CDO handle the migration
om repo level when initializing, considering the ORM strategy and
the evolved EPackage(s) as well as an Edapt.history and/or liquibase DB
change set. After discussing with Eike, he feels this should be store
specific, a good candidate obviously the DBStore.
but..... I would like working together on this (Not on my own). Likely a
branch on CDO, as I expect this will impact various parts of CDO, like
the Package Registry, The store implementation and even more fundamental
in the Revision class.
Cheers Christophe
On 13-06-14 16:45, Per Sterner wrote:
> [quote title=Christophe Bouhier wrote on Fri, 13 June 2014 15:45]On
> 13-06-14 14:39, Per Sterner wrote:
>>> I am using the 'CDOServerExporter' and 'CDOServerImporter' classes for a
>>> DB migration. First everything worked out.
>> How do you migrate your exported data? Have you developped something for
> this?
>
> I have tried different approaches:
> 1. EMFCompare the ecore model and I implemented some DB-Changetasks.
> - But this is incomplete and it's dangerous, especially the column
> 'cdo_feature'. If you add some Attributes in superclasses everything
> gets shiftet
>
> 2. Customized Export and Import (not really depending on CDO). It's not
> fast, especially the import with references and bidirectional references
> is customized.
>
> 3. CDOServerExport/CDOServerImport:
> - I have not looked into xml transformation frameworks. My xml is 2gb
> in a simple test. The real size seems to be 200gb.
> - With the simple test I sucessfully reimported the data, but I made
> some changes to the Exporter and to the Importer.
> - The Importer uses the new Model in the runtime and applies the
> correct 'cdo_feature' value.
>
>>>
>>> Then I tried it with a DB with more items and I got an out of memory
>>> exception. After searching a lot, I figured out, it is the SQL Statement
>>> in 'HorizontalBranchingClassMapping.handleRevisions'.
>>>
>>> If the function 'setFetchSize' isn't set to anything it fetches the
>>> whole data into the memory (At least with postgres).
>>>
>> Interresting, didn't know about this feature.
>
> Me neither, but it works :)
>
>
>>> Perhaps the class 'DelegatingPreparedStatement' should/could set some
>>> value in the constructor?
>>>
>>> After setting the value it works for 10 million rows (in one table).
>>>
>> How do you set it? Through the mapping definition in the cdo-server.xml?
> Or programmatically?
>
> For now I have manually modified the ''org.eclipse.net4j.db.jdbc'' for
> testing.
>
|
|
|
| Re: [CDO] CDOServerExporter Out of memory exception [message #1389212 is a reply to message #1386404] |
Fri, 27 June 2014 13:48 |
Per Sterner
Messages: 91
Registered: October 2011 |
Member |
|
|
Hi,
> without going into the details, it boils down to CDOobject maintaining ClassInfo, and parent EPackage which will mismatch. Your welcome to have a look at this here and run the test maybe.
I tested the Edapt with the examples which really works and looks nice
Furthermore I applied our model modifications for 3 versions. I applied the changes manually (there were not too many changes).
Is it possible to automatically create versions for different ecore files (if the changes aren't too complex)?
> My 2nd attempt is to let Edapt generate liquibase DB-changetasks which would be applied in parallel to the Metamodel migration. It's tricky as the mapping strategy obviously directs the ORM part, which somehow the liquibase task should know about. (Like the cdo-feature issue you mention). I am putting a bit of love into this as we speak.
> A more profound approach, would be to let CDO handle the migration om repo level when initializing, considering the ORM strategy and the evolved EPackage(s) as well as an Edapt.history and/or liquibase DB change set. After discussing with Eike, he feels this should be store specific, a good candidate obviously the DBStore.
> but..... I would like working together on this (Not on my own). Likely a branch on CDO, as I expect this will impact various parts of CDO, like the Package Registry, The store implementation and even more fundamental in the Revision class.
This would be nice. As you mentioned, there would be two (huge) steps?:
- The functionality of Edapt.history and/or liquibase DB change set (I have not looked into liquibase yet)
- CDO Integration
Testing EAdapt I noticed that the order of the attributes/references in an EClass isn't in the history?
Are you intending to share Edapt.history and/or liquibase DB change set functionallity? Of course I'd really like to test this, give feedback and help if I can.
|
|
|
| Re: [CDO] CDOServerExporter Out of memory exception [message #1390927 is a reply to message #1389212] |
Mon, 30 June 2014 07:30 |
Christophe Bouhier
Messages: 937
Registered: July 2009 |
Senior Member |
|
|
On 27-06-14 15:48, Per Sterner wrote:
> Hi,
>
>> without going into the details, it boils down to CDOobject maintaining
>> ClassInfo, and parent EPackage which will mismatch. Your welcome to
>> have a look at this here and run the test maybe.
>
> I tested the Edapt with the examples which really works and looks nice :)
>
> Furthermore I applied our model modifications for 3 versions. I applied
> the changes manually (there were not too many changes).
>
> Is it possible to automatically create versions for different ecore
> files (if the changes aren't too complex)?
The user has to specify when a bunch of operations are considered a new
release by specifying a new nsURI. Not sure what you ask actually...
>
>> My 2nd attempt is to let Edapt generate liquibase DB-changetasks which
>> would be applied in parallel to the Metamodel migration. It's tricky
>> as the mapping strategy obviously directs the ORM part, which somehow
>> the liquibase task should know about. (Like the cdo-feature issue you
>> mention). I am putting a bit of love into this as we speak.
>> A more profound approach, would be to let CDO handle the migration om
>> repo level when initializing, considering the ORM strategy and the
>> evolved EPackage(s) as well as an Edapt.history and/or liquibase DB
>> change set. After discussing with Eike, he feels this should be store
>> specific, a good candidate obviously the DBStore.
>> but..... I would like working together on this (Not on my own). Likely
>> a branch on CDO, as I expect this will impact various parts of CDO,
>> like the Package Registry, The store implementation and even more
>> fundamental in the Revision class.
>
> This would be nice. As you mentioned, there would be two (huge) steps?:
> - The functionality of Edapt.history and/or liquibase DB change set (I
> have not looked into liquibase yet)
> - CDO Integration
>
> Testing EAdapt I noticed that the order of the attributes/references in
> an EClass isn't in the history?
Isn't the order implicit the order in which Change operations are
applied? I am not sure, if during processing of EAttributes, some order
is applied.
>
> Are you intending to share Edapt.history and/or liquibase DB change set
> functionallity?
Yes, the CDO topic branch on Edapt git repo, is EPL ;-)
Of course I'd really like to test this, give feedback
> and help if I can.
This would be nice, I will try to set out some sort of mini-charter...
>
|
|
|
Goto Forum:
Current Time: Sat Apr 27 03:45:23 GMT 2024
Powered by FUDForum. Page generated in 0.03152 seconds |
] 
Search
Help
Register
Login
Home
