Home » Modeling » EMF » [CDO] Raw writes in DB (MySQL)
[CDO] Raw writes in DB (MySQL) [message #989763] |
Fri, 07 December 2012 16:08 |
Christophe MOINE Messages: 34 Registered: February 2012 |
Member |
|
|
Hello,
I first want to apologize to re-open this discussion that I posted one year ago:
I wanted to load large amount of data when initializing the database (150.000 rows), but now we added about 400.000 rows at the initialization process.
I thought about some workarounds, but don't know which one make sense, and which one is safe:
- Make an SQL dump of the concerned tables: should be quick, but annoying to maintain imho (since data are updated monthly...), and doesn't handle nicely the meta model changes (maybe more a feeling than reality).
- Insert directly data in the SQL base, but seems quite risky. Even if we start understanding the CDO data model, we still can miss some subtilities, and we have to ensure that CDO don't use any cache, etc....
- The initialization code runs into the same VM as the CDO Server: isn't it some ways to use directly CDO raw layer classes to initialize the base properly without using the Net4j/Network communication overhead ? For the moment I create a CDO Client into the CDO Server process to initialize the database, but I don't think it is optimal...
- Other ideas... ?
The thing is that my C++ co workers laugh at us because our initialization process takes about 10 minutes, and their takes about 10 sec....
I really hope I could reach some descent performance. I would appreciate some shared experiences.
Thank a lot in advance,
Regards,
Christophe.
Nota: thank you for the automatic migration of the meta model when adding features and tables like Teneo/Hibernate does, this is just great
|
|
| |
Re: [CDO] Raw writes in DB (MySQL) [message #989949 is a reply to message #989763] |
Mon, 10 December 2012 08:44 |
Christophe Bouhier Messages: 937 Registered: July 2009 |
Senior Member |
|
|
Hi Christophe, (Nice name BTW
I have to deal with the same challenge.
On the MM migration, this is unfortunately not possible, there is a bug on the topic, it requires
funding. I think, Eike mentioned the way to migrate now, is to setup a parallel repo, with the new EPackages, and then one-by-one copy the tables over. (With potential customized ECore Util Copier).
On skipping network layer, you can use a JVMConnector. It works like this:
// Prepare container
final IManagedContainer container = IPluginContainer.INSTANCE;
acceptor = JVMUtil.getAcceptor(container, "default");
connector = JVMUtil.getConnector(container, "default");
// Create configuration
final CDOSessionConfiguration sessionConfiguration = CDONet4jUtil
.createSessionConfiguration();
sessionConfiguration.setConnector(connector);
sessionConfiguration.setRepositoryName(REPO_NAME);
Note, this was created for 4.0, and the API changed, but this works with 4.2.
Now on the C++ guys, ok they should have some speed advantage on running the code, but the I/O towards the DB, should be roughly equivalent, so you sure you comparing Pineapples here?
Cheers Christophe
|
|
|
Re: [CDO] Raw writes in DB (MySQL) [message #989960 is a reply to message #989763] |
Mon, 10 December 2012 09:28 |
|
Am 07.12.2012 17:08, schrieb Christophe MOINE:
> Hello,
>
> I first want to apologize to re-open this discussion that I posted one year ago:
> I wanted to load large amount of data when initializing the database (150.000 rows), but now we added about 400.000
> rows at the initialization process.
>
> I thought about some workarounds, but don't know which one make sense, and which one is safe:
> - Make an SQL dump of the concerned tables: should be quick, but annoying to maintain imho (since data are updated
> monthly...), and doesn't handle nicely the meta model changes (maybe more a feeling than reality).
> - Insert directly data in the SQL base, but seems quite risky. Even if we start understanding the CDO data model, we
> still can miss some subtilities, and we have to ensure that CDO don't use any cache, etc....
> - The initialization code runs into the same VM as the CDO Server: isn't it some ways to use directly CDO raw layer
> classes to initialize the base properly without using the Net4j/Network communication overhead ? For the moment I
> create a CDO Client into the CDO Server process to initialize the database, but I don't think it is optimal...
> - Other ideas... ?
>
> The thing is that my C++ co workers laugh at us because our initialization process takes about 10 minutes, and their
> takes about 10 sec....
> I really hope I could reach some descent performance. I would appreciate some shared experiences.
Perhaps you want to try it with the new LissomeStore which I have developed especially to speed up large commits.
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
>
> Thank a lot in advance,
>
> Regards,
>
> Christophe.
>
> Nota: thank you for the automatic migration of the meta model when adding features and tables like Teneo/Hibernate
> does, this is just great ;)
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
| | | |
Re: [CDO] Raw writes in DB (MySQL) [message #990141 is a reply to message #990019] |
Tue, 11 December 2012 06:37 |
|
Am 10.12.2012 15:07, schrieb Christophe MOINE:
> I'm trying LissomeStore, looked at tests.lissome, source code, cdo-light and its cdoperformancetest, and I am a bit
> confused:
>
> If I understand well the LissomeStore comes in place of the relational "db" store type, but both type are called "db":
Oops, I see there's an inconsistency between code and markup:
public class LissomeStore extends Store implements ILissomeStore, CDOAllRevisionsProvider, IDBConnectionProvider,
NoRawAccess, NoLargeObjects, NoQueryXRefs, NoChangeSets, NoDurableLocking
{
public static final String TYPE = "lissome"; //$NON-NLS-1$
---
<extension
point="org.eclipse.emf.cdo.server.storeFactories">
<storeFactory
class="org.eclipse.emf.cdo.server.internal.lissome.LissomeStoreFactory"
storeType="db">
</storeFactory>
</extension>
I've just fixed the latter to "lissome".
You may also want to look at LissomeConfig, which sets up Lissome stores for our tests, see
org.eclipse.emf.cdo.tests.lissome.AllTestsLissome.
> does rely only on what it is found on the osgi "classpath" ?
No.
> Or does it behave as a QueryHandler such as executeQuery("lissome", ...) ?
No. There's no lissome-specific query handler at all.
> Finally I cannot figure out yet why Lissome deals with a File... ?
The LissomeStore stores data in two random access files, a journal file where the commit change sets are simply appended
and an optimizer file (also random access). The optimizer runs asynchronously (the commit thread doesn't have to wait)
on an in-memory queue of the commit change sets creates new revisions and updates search indexes in the RDB.
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
>
> Anyway the concept sounds really nice, but I am a bit lost so far :)
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
| | | | |
Re: [CDO] Raw writes in DB (MySQL) [message #990939 is a reply to message #990855] |
Sat, 15 December 2012 07:15 |
|
Am 14.12.2012 15:08, schrieb Christophe MOINE:
>> > I have another question to help my understanding:
>> > does the LissomeStore lives with the DBStore ? I don't understand this question. Can you elaborate?
>
> Sorry Eike. I mean: does the lissome stores can be setup with the DBStore in the same CDO Server instance for instance
> ? In other words, do I have to configure the store "db" AND "lissome" in the cdo-server.xml ? Or if I configure the
> lissome store, I have to remove the db store configuration ?
The term "CDO server" is not exactly defined, but it makes most sense to see it like this:
A CDO server is a computer with one or more IRepositories and one or more IAcceptors. One repository must have exactly
one IStore (one of DBStore, MEMStore, LissomeStore, etc.). If the stores of different repositories need access to a
database this database can possibly be shared depending on the capabilities of the database engine, but not the same
tables (although even this is thearetically store dependent (as of now I know of no stores that can do this).
> Otherwise I guess I can configure it on another repo name...
Yes, on a different repository, if you need multiple repos with different store types (less common).
>
> So far, I added lissome store in my cdo-server.xml:
> <?xml version="1.0" encoding="UTF-8"?>
> <cdoServer>
> <acceptor type="tcp" listenAddr="0.0.0.0" port="2036"/>
>
> <repository name="repo1">
> <property name="overrideUUID" value=""/>
> <property name="supportingAudits" value="false"/>
>
> <store type="db">
> <mappingStrategy type="horizontal">
> <property name="qualifiedNames" value="false"/>
> <property name="toManyReferences" value="ONE_TABLE_PER_REFERENCE"/>
> <property name="toOneReferences" value="LIKE_ATTRIBUTES"/>
> </mappingStrategy>
>
> <dbAdapter name="mysql"/>
> <dataSource class="com.mysql.jdbc.jdbc2.optional.MysqlDataSource"
> url="jdbc:mysql://${db.host}:${db.port}/${db.schema}"
> user="${db.user}" password="${db.password}"/>
> </store>
>
> <store type="lissome">
> </store>
Only the first <store> element is used. Others are just ignored to make it easier to play with different store types (at
different times).
> </repository>
>
> </cdoServer>
>
> I then configure lissome store on the main container:
> final IManagedContainer container=ContainerUtil.createContainer();
> Net4jUtil.prepareContainer(container); // Register Net4j factories
> TCPUtil.prepareContainer(container); // Register TCP factories
> [...]
> CDONet4jUtil.prepareContainer(container); // Register CDO factories
> LissomeStoreUtil.prepareContainer(container);
The XML-based repository configuration is best supported when OSGi is running. In that case no cutom container has to be
configured/used because the singleton IPluginContainer.INSTANCE is automatically configured through Eclipse's extension
registry.
>
> I cannot see where do I decide to use lissome or db store, but it seems to me more and more clear to configure the
> lissome store on another repository... But then I'm afraid for the CDOID consistency :)
Yes, referential integrity can only be enforced within a single repository. But you can still create "external
references" between the repos (or other resources with EObjects). Use them with care.
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
|
Re: [CDO] Raw writes in DB (MySQL) [message #990940 is a reply to message #990867] |
Sat, 15 December 2012 07:16 |
|
Am 14.12.2012 16:15, schrieb Christophe MOINE:
> I tried with a different repo, and I get:
> !ENTRY org.eclipse.osgi 4 0 2012-12-14 16:12:48.388
> !MESSAGE Erreur liée à l'application
> !STACK 1
> java.lang.IllegalStateException: Revision temporality not supported: NONE
Yes, the LissomeStore is in an early development state and does currently not support other repo modes than branching
(IIRC.), which includes auditing.
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
> at org.eclipse.net4j.util.CheckUtil.checkState(CheckUtil.java:53)
> at org.eclipse.net4j.util.lifecycle.Lifecycle.checkState(Lifecycle.java:229)
> at org.eclipse.emf.cdo.spi.server.Store.setRevisionTemporality(Store.java:193)
> at org.eclipse.emf.cdo.internal.server.Repository.doActivate(Repository.java:1883)
> at org.eclipse.net4j.util.lifecycle.Lifecycle.internalActivate(Lifecycle.java:76)
> at org.eclipse.net4j.util.lifecycle.ShareableLifecycle.internalActivate(ShareableLifecycle.java:43)
> at org.eclipse.net4j.util.lifecycle.Lifecycle.activate(Lifecycle.java:162)
> at org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(LifecycleUtil.java:114)
> at org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(LifecycleUtil.java:104)
> at org.eclipse.emf.cdo.server.CDOServerUtil.addRepository(CDOServerUtil.java:212)
> at org.eclipse.emf.cdo.spi.server.RepositoryConfigurator.configure(RepositoryConfigurator.java:113)
> at org.eclipse.emf.cdo.internal.server.bundle.CDOServerApplication.doStart(CDOServerApplication.java:61)
> at com.robotikdose.importer.ImporterApplication.doStart(ImporterApplication.java:220)
> at org.eclipse.net4j.util.om.OSGiApplication.start(OSGiApplication.java:63)
> at com.robotikdose.app.MLSApplication.start(MLSApplication.java:38)
> at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
> at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
> at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
> at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:353)
> at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:180)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:629)
> at org.eclipse.equinox.launcher.Main.basicRun(Main.java:584)
> at org.eclipse.equinox.launcher.Main.run(Main.java:1438)
> at org.eclipse.equinox.launcher.Main.main(Main.java:1414)
>
>
> Does that mean that I have to enable versioning with supportingAudits=true ?
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
| | | | |
Re: [CDO] Raw writes in DB (MySQL) [message #994071 is a reply to message #991377] |
Wed, 26 December 2012 10:02 |
Christophe MOINE Messages: 34 Registered: February 2012 |
Member |
|
|
Eike Stepper wrote on Tue, 18 December 2012 15:14Am 18.12.2012 17:11, schrieb Christophe MOINE:
> It seems - in an SQL point of view - that the fastest way to insert data is LOAD DATA LOCAL INFILE ...
I've never heard of that one. Is that vendor-specific?
I'm not sure; it might be. I am using MySQL. And it seems to work with Oracle as well.
Eike Stepper wrote on Tue, 18 December 2012 15:14> Just to be sure: the lissomestore don't use this statement right ?
No.
> I would like to use this only before CDO Server starts, so that there is no potential integrity problem.
I don't understand what problem you want to solve and how exactly. Can you please elaborate?
Well, if I modify MySQL tables directly, I must ensure that CDO Server & Client does not hold any cached data, since they might not be up to date. What I mean is that I know what I do is not "good", but I would like to prevent nightmare issues...
Cheers,
Christophe.
|
|
|
Re: [CDO] Raw writes in DB (MySQL) [message #994399 is a reply to message #994071] |
Thu, 27 December 2012 07:25 |
|
Am 26.12.2012 11:02, schrieb Christophe MOINE:
> Eike Stepper wrote on Tue, 18 December 2012 15:14
>> Am 18.12.2012 17:11, schrieb Christophe MOINE:
>> > It seems - in an SQL point of view - that the fastest way to insert data is LOAD DATA LOCAL INFILE ...
>> I've never heard of that one. Is that vendor-specific?
>
>
> I'm not sure; it might be. I am using MySQL. And it seems to work with Oracle as well.
>
> Eike Stepper wrote on Tue, 18 December 2012 15:14
>> > Just to be sure: the lissomestore don't use this statement right ?
>> No.
>>
>> > I would like to use this only before CDO Server starts, so that there is no potential integrity problem.
>> I don't understand what problem you want to solve and how exactly. Can you please elaborate?
>
>
> Well, if I modify MySQL tables directly, I must ensure that CDO Server & Client does not hold any cached data, since
> they might not be up to date. What I mean is that I know what I do is not "good", but I would like to prevent
> nightmare issues...
When the CDO server (i.e., a repository) is running there's most certainly cached data (EPackages and CDORevisions) in it.
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
|
Re: [CDO] Raw writes in DB (MySQL) [message #1000713 is a reply to message #994399] |
Tue, 15 January 2013 10:05 |
Christophe MOINE Messages: 34 Registered: February 2012 |
Member |
|
|
Ok, it still under testing, but it seems on the good way: the import took 8 min, and now takes about 3 minutes (not as fast as I hoped, but prevents many timeouts, and other head overflows...).
The prerequisites so far are the following:
- requires Guava library, and maybe others one
- requires a direct JDBC connection (I query it throught a CDO Query Handler...)
- must be called before the CDOServer load any data in its cache....
- all objects passed as parameters must belongs the the same existing CDOResource
- a new CDOTransaction must be created before, and then closed without being commited (a bit ugly I admit)
- all objects must be attached the the CDOResource content of the 'temporary' transaction.
This is subjects to improvements, but not sure I'll have time to 'unpigify' it
Cheers,
/Christophe
|
|
|
Re: [CDO] Raw writes in DB (MySQL) [message #1000735 is a reply to message #1000713] |
Tue, 15 January 2013 10:52 |
|
Hi Christoph,
Thank you for sharing this! What about sharing your code in the wiki? You could link it from
http://wiki.eclipse.org/CDO/User_Contributed_Documentation#Snippets
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Am 15.01.2013 11:05, schrieb Christophe MOINE:
> Ok, it still under testing, but it seems on the good way: the import took 8 min, and now takes about 3 minutes (not as fast as I hoped, but prevents many timeouts, and other head overflows...).
>
> The prerequisites so far are the following:
> - requires Guava library, and maybe others one
> - requires a direct JDBC connection (I query it throught a CDO Query Handler...)
> - must be called before the CDOServer load any data in its cache....
> - all objects passed as parameters must belongs the the same existing CDOResource
> - a new CDOTransaction must be created before, and then closed without being commited (a bit ugly I admit)
> - all objects must be attached the the CDOResource content of the 'temporary' transaction.
>
> This is subjects to improvements, but not sure I'll have time to 'unpigify' it :)
>
> Cheers,
>
> /Christophe
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
| | | | | |
Goto Forum:
Current Time: Fri Mar 29 09:01:45 GMT 2024
Powered by FUDForum. Page generated in 0.06414 seconds
|