Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF » [CDO] Raw writes in DB (MySQL)
[CDO] Raw writes in DB (MySQL) [message #989763] Fri, 07 December 2012 16:08 Go to next message
Christophe MOINE is currently offline Christophe MOINEFriend
Messages: 34
Registered: February 2012
Member
Hello,

I first want to apologize to re-open this discussion that I posted one year ago:
I wanted to load large amount of data when initializing the database (150.000 rows), but now we added about 400.000 rows at the initialization process.

I thought about some workarounds, but don't know which one make sense, and which one is safe:
- Make an SQL dump of the concerned tables: should be quick, but annoying to maintain imho (since data are updated monthly...), and doesn't handle nicely the meta model changes (maybe more a feeling than reality).
- Insert directly data in the SQL base, but seems quite risky. Even if we start understanding the CDO data model, we still can miss some subtilities, and we have to ensure that CDO don't use any cache, etc....
- The initialization code runs into the same VM as the CDO Server: isn't it some ways to use directly CDO raw layer classes to initialize the base properly without using the Net4j/Network communication overhead ? For the moment I create a CDO Client into the CDO Server process to initialize the database, but I don't think it is optimal...
- Other ideas... ?

The thing is that my C++ co workers laugh at us because our initialization process takes about 10 minutes, and their takes about 10 sec....

I really hope I could reach some descent performance. I would appreciate some shared experiences.

Thank a lot in advance,

Regards,

Christophe.

Nota: thank you for the automatic migration of the meta model when adding features and tables like Teneo/Hibernate does, this is just great Wink
Re: [CDO] Raw writes in DB (MySQL) [message #989905 is a reply to message #989763] Sun, 09 December 2012 19:10 Go to previous messageGo to next message
Christophe MOINE is currently offline Christophe MOINEFriend
Messages: 34
Registered: February 2012
Member
Sorry to mention "competitor" solution, but should I try something like http://code.google.com/p/emf-fragments/ instead ?
Re: [CDO] Raw writes in DB (MySQL) [message #989949 is a reply to message #989763] Mon, 10 December 2012 08:44 Go to previous messageGo to next message
Christophe Bouhier is currently offline Christophe BouhierFriend
Messages: 937
Registered: July 2009
Senior Member
Hi Christophe, (Nice name BTW Smile

I have to deal with the same challenge.

On the MM migration, this is unfortunately not possible, there is a bug on the topic, it requires
funding. I think, Eike mentioned the way to migrate now, is to setup a parallel repo, with the new EPackages, and then one-by-one copy the tables over. (With potential customized ECore Util Copier).

On skipping network layer, you can use a JVMConnector. It works like this:

// Prepare container
		final IManagedContainer container = IPluginContainer.INSTANCE;
		acceptor = JVMUtil.getAcceptor(container, "default");
		connector = JVMUtil.getConnector(container, "default");

		// Create configuration
		final CDOSessionConfiguration sessionConfiguration = CDONet4jUtil
				.createSessionConfiguration();

		sessionConfiguration.setConnector(connector);
		sessionConfiguration.setRepositoryName(REPO_NAME);


Note, this was created for 4.0, and the API changed, but this works with 4.2.

Now on the C++ guys, ok they should have some speed advantage on running the code, but the I/O towards the DB, should be roughly equivalent, so you sure you comparing Pineapples here?

Cheers Christophe


Re: [CDO] Raw writes in DB (MySQL) [message #989960 is a reply to message #989763] Mon, 10 December 2012 09:28 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Am 07.12.2012 17:08, schrieb Christophe MOINE:
> Hello,
>
> I first want to apologize to re-open this discussion that I posted one year ago:
> I wanted to load large amount of data when initializing the database (150.000 rows), but now we added about 400.000
> rows at the initialization process.
>
> I thought about some workarounds, but don't know which one make sense, and which one is safe:
> - Make an SQL dump of the concerned tables: should be quick, but annoying to maintain imho (since data are updated
> monthly...), and doesn't handle nicely the meta model changes (maybe more a feeling than reality).
> - Insert directly data in the SQL base, but seems quite risky. Even if we start understanding the CDO data model, we
> still can miss some subtilities, and we have to ensure that CDO don't use any cache, etc....
> - The initialization code runs into the same VM as the CDO Server: isn't it some ways to use directly CDO raw layer
> classes to initialize the base properly without using the Net4j/Network communication overhead ? For the moment I
> create a CDO Client into the CDO Server process to initialize the database, but I don't think it is optimal...
> - Other ideas... ?
>
> The thing is that my C++ co workers laugh at us because our initialization process takes about 10 minutes, and their
> takes about 10 sec....
> I really hope I could reach some descent performance. I would appreciate some shared experiences.
Perhaps you want to try it with the new LissomeStore which I have developed especially to speed up large commits.

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper


>
> Thank a lot in advance,
>
> Regards,
>
> Christophe.
>
> Nota: thank you for the automatic migration of the meta model when adding features and tables like Teneo/Hibernate
> does, this is just great ;)


Re: [CDO] Raw writes in DB (MySQL) [message #989961 is a reply to message #989905] Mon, 10 December 2012 09:30 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Am 09.12.2012 20:10, schrieb Christophe MOINE:
> Sorry to mention "competitor" solution, but should I try something like http://code.google.com/p/emf-fragments/ instead ?
I don't know that one. That they call CDO an ORM shows that they did not research correctly (universities are more
interested in making new solutions/attempts than enhancing existing ones).

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper


Re: [CDO] Raw writes in DB (MySQL) [message #989968 is a reply to message #989949] Mon, 10 December 2012 10:06 Go to previous messageGo to next message
Christophe MOINE is currently offline Christophe MOINEFriend
Messages: 34
Registered: February 2012
Member
Christophe Bouhier wrote on Mon, 10 December 2012 03:44
Hi Christophe, (Nice name BTW Smile

I have to deal with the same challenge. ...


Very Happy Thank you very much for the compliment and the hint Christophe Smile

I tried the JVMUtil: very interesting, but it doesn't speed out enough unfortunnatly.


Quote:
Perhaps you want to try it with the new LissomeStore which I have developed especially to speed up large commits.


Sound really good Eike: I hope it is not too recent since I need to be compatible with Eclipse 3.7. I'll give it a try right now and let you know.

About EMF Fragment I figured out it is a too recent project, so too risky to adopt for the moment Smile

Cheers,

/Christophe.
Re: [CDO] Raw writes in DB (MySQL) [message #990019 is a reply to message #989968] Mon, 10 December 2012 14:07 Go to previous messageGo to next message
Christophe MOINE is currently offline Christophe MOINEFriend
Messages: 34
Registered: February 2012
Member
I'm trying LissomeStore, looked at tests.lissome, source code, cdo-light and its cdoperformancetest, and I am a bit confused:

If I understand well the LissomeStore comes in place of the relational "db" store type, but both type are called "db": does rely only on what it is found on the osgi "classpath" ? Or does it behave as a QueryHandler such as executeQuery("lissome", ...) ?
Finally I cannot figure out yet why Lissome deals with a File... ?

Anyway the concept sounds really nice, but I am a bit lost so far Smile

Cheers,

/Christophe.
Re: [CDO] Raw writes in DB (MySQL) [message #990141 is a reply to message #990019] Tue, 11 December 2012 06:37 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Am 10.12.2012 15:07, schrieb Christophe MOINE:
> I'm trying LissomeStore, looked at tests.lissome, source code, cdo-light and its cdoperformancetest, and I am a bit
> confused:
>
> If I understand well the LissomeStore comes in place of the relational "db" store type, but both type are called "db":
Oops, I see there's an inconsistency between code and markup:

public class LissomeStore extends Store implements ILissomeStore, CDOAllRevisionsProvider, IDBConnectionProvider,
NoRawAccess, NoLargeObjects, NoQueryXRefs, NoChangeSets, NoDurableLocking
{
public static final String TYPE = "lissome"; //$NON-NLS-1$

---

<extension
point="org.eclipse.emf.cdo.server.storeFactories">
<storeFactory
class="org.eclipse.emf.cdo.server.internal.lissome.LissomeStoreFactory"
storeType="db">
</storeFactory>
</extension>

I've just fixed the latter to "lissome".

You may also want to look at LissomeConfig, which sets up Lissome stores for our tests, see
org.eclipse.emf.cdo.tests.lissome.AllTestsLissome.

> does rely only on what it is found on the osgi "classpath" ?
No.

> Or does it behave as a QueryHandler such as executeQuery("lissome", ...) ?
No. There's no lissome-specific query handler at all.

> Finally I cannot figure out yet why Lissome deals with a File... ?
The LissomeStore stores data in two random access files, a journal file where the commit change sets are simply appended
and an optimizer file (also random access). The optimizer runs asynchronously (the commit thread doesn't have to wait)
on an in-memory queue of the commit change sets creates new revisions and updates search indexes in the RDB.

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper



>
> Anyway the concept sounds really nice, but I am a bit lost so far :)


Re: [CDO] Raw writes in DB (MySQL) [message #990806 is a reply to message #990141] Fri, 14 December 2012 10:38 Go to previous messageGo to next message
Christophe MOINE is currently offline Christophe MOINEFriend
Messages: 34
Registered: February 2012
Member
Hi,

I continue testing Lissome Store every now and again Smile

I used also this snippet:
http://code.google.com/a/eclipselabs.org/p/cdo-light/source/browse/trunk/LibraryCreator/src/CDOPerformanceTests.java?spec=svn79&r=79

It seems that the LissomeStore comes from "cdo light" project is that correct ?

I have another question to help my understanding:
does the LissomeStore lives with the DBStore ? if so, is the next CDOID computed correctly between each other store ? I can't see how...

Thanks a lot,

/Christophe.
Re: [CDO] Raw writes in DB (MySQL) [message #990828 is a reply to message #990806] Fri, 14 December 2012 11:18 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Am 14.12.2012 11:38, schrieb Christophe MOINE:
> Hi,
>
> I continue testing Lissome Store every now and again :)
>
> I used also this snippet:
> http://code.google.com/a/eclipselabs.org/p/cdo-light/source/browse/trunk/LibraryCreator/src/CDOPerformanceTests.java?spec=svn79&r=79
>
Interesting ;-)

>
> It seems that the LissomeStore comes from "cdo light" project is that correct ?
No, the LissomeStore has been developed by myself. I'm in contact with Andy Rytina, though. I asked him to use his
performance suite on my LissomeStore.

> I have another question to help my understanding:
> does the LissomeStore lives with the DBStore ?
I don't understand this question. Can you elaborate?

> if so, is the next CDOID computed correctly between each other store ? I can't see how...
Need to understand the first question first :P

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper


Re: [CDO] Raw writes in DB (MySQL) [message #990855 is a reply to message #990828] Fri, 14 December 2012 14:08 Go to previous messageGo to next message
Christophe MOINE is currently offline Christophe MOINEFriend
Messages: 34
Registered: February 2012
Member
> > I have another question to help my understanding:
> > does the LissomeStore lives with the DBStore ?
> I don't understand this question. Can you elaborate?

Sorry Eike. I mean: does the lissome stores can be setup with the DBStore in the same CDO Server instance for instance ? In other words, do I have to configure the store "db" AND "lissome" in the cdo-server.xml ? Or if I configure the lissome store, I have to remove the db store configuration ?
Otherwise I guess I can configure it on another repo name...

So far, I added lissome store in my cdo-server.xml:
<?xml version="1.0" encoding="UTF-8"?>
<cdoServer>
	<acceptor type="tcp" listenAddr="0.0.0.0" port="2036"/>

	<repository name="repo1">
		<property name="overrideUUID" value=""/>
		<property name="supportingAudits" value="false"/>

		<store type="db">
			<mappingStrategy type="horizontal">
				<property name="qualifiedNames" value="false"/>
				<property name="toManyReferences" value="ONE_TABLE_PER_REFERENCE"/>
				<property name="toOneReferences" value="LIKE_ATTRIBUTES"/>
			</mappingStrategy>

			<dbAdapter name="mysql"/>
			<dataSource class="com.mysql.jdbc.jdbc2.optional.MysqlDataSource" 
				url="jdbc:mysql://${db.host}:${db.port}/${db.schema}"
				user="${db.user}" password="${db.password}"/>
		</store>
		
		<store type="lissome">
		</store>
	</repository>

</cdoServer>


I then configure lissome store on the main container:
final IManagedContainer container=ContainerUtil.createContainer();
Net4jUtil.prepareContainer(container); // Register Net4j factories
TCPUtil.prepareContainer(container); // Register TCP factories
[...]
CDONet4jUtil.prepareContainer(container); // Register CDO factories
LissomeStoreUtil.prepareContainer(container);


I cannot see where do I decide to use lissome or db store, but it seems to me more and more clear to configure the lissome store on another repository... But then I'm afraid for the CDOID consistency Smile

Thanks a lot for your quick answers!

Regards,

/Christophe.

[Updated on: Fri, 14 December 2012 14:09]

Report message to a moderator

Re: [CDO] Raw writes in DB (MySQL) [message #990867 is a reply to message #990855] Fri, 14 December 2012 15:15 Go to previous messageGo to next message
Christophe MOINE is currently offline Christophe MOINEFriend
Messages: 34
Registered: February 2012
Member
I tried with a different repo, and I get:
!ENTRY org.eclipse.osgi 4 0 2012-12-14 16:12:48.388
!MESSAGE Erreur liée à l'application
!STACK 1
java.lang.IllegalStateException: Revision temporality not supported: NONE
	at org.eclipse.net4j.util.CheckUtil.checkState(CheckUtil.java:53)
	at org.eclipse.net4j.util.lifecycle.Lifecycle.checkState(Lifecycle.java:229)
	at org.eclipse.emf.cdo.spi.server.Store.setRevisionTemporality(Store.java:193)
	at org.eclipse.emf.cdo.internal.server.Repository.doActivate(Repository.java:1883)
	at org.eclipse.net4j.util.lifecycle.Lifecycle.internalActivate(Lifecycle.java:76)
	at org.eclipse.net4j.util.lifecycle.ShareableLifecycle.internalActivate(ShareableLifecycle.java:43)
	at org.eclipse.net4j.util.lifecycle.Lifecycle.activate(Lifecycle.java:162)
	at org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(LifecycleUtil.java:114)
	at org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(LifecycleUtil.java:104)
	at org.eclipse.emf.cdo.server.CDOServerUtil.addRepository(CDOServerUtil.java:212)
	at org.eclipse.emf.cdo.spi.server.RepositoryConfigurator.configure(RepositoryConfigurator.java:113)
	at org.eclipse.emf.cdo.internal.server.bundle.CDOServerApplication.doStart(CDOServerApplication.java:61)
	at com.robotikdose.importer.ImporterApplication.doStart(ImporterApplication.java:220)
	at org.eclipse.net4j.util.om.OSGiApplication.start(OSGiApplication.java:63)
	at com.robotikdose.app.MLSApplication.start(MLSApplication.java:38)
	at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:353)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:180)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:629)
	at org.eclipse.equinox.launcher.Main.basicRun(Main.java:584)
	at org.eclipse.equinox.launcher.Main.run(Main.java:1438)
	at org.eclipse.equinox.launcher.Main.main(Main.java:1414)


Does that mean that I have to enable versioning with supportingAudits=true ?
Re: [CDO] Raw writes in DB (MySQL) [message #990939 is a reply to message #990855] Sat, 15 December 2012 07:15 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Am 14.12.2012 15:08, schrieb Christophe MOINE:
>> > I have another question to help my understanding:
>> > does the LissomeStore lives with the DBStore ? I don't understand this question. Can you elaborate?
>
> Sorry Eike. I mean: does the lissome stores can be setup with the DBStore in the same CDO Server instance for instance
> ? In other words, do I have to configure the store "db" AND "lissome" in the cdo-server.xml ? Or if I configure the
> lissome store, I have to remove the db store configuration ?
The term "CDO server" is not exactly defined, but it makes most sense to see it like this:

A CDO server is a computer with one or more IRepositories and one or more IAcceptors. One repository must have exactly
one IStore (one of DBStore, MEMStore, LissomeStore, etc.). If the stores of different repositories need access to a
database this database can possibly be shared depending on the capabilities of the database engine, but not the same
tables (although even this is thearetically store dependent (as of now I know of no stores that can do this).

> Otherwise I guess I can configure it on another repo name...
Yes, on a different repository, if you need multiple repos with different store types (less common).

>
> So far, I added lissome store in my cdo-server.xml:
> <?xml version="1.0" encoding="UTF-8"?>
> <cdoServer>
> <acceptor type="tcp" listenAddr="0.0.0.0" port="2036"/>
>
> <repository name="repo1">
> <property name="overrideUUID" value=""/>
> <property name="supportingAudits" value="false"/>
>
> <store type="db">
> <mappingStrategy type="horizontal">
> <property name="qualifiedNames" value="false"/>
> <property name="toManyReferences" value="ONE_TABLE_PER_REFERENCE"/>
> <property name="toOneReferences" value="LIKE_ATTRIBUTES"/>
> </mappingStrategy>
>
> <dbAdapter name="mysql"/>
> <dataSource class="com.mysql.jdbc.jdbc2.optional.MysqlDataSource"
> url="jdbc:mysql://${db.host}:${db.port}/${db.schema}"
> user="${db.user}" password="${db.password}"/>
> </store>
>
> <store type="lissome">
> </store>
Only the first <store> element is used. Others are just ignored to make it easier to play with different store types (at
different times).

> </repository>
>
> </cdoServer>
>
> I then configure lissome store on the main container:
> final IManagedContainer container=ContainerUtil.createContainer();
> Net4jUtil.prepareContainer(container); // Register Net4j factories
> TCPUtil.prepareContainer(container); // Register TCP factories
> [...]
> CDONet4jUtil.prepareContainer(container); // Register CDO factories
> LissomeStoreUtil.prepareContainer(container);
The XML-based repository configuration is best supported when OSGi is running. In that case no cutom container has to be
configured/used because the singleton IPluginContainer.INSTANCE is automatically configured through Eclipse's extension
registry.

>
> I cannot see where do I decide to use lissome or db store, but it seems to me more and more clear to configure the
> lissome store on another repository... But then I'm afraid for the CDOID consistency :)
Yes, referential integrity can only be enforced within a single repository. But you can still create "external
references" between the repos (or other resources with EObjects). Use them with care.

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper


Re: [CDO] Raw writes in DB (MySQL) [message #990940 is a reply to message #990867] Sat, 15 December 2012 07:16 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Am 14.12.2012 16:15, schrieb Christophe MOINE:
> I tried with a different repo, and I get:
> !ENTRY org.eclipse.osgi 4 0 2012-12-14 16:12:48.388
> !MESSAGE Erreur liée à l'application
> !STACK 1
> java.lang.IllegalStateException: Revision temporality not supported: NONE
Yes, the LissomeStore is in an early development state and does currently not support other repo modes than branching
(IIRC.), which includes auditing.

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper


> at org.eclipse.net4j.util.CheckUtil.checkState(CheckUtil.java:53)
> at org.eclipse.net4j.util.lifecycle.Lifecycle.checkState(Lifecycle.java:229)
> at org.eclipse.emf.cdo.spi.server.Store.setRevisionTemporality(Store.java:193)
> at org.eclipse.emf.cdo.internal.server.Repository.doActivate(Repository.java:1883)
> at org.eclipse.net4j.util.lifecycle.Lifecycle.internalActivate(Lifecycle.java:76)
> at org.eclipse.net4j.util.lifecycle.ShareableLifecycle.internalActivate(ShareableLifecycle.java:43)
> at org.eclipse.net4j.util.lifecycle.Lifecycle.activate(Lifecycle.java:162)
> at org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(LifecycleUtil.java:114)
> at org.eclipse.net4j.util.lifecycle.LifecycleUtil.activate(LifecycleUtil.java:104)
> at org.eclipse.emf.cdo.server.CDOServerUtil.addRepository(CDOServerUtil.java:212)
> at org.eclipse.emf.cdo.spi.server.RepositoryConfigurator.configure(RepositoryConfigurator.java:113)
> at org.eclipse.emf.cdo.internal.server.bundle.CDOServerApplication.doStart(CDOServerApplication.java:61)
> at com.robotikdose.importer.ImporterApplication.doStart(ImporterApplication.java:220)
> at org.eclipse.net4j.util.om.OSGiApplication.start(OSGiApplication.java:63)
> at com.robotikdose.app.MLSApplication.start(MLSApplication.java:38)
> at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
> at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110)
> at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79)
> at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:353)
> at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:180)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:629)
> at org.eclipse.equinox.launcher.Main.basicRun(Main.java:584)
> at org.eclipse.equinox.launcher.Main.run(Main.java:1438)
> at org.eclipse.equinox.launcher.Main.main(Main.java:1414)
>
>
> Does that mean that I have to enable versioning with supportingAudits=true ?


Re: [CDO] Raw writes in DB (MySQL) [message #991348 is a reply to message #990940] Tue, 18 December 2012 16:11 Go to previous messageGo to next message
Christophe MOINE is currently offline Christophe MOINEFriend
Messages: 34
Registered: February 2012
Member
It seems - in an SQL point of view - that the fastest way to insert data is LOAD DATA LOCAL INFILE ...

Just to be sure: the lissomestore don't use this statement right ?

I would like to use this only before CDO Server starts, so that there is no potential integrity problem.

Regards,

/Christophe.
Re: [CDO] Raw writes in DB (MySQL) [message #991365 is a reply to message #991348] Tue, 18 December 2012 18:00 Go to previous messageGo to next message
Erdal Karaca is currently offline Erdal KaracaFriend
Messages: 854
Registered: July 2009
Senior Member
Hm... does that mean that the LissomeStore uses a DBStore as back-end and it can be switched on/off as needed?
Re: [CDO] Raw writes in DB (MySQL) [message #991377 is a reply to message #991348] Tue, 18 December 2012 20:14 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Am 18.12.2012 17:11, schrieb Christophe MOINE:
> It seems - in an SQL point of view - that the fastest way to insert data is LOAD DATA LOCAL INFILE ...
I've never heard of that one. Is that vendor-specific?

> Just to be sure: the lissomestore don't use this statement right ?
No.

> I would like to use this only before CDO Server starts, so that there is no potential integrity problem.
I don't understand what problem you want to solve and how exactly. Can you please elaborate?

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper


Re: [CDO] Raw writes in DB (MySQL) [message #991378 is a reply to message #991365] Tue, 18 December 2012 20:15 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Am 18.12.2012 19:00, schrieb Erdal Karaca:
> Hm... does that mean that the LissomeStore uses a DBStore as back-end and it can be switched on/off as needed?
No, they're totally different. But each on their own they use net4j.db for the DDL abstractions.

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper


Re: [CDO] Raw writes in DB (MySQL) [message #994071 is a reply to message #991377] Wed, 26 December 2012 10:02 Go to previous messageGo to next message
Christophe MOINE is currently offline Christophe MOINEFriend
Messages: 34
Registered: February 2012
Member
Eike Stepper wrote on Tue, 18 December 2012 15:14
Am 18.12.2012 17:11, schrieb Christophe MOINE:
> It seems - in an SQL point of view - that the fastest way to insert data is LOAD DATA LOCAL INFILE ...
I've never heard of that one. Is that vendor-specific?


I'm not sure; it might be. I am using MySQL. And it seems to work with Oracle as well.

Eike Stepper wrote on Tue, 18 December 2012 15:14
> Just to be sure: the lissomestore don't use this statement right ?
No.

> I would like to use this only before CDO Server starts, so that there is no potential integrity problem.
I don't understand what problem you want to solve and how exactly. Can you please elaborate?


Well, if I modify MySQL tables directly, I must ensure that CDO Server & Client does not hold any cached data, since they might not be up to date. What I mean is that I know what I do is not "good", but I would like to prevent nightmare issues...

Cheers,

Christophe.
Re: [CDO] Raw writes in DB (MySQL) [message #994399 is a reply to message #994071] Thu, 27 December 2012 07:25 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Am 26.12.2012 11:02, schrieb Christophe MOINE:
> Eike Stepper wrote on Tue, 18 December 2012 15:14
>> Am 18.12.2012 17:11, schrieb Christophe MOINE:
>> > It seems - in an SQL point of view - that the fastest way to insert data is LOAD DATA LOCAL INFILE ...
>> I've never heard of that one. Is that vendor-specific?
>
>
> I'm not sure; it might be. I am using MySQL. And it seems to work with Oracle as well.
>
> Eike Stepper wrote on Tue, 18 December 2012 15:14
>> > Just to be sure: the lissomestore don't use this statement right ?
>> No.
>>
>> > I would like to use this only before CDO Server starts, so that there is no potential integrity problem.
>> I don't understand what problem you want to solve and how exactly. Can you please elaborate?
>
>
> Well, if I modify MySQL tables directly, I must ensure that CDO Server & Client does not hold any cached data, since
> they might not be up to date. What I mean is that I know what I do is not "good", but I would like to prevent
> nightmare issues...
When the CDO server (i.e., a repository) is running there's most certainly cached data (EPackages and CDORevisions) in it.

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper


Re: [CDO] Raw writes in DB (MySQL) [message #1000713 is a reply to message #994399] Tue, 15 January 2013 10:05 Go to previous messageGo to next message
Christophe MOINE is currently offline Christophe MOINEFriend
Messages: 34
Registered: February 2012
Member
Ok, it still under testing, but it seems on the good way: the import took 8 min, and now takes about 3 minutes (not as fast as I hoped, but prevents many timeouts, and other head overflows...).

The prerequisites so far are the following:
- requires Guava library, and maybe others one
- requires a direct JDBC connection (I query it throught a CDO Query Handler...)
- must be called before the CDOServer load any data in its cache....
- all objects passed as parameters must belongs the the same existing CDOResource
- a new CDOTransaction must be created before, and then closed without being commited (a bit ugly I admit)
- all objects must be attached the the CDOResource content of the 'temporary' transaction.

This is subjects to improvements, but not sure I'll have time to 'unpigify' it Smile

Cheers,

/Christophe
Re: [CDO] Raw writes in DB (MySQL) [message #1000735 is a reply to message #1000713] Tue, 15 January 2013 10:52 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Hi Christoph,

Thank you for sharing this! What about sharing your code in the wiki? You could link it from
http://wiki.eclipse.org/CDO/User_Contributed_Documentation#Snippets

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper


Am 15.01.2013 11:05, schrieb Christophe MOINE:
> Ok, it still under testing, but it seems on the good way: the import took 8 min, and now takes about 3 minutes (not as fast as I hoped, but prevents many timeouts, and other head overflows...).
>
> The prerequisites so far are the following:
> - requires Guava library, and maybe others one
> - requires a direct JDBC connection (I query it throught a CDO Query Handler...)
> - must be called before the CDOServer load any data in its cache....
> - all objects passed as parameters must belongs the the same existing CDOResource
> - a new CDOTransaction must be created before, and then closed without being commited (a bit ugly I admit)
> - all objects must be attached the the CDOResource content of the 'temporary' transaction.
>
> This is subjects to improvements, but not sure I'll have time to 'unpigify' it :)
>
> Cheers,
>
> /Christophe


Re: [CDO] Raw writes in DB (MySQL) [message #1000859 is a reply to message #1000735] Tue, 15 January 2013 15:00 Go to previous messageGo to next message
Christophe MOINE is currently offline Christophe MOINEFriend
Messages: 34
Registered: February 2012
Member
Of course, however, I fixed a bug this morning Wink

How do I share this ? You do it for me ? Or can I do it myself ?
Re: [CDO] Raw writes in DB (MySQL) [message #1000912 is a reply to message #1000859] Tue, 15 January 2013 17:07 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Am 15.01.2013 16:00, schrieb Christophe MOINE:
> Of course, however, I fixed a bug this morning ;)
>
> How do I share this ? You do it for me ? Or can I do it myself ?
What is "this"? The fix to the bug you just mentioned? What is that bug? Or are you talking about editing the wiki?

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper


Re: [CDO] Raw writes in DB (MySQL) [message #1000917 is a reply to message #1000912] Tue, 15 January 2013 17:29 Go to previous messageGo to next message
Christophe MOINE is currently offline Christophe MOINEFriend
Messages: 34
Registered: February 2012
Member
this=the Java file
yes I'm talkign about editing the wiki Smile

Cheers,

/Christophe.
Re: [CDO] Raw writes in DB (MySQL) [message #1000971 is a reply to message #1000917] Tue, 15 January 2013 19:50 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Am 15.01.2013 18:29, schrieb Christophe MOINE:
> this=the Java file
> yes I'm talkign about editing the wiki :)
Of course you're not only allowed but also encouraged to edit it. It's a community resource ;-)

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper


Re: [CDO] Raw writes in DB (MySQL) [message #1001689 is a reply to message #1000971] Thu, 17 January 2013 08:11 Go to previous message
Christophe MOINE is currently offline Christophe MOINEFriend
Messages: 34
Registered: February 2012
Member
Eike Stepper wrote on Tue, 15 January 2013 14:50
Am 15.01.2013 18:29, schrieb Christophe MOINE:
> this=the Java file
> yes I'm talkign about editing the wiki Smile
Of course you're not only allowed but also encouraged to edit it. It's a community resource Wink

Cheers
/Eike


Ok Wink

It's done, not sure it is industrialized yet, but it's a first shot.

Regards,

/Christophe.
Previous Topic:[GWT] SerializationException
Next Topic:[CDO] Exception on commit
Goto Forum:
  


Current Time: Fri Mar 29 09:01:45 GMT 2024

Powered by FUDForum. Page generated in 0.06414 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top