Home » Modeling » EMF » [CDO] Offline mode in an SVN like manner
| [CDO] Offline mode in an SVN like manner [message #639766] |
Wed, 17 November 2010 12:21  |
Eclipse User |
|
|
|
Hi,
I am using the latest release of CDO v3.0
I would like to use the CDO offline capability, but I have the the same problem as written in this post: http://www.eclipse.org/forums/index.php?t=msg&th=199312& amp;start=0&S=8b634d1e14981b7a51b3194b399eb010
I tried to work with the org.eclipse.emf.cdo.examples.server.OfflineExample found in CVS, but once the offline clone losts connection to the master, it can never connect back. I tried with the latest build as well, doesn't work either, I'm receiving Timeout exceptions. Is the error on my side or the example is not a complete one for testing the offline capability?
On the other hand I would like to use the CDO offline mode in an SVN like manner.
The requirements are the following:
- users have to be able to work offline
- the offline clone repository should not commit changes and be updated automatically if connected to the master, these should be handled by manual commands.
My first approach was to tweak the OfflineClone and the RepositorySynchronizer implemetation in the following way:
- Connection to the master, updating the clone and commiting to the master are done always manually
- The user always works into a dedicated work branch of the OfflineClone.
- Update clone -> connect to master, call update, merge result to the work branch
- Commit to master -> connect to master, merge branch back to clone main branch, write through commit copies the change to master
This works fine until a certain point.
The problems are the following:
- when I start up and update the clone for the first time and try to commit something into the work branch, I receive: java.lang.IllegalStateException: Revision was not registered: CDOResource@OID1:1v1. What can be the problem here? If I restart the offline clone repository, I can commit without any problem.
- it is possible that a resource in the master and the client are created with the same cdoID. When it happens, the two repositories cannot be merged again. I suppose this is meant to be solved by the offline branching mechanism, however by default the offline clone and master synchroniztation doesn't work for me. If it should work, could you point me to the code that does the offline branch merge back to the master repository? By the way is there a way to delete branches?
Is CDO meant to work this way at all? Is it a good approach or do you know a better one perhaps?
Thanks for you comments in advance.
Regards,
Szabolcs
|
|
|
| Re: [CDO] Offline mode in an SVN like manner [message #639782 is a reply to message #639766] |
Wed, 17 November 2010 13:18  |
| Eclipse User |
|
|
|
Am 17.11.2010 18:21, schrieb Szabolcs Bardy:
> Hi,
>
> I am using the latest release of CDO v3.0
>
> I would like to use the CDO offline capability, but I have the the same problem as written in this post: http://www.eclipse.org/forums/index.php?t=msg&th=199312& amp;start=0&S=8b634d1e14981b7a51b3194b399eb010
>
> I tried to work with the org.eclipse.emf.cdo.examples.server.OfflineExample found in CVS, but once the offline clone losts connection to the master, it can never connect back. I tried with the latest build as well, doesn't work either, I'm receiving Timeout exceptions. Is the error on my side or the example is not a complete one for testing the offline capability?
Sorry, it's possible that the example is not fully working, yet. I did not have enough time before the summit. I think I can look at it tomorrow and then I'll come back with more infos...
>
> On the other hand I would like to use the CDO offline mode in an SVN like manner.
> The requirements are the following:
> - users have to be able to work offline
> - the offline clone repository should not commit changes and be updated automatically if connected to the master, these should be handled by manual commands.
That's what I'm currently devloping the new CDOWorkspace for (see org.eclipse.emf.cdo.workspace.CDOWorkspace). Some examples are in org.eclipse.emf.cdo.tests.WorkspaceTest. But it's also still considered work in progress. I'd appreciate any feedback regarding these new features ;-)
>
> My first approach was to tweak the OfflineClone and the RepositorySynchronizer implemetation in the following way:
> - Connection to the master, updating the clone and commiting to the master are done always manually
> - The user always works into a dedicated work branch of the OfflineClone.
> - Update clone -> connect to master, call update, merge result to the work branch
> - Commit to master -> connect to master, merge branch back to clone main branch, write through commit copies the change to master
>
> This works fine until a certain point.
> The problems are the following:
> - when I start up and update the clone for the first time and try to commit something into the work branch, I receive: java.lang.IllegalStateException: Revision was not registered: mailto:CDOResource@OID1:1v1. What can be the problem here? If I restart the offline clone repository, I can commit without any problem.
> - it is possible that a resource in the master and the client are created with the same cdoID. When it happens, the two repositories cannot be merged again. I suppose this is meant to be solved by the offline branching mechanism, however by default the offline clone and master synchroniztation doesn't work for me. If it should work, could you point me to the code that does the offline branch merge back to the master repository?
Merges are always done either in CDOTransaction.merge(source, merger) or in a CDOMergingConflictResolver. But I strongly recommend against using an OfflineClone if you want an SVN-like workflow!
> By the way is there a way to delete branches?
No. We can think about this if you submit an enhancement request.
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
| | | | | | |
| Re: [CDO] Offline mode in an SVN like manner [message #642066 is a reply to message #641692] |
Mon, 29 November 2010 12:46 |
| Eclipse User |
|
|
|
Hi,
I had a look at EMFStore, however as I see EMFStore stores the models as XMI files, which is not really sufficient for large models, as Eike mentioned.
My next question is related to this topic. I need to store quite a lot of objects in the repository. At the moment I am trying to import my model into the CDO repository. My metamodel is not too complex,
it contains nodes with relationships between them, a node contains its outbound relationships. However there are hundred thousands of nodes and millions of relationships.
From the CDO point of view, which is the better approach? Having each node in different CDOResources, or having a large model with all of the contained nodes?
I went for the first option, but when I am trying to import a large amount of data into the repository (commiting 1000 nodes at once), after 30-40000 committed objects it takes a lot of time to commit new objects (more minutes for 1000 object). After profiling the code, it seems that the bottleneck is the database, as the database grows, the process is getting slower and slower. I tried with H2 and MySQL databases. Is it normal, or is there any tricks or options that helps the import of such large amount of data?
I tried to do the import in multiple threads, in this case I got obviously conlicts, when trying to commit into the repository. Maybe a dummy question, but is there perhaps a way to switch off the conflict mechanism?
I am working now with the latest CDO code from the CVS.
Thanks,
Szabolcs
|
|
|
| Re: [CDO] Offline mode in an SVN like manner [message #642148 is a reply to message #642066] |
Tue, 30 November 2010 01:11 |
| Eclipse User |
|
|
|
Am 29.11.2010 18:46, schrieb Szabolcs Bardy:
> Hi,
>
> I had a look at EMFStore, however as I see EMFStore stores the models as XMI files, which is not really sufficient for large models, as Eike mentioned.
>
> My next question is related to this topic. I need to store quite a lot of objects in the repository. At the moment I am trying to import my model into the CDO repository. My metamodel is not too complex, it contains nodes with relationships between them, a node contains its outbound relationships. However there are hundred thousands of nodes and millions of relationships.
>
> From the CDO point of view, which is the better approach? Having each node in different CDOResources, or having a large model with all of the contained nodes?
From a pure CDO point of view the "fewer resources" approach is preferrable. Both should work but each resource is a new object and they all need to be contained somewhere. Large lists (like CDOResourceFolder.nodes) are generally less than ideal.
>
> I went for the first option, but when I am trying to import a large amount of data into the repository (commiting 1000 nodes at once), after 30-40000 committed objects it takes a lot of time to commit new objects (more minutes for 1000 object). After profiling the code, it seems that the bottleneck is the database, as the database grows, the process is getting slower and slower. I tried with H2 and MySQL databases. Is it normal, or is there any tricks or options that helps the import of such large amount of data?
Make sure you disable branching and auditing in the repo if you're sure you won't need them. Switch off referential integrity checks for the initial import. If possible split large commits into multiple smaller ones. Do any of these hints help? Maybe Stefan has more recommendations for the DBStore...
>
> I tried to do the import in multiple threads, in this case I got obviously conlicts, when trying to commit into the repository.
That only seems a good idea if these threads do not modify the same objects, e.g. common containers or through bidirectional references.
> Maybe a dummy question, but is there perhaps a way to switch off the conflict mechanism?
Not in the server and I doubt you have explicit CDOConflictResolvers in your client threads.
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
>
> I am working now with the latest CDO code from the CVS.
>
> Thanks,
> Szabolcs
|
|
| |
| Re: [CDO] Offline mode in an SVN like manner [message #643938 is a reply to message #643636] |
Wed, 08 December 2010 13:00 |
| Eclipse User |
|
|
|
Am 07.12.2010 18:18, schrieb Szabolcs Bardy:
> Hi Eike,
>
> Thanks for the tips, they have been very useful.
> It seems that creating separate resources for all of the objects is really not the rigth way. As I wrote, that way the import gets slower and slower as the number of resources grow. When adding each object to a single resource, the import time for one commit is nearly constant, so this solved my problem at least. However I would have another question related to this: I have tried to import ~600000 simple objects with a few attributes to the repository. When using a MEMStore, the commit is executed really fast, in ~26 seconds. When using a MySQL DBStore the same import takes almost an hour. Is such a significant difference expected?
That's hard to say in general. Of course the MEMStore does not really persist anything. That should make a difference. If you send me a zip with an example that I can unpack and run immediately I could try to see if all your settings are optimal.
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
|
Goto Forum:
Current Time: Thu Jul 03 03:26:10 EDT 2025
Powered by FUDForum. Page generated in 0.22299 seconds |
Home