Home » Modeling » EMF » [CDO] Offline mode in an SVN like manner
| [CDO] Offline mode in an SVN like manner [message #639766] |
Wed, 17 November 2010 17:21  |
 Szabolcs Bardy
Messages: 9
Registered: July 2009 |
Junior Member |
|
|
Hi,
I am using the latest release of CDO v3.0
I would like to use the CDO offline capability, but I have the the same problem as written in this post: http://www.eclipse.org/forums/index.php?t=msg&th=199312& amp;start=0&S=8b634d1e14981b7a51b3194b399eb010
I tried to work with the org.eclipse.emf.cdo.examples.server.OfflineExample found in CVS, but once the offline clone losts connection to the master, it can never connect back. I tried with the latest build as well, doesn't work either, I'm receiving Timeout exceptions. Is the error on my side or the example is not a complete one for testing the offline capability?
On the other hand I would like to use the CDO offline mode in an SVN like manner.
The requirements are the following:
- users have to be able to work offline
- the offline clone repository should not commit changes and be updated automatically if connected to the master, these should be handled by manual commands.
My first approach was to tweak the OfflineClone and the RepositorySynchronizer implemetation in the following way:
- Connection to the master, updating the clone and commiting to the master are done always manually
- The user always works into a dedicated work branch of the OfflineClone.
- Update clone -> connect to master, call update, merge result to the work branch
- Commit to master -> connect to master, merge branch back to clone main branch, write through commit copies the change to master
This works fine until a certain point.
The problems are the following:
- when I start up and update the clone for the first time and try to commit something into the work branch, I receive: java.lang.IllegalStateException: Revision was not registered: CDOResource@OID1:1v1. What can be the problem here? If I restart the offline clone repository, I can commit without any problem.
- it is possible that a resource in the master and the client are created with the same cdoID. When it happens, the two repositories cannot be merged again. I suppose this is meant to be solved by the offline branching mechanism, however by default the offline clone and master synchroniztation doesn't work for me. If it should work, could you point me to the code that does the offline branch merge back to the master repository? By the way is there a way to delete branches?
Is CDO meant to work this way at all? Is it a good approach or do you know a better one perhaps?
Thanks for you comments in advance.
Regards,
Szabolcs
|
|
|
| Re: [CDO] Offline mode in an SVN like manner [message #639782 is a reply to message #639766] |
Wed, 17 November 2010 18:18  |
|
Am 17.11.2010 18:21, schrieb Szabolcs Bardy:
> Hi,
>
> I am using the latest release of CDO v3.0
>
> I would like to use the CDO offline capability, but I have the the same problem as written in this post: http://www.eclipse.org/forums/index.php?t=msg&th=199312& amp;start=0&S=8b634d1e14981b7a51b3194b399eb010
>
> I tried to work with the org.eclipse.emf.cdo.examples.server.OfflineExample found in CVS, but once the offline clone losts connection to the master, it can never connect back. I tried with the latest build as well, doesn't work either, I'm receiving Timeout exceptions. Is the error on my side or the example is not a complete one for testing the offline capability?
Sorry, it's possible that the example is not fully working, yet. I did not have enough time before the summit. I think I can look at it tomorrow and then I'll come back with more infos...
>
> On the other hand I would like to use the CDO offline mode in an SVN like manner.
> The requirements are the following:
> - users have to be able to work offline
> - the offline clone repository should not commit changes and be updated automatically if connected to the master, these should be handled by manual commands.
That's what I'm currently devloping the new CDOWorkspace for (see org.eclipse.emf.cdo.workspace.CDOWorkspace). Some examples are in org.eclipse.emf.cdo.tests.WorkspaceTest. But it's also still considered work in progress. I'd appreciate any feedback regarding these new features ;-)
>
> My first approach was to tweak the OfflineClone and the RepositorySynchronizer implemetation in the following way:
> - Connection to the master, updating the clone and commiting to the master are done always manually
> - The user always works into a dedicated work branch of the OfflineClone.
> - Update clone -> connect to master, call update, merge result to the work branch
> - Commit to master -> connect to master, merge branch back to clone main branch, write through commit copies the change to master
>
> This works fine until a certain point.
> The problems are the following:
> - when I start up and update the clone for the first time and try to commit something into the work branch, I receive: java.lang.IllegalStateException: Revision was not registered: mailto:CDOResource@OID1:1v1. What can be the problem here? If I restart the offline clone repository, I can commit without any problem.
> - it is possible that a resource in the master and the client are created with the same cdoID. When it happens, the two repositories cannot be merged again. I suppose this is meant to be solved by the offline branching mechanism, however by default the offline clone and master synchroniztation doesn't work for me. If it should work, could you point me to the code that does the offline branch merge back to the master repository?
Merges are always done either in CDOTransaction.merge(source, merger) or in a CDOMergingConflictResolver. But I strongly recommend against using an OfflineClone if you want an SVN-like workflow!
> By the way is there a way to delete branches?
No. We can think about this if you submit an enhancement request.
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
| |
| Re: [CDO] Offline mode in an SVN like manner [message #640677 is a reply to message #640652] |
Mon, 22 November 2010 19:05 |
|
Am 22.11.2010 18:16, schrieb Szabolcs Bardy:
> Hi Eike,
>
> thanks for the prompt answer and the clarification on this issue. I wasn't aware of the functionality of CDOWorkspace, but as I see it is really what I'm looking for. I started to play around with the examples in WorkspaceTest. Everything works fine when using a MEMStore for both the remote and the local repository. However when I try to use a DBStore I get such exceptions, when trying to commit to the remote repository, or update the local.
> Caused by: java.lang.IllegalStateException: No type found for OID9
> at org.eclipse.emf.cdo.server.internal.db.DBStoreAccessor.getOb jectType(DBStoreAccessor.java:194)
> at org.eclipse.emf.cdo.server.internal.db.DBStoreAccessor.readR evision(DBStoreAccessor.java:207)
> at org.eclipse.emf.cdo.internal.server.Repository.loadRevisions (Repository.java:425)
> at org.eclipse.emf.cdo.internal.common.revision.CDORevisionMana gerImpl.loadRevisions(CDORevisionManagerImpl.java:352)
> at org.eclipse.emf.cdo.internal.common.revision.CDORevisionMana gerImpl.getRevisions(CDORevisionManagerImpl.java:263)
> at org.eclipse.emf.cdo.internal.common.revision.CDORevisionMana gerImpl.getRevision(CDORevisionManagerImpl.java:246)
> at org.eclipse.emf.cdo.spi.common.revision.RevisionInfo.execute (RevisionInfo.java:131)
> at org.eclipse.emf.cdo.server.internal.net4j.protocol.LoadRevis ionsIndication.responding(LoadRevisionsIndication.java:169)
> at org.eclipse.emf.cdo.server.internal.net4j.protocol.CDOServer Indication.responding(CDOServerIndication.java:133)
> at org.eclipse.net4j.signal.IndicationWithResponse.doExtendedOu tput(IndicationWithResponse.java:96)
> at org.eclipse.net4j.signal.Signal.doOutput(Signal.java:286)
> at org.eclipse.net4j.signal.IndicationWithResponse.execute(Indi cationWithResponse.java:65)
> at org.eclipse.emf.cdo.server.internal.net4j.protocol.CDOReadIn dication.execute(CDOReadIndication.java:36)
> ... 5 more
>
> Is it expected? I suppose this is because as you wrote it is still work in progress.
I've seen this bug before but didn't have time to fix it in the DBStore. It only affects the CDOWorkspace.
> I'm looking forward to the further developments.
It's currently on hold because the EclipseCon proposal deadline is approaching. But CDOWorkspace is definitely something we want to have stable before Indigo ;-)
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
| | | |
| Re: [CDO] Offline mode in an SVN like manner [message #641692 is a reply to message #641651] |
Fri, 26 November 2010 18:57 |
|
Am 26.11.2010 15:11, schrieb Guillaume P.:
> Hi Eike,
>
> What do you mean by Online mode ? EMFStore seems to work offline and online too...
Maybe I've missed a new feature there.
>
> Hem... and I just would like to know if you can help me on the problem I have on the offline mode (I had posted a message http://www.eclipse.org/forums/index.php?t=msg&S=a5af3e26 8109319f08abc1cdce4fb2a6&th=199312&goto=638231#msg_6 38231 but it seems that it has been forgotten ^^)
Strange, I've double checked and I can not find your post in my Thunderbird news subscription ;-(
Anyway, it looks like you used either the wrong mapping strategy in the DBStore or the wrong repository mode configuration. These two things must match together. If you set up a repository with branching support (which is needed for an offline clone and its master) you need to use either HorizontalBranchingMappingStrategy or HorizontalBranchingMappingStrategyWithRanges. Are you starting your repo with the cdo-server.xml? If so, use one of these "types" (taken from cdo.server.db/plugin.xml):
<extension
point="org.eclipse.emf.cdo.server.db.mappingStrategies">
<mappingStrategy
class=" org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.Ho rizontalAuditMappingStrategy "
type="horizontal">
</mappingStrategy>
<mappingStrategy
class=" org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.Ho rizontalNonAuditMappingStrategy "
type="horizontalNonAudit">
</mappingStrategy>
<mappingStrategy
class=" org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.Ho rizontalBranchingMappingStrategy "
type="horizontalBranching">
</mappingStrategy>
<mappingStrategy
class=" org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.Ho rizontalAuditMappingStrategyWithRanges "
type="horizontalAuditWithRanges">
</mappingStrategy>
<mappingStrategy
class=" org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.Ho rizontalBranchingMappingStrategyWithRanges "
type="horizontalBranchingWithRanges">
</mappingStrategy>
</extension>
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
|
| Re: [CDO] Offline mode in an SVN like manner [message #642066 is a reply to message #641692] |
Mon, 29 November 2010 17:46 |
Szabolcs Bardy
Messages: 9
Registered: July 2009 |
Junior Member |
|
|
Hi,
I had a look at EMFStore, however as I see EMFStore stores the models as XMI files, which is not really sufficient for large models, as Eike mentioned.
My next question is related to this topic. I need to store quite a lot of objects in the repository. At the moment I am trying to import my model into the CDO repository. My metamodel is not too complex,
it contains nodes with relationships between them, a node contains its outbound relationships. However there are hundred thousands of nodes and millions of relationships.
From the CDO point of view, which is the better approach? Having each node in different CDOResources, or having a large model with all of the contained nodes?
I went for the first option, but when I am trying to import a large amount of data into the repository (commiting 1000 nodes at once), after 30-40000 committed objects it takes a lot of time to commit new objects (more minutes for 1000 object). After profiling the code, it seems that the bottleneck is the database, as the database grows, the process is getting slower and slower. I tried with H2 and MySQL databases. Is it normal, or is there any tricks or options that helps the import of such large amount of data?
I tried to do the import in multiple threads, in this case I got obviously conlicts, when trying to commit into the repository. Maybe a dummy question, but is there perhaps a way to switch off the conflict mechanism?
I am working now with the latest CDO code from the CVS.
Thanks,
Szabolcs
|
|
|
| Re: [CDO] Offline mode in an SVN like manner [message #642148 is a reply to message #642066] |
Tue, 30 November 2010 06:11 |
|
Am 29.11.2010 18:46, schrieb Szabolcs Bardy:
> Hi,
>
> I had a look at EMFStore, however as I see EMFStore stores the models as XMI files, which is not really sufficient for large models, as Eike mentioned.
>
> My next question is related to this topic. I need to store quite a lot of objects in the repository. At the moment I am trying to import my model into the CDO repository. My metamodel is not too complex, it contains nodes with relationships between them, a node contains its outbound relationships. However there are hundred thousands of nodes and millions of relationships.
>
> From the CDO point of view, which is the better approach? Having each node in different CDOResources, or having a large model with all of the contained nodes?
From a pure CDO point of view the "fewer resources" approach is preferrable. Both should work but each resource is a new object and they all need to be contained somewhere. Large lists (like CDOResourceFolder.nodes) are generally less than ideal.
>
> I went for the first option, but when I am trying to import a large amount of data into the repository (commiting 1000 nodes at once), after 30-40000 committed objects it takes a lot of time to commit new objects (more minutes for 1000 object). After profiling the code, it seems that the bottleneck is the database, as the database grows, the process is getting slower and slower. I tried with H2 and MySQL databases. Is it normal, or is there any tricks or options that helps the import of such large amount of data?
Make sure you disable branching and auditing in the repo if you're sure you won't need them. Switch off referential integrity checks for the initial import. If possible split large commits into multiple smaller ones. Do any of these hints help? Maybe Stefan has more recommendations for the DBStore...
>
> I tried to do the import in multiple threads, in this case I got obviously conlicts, when trying to commit into the repository.
That only seems a good idea if these threads do not modify the same objects, e.g. common containers or through bidirectional references.
> Maybe a dummy question, but is there perhaps a way to switch off the conflict mechanism?
Not in the server and I doubt you have explicit CDOConflictResolvers in your client threads.
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
>
> I am working now with the latest CDO code from the CVS.
>
> Thanks,
> Szabolcs
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
| |
| Re: [CDO] Offline mode in an SVN like manner [message #643938 is a reply to message #643636] |
Wed, 08 December 2010 18:00 |
|
Am 07.12.2010 18:18, schrieb Szabolcs Bardy:
> Hi Eike,
>
> Thanks for the tips, they have been very useful.
> It seems that creating separate resources for all of the objects is really not the rigth way. As I wrote, that way the import gets slower and slower as the number of resources grow. When adding each object to a single resource, the import time for one commit is nearly constant, so this solved my problem at least. However I would have another question related to this: I have tried to import ~600000 simple objects with a few attributes to the repository. When using a MEMStore, the commit is executed really fast, in ~26 seconds. When using a MySQL DBStore the same import takes almost an hour. Is such a significant difference expected?
That's hard to say in general. Of course the MEMStore does not really persist anything. That should make a difference. If you send me a zip with an example that I can unpack and run immediately I could try to see if all your settings are optimal.
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
|
Goto Forum:
Current Time: Thu Sep 19 00:43:03 GMT 2024
Powered by FUDForum. Page generated in 0.05350 seconds |
] 
Search
Help
Register
Login
Home
