[CDO] Implement an alternative store [message #809937] |
Wed, 29 February 2012 13:18 |
|
Hi all,
I am looking into the possibility to implement an alternative IStore for CDO. At this tage I am mainly considering feasibility and the effort required. Can you point me to any useful documentation? Do you suggest to just take a look at the code of DBStore & other already existing stores?
Cheers,
Federico
website: http://www.federico-tomassetti.it
|
|
|
Re: [CDO] Implement an alternative store [message #810000 is a reply to message #809937] |
Wed, 29 February 2012 14:49 |
|
Am 29.02.2012 14:18, schrieb Federico Tomassetti:
> Hi all,
> I am looking into the possibility to implement an alternative IStore for CDO. At this tage I am mainly considering
> feasibility
That's definitely feasible and I know of some who've managed.
> and the effort required.
That's impossible to tell in general, i.e., without knowing for what type of backend technology. As an example, it took
me a little less than 10 days to implement the MongoDBStore. But the effort could have been lower if MongoDB offered the
concept of transactions.
And then it depends on the set of optional CDO functionalities you want to support. Here is an excerpt from a former
conversation:
Base functionality
=================
- MongoDB implements IStore
- ID strategy (long IDs? UUID? both? pluggable?)
- Accessor management (connection pooling?)
- System info storage (creation time, first time flag, 2x ID counters, 2x branch counters, last commit time)
- Application property storage
- MongoDBAccessor implements IStoreAccessor
- Storage of package units
- Preloading of package infos
- Lazy loading of packages
- Revision loading
- Revision iteration
- Resource queries
- Commit (including TempID mapping for client-side assigned IDs)
- Rollback
--> All that is comparingly easy but high volume. The main challenge is to find a good and performant mapping to a
MongoDB schema. If we want to have this flexible to a certain degree, it could be a little harder.
Auditing
========
Also not too hard.
Branching
=========
Medium. Hardest impact probably efficient detection of duplicate resources and revision loading. Also high volume as it
normally includes merging support and changeset calculation.
Offline / Raw Access
==================
Medium. (But possibly high volume)
Partial Collection Loading
=======================
Probably very hard with MongoDB.
Automatic crash repair
====================
Easy.
Cross Referencing
=================
Hard to predict for MongoDB. How would MongoDB answer the question "What objects point through a set of given
EReferences to a set of target objects?"
Large Objects
=============
Low volume but making it efficient depends on MongoDB APIs.
Query Handler
==============
Probably easy. Depends on MongoDB API
Commit Infos
=============
Easy
--> That's it, I think. Note that some capabilites depend on each other, like "no offline clone commits without branching".
> Can you point me to any useful documentation? Do you suggest to just take a look at the code of DBStore & other
> already existing stores?
We should have more SPI documentation but as of now the existing store implementations are really your best friend in
understanding. If you have specific questions regarding those or your own attempts, don't hesitate to ask here.
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
|
Re: [CDO] Implement an alternative store [message #810033 is a reply to message #810000] |
Wed, 29 February 2012 15:44 |
|
Thank you Eike.
I would like to save resources as XMI files on a Subversion repository. I would be interested in Auditing and eventually in Branching (but it is less important at the moment). I am starting from MemStore.
At the moment I implemnted writePackageUnits in SvnStoreAccessor (MemStoreAccessor renamed) saving the content of EPackage, now I am trying to understand which resources were affected by changes and save them as XMI file. I think I have to implement this logic in SvnStoreAccessor.doWrite
website: http://www.federico-tomassetti.it
|
|
|
|
Re: [CDO] Implement an alternative store [message #815137 is a reply to message #810062] |
Wed, 07 March 2012 09:21 |
|
At the moment I implemented the "write down data" part, so new metamodels and new revisions of objects are saved.
Now it comes the next part... loading the data when the repository is restarted.
My strategy is to load informations about revisions as soon as the store is loaded.
@Override
protected void doAfterActivate() throws Exception {
System.out.println("LOADING STORE...");
File dataDir = getDataDir();
for (File file : dataDir.listFiles()){
if (file.getName().endsWith(".emf_dump")){
System.out.println("Loading "+file);
String sn = file.getName();
String regex = "CDOID_OID(\\d+)_version_(\\d+)\\.emf_dump";
Pattern p = Pattern.compile(regex);
Matcher m = p.matcher(sn);
if (m.matches()){
int cdoid = Integer.parseInt(m.group(1));
int version = Integer.parseInt(m.group(2));
loadCDOObject(cdoid, version, file);
}
}
}
}
Does it seem reasonable?
My problem currently is that I don't know how to instantiate InternalCDORevision.
@Victor maybe we can join forces
website: http://www.federico-tomassetti.it
|
|
|
Re: [CDO] Implement an alternative store [message #815152 is a reply to message #815137] |
Wed, 07 March 2012 09:37 |
Victor Roldan Betancort Messages: 524 Registered: July 2009 |
Senior Member |
|
|
Hi Federico,
Some comments below
Federico Tomassetti escribió:
> At the moment I implemented the "write down data" part, so new
> metamodels and new revisions of objects are saved.
>
> Now it comes the next part... loading the data when the repository is
> restarted.
>
> My strategy is to load informations about revisions as soon as the store
> is loaded.
>
>
> @Override
> protected void doAfterActivate() throws Exception {
> System.out.println("LOADING STORE...");
> File dataDir = getDataDir();
> for (File file : dataDir.listFiles()){
> if (file.getName().endsWith(".emf_dump")){
> System.out.println("Loading "+file);
> String sn = file.getName();
> String regex = "CDOID_OID(\\d+)_version_(\\d+)\\.emf_dump";
> Pattern p = Pattern.compile(regex);
> Matcher m = p.matcher(sn);
> if (m.matches()){
> int cdoid = Integer.parseInt(m.group(1));
> int version = Integer.parseInt(m.group(2));
> loadCDOObject(cdoid, version, file);
> }
>
> }
> }
> }
>
>
> Does it seem reasonable?
> My problem currently is that I don't know how to instantiate
> InternalCDORevision.
My experience with custom IStores is the DB4OStore which I developed
myself. You should find there some examples of the several parts of
IStore implementation.
Regarding CDORevision instantiation, you could take a look at:
org.eclipse.emf.cdo.server.internal.db4o.DB4ORevision.getCDORevision(IStore,
DB4ORevision)
You only need to get the CDORevisionFactory instance, provided by the
IRepository:
IRepository repository = store.getRepository();
CDORevisionFactory factory =
((InternalCDORevisionManager)repository.getRevisionManager()).getFactory();
.....
InternalCDORevision revision =
(InternalCDORevision)factory.createRevision(eClass);
> @Victor maybe we can join forces
I'm working with Postgresql DBStore support at the moment, but yeah, it
should be feasible. Are you planning to publish your implementation?
Have you thought about contributing to CDO? It would be easier to
collaborate if we had it somewhere like a git repository (yours or
cdo's). Just let me know.
Cheers,
Víctor.
|
|
|
|
|
Re: [CDO] Implement an alternative store [message #815227 is a reply to message #815175] |
Wed, 07 March 2012 11:44 |
|
Am 07.03.2012 11:16, schrieb Federico Tomassetti:
>> did you know EMF has a BinaryResource? It would help you achieve the same results (serializing into a file), but
>> without the hazzle of implementing your own serialization ;)
>
>> org.eclipse.emf.ecore.resource.impl.BinaryResourceImpl
>
> I am going for JSON: it seems to be more interoperable.
>
>> IRepository repository = store.getRepository();
>> CDORevisionFactory factory = ((InternalCDORevisionManager)repository.getRevisionManager()).getFactory();
>
> the revision manager returns a null when I ask for the factory. This seems to me a warning that I am trying to do
> something in the wrong moment...
>
> I could circumvent this issue using CDORevisionFactory.DEFAULT, time will tell if that is wise...
Please have a look at org.eclipse.emf.cdo.internal.server.Repository.doActivate():
LifecycleUtil.activate(store);
LifecycleUtil.activate(packageRegistry);
LifecycleUtil.activate(sessionManager);
LifecycleUtil.activate(revisionManager);
That explains why you can't use the revision manager (nor any other repository infra structure in Store.do(After)Activate.
If you really want to load resources eagerly I recommend to hook into *the first call* to this method:
org.eclipse.emf.cdo.server.IStoreAccessor.readResourceID(CDOID, String, CDOBranchPoint)
It gets called by
--> org.eclipse.emf.cdo.internal.server.Repository.readRootResource()
--> org.eclipse.emf.cdo.internal.server.Repository.doActivate()
in case of server restarts.
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
|
|
|
|
Powered by
FUDForum. Page generated in 0.02917 seconds