Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF » [CDO] Implement an alternative store
[CDO] Implement an alternative store [message #809937] Wed, 29 February 2012 08:18 Go to next message
Federico Tomassetti is currently offline Federico Tomassetti
Messages: 186
Registered: July 2009
Location: Torino
Senior Member

Hi all,
I am looking into the possibility to implement an alternative IStore for CDO. At this tage I am mainly considering feasibility and the effort required. Can you point me to any useful documentation? Do you suggest to just take a look at the code of DBStore & other already existing stores?

Cheers,

Federico


Re: [CDO] Implement an alternative store [message #810000 is a reply to message #809937] Wed, 29 February 2012 09:49 Go to previous messageGo to next message
Eike Stepper is currently offline Eike Stepper
Messages: 5499
Registered: July 2009
Senior Member
Am 29.02.2012 14:18, schrieb Federico Tomassetti:
> Hi all,
> I am looking into the possibility to implement an alternative IStore for CDO. At this tage I am mainly considering
> feasibility
That's definitely feasible and I know of some who've managed.

> and the effort required.
That's impossible to tell in general, i.e., without knowing for what type of backend technology. As an example, it took
me a little less than 10 days to implement the MongoDBStore. But the effort could have been lower if MongoDB offered the
concept of transactions.

And then it depends on the set of optional CDO functionalities you want to support. Here is an excerpt from a former
conversation:

Base functionality
=================

- MongoDB implements IStore
- ID strategy (long IDs? UUID? both? pluggable?)
- Accessor management (connection pooling?)
- System info storage (creation time, first time flag, 2x ID counters, 2x branch counters, last commit time)
- Application property storage

- MongoDBAccessor implements IStoreAccessor
- Storage of package units
- Preloading of package infos
- Lazy loading of packages
- Revision loading
- Revision iteration
- Resource queries
- Commit (including TempID mapping for client-side assigned IDs)
- Rollback

--> All that is comparingly easy but high volume. The main challenge is to find a good and performant mapping to a
MongoDB schema. If we want to have this flexible to a certain degree, it could be a little harder.

Auditing
========

Also not too hard.

Branching
=========

Medium. Hardest impact probably efficient detection of duplicate resources and revision loading. Also high volume as it
normally includes merging support and changeset calculation.

Offline / Raw Access
==================

Medium. (But possibly high volume)

Partial Collection Loading
=======================

Probably very hard with MongoDB.

Automatic crash repair
====================

Easy.

Cross Referencing
=================

Hard to predict for MongoDB. How would MongoDB answer the question "What objects point through a set of given
EReferences to a set of target objects?"

Large Objects
=============

Low volume but making it efficient depends on MongoDB APIs.

Query Handler
==============

Probably easy. Depends on MongoDB API

Commit Infos
=============

Easy

--> That's it, I think. Note that some capabilites depend on each other, like "no offline clone commits without branching".

> Can you point me to any useful documentation? Do you suggest to just take a look at the code of DBStore & other
> already existing stores?
We should have more SPI documentation but as of now the existing store implementations are really your best friend in
understanding. If you have specific questions regarding those or your own attempts, don't hesitate to ask here.

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Re: [CDO] Implement an alternative store [message #810033 is a reply to message #810000] Wed, 29 February 2012 10:44 Go to previous messageGo to next message
Federico Tomassetti is currently offline Federico Tomassetti
Messages: 186
Registered: July 2009
Location: Torino
Senior Member

Thank you Eike.

I would like to save resources as XMI files on a Subversion repository. I would be interested in Auditing and eventually in Branching (but it is less important at the moment). I am starting from MemStore.

At the moment I implemnted writePackageUnits in SvnStoreAccessor (MemStoreAccessor renamed) saving the content of EPackage, now I am trying to understand which resources were affected by changes and save them as XMI file. I think I have to implement this logic in SvnStoreAccessor.doWrite


Re: [CDO] Implement an alternative store [message #810062 is a reply to message #810033] Wed, 29 February 2012 11:28 Go to previous messageGo to next message
Victor Roldan Betancort is currently offline Victor Roldan Betancort
Messages: 524
Registered: July 2009
Senior Member
Federico,

Federico Tomassetti escribió:
> Thank you Eike.
>
> I would like to save resources as XMI files on a Subversion repository.
> I would be interested in Auditing and eventually in Branching (but it is
> less important at the moment). I am starting from MemStore.
>
> At the moment I implemnted writePackageUnits in SvnStoreAccessor
> (MemStoreAccessor renamed) saving the content of EPackage, now I am
> trying to understand which resources were affected by changes and save
> them as XMI file. I think I have to implement this logic in
> SvnStoreAccessor.doWrite

just FYI, there is a bug already raised long time ago, but not progress
in it :(

263934: [FileStore] Create a filestore
https://bugs.eclipse.org/bugs/show_bug.cgi?id=263934
Re: [CDO] Implement an alternative store [message #815137 is a reply to message #810062] Wed, 07 March 2012 04:21 Go to previous messageGo to next message
Federico Tomassetti is currently offline Federico Tomassetti
Messages: 186
Registered: July 2009
Location: Torino
Senior Member

At the moment I implemented the "write down data" part, so new metamodels and new revisions of objects are saved.

Now it comes the next part... loading the data when the repository is restarted.

My strategy is to load informations about revisions as soon as the store is loaded.

	@Override
	protected void doAfterActivate() throws Exception {
		System.out.println("LOADING STORE...");
		File dataDir = getDataDir();
		for (File file : dataDir.listFiles()){
			if (file.getName().endsWith(".emf_dump")){
				System.out.println("Loading "+file);
				String sn = file.getName();
				String regex = "CDOID_OID(\\d+)_version_(\\d+)\\.emf_dump";
				Pattern p = Pattern.compile(regex);
				Matcher m = p.matcher(sn);
				if (m.matches()){
					int cdoid = Integer.parseInt(m.group(1));
					int version = Integer.parseInt(m.group(2));
					loadCDOObject(cdoid, version, file);
				}
				
			}
		}
	}


Does it seem reasonable?
My problem currently is that I don't know how to instantiate InternalCDORevision.


@Victor maybe we can join forces


Re: [CDO] Implement an alternative store [message #815152 is a reply to message #815137] Wed, 07 March 2012 04:37 Go to previous messageGo to next message
Victor Roldan Betancort is currently offline Victor Roldan Betancort
Messages: 524
Registered: July 2009
Senior Member
Hi Federico,

Some comments below

Federico Tomassetti escribió:
> At the moment I implemented the "write down data" part, so new
> metamodels and new revisions of objects are saved.
>
> Now it comes the next part... loading the data when the repository is
> restarted.
>
> My strategy is to load informations about revisions as soon as the store
> is loaded.
>
>
> @Override
> protected void doAfterActivate() throws Exception {
> System.out.println("LOADING STORE...");
> File dataDir = getDataDir();
> for (File file : dataDir.listFiles()){
> if (file.getName().endsWith(".emf_dump")){
> System.out.println("Loading "+file);
> String sn = file.getName();
> String regex = "CDOID_OID(\\d+)_version_(\\d+)\\.emf_dump";
> Pattern p = Pattern.compile(regex);
> Matcher m = p.matcher(sn);
> if (m.matches()){
> int cdoid = Integer.parseInt(m.group(1));
> int version = Integer.parseInt(m.group(2));
> loadCDOObject(cdoid, version, file);
> }
>
> }
> }
> }
>
>
> Does it seem reasonable?
> My problem currently is that I don't know how to instantiate
> InternalCDORevision.

My experience with custom IStores is the DB4OStore which I developed
myself. You should find there some examples of the several parts of
IStore implementation.

Regarding CDORevision instantiation, you could take a look at:

org.eclipse.emf.cdo.server.internal.db4o.DB4ORevision.getCDORevision(IStore,
DB4ORevision)

You only need to get the CDORevisionFactory instance, provided by the
IRepository:

IRepository repository = store.getRepository();
CDORevisionFactory factory =
((InternalCDORevisionManager)repository.getRevisionManager()).getFactory();

.....

InternalCDORevision revision =
(InternalCDORevision)factory.createRevision(eClass);

> @Victor maybe we can join forces

I'm working with Postgresql DBStore support at the moment, but yeah, it
should be feasible. Are you planning to publish your implementation?
Have you thought about contributing to CDO? It would be easier to
collaborate if we had it somewhere like a git repository (yours or
cdo's). Just let me know.


Cheers,
Víctor.
Re: [CDO] Implement an alternative store [message #815153 is a reply to message #815137] Wed, 07 March 2012 04:40 Go to previous messageGo to next message
Victor Roldan Betancort is currently offline Victor Roldan Betancort
Messages: 524
Registered: July 2009
Senior Member
By the way,

did you know EMF has a BinaryResource? It would help you achieve the
same results (serializing into a file), but without the hazzle of
implementing your own serialization ;)

org.eclipse.emf.ecore.resource.impl.BinaryResourceImpl
Re: [CDO] Implement an alternative store [message #815175 is a reply to message #815152] Wed, 07 March 2012 05:16 Go to previous messageGo to next message
Federico Tomassetti is currently offline Federico Tomassetti
Messages: 186
Registered: July 2009
Location: Torino
Senior Member

>did you know EMF has a BinaryResource? It would help you achieve the
>same results (serializing into a file), but without the hazzle of
>implementing your own serialization Wink

>org.eclipse.emf.ecore.resource.impl.BinaryResourceImpl

I am going for JSON: it seems to be more interoperable.

> IRepository repository = store.getRepository();
> CDORevisionFactory factory =
> ((InternalCDORevisionManager)repository.getRevisionManager()).getFactory();

the revision manager returns a null when I ask for the factory. This seems to me a warning that I am trying to do something in the wrong moment...

I could circumvent this issue using CDORevisionFactory.DEFAULT, time will tell if that is wise...

>Are you planning to publish your implementation?
>Have you thought about contributing to CDO? It would be easier to
>collaborate if we had it somewhere like a git repository (yours or
>cdo's). Just let me know.

Definitely I would share my implementation. The only problem is that I am working some hours per week on this project for a company and at this stage we are just evaluating the possibility to implement this store, so I can not guarantee I will work on that in the future.


Re: [CDO] Implement an alternative store [message #815227 is a reply to message #815175] Wed, 07 March 2012 06:44 Go to previous messageGo to next message
Eike Stepper is currently offline Eike Stepper
Messages: 5499
Registered: July 2009
Senior Member
Am 07.03.2012 11:16, schrieb Federico Tomassetti:
>> did you know EMF has a BinaryResource? It would help you achieve the same results (serializing into a file), but
>> without the hazzle of implementing your own serialization ;)
>
>> org.eclipse.emf.ecore.resource.impl.BinaryResourceImpl
>
> I am going for JSON: it seems to be more interoperable.
>
>> IRepository repository = store.getRepository();
>> CDORevisionFactory factory = ((InternalCDORevisionManager)repository.getRevisionManager()).getFactory();
>
> the revision manager returns a null when I ask for the factory. This seems to me a warning that I am trying to do
> something in the wrong moment...
>
> I could circumvent this issue using CDORevisionFactory.DEFAULT, time will tell if that is wise...
Please have a look at org.eclipse.emf.cdo.internal.server.Repository.doActivate():

LifecycleUtil.activate(store);
LifecycleUtil.activate(packageRegistry);
LifecycleUtil.activate(sessionManager);
LifecycleUtil.activate(revisionManager);

That explains why you can't use the revision manager (nor any other repository infra structure in Store.do(After)Activate.

If you really want to load resources eagerly I recommend to hook into *the first call* to this method:

org.eclipse.emf.cdo.server.IStoreAccessor.readResourceID(CDOID, String, CDOBranchPoint)

It gets called by

--> org.eclipse.emf.cdo.internal.server.Repository.readRootResource()
--> org.eclipse.emf.cdo.internal.server.Repository.doActivate()

in case of server restarts.

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Re: [CDO] Implement an alternative store [message #815262 is a reply to message #815175] Wed, 07 March 2012 07:38 Go to previous messageGo to next message
Victor Roldan Betancort is currently offline Victor Roldan Betancort
Messages: 524
Registered: July 2009
Senior Member
Federico Tomassetti escribió:
>> did you know EMF has a BinaryResource? It would help you achieve the
>> same results (serializing into a file), but without the hazzle of
>> implementing your own serialization ;)
>
>> org.eclipse.emf.ecore.resource.impl.BinaryResourceImpl
>
> I am going for JSON: it seems to be more interoperable.

Fair enough ;)

>
>> IRepository repository = store.getRepository();
>> CDORevisionFactory factory =
>> ((InternalCDORevisionManager)repository.getRevisionManager()).getFactory();
>>
>
> the revision manager returns a null when I ask for the factory. This
> seems to me a warning that I am trying to do something in the wrong
> moment...
>
> I could circumvent this issue using CDORevisionFactory.DEFAULT, time
> will tell if that is wise...
>
>> Are you planning to publish your implementation? Have you thought
>> about contributing to CDO? It would be easier to collaborate if we had
>> it somewhere like a git repository (yours or cdo's). Just let me know.
>
> Definitely I would share my implementation. The only problem is that I
> am working some hours per week on this project for a company and at this
> stage we are just evaluating the possibility to implement this store, so
> I can not guarantee I will work on that in the future.

Well, as long as you put it open source, if it raises interest, there
will sure be maintenance of the code somehow ;)
Re: [CDO] Implement an alternative store [message #932037 is a reply to message #809937] Wed, 03 October 2012 15:20 Go to previous messageGo to next message
Jim Foscue is currently offline Jim Foscue
Messages: 14
Registered: June 2012
Junior Member
Have you had any success with your subversion CDO store? if so I'd like to get some of your ideas. I'm looking into create a similar CDO store for git.

Thanks,

jim
Re: [CDO] Implement an alternative store [message #932506 is a reply to message #932037] Thu, 04 October 2012 02:51 Go to previous message
Federico Tomassetti is currently offline Federico Tomassetti
Messages: 186
Registered: July 2009
Location: Torino
Senior Member

Hi Jim!
Let's say I am almost there. Actually I think the plugins could share most of the code: the hard-work is to store and load nodes as files, the actual commit/checkout commands can be added later.

If you are interested we could put our efforts together.

Federico


Previous Topic:Adapter Factories in standalone mode.
Next Topic:Sort actions in menu
Goto Forum:
  


Current Time: Sat Aug 23 16:03:05 EDT 2014

Powered by FUDForum. Page generated in 0.01970 seconds