Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF "Technology" (Ecore Tools, EMFatic, etc)  » [Edapt] is bidirectional migration possible/planned/feasible?
[Edapt] is bidirectional migration possible/planned/feasible? [message #1717091] Wed, 09 December 2015 21:12 Go to next message
Paul Reuter is currently offline Paul ReuterFriend
Messages: 6
Registered: December 2015
Junior Member
Hello,

I am looking to replace our somewhat shoddy migration code with a more robust framework, and spent some time to evaluate Edapt.

One requirement is the ability to export older versions of the data model, since there is old hardware that won't accept newer versions.

From the Operations I inspected, they are clearly one-way only, unless I am missing something.

Since the migration process follows the metamodel changes, would the architecture be able to handle some kind of reverse migration? (Presuming that corresponding operations are specified, or existing operations extended with a reverse operation)

Kind regards,
Paul
Re: [Edapt] is bidirectional migration possible/planned/feasible? [message #1717318 is a reply to message #1717091] Fri, 11 December 2015 16:30 Go to previous messageGo to next message
Johannes Faltermeier is currently offline Johannes FaltermeierFriend
Messages: 97
Registered: December 2013
Member

Hi,

operations/custom migrations in Edapt are not reversable.
Also it is not possible to create perfect reverse operations for all operations, e.g. for the "Delete Feature" operation.
Once we removed the value for the deleted feature from the model instance in the normal migration process, it is gone. When we make the reverse migration later, we have lost this value.

But taking these issues aside (since we could agree that reverse operations are best-effort), from an architectural point-of-view adding reverse operations/custom migrations should be possible.

However I think that a better approach could be to generate some kind of reverse-history.
The generator could replace operations with reverse operations where applicable. Also it would allow users to attach custom-migrations along the way. This approach would give more control to users.

Regards
Johannes


Johannes Faltermeier

Get professional Eclipse developer support:
http://eclipsesource.com/en/services/developer-support/
Re: [Edapt] is bidirectional migration possible/planned/feasible? [message #1717576 is a reply to message #1717318] Tue, 15 December 2015 01:33 Go to previous messageGo to next message
Paul Reuter is currently offline Paul ReuterFriend
Messages: 6
Registered: December 2015
Junior Member
Many thanks for your answer, Johannes.

I have spent some time following the debugger through a migration process, and have a much clearer picture of what is happening.

I do have a couple of new questions, though. (sorry!)

On a conceptual level, I am not sure about the difference between a Change executed in the EMF editor and recorded in the history, and an operation that does the same thing ( delete a class, for example ).

Both seem to implement their behaviour with a call to metamodel.delete(eClass), but one executes from the EcoreForwardReconstructor, and one from the MigrationReconstructor.

Is this a matter of preference when to use which one?



The second concerns the API. I can use either migrateAndSave or migrateAndLoad on a Resource level. Unfortunately, most of our migration is done on export, i.e. starting from EObjects in memory or on parts of the model only.

I can see why there is this limitation. As far as I understand, the model to migrate needs to conform to a reconstructed source metamodel that will be destructively migrated alongside the model to a different release.

However, in between loading and saving there seems to be a conversion from DynamicEObjectImpl to InstanceImpl, which appears to be a different Implementation of EObject that also provides access to all instances of an EType. The new InstanceImpl replace references to model EObjects with references to the corresponding InstanceImpl, but keep a reference to the metamodel.

It seems not too difficult to provide an alternate creation mechanism for such an Instance model that takes a model from a different ( possibly generated ) EPackage, but replaces the metamodel references with references to the reconstructed metamodel. With such a hook we could probably cover most of our requirements.

This would require some opening up on the API side, though. Is this something that could be considered?

If not, I suppose it would be possible to save/load the resources everytime a migration needs to be done, but I am concerned over memory ( and perm space ) usage due to all the model duplication. Our metamodel is huge, and model xmi files can grow to several hundred MB.



A third question is about the release ordering. In the release menu there is a Label to Match field. Does this mean that releases do not need to be chronologically ordered, or even part of the same sequence? Is it possible to have some versions branch off/specialise within the same history file?



Finally, I stumbled accross a class called EcoreBackwardReconstructor which is unused, but seems to be a good starting point for reverse history. Is there something similar for the MigrationReconstructor?



I have some ideas how to implement a reverse migration scheme on operation or reconstructor level, but I have no clue how and where to start looking at a reverse history generation. Do you have a few hints on good starting points?

Kind Regards,
Paul
Re: [Edapt] is bidirectional migration possible/planned/feasible? [message #1717773 is a reply to message #1717576] Wed, 16 December 2015 15:56 Go to previous messageGo to next message
Johannes Faltermeier is currently offline Johannes FaltermeierFriend
Messages: 97
Registered: December 2013
Member

Hi Paul,

please find my comment below:

Paul Reuter wrote on Tue, 15 December 2015 01:33

I have spent some time following the debugger through a migration process, and have a much clearer picture of what is happening.

I do have a couple of new questions, though. (sorry!)

On a conceptual level, I am not sure about the difference between a Change executed in the EMF editor and recorded in the history, and an operation that does the same thing ( delete a class, for example ).

Both seem to implement their behaviour with a call to metamodel.delete(eClass), but one executes from the EcoreForwardReconstructor, and one from the MigrationReconstructor.

Is this a matter of preference when to use which one?


A change recorded in the history is simply the information about how the Ecore was changed.
Using the list of changes in the history we can recreate any state the Ecore-model was in (in both directions).
We use this to recreate an in-memory Ecore, which is used to load the old XMI-model-file.
After the old model was loaded, we can start the migration process.
Here we use the changes again to step through the changes and accordingly change our in-memory ecore.
The operations (and custom migrations, if any) are used to migrate the loaded model-instance, so that it conforms to the current metamodel.

So on the conceptual level operations are a list of regular changes (just like the recorded changes = metamodel migration information) + model migration information.

Paul Reuter wrote on Tue, 15 December 2015 01:33

The second concerns the API. I can use either migrateAndSave or migrateAndLoad on a Resource level. Unfortunately, most of our migration is done on export, i.e. starting from EObjects in memory or on parts of the model only.

I can see why there is this limitation. As far as I understand, the model to migrate needs to conform to a reconstructed source metamodel that will be destructively migrated alongside the model to a different release.

However, in between loading and saving there seems to be a conversion from DynamicEObjectImpl to InstanceImpl, which appears to be a different Implementation of EObject that also provides access to all instances of an EType. The new InstanceImpl replace references to model EObjects with references to the corresponding InstanceImpl, but keep a reference to the metamodel.

It seems not too difficult to provide an alternate creation mechanism for such an Instance model that takes a model from a different ( possibly generated ) EPackage, but replaces the metamodel references with references to the reconstructed metamodel. With such a hook we could probably cover most of our requirements.

This would require some opening up on the API side, though. Is this something that could be considered?

If not, I suppose it would be possible to save/load the resources everytime a migration needs to be done, but I am concerned over memory ( and perm space ) usage due to all the model duplication. Our metamodel is huge, and model xmi files can grow to several hundred MB.


The main use case of Edapt is to be able to load models which have been created with an older version of the Ecore. So at the start, there is always a loading process + the migration to the given metamodel state. These requirements fit the current migrateAndSave or migrateAndLoad methods.
Since backward migration is not supported yet, there is no API starting point for an already loaded model.
Opening up on the API shouldn't be a blocker. Also the Migrator class already offers a lot of customization probabilities, e.g. providing custom operations/libraries. So we could maybe align this extension with existing ones.

Paul Reuter wrote on Tue, 15 December 2015 01:33

A third question is about the release ordering. In the release menu there is a Label to Match field. Does this mean that releases do not need to be chronologically ordered, or even part of the same sequence? Is it possible to have some versions branch off/specialise within the same history file?


The Label to match is simply the part of the current NS-URI which will be replaced with given text when a new release is created.

Paul Reuter wrote on Tue, 15 December 2015 01:33

Finally, I stumbled accross a class called EcoreBackwardReconstructor which is unused, but seems to be a good starting point for reverse history. Is there something similar for the MigrationReconstructor?


The EcoreBackwardReconstructor is used by the Reconstruction View. When you open this view and select any child of your history, it will show you the Ecore-state at this point of time.
The reconstructor, which should be the fastest one to create the ecore state is used.
Afaik there is no such thing for Migration changes.

Paul Reuter wrote on Tue, 15 December 2015 01:33

I have some ideas how to implement a reverse migration scheme on operation or reconstructor level, but I have no clue how and where to start looking at a reverse history generation. Do you have a few hints on good starting points?


I would start with a simple history file which only contains operations, for which it is kinda easy to identify a reverse operation.

Delete Feature <-> Create Attribute / Create Reference
(this is interesting because there may be different reverse operation)

and

Rename <-> Rename
(Interesting because same operation, but different order)

Then I would start to write some code which will read in this history and starts to translate this to a reverse-history file.
I would test the generated reverse history like a regular migration at first (so keeping the generated projects for all the releases).
When the generated history is fine, I would check what needs to be changed in the Migrator class to make the backward migration work.

This should give us a feeling for how the operation<->reverse-operation mapping must look like and how it can be implemented and if the whole approach is feasible at all.

We would be glad to accept a contribution with this feature and to review your implementation. Also we as a company offer sponsored open-source development for Edapt. So if you would like to get help with the implementation, you may of course contact us Wink

Best regards
Johannes


Johannes Faltermeier

Get professional Eclipse developer support:
http://eclipsesource.com/en/services/developer-support/
Re: [Edapt] is bidirectional migration possible/planned/feasible? [message #1718244 is a reply to message #1717773] Mon, 21 December 2015 14:51 Go to previous message
Paul Reuter is currently offline Paul ReuterFriend
Messages: 6
Registered: December 2015
Junior Member
Thank you for your support, Johannes!

For now, I will use the change recording in Edapt to build a complete history of our data model, and use that to bullet-proof the current migration code. I'll also take note of any required features that might be problematic.

With that I'll try to prototype a reverse migration toy example and get back to you.

Kind regards
Paul

Previous Topic:[EDAPT] Use Edapt with binary resources
Next Topic:[TEXO] Is it possible to use Texo without DB?
Goto Forum:
  


Current Time: Wed Aug 05 17:13:41 GMT 2020

Powered by FUDForum. Page generated in 0.01450 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top