Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF "Technology" (Ecore Tools, EMFatic, etc)  » [EMFClient] Advice on merging 3 ex CDO exported models
[EMFClient] Advice on merging 3 ex CDO exported models [message #1434133] Mon, 29 September 2014 18:16 Go to next message
David Wynter is currently offline David WynterFriend
Messages: 4624
Registered: July 2009
Senior Member
Hi,

We have serious problem with CDO, it corrupts the DBStore at random,
test cases pass. Eike thinks this might be a multi threading bug. But
given the support for this complex bug is impossible to get now we have
no choice other than to use EMFStore.

We modified CDO libraries to ignore the corrupted parts of the model
instance and have managed to export our 3 CDO repositories to xmi files.
Can someone explain how we might load them into EMFStore and merge these
model instances, the actual model is consistent and they have a common
parent model instance.

My thoughts were to somehow (howw do we load xmi into EMFStore?) the
common parent model instance into EMFStore. Then import each of our
derived model instance from xmi file to a branch each then merge those
branches. There are collectively several hundred hours of work in these
models.

Thx

David
Re: [EMFClient] Advice on merging 3 ex CDO exported models [message #1434522 is a reply to message #1434133] Tue, 30 September 2014 07:43 Go to previous messageGo to next message
Jonas Helming is currently offline Jonas HelmingFriend
Messages: 699
Registered: July 2009
Senior Member
Hi David,

what exactly does "merging" mean? Are those three models different
versions of the same model? Or do they just need to be combined into one
model instance.
Is the multi threading issue within your application?

Best regards

Jonas


Am 29.09.2014 20:16, schrieb David Wynter:
> Hi,
>
> We have serious problem with CDO, it corrupts the DBStore at random,
> test cases pass. Eike thinks this might be a multi threading bug. But
> given the support for this complex bug is impossible to get now we have
> no choice other than to use EMFStore.
>
> We modified CDO libraries to ignore the corrupted parts of the model
> instance and have managed to export our 3 CDO repositories to xmi files.
> Can someone explain how we might load them into EMFStore and merge these
> model instances, the actual model is consistent and they have a common
> parent model instance.
>
> My thoughts were to somehow (howw do we load xmi into EMFStore?) the
> common parent model instance into EMFStore. Then import each of our
> derived model instance from xmi file to a branch each then merge those
> branches. There are collectively several hundred hours of work in these
> models.
>
> Thx
>
> David
Re: [EMFClient] Advice on merging 3 ex CDO exported models [message #1434537 is a reply to message #1434522] Tue, 30 September 2014 08:06 Go to previous messageGo to next message
David Wynter is currently offline David WynterFriend
Messages: 4624
Registered: July 2009
Senior Member
Hi,

The 3 models are the same, the 3 model instances ( i.e. model data) all derive from a single parent model instance. They should not have any overlapping model objects across these 3 model instances.

We want to merge these 3 model instances to end up with one model instance containing all objects from all 3 source model instances.

The issue with CDO is after a variable amount of time it causes a corruption, it does not seem to be associated with our code, because it is at random which model class is being persisted at the time of the corruption, we do not use multi threading, we use our GUI design tool 1 to 1 to a single DbStore on H2. We are not aware of the corruption at the time we save the model, i.e. it saves. There are other times it fails to save, but that does not seem to cause an issue, it recovers.

Eike asked us to provide a test case, but they pass in general. Although a test class that copies the model will fail consistently once a repo is corrupted. Eike no longer has time to investigate, even when offered paid consulting, he has access to the corrupted repos on a google drive, but no time to look at them. Commits on CDO have really fallen off since the end of May. At this stage I think CDO is no longer tenable as a project, I have a raised the issues in the EMF forum, they remain unresolved. Not being able to provide a test case that shows the process that causes corruption means I cannot raise a bug. So we are trapped, with hundreds of hours of modelling work in corrupted CDO repos. We are just trying to recover the situation. EMFStore is our destination. As a startup and being so dependent on CDO this has been a massive problem.
Re: [EMFClient] Advice on merging 3 ex CDO exported models [message #1434584 is a reply to message #1434537] Tue, 30 September 2014 09:37 Go to previous messageGo to next message
Jonas Helming is currently offline Jonas HelmingFriend
Messages: 699
Registered: July 2009
Senior Member
Hi David,

how were those three instances divided? If there are no common,
overlapping model elements, how should they be combined? Is there a
common containment structure, elements should be put in? Depending on
your case, I think it might be fairly simple to programmatically combine
the models, especially if there is no overlapping.

The merging algorithm of EMF Store works change-based. In your case, you
are comparing states (without having the history available), so it will
not solve your case.

Best regards

Jonas

Am 30.09.2014 10:06, schrieb David Wynter:
> Hi,
>
> The 3 models are the same, the 3 model instances ( i.e. model data) all
> derive from a single parent model instance. They should not have any
> overlapping model objects across these 3 model instances.
> We want to merge these 3 model instances to end up with one model
> instance containing all objects from all 3 source model instances.
>
> The issue with CDO is after a variable amount of time it causes a
> corruption, it does not seem to be associated with our code, because it
> is at random which model class is being persisted at the time of the
> corruption, we do not use multi threading, we use our GUI design tool 1
> to 1 to a single DbStore on H2. We are not aware of the corruption at
> the time we save the model, i.e. it saves. There are other times it
> fails to save, but that does not seem to cause an issue, it recovers.
> Eike asked us to provide a test case, but they pass in general. Although
> a test class that copies the model will fail consistently once a repo is
> corrupted. Eike no longer has time to investigate, even when offered
> paid consulting, he has access to the corrupted repos on a google drive,
> but no time to look at them. Commits on CDO have really fallen off since
> the end of May. At this stage I think CDO is no longer tenable as a
> project, I have a raised the issues in the EMF forum, they remain
> unresolved. Not being able to provide a test case that shows the process
> that causes corruption means I cannot raise a bug. So we are trapped,
> with hundreds of hours of modelling work in corrupted CDO repos. We are
> just trying to recover the situation. EMFStore is our destination. As a
> startup and being so dependent on CDO this has been a massive problem.
Re: [EMFClient] Advice on merging 3 ex CDO exported models [message #1434803 is a reply to message #1434584] Tue, 30 September 2014 16:19 Go to previous messageGo to next message
David Wynter is currently offline David WynterFriend
Messages: 4624
Registered: July 2009
Senior Member
Hi,

To answer each in turn.
"how were those three instances divided? If there are no common,
overlapping model elements, how should they be combined?" As I stated there is a common parent, since branching there are no common overlapping model elements,
but of course all 3 have all the same original model elements.

I now have the makeithappen EMFClient running with our ecore model registered. I try to Import a project, delighted to see it is xmi format. But it runs for about 10 minutes on each of our 3 model instances in xmi and then fails without any console message and no error message.

I found the log, now attached.

Console output
Configuration location:
    file:/Users/david/Documents/workspace/.metadata/.plugins/org.eclipse.pde.core/ECP View and Domain Model Editor/
Configuration file:
    file:/Users/david/Documents/workspace/.metadata/.plugins/org.eclipse.pde.core/ECP View and Domain Model Editor/config.ini loaded
Install location:
    file:/Users/david/eclipse/
Framework located:
    file:/Users/david/eclipse/plugins/org.eclipse.osgi_3.9.1.v20140110-1610.jar
Framework classpath:
    file:/Users/david/eclipse/plugins/org.eclipse.osgi_3.9.1.v20140110-1610.jar
Splash location:
    null
Debug options:
    file:/Users/david/eclipse/Eclipse.app/Contents/MacOS/.options not found
Time to load bundles: 1335
Starting application: 5640
UIProviderDescriptor: org.eclipse.emf.ecp.emfstore.ui [27] --> file:/Users/david/eclipse/Eclipse.app/Contents/MacOS/plugins/org.eclipse.emf.ecp.emfstore.ui_1.4.0.20140916-1322.jar
UIProviderDescriptor: org.eclipse.emf.ecp.workspace.ui [97] --> file:/Users/david/eclipse/Eclipse.app/Contents/MacOS/plugins/org.eclipse.emf.ecp.workspace.ui_1.4.0.20140916-1322.jar
Loading ECPProjectManagerImpl from /Users/david/Documents/runtime-ECPViewModelAndDomainEditor/.metadata/.plugins/org.eclipse.emf.ecp.core/projects
ProviderDescriptor: org.eclipse.emf.ecp.emfstore.core [130] --> file:/Users/david/eclipse/Eclipse.app/Contents/MacOS/plugins/org.eclipse.emf.ecp.emfstore.core_1.4.0.20140916-1322.jar
ProviderDescriptor: org.eclipse.emf.ecp.workspace.core [181] --> file:/Users/david/eclipse/Eclipse.app/Contents/MacOS/plugins/org.eclipse.emf.ecp.workspace.core_1.4.0.20140916-1322.jar
Loading ECPRepositoryManagerImpl from /Users/david/Documents/runtime-ECPViewModelAndDomainEditor/.metadata/.plugins/org.eclipse.emf.ecp.core/repositories
RepositoryDescriptor: org.eclipse.emf.ecp.emfstore.ui [27] --> file:/Users/david/eclipse/Eclipse.app/Contents/MacOS/plugins/org.eclipse.emf.ecp.emfstore.ui_1.4.0.20140916-1322.jar
Application Started: 10614

...

No location provider or error while instantiating location provider, switching to default location!
Property file read. (/Users/david/.emfstore/server/profiles/demoClient/./conf/es.properties)
Server data home location: /Users/david/.emfstore/server/profiles/demoClient/
JVM Max Memory: 1,037 MByte
Couldn't load password file from path: /Users/david/.emfstore/server/profiles/demoClient/./conf/user.properties
Using WebServerProvider class org.eclipse.emf.emfstore.internal.server.connection.xmlrpc.XmlRpcBuiltinWebServer at port 8080
Started XML RPC Webserver on port: 8080
Notifying post startup listener
Initialitation COMPLETE.
Server is RUNNING...Time to relax...
Resource save options initialized:
	USE_ENCODED_ATTRIBUTE_STYLE: true
	USE_CACHED_LOOKUP_TABLE: []
	ENCODING: UTF-8
	FLUSH_THRESHOLD: 100000
	USE_FILE_BUFFER: true


  • Attachment: log
    (Size: 373.63KB, Downloaded 91 times)

[Updated on: Tue, 30 September 2014 16:30]

Report message to a moderator

Re: [EMFClient] Advice on merging 3 ex CDO exported models [message #1434826 is a reply to message #1434803] Tue, 30 September 2014 16:59 Go to previous messageGo to next message
David Wynter is currently offline David WynterFriend
Messages: 4624
Registered: July 2009
Senior Member
I thought I corrected the model uri and awaited for the model to complete loading.

But still get this

!ENTRY org.eclipse.ui 4 0 2014-09-30 18:07:03.374
!MESSAGE Unhandled event loop exception
!STACK 0
org.eclipse.e4.core.di.InjectionException: org.eclipse.emf.common.util.WrappedException: java.lang.ClassNotFoundException: com.yambina.edm.edmmodel.modelPackage cannot be found by com.eclipsesource.makeithappen.model_0.1.0.qualifier
	at org.eclipse.e4.core.internal.di.MethodRequestor.execute(MethodRequestor.java:63)



This is the correct package name and the package is in the com.eclipsesource.makeithappen.model project?

[Updated on: Tue, 30 September 2014 17:14]

Report message to a moderator

Re: [EMFClient] Advice on merging 3 ex CDO exported models [message #1435236 is a reply to message #1434826] Wed, 01 October 2014 07:42 Go to previous messageGo to next message
Jonas Helming is currently offline Jonas HelmingFriend
Messages: 699
Registered: July 2009
Senior Member
Did it work now? How big are your models?

Am 30.09.2014 18:59, schrieb David Wynter:
> I have corrected the model uri and am now awaiting the model to complete
> loading.
Re: [EMFClient] Advice on merging 3 ex CDO exported models [message #1435243 is a reply to message #1435236] Wed, 01 October 2014 07:52 Go to previous messageGo to next message
David Wynter is currently offline David WynterFriend
Messages: 4624
Registered: July 2009
Senior Member
Hi,

No it did not work I get the InjectException I showed above. The xmi files are al between 6 and 9MB in size. Thousands of model elements. Will read more to see if I can find out why it does not pickup the com.yambina.edm.edmmodel.modelPackage.

[Updated on: Wed, 01 October 2014 08:51]

Report message to a moderator

Re: [EMFClient] Advice on merging 3 ex CDO exported models [message #1435362 is a reply to message #1435243] Wed, 01 October 2014 11:40 Go to previous messageGo to next message
David Wynter is currently offline David WynterFriend
Messages: 4624
Registered: July 2009
Senior Member
OK, Took me a while to modify the xmi files and work out all the uri and package names and align them all. But they are aligned now but now get this:
!ENTRY org.eclipse.emf.ecp.emfstore.ui 4 0 2014-10-01 12:32:56.072
!MESSAGE Package with uri 'http://com/yambina/edmmodel' not found. (file:/Users/david/Documents/workspace/com.eclipsesource.makeithappen.model/SavedModels/repo_full_dseq2.xmi, 3, 59)
!STACK 0
org.eclipse.emf.ecore.resource.Resource$IOWrappedException: Package with uri 'http://com/yambina/edmmodel' not found. (file:/Users/david/Documents/workspace/com.eclipsesource.makeithappen.model/SavedModels/repo_full_dseq2.xmi, 3, 59)



First 2 lines of repo_full_dseq2.xmi looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<xmi:XMI xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:edmmodel="http://com/yambina/edmmodel">


Extension Element Details in the config have
uri: http://com/yambina/edmmodel
class: com.yambina.edmmodel.ModelPackage
genmodel: model/edmmodel.genmodel

Since these seem to line up what could it be?

Re: [EMFClient] Advice on merging 3 ex CDO exported models [message #1435434 is a reply to message #1435362] Wed, 01 October 2014 13:34 Go to previous messageGo to next message
David Wynter is currently offline David WynterFriend
Messages: 4624
Registered: July 2009
Senior Member
I missed a step, adding the model and model.edit plugins in the Debug Configuration, but they are not listed to be selected in the list of plugins. Trying to find out why not now.
Re: [EMFClient] Advice on merging 3 ex CDO exported models [message #1435968 is a reply to message #1435434] Thu, 02 October 2014 07:50 Go to previous messageGo to next message
Jonas Helming is currently offline Jonas HelmingFriend
Messages: 699
Registered: July 2009
Senior Member
Are them in your workspace or in the target platform?

Am 01.10.2014 15:34, schrieb David Wynter:
> I missed a step, adding the model and model.edit plugins in the Debug
> Configuration, but they are not listed to be selected in the list of
> plugins. Trying to find out why not now.
Re: [EMFClient] Advice on merging 3 ex CDO exported models [message #1436022 is a reply to message #1435434] Thu, 02 October 2014 09:14 Go to previous messageGo to next message
David Wynter is currently offline David WynterFriend
Messages: 4624
Registered: July 2009
Senior Member
Hi,

I have spent all day yesterday trying to get EMFClient to import our xmi files.
I started from scratch late yesterday checking out makeithappen and simply adding our existing ecore model to it. But now I get this:
!ENTRY org.eclipse.emf.ecp.emfstore.ui 4 0 2014-10-02 09:59:35.535
!MESSAGE File is corrupt, does not contain a org.eclipse.emf.emfstore.internal.client.model.ProjectSpace.
!STACK 0
java.io.IOException: File is corrupt, does not contain a org.eclipse.emf.emfstore.internal.client.model.ProjectSpace.
	at org.eclipse.emf.emfstore.internal.client.model.util.ResourceHelper.getElementFromResource(ResourceHelper.java:69)
	at org.eclipse.emf.emfstore.internal.client.model.util.ResourceHelper.getElementFromResource(ResourceHelper.java:93)
	at org.eclipse.emf.emfstore.internal.client.model.impl.WorkspaceBase.importProjectSpace(WorkspaceBase.java:245)
	at org.eclipse.emf.emfstore.internal.client.importexport.impl.ImportProjectSpaceController.execute(ImportProjectSpaceController.java:80)
	at org.eclipse.emf.emfstore.internal.client.importexport.ExportImportControllerExecutor$1.doRun(ExportImportControllerExecutor.java:64)
	at org.eclipse.emf.emfstore.internal.client.model.util.EMFStoreCommand.commandBody(EMFStoreCommand.java:51)
	at org.eclipse.emf.emfstore.internal.client.model.util.AbstractEMFStoreCommand.execute(AbstractEMFStoreCommand.java:64)
	at org.eclipse.emf.common.command.BasicCommandStack.execute(BasicCommandStack.java:78)
	at org.eclipse.emf.emfstore.internal.client.model.changeTracking.commands.EMFStoreBasicCommandStack.execute(EMFStoreBasicCommandStack.java:72)
	at org.eclipse.emf.emfstore.internal.client.model.util.AbstractEMFStoreCommand.aRun(AbstractEMFStoreCommand.java:108)
	at org.eclipse.emf.emfstore.internal.client.model.util.EMFStoreCommand.run(EMFStoreCommand.java:60)
	at org.eclipse.emf.emfstore.internal.client.importexport.ExportImportControllerExecutor.execute(ExportImportControllerExecutor.java:73)
	at org.eclipse.emf.ecp.emfstore.internal.ui.handler.ImportProjectHelper.importProject(ImportProjectHelper.java:81)
	at org.eclipse.emf.ecp.emfstore.internal.ui.handler.ImportProjectHandler.execute(ImportProjectHandler.java:35)


Any ideas on why this might occur?

btw I noticed that on OSX that the first time the import dialog opens it does not display the file in the directory. I have to close it and exercise the "Import Project" a second time before it works correctly.


Re: [EMFClient] Advice on merging 3 ex CDO exported models [message #1436059 is a reply to message #1436022] Thu, 02 October 2014 10:26 Go to previous messageGo to next message
David Wynter is currently offline David WynterFriend
Messages: 4624
Registered: July 2009
Senior Member
Ok, I traced. now I get it. Import Project is not for general ecore models that have been exported to xmi, it is some sort of specific model that contains an outer element ProjectSpace. Line 68 of org.eclipse.emf.emfstore.internal.client.model.util.ResourceHelper tells me this.

So how do you import ecore models that are in xmi format to EMFClient?

Also interested in resources explaining how to merge instance of the same model with a common parent dataset.



[Updated on: Thu, 02 October 2014 10:27]

Report message to a moderator

Re: [EMFClient] Advice on merging 3 ex CDO exported models [message #1436751 is a reply to message #1436059] Fri, 03 October 2014 11:02 Go to previous messageGo to next message
David Wynter is currently offline David WynterFriend
Messages: 4624
Registered: July 2009
Senior Member
Proving to be difficult to merge these instances of the same model. EMF Compare fails with
An internal error occurred during: "Compute Model Differences".
loader constraint violation: when resolving method "org.eclipse.emf.compare.scope.FilterComparisonScope.setResourceSetContentFilter(Lcom/google/common/base/Predicate;)V" the class loader (instance of org/eclipse/osgi/internal/baseadaptor/DefaultClassLoader) of the current class, org/eclipse/emf/compare/ide/ui/internal/logical/EMFSynchronizationModel, and the class loader (instance of org/eclipse/osgi/internal/baseadaptor/DefaultClassLoader) for resolved class, org/eclipse/emf/compare/scope/FilterComparisonScope, have different Class objects for the type ope.setResourceSetContentFilter(Lcom/google/common/base/Predicate;)V used in the signature

[Updated on: Fri, 03 October 2014 11:03]

Report message to a moderator

Re: [EMFClient] Advice on merging 3 ex CDO exported models [message #1438032 is a reply to message #1436059] Sun, 05 October 2014 11:12 Go to previous messageGo to next message
Jonas Helming is currently offline Jonas HelmingFriend
Messages: 699
Registered: July 2009
Senior Member
Hi,

did you try the option "import child"?

Best regards

Jonas

Am 02.10.2014 12:26, schrieb David Wynter:
> Ok, I traced. now I get it. Import Project is not for general ecore
> models that have been exported to xmi, it is some sort of specific model
> that contains an outer element ProjectSpace. Line 68 of ResourceHelper
> tells me this.
> So how do you import ecore models that are in xmi format to EMFClient?
>
> Also interested in resources explaining how to merge instance of the
> same model with a common parent dataset.
>
>
>
>
Re: [EMFClient] Advice on merging 3 ex CDO exported models [message #1438772 is a reply to message #1438032] Mon, 06 October 2014 13:19 Go to previous messageGo to next message
David Wynter is currently offline David WynterFriend
Messages: 4624
Registered: July 2009
Senior Member
Hi,

Tried that, it works to a point. It looks like you need a single element at root level under the
<xmi:XMI xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:edmmodel="http://www.yambina.com/edmmodel">
top element. We have lots of peer elements at the same level in the hierarchy under xmi:XMI and it only imports the first of these into the Project.

I added a top level element <parent></parent> under the xmi:XMI element that contains all my model elements as children. But this does not seem to load anything. Is there documentation on what EMFClient expects in the xmi file for it to successfully load an xmi file?

I have a concern on trying to merge these models, ecore uses element offsets to identify related elements e.g.

<?xml version="1.0" encoding="UTF-8"?>
<xmi:XMI xmi:version="2.0" xmlns:xmi="http://www.omg.org/XMI" xmlns:edmmodel="http://www.yambina.com/edmmodel">
<parent>
  <edmmodel:Product Name="Bloomberg Per Security Product">
    <uuid lsb="-8801872204936836175" msb="1090523321334386058"/>
    <workflow product="/0" etlGroups="/575 /579 /736 /740 /868 /892" localName="BBG Bond Loading" valid="true">
      <uuid lsb="-5346265735007129715" msb="-7710296907219514541"/>
    </workflow>
...


You can see etlGroups contained within workflow here are expressed as offsets. The other models which were developed from the same parent will have different elements at these offsets. Will the Merge facility be able to accommodate these different offsets?

[Updated on: Mon, 06 October 2014 15:33]

Report message to a moderator

Re: [EMFClient] Advice on merging 3 ex CDO exported models [message #1438814 is a reply to message #1438772] Mon, 06 October 2014 14:37 Go to previous message
Jonas Helming is currently offline Jonas HelmingFriend
Messages: 699
Registered: July 2009
Senior Member
Hi,

yes it exspects exactly one root element. If you look at the method
ECPImportHandlerHelper#importFile (bundle: org.eclipse.emf.ecp.ui)
you see, how it processes the file. It would be easy to adapt this to
multiple root elements.

Best regards

Jonas

Am 06.10.2014 15:19, schrieb David Wynter:
> Hi,
>
> Tried that, it works to a point. It looks like you need a single element
> at root level under the <xmi:XMI xmi:version="2.0"
> xmlns:xmi="http://www.omg.org/XMI"
> xmlns:edmmodel="http://www.yambina.com/edmmodel">
> top element. We have lots of peer elements at the same level in the
> hierarchy under xmi:XMI and it only imports the first of these into the
> Project.
>
> I added a top level element <parent></parent> under the xmi:XMI element
> that contains all my model elements as children. But this does not seem
> to load anything. Is there documentation on what EMFClient expects in th
> exmi file for it to successfully load an xmi file?
>
>
Previous Topic:Drag & Drop not working in ECP - Model Explorer View
Next Topic:[EMF Forms] EMF Forms and GMF
Goto Forum:
  


Current Time: Sat Mar 28 21:51:02 GMT 2020

Powered by FUDForum. Page generated in 0.02218 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top