Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » Epsilon » Calling ETL from Java with "partial" reset(Trying to merge multiple instances of one meta-model in a single instance of a mother without reloading the latter.)
Calling ETL from Java with "partial" reset [message #1281870] Tue, 01 April 2014 13:39 Go to next message
Philip Makedonski is currently offline Philip MakedonskiFriend
Messages: 10
Registered: May 2013
Junior Member
Hi,

I am trying to merge multiple instances of one meta-model in a single instance of a mother without reloading the latter. Let me try to elaborate on this:

I have a model instance of meta-model A and multiple instances of meta-model B. What I want to do is execute the same ETL module on all instances of B and the single instance of A. Typically, a self-contained execution is not a problem:

* 0. load ETL,
* 1. load A,
* 2. load Bi,
* 3. execute ETL,
* 4. reset (dispose of A, Bi, ETL),
* 5. repeat (go to 0.)

But since with time A grows, loading and disposing A for each B comprises a major impact on performance. Thus, I've been trying to modify the workflow at least for a set of B instances (B1 ... Bn), where it becomes:

* 0. load ETL,
* 1. load A,
* 2. load Bi,
* 3. execute ETL,
* 4. dispose of Bi,
* 5. repeat (go to 2),
* 6. dispose of A, ETL,

Now I'm not quite sure whether this is possible at all, looking at the implementations for IModule.reset() does not really help so far, but maybe I am looking at the wrong place. Any advice will be appreciated. Here is some sample code illustrating what I am after:


		//initialise and load ETL source
		module = createModule();
		module.parse(getFile(getSource()));

		//load merged model (A)
		module.getContext().getModelRepository().addModel(getDecentModel(location));
		
		//iterate over models to be merged  (Bi)
		for (int i : list) {
			//load Bi instance 
			IModel famixModel = getFamixModel(location,i);
			module.getContext().getModelRepository().addModel(famixModel);

			//execute ETL
			preProcess();
			result = execute(module);
			postProcess();
			
			//remove the merged instance Bi - neither			
			//module.getContext().getModelRepository().getModelByName("FAMIX").dispose();
			//nor 
			//module.getContext().getModelRepository().removeModel(famixModel);
			//seems to work as the ETL transformations are not executed after the first iteration
			//perhaps I am missing reset the module without disposing of the source or at least of the merged target A
			
			//default approach which works but has potential performance implications
			module.reset();
			//reload
			module.parse(getFile(getSource()));
			module.getContext().getModelRepository().addModel(getDecentModel(location));
		}
		module.getContext().getModelRepository().dispose();


Re: Calling ETL from Java with "partial" reset [message #1282080 is a reply to message #1281870] Tue, 01 April 2014 21:04 Go to previous messageGo to next message
Dimitris Kolovos is currently offline Dimitris KolovosFriend
Messages: 2154
Registered: July 2009
Location: York, UK
Senior Member

Hi,

Does the following do the trick?

IModel a = ...;
a.load();

for every Bi {
EtlModule module = new EtlModule();
module.parse(...);
IModel b = ...;
b.load();
module.getContext().getModelRepository().addModel(a);
module.getContext().getModelRepository().addModel(b);
module.execute();
b.dispose();
}

a.dispose();

Cheers,
Dimitris
Re: Calling ETL from Java with "partial" reset [message #1282418 is a reply to message #1282080] Wed, 02 April 2014 09:18 Go to previous messageGo to next message
Philip Makedonski is currently offline Philip MakedonskiFriend
Messages: 10
Registered: May 2013
Junior Member
Yes, perfectly! Now that I look at it, it seems obvious that this is how it should be done, but it took me quite a while to figure out all the possible ways it shouldn't be done Wink Thanks a lot!

Cheers,

Philip
Re: Calling ETL from Java with "partial" reset [message #1282456 is a reply to message #1282418] Wed, 02 April 2014 10:30 Go to previous messageGo to next message
Dimitris Kolovos is currently offline Dimitris KolovosFriend
Messages: 2154
Registered: July 2009
Location: York, UK
Senior Member

Great! Glad this helped.

Cheers,
Dimitris
Re: Calling ETL from Java with "partial" reset [message #1357495 is a reply to message #1282456] Thu, 15 May 2014 14:52 Go to previous messageGo to next message
Jens Neuschulz is currently offline Jens NeuschulzFriend
Messages: 3
Registered: May 2014
Junior Member
Hi,

I have a similar problem. I'm trying to transform many source files of one metamodel into many target files of another one metamodel using one ETL. Creating a new ETL Module for every transformation takes a long time and decreases the performance. Using one ETL Module and reloading the source and the target models doesn't work. The content of the source file is changing before every load.
Here is my code:

module = createModule();
module.parse(getFile(getSource()));

for (int i = 0; i < 10000; i++) {
preProcess();
for (IModel model : getModels()) {
module.getContext().getModelRepository().addModel(model);
}
result = execute(module);
postProcess();
module.getContext().getModelRepository().dispose();
}

public List<IModel> getModels() throws Exception {
List<IModel> models = new ArrayList<IModel>();
models.add(createEmfModel("fin5412013", "models/MT541Instance.model", "models/fin5412013.ecore", true, false));
models.add(createEmfModel("sese02300105", "models/sese023Transformed.model", "models/sese02300105.ecore", false, true));
return models;
}

Any help?

Many thanks Jens

Re: Calling ETL from Java with "partial" reset [message #1358293 is a reply to message #1357495] Thu, 15 May 2014 22:52 Go to previous messageGo to next message
Dimitris Kolovos is currently offline Dimitris KolovosFriend
Messages: 2154
Registered: July 2009
Location: York, UK
Senior Member

Hi Jens,

Off the top of my head, you could remove the old models from the module's model repository and add fresh ones in each iteration. Please let me know if this is unclear and I'll provide some example code.

Cheers,
Dimitris
Re: Calling ETL from Java with "partial" reset [message #1359035 is a reply to message #1358293] Fri, 16 May 2014 06:26 Go to previous messageGo to next message
Jens Neuschulz is currently offline Jens NeuschulzFriend
Messages: 3
Registered: May 2014
Junior Member
Hi Dimitris,

could you provide an example please.

By the way could you provide an example for multi threading please.

Many thanks for your support Jens
Re: Calling ETL from Java with "partial" reset [message #1366277 is a reply to message #1359035] Mon, 19 May 2014 08:32 Go to previous messageGo to next message
Dimitris Kolovos is currently offline Dimitris KolovosFriend
Messages: 2154
Registered: July 2009
Location: York, UK
Senior Member

Hi Jens,

Sure - I'll try to do this within the next few days.

Cheers,
Dimitris
Re: Calling ETL from Java with "partial" reset [message #1386274 is a reply to message #1366277] Mon, 16 June 2014 10:11 Go to previous messageGo to next message
Jens Neuschulz is currently offline Jens NeuschulzFriend
Messages: 3
Registered: May 2014
Junior Member
Hi Dimitris,

any update on this?

Best regards Jens
Re: Calling ETL from Java with "partial" reset [message #1386639 is a reply to message #1386274] Wed, 18 June 2014 23:16 Go to previous messageGo to next message
Dimitris Kolovos is currently offline Dimitris KolovosFriend
Messages: 2154
Registered: July 2009
Location: York, UK
Senior Member

Hi Jens,

Apologies for the radio silence - this somehow fell through the cracks. I'll have a look at this within the next couple of days and get back to you.

Cheers,
Dimitris
Re: Calling ETL from Java with "partial" reset [message #1390487 is a reply to message #1386639] Sun, 29 June 2014 14:24 Go to previous messageGo to next message
Dimitris Kolovos is currently offline Dimitris KolovosFriend
Messages: 2154
Registered: July 2009
Location: York, UK
Senior Member

Hi Jens,

I've had a look at this and it appears that it is not safe to reuse / execute the same ETL module in different threads. I've filed an enhancement request for this: https://bugs.eclipse.org/bugs/show_bug.cgi?id=438490. Please feel free to CC yourself to the request to keep track of any progress on this.

Cheers,
Dimitris
Re: Calling ETL from Java with "partial" reset [message #1802937 is a reply to message #1390487] Tue, 19 February 2019 10:53 Go to previous messageGo to next message
Kelsey Brennan is currently offline Kelsey BrennanFriend
Messages: 7
Registered: February 2019
Junior Member
Hi Dimitris,

I know this is a fairly old thread but I seem to have run into the same problem in trying to reuse an EVL module. I'd like to run the validation multiple times without re-parsing the .evl file but it doesn't seem to work correctly after the first execution. Has there been a fix for this, and if so, in what version?

Thanks,
Kelsey
Re: Calling ETL from Java with "partial" reset [message #1802938 is a reply to message #1802937] Tue, 19 February 2019 11:03 Go to previous messageGo to next message
Dimitris Kolovos is currently offline Dimitris KolovosFriend
Messages: 2154
Registered: July 2009
Location: York, UK
Senior Member

Hi Kelsey,

EVL modules are not designed to be recyclable so I'm afraid that you'll need to re-parse every time.

Cheers,
Dimitris
Re: Calling ETL from Java with "partial" reset [message #1802995 is a reply to message #1802938] Wed, 20 February 2019 12:39 Go to previous messageGo to next message
Kelsey Brennan is currently offline Kelsey BrennanFriend
Messages: 7
Registered: February 2019
Junior Member
Hi Dimitris,

It's unfortunate that the EVL modules are not recyclable. My team is really interested in using EVL but we're finding that having to re-parse every time is too slow for our purposes. We might be interested in contributing to the library to make it reusable. Are there any major impediments that would make rewriting the module, or maybe just extracting the parsing step, unfeasible?

Thanks,
Kelsey
Re: Calling ETL from Java with "partial" reset [message #1803027 is a reply to message #1802995] Thu, 21 February 2019 08:58 Go to previous messageGo to next message
Dimitris Kolovos is currently offline Dimitris KolovosFriend
Messages: 2154
Registered: July 2009
Location: York, UK
Senior Member

Hi Kelsey,

I've had a closer look at the EVL code and while the intention was never to be able to reuse modules it appears that providing the module with a fresh context, as follows, should do the trick:

EvlModule module = new EvlModule();
module.parse(...);
...
module.execute(); // First execution

EvlContext context = new EvlContext();
module.setContext(context);
context.setModule(module);

module.execute(); // Second execution without re-parsing


Could you please give it a try and let me know if it works for you?

Cheers,
Dimitris
Re: Calling ETL from Java with "partial" reset [message #1803102 is a reply to message #1803027] Fri, 22 February 2019 12:42 Go to previous messageGo to next message
Kelsey Brennan is currently offline Kelsey BrennanFriend
Messages: 7
Registered: February 2019
Junior Member
Hi Dimitris,

Unfortunately that doesn't seem to work. The constraints are executed but it would appear that the model from the previous execution isn't removed properly, and the constraints are always executed on the original model instance.

Maybe I've misunderstood and it's only possible to rerun the validations on the same model instance. If that's the case, then my problem is not solved by this.

Here's the small example I've tried. My expected outcome is some logs like below, but what I'm actually seeing is "ID is 0" every time, and when I step through with a debugger it has a reference to the first book object I added to the model.

1 constraint(s) have not been satisfied
An book ID must be greater than 100 but ID is 0
1 constraint(s) have not been satisfied
An book ID must be greater than 100 but ID is 1
1 constraint(s) have not been satisfied
An book ID must be greater than 100 but ID is 2
1 constraint(s) have not been satisfied
An book ID must be greater than 100 but ID is 3
1 constraint(s) have not been satisfied
An book ID must be greater than 100 but ID is 4


Java:
Resource.Factory.Registry reg = Resource.Factory.Registry.INSTANCE;
        Map<String, Object> m = reg.getExtensionToFactoryMap();
        m.put("test", new XMIResourceFactoryImpl());

        String modelPath = "src/main/resources/metamodel/MyModel";

        EvlModule evlModule = new EvlModule();
        try {
            evlModule.parse(EVLRunner.class.getResource("/evl/Validate.evl").toURI());

            if (evlModule.getParseProblems().size() > 0) {
                System.err.println("Parse errors occured...");
                for (ParseProblem problem : evlModule.getParseProblems()) {
                    System.err.println(problem.toString());
                }
                return;
            }
        } catch (Exception e) {
            e.printStackTrace();
            return;
        }

        for (int i = 0; i < 5; i++) {
            // create a resource
            ResourceSet resSet = new ResourceSetImpl();
            Resource resource = resSet.createResource(URI.createURI(modelPath + i + ".test"));

            // Reset context
            EvlContext context = new EvlContext();
            evlModule.setContext(context);
            context.setModule(evlModule);

            // Add to model
            TestFactory factory = new TestFactoryImpl();
            Book book = factory.createBook();
            book.setBookId(i);
            resource.getContents().add(book);

            InMemoryEmfModel model = new InMemoryEmfModel(resource);
            evlModule.getContext().getModelRepository().addModel(model);

            try {
                evlModule.execute();
            } catch (EolRuntimeException e) {
                e.printStackTrace();
            }

            Collection<UnsatisfiedConstraint> unsatisfied = evlModule.getContext().getUnsatisfiedConstraints();

            if (unsatisfied.size() > 0) {
                System.out.println(unsatisfied.size() + " constraint(s) have not been satisfied");
                for (UnsatisfiedConstraint uc : unsatisfied) {
                    System.out.println(uc.getMessage());
                }
            } else {
                System.out.println("All constraints have been satisfied");
            }
        }
    }


EVL:
context Book {
    constraint ID {
        check : self.bookId > 100
        message : "An book ID must be greater than 100 but ID is " + self.bookId
    }
}
Re: Calling ETL from Java with "partial" reset [message #1803140 is a reply to message #1803102] Sat, 23 February 2019 10:58 Go to previous messageGo to next message
Dimitris Kolovos is currently offline Dimitris KolovosFriend
Messages: 2154
Registered: July 2009
Location: York, UK
Senior Member

Hi Kelsey,

Could you please share a minimal demo project [1] that includes all relevant models and metamodels that I can run and reproduce this locally?

Cheers,
Dimitris

[1] https://www.eclipse.org/epsilon/doc/articles/minimal-examples/
Re: Calling ETL from Java with "partial" reset [message #1803145 is a reply to message #1803140] Sat, 23 February 2019 14:16 Go to previous messageGo to next message
Sina MadaniFriend
Messages: 160
Registered: November 2015
Location: York, UK
Senior Member
Hi Kelsey,

Possibly related / root cause: https://bugs.eclipse.org/bugs/show_bug.cgi?id=538175.

Out of interest, is it the actual parsing which takes too long or the execution? Such a small script should take milliseconds to parse. I suspect the execution of the script is what takes most of the time, not the parsing. In which case you may consider using parallel EVL (replace EvlModule with EvlModuleParallelElements) to improve performance.

Thanks,
Sina
Re: Calling ETL from Java with "partial" reset [message #1803211 is a reply to message #1803145] Mon, 25 February 2019 11:05 Go to previous messageGo to next message
Kelsey Brennan is currently offline Kelsey BrennanFriend
Messages: 7
Registered: February 2019
Junior Member
Sina-

Yes, that bug sounds like it's the problem I'm running into. It actually is the parsing that's taking too long, our application needs to be able to process incoming data within a few milliseconds. Running the parsing every time made our average processing time about 10x longer than it was before so if we could just do it once during the application startup that would work much better for us.

Dimitris-

Minimal example here: https://github.com/kelseybrennan/evl-reuse-test

The main class is EVLRunner.java. Build with Maven and then it shouldn't need anything special to run the main method and see that it is validating the original model every time.



Re: Calling ETL from Java with "partial" reset [message #1803221 is a reply to message #1803211] Mon, 25 February 2019 12:32 Go to previous messageGo to next message
Dimitris Kolovos is currently offline Dimitris KolovosFriend
Messages: 2154
Registered: July 2009
Location: York, UK
Senior Member

Hi Kelsey,

Replacing your EvlModule evlModule = new EvlModule() line with the following should do the trick

EvlModule evlModule = new EvlModule() {
	
	@Override
	public ModuleElement adapt(AST cst, ModuleElement parentAst) {
		if (cst.getType() == EvlParser.CONTEXT) {
			return new ConstraintContext() {
				protected EolModelElementType getType(IEvlContext context) throws EolModelNotFoundException, EolModelElementTypeNotFoundException {
					type = new EolModelElementType(getTypeName(), context);
					return type;
				};
			};
		}
		else return super.adapt(cst, parentAst);
	}
};


Line 177 of the original code of ConstraintContext's getType() [1] caches "type", which causes the issue you're observing as correctly pointed out by Sina.

Cheers,
Dimitris

[1] https://git.eclipse.org/c/epsilon/org.eclipse.epsilon.git/tree/plugins/org.eclipse.epsilon.evl.engine/src/org/eclipse/epsilon/evl/dom/ConstraintContext.java?h=1.5#n177

[Updated on: Mon, 25 February 2019 12:41]

Report message to a moderator

Re: Calling ETL from Java with "partial" reset [message #1803238 is a reply to message #1803221] Mon, 25 February 2019 14:29 Go to previous message
Kelsey Brennan is currently offline Kelsey BrennanFriend
Messages: 7
Registered: February 2019
Junior Member
Hi Dimitris,

Great, that fix works! Thank you very much for all your help with this.

Previous Topic:[EVL] update in-memory model modified programmatically
Next Topic:ECL in the standalone mode
Goto Forum:
  


Current Time: Thu Mar 28 16:40:36 GMT 2024

Powered by FUDForum. Page generated in 0.06756 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top