Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » Epsilon » ETL execution engine
ETL execution engine [message #1765197] Wed, 07 June 2017 12:44 Go to next message
Banafsheh A is currently offline Banafsheh AFriend
Messages: 262
Registered: July 2016
Senior Member
Hi,

I want to know about execution engine of ETL. Is there any reference to help in this regard?

e.g., In the below code when 'Element' class is created? When we execute 'Statemachine2Petrinet' rule and arrive to 'pn.elements ::= sm.elements' statement or when we execute `ModelElem2Element` rule?

rule Statemachine2Petrinet
	transform sm: Statemachine!Statemachine
 	to pn : Petrinet!Petrinet {
 		pn.elements ::= sm.elements;
}

rule ModelElem2Element
	transform mElem : Statemachine!ModelElem
	to elem : Petrinet!Element {
		guard : mElem.name <> null
		elem.name := mElem.name;
}

[Updated on: Wed, 07 June 2017 12:45]

Report message to a moderator

Re: ETL execution engine [message #1765200 is a reply to message #1765197] Wed, 07 June 2017 13:16 Go to previous messageGo to next message
Bugra M. Yildiz is currently offline Bugra M. YildizFriend
Messages: 105
Registered: July 2014
Senior Member
Hello,

1. This is the declarative side of ETL. The engine automatically schedules the execution of the rules. You are not supposed to know, or better saying, you do/should not assume any ordering of the execution of the rules. This is decided by the engine.

2. A question to be asked is, I also do not know the answer, is the following a valid ETL statement:

pn.elements ::= sm.elements;


Does it trigger the execution of ModelElem2Element rule for each element in pn's list? Or should you write a for loop to iterate each list element individually?

3. If the ModelElem2Element rule is executed for every element in the list before the Statemachine2Petrinet rule, then the engine uses the result of this execution in the code part

pn.elements ::= sm.elements;


, rather than executing the rule again. Otherwise (if the ModelElem2Element rule is not executed yet), then the ModelElem2Element rule is called when needed.
Re: ETL execution engine [message #1765209 is a reply to message #1765200] Wed, 07 June 2017 14:24 Go to previous messageGo to next message
Dimitris Kolovos is currently offline Dimitris KolovosFriend
Messages: 1498
Registered: July 2009
Location: York, UK
Senior Member

Hi both,

In the absence of lazy rules, ETL runs in 2 passes. In the first pass it creates all target objects and traceability links back to their equivalent source objects and in the second pass, it populates the contents of the target objects by running the bodies of the respective rules.

> Does it trigger the execution of ModelElem2Element rule for each element in pn's list? Or should you write a for loop to iterate each list element individually?

Multi-valued features are handled using a built-in loop so you wouldn't need to write one.

Cheers,
Dimitris
Re: ETL execution engine [message #1765252 is a reply to message #1765209] Wed, 07 June 2017 20:56 Go to previous messageGo to next message
Banafsheh A is currently offline Banafsheh AFriend
Messages: 262
Registered: July 2016
Senior Member
Thank you very much indeed. It's really helpful.
The traceability links that you mentioned are the same as the post part of OO2DB transformation [1] ?
In OO2DB transformation why do we need to create the traceability links?

[1]. https://www.eclipse.org/epsilon/examples/index.php?example=org.eclipse.epsilon.examples.oo2db
Re: ETL execution engine [message #1765253 is a reply to message #1765252] Wed, 07 June 2017 20:59 Go to previous messageGo to next message
Dimitris Kolovos is currently offline Dimitris KolovosFriend
Messages: 1498
Registered: July 2009
Location: York, UK
Senior Member

No - I was referring to the internal traceability links that the ETL engine maintains to be able to compute .equivalent[s](). The trace model in OO2DB is there only to demonstrate that an ETL transformation is not limited to one source and one target model but I can now see how it can be more confusing than helpful :/

Cheers,
Dimitris
Re: ETL execution engine [message #1765259 is a reply to message #1765253] Wed, 07 June 2017 21:06 Go to previous messageGo to next message
Banafsheh A is currently offline Banafsheh AFriend
Messages: 262
Registered: July 2016
Senior Member
Ok. Thank you.
In the aspect of functionality, I mean; do they the same work?

Is there any reference that help to know more about ETL execution engine?
Re: ETL execution engine [message #1765261 is a reply to message #1765259] Wed, 07 June 2017 21:08 Go to previous messageGo to next message
Dimitris Kolovos is currently offline Dimitris KolovosFriend
Messages: 1498
Registered: July 2009
Location: York, UK
Senior Member

> In the aspect of functionality, I mean; do they the same work?

No. As far as the execution engine is concerned, the Trace model has no special semantics.

> Is there any reference that help to know more about ETL execution engine?

Not really but please feel free to keep asking questions. Perhaps at the end of this discussion you can even put together a relevant article for the Epsilon website ;)

Cheers,
Dimitris
Re: ETL execution engine [message #1766455 is a reply to message #1765261] Wed, 21 June 2017 15:11 Go to previous messageGo to next message
Banafsheh A is currently offline Banafsheh AFriend
Messages: 262
Registered: July 2016
Senior Member
Thanks for your kind help.

1. How the engine distinguish between two classes with the same name?
e.g., In these two rules 4 different Column classes should be created and have their own features:
rule Class2Table
  transform c : OO!Class 
  to t : DB!Table, pk : DB!Column {
  
   if (c.`extends`.isDefined()){
   
    var childFkCol : new DB!Column;
    var parentFkCol : DB!Column;

  }

}

rule SingleValuedAttribute2Column
  transform a : OO!Attribute
  to c : DB!Column {
....
}

EClass [name=Column, instanceClassName=null, instanceClass=null, defaultValue=null, instanceTypeName=null, abstract=false, interface=false, ] cannot specifies the difference of these classes.
2. When we have an assignment statement that its lhs is a class, what does it do?
parentFkCol = parentTable.primaryKeys.first();

[Updated on: Thu, 22 June 2017 15:35]

Report message to a moderator

Re: ETL execution engine [message #1766532 is a reply to message #1766455] Thu, 22 June 2017 17:20 Go to previous messageGo to next message
Dimitris Kolovos is currently offline Dimitris KolovosFriend
Messages: 1498
Registered: July 2009
Location: York, UK
Senior Member

Hi,

I'm not sure I understand what you're asking. Could you please elaborate a bit more?

Cheers,
Dimitris
Re: ETL execution engine [message #1766534 is a reply to message #1766532] Thu, 22 June 2017 18:28 Go to previous messageGo to next message
Banafsheh A is currently offline Banafsheh AFriend
Messages: 262
Registered: July 2016
Senior Member
1. Does the execution engine create 4 columns (EClass) for above code?

2. It is another question that is not related to the first question.
In the below assignment statement what happens?
parentFkCol = parentTable.primaryKeys.first();

[Updated on: Fri, 23 June 2017 08:56]

Report message to a moderator

Re: ETL execution engine [message #1766593 is a reply to message #1766534] Fri, 23 June 2017 11:20 Go to previous messageGo to next message
Dimitris Kolovos is currently offline Dimitris KolovosFriend
Messages: 1498
Registered: July 2009
Location: York, UK
Senior Member

> 1. Does the execution engine create 4 columns (EClass) for above code?

It creates 4 instances of Column.

> In the below assignment statement what happens? parentFkCol = parentTable.primaryKeys.first();

The value of the parentFkCol variable becomes the result of evaluating "parentTable.primaryKeys.first()"

Cheers,
Dimitris
Re: ETL execution engine [message #1766594 is a reply to message #1766593] Fri, 23 June 2017 11:38 Go to previous messageGo to next message
Banafsheh A is currently offline Banafsheh AFriend
Messages: 262
Registered: July 2016
Senior Member
Dimitris Kolovos wrote on Fri, 23 June 2017 11:20

It creates 4 instances of Column.

That means, we have 4 instances of EClass with the same name in the same package. Since they have not id, how can we distinguish between them?
e.g. I want to add an attribute to EClass that is created with "var parentFkCol : DB!Column;". How can I select this specific class?

Dimitris Kolovos wrote on Fri, 23 June 2017 11:20
>
The value of the parentFkCol variable becomes the result of evaluating "parentTable.primaryKeys.first()"

Since parentFkCol is a class (not feature of a class), what happens in the model?
The class should rename and all its features must remove or not happens in the model? if not happens, why?

[Updated on: Fri, 23 June 2017 11:50]

Report message to a moderator

Re: ETL execution engine [message #1766596 is a reply to message #1766594] Fri, 23 June 2017 11:50 Go to previous messageGo to next message
Dimitris Kolovos is currently offline Dimitris KolovosFriend
Messages: 1498
Registered: July 2009
Location: York, UK
Senior Member

It does not create 4 EClasses - it creates 4 *instances* of the Column EClass. As such, you cannot add "attributes" to parentFkCol as it's a Column, not an EClass.

Cheers,
Dimitris
Re: ETL execution engine [message #1766597 is a reply to message #1766596] Fri, 23 June 2017 11:51 Go to previous messageGo to next message
Banafsheh A is currently offline Banafsheh AFriend
Messages: 262
Registered: July 2016
Senior Member
Ok, How can we distinguish between these instances?
Re: ETL execution engine [message #1766598 is a reply to message #1766597] Fri, 23 June 2017 11:58 Go to previous messageGo to next message
Dimitris Kolovos is currently offline Dimitris KolovosFriend
Messages: 1498
Registered: July 2009
Location: York, UK
Senior Member

You can 1) access it while the parentFkCol variable is still available, 2) reach it through a navigation path or 3) assign one of its properties and then use that property to look it up.

Cheers,
Dimitris
Re: ETL execution engine [message #1766602 is a reply to message #1766598] Fri, 23 June 2017 12:05 Go to previous messageGo to next message
Banafsheh A is currently offline Banafsheh AFriend
Messages: 262
Registered: July 2016
Senior Member
Thank you very much.
Dimitris Kolovos wrote on Fri, 23 June 2017 11:20

The value of the parentFkCol variable becomes the result of evaluating "parentTable.primaryKeys.first()"

Since parentFkCol is a class (not feature of a class), what happens in the model?
The instance of the class should rename and all its properties must remove or not happens in the model? if not happens, why?

[Updated on: Fri, 23 June 2017 12:07]

Report message to a moderator

Re: ETL execution engine [message #1766604 is a reply to message #1766602] Fri, 23 June 2017 12:19 Go to previous messageGo to next message
Dimitris Kolovos is currently offline Dimitris KolovosFriend
Messages: 1498
Registered: July 2009
Location: York, UK
Senior Member

Nothing happens to the model. It's just a local variable assignment.

Cheers,
Dimitris
Re: ETL execution engine [message #1767133 is a reply to message #1766604] Sat, 01 July 2017 17:37 Go to previous messageGo to next message
Banafsheh A is currently offline Banafsheh AFriend
Messages: 262
Registered: July 2016
Senior Member
Hi,

Thanks a lot.
I have some more questions:
1.
Dimitris Kolovos wrote on Wed, 07 June 2017 14:24

In the absence of lazy rules, ETL runs in 2 passes. In the first pass it creates all target objects and traceability links back to their equivalent source objects and in the second pass, it populates the contents of the target objects by running the bodies of the respective rules.

when we have lazy rules what changes occur?


2. In the "SpecialAssignmentStatements" or "AssignmentStatements" which have equivalent(s) in their rhs, the engine finds TransformationRule which its source has the special type and selects its target. I want to know:
2.1. If the TransformationRule has more than one target, does the engine select all of its target?
2.2. If we have more than one TrasformationRule that matches the condition, do all of them select?

3. In the "parentTable.primaryKeys.first()" expression, how the execution engine find out to which class "primaryKeys" refers?
My own perception is it finds the eType of primarykeys reference. By the way, I have no idea about the metamodel file it belongs to.

Many thanks for your time,

[Updated on: Sat, 01 July 2017 17:45]

Report message to a moderator

Re: ETL execution engine [message #1767192 is a reply to message #1767133] Mon, 03 July 2017 11:52 Go to previous message
Dimitris Kolovos is currently offline Dimitris KolovosFriend
Messages: 1498
Registered: July 2009
Location: York, UK
Senior Member

Hi,

> when we have lazy rules what changes occur?

In the presence of lazy rules, rules are fully executed (element creation + population) in one go when a .equivalent() is encountered.

2.1 and 2.2 are easy to figure out through experimentation.

Regarding 3, EOL uses reflection on the "parentTable" EObject to retrieve the values of its "primaryKeys" reference.

Cheers,
Dimitris
Previous Topic:ETL and ATL
Next Topic:jdbc emc problem
Goto Forum:
  


Current Time: Sun Jan 21 13:00:45 GMT 2018

Powered by FUDForum. Page generated in 0.02317 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software