Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Archived » M2M (model-to-model transformation) » [QVTO] Any way to speed up transformation?(Transformation takes too much time, unacceptable for user experience)
[QVTO] Any way to speed up transformation? [message #736985] Fri, 14 October 2011 10:41 Go to next message
Vitaly Savickas is currently offline Vitaly Savickas
Messages: 61
Registered: March 2010
Member
Hi,

I am using QVTO transformation to generate something out of a diagram in a custom editor. The QVTO syntax and semantics suit great for the purpose, but the transformation takes too much time for some reason. Even for the simplest diagram with a few elements. My (only a) guess is that it happens because I have a lot of scripts and they have many rules. I don't know how QVTO engine works, but I assume it compiles them at each run time, so that can be a slowdown. Is there a way to speed it up, maybe by precompiling the scripts once and for all as they don't change anyway?

Regards,
Vitaly
Re: [QVTO] Any way to speed up transformation? [message #737023 is a reply to message #736985] Fri, 14 October 2011 11:28 Go to previous messageGo to next message
Ed Willink is currently offline Ed Willink
Messages: 4010
Registered: July 2009
Senior Member
Hi

I'm not an expert on the way QVTo works, but have observed some of its
operation.

Unless you have some strange outer loop, I would certainly expect that
all the parsing and analysis would occur before execution starts, or
just possibly lazily on the first encounter. Observing when and what
Resources are created can be very instructive.

The QVTo engine extends the OCL evaluator and so gives an interpreted
execution. I don't think that there is a great deal of
compilation/optimisation as such.

The almost declarative nature of QVTo makes it quite easy to
accidentally do things repeatedly; it is worth understanding what is
happening and perhaps caching some results in let variables or Dicts. I
find the resolve capability very confusing and usually indaequate and
always maintain my own in2out Dict. This might help you. If you have
many rules, you may be performing long is-it-A, is-it-B, is-it-C match
searches. You may be able to structure these to help the naive
interpreted execution.

The QVTo engine maintains an EMF resource internally, so in principle it
should be possible to serialize this after parsing and analysis and
reuse it later. However beware that such a serialisation is necessarily
a proprietary format since the OMG specification is incomplete and the
related Eclipse tooling is Ecore rather than EMOF-based.

[The OCL tooling will support generation of Java in the Juno release
giving one to two orders of magnitude speed improvement, however it is
not clear when or if QVTo will be able to exploit this.]

Regards

Ed Willink


On 14/10/2011 15:41, Vitaly Savickas wrote:
> Hi,
>
> I am using QVTO transformation to generate something out of a diagram
> in a custom editor. The QVTO syntax and semantics suit great for the
> purpose, but the transformation takes too much time for some reason.
> Even for the simplest diagram with a few elements. My (only a) guess
> is that it happens because I have a lot of scripts and they have many
> rules. I don't know how QVTO engine works, but I assume it compiles
> them at each run time, so that can be a slowdown. Is there a way to
> speed it up, maybe by precompiling the scripts once and for all as
> they don't change anyway?
>
> Regards,
> Vitaly
Re: [QVTO] Any way to speed up transformation? [message #737116 is a reply to message #737023] Fri, 14 October 2011 13:20 Go to previous messageGo to next message
Vitaly Savickas is currently offline Vitaly Savickas
Messages: 61
Registered: March 2010
Member
Thank you for a detailed response, Ed!

I know I lack experience in QVTO and the way I have written the rules may be very inefficient. This is another aspect:) But what I have noticed by experimenting with it is that if let's say I use only one library script (imported) in my main script the transformation is 3 times faster than if I have all my libraries included. Even if the actual rules that are called are only few. This means to me that script files get read and parsed every time I call the transformation. I was wondering if this process can be done once as scripts don't change anyway. Now I am wondering if I run the transformation correctly:) I am using TransformationExecutor which takes a URI of a script file, and I got this approach from a QVTO forum suggestion about running transformation programmatically.

Regards,
Vitaly
Re: [QVTO] Any way to speed up transformation? [message #737126 is a reply to message #737116] Fri, 14 October 2011 13:34 Go to previous messageGo to next message
Ed Willink is currently offline Ed Willink
Messages: 4010
Registered: July 2009
Senior Member
Hi Vitaly

I would be surprised if there are miltiple parses. A breakpoint on
XMLResourceImpl copnstruction should soon settle the issue.

What is more plausible, but disappointing, is if the matching loop
search time increases linearly (or worse quadratically) with the number
of candidate rules. I would expect a sub-linear increase since most of
the rules should be dismissable by a simple type check that could be
optimised.

I think you'll have to start instrumenting to see what costs and what
you can avoid.

Regards

Ed Willink



On 14/10/2011 18:20, Vitaly Savickas wrote:
> Thank you for a detailed response, Ed!
>
> I know I lack experience in QVTO and the way I have written the rules
> may be very inefficient. This is another aspect:) But what I have
> noticed by experimenting with it is that if let's say I use only one
> library script (imported) in my main script the transformation is 3
> times faster than if I have all my libraries included. Even if the
> actual rules that are called are only few. This means to me that
> script files get read and parsed every time I call the transformation.
> I was wondering if this process can be done once as scripts don't
> change anyway. Now I am wondering if I run the transformation
> correctly:) I am using TransformationExecutor which takes a URI of a
> script file, and I got this approach from a QVTO forum suggestion
> about running transformation programmatically.
>
> Regards,
> Vitaly
Re: [QVTO] Any way to speed up transformation? [message #737265 is a reply to message #737126] Fri, 14 October 2011 17:15 Go to previous messageGo to next message
Vitaly Savickas is currently offline Vitaly Savickas
Messages: 61
Registered: March 2010
Member
Hi Ed,

I am sorry for probably a silly question - I didn't get why you mentioned a resource. Do you mean somehow watching for a read of a script file? I believe it is read every time I execute the script, as I programmatically create the TransformationExecutor and pass a script URI to it, then execute the executor and save the result.

Shall I be more precise about my observation, what I meant is that after twiddling with the main script file I have commented most of the rule calls and include statements of libraries. Transformation ran quite fast, obviously:) Now when I uncommented includes so that the whole script to be read became huge, the speed decreased significantly. I don't see a reason in this case to have an increase in matching loop search time - I include the libraries, but don't use them:) Basically I have around 8 or so script files - one with with only a main rule and rest are libraries that are included and used in main. All rules are in libraries.

So I guess there is no way to prepare the script once and cache or save it in this state for later executions so to speed them up? Very disappointing:(

Regards,
Vitaly
Re: [QVTO] Any way to speed up transformation? [message #737687 is a reply to message #737265] Sat, 15 October 2011 05:58 Go to previous messageGo to next message
Ed Willink is currently offline Ed Willink
Messages: 4010
Registered: July 2009
Senior Member
Hi

I initially wrote "EMF resource" then I wrote "XMIResourceImpl" which is
one of the classes that should not be constructed very often; once for
each file/model loaded. You may find about 20 Resources get created for
detailed XMI and OCL requirements, but no Resource should get loaded
more than twice, preferably not more than once.

Other classes that shouldn't be constructed often are ResourceSetImpl
(at most five) and OCLAnalyzer (possibly one per parse, ideally reused).

Instrumenting the major actions of these with just a System.println of
System.currentMilliseconds may provide a more useful indication of where
the time goes than the vast detail that VisualVM can provide.

Regards

Ed Willink


On 14/10/2011 22:15, Vitaly Savickas wrote:
> Hi Ed,
>
> I am sorry for probably a silly question - I didn't get why you
> mentioned a resource. Do you mean somehow watching for a read of a
> script file? I believe it is read every time I execute the script, as
> I programmatically create the TransformationExecutor and pass a script
> URI to it, then execute the executor and save the result.
>
> Shall I be more precise about my observation, what I meant is that
> after twiddling with the main script file I have commented most of the
> rule calls and include statements of libraries. Transformation ran
> quite fast, obviously:) Now when I uncommented includes so that the
> whole script to be read became huge, the speed decreased
> significantly. I don't see a reason in this case to have an increase
> in matching loop search time - I include the libraries, but don't use
> them:) Basically I have around 8 or so script files - one with with
> only a main rule and rest are libraries that are included and used in
> main. All rules are in libraries.
>
> So I guess there is no way to prepare the script once and cache or
> save it in this state for later executions so to speed them up? Very
> disappointing:(
>
> Regards,
> Vitaly
Re: [QVTO] Any way to speed up transformation? [message #737712 is a reply to message #737687] Sat, 15 October 2011 06:43 Go to previous messageGo to next message
Vitaly Savickas is currently offline Vitaly Savickas
Messages: 61
Registered: March 2010
Member
Ed, thank you very much for being patient to my stupid questions:)

I try to do your suggested approach. Before that, if I understand you correctly at the execution there are many intermediate resources created and loaded by the QVTO framework. In that case I guess one potential speed up solution could be if those resources got persisted in my plug-in and every time I execute the translation they could be reused instead of created again, right? Do you think it is possible and if you have any hint on how to implement that?

Regards,
Vitaly
Re: [QVTO] Any way to speed up transformation? [message #737755 is a reply to message #737712] Sat, 15 October 2011 07:53 Go to previous messageGo to next message
Ed Willink is currently offline Ed Willink
Messages: 4010
Registered: July 2009
Senior Member
Hi

You seem to be speculating about solutions beyond your knowledge.

In the absence of a solution from someone else

you must determine what is happening
determine the costs of those activities
determine what might be improved
estimate the costs of those improvements
compromise the costs to achieve the most effective solution

You are very unlikely to come to a good solution by hypothesizing rather
than measuring.

Regards

Ed Willink


On 15/10/2011 11:43, Vitaly Savickas wrote:
> Ed, thank you very much for being patient to my stupid questions:)
>
> I try to do your suggested approach. Before that, if I understand you
> correctly at the execution there are many intermediate resources
> created and loaded by the QVTO framework. In that case I guess one
> potential speed up solution could be if those resources got persisted
> in my plug-in and every time I execute the translation they could be
> reused instead of created again, right? Do you think it is possible
> and if you have any hint on how to implement that?
>
> Regards,
> Vitaly
Re: [QVTO] Any way to speed up transformation? [message #737774 is a reply to message #737755] Sat, 15 October 2011 08:30 Go to previous messageGo to next message
Vitaly Savickas is currently offline Vitaly Savickas
Messages: 61
Registered: March 2010
Member
Sorry, Ed:) I fully understand your critique and agree with that. But I don't see a lot of power in the hands of a developer in this case, to be honest. The only thing I can control is to rewrite/restructure the script files. That's why I initially asked if I have control over the transformation workflow & can I optimise it by any other means than just script file content itself. It seems quite obvious from my experiments that what is happening is the bigger the size of the script files, the worse is performance, independing of the actual rules used/called - this isn't good. And what I can do about it - I guess nothing, as I use all the rules in the final product. But this doesn't mean the performance is bad because of the big number of rules, but because they are read every time on execution. And this is where I have no control to improve something. But I would be very happy if I am wrong:) That's why I asked about optimisation in the first place.

Regards,
Vitaly
Re: [QVTO] Any way to speed up transformation? [message #737781 is a reply to message #737774] Sat, 15 October 2011 08:39 Go to previous message
Ed Willink is currently offline Ed Willink
Messages: 4010
Registered: July 2009
Senior Member
Hi

QVTo is Open Source Software. You can run it under the debugger, set
breakpoints on interesting events, add println()'s for diagnostics.

Regards

Ed Willink

On 15/10/2011 13:30, Vitaly Savickas wrote:
> Sorry, Ed:) I fully understand your critique and agree with that. But
> I don't see a lot of power in the hands of a developer in this case,
> to be honest. The only thing I can control is to rewrite/restructure
> the script files. That's why I initially asked if I have control over
> the transformation workflow & can I optimise it by any other means
> than just script file content itself. It seems quite obvious from my
> experiments that what is happening is the bigger the size of the
> script files, the worse is performance, independing of the actual
> rules used/called - this isn't good. And what I can do about it - I
> guess nothing, as I use all the rules in the final product. But this
> doesn't mean the performance is bad because of the big number of
> rules, but because they are read every time on execution. And this is
> where I have no control to improve something. But I would be very
> happy if I am wrong:) That's why I asked about optimisation in the
> first place.
>
> Regards,
> Vitaly
Previous Topic:how generate string from a Sequence of Tuple
Next Topic:labrary of rules in ATL
Goto Forum:
  


Current Time: Tue Jul 29 19:11:26 EDT 2014

Powered by FUDForum. Page generated in 0.04659 seconds