Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » M2T (model-to-text transformation) » [Acceleo] Performance Profiling
[Acceleo] Performance Profiling [message #847652] Tue, 17 April 2012 10:21 Go to next message
Bart Theelen is currently offline Bart Theelen
Messages: 20
Registered: February 2012
Junior Member
We have created a M2T transformation in Acceleo which is fairly simpel and straightforward but has unacceptable scalability issues. Are there any means to profile the performance of Acceleo code? Has anybody experienced performance issues with Acceleo before?
Re: [Acceleo] Performance Profiling [message #847656 is a reply to message #847652] Tue, 17 April 2012 10:26 Go to previous messageGo to next message
Stephane Begaudeau is currently offline Stephane Begaudeau
Messages: 406
Registered: April 2010
Location: Nantes (France)
Senior Member

Hi,

Which version of Acceleo are you using? Can you post here the code of your transformation? Some operations are browsing all the model to find some result so they may be time consuming with large models. You can also launch your transformation with the profiler activated to see the bottleneck of your generation. Finally, what is the size of your model? How many elements are there inside?

Regards,

Stephane Begaudeau, Obeo

--
Twitter: @sbegaudeau & @acceleo
Google+: stephane.begaudeau & acceleo
Blog: http://stephanebegaudeau.tumblr.com
Acceleo Documentation: http://docs.obeonetwork.com/acceleo



Re: [Acceleo] Performance Profiling [message #848901 is a reply to message #847656] Wed, 18 April 2012 13:35 Go to previous messageGo to next message
Bart Theelen is currently offline Bart Theelen
Messages: 20
Registered: February 2012
Junior Member
I am using Acceleo version 3.2.1.

It is rather difficult to post anonymized code without a lot of effort. But I can give a few statistics on the size of some instances of the model that needs transforming. The transformation requires information from 5 related ecore models and consists of about 90 lines of Acceleo code. There are several loops which go through a large number of objects. This cannot be avoided as all objects need to be translated to a textual representation. For our killer case, there are 8000 objects of a class T. Moreover, there are 129 objects of a class SG, which together encapsulate about 14000 objects of a class B. There are furthermore about 100000 objects of a class C, which are encapsulated in either the one root object or one of the 129 SG objects. Finally, there are an (estimated) 250000 objects of a class P. All of this needs to be translated to text where all objects need to be visited at least once and the P objects twice (this can't be avoided to produce the correct text). The tree-editor does not have too much difficulty with opening this killer case and browsing through it.

The size of the example above makes the transformation take 2.5 hours, while our current implementation in C++ basically does the same transformation to text in about 1 second (including all file operations). The Acceleo code is nearly a 100% re-syntaxed verion of the original C++ code, with one exception for objects of class C. This requires some scoping which is realised with a select on an OrderedSet to find those C objects that have a reference to a specific P object within the same scope. To me the difference between 2.5h versus 1s is unexplanable for the given similarity in code. So, any help is welcome...

What exactly do you mean with 'launch with THE profiler activated'? Which profiler? What does it profile? We would like to see some statistics on how much time is spent on the execution of each of the different templates in our Acceleo code.

[Updated on: Wed, 18 April 2012 13:38]

Report message to a moderator

Re: [Acceleo] Performance Profiling [message #849478 is a reply to message #848901] Thu, 19 April 2012 03:27 Go to previous messageGo to next message
Laurent Goubet is currently offline Laurent Goubet
Messages: 1609
Registered: July 2009
Senior Member
Hi Bart,

The Acceleo profiler aims at providing such statistics : http://wiki.eclipse.org/Acceleo/Profiler . 2.5 hours for a generation clearly is strange at best, especially considering the small size of your templates. Could you provide us with these templates and a sample model on which we could try and profile Acceleo itself to see if we can make things better?

Laurent Goubet
Obeo
Re: [Acceleo] Performance Profiling [message #849550 is a reply to message #849478] Thu, 19 April 2012 04:41 Go to previous messageGo to next message
Ed Willink is currently offline Ed Willink
Messages: 3844
Registered: July 2009
Senior Member
Hi Laurent

An idea;

Eclipse OCL supports evaluation of individual queries and has no caching
to improve reuse of partial results. (The Impact Analyzer provides
caches for re-evaluation.)

Avoiding re-evaluation is a user/consumer responsibility, so a user can
use let-variables.

There is a pending patch for QVTo from JPL that gives massive
improvements in compilation time by caching all the meta-model lookups
that are otherwise repeated needlessly. (The OCL to Java generator
creates a flat meta-model representation that could be enabled to give
similar compilation time benefits with the new pivot model.)

Acceleo provides query evaluation caches to ensure that queries are not
re-evaluated needlessly.

The user problem seems to be small templates and large data, so I wonder
whether the query caches are malfunctioning, or whether uncached derived
features are in use that could benefit from use of cached Acceleo queries.

Regards

Ed Willink


On 19/04/2012 08:27, Laurent Goubet wrote:
> Hi Bart,
>
> The Acceleo profiler aims at providing such statistics :
> http://wiki.eclipse.org/Acceleo/Profiler . 2.5 hours for a generation
> clearly is strange at best, especially considering the small size of
> your templates. Could you provide us with these templates and a sample
> model on which we could try and profile Acceleo itself to see if we
> can make things better?
>
> Laurent Goubet
> Obeo
Re: [Acceleo] Performance Profiling [message #849684 is a reply to message #849550] Thu, 19 April 2012 07:17 Go to previous messageGo to next message
Bart Theelen is currently offline Bart Theelen
Messages: 20
Registered: February 2012
Junior Member
Thanks for the responses. I will have a look at the profiling tool.

Unfortunately, I cannot give a concrete example, nor the models and actual transformations as they are confidential...

There are no queries at all in our Acceleo code (well there is one, but that is called only once to identify the file name for storing the output). There is exaclty one let-variable in the 90 lines of Acceleo code (in the template for class C).
Re: [Acceleo] Performance Profiling [message #849730 is a reply to message #849550] Thu, 19 April 2012 08:01 Go to previous messageGo to next message
Laurent Goubet is currently offline Laurent Goubet
Messages: 1609
Registered: July 2009
Senior Member
Bart,

You might be interested in Ed's last point : Acceleo caches the result of query evaluations. If you can pinpoint specific expressions in your templates that are (or could be) extremely costly/executed so many times that it hurts, you should extract these expressions into Acceleo queries to avoid re-evaluation.

Since your models and templates are confidential, you might be interested in contacting info@obeo.fr to discuss the possibility of signing an NDA with us so that we can help you with the profiling and enhancing of your templates? These use case would also be beneficial to us since it would help us in determining the bottlenecks in Acceleo itself.

Ed,

Edward Willink wrote on Thu, 19 April 2012 04:41

Eclipse OCL supports evaluation of individual queries and has no caching
to improve reuse of partial results. (The Impact Analyzer provides
caches for re-evaluation.)

Avoiding re-evaluation is a user/consumer responsibility, so a user can
use let-variables.


In my experience, the main problem with Acceleo and OCL is their eager need to create new Collections every time something is called : set->select(expression)->filter(EClass)->collect(name) will cause the creation of at least four separate colleciton : a first when creating the initial "set" (IIRC, even a call to "package.eClassifiers" will return a new list instead of the "BasicEList" returned by a property call to "eClassifiers), a second for the select, a third for the filter, and finally a fourth for the collect. For Sets/LinkedSets, these creations are extremely costly. Using guava in the implementation would be extremely beneficial, but that's something I haven't gotten to try yet.

Edward Willink wrote on Thu, 19 April 2012 04:41

There is a pending patch for QVTo from JPL that gives massive
improvements in compilation time by caching all the meta-model lookups
that are otherwise repeated needlessly. (The OCL to Java generator
creates a flat meta-model representation that could be enabled to give
similar compilation time benefits with the new pivot model.)


I don't think QVTo is relevant here?

Edward Willink wrote on Thu, 19 April 2012 04:41

Acceleo provides query evaluation caches to ensure that queries are not
re-evaluated needlessly.

The user problem seems to be small templates and large data, so I wonder
whether the query caches are malfunctioning, or whether uncached derived
features are in use that could benefit from use of cached Acceleo queries.


Indeed, this is a potential improvement axis that can be explored.

Laurent Goubet
Obeo
Re: [Acceleo] Performance Profiling [message #849785 is a reply to message #849730] Thu, 19 April 2012 08:56 Go to previous messageGo to next message
Bart Theelen is currently offline Bart Theelen
Messages: 20
Registered: February 2012
Junior Member
Ok, I got it now. There are indeed some ->filter and one ->select in all the code, but they all do different filterings/selections and basically only once for each context in which they are called. But I will look into creating queries for those.

Thanks for the offer for which an NDA is needed, but this could be rather difficult to arrange on a short term. I will however put it on the agenda here.

EDIT: Well, I created queries and let-variables for all sletect, filter and exist selection methods without reducing the run-time substantially (I killed the running transformation after an hour). In fact, on a smaller example it seems to perform worse...

[Updated on: Thu, 19 April 2012 09:58]

Report message to a moderator

Re: [Acceleo] Performance Profiling [message #850849 is a reply to message #849730] Fri, 20 April 2012 07:35 Go to previous messageGo to next message
Ed Willink is currently offline Ed Willink
Messages: 3844
Registered: July 2009
Senior Member
On 19/04/2012 13:01, Laurent Goubet wrote:
> Ed,
>
> In my experience, the main problem with Acceleo and OCL is their eager
> need to create new Collections every time something is called :
> set->select(expression)->filter(EClass)->collect(name) will cause the
> creation of at least four separate colleciton : a first when creating
> the initial "set" (IIRC, even a call to "package.eClassifiers" will
> return a new list instead of the "BasicEList" returned by a property
> call to "eClassifiers), a second for the select, a third for the
> filter, and finally a fourth for the collect. For Sets/LinkedSets,
> these creations are extremely costly. Using guava in the
> implementation would be extremely beneficial, but that's something I
> haven't gotten to try yet.
>
The polymorphic Value collections in the new pivot evaluator are
partially mutable which allows an in-place update for an iterate.
Associated iterator variables also have mutable values avoiding the need
for unregistration and reregistration of iterator variable bindings on
every iteration step.

A relatively simple optimisation
(https://bugs.eclipse.org/bugs/show_bug.cgi?id=339783) would allow
in-place update for the last access to any collection giving a massive
reduction in collection churning. This is of only moderate priority for
me in the current interpreter. It will be much more useful as part of
the OCL to Java code generator.

If guava collections are more efficient, then switching to them as the
base implementation should be very easy too.

However I'm not yet confident enough in the basic naive OCL to Java code
generator to start optimising.

Regards

Ed Willink
Re: [Acceleo] Performance Profiling [message #853636 is a reply to message #850849] Mon, 23 April 2012 03:35 Go to previous messageGo to next message
Bart Theelen is currently offline Bart Theelen
Messages: 20
Registered: February 2012
Junior Member
Is the problem that you are talking about due to combining several of the select, filter, collect, exist etc in one code line? Or does the creation of new collections also occur for each of these methods individually, i.e., when they are used 'alone' in a code line (as in our code)?

[Updated on: Mon, 23 April 2012 03:36]

Report message to a moderator

Re: [Acceleo] Performance Profiling [message #853720 is a reply to message #853636] Mon, 23 April 2012 05:13 Go to previous messageGo to next message
Ed Willink is currently offline Ed Willink
Messages: 3844
Registered: July 2009
Senior Member
Hi

Combining several iterations on a single code line is slightly
beneficial since the evaluation is often depth first, so you may benefit
from some tree pruning.

The OCL evaluator has no caching so if you recalculate you recalculate
and on large models the inefficiency is noticeable.

Your indication that you have many iterations on one line suggests that
you may benefit from creating an intermediate model of partial analysis
results that can then be reused to create the final results. The
intermediate may be an overtly distinct model using an MtoM
transformation, or an implicit intermediate using Acceleo queries to
cache the intermediate nodes. But unless you show us the code we can
only guess at where your problems lie.

Regards

Ed Willink

On 23/04/2012 08:35, Bart Theelen wrote:
> Is the problem that you are talking about due to combining several of
> the select, filter, collect, exist etc in one code line? Or does the
> creation of new collections also occur for each of these methods in
> case they are used 'alone' in a code line (as in our code)?
Re: [Acceleo] Performance Profiling [message #855061 is a reply to message #853720] Tue, 24 April 2012 09:44 Go to previous messageGo to next message
Bart Theelen is currently offline Bart Theelen
Messages: 20
Registered: February 2012
Junior Member
Our Acceleo code has a few constructs like:

[for Object: Class | AbstractClass->filter(Class)]
   [PrintSomeString/]
[/for]


Here is one selection/filtering per code line, which is typical in our Acceleo code. What is also typical is that [PrintSomeString/] is includes similar loops on more detailed subobjects in each Object (see above for the SuperClass/SubClass relations and their sizes).

The most complex piece of Acceleo code is for objects of class C in a specific context. This piece of code looks like:

[for (Object: Class| SetOfC.Attribute->asOrderedSet())]
  [let x : OrderedSet(ClassC) = SetOfC->select(Attribute = Object)]
GenerateCommonOutputStringsFor(x)
    [for (y: ClassC | x)]
GenerateSpecificOutputStringsFor(y)
    [/for]
  [/let]
[/for]


Today we got some new suggestions that were suspected to potentially impact performance:

- first do a validation of the model to ensure that is it really completely in memory
- use binary file formats for the models (not sure yet how that is going to work for our case as the models are distributed over multiple files, with references between them)
- we use custom xmi:id's to identify objects and maybe they have some internal use that impacts the performance

If any of these ring some bells, please let me know...

[Updated on: Tue, 24 April 2012 09:54]

Report message to a moderator

Re: [Acceleo] Performance Profiling [message #856328 is a reply to message #855061] Wed, 25 April 2012 11:11 Go to previous messageGo to next message
Ronald Krijgsheld is currently offline Ronald Krijgsheld
Messages: 41
Registered: November 2010
Member
Hi

How do you invoke the templates. Does it have one entry point or do you invoke a lot of templates?

I ask, because I noticed that the default AbstractAcceleoGenerator.postGenerate() method unloads all models. That means that every time you create an instance of your YourConcreteGenerator (your subclass derived from the AbstractAcceleoGenerator) and call doGenerate() your models are read into memory.

- Ronald
(DISCLAIMER: I am probably not using the latest version, so this may have changed).
Re: [Acceleo] Performance Profiling [message #857324 is a reply to message #856328] Thu, 26 April 2012 09:01 Go to previous messageGo to next message
Laurent Goubet is currently offline Laurent Goubet
Messages: 1609
Registered: July 2009
Senior Member
Ronald,

postGenerate() still unloads all models in its latest version. We just can't let everything sleep in memory when we don't need it anymore.

We do advise users to use a single "entry point" template instead of multiple "main" (see C.14 of the Best Practices guide.

Bart,

Did you try profiling your generation to pinpoint the problematic expressions? As is the case with Java, it is difficult (if not useless) to try and optimize something without a profiler.

[for (Object: Class| SetOfC.Attribute->asOrderedSet())]
  [let x : OrderedSet(ClassC) = SetOfC->select(Attribute = Object)]
GenerateCommonOutputStringsFor(x)
    [for (y: ClassC | x)]
GenerateSpecificOutputStringsFor(y)
    [/for]
  [/let]
[/for]


Iterate on all attribute values of a set of "C", then find the set of "C" that have this possible value and print something?

Taking into account the numbers you gave us before (100 000 'C'), that's quite a number of iteration over a big collection (not to mention that creating an OrderedSet is also costly (asOrderedSet) if you do it often) : suppose you have 10 possible values for "Attribute" : you'll have a first iteration over 100 000 elements to find them, then 10 more iterations over these elements to find back those that match this attribute value... And finally a new iteration over these "matching" classes to output "GenerateSpecificOutputString..." Whatever the language, more than a million iterations is bound to take time.

For a need such as this one, Acceleo/OCL don't offer good alternative to what you have (or so I think). The best I can think of is to use a Java service to find the lists you need. Something like this :

public Collection<List<ClassC>> orderByAttributeValue(Set<ClassC> setOfC) {
    Map<ClassC, List<ClassC>> result = new HashMap<ClassC, List<ClassC>>(10);
    for (ClassC candidate : setOfC) {
        ClassC attribute = candidate.getAttribute();
        List<ClassC> list = result.get(attribute);
        if (list == null) {
            list = new ArrayList<ClassC>(10000);
            result.put(attribute, list);
        }
        list.add(candidate);
    }
    return result.values();
}


Which only does a single iteration over your "setOfC" and which you could then use like this :

[for (x : Sequence(ClassC) | setOfC.orderByAttributeValue())]
GenerateCommonOutputStringsFor(x)
    [for (y: ClassC | x)]
GenerateSpecificOutputStringsFor(y)
    [/for]
[/for]


this should only iterate twice over the full set (once in the java service, a second time by iterating over each of the subsequences).

Once again, these are optimization you should only make after profiling, to make sure your optimizing the correct expressions.

Laurent Goubet
Obeo

[Updated on: Thu, 26 April 2012 09:04]

Report message to a moderator

Re: [Acceleo] Performance Profiling [message #857345 is a reply to message #857324] Thu, 26 April 2012 09:19 Go to previous messageGo to next message
Ed Willink is currently offline Ed Willink
Messages: 3844
Registered: July 2009
Senior Member
Hi

[for (Object: Class| SetOfC.Attribute->asOrderedSet())]
[let x : OrderedSet(ClassC) = SetOfC->select(Attribute = Object)]

is almost certainly inefficient.

->asOrderedSet()

probably requires a total copy from the source collection into a
LinkedHashSet which is a linked list so has link objects and might do
linear duplicate detection. Since your ordering is not determinate
anyway, asSet() will be better. Similarly doing select on what I think
must be an OrderedSet to avoid a compilation error may also involve
linear rather than hashed access.

Don't use OrderedSet unless both uniqueness and ordering are relevant.

Regards

Ed Willink



On 26/04/2012 14:02, Laurent Goubet wrote:
> Ronald,
>
> postGenerate() still unloads all models in its latest version. We just
> can't let everything sleep in memory when we don't need it anymore.
>
> We do advise users to use a single "entry point" template instead of
> multiple "main" (see C.14 of the
> http://www.obeonetwork.com/page/acceleo-best-practices guide.
>
> Bart,
>
> Did you try profiling your generation to pinpoint the problematic
> expressions? As is the case with Java, it is difficult (if not
> useless) to try and optimize something without a profiler.
>
> Quote:
>> [for (Object: Class| SetOfC.Attribute->asOrderedSet())]
>> [let x : OrderedSet(ClassC) = SetOfC->select(Attribute = Object)]
>> GenerateCommonOutputStringsFor(x)
>> [for (y: ClassC | x)]
>> GenerateSpecificOutputStringsFor(y)
>> [/for]
>> [/let]
>> [/for]
>
>
> Iterate on all attribute values of a set of "C", then find the set of
> "C" that have this possible value and print something?
>
> Taking into account the numbers you gave us before (100 000 'C'),
> that's quite a number of iteration over a big collection (not to
> mention that creating an OrderedSet is also costly (asOrderedSet) if
> you do it often) : suppose you have 10 possible values for "Attribute"
> : you'll have a first iteration over 100 000 elements to find them,
> then 10 more iterations over these elements to find back those that
> match this attribute value... And finally a new iteration over these
> "matching" classes to output "GenerateSpecificOutputString..."
> Whatever the language, more than a million iterations is bound to take
> time.
>
> For a need such as this one, Acceleo/OCL don't offer good alternative
> to what you have (or so I think). The best I can think of is to use a
> Java service to find the lists you need. Something like this :
>
> Quote:
>> public Collection<List<ClassC>> orderByAttributeValue(Set<ClassC>
>> setOfC) {
>> Map<ClassC, List<ClassC>> result = new HashMap<ClassC,
>> List<ClassC>>(10);
>> for (ClassC candidate : setOfC) {
>> ClassC attribute = candidate.getAttribute();
>> List<ClassC> list = result.get(attribute);
>> if (list == null) {
>> list = new ArrayList<ClassC>(10000);
>> result.put(attribute, list);
>> }
>> list.add(candidate);
>> }
>> return result.values();
>> }
>
>
> Which only does a single iteration over your "setOfC" and which you
> could then use like this :
>
> Quote:
>> [for (x : Sequence(ClassC) | setOfC.orderByAttributeValue())]
>> GenerateCommonOutputStringsFor(x)
>> [for (y: ClassC | x)]
>> GenerateSpecificOutputStringsFor(y)
>> [/for]
>> [/for]
>
>
> this should only iterate twice over the full set (once in the java
> service, a second time by iterating over each of the subsequences).
>
> Once again, these are optimization you should only make after
> profiling, to make sure your optimizing the correct expressions.
>
> Laurent Goubet
> Obeo
Re: [Acceleo] Performance Profiling [message #869032 is a reply to message #857345] Wed, 02 May 2012 17:49 Go to previous messageGo to next message
Bart Theelen is currently offline Bart Theelen
Messages: 20
Registered: February 2012
Junior Member
Thanks to all for the interesting answers.

Here are some responses:

@Ronald: The number of times the given 'complex' piece of Acceleo code is invoked is for our killer case study 131 times. There are three entry points for this piece of code and for our killer case there are only two entry points that are actually used.

@Laurent: The described behavior is indeed basically what we want: For a set 'C', find all elements that have the same Attribute and for each possible unique Attribute, print something for all 'C's that have that Attribute. The 'C' sets that go into an invocation have for our killer case at maximum about 650 'C's in most invocations. The number of possible values for Attribute is for our killer case (by coincidence) the same 650.

We have not yet been able to do proper profiling of the Acceleo code, but we were able to profile the Java implementation. Here we found one problem: the referencing between two particular models in different files. There are however more inter-file references for large(r) numbers of objects but for some unclear reason there was only a problem between two particular files and not between any of the others.

Now, we have done some tests with a completely different model and different transformation etc. Here, we don't have references between files as everything is in one file (so this test rules that out as a possible problem). We have 25000 object and performing the transformation (which is more complex than the ones discussed in this topic so far) also takes hours instead of the expected seconds/few minutes. The experiments we have done for this transformation shows that having more than about 5000 objects gives run times over 30min, which is again far beyond what we expected. 5000 objects is 'small' compared to the real-life models that we have to handle. Our killer case will at some point also go through this transformation and will have 25000+ objects...

The proposed kind of optimizations are far beyond our Acceleo skills Wink But they could be a good idea if the problem is really located in this part of the code. If I completely comment the 'complex' piece of code out, things do get better but not as substantial as we expect (it's still 30+ min).

We slowly start to believe that there might be a deeper underlying problem (somewhere in the EMF core) but we still need to further investigate things.

@Ed: I don't really understand why using ->asOrderedSet would require creating copies. I mean, there is no specification of what the order should be and hence any ordering is a good one, also the one that happened to be the case for the retrieved set / collection. So, I just see a cast and that's it. But apparently I am missing something here? Anyway, I agree that we don't need any particular ordering. Unfortunately, changing ->asOrderedSet into ->asSet does not help any bit in improving the performance...

[Updated on: Wed, 02 May 2012 17:56]

Report message to a moderator

Re: [Acceleo] Performance Profiling [message #869299 is a reply to message #869032] Thu, 03 May 2012 03:45 Go to previous messageGo to next message
Laurent Goubet is currently offline Laurent Goubet
Messages: 1609
Registered: July 2009
Senior Member
Bart,

We're basically shooting in the dark here. Wild guesses are probably not going to help you much (I'd wager it'll do the oppposite).

We have used Acceleo for large models to make transformations that sound much more complex that what you describe here, though your models are in the same "large" range as ours. We did not encounter such huge performance drawbacks as what you're describing here, thus I truly believe that there are ways to improve your transformation, and most likely one or two "big" bottlenecks that should be improved.

However as I already mentionned, as long as you can't provide us with profiling results or the (or "one of" the) problematic OCL/Acceleo expressions, we can only provide you with general information that may or may not be related to the true issue. We can help with Acceleo, OCL, EMF or your own code if that's where the enhancements must be made, but we need the profiling results to be able to do anything at all. It might sound like I'm just insisting, but concluding an NDA with Obeo is a good trade-off for that : it does not compel you to paying anything or subscribing to any commercial offer, it only allows you to provide us with private information by making sure we won't divulge it to any third party. At least it would allow us to quickly pinpoint the exact bottleneck and provide you with workarounds and/or enhancements.

As for Ed's mention of the problem with "asOrderedSet" : both "asSet" and "asOrderedSet" ("asSequence" and "asBag" too for that matter) work by creating a whole new collection as a copy of the input. In terms of Java, calling the OCL "collection->asOrderedSet()" is the equivalent of "new LinkedHashSet(collection)" which iterates over the whole input collection (even if it is already a linked hash set), computes the has of every entry, checks if the new set already contains it, adds it if not ... which is why it is costly. "asSet" is the equivalent of Java's "new HashSet(collection)" which is a little less costly, but still too much.

Laurent Goubet
Obeo
Re: [Acceleo] Performance Profiling [message #869333 is a reply to message #869299] Thu, 03 May 2012 04:40 Go to previous messageGo to next message
Ed Willink is currently offline Ed Willink
Messages: 3844
Registered: July 2009
Senior Member
Hi

Echoing Laurent's comment; we can't help when you don't let us.

From, the limited information, I strongly suspect that it is either an
OCL limitation or an EMF access issue aggravated by the Acceleo context
and your usage.

That you report no change from asOrderedSet makes me think that the
problem is localised in a very small area, which should show up through
straightforward Java profiling with eg VirtualVM.

That your problems have an at least 650 by 650 dimension indicates that
it will be very sensitive to accidental quadratic, or worse, rather than
linear behaviour.

Partition your program so that you measure:

Load time
Transform time
Save time
ReLoad time
ReTransform time
ReSave time

and see how these vary between your current model and one one tenth of
the size.

Regards

Ed Willink

On 03/05/2012 08:45, Laurent Goubet wrote:
> Bart,
>
> We're basically shooting in the dark here. Wild guesses are probably
> not going to help you much (I'd wager it'll do the oppposite).
>
> We have used Acceleo for large models to make transformations that
> sound much more complex that what you describe here, though your
> models are in the same "large" range as ours. We did not encounter
> such huge performance drawbacks as what you're describing here, thus I
> truly believe that there are ways to improve your transformation, and
> most likely one or two "big" bottlenecks that should be improved.
>
> However as I already mentionned, as long as you can't provide us with
> profiling results or the (or "one of" the) problematic OCL/Acceleo
> expressions, we can only provide you with general information that may
> or may not be related to the true issue. We can help with Acceleo,
> OCL, EMF or your own code if that's where the enhancements must be
> made, but we need the profiling results to be able to do anything at
> all. It might sound like I'm just insisting, but concluding an NDA
> with Obeo is a good trade-off for that : it does not compel you to
> paying anything or subscribing to any commercial offer, it only allows
> you to provide us with private information by making sure we won't
> divulge it to any third party. At least it would allow us to quickly
> pinpoint the exact bottleneck and provide you with workarounds and/or
> enhancements.
>
> As for Ed's mention of the problem with "asOrderedSet" : both "asSet"
> and "asOrderedSet" ("asSequence" and "asBag" too for that matter) work
> by creating a whole new collection as a copy of the input. In terms of
> Java, calling the OCL "collection->asOrderedSet()" is the equivalent
> of "new LinkedHashSet(collection)" which iterates over the whole input
> collection (even if it is already a linked hash set), computes the has
> of every entry, checks if the new set already contains it, adds it if
> not ... which is why it is costly. "asSet" is the equivalent of Java's
> "new HashSet(collection)" which is a little less costly, but still too
> much.
>
> Laurent Goubet
> Obeo
Re: [Acceleo] Performance Profiling [message #869511 is a reply to message #869333] Thu, 03 May 2012 10:01 Go to previous messageGo to next message
Bart Theelen is currently offline Bart Theelen
Messages: 20
Registered: February 2012
Junior Member
I am fully aware of the fact that you can't help much without proper insight in the model and transformation details. For me it is obvious that we should arrange an NDA to have you experts look at it. Unfortunately, the ones who make the decisions about these things aren't convinced (yet)...
Re: [Acceleo] Performance Profiling [message #870916 is a reply to message #869511] Thu, 10 May 2012 05:07 Go to previous messageGo to next message
Jack Sleuters is currently offline Jack Sleuters
Messages: 6
Registered: January 2012
Junior Member
Hi, I'm a colleague of Bart Theelen and noticed the following Acceleo behavior which could be related to the performance issue:

I created a template file which has the following calling structure:

Step 1: Main calls template 1
Step 2: Template 1 calls template 2
Step 3: Template 2 calls template 3
Step 4: Template 3 calls template 4
Step 5: Template 4 calls template 5
Step 6: Template 5 calls template 6

Now, when I generate an input instance model with a large number of objects that will be processed in template 6 then the performance problem appears.
When I change Template 5 to incorporate the contents of template 6, and effectively changing the calling depth to 5 (Step 6 is no longer performed) the performance is again up to an acceptable level!

I created a sample ecore model and a Acceleo template file for that and attached it. It contains:

"AcceleoScaling" project -> "Model": contains the ecore model used

"AcceleoTest" project contains:
• the "test.acceleotest" input file with 20000 "Records"
• "generate.mtl", the Acceleo template file used

Input file consisting of 20000 objects to be processed in Step 6, leaving the calling depth of 6 (Step 6 is executed) this results in a generation time of 34s.
Changing the calling depth to 5 (as described above) results in a generation time of 3s including the loading of the input file.

I don't know it this is only appearing in my eclipse environment but it might help in tracing the performance problem. B.t.w. I'm using Acceleo version 3.3.0M7

Kind regards,
Jack
Re: [Acceleo] Performance Profiling [message #870933 is a reply to message #870916] Thu, 10 May 2012 05:39 Go to previous messageGo to next message
Ed Willink is currently offline Ed Willink
Messages: 3844
Registered: July 2009
Senior Member
Hi

Seems rather unlikely so rather interesting and a very useful repro, but
to avoid pursuing blind alleys, can you post your launch configurations...

If you go to the "Common" tab of the Launch Configuration editor you can
specify a shared file, typically at the root or .settings or .launches
folder of your project.

Then we can just run e.g. 6-level launch, or 5-level launch and be sure
that we're replicating your scenario.

Regards

Ed Willink



On 10/05/2012 10:07, Jack Sleuters wrote:
> Hi, I'm a colleague of Bart Theelen and noticed the following Acceleo behavior which could be related to the performance issue:
>
> I created a template file which has the following calling structure:
>
> Step 1: Main calls template 1
> Step 2: Template 1 calls template 2
> Step 3: Template 2 calls template 3
> Step 4: Template 3 calls template 4
> Step 5: Template 4 calls template 5
> Step 6: Template 5 calls template 6
>
> Now, when I generate an input instance model with a large number of objects that will be processed in template 6 then the performance problem appears.
> When I change Template 5 to incorporate the contents of template 6, and effectively changing the calling depth to 5 (Step 6 is no longer performed) the performance is again up to an acceptable level!
>
> I created a sample ecore model and a Acceleo template file for that and attached it. It contains:
>
> "AcceleoScaling" project -> "Model": contains the ecore model used
>
> "AcceleoTest" project contains:
> • the "test.acceleotest" input file with 20000 "Records"
> • "generate.mtl", the Acceleo template file used
>
> Input file consisting of 20000 objects to be processed in Step 6, leaving the calling depth of 6 (Step 6 is executed) this results in a generation time of 34s.
> Changing the calling depth to 5 (as described above) results in a generation time of 3s including the loading of the input file.
>
> I don't know it this is only appearing in my eclipse environment but it might help in tracing the performance problem. B.t.w. I'm using Acceleo version 3.3.0M7
>
> Kind regards,
> Jack
Re: [Acceleo] Performance Profiling [message #870995 is a reply to message #870933] Thu, 10 May 2012 09:42 Go to previous messageGo to next message
Jack Sleuters is currently offline Jack Sleuters
Messages: 6
Registered: January 2012
Junior Member
I'm not exactly sure what you want me to do so I just add the text how I run the examples:

1. In the Acceleo test project open the "src" folder.
2. Open the "AcceleoTest.main" package.
3. Right click the "generate.mtl" file and select "Run as -> Run configurations".
4. In the dialog select the "Acceleo" tab and make sure that the fields are filled in as follows:

Project: AcceleoTest
Main Class: AcceleoTest.main.Generate
Model: /AcceleoTest/test.acceleotest
Target: /AcceleoTest/generated

This should give you the exact Run configuration I used.

Furthermore, to run the 6 level example, edit generate.mtl such that:

[template public generateRecords(records : OrderedSet(Record)) post(trim())]
[comment level 5/]
[for (record : Record | records)]
[generateRecord(record)/]
[/for]
[/template]


[template public generateRecord(record : Record) post(trim())]
[comment level 6/]
<record name="[record.name/]" address="[record.address/]">
</record>
[/template]


To run the 5 level example, edit generate.mtl such that:

[template public generateRecords(records : OrderedSet(Record)) post(trim())]
[comment level 5/]
[for (record : Record | records)]
<record name="[record.name/]" address="[record.address/]">
</record>
[/for]
[/template]


[template public generateRecord(record : Record) post(trim())]
[comment level 6/]
<record name="[record.name/]" address="[record.address/]">
</record>
[/template]


That should do the trick.

Kind regards,
Jack
Re: [Acceleo] Performance Profiling [message #871560 is a reply to message #870995] Mon, 14 May 2012 08:21 Go to previous messageGo to next message
Jack Sleuters is currently offline Jack Sleuters
Messages: 6
Registered: January 2012
Junior Member
To rule out that my eclipse environment introduces this behaviour, I also tested on a colleague's Pc with eclipse configuration. The result was the same. At template depth of 6 the generation takes 34s. At template depth of 5 the generation takes about 1s.

Did anyone try my test projects already and experience the same behaviour?

Kind regards,
Jack
Re: [Acceleo] Performance Profiling [message #873189 is a reply to message #871560] Thu, 17 May 2012 12:43 Go to previous messageGo to next message
Ed Willink is currently offline Ed Willink
Messages: 3844
Registered: July 2009
Senior Member
Hi Jack

Your original description was misleading, implying that there were five
levels of recursion and that a sixth was costly.

Your problem is actually very simple.

In the fast case, a for loop appends 20000 contributions directly; there
are tens of template invocations.

In the slow case, a for loop appends 20000 contributions via nested
call; there are 20000 + tens of template invocations.

So on my modest laptop, time in ms:

5 level

main 0.0
pre 1.841
post 4.914

6 level

main 0.0
pre 1.95
post 80.636

i.e 2.0 s to start and load
3.0 s to append directly
75.0 s for 20000 extra template invocations, so 3.7ms/template invocation

This is disappointing, but not unsurprising, since Acceleo is
interpreted and a template invocation is a complex activity.

In Kepler, the direct OCL 2 Java code generator may be in place and
exploited by Acceleo offering massive opportunities for improvement.

Regards

Ed Willink


On 14/05/2012 13:21, Jack Sleuters wrote:
> To rule out that my eclipse environment introduces this behaviour, I
> also tested on a colleague's Pc with eclipse configuration. The result
> was the same. At template depth of 6 the generation takes 34s. At
> template depth of 5 the generation takes about 1s.
>
> Did anyone try my test projects already and experience the same
> behaviour?
>
> Kind regards,
> Jack
Re: [Acceleo] Performance Profiling [message #874562 is a reply to message #873189] Mon, 21 May 2012 02:55 Go to previous messageGo to next message
Jack Sleuters is currently offline Jack Sleuters
Messages: 6
Registered: January 2012
Junior Member
Hi Edward,

Thanks for your reply. Indeed the 20000 times calling a template is the problem. I changed the generateBook template as follows:

[template public generateBook(book : Book)]
[comment level 2/]
<book "[book.name/]">
[comment level 3/]
[for (aFolder : Folder | book.folder)]
[comment level 4/]
<folder name="[aFolder.name/]">
[comment level 5/]
[for (record : Record | aFolder.record)]
[generateRecord(record)/]
[/for]
</folder>
[/for]
</book>
[/template]


This means that the 20000 invocations of the generateRecord template are now done at depth 2. The result is indeed the same taking about 32s to generate.

At least we now know where the performance drop comes from. We now have to think about a good solution for our project.

Kind regards,
Jack
Re: [Acceleo] Performance Profiling [message #876224 is a reply to message #874562] Thu, 24 May 2012 03:28 Go to previous messageGo to next message
Thomas Buchmann is currently offline Thomas Buchmann
Messages: 53
Registered: November 2010
Member
I appologize if this is not the proper thread, but i'm also facing massive performance issues since the update to Acceleo 3.2.1. Everything was fine with Acceleo 3.2.0, so maybe some performance glitches are not only related to bad template design?

An Example: With Acceleo 3.2.0, a model with approx. 300 Classes has been generated in less than 10 seconds. With Acceleo 3.2.1 the transformation is running since more than 10 minutes now and it is not finished as i'm typing these lines. Please note that there was no change in the templates!

Kind regards,
Thomas
Re: [Acceleo] Performance Profiling [message #876252 is a reply to message #876224] Thu, 24 May 2012 04:30 Go to previous messageGo to next message
Ed Willink is currently offline Ed Willink
Messages: 3844
Registered: July 2009
Senior Member
Hi

I recommend trying 3.3RC1 and some forceful refreshes, cleans and
rebuilds. Hopefully the problem is already fixed; I find 3.3 much better
now; I couldn't use 3.2 at all.

If the problem persists, any repro or VisualVM performance information
could be useful.

When things go really slow, it can be worth checking with Wireshark that
there is no untoward Ethernet traffic.

I know that the Acceleo team are pretty busy at present preparing the
Juno release. Getting a fix for this in Juno would be really good, but
probably needs help from you to make the problem as easily reproducible
as possible. Ideally a simplified template as part of a zipped project
with clear how to launch instructions; again ideally, a saved launch
config in the project.

Regards

Ed Willink


On 24/05/2012 08:28, Thomas Buchmann wrote:
> I appologize if this is not the proper thread, but i'm also facing
> massive performance issues since the update to Acceleo 3.2.1.
> Everything was fine with Acceleo 3.2.0, so maybe some performance
> glitches are not only related to bad template design?
>
> An Example: With Acceleo 3.2.0, a model with approx. 300 Classes has
> been generated in less than 10 seconds. With Acceleo 3.2.1 the
> transformation is running since more than 10 minutes now and it is not
> finished as i'm typing these lines. Please note that there was no
> change in the templates!
>
> Kind regards,
> Thomas
Re: [Acceleo] Performance Profiling [message #876260 is a reply to message #876252] Thu, 24 May 2012 04:58 Go to previous messageGo to next message
Thomas Buchmann is currently offline Thomas Buchmann
Messages: 53
Registered: November 2010
Member
Hi Ed,

thanks a lot for your quick reply. I will give Acceleo 3.3RC1 a try and report the results.

Regards,
Thomas

[Updated on: Thu, 24 May 2012 06:21]

Report message to a moderator

Re: [Acceleo] Performance Profiling [message #876349 is a reply to message #847652] Thu, 24 May 2012 08:42 Go to previous messageGo to next message
Michaël Melchiore is currently offline Michaël Melchiore
Messages: 39
Registered: April 2012
Member
Hi,

I sincerely hope that switching to Juno does solve Thomas' problem. Yet, even if it does, providing Obeo with a sample test case will help fix the issue in the 3.2.x branch.

Some of us work in a controlled environment and will still be stuck with Indigo releases for the upcoming months. If Acceleo 3.2.x is not usable, then it should be fixed

Please think of our poor souls Smile

Michaël

[Updated on: Thu, 24 May 2012 08:42]

Report message to a moderator

Re: [Acceleo] Performance Profiling [message #876353 is a reply to message #876260] Thu, 24 May 2012 08:44 Go to previous messageGo to next message
Thomas Buchmann is currently offline Thomas Buchmann
Messages: 53
Registered: November 2010
Member
Hi Ed,

i'm stuck. I upgraded to Acceleo 3.3.0RC1 as suggested. But the performance issues still remain. I setup a launch configuration with profiling support enabled. However, the profiling result looks fine (see attached screenshot). Nevertheless, it took 57s to complete the M2T transformation... At least i think the problem is not within the templates... But where could it be?

index.php/fa/9896/0/

Ah and btw: i tried Ed's suggestion and ran wireshark. But it didn't reveal anything suspicious...

[Updated on: Thu, 24 May 2012 08:56]

Report message to a moderator

Re: [Acceleo] Performance Profiling [message #876366 is a reply to message #876353] Thu, 24 May 2012 09:07 Go to previous messageGo to next message
Ed Willink is currently offline Ed Willink
Messages: 3844
Registered: July 2009
Senior Member
Hi

Try increasing the number of open/closed projects, installed modeling
projects.

There a number of 'helpful' modeling applications that endeavour to
discover models and model elements so that you can access them easily.

These can scale very badly. Be suspicious of at least MWE, Xtext,
EMF-Index, Modisco, Acceleo.

If direct profiling doesn't help, try putting in a few
System.currentTime's to see which phase is going slow.

Try using a Java application rather than plugin launch.

If you're happy to use Acceleo 3.3.0RC1 then are you using Eclipse
4.2RC1? It seems like you're seeing something that we really want to fix.

Regards

Ed Willink


On 24/05/2012 13:44, Thomas Buchmann wrote:
> Hi Ed,
>
> i'm stuck. I upgraded to Acceleo 3.3.0RC1 as suggested. But the performance issues still remain. I setup a launch configuration with profiling support enabled. However, the profiling result looks fine (see attached screenshot). Nevertheless, it took 57s to complete the M2T transformation... At least i think the problem is not within the templates... But where could it be?
>
>
Re: [Acceleo] Performance Profiling [message #876385 is a reply to message #876366] Thu, 24 May 2012 09:49 Go to previous messageGo to next message
Thomas Buchmann is currently offline Thomas Buchmann
Messages: 53
Registered: November 2010
Member
Edward Willink wrote on Thu, 24 May 2012 09:07

Try increasing the number of open/closed projects, installed modeling
projects.

There a number of 'helpful' modeling applications that endeavour to
discover models and model elements so that you can access them easily.

These can scale very badly. Be suspicious of at least MWE, Xtext,
EMF-Index, Modisco, Acceleo.


Thanks a lot for the hint Ed. I created a launch configuration for my toolset with only minimal requirements (Modisco & Acceleo are included though) and everything works fine now!! Now i'm trying to figure out, which plugin causes the explosion of the runtime complexity.
Re: [Acceleo] Performance Profiling [message #876446 is a reply to message #876385] Thu, 24 May 2012 11:52 Go to previous messageGo to next message
Ed Willink is currently offline Ed Willink
Messages: 3844
Registered: July 2009
Senior Member
Hi

If you experience the same speed ration in a nested Eclipse, I suggest
instrumenting ResourceSetImpl.loadResource with some println's to see
what resources are loaded. The problem is almost certainly something
that loads everything.

Regards

Ed Willink


On 24/05/2012 14:49, Thomas Buchmann wrote:
> Edward Willink wrote on Thu, 24 May 2012 09:07
>> Try increasing the number of open/closed projects, installed modeling
>> projects.
>>
>> There a number of 'helpful' modeling applications that endeavour to
>> discover models and model elements so that you can access them easily.
>>
>> These can scale very badly. Be suspicious of at least MWE, Xtext,
>> EMF-Index, Modisco, Acceleo.
>
>
> Thanks a lot for the hint Ed. I created a launch configuration for my
> toolset with only minimal requirements (Modisco & Acceleo are included
> though) and everything works fine now!! Now i'm trying to figure out,
> which plugin causes the explosion of the runtime complexity.
>
Re: [Acceleo] Performance Profiling [message #876511 is a reply to message #876446] Thu, 24 May 2012 14:41 Go to previous messageGo to next message
Thomas Buchmann is currently offline Thomas Buchmann
Messages: 53
Registered: November 2010
Member
Ed,

thanks again for your valuable hints!! I tried your suggestion and found out that the EEF plugin causes the performance problems. Once it is deactivated, everything is fine.

I'm using Indigo SR2 Modeling Distribution. EEF Version 1.0.2.v20120216-1513

Kind regards,
Thomas

Re: [Acceleo] Performance Profiling [message #876531 is a reply to message #876511] Thu, 24 May 2012 15:36 Go to previous messageGo to next message
Ed Willink is currently offline Ed Willink
Messages: 3844
Registered: July 2009
Senior Member
Hi

EEF is part of the Simultaneous Release so please raise a 'major' Bugzilla.

Regards

Ed Willink

On 24/05/2012 19:41, Thomas Buchmann wrote:
> Ed,
>
> thanks again for your valuable hints!! I tried your suggestion and
> found out that the EEF plugin causes the performance problems. Once it
> is deactivated, everything is fine.
>
> I'm using Indigo SR2 Modeling Distribution. EEF Version
> 1.0.2.v20120216-1513
>
> Kind regards,
> Thomas
>
>
Re: [Acceleo] Performance Profiling [message #876555 is a reply to message #876531] Thu, 24 May 2012 16:34 Go to previous messageGo to next message
Thomas Buchmann is currently offline Thomas Buchmann
Messages: 53
Registered: November 2010
Member
Hi,

done: https://bugs.eclipse.org/bugs/show_bug.cgi?id=380589

Regards,
Thomas
Re: [Acceleo] Performance Profiling [message #878963 is a reply to message #847652] Wed, 30 May 2012 05:25 Go to previous messageGo to next message
Laurent Goubet is currently offline Laurent Goubet
Messages: 1609
Registered: July 2009
Senior Member
Hi Thomas,

Sorry for the late response, we have been really busy with the incoming release of Juno.

Having an old version of EEF (anything prior to march IIRC) in your environment will slow down every single Acceleo generation you launch as a "plugin application", whatever the target, input model or template complexity. This was a major issue with one of Acceleo's extension points that needed to be fixed in both Acceleo (done since late 2011) and EEF (they updated their code around march/april).

This is a likely culprit for any slow down, but it should already be fixed in the later versions.

Laurent Goubet
Obeo
Re: [Acceleo] Performance Profiling [message #881795 is a reply to message #878963] Tue, 05 June 2012 05:11 Go to previous message
Thomas Buchmann is currently offline Thomas Buchmann
Messages: 53
Registered: November 2010
Member
Hi Laurent,

thanks for your reply. I will update my environment accordingly.

Regards,
Thomas
Previous Topic:Porting Xpand to Xtend2
Next Topic:[Ant/Xpand]problem with mwe script that uses xpt script
Goto Forum:
  


Current Time: Wed Apr 23 02:43:15 EDT 2014

Powered by FUDForum. Page generated in 0.03069 seconds