Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » M2T (model-to-text transformation) » Best practice to test M2T transformation(What is the best practice to (automatically) test M2T transformations. Are there any frameworks that might help?)
Best practice to test M2T transformation [message #656416] Fri, 25 February 2011 09:11 Go to next message
Niels Brouwers is currently offline Niels Brouwers
Messages: 67
Registered: July 2009
Member
We are currently incorporating M2T transformations within transformation sequences to transform models at low level of abstractions to program code (C/C++, Python, etc.). To assure better quality and ensure that the transformation results in working implementations, we would like to test our generators better than we currently do; currently we manually inspect the generated source code and try to compile it. If our customers don't complain, it is assumed to work fine! Wink

I started this topic to learn about the practices currently used to test the code generators created by the community and I hope this will be the start of a fruitful discussion.

Thinking about a better way to test our M2T transformations than described earlier, resulted in the following alternatives:

  1. Creating a framework which automatically executes the transformation for a set of test models and compares the output of the transformation with a set of reference texts.
  2. Create one big test model which covers the majority of the requirements, generate code from this model, execute the implementation and test this implementation with a (conventional) automatic tester.


The first alternative allows to test the generator very extensively, but will also result in a large amount of test models and corresponding reference text files. Furthermore, I am not aware of a framework which simplifies this task. Last but not least, I think it requires a lot of effort to prevent false positives due to irrelevant formatting differences (e.g. white spaces & comments in the code) and to non-deterministic code generation (e.g. the order of variable declarations might not be relevant to the compiler, but it will be for the M2T tester).

The second alternative is perceived to be less challenging, as we can easily generate code from one big model, we can use existing unit tester frameworks (e.g. PyUnit or CxxUnit) and we can better test if the resulting implementation behaves according to our expectations in stead being forced to prevent the false-positives. However, choosing this alternative will likely not result in a high level of test coverage.

It is my guess that the second alternative is good enough for us and provides the best benefit / effort ratio. What do you think about this? Is there a better alternative that I've missed?

Best regards,
Niels Brouwers.







Kind regards,
Niels Brouwers.
Re: Best practice to test M2T transformation [message #656714 is a reply to message #656416] Mon, 28 February 2011 03:53 Go to previous messageGo to next message
Stephane Begaudeau is currently offline Stephane Begaudeau
Messages: 406
Registered: April 2010
Location: Nantes (France)
Senior Member

Hi Niels,

My opinion would be to go for your first approach:
"Creating a framework which automatically executes the transformation for a set of test models and compares the output of the transformation with a set of reference texts."

Because with this approach, you can also test how your generator should handle an invalid input model. For example, if you are generating Java source code from UML and if you have to classes with the same name in the same package, you could test that your generator has created a log file with a message indicating that your model contains a problem. Same thing with a Java class with two identical methods or anything that would not compile.

Stephane Begaudeau, Obeo

--
Twitter: @sbegaudeau
Acceleo wiki: http://wiki.eclipse.org/Acceleo
Blogs: http://stephanebegaudeau.tumblr.com & http://sbegaudeau.tumblr.com



Re: Best practice to test M2T transformation [message #657693 is a reply to message #656416] Thu, 03 March 2011 11:24 Go to previous messageGo to next message
Niels Brouwers is currently offline Niels Brouwers
Messages: 67
Registered: July 2009
Member
Hi Stephane,

thank you for your response. I especially like your comment on testing the generator when handling an invalid input model. In my opinion, this situation should never happen. Models fed to a generator must always be correct, verified by running a set of audits on the model.

In general, I think you are right that the first approach to test M2T transformations is the best alternative. Can you think of a framework that can help me conveniently creating test cases and running them automatically? I have seen some examples of chains (.chain files) on the acceleo.org site, but I suspect this was available in the pre-Eclipse release of Acceleo. Any other frameworks spring to mind?

Thanx.

Niels.



Kind regards,
Niels Brouwers.
Re: Best practice to test M2T transformation [message #657819 is a reply to message #656416] Fri, 04 March 2011 03:50 Go to previous messageGo to next message
Stephane Begaudeau is currently offline Stephane Begaudeau
Messages: 406
Registered: April 2010
Location: Nantes (France)
Senior Member

Hi Niels,

I looked for M2T testing framework in the past and I didn't find anything great.
I realized a prototype of M2T testing framework for Acceleo: http://stephanebegaudeau.tumblr.com/post/2379262223/having-f un-with-junit-and-acceleo
It's just a small prototype created during a week end and there are no development on this right now but we are thinking about this problem too.

Stephane Begaudeau, Obeo

--
Twitter: @sbegaudeau
Acceleo wiki: http://wiki.eclipse.org/Acceleo
Blogs: http://stephanebegaudeau.tumblr.com & http://sbegaudeau.tumblr.com



Re: Best practice to test M2T transformation [message #658198 is a reply to message #656416] Mon, 07 March 2011 06:40 Go to previous message
Niels Brouwers is currently offline Niels Brouwers
Messages: 67
Registered: July 2009
Member
Hi Stephane,

wow, as far as I am concerned, you are definitely proceeding in the right direction. This is exactly what I thought of when thinking about how to test my generator. OK, to be honest: it is better than what I thought of Wink
I especially like the UI addition.

So, when will it be ready? Will it be part of the next Eclipse release? Wink
More seriously, embedding code generators in a professional organization definitely requires the ability to extensively test the generators. IMO it would be extremely useful if Acceleo would provide a framework to help the users with this. Your prototype seems like a good starting point.

If you are looking for beta testers, I will be happy to volunteer!

Cheers Niels.



Kind regards,
Niels Brouwers.
Previous Topic:acceleo - cast of variables
Next Topic:Bug in my code or in OCL
Goto Forum:
  


Current Time: Wed Aug 20 18:44:19 EDT 2014

Powered by FUDForum. Page generated in 0.01576 seconds