Hi


This is a standard research topic on which you can find many articles in the literature; try Googling "test model generator". However they tend to be metamodel blind and so take an unpalatably long time.

Using random models seems like a good idea, but is absolutely not what you want.

Mindlessly random test models will have a very low chance of hitting interesting test cases so that your testing will take forever and perhaps "find" nothing.

"find". Given a random test model how can you tell whether the result is good or bad? Ok; a smoke test is useful, but unless you have an Oracle to predict the correct output for comparison, or a sub-Oracle to check e.g. that size(Min+Nin) == size(Qout), your tests reveal little. If you are lucky, you have a golden brick reference implementation that you can compare against. More likely you have a flaky legacy implementation. There will always be tolerable differences so you will need an intelligent comparator. For my transformations, I try to ensure that they can be bidirectional so that I can do a round-trip test and so use the reverse transformation as an oracle for the forward and vice-versa. To accommodate acceptable differences I run the result and reference model through a normalizer that may zero optional fields, alphabeticise unordered collections etc to make comparison viable.

Intelligently random test models are the goal of much of the better research relying on the observation that the commonest errors are off-by-one or more generally recognizing that each test model is a point in the hyperspace of all possible models. The hyperspace is partitioned by hyperplanes each of which separates good from bad. Consequently an intelligently random test model will pick test models that are marginally good and marginally bad. While the hyperplane for e.g. "age >= 0" is easy to identify, the full set of hyperplanes is very challenging. IMHO this is where the tools go wrong attempting to reformulate the metamodels and OCL into the language of a SAT solver. My experience implementing a QVTr compiler convinces me that metamodels are very very restrictive consequently much of the arbitrary generality of SAT constraints is inappropriate for metamodels;the migration to SAT solvers requires the metamodel characteristics to be reformulated as additional problem dimensions rather than restrictive solutions. Anyway I have yet to see a test model generator that is not really really slow for big models. Of course you don't want big models for functionality testing. The minimum model that locates the marginal point in the hyperplane will do.



For my own purposes such as Fig 5 of [1] I want a family of structurally similar models ranging from 100 to 10,000,000 model elements so that I can see whether performance scales linearly or not. I absolutely do not want random models. A custom generator for structurally similar models is pretty easy to write for the small metamodels that my tests have instrumented so far. The models must be generated on the fly since the time and space to load a 10,000,000 element model may dwarf all useful testing. For larger metamodels you have the challenge of how the N elements are to be realized over perhaps 10 different metamodel multiplicities. If you allocate too much N to depth, your transformation may well hit a StackOverflow. If you allocate too much N to width you may find that you are just demonstrating that EMF performance for large Sets implemented by Lists is really disappointing.

Of course model elements are only part of the story. Nearly all models have Strings and Integers for which some values may be significant. e.g. if checking on URIs you need to have some string with sensitive %, \, $ combinations and ensure that you go through a transformation path that can detect the erroneous double encoding of a URI. The chances of any random model generator happening to generate a suitable String in a suitable context are infinitessimal.

I recommend a suite of judicious manual test models with a bidirectional Oracle. For Eclipse QVTo there is a test coverage tool that may help you assess your test models.
Regards

Ed Willink

[1] http://www.eclipse.org/mmt/qvt/docs/BigMDE2016/QVTcFirstResults.pdf