Hi Dejan,
it is intentional that suite_generate.py creates a different set of
tests each time. In theory, any generated test suite should pass
on any broker. I did intend to record and include a pre-generated
test suite or suites so that they could be just run. However, a
complication is that there are a handful of optional behaviours
which the specification allows. They can be configured in the test
broker and are:
INFO 20150108 150621 Optional behaviour, publish on pubrel: True
INFO 20150108 150621 Optional behaviour, single publish on
overlapping topics: True
INFO 20150108 150621 Optional behaviour, drop QoS 0 publications to
disconnected clients: True
INFO 20150108 150621 Optional behaviour, support zero length
clientids: True
so we would have to have 16 different test suites to cover all
variations of these.
At the moment, the test suites created by suite_generate.py achieve
about 62% coverage of conformance statements, and 16% coverage of
exceptions. What I would like to do is get those figures up to as
high as I can get them (>90%), and save those as pre-generated
test suites. To do this involves changing the input model MQTTV311_spec.py.
The next part of the plan is to create similar tests for client
libraries. To do this, I plan to generate the tests in Cucumber
format (http://cukes.info/), using a new input model (similar to
MQTTV311_spec.py). Then for any particular API a translation layer
will have to be written to allow the tests to be executed.
Ian
On 01/08/2015 11:27 AM, Dejan Bosanac
wrote:
Hi Ian,
it works much better on my laptop
now. Thanks a lot for these improvements.
I didn’t work on the tests as I
found a Linux box that runs them properly and was focused on
broker testing. It revealed couple of configuration changes
that were needed for them to pass. I still have couple of them
to work out.
I noticed that you now changed how
the test suite is generated. Each time I run suite_generate.py
I get different set of test files. Is this intentional? It
would be good to have a test suite we’re planning to use
stabilised soon, so people can focus on the product testing.
Thanks again for the fix, it will
make testing much easier. I also submitted one patch for
suite_generate.py script which should make it stop properly
(at least on osx).
_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/paho-dev
--
Ian Craggs
icraggs@xxxxxxxxxx IBM United Kingdom
Paho Project Lead; Committer on Mosquitto
|