| Re: [paho-dev] Interop test questions |
Hi Dejan,
I agree that dealing with failures can be harder. The payoff is having a more complete and maintainable test suite.
However, I have thoughts about making problem determination easier. The main one is to use a binary chop sort of approach "delta-debugging" (https://www.st.cs.uni-saarland.de/dd/) to produce the shortest test sequence which will reproduce a failure. This often will lead to an explanation of why the problem occurred.
If a server failure caused the test failure, then the server problem determination facilities ought to be able to pinpoint the cause (ideally :-).
Ian
On 01/08/2015 05:30 PM, Dejan Bosanac wrote:
Hi Ian,
sounds good. The only thing is that it makes a bit harder for folks to focus on the test failures in this phase (as it adds randomness). The one thing that I observed is that it’s a bit hard to deal with failures in general, as it takes an effort to narrow things down and find what exactly led to the failure. That’s probably something we can work on in the future.
On Thu, Jan 8, 2015 at 4:14 PM, Ian Craggs <icraggs@xxxxxxxxxxxxxxxxxxxxxxx> wrote:
Hi Dejan,
it is intentional that suite_generate.py creates a different set of tests each time. In theory, any generated test suite should pass on any broker. I did intend to record and include a pre-generated test suite or suites so that they could be just run. However, a complication is that there are a handful of optional behaviours which the specification allows. They can be configured in the test broker and are:
INFO 20150108 150621 Optional behaviour, publish on pubrel: True
INFO 20150108 150621 Optional behaviour, single publish on overlapping topics: True
INFO 20150108 150621 Optional behaviour, drop QoS 0 publications to disconnected clients: True
INFO 20150108 150621 Optional behaviour, support zero length clientids: True
so we would have to have 16 different test suites to cover all variations of these.
At the moment, the test suites created by suite_generate.py achieve about 62% coverage of conformance statements, and 16% coverage of exceptions. What I would like to do is get those figures up to as high as I can get them (>90%), and save those as pre-generated test suites. To do this involves changing the input model MQTTV311_spec.py.
The next part of the plan is to create similar tests for client libraries. To do this, I plan to generate the tests in Cucumber format (http://cukes.info/), using a new input model (similar to MQTTV311_spec.py). Then for any particular API a translation layer will have to be written to allow the tests to be executed.
Ian
On 01/08/2015 11:27 AM, Dejan Bosanac wrote:
Hi Ian,
it works much better on my laptop now. Thanks a lot for these improvements.
I didn’t work on the tests as I found a Linux box that runs them properly and was focused on broker testing. It revealed couple of configuration changes that were needed for them to pass. I still have couple of them to work out.
I noticed that you now changed how the test suite is generated. Each time I run suite_generate.py I get different set of test files. Is this intentional? It would be good to have a test suite we’re planning to use stabilised soon, so people can focus on the product testing.
Thanks again for the fix, it will make testing much easier. I also submitted one patch for suite_generate.py script which should make it stop properly (at least on osx).
On Mon, Jan 5, 2015 at 12:15 PM, Ian Craggs <icraggs@xxxxxxxxxxxxxxxxxxxxxxx> wrote:
Hi Dejan,
I've just checked in a change to address this problem, and handily, some others too :-)
Instead of serializing the socket object, I've replaced that with a sequence number, which is just the order in which the sockets are created. This makes the test log easier to follow, and should make the tests platform independent. The tests don't rely on any internal details of the sockets, so this solution should work fine.
Let me know what you think.
Ian
On 12/17/2014 01:18 PM, Dejan Bosanac wrote:
Hi,
I started playing with mqtt interop test suite. I ran into some problems in how we serialize socket between steps.
On my system (OSX), the socket object serialized to log file looks like
<socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketType.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 51240), raddr=('127.0.0.1', 1883)>
There are two problems with it:
1. it can’t be easily “matched” against results as laddr property is different on every new socket.
2. to must be put in quotes to be able to eval it
I think we need to find a better way to pass the socket object between the steps and verify it’s in proper state. I’m not sure of the proper solution yet, so just wanted to share it to start a discussion around it.Regards
--
Dejan Bosanac
----------------------
Red Hat, Inc.
dbosanac@xxxxxxxxxx
Twitter: @dejanb
Blog: http://sensatic.net
ActiveMQ in Action: http://www.manning.com/snyder/
_______________________________________________ paho-dev mailing list paho-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/paho-dev
-- Ian Craggs icraggs@xxxxxxxxxx IBM United Kingdom Paho Project Lead; Committer on Mosquitto
_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/paho-dev
_______________________________________________ paho-dev mailing list paho-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/paho-dev
-- Ian Craggs icraggs@xxxxxxxxxx IBM United Kingdom Paho Project Lead; Committer on Mosquitto
_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/paho-dev
_______________________________________________ paho-dev mailing list paho-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/paho-dev
-- Ian Craggs icraggs@xxxxxxxxxx IBM United Kingdom Paho Project Lead; Committer on Mosquitto
_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/paho-dev