| Re: [paho-dev] MQTT Interoperability Testing |
I think that would be great. I'd love to see these turned
into a full MQTT interop test.
As for timing, I tend to agree you want to start testing before the standard is finalized. You can also use the time to build and finish any test suites. One downside is that you would not be ‘certifying’ one implementation since the standard would not be final.
There is a solution to this, which is that the OASIS TC
can approve and publish any number of "Committee Specifications".
So for example, the implementations could be certified to support MQTT
vX.Z CS2, where CS2 indicates conformance with Committee Specification
2.
Paul
Ian
From: paho-dev-bounces@xxxxxxxxxxx
[mailto:paho-dev-bounces@xxxxxxxxxxx]
On Behalf Of Paul Fremantle
Sent: August-22-13 5:00 AM
To: General development discussions for paho project
Cc: paho-dev-bounces@xxxxxxxxxxx
Subject: Re: [paho-dev] MQTT Interoperability Testing
Dave
It is quite common to do interop testing *during* an OASIS TC's lifecycle rather than after. Interop testing is a great way of uncovering issues in the specification and unclarities. I remember an interesting case in WSRX where we had two ways of doing one thing (yes I know it seems a bad idea put that way!) and it was only during interop testing that a side effect of this became known.
So I think it would be good to do interop testing whenever we can. In addition, it takes some effort to produce an interop test suite and we should definitely get on with that.
In my experience there are two ways of doing that:
1) create a list of scenarios and then a matrix of implX<-->implY on scenario Z, so you end up with a three dimensional matrix where every cell is either a pass / fail / not tested. The scenarios need to be complete enough so that they cover all the conformance clauses.
2) create a independent test suite that tests all the conformance clauses of the spec.
They both have advantages and disadvantages. #2 is easier to test, more repeatable to test and more self contained. However it has two big disadvantages: firstly, it is a lot of upfront work independent of any implementation. Secondly it has to be very complete to really work, because its not actually testing any two real implementations together!
Paul
PS I think I probably should cross-post this to OASIS as well.
On 22 August 2013 09:35, Dave Locke <locke@xxxxxxxxxx> wrote:
like the idea with a caveat :-) If inter-operability
was to be based around the MQTT OASIS standard then the timing may make
not be practical. Good progress is being made at OASIS but it is not planned
te finish until early March 2014 (and could be later). At that point I
am not sure how many MQTT implementations will have been updated to meet
the standard. Is March 17-20 a little too early for an MQTT inter-op testing
hackathon?
All the best
Dave
From: "Ian
Skerrett" <ian.skerrett@xxxxxxxxxxx>
To: "General
development discussions for paho project" <paho-dev@xxxxxxxxxxx>
Date: 21/08/2013
20:49
Subject: [paho-dev]
MQTT Interoperability Testing
Sent by: paho-dev-bounces@xxxxxxxxxxx
All,
Is there any interest in the MQTT community to organize interoperability
testing with different implementations of MQTT clients and servers. It
seems there to be a growing list of open source projects and vendors that
are providing MQTT support so interoperability testing seems like a good
idea.
If there is interest, we could look at hosting an MQTT interoperability
testing hackathon at EclipseCon in March 17-20 in San Francisco.
Thoughts?
Ian
_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/paho-dev
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/paho-dev
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU