Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [paho-dev] MQTT Interoperability Testing

Excellent. I think it will be really important for the MQTT community to have this type of interoperability testing.


Can I suggest that on one of the next Paho project calls we add MQTT interop testing to the agenda. Anyone interested in helping would be invited to attend. I don’t know when the regular Paho calls are held so if someone can suggest a time that would be great.






From: Ian Craggs [mailto:icraggs@xxxxxxxxxxxxxxxxxxxxxxx]
Sent: September-18-13 11:53 AM
To: General development discussions for paho project
Cc: Ian Skerrett; paho-dev-bounces@xxxxxxxxxxx
Subject: Re: [paho-dev] MQTT Interoperability Testing



we're definitely interested in an MQTT interoperability hackathon at EclipseCon on March.   Dave didn't mean to give the impression of being negative, he tells me.

The Paho tests would be one place to start.  As they stand, they would check that a server implementation reacts the same as our test server to the inputs from the Paho MQTT clients.  Of course, right now, they "conform" to the MQTT 3.1 spec.

I would hope to have the Mosquitto MQTT server project up and running by then.  Part of that will be the contribution of more specific server tests, which we could also use.

Mosquitto and RSMB also have a bridging feature which allows MQTT servers to be connected together.  Any other servers supporting bridging could be connected together, with some specific scenarios.

We also plan to have tests or test scenarios which correspond to the conformance section of the OASIS specification, after it has been written.


On 22/08/13 13:39, Ian Skerrett wrote:

If we pursued option #2, is it feasible to use Paho test suites as the starting point for interoperability test suites?


As for timing, I tend to agree you want to start testing before the standard is finalized. You can also use the time to build and finish any test suites. One downside is that you would not be ‘certifying’ one implementation since the standard would not be final.






From: paho-dev-bounces@xxxxxxxxxxx [mailto:paho-dev-bounces@xxxxxxxxxxx] On Behalf Of Paul Fremantle
Sent: August-22-13 5:00 AM
To: General development discussions for paho project
Cc: paho-dev-bounces@xxxxxxxxxxx
Subject: Re: [paho-dev] MQTT Interoperability Testing




It is quite common to do interop testing *during* an OASIS TC's lifecycle rather than after. Interop testing is a great way of uncovering issues in the specification and unclarities. I remember an interesting case in WSRX where we had two ways of doing one thing (yes I know it seems a bad idea put that way!) and it was only during interop testing that a side effect of this became known.


So I think it would be good to do interop testing whenever we can. In addition, it takes some effort to produce an interop test suite and we should definitely get on with that. 


In my experience there are two ways of doing that:

1) create a list of scenarios and then a matrix of implX<-->implY on scenario Z, so you end up with a three dimensional matrix where every cell is either a pass / fail / not tested. The scenarios need to be complete enough so that they cover all the conformance clauses.

2) create a independent test suite that tests all the conformance clauses of the spec. 


They both have advantages and disadvantages. #2 is easier to test, more repeatable to test and more self contained. However it has two big disadvantages: firstly, it is a lot of upfront work independent of any implementation. Secondly it has to be very complete to really work, because its not actually testing any two real implementations together! 



PS I think I probably should cross-post this to OASIS as well.


On 22 August 2013 09:35, Dave Locke <locke@xxxxxxxxxx> wrote:

like the idea with a caveat :-) If inter-operability was to be based around the MQTT OASIS standard then the timing may make not be practical. Good progress is being made at OASIS but it is not planned te finish until early March 2014 (and could be later). At that point I am not sure how many MQTT implementations will have been updated to meet the standard. Is March 17-20 a little too early for an MQTT inter-op testing hackathon?

All the best


From:        "Ian Skerrett" <ian.skerrett@xxxxxxxxxxx>
To:        "General development discussions for paho project" <paho-dev@xxxxxxxxxxx>
Date:        21/08/2013 20:49
Subject:        [paho-dev] MQTT Interoperability Testing
Sent by:        paho-dev-bounces@xxxxxxxxxxx

Is there any interest in the MQTT community to organize interoperability testing with different implementations of MQTT clients and servers. It seems there to be a growing list of open source projects and vendors that are providing MQTT support so interoperability testing seems like a good idea.
If there is interest, we could look at hosting an MQTT interoperability testing hackathon at EclipseCon in March 17-20 in San Francisco.


paho-dev mailing list

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

paho-dev mailing list



Paul Fremantle
CTO and Co-Founder, WSO2
OASIS WS-RX TC Co-chair, Apache Member

UK: +44 207 096 0336
US: +1 646 595 7614

paul@xxxxxxxx Lean Enterprise Middleware

Disclaimer: This communication may contain privileged or other confidential information and is intended exclusively for the addressee/s. If you are not the intended recipient/s, or believe that you may have received this communication in error, please reply to the sender indicating that fact and delete the copy you received and in addition, you should not print, copy, retransmit, disseminate, or otherwise use the information contained in this communication. Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions.

paho-dev mailing list


Back to the top