[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
Re: [paho-dev] MQTT Interoperability Testing
|
yep it is definitely of interest/ a good
idea. My initial reservation was solely on the timing in relation to the
timeline of the standard. Based on the email discussions I see no
problem with starting to create conformance tests once the spec is in reasonable
shape (maybe @ first public draft) and then evolve them as the spec is
finalised.
In summary would be good to have Paho:
- Host standards compliant MQTT clients
- Host conformance tests. The nature
of the conformance tests will need to be worked out. Initial thoughts are;-
to fully test a server for compliance will require wire level tests that
can drive any server (The MQTT clients do not allow full compliance of
a server to be validated). In addition client specific tests can
be created that will test features of the clients that the wire line tests
cannot test.
All the best
Dave
From:
Ian Craggs <icraggs@xxxxxxxxxxxxxxxxxxxxxxx>
To:
paho-dev@xxxxxxxxxxx
Date:
18/09/2013 16:14
Subject:
Re: [paho-dev]
MQTT Interoperability Testing
Sent by:
paho-dev-bounces@xxxxxxxxxxx
But that's only a "nice to have". If the
server at the other end doesn't understand, you can fall back to bog-standard
MQTT - you don't need more than that to make the basics work.
Ian
On 18/09/13 16:07, Andy Piper wrote:
http://mqtt.org/wiki/doku.php/bridge_protocol
On Wed, Sep 18, 2013 at 12:15 PM, Paul Fremantle <paul@xxxxxxxx>
wrote:
Ian
Is there any documentation on the bridging model in case
other servers wish to implement the same?
Paul
On 18 September 2013 10:53, Ian Craggs <icraggs@xxxxxxxxxxxxxxxxxxxxxxx>
wrote:
Ian,
we're definitely interested in an MQTT interoperability hackathon at EclipseCon
on March. Dave didn't mean to give the impression of being negative,
he tells me.
The Paho tests would be one place to start. As they stand, they would
check that a server implementation reacts the same as our test server to
the inputs from the Paho MQTT clients. Of course, right now, they
"conform" to the MQTT 3.1 spec.
I would hope to have the Mosquitto MQTT server project up and running by
then. Part of that will be the contribution of more specific server
tests, which we could also use.
Mosquitto and RSMB also have a bridging feature which allows MQTT servers
to be connected together. Any other servers supporting bridging could
be connected together, with some specific scenarios.
We also plan to have tests or test scenarios which correspond to the conformance
section of the OASIS specification, after it has been written.
Ian
On 22/08/13 13:39, Ian Skerrett wrote:
If we pursued option #2, is
it feasible to use Paho test suites as the starting point for interoperability
test suites?
As for timing, I tend to agree
you want to start testing before the standard is finalized. You can also
use the time to build and finish any test suites. One downside is that
you would not be ‘certifying’ one implementation since the standard would
not be final.
Ian
From: paho-dev-bounces@xxxxxxxxxxx
[mailto:paho-dev-bounces@xxxxxxxxxxx]
On Behalf Of Paul Fremantle
Sent: August-22-13 5:00 AM
To: General development discussions for paho project
Cc: paho-dev-bounces@xxxxxxxxxxx
Subject: Re: [paho-dev] MQTT Interoperability Testing
Dave
It is quite common to do interop testing *during* an OASIS
TC's lifecycle rather than after. Interop testing is a great way of uncovering
issues in the specification and unclarities. I remember an interesting
case in WSRX where we had two ways of doing one thing (yes I know it seems
a bad idea put that way!) and it was only during interop testing that a
side effect of this became known.
So I think it would be good to do interop testing whenever
we can. In addition, it takes some effort to produce an interop test suite
and we should definitely get on with that.
In my experience there are two ways of doing that:
1) create a list of scenarios and then a matrix of implX<-->implY
on scenario Z, so you end up with a three dimensional matrix where every
cell is either a pass / fail / not tested. The scenarios need to be complete
enough so that they cover all the conformance clauses.
2) create a independent test suite that tests all the conformance
clauses of the spec.
They both have advantages and disadvantages. #2 is easier
to test, more repeatable to test and more self contained. However it has
two big disadvantages: firstly, it is a lot of upfront work independent
of any implementation. Secondly it has to be very complete to really work,
because its not actually testing any two real implementations together!
Paul
PS I think I probably should cross-post this to OASIS as well.
On 22 August 2013 09:35, Dave Locke <locke@xxxxxxxxxx>
wrote:
like the idea with a caveat :-) If inter-operability
was to be based around the MQTT OASIS standard then the timing may make
not be practical. Good progress is being made at OASIS but it is not planned
te finish until early March 2014 (and could be later). At that point I
am not sure how many MQTT implementations will have been updated to meet
the standard. Is March 17-20 a little too early for an MQTT inter-op testing
hackathon?
All the best
Dave
From: "Ian
Skerrett" <ian.skerrett@xxxxxxxxxxx>
To: "General
development discussions for paho project" <paho-dev@xxxxxxxxxxx>
Date: 21/08/2013
20:49
Subject: [paho-dev]
MQTT Interoperability Testing
Sent by: paho-dev-bounces@xxxxxxxxxxx
All,
Is there any interest in the MQTT community to organize interoperability
testing with different implementations of MQTT clients and servers. It
seems there to be a growing list of open source projects and vendors that
are providing MQTT support so interoperability testing seems like a good
idea.
If there is interest, we could look at hosting an MQTT interoperability
testing hackathon at EclipseCon in March 17-20 in San Francisco.
Thoughts?
Ian
_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/paho-dev
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/paho-dev
--
Paul Fremantle
CTO and Co-Founder, WSO2
OASIS WS-RX TC Co-chair, Apache Member
UK: +44
207 096 0336
US: +1
646 595 7614
blog: http://pzf.fremantle.org
twitter.com/pzfreo
paul@xxxxxxxx
wso2.com
Lean Enterprise Middleware
Disclaimer: This communication may contain privileged or other confidential
information and is intended exclusively for the addressee/s. If you are
not the intended recipient/s, or believe that you may have received this
communication in error, please reply to the sender indicating that fact
and delete the copy you received and in addition, you should not print,
copy, retransmit, disseminate, or otherwise use the information contained
in this communication. Internet communications cannot be guaranteed to
be timely, secure, error or virus-free. The sender does not accept liability
for any errors or omissions.
_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/paho-dev
_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/paho-dev
--
Paul Fremantle
CTO and Co-Founder, WSO2
OASIS WS-RX TC Co-chair, Apache Member
UK: +44
207 096 0336
US: +1
646 595 7614
blog: http://pzf.fremantle.org
twitter.com/pzfreo
paul@xxxxxxxx
wso2.com
Lean Enterprise Middleware
Disclaimer: This communication may contain privileged or other confidential
information and is intended exclusively for the addressee/s. If you are
not the intended recipient/s, or believe that you may have received this
communication in error, please reply to the sender indicating that fact
and delete the copy you received and in addition, you should not print,
copy, retransmit, disseminate, or otherwise use the information contained
in this communication. Internet communications cannot be guaranteed to
be timely, secure, error or virus-free. The sender does not accept liability
for any errors or omissions.
_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/paho-dev
--
Andy Piper | Kingston upon Thames, London (UK)
blog: http://andypiper.co.uk
| skype: andypiperuk
twitter: @andypiper | images: http://www.flickr.com/photos/andypiper
--
Andy Piper | Kingston upon Thames, London (UK)
blog: http://andypiper.co.uk
| skype: andypiperuk
twitter: @andypiper | images: http://www.flickr.com/photos/andypiper
_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/paho-dev
_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
https://dev.eclipse.org/mailman/listinfo/paho-dev
Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU