[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
[paho-dev] MQTT Interoperability Tests
|
Hello all,
I'm in the process of developing some interoperability tests suites for
the Interops Testing Day, and for general MQTT "conformance". It's taken
a while longer to get started than I anticipated partly because I was
waiting for the first public consultation draft of the OASIS MQTT
specification to be ready.
These tests are based on CSD03 of the spec, which can be found here:
https://www.oasis-open.org/committees/documents.php?wg_abbrev=mqtt.
I've created a new Paho repository to hold the artifacts
http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.testing.git/. The
first artifact in there is a spreadsheet of the conformance statements
in the spec, along with an outline of tests needed for each. I
anticipate adding some more statements to this list, as there are some
more aspects that need to be tested than are included (such as longer
payload lengths).
The short description of the approach I am taking is:
1. Python package mqtt.formats contains MQTT de/serialization and
checking routines
2. package mqtt.client contains a minimal Python MQTT client
implementation I use for ad-hoc checking of the broker
3. package mqtt.broker contains a "model" Python broker implementation
with conformance statements embedded in it, so that when one is
encountered, it is logged
4. package mbt contains a test generation package. It works out
dependencies of test inputs and remembers test sequences taken so they
are not duplicated. Ideas
In the main directory,
a) startbroker.py will start the "model" broker. You can connect to it
to see it working
b) client_test.py is just for playing with various client interactions
with the broker
c) MQTTV311.py is a client test "model". It does not use the
mqtt.client package, but generates MQTT inputs to the broker, which then
responds with expected results. These inputs and expected results are
combined into tests.
d) suite_generate.py will eventually generate a whole test suite.
To generate a whole test suite for brokers, the approach will be:
1. Use MQTTV311.py to generate test sequences and results.
2. Finish a test sequence when one or more conformance statements have
been hit. Record that as one test.
3. Keep generating tests until all conformance statements have been
encountered.
4. Prune the test suite by removing tests which found duplicate
conformance statements.
5. Optionally use the Delta Debugging to shorten the test sequences.
Client libraries will also be able to use the model broker to test
against. The tests generated above can be translated into abstract
tests for clients using a notation something like cucumber (or behave
for Python http://pythonhosted.org/behave/index.html). I haven't quite
worked out that bit yet.
Oh - these are all Python 3 programs, not Python 2.
I will write a better description and documentation as I progress.
Suggestions or questions welcome.
Ian
