Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[paho-dev] MQTT Interoperability Tests

Hello all,

I'm in the process of developing some interoperability tests suites for the Interops Testing Day, and for general MQTT "conformance". It's taken a while longer to get started than I anticipated partly because I was waiting for the first public consultation draft of the OASIS MQTT specification to be ready.

These tests are based on CSD03 of the spec, which can be found here:

I've created a new Paho repository to hold the artifacts The first artifact in there is a spreadsheet of the conformance statements in the spec, along with an outline of tests needed for each. I anticipate adding some more statements to this list, as there are some more aspects that need to be tested than are included (such as longer payload lengths).

The short description of the approach I am taking is:

1. Python package mqtt.formats contains MQTT de/serialization and checking routines 2. package mqtt.client contains a minimal Python MQTT client implementation I use for ad-hoc checking of the broker 3. package contains a "model" Python broker implementation with conformance statements embedded in it, so that when one is encountered, it is logged 4. package mbt contains a test generation package. It works out dependencies of test inputs and remembers test sequences taken so they are not duplicated. Ideas

In the main directory,

a) will start the "model" broker. You can connect to it to see it working b) is just for playing with various client interactions with the broker c) is a client test "model". It does not use the mqtt.client package, but generates MQTT inputs to the broker, which then responds with expected results. These inputs and expected results are combined into tests.
d) will eventually generate a whole test suite.

To generate a whole test suite for brokers, the approach will be:

1. Use to generate test sequences and results.
2. Finish a test sequence when one or more conformance statements have been hit. Record that as one test. 3. Keep generating tests until all conformance statements have been encountered. 4. Prune the test suite by removing tests which found duplicate conformance statements.
5. Optionally use the Delta Debugging to shorten the test sequences.

Client libraries will also be able to use the model broker to test against. The tests generated above can be translated into abstract tests for clients using a notation something like cucumber (or behave for Python I haven't quite worked out that bit yet.

Oh - these are all Python 3 programs, not Python 2.

I will write a better description and documentation as I progress. Suggestions or questions welcome.


Back to the top