Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [paho-dev] MQTT Conformance/Interoperability Testing status
As an addendum to that, I need to update the documentation page to the current state of the test suite. The approach you are using was meant as a stop gap until the proper, full test suite was finished.
I think it's in reasonable condition, but I wanted to improve the search algorithm (the tests are generated) to improve the coverage before declaring success for the server tests. Your question has given me an incentive to look at this again :-) 

Ian

On 07/15/2015 09:35 PM, Nicolas Jouanin wrote:

Hi,

Just tested on a private instance, everything is fine. Sorry about that.

$ python3 client_test.py —hostname=localhost -p 1883
clean up starting
clean up finished
Basic test starting
Basic test succeeded
Retained message test starting
Retained message test succeeded
This server is not queueing QoS 0 messages for offline clients
Offline message queueing test succeeded
Will message test succeeded
Overlapping subscriptions test starting
This server is publishing one message for all matching overlapping subscriptions, not one for each.
Overlapping subscriptions test succeeded
Keepalive test starting
Keepalive test succeeded
Redelivery on reconnect test starting
Redelivery on reconnect test succeeded
test suite succeeded



Le 15 juil. 2015 à 22:27, Roger Light <roger@xxxxxxxxxx> a écrit :

Hi Nicolas,

I've replied on irc, but for others - the answer for those errors is
that you can't run the test suite on a broker that is doing other
tasks. Run a local copy and test against it.

I can't comment on how complete the tests are.

Cheers,

Roger


On Wed, Jul 15, 2015 at 8:59 PM, Nicolas Jouanin <nico@xxxxxxxxxxxxxxx> wrote:

Hi,

I was wondering what was the current status of broker testing suite. I mean is it currently reliable for a broker compliance testing ?
I’d like to use it to validate my own broker implementation, but some tests are failing. Also I’ve tried to run the conformance tests on test.mosquito.org and it also fails on some test (see results below).


python3 client_test.py --hostname=test.mosquitto.org -p 1883
hostname test.mosquitto.org port 1883
clean up starting
ERROR:root:receive: unexpected exception (<class 'UnicodeDecodeError'>, UnicodeDecodeError('utf-8', b'/v1/p0/atmeld/aeef/stat\x9d', 23, 24, 'invalid start byte'), <traceback object at 0x104826208>)
Traceback (most recent call last):
  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/client/internal.py", line 44, in receive
    packet = MQTTV3.unpackPacket(MQTTV3.getPacket(self.socket))
  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/formats/MQTTV311.py", line 892, in unpackPacket
    packet.unpack(buffer)
  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/formats/MQTTV311.py", line 460, in unpack
    self.topicName = readUTF(buffer[fhlen:], packlen - curlen)
  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/formats/MQTTV311.py", line 178, in readUTF
    buf = buffer[2:2+length].decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9d in position 23: invalid start byte
ERROR:root:call: unexpected exception (<class 'UnicodeDecodeError'>, UnicodeDecodeError('utf-8', b'/v1/p0/atmeld/aeef/stat\x9d', 23, 24, 'invalid start byte'), <traceback object at 0x104826248>)
Traceback (most recent call last):
  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/client/internal.py", line 155, in __call__
    self.receive(callback)
  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/client/internal.py", line 44, in receive
    packet = MQTTV3.unpackPacket(MQTTV3.getPacket(self.socket))
  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/formats/MQTTV311.py", line 892, in unpackPacket
    packet.unpack(buffer)
  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/formats/MQTTV311.py", line 460, in unpack
    self.topicName = readUTF(buffer[fhlen:], packlen - curlen)
  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/formats/MQTTV311.py", line 178, in readUTF
    buf = buffer[2:2+length].decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9d in position 23: invalid start byte
deleting retained message for topic frl/2130
deleting retained message for topic /ohotkuba/19791981jh
deleting retained message for topic /ohotkuba/19791981ih
deleting retained message for topic /casa/corredor/luz
deleting retained message for topic /casa/sala/controlador/luz
deleting retained message for topic /casa/sala/luz/cor
deleting retained message for topic /casa/sala/slider
deleting retained message for topic /casa/jardim/luz
deleting retained message for topic /casa/garagem/luz
deleting retained message for topic /Schlenz/AndroidCI
deleting retained message for topic /Schlenz/WebCI
deleting retained message for topic /Schlenz/iOSCI
deleting retained message for topic /jolicode/coffee_pot
deleting retained message for topic /node0/dht0/temperature
deleting retained message for topic /node0/dht0/humidity
deleting retained message for topic /node0/gas0/pollution
deleting retained message for topic /dcc/jodo
deleting retained message for topic /_mytopic/date
deleting retained message for topic /openhab/in/Date/state
deleting retained message for topic /openhab/in/Weather_Temperature/state
deleting retained message for topic /openhab/in/Light_GF_Living_Table/state
deleting retained message for topic /openhab/in/Light_GF_Corridor_Ceiling/state
deleting retained message for topic /openhab/in/Light_GF_Kitchen_Ceiling/state
deleting retained message for topic /openhab/in/Light_GF_Kitchen_Table/state
deleting retained message for topic /openhab/in/Light_GF_Corridor_Wardrobe/state
deleting retained message for topic /openhab/in/Light_GF_Toilet_Ceiling/state
deleting retained message for topic /openhab/in/Light_GF_Toilet_Mirror/state
deleting retained message for topic /openhab/in/Light_FF_Bath_Ceiling/state
deleting retained message for topic /openhab/in/Light_FF_Bath_Mirror/state
deleting retained message for topic /openhab/in/Light_FF_Corridor_Ceiling/state
deleting retained message for topic /openhab/in/Light_FF_Office_Ceiling/state
deleting retained message for topic /openhab/in/Light_FF_Child_Ceiling/state
deleting retained message for topic /openhab/in/Light_FF_Bed_Ceiling/state
deleting retained message for topic /openhab/in/Light_C_Corridor_Ceiling/state
deleting retained message for topic /openhab/in/Light_C_Staircase/state
deleting retained message for topic /openhab/in/Light_C_Washing_Ceiling/state
deleting retained message for topic /openhab/in/Light_C_Workshop/state
deleting retained message for topic /openhab/in/Light_Outdoor_Garage/state
deleting retained message for topic /openhab/in/Light_Outdoor_Terrace/state
deleting retained message for topic /openhab/in/Light_Outdoor_Frontdoor/state
deleting retained message for topic /openhab/in/Heating_GF_Corridor/state
deleting retained message for topic /openhab/in/Heating_GF_Toilet/state
deleting retained message for topic /openhab/in/Heating_GF_Living/state
deleting retained message for topic /openhab/in/Heating_GF_Kitchen/state
deleting retained message for topic /openhab/in/Heating_FF_Bath/state
deleting retained message for topic /openhab/in/Heating_FF_Office/state
deleting retained message for topic /openhab/in/Heating_FF_Child/state
deleting retained message for topic /openhab/in/Heating_FF_Bed/state
deleting retained message for topic /openhab/in/Temperature_Setpoint/state
deleting retained message for topic /openhab/in/Window_GF_Frontdoor/state
deleting retained message for topic /openhab/in/Window_GF_Kitchen/state
deleting retained message for topic /openhab/in/Window_GF_Living/state
deleting retained message for topic /openhab/in/Window_GF_Toilet/state
deleting retained message for topic /openhab/in/Window_FF_Bath/state
deleting retained message for topic /openhab/in/Window_FF_Bed/state
deleting retained message for topic /openhab/in/Window_FF_Office_Window/state
deleting retained message for topic /openhab/in/Window_FF_Office_Door/state
deleting retained message for topic /openhab/in/Garage_Door/state
deleting retained message for topic /openhab/in/Temperature_GF_Corridor/state
deleting retained message for topic /openhab/in/Temperature_GF_Toilet/state
deleting retained message for topic /openhab/in/Temperature_GF_Living/state
deleting retained message for topic /openhab/in/Temperature_GF_Kitchen/state
deleting retained message for topic /openhab/in/Temperature_FF_Bath/state
deleting retained message for topic /openhab/in/Temperature_FF_Office/state
deleting retained message for topic /openhab/in/Temperature_FF_Child/state
deleting retained message for topic /openhab/in/Temperature_FF_Bed/state
deleting retained message for topic /openhab/in/Weather_Temp_Max/state
deleting retained message for topic /openhab/in/Weather_Temp_Min/state
deleting retained message for topic /home
deleting retained message for topic /home/temperature
deleting retained message for topic /openHABBruegge/out/Slim_motion/state
deleting retained message for topic /openHABBruegge/out/Slim_temp/state
deleting retained message for topic /openHABBruegge/out/Slim_door/state
deleting retained message for topic /openHABBruegge/out/Slim_battery/state
deleting retained message for topic /v1/p0/starling/3810/status
deleting retained message for topic /v1/p0/buyAtmel/aeef/status
deleting retained message for topic /v1/p0/atmeld/aeef/status
clean up finished
Basic test starting
Basic test succeeded
Retained message test starting
Traceback (most recent call last):
  File "client_test.py", line 142, in retained_message_test
    aclient.disconnect()
  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/client/main.py", line 187, in disconnect
    assert self.__receiver.inMsgs == {}, self.__receiver.inMsgs
AssertionError: {47: Publishes(DUP=False, QoS=2, Retain=True, MsgId=47, TopicName='PREFIX/DeviceList', Payload=b'{\n   "2,1" : {\n      "Description" : "N/A",\n      "Manufacturer" : "Sogeti HT",\n      "Name" : "SignalGenerator"\n   }\n}\n'), 48: Publishes(DUP=False, QoS=2, Retain=True, MsgId=48, TopicName='SETest/DeviceList', Payload=b'{\n   "6,1" : {\n      "Description" : "N/A",\n      "Manufacturer" : "Sogeti HT",\n      "Name" : "SignalGenerator"\n   }\n}\n'), 51: Publishes(DUP=False, QoS=2, Retain=True, MsgId=51, TopicName='hello/world', Payload=b'yooooooooooooooooooo'), 52: Publishes(DUP=False, QoS=2, Retain=True, MsgId=52, TopicName='hello/emqtt', Payload=b'Example: QoS 2'), 53: Publishes(DUP=False, QoS=2, Retain=True, MsgId=53, TopicName='hello/1234', Payload=b'rep'), 54: Publishes(DUP=False, QoS=2, Retain=True, MsgId=54, TopicName='hello/1126', Payload=b'7c:dd:90:78:51:23'), 55: Publishes(DUP=False, QoS=2, Retain=True, MsgId=55, TopicName='hello/27137', Payload=b'7c:dd:90:78:51:23'), 56: Publishes(DUP=False, QoS=2, Retain=True, MsgId=56, TopicName='hello/4457', Payload=b'7c:dd:90:78:51:23'), 57: Publishes(DUP=False, QoS=2, Retain=True, MsgId=57, TopicName='hello/21670', Payload=b'7c:dd:90:78:51:23'), 58: Publishes(DUP=False, QoS=2, Retain=True, MsgId=58, TopicName='hello/21622', Payload=b'7c:dd:90:78:51:23')}
Retained message test failed
Traceback (most recent call last):
  File "client_test.py", line 218, in offline_message_queueing_test
    aclient.disconnect()
  File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/client/main.py", line 187, in disconnect
    assert self.__receiver.inMsgs == {}, self.__receiver.inMsgs
AssertionError: {47: Publishes(DUP=False, QoS=2, Retain=True, MsgId=47, TopicName='PREFIX/DeviceList', Payload=b'{\n   "2,1" : {\n      "Description" : "N/A",\n      "Manufacturer" : "Sogeti HT",\n      "Name" : "SignalGenerator"\n   }\n}\n'), 48: Publishes(DUP=False, QoS=2, Retain=True, MsgId=48, TopicName='SETest/DeviceList', Payload=b'{\n   "6,1" : {\n      "Description" : "N/A",\n      "Manufacturer" : "Sogeti HT",\n      "Name" : "SignalGenerator"\n   }\n}\n'), 51: Publishes(DUP=False, QoS=2, Retain=True, MsgId=51, TopicName='hello/world', Payload=b'yooooooooooooooooooo'), 52: Publishes(DUP=False, QoS=2, Retain=True, MsgId=52, TopicName='hello/emqtt', Payload=b'Example: QoS 2'), 53: Publishes(DUP=False, QoS=2, Retain=True, MsgId=53, TopicName='hello/1234', Payload=b'rep'), 54: Publishes(DUP=False, QoS=2, Retain=True, MsgId=54, TopicName='hello/1126', Payload=b'7c:dd:90:78:51:23'), 55: Publishes(DUP=False, QoS=2, Retain=True, MsgId=55, TopicName='hello/27137', Payload=b'7c:dd:90:78:51:23'), 56: Publishes(DUP=False, QoS=2, Retain=True, MsgId=56, TopicName='hello/4457', Payload=b'7c:dd:90:78:51:23'), 57: Publishes(DUP=False, QoS=2, Retain=True, MsgId=57, TopicName='hello/21670', Payload=b'7c:dd:90:78:51:23'), 58: Publishes(DUP=False, QoS=2, Retain=True, MsgId=58, TopicName='hello/21622', Payload=b'7c:dd:90:78:51:23')}
Offline message queueing test failed
Traceback (most recent call last):
  File "client_test.py", line 178, in will_message_test
    assert len(callback2.messages) == 1, callback2.messages  # should have the will message
AssertionError: [('Topic/C', b'qos 1', 1, True, 1), ('Topic/C', b'client not disconnected', 2, False, 2)]
Will message test failed
Overlapping subscriptions test starting
Traceback (most recent call last):
  File "client_test.py", line 253, in overlapping_subscriptions_test
    assert len(callback.messages) in [1, 2]
AssertionError
Overlapping subscriptions test failed
Keepalive test starting
Keepalive test succeeded
Redelivery on reconnect test starting
Traceback (most recent call last):
  File "client_test.py", line 309, in redelivery_on_reconnect_test
    assert len(callback2.messages) == 2, "length should be 2: %s" % callback2.messages
AssertionError: length should be 2: [('TopicA/B', b'', 1, False, 2), ('TopicA/C', b'qos 2', 2, True, 1), ('TopicA/C', b'', 2, False, 3)]
Redelivery on reconnect test failed
test suite failed
_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/paho-dev

_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/paho-dev

_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/paho-dev


--
Ian Craggs
icraggs@xxxxxxxxxx                 IBM United Kingdom
Paho Project Lead; Committer on Mosquitto

Back to the top