[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [paho-dev] MQTT Conformance/Interoperability Testing status
|
Hi Nicolas,
I've replied on irc, but for others - the answer for those errors is
that you can't run the test suite on a broker that is doing other
tasks. Run a local copy and test against it.
I can't comment on how complete the tests are.
Cheers,
Roger
On Wed, Jul 15, 2015 at 8:59 PM, Nicolas Jouanin <nico@xxxxxxxxxxxxxxx> wrote:
> Hi,
>
> I was wondering what was the current status of broker testing suite. I mean is it currently reliable for a broker compliance testing ?
> I’d like to use it to validate my own broker implementation, but some tests are failing. Also I’ve tried to run the conformance tests on test.mosquito.org and it also fails on some test (see results below).
>
>
> python3 client_test.py --hostname=test.mosquitto.org -p 1883
> hostname test.mosquitto.org port 1883
> clean up starting
> ERROR:root:receive: unexpected exception (<class 'UnicodeDecodeError'>, UnicodeDecodeError('utf-8', b'/v1/p0/atmeld/aeef/stat\x9d', 23, 24, 'invalid start byte'), <traceback object at 0x104826208>)
> Traceback (most recent call last):
> File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/client/internal.py", line 44, in receive
> packet = MQTTV3.unpackPacket(MQTTV3.getPacket(self.socket))
> File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/formats/MQTTV311.py", line 892, in unpackPacket
> packet.unpack(buffer)
> File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/formats/MQTTV311.py", line 460, in unpack
> self.topicName = readUTF(buffer[fhlen:], packlen - curlen)
> File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/formats/MQTTV311.py", line 178, in readUTF
> buf = buffer[2:2+length].decode("utf-8")
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9d in position 23: invalid start byte
> ERROR:root:call: unexpected exception (<class 'UnicodeDecodeError'>, UnicodeDecodeError('utf-8', b'/v1/p0/atmeld/aeef/stat\x9d', 23, 24, 'invalid start byte'), <traceback object at 0x104826248>)
> Traceback (most recent call last):
> File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/client/internal.py", line 155, in __call__
> self.receive(callback)
> File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/client/internal.py", line 44, in receive
> packet = MQTTV3.unpackPacket(MQTTV3.getPacket(self.socket))
> File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/formats/MQTTV311.py", line 892, in unpackPacket
> packet.unpack(buffer)
> File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/formats/MQTTV311.py", line 460, in unpack
> self.topicName = readUTF(buffer[fhlen:], packlen - curlen)
> File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/formats/MQTTV311.py", line 178, in readUTF
> buf = buffer[2:2+length].decode("utf-8")
> UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9d in position 23: invalid start byte
> deleting retained message for topic frl/2130
> deleting retained message for topic /ohotkuba/19791981jh
> deleting retained message for topic /ohotkuba/19791981ih
> deleting retained message for topic /casa/corredor/luz
> deleting retained message for topic /casa/sala/controlador/luz
> deleting retained message for topic /casa/sala/luz/cor
> deleting retained message for topic /casa/sala/slider
> deleting retained message for topic /casa/jardim/luz
> deleting retained message for topic /casa/garagem/luz
> deleting retained message for topic /Schlenz/AndroidCI
> deleting retained message for topic /Schlenz/WebCI
> deleting retained message for topic /Schlenz/iOSCI
> deleting retained message for topic /jolicode/coffee_pot
> deleting retained message for topic /node0/dht0/temperature
> deleting retained message for topic /node0/dht0/humidity
> deleting retained message for topic /node0/gas0/pollution
> deleting retained message for topic /dcc/jodo
> deleting retained message for topic /_mytopic/date
> deleting retained message for topic /openhab/in/Date/state
> deleting retained message for topic /openhab/in/Weather_Temperature/state
> deleting retained message for topic /openhab/in/Light_GF_Living_Table/state
> deleting retained message for topic /openhab/in/Light_GF_Corridor_Ceiling/state
> deleting retained message for topic /openhab/in/Light_GF_Kitchen_Ceiling/state
> deleting retained message for topic /openhab/in/Light_GF_Kitchen_Table/state
> deleting retained message for topic /openhab/in/Light_GF_Corridor_Wardrobe/state
> deleting retained message for topic /openhab/in/Light_GF_Toilet_Ceiling/state
> deleting retained message for topic /openhab/in/Light_GF_Toilet_Mirror/state
> deleting retained message for topic /openhab/in/Light_FF_Bath_Ceiling/state
> deleting retained message for topic /openhab/in/Light_FF_Bath_Mirror/state
> deleting retained message for topic /openhab/in/Light_FF_Corridor_Ceiling/state
> deleting retained message for topic /openhab/in/Light_FF_Office_Ceiling/state
> deleting retained message for topic /openhab/in/Light_FF_Child_Ceiling/state
> deleting retained message for topic /openhab/in/Light_FF_Bed_Ceiling/state
> deleting retained message for topic /openhab/in/Light_C_Corridor_Ceiling/state
> deleting retained message for topic /openhab/in/Light_C_Staircase/state
> deleting retained message for topic /openhab/in/Light_C_Washing_Ceiling/state
> deleting retained message for topic /openhab/in/Light_C_Workshop/state
> deleting retained message for topic /openhab/in/Light_Outdoor_Garage/state
> deleting retained message for topic /openhab/in/Light_Outdoor_Terrace/state
> deleting retained message for topic /openhab/in/Light_Outdoor_Frontdoor/state
> deleting retained message for topic /openhab/in/Heating_GF_Corridor/state
> deleting retained message for topic /openhab/in/Heating_GF_Toilet/state
> deleting retained message for topic /openhab/in/Heating_GF_Living/state
> deleting retained message for topic /openhab/in/Heating_GF_Kitchen/state
> deleting retained message for topic /openhab/in/Heating_FF_Bath/state
> deleting retained message for topic /openhab/in/Heating_FF_Office/state
> deleting retained message for topic /openhab/in/Heating_FF_Child/state
> deleting retained message for topic /openhab/in/Heating_FF_Bed/state
> deleting retained message for topic /openhab/in/Temperature_Setpoint/state
> deleting retained message for topic /openhab/in/Window_GF_Frontdoor/state
> deleting retained message for topic /openhab/in/Window_GF_Kitchen/state
> deleting retained message for topic /openhab/in/Window_GF_Living/state
> deleting retained message for topic /openhab/in/Window_GF_Toilet/state
> deleting retained message for topic /openhab/in/Window_FF_Bath/state
> deleting retained message for topic /openhab/in/Window_FF_Bed/state
> deleting retained message for topic /openhab/in/Window_FF_Office_Window/state
> deleting retained message for topic /openhab/in/Window_FF_Office_Door/state
> deleting retained message for topic /openhab/in/Garage_Door/state
> deleting retained message for topic /openhab/in/Temperature_GF_Corridor/state
> deleting retained message for topic /openhab/in/Temperature_GF_Toilet/state
> deleting retained message for topic /openhab/in/Temperature_GF_Living/state
> deleting retained message for topic /openhab/in/Temperature_GF_Kitchen/state
> deleting retained message for topic /openhab/in/Temperature_FF_Bath/state
> deleting retained message for topic /openhab/in/Temperature_FF_Office/state
> deleting retained message for topic /openhab/in/Temperature_FF_Child/state
> deleting retained message for topic /openhab/in/Temperature_FF_Bed/state
> deleting retained message for topic /openhab/in/Weather_Temp_Max/state
> deleting retained message for topic /openhab/in/Weather_Temp_Min/state
> deleting retained message for topic /home
> deleting retained message for topic /home/temperature
> deleting retained message for topic /openHABBruegge/out/Slim_motion/state
> deleting retained message for topic /openHABBruegge/out/Slim_temp/state
> deleting retained message for topic /openHABBruegge/out/Slim_door/state
> deleting retained message for topic /openHABBruegge/out/Slim_battery/state
> deleting retained message for topic /v1/p0/starling/3810/status
> deleting retained message for topic /v1/p0/buyAtmel/aeef/status
> deleting retained message for topic /v1/p0/atmeld/aeef/status
> clean up finished
> Basic test starting
> Basic test succeeded
> Retained message test starting
> Traceback (most recent call last):
> File "client_test.py", line 142, in retained_message_test
> aclient.disconnect()
> File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/client/main.py", line 187, in disconnect
> assert self.__receiver.inMsgs == {}, self.__receiver.inMsgs
> AssertionError: {47: Publishes(DUP=False, QoS=2, Retain=True, MsgId=47, TopicName='PREFIX/DeviceList', Payload=b'{\n "2,1" : {\n "Description" : "N/A",\n "Manufacturer" : "Sogeti HT",\n "Name" : "SignalGenerator"\n }\n}\n'), 48: Publishes(DUP=False, QoS=2, Retain=True, MsgId=48, TopicName='SETest/DeviceList', Payload=b'{\n "6,1" : {\n "Description" : "N/A",\n "Manufacturer" : "Sogeti HT",\n "Name" : "SignalGenerator"\n }\n}\n'), 51: Publishes(DUP=False, QoS=2, Retain=True, MsgId=51, TopicName='hello/world', Payload=b'yooooooooooooooooooo'), 52: Publishes(DUP=False, QoS=2, Retain=True, MsgId=52, TopicName='hello/emqtt', Payload=b'Example: QoS 2'), 53: Publishes(DUP=False, QoS=2, Retain=True, MsgId=53, TopicName='hello/1234', Payload=b'rep'), 54: Publishes(DUP=False, QoS=2, Retain=True, MsgId=54, TopicName='hello/1126', Payload=b'7c:dd:90:78:51:23'), 55: Publishes(DUP=False, QoS=2, Retain=True, MsgId=55, TopicName='hello/27137', Payload=b'7c:dd:90:78:51:23'), 56: Publishes(DUP=False, QoS=2, Retain=True, MsgId=56, TopicName='hello/4457', Payload=b'7c:dd:90:78:51:23'), 57: Publishes(DUP=False, QoS=2, Retain=True, MsgId=57, TopicName='hello/21670', Payload=b'7c:dd:90:78:51:23'), 58: Publishes(DUP=False, QoS=2, Retain=True, MsgId=58, TopicName='hello/21622', Payload=b'7c:dd:90:78:51:23')}
> Retained message test failed
> Traceback (most recent call last):
> File "client_test.py", line 218, in offline_message_queueing_test
> aclient.disconnect()
> File "/Users/nico/Dev/beerfactory/org.eclipse.paho.mqtt.testing/interoperability/mqtt/client/main.py", line 187, in disconnect
> assert self.__receiver.inMsgs == {}, self.__receiver.inMsgs
> AssertionError: {47: Publishes(DUP=False, QoS=2, Retain=True, MsgId=47, TopicName='PREFIX/DeviceList', Payload=b'{\n "2,1" : {\n "Description" : "N/A",\n "Manufacturer" : "Sogeti HT",\n "Name" : "SignalGenerator"\n }\n}\n'), 48: Publishes(DUP=False, QoS=2, Retain=True, MsgId=48, TopicName='SETest/DeviceList', Payload=b'{\n "6,1" : {\n "Description" : "N/A",\n "Manufacturer" : "Sogeti HT",\n "Name" : "SignalGenerator"\n }\n}\n'), 51: Publishes(DUP=False, QoS=2, Retain=True, MsgId=51, TopicName='hello/world', Payload=b'yooooooooooooooooooo'), 52: Publishes(DUP=False, QoS=2, Retain=True, MsgId=52, TopicName='hello/emqtt', Payload=b'Example: QoS 2'), 53: Publishes(DUP=False, QoS=2, Retain=True, MsgId=53, TopicName='hello/1234', Payload=b'rep'), 54: Publishes(DUP=False, QoS=2, Retain=True, MsgId=54, TopicName='hello/1126', Payload=b'7c:dd:90:78:51:23'), 55: Publishes(DUP=False, QoS=2, Retain=True, MsgId=55, TopicName='hello/27137', Payload=b'7c:dd:90:78:51:23'), 56: Publishes(DUP=False, QoS=2, Retain=True, MsgId=56, TopicName='hello/4457', Payload=b'7c:dd:90:78:51:23'), 57: Publishes(DUP=False, QoS=2, Retain=True, MsgId=57, TopicName='hello/21670', Payload=b'7c:dd:90:78:51:23'), 58: Publishes(DUP=False, QoS=2, Retain=True, MsgId=58, TopicName='hello/21622', Payload=b'7c:dd:90:78:51:23')}
> Offline message queueing test failed
> Traceback (most recent call last):
> File "client_test.py", line 178, in will_message_test
> assert len(callback2.messages) == 1, callback2.messages # should have the will message
> AssertionError: [('Topic/C', b'qos 1', 1, True, 1), ('Topic/C', b'client not disconnected', 2, False, 2)]
> Will message test failed
> Overlapping subscriptions test starting
> Traceback (most recent call last):
> File "client_test.py", line 253, in overlapping_subscriptions_test
> assert len(callback.messages) in [1, 2]
> AssertionError
> Overlapping subscriptions test failed
> Keepalive test starting
> Keepalive test succeeded
> Redelivery on reconnect test starting
> Traceback (most recent call last):
> File "client_test.py", line 309, in redelivery_on_reconnect_test
> assert len(callback2.messages) == 2, "length should be 2: %s" % callback2.messages
> AssertionError: length should be 2: [('TopicA/B', b'', 1, False, 2), ('TopicA/C', b'qos 2', 2, True, 1), ('TopicA/C', b'', 2, False, 3)]
> Redelivery on reconnect test failed
> test suite failed
> _______________________________________________
> paho-dev mailing list
> paho-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/paho-dev
