Re: [paho-dev] MQTT <-> REST+websocket mapping

[Matteo Collina <hello@xxxxxxxxxxxxxxxxx>]

Hi Nick,

You make very good points, I'll reply inline:

> 1) The websocket tunneling issue is longstanding in this community. Some
> previous attempt has been done, and there was even a proposal for routing
> binary MQTT inside a websocket tunnel.
> I believe that the best approach to make this work is to map MQTT to JSON in
> a standard way, and then route it inside websocket & polyfills (es.
> socket.io or sock.js and the like)
> This will simplify implementations and diffusion.

There is already a proposal for using MQTT with Websockets on the Paho
wiki (which I would link to if the entire eclipse wiki wasn't down
right now). It takes the approach of tunnelling the MQTT binary
packets through the websocket.
This is the approach taken by the Mosquitto websocket implementation afaik.

In my opinion, mapping it to a JSON payload across the websocket
(which is what we did with earlier versions when browser support for
ArrayBuffer etc was not as wide spread as it is today) means you lose
the benefit of the light-weight packet size over the network - one of
the core principles of the protocol.

The real problem with this approach is that it is not backward-compatible,

and you cannot build a solution that works mostly everywhere.

The Socket.io team is dropping WebSocket has a first transport, because is still

not well supported and it takes too much time to detect if it is working or not (~20s). 

In Socket.io 1.0, they are going to establish a long-polling connection,

and then try to upgrade it to WebSocket.

Moreover, WebSocket is still not compatible with SPDY.

In the end, we are on the web, not on low-powered constrained devices.

Have you looked at the WebSocket protocol? How could you call that lightweight?

A different format will not change its simplicity: JSON parsing and formatting 

is fast on browsers and servers.

> 2) While creating an interface for publishing messages is easy, the core
> issue is on reading them.
> For example, ActiveMQ uses a DELETE to read a message, which can timeout if
> no message is published in a given period of time. This approach is not
> really web-compatible,

can you explain what you mean by "not really web-compatible". DELETE
is absolutely the right verb to use if you request a resource that as
a side-effect removes it from the server. From _javascript_ you can
issue a DELETE XHR request just as easily as any of the other verbs.

I am taking a different approach, which is representing devices on the web.

I think that a way to map binary-speaking devices to REST endpoints is what it is 

missing to scale the IoT to the web world.

Developers want to use the very same design approach they used for web and mobile apps.

There should be no waiting involved, no "queuing" of messages, and something similar.

The "pure" pub/sub logic should be given by the Websocket+alternatives solution.

> so QEST adopt a standard GET approach, exposing to
> the web every published message (here you can find is my home temperature:
> http://qest.me/topics/temp). Every message is stored on a Redis database.
> However, exposing the last message on the web is extremely similar to the
> retained message behavior. In QEST, I did assimilate the two concepts
> stating that every message is retained, even on MQTT.
> At the moment, I am revising this decision to be MQTT-compliant. Still, the
> web behavior needs to be clear. One possible solution is to expose on the
> web only retained messages, or ignore the 'retained' flag at the web level
> and expose everything.

You cannot only expose retained messages and call that MQTT-compliant.

Exactly. That's what this point is all about, supporting both of them,

while keeping a low response-time (<200ms).

 

A possible suitable approach would be: When an http-client issues its
first 'subscribe' to a topic, return any retained message on that
topic. Then, as you would with an MQTT non-clean session client,
subscribe to the topic in the MQTT-sense for them and start queuing up
messages for that client. Subsequent requests from that client will
then consume messages off that client-specific delivery queue. A set
of custom HTTP headers could be defined to say how long that
subscription should be maintained for before discarding the queued up
messages.

This is interesting.

The problem then came when you want to handle the non-retained messages.

For a web client issuing a "GET" request, waiting X seconds is not an option.

How about serving the last non-retained message?

It is worth noting this is well trodden ground. A few years ago, we
released a support pack for WebSphere MQ that provided an HTTP
interface. This isn't explicitly MQTT as it covers queues as well, but
it did examine a lot of these issues. This has since been rolled into
the base product, but you can see the original support pack
documentation here to see how they defined the interface
http://www-01.ibm.com/support/docview.wss?rs=171&uid=swg24016142.
Again - this was for a more complex messaging system than MQTT in
terms of options and properties, but the basic concepts are there. I
should add, I don't like how they chose to do subscriptions in the
support pack - they went for the 'do a request with a timeout and only
deliver a message that happens to arrive in that period' approach. But
interesting background material.

I know there have been previous attemps, but these were designed taking in mind

the requirements of the broker/infrastructure, not the applications. That subscription

mechanism is the right way for a broker, but it is a big problem for an application

developer.

That's the point I am making for the Internet of Things support.

Cheers,

Matteo