Re: [paho-dev] MQTT <-> REST+websocket mapping

[Raphael Cohn <raphael.cohn@xxxxxxxxxxx>]

Matteo,

I don't think WebSockets are extremely complicated, but they did confuse some ideas. In essence, the best way to understand WebSockets is to strip away all the guff in the standard and think solely in those terms that make sense for TCP. Then simply implement whatever protocol one needs on top. To my simple mind, WebSockets are nothing more than a gateway that tunnels through a cruddy HTTP front-door to a TCP back-door. Indeed, it should be quite possible to write the equivalent of HAProxy for WebSockets that does 100,000+ connections on a box. The only bottleneck is the use of XOR, which prevents one using vmsplice() / splice() functions in modern Linux kernels on upstream data. Client-side, just treat the WebSocket 'frames' as a TCP stream of data, ie forget about the framing meaning anything. It's all just chunks of bytes.

Once you see WebSockets as a TCP stream, you just keep a buffer and respond to your callback for read - it's no different to epoll's eventing. Move over the buffer parsing data structures. Occasionally compact the buffer. Done. Debugging binary data doesn't have to be hard at all. You just need to build up your data structures from first principles. And if you do it right, one developer or two has to understand it; the rest just use the library. If you don't want to use binary data, then you might want to use the STOMP protocol. Having said that, I think MQTT would be far better for the majority of situations.

Long-polling is, to my mind, quite retrograde. That's what a WebSocket is trying to be for. And then you should either use your protocol's 'liveness' (which MQTT has) or TCP keep-alive if it doesn't (not good). WebSockets ping-pong is quite useless, as it knows nothing of the protocol layer above it and its behaviour.

I really struggle to see the use case for Engine.IO. To my mind, Socket.IO should probably start to die. All the major browsers now support the standard WebSockets RFC. And IE 8 / 9 can use a flash shim. The only bugbear is older versions of Android and iPhone (nearly all dead). And running a binary protocol over node.js? Clever, but wrong. Ultimately, for real scale, you're never going to do it with a dynamic language. The memory management and lack of sticky threading kill you. 

I really, really hazard against using HTTP for MQ. You end up having to use a lot of persistent state. If it is only to ship data to browsers, use WebSockets. If you want REST for messages, think long and hard. But remember that even HTTP headers are badly supported by most simpler http libs and browser plug ins. There's a good reason Amazon SQS has never really taken off...

Raphael Cohn
Chief Architect, stormmq

Secretary, OASIS AMQP Standard
raphael.cohn@xxxxxxxxxxx

+44 7590 675 756

UK Office:
Hamblethorpe Farm, Crag Lane, Bradley BD20 9DB, North Yorkshire, United Kingdom
Telephone: +44 845 3712 567

Registered office:
16 Anchor Street, Chelmsford, Essex, CM2 0JY, United Kingdom
StormMQ Limited is Registered in England and Wales under Company Number 07175657
StormMQ.com

On 15 March 2013 18:46, Matteo Collina <hello@xxxxxxxxxxxxxxxxx> wrote:

2013/3/15 Raphael Cohn <raphael.cohn@xxxxxxxxxxx>

Interesting stuff. We're very much in favour of keeping MQTT with a binary mapping. It makes it trivial for implementations to add a WebSockets tunnel, using a wide range of different technologies that aren't tidied to a broker, etc.

The challenge is doing a little binary handling in _javascript_, but done once, it's done.

It is already done for node.js (it is what I am using): https://github.com/adamvr/MQTT.js.

It should not be harder to port it to the browser with ArrayBuffer (there are polyfills for that).

The only weakness with WebSockets is the blasted framing (XOR? Text frames? What where they smoking? Oh well, that's a committee for you). Many of the more generic WebSockets <=> TCP implementations (correctly, in my view), scoop up as many bytes as possible and send them on as WS binary frames. Thus there's no delineations in the frames so that they align with MQTT data structures.

And the W3C's WS API is naive and assumes one should present complete frames at each callback event. This sort of thing seems to make a js' programmers job harder, until he realises that it's just life, and a WebSocket is nothing more than a TCP stream. All he has to do is keep a buffer, and pre-pend it to the next WS event...

The WebSocket specification is extremely complicated, and still it has jeopardy support.

I am not a fan of it, and I prefer the much simpler Server-Sent Events one.

The Socket.IO devised a protocol to run inside a websocket or a tcp stream to fix these problems. It's called Engine.IO.

Do you think it might be possible to route MQTT inside a long-polling connection? 

It seems it can work, and it can be a good solution.

After Nick and your email I am getting more convinced that routing binary data to browsers might be a feasible idea.

From a dev perspective, it will be much harder to debug.

I'd agree with Nick on the challenges of using HTTP for messaging. It's quarts into pint pots - messaging is async, HTTP is sync and typically short-lived. It's hard to scale a broker that does HTTP mq.

I know that's very hard, but I am trying try to solve these problems. :)

Cheers,

Matteo 

_______________________________________________
paho-dev mailing list
paho-dev@xxxxxxxxxxx
http://dev.eclipse.org/mailman/listinfo/paho-dev