Eclipse Paho - Writing The Last Ever C/C++ MQTT Client Library

If you are as old as I am, and lived in Britain in 1981, you might remember advertisements for a product called "The Last One". It promised to be the last piece of software you would ever need to buy, as it would write all your programs for you. Needless to say, its claims were overblown -- I've written plenty of programs since then, all without its aid. (If you want to know more see this page).

When I was asked to write an MQTT client library for ARM's mbed platform (mbed.com) last year, it brought "The Last One" to my mind, because I've been writing MQTT tools and libraries in C for the past 10 years or more, for IBM, and lately for the Eclipse Paho project. On the whole, I like that work, but writing yet another C MQTT library was starting to get a bit old. And I could see that after mbed, there were other embedded devices and operating systems for which an MQTT solution would be needed. So rather than writing a client library specifically for mbed, I decided to see if I could write a really portable client library, that would work wherever anyone needed it to, but still cater for just about everyone's requirements.

Arcom Viper - one of the first embedded computers to run MQTT, using Java

The standard C client libraries in Paho have a lineage that stretches back to the early days of MQTT in 2000. A colleague of mine, Ian Harwood, wrote a portable client library for IBM, called IA93. The major drawback as I saw it was that the API it presented was not very user friendly. There was a lot of bit manipulation and buffer mangling going on. When another colleague of mine, Dave Locke, suggested I write a new C client to go along with my then new MQTT server (Really Small Message Broker - RSMB), I thought I would try to make it look like a user friendly Java API. But what I also did was write it primarily for Linux and Windows, as it seemed that even embedded devices were consolidating on those operating systems. And I also threw in the kitchen sink functionally, partly because a committee was involved in defining the requirements. For mbed and similar platforms, we needed a different approach -- minimalist and portable.

I used the Paho mailing list to help me determine what criteria I should adhere to. As usual on that mailing list, I got some very helpful advice, which resulted in these principles:

1. No dynamic memory allocation. Embedded operating systems and devices sometimes have idiosyncratic memory models which make porting difficult. In any case, we would like to make memory use as predictable as possible.
2. No specific networking calls. TCP stacks in the embedded world are not standardized - we'd like to copy with any of them without changing the core client code.
3. No operating system specific calls. No mallocs or TCP calls in the core code obviously helps with this. We should try to use ANSI standard library calls only, and even minimize the use of them.

ARM mbed IoT Starter Kit for IBM Internet of Things Foundation - using the Paho embedded MQTT client

To make the libraries as flexible as possible, I decided to structure them into 3 layers.

1. The lowest level very simple layer, called MQTTPacket, simply serializes data structures into MQTT wire format. The resulting buffer is ready to be written directly to the network. It can also accomplish the opposite: deserializing the MQTT wire format back into data structures. There are some helper functions for reading from and writing to the network, which I originally meant to be just examples, but have become a more integral part of the package since then. MQTTPacket is very small and simple, and limited in its function.
2. A synchronous, blocking API layer, called MQTTClient, using MQTTPacket underneath. This was first written in C++, as that is the default language on the mbed platform. Each MQTT call, like subscribe, or publish, waits for the MQTT protocol exchange to complete before returning. Thus it is easy to write an application - you don't have to worry about synchronization. Again, ease of use is paramount in the mbed environment. Network and OS-specific API calls for timing are relegated to two replaceable classes, so that porting to a different platform does not touch the core code. Another colleague of mine, Allan Stockdill-Mander, translated this layer into C, for a Texas Instruments environment where C++ was not well supported. This C version is now called MQTTClient-C.
3. An asynchronous, non-blocking API layer, called MQTTAsync, also built on top of MQTTPacket. This needs a background thread and syncronization facilities to work, both of which are OS specific and placed in replaceable classes. As of the time of writing, I have not completed this level, primarily I think because MQTTPacket and MQTTClient seemed to satisfy most people's needs. I'm open to any offers to help me complete this work :-)

Now these libraries in their different forms have been ported to Linux, mbed and Arduino platforms amongst others, and are used in a variety of SDKs, including the recent Amazon Web Services IoT embedded SDK and IBM's IoT Foundation embedded toolkit. I still have quite a bit of work to do to complete my vision, including getting full support for FreeRTOS and other popular embedded operating systems. However I'm very happy that my original goals seem to be being fulfilled, and that I may not have to write another totally new MQTT client again!

For more information, see https://www.eclipse.org/paho/clients/c/embedded/. If you would like to contribute any work on Paho, please check out the contributing guide at http://git.eclipse.org/c/paho/org.eclipse.paho.mqtt.c.git/about/.