Towards The Next Version of MQTT

MQTT is a transport (not protocol as Andy Stanford-Clark would say), which allows applications to easily communicate with each other over TCP/IP. I have been working with MQTT in some regard since the early 2000s, and am now lead of the Eclipse Paho project, which supplies client libraries in a variety of programming languages. I've seen MQTT versions progress from 3.0, to 3.1 and last year 3.1.1, the first standardized version not published by IBM, but by the OASIS consortium. In fact when I implemented an early version of MQTT software, Andy Stanford-Clark asked me to include support for version 2.0, it was that long ago.

The brief for the MQTT 3.1.1 standardization process was to make minimal changes to ensure that it could be completed quickly. But when it was done, I think that just about every committee member had some burning issues. As a result, we embarked on the next round of updates, attempting to balance the competing forces of stability and progress. Personally, I envision that the next version of MQTT should be good enough to be fundamentally untouched for perhaps a decade, and that 3.1.1 will coexist with it.

One of the first problems is what to call this version. This seems simple, but there is only one byte to represent the protocol version in the MQTT connect packet. MQTT 3.1 used 3. MQTT 3.1.1 uses 4. In writing Paho embedded client libraries, I've even referred to MQTT 3.1.1 as version 4 myself. After not inconsiderable deliberation, it has been agreed this time to make the document and the wire versions the same. This means they are both 5, and that skipping 4 has no more significance than it has been used already. We expect this principle to be followed in the future, but time will tell. (Actually the document version will be 5.0, because we've been told that the .0 must exist).

For this revision of MQTT, the committee has had the welcome addition of members from Microsoft. This is one recognition of the importance that MQTT is attaining in the Internet of Things (IoT). One consequence is that we have set ourselves a quite aggressive schedule, to complete version 5 in the first quarter of 2017. This means having an implementable first public draft in the Autumn of 2016.

A good way to understand the thinking driving the changes in MQTT 5.0 is the "Big Ideas" document. The ideas are:

Improvements for Scalability and Large Scale Implementations

Retained messages, QoS 2 and cleansession false all require state to be held which can make it more difficult to implement a solution where clients may reconnect to a different endpoint. It will be possible for a server to indicate a lack of support for these features at connect time. The ability for multiple clients to share a single subscription will allow applications to be horizontally scaled.

Another issue to be addressed under this banner is called Simplified State Management. This has at least two major advantages.

1) As an application, I only want my session state to be discarded when I've completed all my work, not when my network connection fails. This was inconveniently hard in all previous versions of MQTT - not in version 5.

2) The ability for session state to have an expiry time. If the client does not connect within a certain amount of time, the session state can be deleted by the server. This obviates the need for a client to reconnect just to clean up session state.

The connection state diagram* for MQTT 3.1.1 is shown here:

MQTT v3 State Transition

The new state diagram* for 5.0 is here:

MQTT v5 State Transition

As you can see, as well as having more functionality, the transitions are simpler.

Improved Error Reporting

In 3.1.1, negative acknowledgements (nacks) were added for subscribe commands. This still left publish requests without - this will now be fixed. All response packets will now be able to carry error codes, and the specification will list them and their meanings. MQTT servers will be allowed to send disconnect commands with reason codes, such as in the case of a server shutdown, which should help MQTT clients take more appropriate action.

Metadata for the Publish Command

The format of the payload can be indicated, and some other possible fields such as time to live (message expiry), reply-to topic and correlation identifier for a request reply mode of operation. At the moment, it is planned that the available metadata fields will be predefined, for the sake of simplicity.

Resource Constrained Clients and Performance Improvements

Nolocal subscriptions - the ability for a subscriber to not receive its own publications. This will make it easier for the perennial first MQTT application to be written - the chat room! Topic registration also falls into this category, although the current proposal is a simplified version, where using a topic of length 0 indicates "re-use the last one I sent". That may yet be revised.

Working draft 4 is available now. At the end of September the committee is meeting face to face at IBM Hursley - the agenda. If you are not on the committee but would like to submit a comment about the draft or any other subject, instructions for doing so can be found here.

*Both MQTT diagrams created by Andy Banks (IBM)

About the Authors

Ian Craggs

Ian Craggs
IBM