One of the most important reasons that I considered using MQTT for my haptic wristband project was because of its ability to handle devices that are intermittently connected. IOT devices, even if nominally switched on, may at any given time be unavailable due to sleep modes (even although this direction did not prove as promising on the ESP8266 for my application as I’d initially hoped), or simply because the device has temporarily lost Wi-Fi connectivity.

Whatever the cause of disruption, the MQTT architecture provides for the broker to store any important messages (i.e. those flagged with high qos or “quality of service”) to be later delivered to a client that had been previously subscribed, also with high qos, but is at that point in time disconnected.

For my project this is crucial. I have always envisaged my wristband to be an additional “priority” channel to cut through the barrier of noise created by the visual overload of our multiple screens, through the beeps and buzzes representing emails, meeting requests, tweets, texts, phone calls and missed calls – which are themselves competing for (hopefully) the majority of your attention directed at real people and real objects around you as you do your job and live your life!

Therefore, as a device that handles relatively few but very important notifications, a missed message is unacceptable.

Unfortunately, as I hinted at last week, I struggled to get this MQTT mechanism to work. The main problem I was having was that I was not sure where the fault lay – and even wondered if I was misunderstanding the way that MQTT should work. I began to suspect that the free MQTT broker that I was using did not support this more advanced behaviour. As it transpired, and thanks to some timely help from Carl at CloudMQTT support, the MQTT broker turned out to be fine – it was both the ESP8266 client and the MQTT test software that I was using that were dropping the ball.

After wading through many documents on the web, I also realised that this whole area of MQTT persistence is quite confusing and not well explained from the point of view of someone actually wanting to make use of it. So, in this post, I aim to first explain how persistence should work, and then afterwards provide all the details for someone to actually try it out for themselves. Along the way I also discover a couple of great new products.

MQTT Persistence

There are actually two separate and parallel sets of functionality when considering messages that persist on the broker – something that was certainly not clear to me at the outset.

The first set of functionality is that described above, and what I was looking for – which is to say that the broker will store messages until an offline device later reconnects to retrieve them.

To do this requires the following steps and associated MQTT parameters:

1. The receiving client connects to the broker with the cleanSession parameter set to 0. Without this, a “clean” session will be started, which ignores any stored messages on the broker.
2. The client MUST connect each time with the same “client id” . By default, many clients use a randomised client id, which will not link back to the previous session that was disconnected.
3. Subscribe to your chosen topic with the qos parameter set to 1. “1” instructs the broker to deliver any messages that the client may have missed. “2” goes one step further by requiring that the client acknowledge receipt of a message, to ensure that a message is not delivered more than once.
4. The client should execute the above at least once to subscribe to the chosen topic with qos >= 1.
5. The client can then be disconnected (e.g. by removing power, stopping the software, disconnecting network, etc).
6. Using another MQTT client, publish a few test messages to your chosen MQTT topic, ensuring that the qos flag is set to 1 (or 2)  for the published message.
7. When the receiving client is reconnected, and as soon as it resubscribes to the chosen topic, the broker will send it those stored messages.

In summary, this is really just a FIFO queue that is stored on the broker per topic and for each client that has subscribed to the topic with qos >= 1. It is useful (and necessary) whenever you have events or sensor readings that you do not want to miss, even if the receiving client is temporarily unavailable.

On the surface, the second set of functionality sounds similar, but actually behaves quite differently. There is another option available when you publish a message, namely the “retain” flag. It would be more helpful to think of this as the “last known good” flag. What happens is that the last “retained” message that has been published to each topic is always kept. Whenever a client subscribes to that topic, it will receive the “last known good” message. In fact, whenever it resubscribes it will always receive that message, even if it has already been sent that same message before.
Step by step, this works as follows:

1. Publish a test message to your chosen MQTT topic, ensuring that the “retain” flag is set 1. Note that if you publish more than one, only the last will be retained.
2. When the receiving client subscribes to the chosen topic, it will receive the “last known good” message. Note that this happens irrespective of the cleansession flag setting.
3. If the client disconnects and then resubscribes to the chosen topic, it will receive that same “last known good” message.
4. In fact, the only way to get rid of that “last known good” message is to send a blank message, with the retain flag set to 1, to the topic.

How might this be used? It is ideal for whenever the MQTT message conveys state (as opposed to an event). For example, where the device is a plug, it needs to know and restore its state (in the plug example on or off) every time it reconnects, irrespective of whether or not the state has changed in between.

Getting into the Details

I tried two alternative clients: the first is hardcore, command line based, and the second is tap-your-finger iOS (yet fully featured) – but costs a couple of dollars.

First: to test the concept, and because I had been getting nowhere and was frustrated, I downloaded the MQTT Paho python client (with documentation here), which is part of the iot.eclipse.org project. Although there are clients for many platforms, I chose the python option because it is easy to use on both Windows and Linux – and there is no native WIndows client. In reality, I first ran these tests within a Linux (Mint) VM, but I provide the instructions here for Windows:

1. First you will need to download Python and install. I chose the latest 3.x version. When you install, choose “Add python.exe to Path”.
2. Next, start a command prompt as administrator, and  install the paho client using pip install paho-mqtt.
3. Create a new test folder for pub.py.
4. Also place sub.py in that folder.
5. cd to that folder from within the command prompt.
6. Start a MQTT receiving client by running python sub.py.
7. stop the client using CTRL-C (maybe hit a few times).
8. Start another command prompt as administrator.
9. cd to your test folder from within the command prompt.
10. Publish a message (or a few) with qos=1 by running python sub.py.
11. Restart your receiving client by again running python sub.py from your first command prompt.
12. The message(s) you published when the client was offline will be picked up.

The second alternative is to use MQTT Inspector for IOS. This is a beautiful piece of software that just seems to work effortlessly. The reason I bought it was that it has a great “Low Level” option that allows you to see what is being sent back and forth.

It also has all the required qos and retain options, and correctly picks up stored messages as expected.

Problem with the ESP8266 client

At this point in time, I cannot get the persistence (i.e. qos / queuing mode) to work using the ESP8266 MQTT client, but have communicated with tuanpm through the forums, and have every confidence that he will sort it out.

In line with all the explanations above, this is how I experimented with persistence using esp_mqtt on the ESP8266 (all of this is done within mqtt.c):

Set cleanSession to 0 when connecting in user_int:

MQTT_InitClient(MQTT_Client *mqttClient, uint8_t* client_id, uint8_t* client_user, uint8_t* client_pass, uint32_t keepAliveTime, uint8_t cleanSession)

Set qos =1 when subscribing in inmqttConnectedCb:

MQTT_Subscribe(MQTT_Client *client, char* topic, uint8_t qos)

As explained, I used other software to publish, but if using the ESP8266 as a publisher, set qos to 1:

MQTT_Publish(MQTT_Client *client, const char* topic, const char* data, int data_length, int qos, int retain)

For the second mode of persistence that I described (“last known good”), you need to set the retain flag to 1:

MQTT_Publish(MQTT_Client *client, const char* topic, const char* data, int data_length, int qos, int retain)

Oh yes, I dont think it makes any difference, but I also configured the newer MQTT protocol from within user_config.h:

#define PROTOCOL_NAMEv311 /*MQTT version 3.11 */