ESP8266 Smart Shelf – Part 2 (NodeMCU + Arduino IDE)

In Part 1, I introduced the Smart Shelf concept, the sensor used, and the web page for displaying real-time updates.

Since then, I have also created a short video to demonstrate the project:

Part 1 explained how I used the Sparkfun ESP8266 Thing and the C SDK. For reliability reasons (in my case, when presenting / demonstrating in front of a live audience you ideally want things to work!), I have always preferred to use the C SDK, especially after the problems I experienced with Lua causing frequent reboots when I tried that approach a year ago.

IMG_4663mBut with the arrival earlier this year of the Arduino IDE, there is now a compelling alternative, which seems to work well (based on my limited experiments). So, as illustrated in this post, if you are an Arduino user, my view is that connecting a NodeMCU board and programming using the Arduino IDE is no more difficult than developing using an official Arduino board. The advantage is that you get built-in Wi-Fi, a smaller form factor and all this at a fraction of the cost. So for many applications, using an ESP8266 dev board like the NodeMCU might be the approach to consider first. Continue reading

ESP8266 Smart Shelf – Part 1

ESP8266 Dev Boards

I have realised, after spending some time building small projects using the ESP-01, ESP-03 and ESP-12 modules running off battery power, that it can be quite time consuming and tedious to build the necessary power supply circuit and wire up serial connectors each time. Also, where space is not critical (unlike in my train project), it becomes quite compelling to consider dev boards that integrate an ESP8266, together with other features, in an-easy to use (bread-boardable) form factor.

This is even more the case if you look at the falling prices of, for example, the NodeMCU board on AliExpress, You are not likely to be able to build your own equivalent at a lower cost, even discounting the time it would take.

In my type of interactive projects (where the ESP8266 cannot be put into deep sleep for extended periods), operating off battery power realistically means using LiPo/LiIon. As a result, I found myself having to disconnect the battery, attach it to a LiPo charger and then reconnect for every charge cycle.

Sparkfun ESP8266 Thing

This is when I saw that the Sparkfun ESP8266 Thing “includes a LiPo charger, power supply, and all of the other supporting circuitry”. Very convenient for what I had in mind…

But before I get to that, I want to be clear that the Sparkfun ESP8266 Thing is actually not my favourite module for the following reasons:

  • It is relatively expensive (more than 5 times the price of one of those NodeMCU boards listed above, including shipping)
  • Programming is not done neatly via the USB port like in the NodeMCU, so requires an additional 3.3V FTDI USB converter (or equivalent).
  • Unfortunately, the serial programming lines interfere with debugging via the serial port after programming, which requires that you install terminal software (e.g. RealTerm) that can disable toggling the DTR line, as described in the Sparkfun Hookup Guide. I have not yet tried the Adafruit Huzzah, but it does not appear to have this problem, although requires you to fiddle about with buttons to get into programming mode. In contrast, the NodeMCU board simply handles (transparently in the background) toggling GPIO0 low for a programming cycle.

It is worth noting that Sparkfun’s newer ESP8266 Thing – Dev Board fixes these issues (other than the price) – but then does not include a LiPo charger!

It would be interesting to find an inexpensive LiPo charger/management board that one could pair with a NodeMCU, if anyone knows of such a thing?

Smart Shelf Concept

Continue reading

A “Thomas” toy train joins the Internet-of-Things

I am intrigued by the notion put forward by David Rose (seen here presenting at TEDx) of “putting magic in the mundane – that is, having ordinary items that can do extraordinary things”.

I find compelling the notion that for everyday activities there will be a swing in popularity away from having to interact with complex abstract worlds behind glass screens toward a more “authentic” physical experience, for which we humans are so well adapted. But this shift will only be made possible by means of even more advanced technology – able to hide its own complexity.

This month the popular “Thomas the Tank Engine” toy celebrates its 70 anniversary. As a fun project, what could be more appropriate than to bring this traditional toy into the age of IoT, while preserving its physical appearance and simple charm?

For a short overview / teaser, watch the following video (be sure to select HD):


Inevitably there will be those who see this and ask “why?”

The short answer is that, as Hackaday believes,  “why” is the wrong question.

A longer answer is that the train is simply BETTER when you can control its speed (remotely over wi-fi), more consistent to operate, especially at low speeds because of an auto “throttle-boost” that kicks in when the accelerometer senses that the train is going uphill, and more user-friendly with remote monitoring of the battery state.

And if you need still more justification, the Train appeared this week at a trade exhibition as a source of IoT data to demonstrate business software (go to YouTube here if you are interested, where one of my colleagues explains).

The main challenge was to be able to fit the required electronics into an existing form factor. As it transpired, the toy train was a lot more compact than I had anticipated. I had to undertake some delicate “surgery” to free up space underneath the train and behind the front face, but this was done in a way that a casual observer would not notice.

Follow the link to a full write-up to discover more of the details:

Continue reading ->

Simple (Browser Based) graphical display of MQTT data

Recently, I have been using MQTT to stream ESP8266 sensor data to a Mosquitto broker running on a Raspberry Pi. I wanted to develop something to show the stream graphically.

I have previously found Dygraphs to be a great piece of software. I like the way it consists purely of client-side javascript, which runs entirely within a web browser. In fact, you can even open an HTML file (containing javascript) from your local disk, without requiring a web server.

DygraphDemoI specifically wanted to build something along the lines of Dygraph’s dynamic update demo, except that I wanted a constant scrolling “window” of values – like a heart monitor for example. (Really, what is the use of having all your data scrunch up against the left of the window?).

Having solved that issue, I then had to connect the graph to my MQTT broker to be able to receive live data to display.

This post goes into details of how I did this, but before going further, here is the result:

[wpvideo uIHfHtPH]

Continue reading

ESP8266 and IoT: Streaming serial accelerometer data via MQTT


Over the last couple of weeks I have been sidetracked by another project – about which I’ll certainly share more details when it is done. The project has a requirement to stream readings to the Internet from an accelerometer (in a very small form factor), so I decided to pair one with an ESP8266.

IMG_4377As I work more with the ESP8266, I am finding a common theme when using this chip.

On the one hand its low cost and extreme functionality in a very small package makes it a real game changer, and I was glad to see an article in Hackaday a week or so ago bringing this message to a wider audience. Essentially, the ESP8266 is not much more difficult to program than an Arduino, yet gives you a tiny, single-MCU solution with Wi-Fi included.

BUT, on the other hand, it can be really difficult to find good, clear information – at which point it suddenly becomes NOT an easy task to get something done. Although this situation is changing, it currently costs me more hours than I would have imagined to accomplish certain tasks.

I found myself in this wilderness recently after selecting the ADXL345 accelerometer chip, which I chose mainly because it works off 3.3V and because it has a good reputation. It also supports both SPI and I2C, which gives greater flexibility when interfacing to an MCU. So far, so good.

IMG_4378This blog post details the different approaches I tried, and the challenges I faced. These were:

  • I2C
  • Hardware SPI
  • Software SPI

At the end I include a video showing a working ESP8266 (ESP-12 version, because I will later need an ADC) streaming accelerometer readings to an MQTT broker running on a Raspberry Pi, with results shown graphically on a PC, which is connecting to that broker.

Continue reading

Investigating MQTT persistence in an IOT protocol on the ESP8266

One of the most important reasons that I considered using MQTT for my haptic wristband project was because of its ability to handle devices that are intermittently connected. IOT devices, even if nominally switched on, may at any given time be unavailable due to sleep modes (even although this direction did not prove as promising on the ESP8266 for my application as I’d initially hoped), or simply because the device has temporarily lost Wi-Fi connectivity.

MQTTInspector_sWhatever the cause of disruption, the MQTT architecture provides for the broker to store any important messages (i.e. those flagged with high qos or “quality of service”) to be later delivered to a client that had been previously subscribed, also with high qos, but is at that point in time disconnected.

For my project this is crucial. I have always envisaged my wristband to be an additional “priority” channel to cut through the barrier of noise created by the visual overload of our multiple screens, through the beeps and buzzes representing emails, meeting requests, tweets, texts, phone calls and missed calls – which are themselves competing for (hopefully) the majority of your attention directed at real people and real objects around you as you do your job and live your life!

Therefore, as a device that handles relatively few but very important notifications, a missed message is unacceptable.

Unfortunately, as I hinted at last week, I struggled to get this MQTT mechanism to work. The main problem I was having was that I was not sure where the fault lay – and even wondered if I was misunderstanding the way that MQTT should work. I began to suspect that the free MQTT broker that I was using did not support this more advanced behaviour. As it transpired, and thanks to some timely help from Carl at CloudMQTT support, the MQTT broker turned out to be fine – it was both the ESP8266 client and the MQTT test software that I was using that were dropping the ball.

After wading through many documents on the web, I also realised that this whole area of MQTT persistence is quite confusing and not well explained from the point of view of someone actually wanting to make use of it. So, in this post, I aim to first explain how persistence should work, and then afterwards provide all the details for someone to actually try it out for themselves. Along the way I also discover a couple of great new products. Continue reading

Investigating MQTT as an “IoT protocol” for the ESP8266 wristband

I have recently had a rethink about how I will link my wristband to an alert source.

To demo my first prototype, I hosted a web page on the ESP8266 device itself. Selecting a button on the web page posted a value, which activated the chosen vibration pattern. This is a very convenient way to demo the concept, but, in the real world it has some significant drawbacks.

The first drawback is that the whole submit / refresh cycle of a web page is not particularly fast, which also probably means that the web server on the ESP8266 is having to do a relatively large amount of work. I therefore had originally planned to eliminate some of this overhead by following the approach in the ESP8266 IoT example of using http methods and json, which can be called from anything (e.g. from another pc/server by using the curl command).

But this does not solve a second issue, which is that the ESP8266 server has to be “always on”, waiting for a command from the remote web page. This is likely to make power savings measures (sleep cycles) difficult, which is less than ideal given that Wi-Fi is not as power efficient as Bluetooth – notwithstanding that Wi-Fi continues to evolve.

The third problem is the direction of the link. To communicate with the wristband requires that you initiate a connection. This is not a problem if you are on the same network (and also assuming you always know the ip address of the wristband). However, if the source of the alert is from outside the network, a firewall port will need to be opened and forwarded – something that will usually be out of the question on most corporate networks.

With this background, and inspired by Peter Scargill’s exploits, I decided to investigate MQTT. Potentially if offered to solve all three of my issues:

  • It is a lightweight protocol, optimized for resource-constrained devices (like the ESP8266) – refer to here and here.
  • Because it uses a broker in the middle, both the client and the (wristband) device talk to the broker, not to each other. This allows for an asynchronous communication scenario in which I could allow the wristband to, for example, check the broker for messages and then go to sleep for a second, yet be guaranteed not to have missed a message when it next checks at the end of that second. Such a duty-cycle could dramatically lower the overall power consumption.


  • The broker architecture also solves the firewall problem by reversing the direction of the connections. Now the wristband will initiate the connection to the broker, which can thereafter communicate without the need for port forwarding.

Continue reading