A description of “config” and “state” MQTT topics of GCP-Cloud IoT Core and an illustration of ESP32 flash memory encryption w/Mongoose OS.

TL;DR

My Medium home page is here.

This short post is an extension to the previous one called “How to check the weather using GCP-Cloud IoT Core with ESP32 and Mongoose OS”: [link]. Today, we tackle config and state special MQTT topics of Google Cloud Platform-Cloud IoT Core and we also talk about actions countering local attacks on the device, the most important being the ESP32 flash memory encryption. At last, simple measures of ESP32 current/power consumption are made.

Note: Most of the informations given here come from Mongoose OS web site ([link]) and from Cloud IOT Core documentation ([link]).

A) “config” and “state” special MQTT topics

A) 1) Presentation

Basic understanding of the role of these two topics is given here: https://cloud.google.com/iot/docs/concepts/devices. It can be good to know that their use is a mean to obtain an equivalent of AWS IoT device shadow.

On the project architecture diagram given at the beginning of our previous post ([link]), we see besides telemetry two other data flows, Config and State:

• Indeed, the Cloud IoT Core service may publish configuration update messages to a special topic the device has subscribed to ([link]). It is useful when we need the device to go to a new state, e.g. by updating a parameter of its associated sensor, by changing a deep sleep period, moving a servomotor, etc. For efficiency, there shouldn’t be more than one message of that type per second per device. Such a message is an arbitrary user-defined blob (we’ll use JSON), up to 64 kiB. At last, the name of this special MQTT topic is imperatively:

/devices/{device-id}/config

• On the other direction, a device may publish to a special topic — that Cloud IoT Core has automatically subscribed to — messages concerning its state ([link]), e.g. quantity of RAM available, state of a button, etc. It is often used to see if the previous config message sent to the device had the desired effect. For efficiency, this kind of publication shouldn’t be done more than once per second per device. Such a message is an arbitrary user-defined blob (we’ll use JSON), up to 64 kiB. At last, the topic to which the device publishes its state data has imperatively this name:

/devices/{device-id}/state

Note: Sending commands to devices is also possible from Cloud IoT Core: see [link] but we won’t illustrate it.

The following two sections A) 2 and A) 3) illustrate simple use cases of config and state special MQTT topics:

• Illustrating config will use the blue built-in LED connected to GPIO2 of ESP32 DEVKIT V1.
• Illustrating state will use a push button we chose to connect to GPIO4:

A) 2) Updating device config from Cloud IoT Core

Context

We use the blue built-in LED connected to GPIO2 of ESP32 DEVKIT V1.

The config we want to impose from GCP-Cloud IoT Core to the device is the desired state of the blue built-in LED.

So, if we want to switch on (respectively off) the device blue LED from Cloud IoT Core, we choose to publish to the topic whose name is /devices/{device-id}/config the JSON message {"on": true} ( respectively {"on": false}).

Accordingly, in the device firmware, we should subscribe to that topic and parse the JSON message received to switch on (respectively off) the LED. Really simple!

ESP32 firmware featuring a subscribe to “config” topic

We write the following fs/init.js file to meet our goal. Useful documentation links are provided at the beginning of the source code:

To upload this file to ESP32 file system, from the folder (of the host computer) containing the fs folder containing itself the init.js file, we type:

mos put fs/init.js
mos call Sys.Reboot

Publishing a new config to the device from Google Cloud Console

We head to https://console.cloud.google.com/iot/ and then we follow the different steps indicated below, the green figures indicating where to click or type something:

This will switch the blue LED on ! The device will also send an acknowledgment message to GCP, telling it it has received the new configuration (in that case, there is a green “check” icon, otherwise the icon is a yellow “warning” one):

Note: When you use MQTT, a good point is that, if the device has no connection, the configuration will be propagated to the device when it connects. Try this feature by setting the blue LED on, then switch off the device, then switch it on back: the blue LED will recover its ‘on’ state.

Note: It is also possible to impose a config from a Command Line Terminal, using a gcloud command. For instance, if the device-id is esp32_F70468, if the project-id is hello-cloud-iot-core and if the registry name is weather-devices-registry, the command to switch the LED off is:

gcloud beta iot devices configs update --device esp32_F70468 --project hello-cloud-iot-core --region europe-west1 --registry weather-devices-registry --config-data "{\"on\": false}"

A) 3) Publishing state from device to Cloud IoT Core

Context

We connect one pin of a push button to GPIO4 of an ESP32 DEVKIT V1 and we connect the other pin to 3.3 V.

We chose that the state the device will communicate to GCP-Cloud IoT Core is: “The button connected to my GPIO4 has just been pressed. It was at the Epoch time xxx.”

Let’s consider these 3 points:

• We didn’t add any pull-down resistor between GPIO4 and GND because ESP32 has such built-in resistors (we’ll have to validate programmatically the one for GPIO4).
• When we push and release the button, there is a high to low transition on GPIO4 and this is the type of transition we should detect to initiate a MQTT publication to the state topic.
• With buttons, it’s good to have a debounce time of, let’s say, 200 ms.

The good new is that Mongoose OS “GPIO” API has a method called set_button_handler ([link]) where we can specify all those aspects:

GPIO.set_button_handler(pin, GPIO.PULL_DOWN, GPIO.INT_EDGE_NEG, 200, function(x) {
  print('Button press, pin: ', x);
  // Make MQTT publication to 'state' topic...
}, null);

ESP32 firmware featuring publication to “state” topic

We write the following fs/init.js file to meet our goal. Of course, it could be merged with the previous fs/init.js file we wrote when we dealt with “config” topic. Useful documentation links are provided at the beginning of the source code:

To upload this file to ESP32’s file system, from the folder where there is the fs folder containing the init.js file, we type:

mos put fs/init.js
mos call Sys.Reboot

Seeing the published state in Google Cloud Console

Now, our button is ready to be pressed! So let’s press it! In Cloud IoT Core, where can we see the matching state publication ?

We head to https://console.cloud.google.com/iot/ and then we follow the different steps indicated below, the green figures indicating where to click:

That’s all! Not really hard, is it?

B) ESP32 flash memory encryption and other stuffs for local security

B) 1) Introduction

We already talked about security in the previous post (See [link] and look for “security”): We use TLS for communication between devices and Cloud Iot Core and JWT authentication protected by asymmetrical keys.

Another aspect of security is local security, i.e. the actions undertaken to prevent someone gaining physical access to the device to perform damages : firmware corruption, information theft, credentials retrievals, access to the whole data chain, destruction, etc.

Concerning this topic, there is a lot to say but three points seem the most important to us :

• Consider a device that can have its flash memory encrypted. ESP32 is in this category. Or at least, consider adding to your device a secure element like ATECC608A from Microchip.
• Consider a device handling secure boot. In one word, at each startup the device should check that its firmware is the one that the provisionner uploaded to it. This is especially important after a firmware update. This can be done with an ESP32, not by Mongoose OS but by ESPressif Iot Development Framework (ESP-IDF) : [link].
• In production, prevent access to UART and JTAG connectors by cutting the pins or adding some resin, etc.

Note: This IoT security related video, in french, taken at Devoxx FR 2019, helped us to make this list : [link]. It is made by @disk_91, the runner of the cool IoT blog disk91.com.

In the following paragraphs, we discuss only about the first point. We haven’t tackled the second one yet because of a lack of time, but someday…

B) 2) Reading flash memory

Just to remember, in our 3-post serie dealing with ESP8266 and Firebase, with Arduino style programs (we weren’t using Mongoose OS yet), we showed in the first of these posts (See [link] at section “Security considerations”) how it was easy for an attacker having physical access to the chip to recover sensitive information like Wifi SSID and password. Indeed they were, like everything else, stored in plain text in the flash memory.

With ESP32 and Mongoose OS, if you don’t do anything special, it’s the same situation. For instance, it’s easy to recover the ESP32 private key it uses along with the JSON Web Token to get authenticated with GCP-Cloud IoT Core. It is a serious security issue. Moreover, it is also easy to practice reverse engineering by reading the code stored in plain text. A good point however is that we didn’t manage to recover Wifi credentials.

At the beginning of the following video, Mongoose OS team show these aspects, using their tool called mos to read the memory. It is also detailed in the Mongoose OS web site at the Security section: https://mongoose-os.com/docs/mongoose-os/userguide/security.md

With a 4 MiB flash memory, an ESP32 communicating to host PC thanks to COM18, here are a few examples of commands performing partial or total memory read (from mos tool or from a Command Line Terminal).

# Print to console 2000 bytes from address 0x190000
mos flash-read --port COM18 --platform esp32 0x190000 2000 -

# Save to a file called "flash_content.bin" 2000 bytes from address 0x190000
mos flash-read --port COM18 --platform esp32 0x190000 2000 flash_content.bin

# Save to a file called "flash_content.bin" all bytes
mos flash-read --port COM18 --platform esp32 flash_content.bin

Note: To read our ESP32 DEVKIT V1 development board, we need ESP32 to boot in a specific mode. So before hitting the Enter key to validate the command, we press BOOT and EN buttons on the board. Then we validate the command hitting Enter, then we release EN button, then BOOT button. Some explanations are given here: [link].

Dumping memory content to a file and performing a Search inside it makes private key obtention really easy. For instance, look for the second time the expression -----END EC PRIVATE KEY----- appears in the full file, the private key lies just before it! You can compare it with the content of the file gcp-{device-id}.key, they are the same!

...EzNgaTW1rNxaHeu1lIXg==
-----END EC PRIVATE KEY-----

We also easily retrieve parts of our program, stored in plain text:

// This button handler will trigger function when button is released, with a 200 ms debounce time:
let bh = GPIO.set_button_handler(pin, GPIO.PULL_DOWN, GPIO.INT_EDGE_NEG, 200, function(x) {
  print('Button press,...

B) 3) Encrypting flash memory

Normally you should be now convinced of the necessity to encrypt the memory content!

Happily, ESP32 flash memory is designed to be encrypted and Mongoose OS handles that easily! Another good point for Mongoose OS! Please note that the process of encrypting memory is irreversible and that we have to store the encrypting key in a safe place, if we need someday to reflash the memory.

a. The following command enables flash encryption at next flashing. The device must be connected to the host PC issuing this command. We chose a name for the flash encryption key related to the device id. For instance, if this one is esp32_F70468, we name the key esp32_F70468_fe.key.

mos -X esp32-gen-key flash_encryption_key esp32_F70468_fe.key --esp32-enable-flash-encryption --dry-run=false

This should end up with the message:

Programming eFuses...
Success

b. Then we flash the device. After it reboots, the encryption begins. It takes about 2 minutes long but this will be the only time it is done. The command for our device is:

mos flash esp32 --esp32-encryption-key-file esp32_F70468_fe.key

It should end up with a message like:

All done!

Of course, as we flashed the device, we need to:

• reconfigure Wifi,
• re-upload init.js file to the device,
• register again the device with Cloud IoT core project.

All of this is explained in our previous post ([link]).

Now, a memory read produces nothing readable. Try it yourself!

For security, we encrypted the memory content of all our ESP32s connected to GCP. This is a good habit, don’t postpone this!

C) Bonus: Power considerations

We must be aware that we have a permanent MQTT connection to GCP-Cloud IoT Core (and so a permanent Wifi connection) and we didn’t set up any deep sleep in our ESP32 programs. That being said, with a publication every two minutes, just like in the live demo ([link]), a cheap usb current meter ([link]) says the current consumption is almost 100 mA (under 5 V) and yes, that’s pretty much. 2000 mAh batteries wouldn’t last a full day. As far as we’re concerned, we use mobile phone chargers to power our devices.

Mongoose OS has no API to handle deep sleep but doesn’t prevent from using the ESP32 native APIs. Maybe this solution should be tested to see if consumption reduces significantly. But the programs we wrote would stop being cross platform.

Conclusion

In this small post, we completed the first one dealing with ESP32s running Mongoose OS and managed by GCP-Cloud IoT Core ([link]). Today, we showed the use of special “config” and “state” MQTT topics, a nice way to manage a device. We also showed ESP32 flash memory encryption, making our IoT system even more secure.

My Medium home page is here.

The version of this post you are reading now on www.medium.com is the most up to date. The www.freecodecamp.org version is not maintained.

This post is a step-by-step tutorial for newbies to Google Cloud Platform-Cloud IoT Core. The devices are ESP32 Wi-Fi chips running Mongoose OS. To go through this tutorial, the concepts and then the setup of a simple IoT system measuring weather data are described.

Live demo is here: https://hello-cloud-iot-core.firebaseapp.com/

GitHub for last section (Logging, storing and visualizing weather data with Firebase) is here: https://github.com/olivierlourme/iot-store-display

This post is completed by a second one: see here.

Introduction

1) History

In a previous 3-post series [link, link, link], we used an ESP8266 Wi-Fi chip to regularly measure luminosity and feed a database with the obtained data. The data set was ultimately lively plotted to a web app (see live plot here: [link]). We massively used Firebase products (Realtime Database, Cloud Functions, SDK and Hosting) to meet our goals.

This project worked fine, and thanks to deep sleep it drew very little power. We really enjoyed developing it — but:

• This project was okay to handle just a few connected sensors. Setting up a set of a hundred sensors would require a lot of (rigorous) manual intervention and monitoring/exploiting them would be challenging as well. Indeed, there was no central place where we can manage our system.
• Arduino IDE and Arduino core for ESP8266 were great for discovering ESP8266 but they are quickly insufficient: The IDE file management is really basic, there is only one program in the chip, and there is no Operating System providing useful APIs for IoT.
• FirebaseArduino library, allowing an ESP8266 to push data to a Firebase Realtime Database, was experimental. Some features like authentication should be improved. For now, the “secret” type authentication we used gives ESP8266 admin rights over the whole database!
• Eventually, ESP8266 SPI flash memory was not designed to be encrypted. In our first post [link], we showed how easy it was to recover a Wi-Fi password when reading this memory.

In a word, this past project couldn’t be used in an industrial context. It was more a prototype for a Proof of Concept. We learnt a lot with it but today we’d like to develop a professional and fully secured solution capable of managing in a simple way a lot of connected sensors.

This is why we decided to:

• investigate Google Cloud Platform-Cloud IoT Core [link] to manage our system : devices setup, provision, authentication and monitoring;
• move from ESP8266 to ESP32, which offers memory encryption;
• run Mongoose OS [link] in our ESP32s. This OS accepts programs written in Javascript (JS) and provides a lot of APIs to deal with time, MQTT protocol (the most used in IoT, described a few paragraphs later), sensors, etc. Moreover, Mongoose OS comes with a development tool called mos, allowing to easily set up a device and provision it with Cloud IoT Core.

2) A word about ESP32 Wi-Fi chip

ESP32 Wi-Fi chip is a successor of the famous ESP8266 we described here: [link]. Compared to it, every feature is enhanced (speed up to 240 MHz, two cores, 520 kiB RAM, number of GPIOs, variety of peripherals, etc.) and there are some new ones: Bluetooth (legacy/BLE), 4 MiB-flash memory encryption capability and cryptographic hardware acceleration (AES, SHA-2, RSA, ECC, RNG). There are a lot of resources on the web concerning ESP32. The following one deals with the ESP32 DEVKIT V1 development board that we will use and gives its pinout: [link].

There is also this extensive resource concerning the wide variety of ESP32 chips and development kits : http://esp32.net/ . On their home page, searching for “ESP32 DevKit” or “GeekCreit” leads to a link to the schematic of our ESP32 DEVKIT V1. This development board embeds an official Espressif ESP32-WROOM-32 chip and costs about 6€ at Banggood.

Basic IoT concepts explained through our use case

So, what will be our playground for test all these new tools?

To illustrate IoT concepts through Cloud IoT Core, we chose to build a weather station reporting humidity and temperature from different places.

For simplicity we’ll handle only 2 places: inside our house (“indoor”) and outside our house (“outdoor”). It’s up to you to deal with many more places.

1) Project hardware : ESP32 & DHT22

At each of theses places, we’ll install a connected sensor (“a device”) constituted by a DHT22 humidity/temperature sensor (description: [link], datasheet: [link], 4€ at Banggood) connected to an ESP32 DEVKIT V1 development board. DHT22 observes a kind of “1-Wire” protocol. Each ESP32 will house Mongoose OS for operating system. Its installation on an ESP32, a “Hello, World!” and a test with a DHT22 are given in the section called “Mongoose OS installation on devices”.

Just below are given DHT22 specifications. Afterwards, we think the accuracy figures are a bit optimistic but that’s not our concern today. If you want more accurate sensors, have a look at SHT3X series or BME280.

We can already build the following assembly twice (one for indoor and one for outdoor). For now, power will come from the USB connector connected to our host machine. In production, power may come from a phone charger. A power bank is not a realistic choice as current consumption is pretty high - we show this in the post following this one : [link].

That’s all for hardware! The rest of the project uses serverless solutions from Google. We describe them now…

2) Project architecture : Cloud IoT Core & Firebase

All this “Project architecture” section is theoretical, there is no step to perform. Its aim is to introduce vocabulary and notions related to IoT, more specifically when this domain involves Google Cloud solutions.

Here is the general architecture of our project:

Note: There is no gateways between our devices and Cloud IoT Core because they “speak” MQTT.

Note: Devices can also communicate with Cloud IoT Core via its HTTP bridge. As it is less performant than the MQTT bridge (see a comparison : [link]), we will disallow this communication later during registry configuration. Limiting access just to what is necessary is a good practice.

Let’s explain this architecture in three sections:

• “From Devices to Cloud Pub/Sub” describes the classical Google IoT architecture.
• “From Cloud Pub/Sub to data storage and visualization”, describes the choices we made to exploit data.
• “Additional config and state topics” completes this architecture presentation.

From Devices to Cloud Pub/Sub

• Cloud IoT Core

Cloud IoT Core is the Google Cloud Platform service to which each of our registered devices will send temperature/humidity data. When such a data is sent, we say that the device publishes a telemetry event (sometimes also called a “telemetry message”).

Note: Pricing is detailed here : [link]. For small projects with a few devices, there’s little chance you get charged.

• MQTT

This publication is done through a MQTT connection. MQTT is a publish/subscribe-based message protocol; most of the time it lies over TCP [link] (or better: over TLS, itself being over TCP). The telemetry message has to be published by the device (a MQTT client) to the Cloud Iot Core “MQTT bridge” (a MQTT server) in a MQTT topic whose name imperatively respects this format:

/devices/{device-id}/events

Note: Sub-folders in the topic name are possible. We won’t need this feature here but see [link], as it can sometimes be useful.

{device-id} is unique to each device. In our case, Mongoose OS creates it from the last 3 bytes of the MAC address of the ESP32. For example it could be esp32_ABB3B4.

• Quality of Service (QoS)

The MQTT specification describes three Quality of Service (QoS) levels, when publishing to a topic ([link]):

> QoS 0, the message is delivered at most once;

> QoS 1, the message is delivered at least once;

> QoS 2, the message is delivered exactly once.

Cloud IoT Core does not support QoS 2. And QoS 1 is better than QoS 0. So QoS 1 is the one we will adopt. Mongoose OS can do that.

• Security

Concerning security, in our Mongoose OS/Cloud IoT Core context, MQTT communications are made over TLS ([link]), so (1) the device is assured to be connected to Cloud IoT Core MQTT server (CA’s certificates are stored in Mongoose OS ca.pem file), (2) the data exchange will be private and (3) data integrity will be checked. On the other way, device authentication ([link]) with Cloud IoT Core is performed with a per-device public/private key authentication using JSON Web Token (JWT). The device performs the signature part of the JWT with its private key and Cloud IoT Core validates it using the related public key. Mongoose OS tools handles this keys generation and distribution, we’ll see that soon in the section called “Device registration within the Cloud IoT Core project” lying a few paragraphs below. In this section, we’ll see also how to store securely the private key on the device by performing memory encryption (preventing as well reverse engineering).

Note: Beyond JWT device authentication, for additional security, it’s possible to impose TLS from Cloud IoT Core to devices (so each device has also a public key certificate, etc.). It is an option we won’t use but it’s described here for Mongoose OS side (see “mutual TLS”) and here for Cloud Iot Core side. It’s good to know that AWS IoT imposes this mutual TLS, unconditionally ([link]).

• Registry

Devices sharing the same purpose are regrouped within a registry.

• Cloud Pub/Sub

Telemetry data from all devices belonging to the same registry is then forwarded to a Cloud Pub/Sub topic (Cloud Pub/Sub is a GCP product [link], not specifically a Cloud IoT Core one). The name of the Cloud Pub/Sub topic follows this pattern:

projects/id-of-google-cloud-project/topics/name-of-telemetry-topic

So, if we call our Google Cloud project hello-cloud-iot-core, if we choose weather-telemetry-topic for the name of our Pub/Sub telemetry topic and if finally our registry is called weather-devices-registry, we’ll get sooner or later that kind of view in Google Cloud Console :

But no stress, everything will be explained step by step to reach that.

Note: As it is said here ([link]), each message in the Cloud Pub/Sub topic contains a copy of the telemetry message published by the device but also some message attributes, the most important being probably deviceID, allowing us to match some received data with the device that published it.

Note: We talk a lot about Pub(lish), but where is the Sub(scribe)? In fact, we’ll create quickly with Google Cloud Command Line Interface a Cloud Pub/Sub subscription (a “pull” one) in order to view the messages published to the telemetry topic. Later in this post, we’ll create a Firebase Cloud Function reacting to each publication and this will automatically create another subscription (a “push” one this time).

From Cloud Pub/Sub to data storage and visualization

We’re following the right part of the project architecture diagram given at the beginning of this post:

A publication to the Cloud Pub/Sub topic will trigger a Firebase Cloud Function that will itself fulfill a Firebase Realtime Database with the new data. A web app hosted by Firebase Hosting will lively plot data from the Firebase Realtime Database, in the same way as we did in a previous post: [link].

There are other options in the Google ecosystem to store/treat/visualize data. Alvaro Viebrantz’s really good post [link] that helped us uses Big Query ([link]) and Data Studio ([link]).

Additional “config” and “state” topics

On the project architecture diagram given at the beginning of this post, we see besides telemetry two other data flows: Config ([link]) and State ([link]):

Indeed, the Cloud IoT Core service may publish configuration update messages to a special topic the device has subscribed to ([link]). It is useful when we need the device to go to a new state, e.g. by updating a parameter of its associated sensor, by changing a deep sleep period, moving a servomotor, etc.

For efficiency, there shouldn’t be more than one message of that type per second per device. Such a message is an arbitrary user-defined blob (we’ll use JSON), up to 64 kiB. At last, the name of this special MQTT topic is imperatively:

/devices/{device-id}/config

On the other hand, a device may publish to a special topic — that Cloud IoT Core has automatically subscribed to — messages concerning its state ([link]), e.g. quantity of RAM available, state of a button, etc. It is often used to see if the previous config message sent to the device had the desired effect.

For efficiency, this kind of publication shouldn’t be done more than once per second per device. Such a message is an arbitrary user-defined blob (we’ll use JSON), up to 64 kiB. At last, the topic to which the device publishes its state data has imperatively this name:

/devices/{device-id}/state

Note: Sending commands to devices is also possible from Cloud IoT Core: see [link] but we won’t illustrate it.

But for the moment, we will focus on telemetry. After this journey, in a “coming soon” post we will show how to handle config and state special topics.

UPDATE March 29, 2019: This post about config and state special topics is out: [link].

Mongoose OS installation on devices

1) A short description of Mongoose OS

Mongoose OS ([link], [link]) is a smart IoT-oriented OS, runnable on several chips, including ESP8266 and ESP32. Mongoose OS is in partnership with the major actors in IoT ([link]). It comes with a development tool called mos, really useful to set up the device and then provision it with the IoT platform, working either within a UI or within a Command Line Terminal. In either cases, we’ll write mos commands. There is also a device management app called mDash but we didn’t try it. Numerous APIs dealing with most of the network and sensor protocols are provided. Programs can be written in both C/C++ and JS.

At last, there is a 12-tutorial series on YouTube, really useful:

Note: We used Mongoose OS Community Edition, which is free, licensed under Apache 2.0.

2) Mongoose OS installation on ESP32

This installation has to be performed on each device.

We head to the developers section of Mongoose OS web site ([link]) in order to perform the first seven steps of the list given in this resource:

Step #1, Step #2 and Step #3 are trivial. At Step #3 don’t forget to connect the device to the host machine via a USB cable.

For Step #4 “Create new app”, we choose to call the app app1. When mos clone https://github.com/mongoose-os-apps/demo-js app1 indicated on the web site is completed, mos tool automatically goes to the just created app1/ folder.

In app1/fs/ folder, there is a source file called init.js. It is a demo file capable to communicate with different IoT platforms (if they are configured of course). We will basically test it and soon simplify it for our purposes.

Step #5 “Build app firmware” is launched with mos build command (add --arch esp32 to this command if you launch it from a Command Line Terminal, not from mos tool). It may take a while but normally we have to perform this build only once. After it, we have many more files. One called app1/build/fw.zip contains the binaries of the OS and init.js. It will be flashed to ESP32 in the next step.

Step #6 “Flash firmware” is launched with mos flash command. It has normally to be done just once. Even if later we change some files (like init.js for instance), we will use a mos put command to upload a file from the host machine to the local device’s file system. Of course this command is only available after the flash process.

Note: Firmware flash step can be tricky with a brand new ESP32. With our ESP32 DEVKIT V1, we had messages in console (that’s a first good point!) reporting issues about failing to connect to ESP32 ROM. Retrying to flash by pressing the BOOT button (closed to USB connector) finally turned out to a successful flash. Though, be ready to wait for one minute or two.

Then, the device automatically reboots and executes init.js. We obtained every second the following information in mos console (or in any serial terminal @115200 bds) :

In Step #7 we connect ESP32 to our Wi-Fi network (we use mos tool):

mos wifi WIFI_NETWORK_NAME WIFI_PASSWORD

The device will reboot by itself after getting an IP address and synchronizing time by contacting a SNTP server. We then ping our device to check its internet connexion.

Note: We get device information (IP address for instance) by hitting CTRL+i in mos tool, or by typing mos call Sys.GetInfo.

Note: We reset the device by hitting CTRL+u in mos tool, or by typing mos call Sys.Reboot.

Note: Steps #5, #6 and #7 could be the beginning of a “provisioning script”, useful if we have many devices to setup. It is optional to rerun Step #5 if all devices are the same, e.g. only ESP32.

3) ESP32 “Hello, World!” program with Mongoose OS

To get used to Mongoose OS JS programming style and mos tool, let’s write a small program whose aim is to make the blue built-in LED blink and print messages on console. This led is connected to GPIO2 pin of ESP32 DEVKIT V1 (see Assembly Diagram at the beginning of this post). On our host machine, let’s replace the content of app1/fs/init.js by this one:

From mos tool or from a Command Line Terminal, we upload this file to Mongoose OS file system and finally we reboot the device:

mos put fs/init.js
mos call Sys.Reboot

The blue led should blink and we should see alternatively Tick and Tock printed on console.

4) DHT22 test with Mongoose OS

At the beginning of this post, there is an Assembly Diagram showing how to connect DHT22 sensor with ESP32 DEVKIT V1. We chose to connect DHT22 data pin to GPIO0 of ESP32 DEVKIT V1.

So, here is another short init.js program. This one prints periodically to serial console DHT22 measures (temperature and humidity - as an object in JSON, no MQTT publication yet):

Then:

mos put fs/init.js
mos call Sys.Reboot

And this is the related console we get after uploading the init.js program and rebooting the device. Humidity is in % and temperature is in Celsius degrees:

Numbers seem to have a long decimal part but this will be fixed later within a Cloud Function.

Note: For pedagogical purposes, we chose all over this post explicit long key names like temperature or humidity. This will have consequences on the volume of data stored later in a NoSQL database (Firebase Realtime Database) as those keys will be repeated for each measure. Shorter key names could be a good idea.

5) Let’s publish data to the MQTT telemetry topic

This is our last program, the one ready to work with Cloud IoT Core! On the previous program, we just add a publication to the telemetry topic we already talked about: /devices/{device-id}/events.

Note that messages are published in JSON as it will facilitate later their content retrieval with the Firebase Cloud Function reacting to messages publication.

We name this file init.js, upload it to Mongoose file system, then provoke a reset:

mos put fs/init.js
mos call Sys.Reboot

Note: These commands could be appended to the “provisioning script” we mentioned earlier.

When running, this last program prints data to console but it fails to publish data to the MQTT bridge of Cloud IoT Core (MQTT.pub() returns 0):

Indeed, we haven’t set up any Google Cloud project yet, neither a fortiori registered a single device to it. Let’s do it now!

Cloud IoT Core project setup

1) Google Cloud SDK installation

Firstly we need to install Google Cloud SDK because we will have to type some gcloud commands in a Command Line Terminal. At the time of writing, it requires Python 2.7. It won’t work with Python 3.5. The Google Cloud SDK download page ([link]) offers versions of the SDK with Python bundled inside (if you’re sure you don’t have Python already installed and don’ t want to handle this Python point).

Then, Cloud IoT Core requires some Beta versions of gcloud commands. So in a Command Line Terminal, from any folder, we type:

gcloud components install beta

These two previous steps have to be done just one time!

Note: Most of the following actions on Google IoT Core can be performed in three ways:

• with Google Cloud Console (on the web)
• with some APIs in different languages, and
• with Command Line Interface in a terminal, typing gcloud commands.

We will use the latter to configure things and we’ll check facts with Google Cloud Console (on the web).

2) Google Cloud project setup

We follow now this guide from Mongoose OS web site : [link].

# Commands indicated in this grey frame have to be done just once to configure the Google Cloud project! They can be performed from any folder.

# Get authenticated with Google Cloud
gcloud auth login

# Create cloud project. We chose hello-cloud-iot-core as PROJECT_ID
gcloud projects create hello-cloud-iot-core

# Give Cloud IoT Core permission to publish to Pub/Sub topics
gcloud projects add-iam-policy-binding hello-cloud-iot-core --member=serviceAccount:cloud-iot@system.gserviceaccount.com --role=roles/pubsub.publisher

# Set default project for gcloud
gcloud config set project hello-cloud-iot-core

# Create Pub/Sub topic for device telemetry
gcloud beta pubsub topics create weather-telemetry-topic

# Create a Pub/Sub subscription to the just created topic
gcloud beta pubsub subscriptions create --topic weather-telemetry-topic weather-telemetry-subscription

# Create devices registry (we call it weather-devices-registry)
# Precise Pub/Sub topic name for event notifications
# Disallow device connections to the HTTP bridge
gcloud beta iot registries create weather-devices-registry --region europe-west1 --no-enable-http-config --event-notification-config=topic=weather-telemetry-topic
# Say 'yes' to enable API (if prompted).
# But the last command may however not work
# if you don't enable billing.
# So, follow the link to enable billing and retry last command.
# It should end up to "Created registry [weather-devices-registry]."

3) Device registration within the Cloud IoT Core project

Let’s now register the devices to the project! One at a time of course. mos tool is really helpful for this task. From mos tool launched in its UI or from Command Line Terminal, placed in our app1 folder, we type the following command (project id and registry name are involved, as you see):

# Register device with Cloud IoT Core (do it for each device!)
mos gcp-iot-setup --gcp-project hello-cloud-iot-core --gcp-region europe-west1 --gcp-registry weather-devices-registry

Note: This command could be the last one of the “provisioning script” we mentioned already twice.

This command is a mos command that will itself use gcloud commands. The device about to be registered must be connected via the serial port to our host computer because some information will be uploaded to it just like a private key, MQTT bridge address, etc.

Indeed, we see on mos console that two keys (one private, one public) are generated. We can inspect them in app1 project folder. The private one is for ESP32 and the public one is for Google IoT Core. They are used during the authentication process involving the JSON Web Token we mentioned earlier.

Note concerning security: The private key shouldn’t be stored in plain text in ESP32 flash memory. This is why we describe in the post following this one ([link]) how to encrypt this memory. Also, the private key file shouldn’t be stored in plain text on the host development computer. At least, protect access to its content with a password.

When the device reboots, we see in the console that it successfully connects to the Google MQTT bridge and publishes telemetry messages (MQTT.pub() returns 1):

4) Checking the project setup in Google Cloud Console

We head to https://console.cloud.google.com/iot/ to check that everything was well configured:

Clicking on the Registry ID weather-devices-registry reaches another screen. Clicking on “Devices” on this new screen lists provisioned devices and gives details like the last time they were seen (but this is not live updated, we have to refresh the page):

Otherwise, if we click on the Telemetry Pub/Sub topic name (here: projects/hello-cloud-iot-core/topics/weather-telemetry-topic), we reach the Pub/Sub console, that shows the subscription we created before, i.e. the one related to the telemetry topic:

5) Viewing at last some telemetry data

Now it would be nice to see the data that devices are publishing. For this, we have the subscription we already created. From any folder of the host computer, we type:

gcloud beta pubsub subscriptions pull --auto-ack weather-telemetry-subscription --limit=2

This command ([link]) pulls until 2 Pub/Sub messages from our weather-telemetry-subscription subscription. We can see data in JSON, messages ids and a list of attributes for each message. Among them the deviceId attribute is present. Unfortunately there are no timestamps, we’ll see how to get them later.

If you have reached this milestone, congrats! We’re now ready to write a Firebase Cloud Function that subscribes to the Pub/Sub telemetry topic!

Logging, storing and visualizing weather data with Firebase

1) Introduction

We’re now tackling this part of the project:

On that diagram, we see that our project needs 3 Firebase products :

A Firebase Cloud Function (more exactly “a Cloud Function for Firebase”) must react to any publication to the telemetry topic in order to store the weather data of this publication to a Firebase Realtime Database. This storage allows weather data persistence and is used to feed a web app hosted by Firebase Hosting. This web app draws live plots of this weather data across time.

The good new is that it’s possible to configure all these products with one command.

2) Firebase configuration, GitHub repository

We are still working on the same Google Cloud project called hello-cloud-iot-core. Firebase will just “enhance” this project with its products.

We made a GitHub repository for the Firebase aspects of our project:

olivierlourme/iot-store-display

Clone this repository in your favorite development folder and head to the newly created directory:

c:\_app>git clone https://github.com/olivierlourme/iot-store-display
c:\_app>cd iot-store-display

Global Firebase configuration

Note: We suppose you have Firebase tools installed (i.e. Node.js was installed and then npm install -g firebase-tools was run, see [link] for details).

Let’s perform the Firebase initializations:

c:\_app\iot-store-display>firebase init 

First step is to choose Firebase products we want to use:

We are then prompted to associate the current directory (iot-store-display) with one of the listed Firebase projects. The problem is that our project hello-cloud-iot-core doesn’t appear in the list because before being a Firebase project it’s also a Google Cloud project! Read Doug Stevenson’s posts for relationships between Firebase and Google Cloud: [link] and [link].

To overcome this, first we hit CTRL+C to stop this initialization process and then we go to Firebase Console at https://console.firebase.google.com. We choose “Add a project”:

And we can see our project (with Google Cloud logo) and choose it:

Note: You might then be asked to confirm Firebase billing plan if the Google Cloud project itself has a billing plan.

Great! We restart the Firebase initialization with firebase init command and this time our Google Cloud project hello-cloud-iot-core is listed. We choose it:

Note: If you still don’t see your project you might be logged to Firebase without the correct Google account. In this case, type firebase logout followed by firebase login.

Realtime Database configuration

Then the wizard asks a single question about the Realtime Database and its rules: the name of the file where they will be saved. We opt for the default proposed name database.rules.json. It’s more practical to have these rules in a file lying in the project directory than to go to Firebase Console as we did in past posts. We will detail these rules later.

Cloud Functions configuration

Here are the answers we made to the wizard concerning Functions Setup:

Of course, we choose not to overwrite the functions/index.js file obtained from GitHub.

UPDATE Feb. 21, 2021: Two years after the initial release of this project we had to update the Cloud Function because the version of node.js it used had gone deprecated. For simplicity, we chose not to use Firebase tools but to edit this Cloud Function inline with GCP console. It is summarized here: https://stackoverflow.com/questions/66279936/from-the-gcp-console-can-we-write-a-gcp-cloud-function-interacting-with-firebas

Firebase Hosting configuration

And here are the answers we made to the wizard concerning Hosting Setup:

Of course we choose not to overwrite the public/index.html file obtained from GitHub.

Deploying (not now!)

Later, if we want to deploy some updates we made to our 3 products, we can type globally:

c:\_app\iot-store-display>firebase deploy

But if we want to deploy only, respectively:

• updated database rules,
• updated cloud functions,
• updated web app,

we type, respectively:

firebase deploy --only database
firebase deploy --only functions
firebase serve --only hosting (local deployment) OR firebase deploy --only hosting (remote deployment)

3) Pub/Sub trigger Cloud Function

Introduction

In a past post, we explained that we could write Firebase Cloud Functions triggered on some events happening to some of the Google products.

Post 2 of 3. Our IoT journey through ESP8266, Firebase and Plotly.js

Cloud Pub/Sub is one of these products and so it is possible to trigger a function each time a message is published to a Pub/Sub topic : [link].

So if the Firebase Cloud Function is triggered on each publication to the weather-telemetry-topic topic, watching its log will allow us to watch the telemetry topic’s activity.

The code of the Cloud Function has to store each new published data to the Firebase Realtime Database associated with our project.

Cloud Function source code

The beginning of the source code looks like this:

exports.detectTelemetryEvents = functions.pubsub.topic('weather-telemetry-topic').onPublish(
    (message, context) => {...

The full Cloud Function source code lies in the file named index.js. This file is in the functions folder of our iot-store-display directory on GitHub. It is fully commented, so run and study it, it’s short and not complicated.

Cloud Function deployment

It’s time to deploy the Cloud Function:

c:\_app\iot-store-display>firebase deploy --only functions

Cloud Function validation

Once Cloud Function is deployed, we can watch the Cloud Function logs and among other things, we’ll see the results of the console.log(`Device=${deviceId}...) we wrote at the end of index.js.

Where to see those logs? We have two opportunities:

• in Firebase Console (https://console.firebase.google.com):

• in Google Cloud Console (https://console.cloud.google.com/functions/):

We prefer this latter solution, as logs are clearer:

Concerning storage, here is what lies in Firebase Realtime Database after each device has made 2 telemetry data publication. Data is of course sorted by device as we specified it in index.js:

Note: Don’t forget to delete your Cloud Function on Google servers if you don’t use it, otherwise you might either reach the invocation quota or pay for service you don’t use, as indicated here: [link]. Function deletion is to be performed on Google Cloud Console (see “Delete”, 3 screenshots above).

Note: The Cloud Function has admin rights over the database, whatever is the content of the database.rules.json file. At this step, the database.rules.json file can still be very restrictive. Don’t forget to deploy them, once edited.

{
  "rules": {
    ".read": false,
    ".write": false
  }
}

4) A web app using Firebase and plotlys.js to visualize weather data

Introduction

Note: We’re now building a “homemade” (and satisfying) data visualization solution. For enhanced UI (dashboard, etc.), maybe you should investigate Data Studio we already mentioned elsewhere in this post.

We focus on building a small web app, hosted by Firebase Hosting. This web app lively plots the data stored in the Firebase Realtime Database. We used plotly (https://plot.ly/javascript/) for the plotting library. We are familiar with that work as we already undertook a similar one in a previous post:

Post 3 of 3. Our IoT journey through ESP8266, Firebase and Plotly.js

What’s different today is that we have to:

• draw several charts: temperature vs time and humidity vs time,
• inside each chart, we have one plot per device.

Database rules & devices-ids node

Concerning the database rules, what should be now the database.rules.json file? The device-telemetry node needs to be read by the web app. And if you looked attentively at the Realtime Database screenshot given a few screenshots before, there is another node called devices-ids.

You need to create manually in the Firebase Console this devices-ids node and fill it appropriately for the web app to work properly. It is a simple mean to declare to the web app the devices we want plots for and also to give aliases to the devices. Its role and necessity are fully explained in comments of the public/script.js file given in GitHub.

Note: An improvement could be a form (accessed through authentication) that, once fulfilled, calls a script to generate this devices-ids node.

This devices-ids node also needs to be read by the web app. So the database.rules.json file should eventually become:

{
  "rules": {
    "devices-ids": {
      ".read": true,
      ".write": false
    },
    "devices-telemetry": {
      ".read": true,
      ".write": false
    }
  }
}

These new rules, once edited and saved, must be deployed with:

c:\_app\iot-store-display>firebase deploy --only database

Web app source code

The web app source code lies in the public directory of our hello-cloud-iot-core folder or in GitHub. The content of the folder, especially script.js, is fully commented so you know where to study (and improve!) it.

Note: we have only two devices for this demo but the source code is okay for x devices as long as you declare them in the devices-ids node.

Web app local deployment and validation

For testing purposes, Firebase Hosting can launch a local live server:

c:\_app\iot-store-display>firebase serve --only hosting

We head to http://localhost:5000 and we’re happy to get this:

Web app remote deployment

At last, Firebase offers us the hosting of our web app and an access to it via https:

c:\_app\iot-store-display>firebase deploy --only hosting

We quickly get the public URL of our web app : https://hello-cloud-iot-core.firebaseapp.com

Note: If you have your own domain, you can connect your Firebase web app to it. See [link].

Conclusion

In this post we discovered how to combine ESP32, Mongoose OS and Cloud IoT Core, obtaining a serious, secure and professional IoT project. Now that we know, it can go really fast to provision 10, 100… 1000 devices acquiring weather data all over an area, as long as they can get a Wi-Fi connection. Now, devices are centrally managed, it is easy to provision and monitor them. But we can go further!

Indeed, in addition to this post, there is a second one ([link]). Inside it:

• We’ll focus on ESP32 flash memory encryption, to achieve a fully secured system.
• We’ll see how to use the config special topic, allowing us to trig an action on the device from the Google Cloud Console.
• We’ll see how to use the state special topic, allowing the device to communicate to Google Cloud Console informations about its current state.

We hope you enjoyed this really long post and that you learnt something! Don’t hesitate to ping me if you have any questions or improvement suggestions…

How to check the weather using GCP-Cloud IoT Core with ESP32 and Mongoose OS was originally published in We’ve moved to freeCodeCamp.org/news on Medium, where people are continuing the conversation by highlighting and responding to this story.

TL;DR

My Medium home page is here.

This post stands in a series of 3 as detailed below. These 3 posts reflect the Project Architecture we chose to achieve a simple luminosity logging/live plotting project. We hope they will help developers or students discovering ESP8266 chip and Firebase platform. Moreover, we believe that the solution developed here can be an interesting alternative to the comprehensive Google, AWS and Azure IoT solutions [link, link and link] when we have to manage in a simple way only a few connected devices. (For instance, we won’t set up a MQTT broker neither deal with a device registry.)

Post 1 [link]: We investigate on how an ESP8266 can regularly make acquisition of an analog data and push its 10-bit equivalent value to a Firebase Realtime Database. We finish with considerations on ESP8266 power saving.

Post 2 [link]: We write a Firebase Cloud Function in Typescript language to timestamp the data just pushed to Firebase Realtime Database.

Post 3 (this post): With a web app using plotly.js library, we lively plot in a browser the analog data versus time. The web app is hosted with Firebase Hosting.

Post 3. A web app hosted by Firebase Hosting susbscribes to the data stream coming from a Firebase Realtime Database and plot it.

A) Where we left

At the end of Post 1 [link], an ESP8266 regularly pushed the 10-bit luminosity measure it just made to the measures node of a Firebase Realtime Database:

At the end of Post 2 [link], each time a push was made to measures node, a Firebase Cloud Function associated a timestamp to that push. The obtained object was then pushed to timestamped_measures node of the Firebase Realtime Databse:

Now, that would be nice if a web app, each time a timestamped measure is pushed to timestamped_measures node, could get the last 200 (for instance) timestamped measures in that node in order to replot measures versus timestamps. Firebase SDK makes that subscription really easy and plotly ( https://plot.ly/javascript/) plots data with no pain!

Indeed, at the end of this third and last post describing our ESP8266-based project, we will get this:

B) Angular web app or “classical” web app?

When we hacked this small project (5 months ago), before writing the 3 related Medium posts, we used Angular framework [link] to write the web app. We believed we would use it in this last post of the series, that’s why it is mentioned in the 3 posts URLs. But, as we published our first (4 months ago) and second (2 months ago) Medium posts [link, link], it turned out that this project could be a basis for students discovering IoT, even if they were not initially developers. In that context, we cannot ask them to learn Angular before diving into the IoT topic, that would really be discouraging. So, as our project went eventually more newbies oriented, we will explain here how to write the web app “classically”, i.e. with html, javascript and css files. It’s not possible to change Medium posts URLs afterwards, that’s why “Angular” remains in them and we’re sorry for that. That’s not bad marketing!

Note: If you want to write the web app with Angular all the same, follow the excellent resource mentioned in the Acknowledgments section at the end of this post. We ourselves started with it.

C) Project description and setup

Note: Each time there is an action to perform, it is preceeded by a lowercase letter like a. b. c. etc.

a. Head to the project GitHub repository to see the full source code of our web app: https://github.com/olivierlourme/esp8266webapp. There is even a link to a live demo.

b. Clone the repository in your development folder (or download it if you don’t have git installed):

c:\_APP>git clone https://github.com/olivierlourme/esp8266webapp.git

c. Move to the just generated project directory and launch VS code over it:

c:\_APP>cd esp8266webapp
c:\_APP\esp8266webapp>code .

Note: In Post 2 [link], we had developed the cloud function in a esp8266 folder. We could develop our web app in the same folder as it is just another feature of the project. But to keep presentation clear and posts independent, we chose another folder for the web app.

Note: The files of the web app are all in a public folder in order to prepare Firebase Hosting detailed at the end of this post.

Let’s briefly explain the role of index.html and script.js files present in the public folder. We won’t go much into details as the source code is fully commented on the github repository.

index.html:

• index.html includes the Firebase SDK scripts that allows manipulation of the Firebase Realtime Database of the project. For details, see https://firebase.google.com/docs/web/setup.

<!-- Firebase App is always required and must be first -->
<script src="https://www.gstatic.com/firebasejs/5.5.9/firebase-app.js"></script>
<!-- Add additional services that you want to use -->
<script src="https://www.gstatic.com/firebasejs/5.5.9/firebase-database.js"></script>

• index.html includes the javascript plotting library named plotly.js. This powerful library, built on top of d3.js and stack.gl, makes insertion of sophisticated charts inside webpages really easy. See https://plot.ly/javascript/

<!-- Include Plotly.js -->
<script src="https://cdn.plot.ly/plotly-latest.min.js"></script>

• index.html includes moment.js library. This library can convert timestamps expressed in Epoch time (milliseconds) to human readable dates. See https://momentjs.com. For example: moment(1543658683000).format('YYYY-MM-DD HH:mm:ss') would produce
  '2018–12–01 11:04:43' for a computer with a GMT+01:00 timezone.

<!-- Include the moment.js library -->
<script src="https://momentjs.com/downloads/moment.js"></script>

• index.html has then in its <body> section an identified <div> where the plot will be placed.

<!-- Plotly chart will be drawn inside this div. -->
<div id="myPlot" style="width: 100vw; max-height:75vh"></div>

• index.html includes at last the script.js script containing the essential part of the work.

<!-- Include the script describing the work to do -->
<script src="script.js"></script>

script.js:

• script.js defines in a const named config the Firebase Project Configuration. It doesn’t contain any sensitive information, it is a way for the web app to identify the Firebase project on Google servers. Anyway, it shouldn’t be exposed unnecessarily (for instance, if we use git, we place this const in a file named config.js, we reference it from script.js and we add config.jsto .gitignore).

d. To grab this Firebase Project Configuration, follow the 4 steps below and copy what’s in “5”:

e. Paste these informations to the beginning of script.js. You should end up with something like this:

const config = {
  apiKey: "<API_KEY>",
  authDomain: "<PROJECT_ID>.firebaseapp.com",
  databaseURL: "https://<DATABASE_NAME>.firebaseio.com",
  projectId: "<PROJECT_ID>",
  storageBucket: "<BUCKET>.appspot.com",
  messagingSenderId: "<SENDER_ID>",
};

• script.js then “subscribes” to the value changes in the Firebase Realtime Database node we want to plot: When a change occurs (i.e. a push to the node), an array of the nbOfElts last values of timestamp is made and also an array of the nbOfElts last values of luminosity. Those arrays then feed the x and y part of the plotly trace.

firebase.database()
  .ref('timestamped_measures')
  .limitToLast(nbOfElts) // gives the sliding effect!
  .on('value', ts_measures => {...

Note: The value of nbOfElts is up to you. For instance, if you have a delay of 5 minutes between measures, it makes 12 measures per hour. If you want a sliding plot covering a whole day, you need to assign to nbOfElts the value 24*12 = 288.

Note: Our Firebase web app is made synced with the database since it connects to it through what we call WebSockets [link] and not normal HTTP.

f. Adapt Firebase Realtime Database rules

It’s time to adapt rules that grant access to nodes of our database. At the end of Post 2 [link], we had this:

{
  "rules": {
    "measures": {
      ".read": "false",
      ".write": "false" 
    },
    "timestamped_measures": {
      ".read": "false",
      ".write": "false" 
    }
  }
}

But now, as we want our web app to read timestamped_measures node, with no kind of authentication, the rules have to be changed to this:

{
  "rules": {
    "measures": {
      ".read": "false",
      ".write": "false" 
    },
    "timestamped_measures": {
      ".read": "true",
      ".write": "false" 
    }
  }
}

D) Starting a development server, configuring Firebase Hosting

D) 1) Description

We’ll cover two topics here:

• start a development server to test locally our web app while improving it,
• configure Firebase Hosting in order to have quickly our app hosted on a Google server.

In fact, these topics are linked: As we will configure the use of a Firebase service (here the hosting one), we will benefit from the development server.

a. To be sure to log-in to Firebase with the right account (i.e. the one who created the esp8266-rocksproject), we type first:

c:\_APP\esp8266webapp>firebase logout

Followed by:

c:\_APP\esp8266webapp>firebase login

b. Once logged-in, we associate esp8266webapp folder with the Firebase services we want to use. Here we just want Firebase Hosting. Of course, we use also Firebase Realtime Database in this project, but it has already been included in index.html. So the command is:

c:\_APP\esp8266webapp>firebase init hosting

c. A few questions are asked. Below, we put our responses in bold:

? Are you ready to proceed? Yes
? Select a default Firebase project for this directory: esp8266-rocks
? What do you want to use as your public directory? public
? Configure as a single-page app (rewrite all urls to /index.html)? No
? File public/index.html already exists. Overwrite? No

A few files are generated (and a 404 page as well):

D) 2) Use of Firebase development server

d. Having initiated Firebase allows us to launch a local development server. We do it that way:

c:\_APP\esp8266webapp>firebase serve --only hosting

e. We get the URL to test our web app and we paste it in a browser.

Fantastic! The app works! And each time a new measure arrives, the plot moves to the left.

Note: Now, we cannot have access anymore to the terminal that launched the Firebase development server because it is monopolized by this latter, displaying messages about the incoming connections and requests. If we need to stop the server, we hit CTRL + C. We can also open another terminal if we still need the server to work.

Note: After saving a change made in the source code, we have to refresh the browser to test it because Firebase development server is not a live server like this one : [link].

D) 3) Use of Firebase Hosting

When we’re satisfied with our development, it’s time to host the web app.

f. After hitting CTRL+C to stop Firebase development server, we just have to type this command:

c:\_APP\esp8266webapp>firebase deploy --only hosting

After a few seconds, deploy is complete and we get the public URL of our web app: https://esp8266-rocks.firebaseapp.com . We test it in a browser (you too!), it works perfectly!

Note: If you have your own domain, you can connect your Firebase web app to it. See [link].

D) 4 ) How much data can be downloaded from the database?

With Firebasebase “Spark” pricing plan (the free one), up to 10 GB/month can be downloaded. This can be quickly reached if:

• We have many points in the plot, i.e. if nbOfElts is high.
• We have a high refresh rate for it.
• Many people are connected to our web app!

To know the current downloaded amount of data, we hit Project Overview in Firebase Console:

Conclusion

About this post

That’s all for today! We now have a web app basis to monitor data. But it can be improved in many ways. The first thing to do is to implement Firebase Authentication [link] and roles to handle user/admin access to the web app. Database rules should then be updated adequatly. With authentication feature, the web app could:

• have a form element for the admin to choose the refresh period (that could be a Firebase Realtime Database node written by the web app and read by ESP8266 after each push),
• have a button for the admin to delete the measures and timestamped_measures nodes,
• have a form element for users to set the number of last elements to plot.

Then we could consider the sending of notifications to some registered users. For instance a user could receive a message if luminosity goes below a personalized threshold. This is possible with Firebase Cloud Messaging.

We have already implemented both aspects (authentication/notifications) in other contexts and this was really immediate.

Acknowledgments for this post: We’d like to thank here @angularfirebase (Jeff Delaney) whose videos-lessons are great quality content. Especially, Episode #39 “Realtime Charts With Plot.ly” [link] made us discover plotly.js in an Angular context and gave us the idea to lively plot measures made by an ESP8266.

About this 3-post series

With this third and last post about ESP8266-Firebase topic, we now have the full picture…

• Personally, our system (ESP8266 — Cloud Function — Web App) has been working like a charm for days and days. ESP8266 is supplied with a 10.000 mAh powerbank and just pushes a measure once in a while. It remains in deep sleep mode the rest of the time. Moreover, as timestamps are delegated to a cloud function, at the end ESP8266 consumes really little power.
• Concerning security, though “secret” authentication to Firebase is not perfect, it does the job. The absence of an easy flash memory encryption is the only serious drawback we will regret (cf. post 1, [link]).
• UPDATE (March 15, 2019). To go further and build a more professional, scalable and secure solution, read our next Medium post called GCP-Cloud IoT Core with ESP32 and Mongoose OS.

About writing for Medium

Writing for Medium for the first time was a positive experience. We didn’t realize at first the long process it was to write, choose pictures, refactor source code and comments, etc. to seamlessly present our project. Hacking the whole project took us less than one day. Writing just one post for Medium was a matter of at least 2 full days! But it’s really rewarding when you see that people from many countries, even France :-), enjoyed it and give you positive feedback. And it opened interesting discussions with our job colleagues as well. So, for sure we’ll write some other posts in 2019. Stay tuned, and try yourself writing!

Tip: By default your Medium post title will influence the post URL. If your title is long, you’ll have a long URL for your post provoking sometimes display disagreements due to long URLs. To avoid that, before publishing you can personalize your post URL by selecting Customize story link after clicking on the 3 dots … (close to the bell).

We’ve been really happy to share this newly learned knowledge with you.

Links to the posts of this series: Post 1, Post 2, Post 3, GitHub and Live Demo.

Check also this post: GCP-Cloud IoT Core with ESP32 and Mongoose OS.

TL;DR

My Medium home page is here.

This post stands in a series of 3 as detailed below. These 3 posts reflect the Project Architecture we chose to achieve a simple luminosity logging/live plotting project. We hope they will help developers or students discovering ESP8266 chip and Firebase platform. Moreover, we believe that the solution developped here can be an interesting alternative to the comprehensive Google, AWS and Azure IoT solutions [link, link and link] when we have to manage in a simple way only a few connected devices. (For instance, we won’t set up a MQTT broker neither deal with a device registry.)

Post 1 [link]: We investigate on how an ESP8266 can regularly make acquisition of an analog data and push its 10-bit equivalent value to a Firebase Realtime Database. We finish with considerations on ESP8266 power saving.

Post 2 (this article): We write a Firebase Cloud Function in Typescript language to timestamp the data just pushed to Firebase Realtime Database.

Post 3 [link]: With a web app using plotly.js library, we lively plot in a browser the analog data versus time. The web app is hosted with Firebase Hosting.

Post 2. A Firebase Cloud Function appends a timestamp to each value pushed to a Firebase Realtime Database.

A) The options we had for timestamping data

A) 1) In the previous episode…

At the end of Post 1 [link] of this terrific project, we regularly had a new 10-bit value of luminosity that was pushed into a Firebase RealTime Database — all done by an ESP8266:

As we want to plot luminosity vs time, a timestamp associated with each measured value is necessary. Indeed, we would like something like in the example below:

{
  "timestamped_measures" : {
    "-LHmx0njIsZ3hkCFzjSC" : {
      "timestamp" : 1532010700350,
      "value" : 910
    },
    "-LHmx1PcieyEm04aOo2a" : {
      "timestamp" : 1532010702641,
      "value" : 907
    },
    "-LHmx1leagyrTQxa9zii" : {
      "timestamp" : 1532010703986,
      "value" : 840
    },
    ...
}

Each measure should now be an object (described in JSON) having two key/val. pairs, the keys being "timestamp" and "value". And, at an upper level, this object is the value associated with the key called a “Firebase push ID”, for instance "-LHmx0njIsZ3hkCFzjSC" for the first record in the above example.

The timestamp of a measure should ideally corresponds to the Unix Epoch time of the instant the measure was made.

Note: The Epoch time is the number of seconds that have elapsed since 00:00:00 Coordinated Universal Time (UTC), Thursday, 1 January 1970 [link]. Manual conversions from Epoch time to human time (or vice versa) can be made here [link] for instance. As you may have guessed it in the above example, we will use Epoch time expressed in milliseconds (not seconds) as the tools we’ll use supply timestamps within this unit.

We investigate now two options to associate a timestamp to a 10-bit value just obtained.

A) 2) First option: ESP8266 generates a timestamp and associates it to the just obtained value of luminosity

With this option, ESP8266 should be aware of time, i.e. have a Real Time Clock (RTC, hardware or software recreated). It should loop over the following actions:

1. get the 10-bit luminosity measure.
2. get Epoch time.
3. create an object with "value" and "timestamp" populated with the measure and the Epoch time. The ArduinoJson library we already installed in our previous article [link] can deal with that [link].
4. push this object to our Firebase Realtime Database. This is possible with the FirebaseArduino library [link, link] we already know from our previous article [link]. The function for that purpose has this prototype:

String push(const String &path, const JsonVariant &value)

Steps 1, 3 and 4 should be okay but step 2 is the tricky point. Indeed, when we read this NodeMCU Documentation at RTC Time Module page [link], we realize that RTC in ESP8266:

• is a software module,
• has an accuracy depending on chip temperature,
• needs to synchronize via SNTP or NTP (thus consuming power),
• will lose time in case of an unexpected reset.

Okay! Too much drawbacks! Let’s forget this option and let’s dive into a more realistic one…

Note: Using NTP with ESP8266 is nonetheless described here: [link].

A) 3) Second option: A Firebase Cloud Function issues the timestamp when a value is pushed to the Firebase Realtime Database.

Firstly, what are Firebase Cloud Functions?

Firebase Cloud Functions (or “Cloud Functions for Firebase”, [link]) are part of Google Cloud Platform and are functions executed in a Node environment on Google servers. A Firebase Cloud Function can be:

• called from an app involving Firebase (directly or via a HTTP request),
• automatically triggered upon an event on Firebase products:

You might have already guessed it, what we need is a Realtime Database trigger. Each time a measure is pushed to Firebase Realtime Database by ESP8266, a Firebase Cloud Function will be triggered in order to timestamp the measure. With that solution, we maintain the work done by ESP8266 to the minimum (and thus its power consumption).

The small drawback of this option is that we will timestamp the push to Firebase Realtime Database, not the measure. But the temporal gap between them is only about seconds, depending on network latency. In our previous article [link], we used to make a measure every 5 minutes ; a delay of 1 or 2 seconds on the timestamp is therefore not an issue here.

Note: Firebase Cloud Functions are part of what is sometimes called Functions as a Service (FaaS) and are one aspect of the famous serverless architecture, i.e. we don’t need to provide our own servers to host and run them.

B) Writing our Cloud Function

As written in https://firebase.google.com/docs/functions/ we should:

So let’s follow these steps! But before let’s give a few documentation links:

Get started: write, test, and deploy your first functions | Cloud Functions for Firebase

This page ↑ and its links often include videos from Doug Stevenson. We can find them on YouTube under the following list. It is worth watching all them, Cloud Functions are nicely explained and fundamental concepts like Promises too!

And finally, if you want to become a master in Firebase Cloud Functions 😃, head to this impressive Github repo of Cloud Functions samples (mostly in JavaScript):

firebase/functions-samples

B) 1) Setting up the tools managing Firebase Cloud Functions

To sum up :

Note: Each time there is an action to perform, it is preceeded by a lowercase letter like a. b. c. etc.

a. If not installed yet on our computer, we install Node.js [link]. It will also install Node Package Manager (NPM). Once done, we check version in command line. At the time of writing, it should be ≥8 for Node.js and ≥6 for NPM. To do so, open a terminal. Personnaly, as we develop on a Windows 10 computer, the terminal is cmd.exe. Do it right now with admin rights as some of the future commands will require it. Then, to check installed versions, run from any folder:

node --version

npm --version

b. We install Firebase Command Line Interface (CLI, a necessary set of Firebase command line tools) by running, from any folder:

npm install -g firebase-tools

Again, firebase --version will give the installed version. At the time of writing, it should be ≥4.

-g installs Firebase tools “globally” i.e. we will be able to use the Firebase tools from any project directory. A project directory is a directory on our computer that hosts during its edition the Cloud Function(s) related to the Firebase project we’re working on.

c. We authenticate with Firebase services by running (a few permissions will be asked):

firebase login

Note: Here, we have to authenticate with the same account than the one we used while setting up esp8266-rocks Firebase project in our previous article [link].

Note: Use first firebase logout if you were logged-in with another account.

It should result in this:

d. In our computer development directory (c:\_APP for us), we create a project directory that we call esp8266:

c:\_APP>md esp8266 on Windows (or mkdir esp8266 on Linux).

We navigate to it and run the command to set up the “functions” purpose of the project (same commands for Windows/Linux):

c:\_APP>cd esp8266
c:\_APP\esp8266>firebase init functions

This is what we get:

e. A few questions are asked after we respond Yes to proceed:

• Of course, we choose to associate the esp8266 project directory with the esp8266-rocksFirebase project:

• We then choose TypeScript language. Usually programs for Node environment are written in JavaScript but TypeScript has a lot of advantages over JavaScript, the first one being likely (optional) static typing. In fact, it handles the pillars of Object Oriented Programming. However we have to confess that our function will be so simple today that using TypeScript will provide no real benefit... But the scaffolding for writing later more sophisticated functions in Typescript will have been set up. A short and nice introduction to TypeScript is here: [link].

Note: Check later this blog post [link] from Doug Stevenson for TypeScript pros while writing Cloud Functions. Read also [link] for additional workflow details while using TypeScript with Cloud Functions. You will also learn the key differences between a file in TypeScript and its equivalent in JavaScript ES6 (import <-> require, export const <-> exports.)

• We answer Yes to the next question concerning the use of TSLint. It is a powerful debug tool but unfortunately we won’t have time to tackle with it in this article! At least it will have been set up.
• At last, we accept to install dependencies with NPM: (be prepared to have a bunch of files installed!)

f. Now we’re done! On the graph below we have an (incomplete) tree of the project directory. What’s important to us is that inside the esp8266 project directory, a functions directory was created. In this one we find a src directory that hosts index.ts , the file where we will write our Cloud Function in TypeScript. At the time of deploying the Cloud Function on Google servers, the index.ts file will be transpiled to JavaScript, by a transpiler program called tsc, resulting in the index.js file stored in the lib directory. It’s index.js that will be run by Node.js on Google servers.

The node_modules directory hosts the dependencies (as Node packages/modules) indicated in package.json: firebase-admin and firebase-functions. But it also stores the dependencies of these dependencies… That’s why it is so huge.

esp8266/
 |
 +- .firebaserc
 |
 +- firebase.json
 |
 +- functions/
     |
     +- package.json
     |
     +- src/
     |   |
     |   +- index.ts   # where we will write our Cloud Function
     |
     +- lib/ 
     |   |
     |   +- index.js   # result of index.ts transpilation
     |
     +- node_modules/  # directory where dependencies (declared in
                       # package.json) are installed

B) 2) Writing the Cloud Function

We launch an editor on our project directory. For instance, Visual Studio Code [link] is launched over the esp8266 folder that way:

c:\_app\esp8266>code .

This is what we get:

There is already some (commented) code in index.ts. This is in fact the code of a function called on a HTTP request, not what we need today (but it’s worth testing it along with the first YouTube video embedded above in this post). Remember, we are about to write a Realtime Database trigger function.

Note: If we needed to write several Cloud Functions, they would all fit in index.ts, one after one.

We copy/paste the following code in index.ts. It is fully commented to understand how it proceeds. Of course, we save this file!

B) 3) Deploy, test and monitor the Cloud Function

Cloud Function deployment

In terminal, we run:

c:\_APP\esp8266>firebase deploy --only functions

Unfortunately, there is a known issue while predeploying (i.e. tslint & tsc execution) with Windows 10 (cmd.exe):

This issue is discussed here : [link, link]. We used this sustainable workaround:

• Replace firebase.json current content:

{
  "functions": {
    "predeploy": [
      "npm --prefix \"$RESOURCE_DIR\" run lint",
      "npm --prefix \"$RESOURCE_DIR\" run build"
    ]
  }
}

by:

{
  "functions": {
  "predeploy": "npm --prefix functions run lint && npm --prefix functions run build"
  }
}

• Then, replace in the "scripts" part of package.json the lines:

"lint": "tslint --project tsconfig.json",
"build": "tsc",

by:

"lint": "./node_modules/.bin/tslint -p tslint.json",
"build": "./node_modules/.bin/tsc",

We try again firebase deploy --only functions and this time it’s ok! By curiosity, you can have a look at the obtained index.js file.

Note: It takes about 30 seconds for Cloud Functions to be deployed on Google servers. As we will repeat many times the write-deploy-test process, deploying Cloud Functions locally during development could be an interesting option. We can find more information on Cloud Functions emulator here: [link].

Cloud Function test

In the animation below, we trigger the Cloud Function. To achieve this, we simulate in Firebase Console a new measure pushed to Firebase Realtime Databasemeasures node. Instantly, the Cloud Function detects this creation and push to the timestamped_measures node the object with value and timestamp keys we talked about before. It also assign it a unique push ID. The value of the timestamp is the Epoch time in ms provided by Google servers, it should be accurate.

Cloud Function monitoring

While debugging, it is useful to monitor the execution of the Cloud Function and to console.log() a few informations as we did at the end of our index.ts file:

console.log(`Detected new measure ${original} with pushId ${pushId}`);

Where can we see those logs? In Firebase Console (https://console.firebase.google.com):

How to undeploy a Cloud Function?

Curiously, it’s not possible to delete a Cloud Function from Firebase Console. We have to go to Google Cloud Platform Console (https://console.cloud.google.com), then to choose “Cloud Functions” and then to select the Cloud Function we want to delete:

How many times can we invoke our Cloud Functions per month?

With Firebase “Spark pricing plan” (the free one), we can make up to 125K invocations per month in the limits of 5K invocation per day and 50 invocation per 100 seconds [link]. This thresholds can be quickly reached! To know the current number of invocations (it’s easy to estimate it), we hit Project Overview in Firebase Console:

Note: When the invocation quota is exceeded, the function execution is paused and timestamps will be the ones when the function execution resumes! We should detect the “quota exceeded” event and /or consider the “Blaze pricing plan”.

Hmmh… wait! Do we have to change database access rules ?

First, remember that the Cloud Function we wrote has admin rights and can thus do anything with the database regardless of the rules we’ll set up.

But it’s good practice to set access rules for each node we have in the database, knowing also that .read and .write rules cascade to deeper paths (see [link]).

So, for our new node named timestamped_measures, we set .read and .write to false as, for the moment, no third party needs to access this node. But this will change in the next episode [link].

{
  "rules": {
    "measures": {
      ".read": "false",
      ".write": "false" 
    },
    "timestamped_measures": {
      ".read": "false",
      ".write": "false" 
    }
  }
}

Conclusion

In this article, we wrote our first Firebase Cloud Function to timestamp measures made by ESP8266. No doubt you will adopt them in your own projects as it is an easy way to run code that cannot be run by client side.

As live data is now presented in a good shape, we will plot it dynamically with a web app using plotly.js in the third and last article of this series. Hopefully we’ll post it before Christmas 2018 🎄 🎁! So stay tuned!

Links to the posts: Post 1, Post 2, Post 3, GitHub and Live Demo.

Check also this post: GCP-Cloud IoT Core with ESP32 and Mongoose OS.

TL;DR

My Medium home page is here.

This post stands in a series of 3 as detailed below. These 3 posts reflect the Project Architecture we elected in order to achieve a simple luminosity logging/live plotting project. We hope they will help developers or students discovering ESP8266 chip and Firebase platform. Moreover, we believe that the solution developped here can be an interesting alternative to the comprehensive Google, AWS and Azure IoT solutions [link, link and link] when we have to manage in a simple way only a few connected devices. (For instance, we won’t set up a MQTT broker neither deal with a device registry.)

Post 1 (this post): We investigate on how an ESP8266 can regularly make acquisition of an analog data and push its 10-bit equivalent value to a Firebase Realtime Database. We finish with considerations on ESP8266 power saving.

Post 2 [link]: We write a Firebase Cloud Function in Typescript language to timestamp the data just pushed to Firebase Realtime Database.

Post 3 [link]: With a web app using plotly.js library, we lively plot in a browser the analog data versus time. The web app is hosted with Firebase Hosting.

Post 1. An ESP8266 pushes luminosity measures to a Firebase Realtime Database.

General introduction

It’s been a while since we heard about the ESP8266 component for the first time, its low-cost and its Wi-Fi features. Willing to master it and having at last a little free time, we built a small IoT (Internet of Things) project whose aim is to lively plot an analog data to a web app. Which analog data? Hmmmh, let’s say luminosity in North of France!

A) Hardware considerations

A) 1) ESP8266: a short introduction

ESP8266 [link] is a low cost and low power Wi-Fi SoC integrating a 32-bit Tensilica microcontroller running at 80 MHz. It is designed by ESPRESSIF chinese company and manufactured by companies such as ESPRESSIF, AI-THINKER or DOIT. Due to its embedded Wi-Fi capability, it is widely used in IoT projects and products (in Sonoff or Shelly switches for instance : [link, link]). Until now, ESP8266 is declined in many versions but we use here the widespread one called ESP-12E [link to specs as a pdf].

Note: What do we mean by low cost ? We’ll use a NodeMCU DEVKIT v1.0 board integrating an ESP8266 and many items for development purpose (see a few lines below). It costs between $3 and $4 from classical chinese web sites. A device dedicated to production costs even less.

Note: An ESP8266 successor is ESP32 [link]. Compared to ESP8266, everything is enhanced: speed, memory, peripherics, presence of BLE, memory encryption capability, etc. But it costs about twice.

Besides Wi-Fi, ESP-12E is equipped with all classical peripherals : GPIOs, UART, a single ADC (Analog To Digital Converter), PWMs, I2C, SPI, and so on. It also comes with a 4 MiB SPI Flash memory which we’ll discuss later.

It is important to note that, though ESP-12E is 3.3 V powered, the ADC input (pin #2 on the pinout just below) voltage range is from 0 V to 1 V. The ADC is a 10-bit one so its quantum is about 1 mV (exactly: 1 V /(2¹⁰-1)).

Of course, you can’t use ESP-12E out of the box, a lot of things are missing (at least a supply). So read next paragraph…

A) 2) NodeMCU DEVKIT v1.0 to immediately start development

Really soon (2014) after ESP8266 was released (2013) some people developped (in C and LUA languages) a firmware called NodeMCU [link]. It allows to write scripts for ESP8266 in LUA language. A few months later, an open-hardware platform called NodeMCU DEVKIT v0.9 was added to this project. Now outdated, we used in our project its successor NodeMCU DEVKIT v1.0 [link to schematic as a pdf in github] on which we can find everything to immediately start IoT development:

1. An ESP-12E.
2. A USB to Serial Converter to communicate with a computer (for console and program flashing purposes); most of the time it’s a CP2102 chip but sometimes it’s a CH340. Installing a driver is sometimes needed.
3. A MicroUSB-B connector to connect our NodeMCU DEVKIT v1.0 to our computer during development/debug. You will need a cable.
4. A 3.3 V regulator to make 3.3 V from 5 V USB (typical reference: AMS1117).
5. A “flash” button (we didn’t have to use it) and a useful reset button.
6. An active low built-in blue LED, connected to D0 (GPIO16) (see pinout below, after NodeMCU DEVKIT v1.0 photo)
7. A voltage divider (not referenced on image below) whose role is explained just after.
8. …

On the above pinout, A0 is the (upper left) pin where you connect the analog input you want to measure. A0 is not directly connected to pin #2 of ESP-12E, there is a 100k/(100k+220k) = (1/3.2) voltage divider between them [link to schematic as a pdf in github, see part called “ADC”]. So, on A0 of NodeMCU v1.0, the input voltage range can be from 0 V to 3.2 V.

To conclude this hardware presentation, let’s say it is sometimes confusing to deal with the numerous NodeMCU DEVKIT versions and manufacturers. This post [link] explains things well and also presents ESP8266 alternative developpement boards like Adafruit Feather HUZZAH with ESP8266 (supplied by a 3.7 V LiPo battery or 5 V USB + power management included) or WeMos mini. It also gives links to at last buy our first NodeMCU DEVKIT v1.0!

B) Software choice: Arduino language and environment

LUA is without doubt an interesting language but what if we could program ESP8266 with a language and an IDE we all already know, eliminating long learning hours ? Let’s say something like Arduino. This is possible thanks to “Arduino core for ESP8266”! And it has a fantastic network library!

Note: There are numerous other development options than NodeMCU and Arduino. See “SDKs” and “See also” parts in [link].

B) 1) Installation

The official “Arduino core for ESP8266” github repo is here : [link]. We follow the part called Installing with Boards Manager. At the end, when everything is installed and the NodeMCU DEVKIT v1.0 is selected as our development board, we get something like this:

Note: There is no Arduino firmware flash to care about. And if there is already a firmware installed on our NodeMCU DEVKIT v1.0 when we get it, it will be deleted the first time we upload an Arduino program.

B) 2) Time for “Hello, World!” program

In this “Hello, World!” program (a program is called a “sketch” in the Arduino world), we will make the NodeMCU DEVKIT v1.0 built-in blue LED blink and print a message to Arduino IDE Serial Monitor each time this LED is about to change its state. This program is inspired from:[link] (or we can reach it by: Arduino_IDE>File>Examples>ESP8266>Blink).

Note: Each time there is an action to perform, it is preceeded by a lowercase letter like a. b. c. etc.

a. With Arduino_IDE>File>New, we create a new program and copy/paste the following code. We save it.

b. If not done yet, we connect via USB our NodeMCU DEVKIT v1.0 to our computer.

c. With Arduino_IDE>Tools>Port, we choose the (emulated via USB) serial port we think our computer uses to communicate with NodeMCU DEVKIT v1.0, for instance COM6 on Windows.

d. Select Arduino_IDE>Sketch>Upload (or hit the famous Arduino right arrow icon). If the LED is not blinking after several seconds, we should change serial port and retry.

If the LED remains in the same state, we try changing upload speed with Arduino_IDE>Tools>Upload Speed and retry. Normally it’s 115200 Bd but it depends on the firmware our NodeMCU DEVKIT v1.0 is shipped with.

At this point, the LED should blink!

B) 3) Memory

Let’s say a word about ESP-12E memory. On Arduino IDE status bar, we have this:

• “NodeMCU 1.0 (ESP-12E Module)” is the board we work with.
• “115200” is the program’s upload speed, here 115200 Bd.
• “4M” is the size of SPI flash memory in ESP-12E, i.e. 4 MiB.
• “(3M SPIFFS)”: From the 4 MiB mentionned before, 3 are here reserved (this can be changed) to be handled by SPI Flash File System (SPIFFS). For instance, if we plan to use ESP8266 as a web server, our web site would be stored somewhere in those 3 MiB. So we have 1 MiB left to store our Arduino program and libraries.
• “COM6” is the serial port our computer uses to communicate with NodeMCU DEVKIT v1.0.

After uploading we have also these sentences in Arduino IDE:

Sketch uses 224281 bytes (21%) of program storage space. Maximum is 1044464 bytes.
Global variables use 31768 bytes (38%) of dynamic memory, leaving 50152 bytes for local variables. Maximum is 81920 bytes.

• The first and second sentences effectively show that we have 1 MiB for our program and libraries. We learn that we used 21% of this.
• The third and fourth sentences show that we used 38% of the 80 KiB available RAM.

Note: Some memory can be EEPROM for sketches to store data.

B) 4) End of “Hello, World!” test

Now, let’s see the messages ESP8266 is sending us.

e. We select Arduino_IDE>Tools > Serial Monitor. When the Serial Monitor appears, we set its serial speed at 115200 Bd:

This Serial Monitor speed is of course the same that the one we set for ESP8266 in this line of our program:

Serial.begin(115200);

f. And this is eventually the scroll we get in Serial Monitor:

C) Measuring an analog data like luminosity

Let’s go a step further by measuring an analog data with our NodeMCU DEVKIT v1.0.

The analog data wire should be connected to pin A0 of NodeMCU DEVKIT v1.0. Thanks to the voltage divider we mentionned in A)2), the analog data should be between 0 V and 3.2 V:

• 0 V produces the minimal integer 10-bit value, i.e. 0.
• 3.2 V produces the maximum integer 10-bit value i.e. 1023.

For instance, 2.5 V produces: 2.5 * (1023/3.2) = 799.

As we said before, we want to measure luminosity (or, more accurately, the voltage that is an image of it).

This document from Adafruit [link to pdf] explains how a photo cell (or Light Dependent Resistor, LDR) can be used to measure luminosity (i.e. how you can relate its ohm or voltage values to luminosity) and how you can connect it to a microcontroller analog input. It even gives Arduino program examples.

So this is the assembly we will adopt (no need of a voltage follower between LDR and NodeMCU DEVKIT v1.0, we checked that point):

• Total darkness should produce a 10-bit equivalent value close to 0.
• Direct strong sunlight should produce a 10-bit equivalent value close to 1023.

With a small breadbord, we can get this compact installation (in fact we’ve been given the whole hardware in a @RedisLabs meetup [link] hosted by @ZenikaLille, we thank both of them here!):

Note: A LDR is not the best luminosity sensor in town! But for sure, it’s the cheapest!

g. The program can be as simple as that:

h. And eventually, after we upload this program to NodeMCU DEVKIT v1.0, we get the following sreenshot of Arduino IDE Serial Monitor (we produced different luminosity environments - covering LDR, adding light, … - to get this values):

D) Connecting NodeMCU DEVKIT v1.0 to our LAN and Internet

D) 1) Procedure

Serious things begin now! To connect our NodeMCU DEVKIT v1.0 to our LAN, this latter should of course have a Wi-Fi access point. Also, our LAN gateway should run a DHCP server to give an IPv4 address to our component when it asks for one. And if we want our component to go to Internet though we have only one public IP address, our gateway should implement NAT. Nothing to worry about, almost all domestic Fiber/ADSL boxes do that naturally.

To get connected to our LAN, we just have to copy the void setup() function of one of the “Arduino core for ESP8266” examples, for instance: [link] (or you can reach it by: Arduino_IDE>File>Examples>ESP8266Wifi>WifiClient).

The short program looks like this, don’t forget to replace the values of WIFI_SSID and WIFI_PASSWORD by yours:

And this is the kind of screenshot we get on Serial Monitor:

Now, we encourage you to continue this example and implement totally a TCP client or even a web server, finding help in “Arduino core for ESP8266” examples and Internet. By lack of time, we won’t do it here (but again, do it!). We prefer focussing on a client that connects to a Firebase RealTime Database…

D) 2) Security considerations

Hard coding Wi-Fi SSID and password in the sketch might be a problem:

• if we want to communicate the sketch to others, we have to mask these informations,
• if the Wi-Fi configuration changes, we have to reprogram the device.

To avoid these drawbacks, there is a solution called WifiManager [link], we didn’t personally test it but it has many stars on github.

Anyway, whatever the manner Wi-Fi SSID and password are specified with, at the end they lie in plain text in the flash memory. If a person with bad intention has physical access to the chip, it is really easy for it to get all credentials.

For instance, we dumped our 4MiB flash memory to a file with the use of esptool [link], an official Espressif tool:

esptool.py -p COM6 read_flash 0 0x400000 flash_contents.bin

Then, searching in an editor the Wi-Fi SSID, we find close to it the Wi-Fi password:

Later, we will also have to store credentials that give access to a Firebase database. The problem will be the same as here… they will be easily discoverable.

How can we protect credentials from hackers having physical access to the chip ?

• ESP8266 doesn’t support native flash encryption. A possibility is to make it work with an encrypting device like ATECC508A or ATECC608A [link] from Microchip/Atmel. Mongoose OS (a step further from this present post) uses this solution to encrypt sensitive information when dealing with ESP8266 [link].
• But maybe it would be simpler to upgrade to a ESP32 chip (we talked about this ESP8266 successor at the beginning of this post), a device with native encryption capability. Still with Mongoose OS, we explain in this post [link] how to encrypt an ESP32 SPI flash memory.

E) Connecting and pushing to a Firebase Realtime Database

E) 1) What is Firebase Realtime Database?

Firebase [link] is a huge Google platform to help build sophisticated, mobile and web realtime apps. Each Firebase project has one online NoSQL databases (to chose between Firebase Realtime Database, a simple yet powerful database, or Cloud Firestore, a more sophisticated, document oriented database), handles authentication, cloud messaging, cloud storage, cloud functions, etc.

Note: Firebase is deeply linked to Google Cloud. To learn more about their relationship, read this post from Doug Stevenson : [link].

What interests us here is the first online NoSQL database called Firebase Realtime Database because we will push into it each new (10-bit equivalent) analog data NodeMCU DEVKIT v1.0 has just measured.

For the moment, this is a basic usage of this database (it is capable of much more!) but the good point is that new data won’t be stored in NodeMCU DEVKIT v1.0. Thus, we can have a really big measurement history without being concerned by our component memory limitation.

Another good point is that we will use later the “real time” functionality of this database to :

• add a timestamp to each new pushed value. This is discussed in a second post dealing with Firebase Cloud Functions [link].
• plot to a web app data as it arrives. This is discussed in a third post dealing with Firebase SDK [link].

E) 2) Let’s store data!

After NodeMCU DEVKIT v1.0 obtained an Internet connection, it will connect over a secure connection and with credentials to our Firebase Realtime Database.

Then, for each captured value of luminosity, it will push it to the Firebase Realtime Database with the following process: In the database, each incoming value is associated with a unique key, called a push ID. Then, this key/value pair is stored in a node of the database.

Each key is unique, has 20 characters and keys are chronological. If you’re curious, this post [link] describes how push IDs are generated.

For instance, after NodeMCU DEVKIT v1.0 made 3 pushes in the node named measures in the database, the latter looks like this, in JSON:

{
  "measures" : {
    "-LH2fJPeouqr_JPE6XXZ" : 852,
    "-LH2gSuEplF4sAHWaGaE" : 801,
    "-LH2hbPTIfpIj0Ta6OsK" : 728
  },
  "another_node" : {
    ...
  }
}

Of course, push is not the only avalaible operation in Firebase Realtime Database. You can update a node, listen to events on it (e.g. child added), delete it... There is a Firebase SDK to perform all these actions for several languages (JavaScript, C++ for instance) and operating systems (Android, iOS).

Unfortunately, in Firebase documentation, there is no mention of something like “Firebase SDK for Arduino core for ESP8266”. But, but… a library called FirebaseArduino exists and its github repo is here [link]!

Nevertheless, it’s written:

“ The Arduino library is under heavy development, experimental, unversioned and its API is not stable.”

Ok! That’s why experimental projects are for! Let’s go ahead.

E) 3) Setting up a Firebase project and its Realtime Database

Note: We won’t do it here but it is good practice to have a Firebase project for development and another one for production.

Below are detailed the steps we went through:

a. Create a Firebase project by going to Firebase console: https://console.firebase.google.com

b. Click the Add project blue icon and then give a name to it, for us, this will be esp8266-rocks:

c. Click on Database on the aside menu:

d. Choose Realtime Database, not Cloud Firestore. Then Create database:

e. Set read/write rules to true for the moment. It’s not really important as we will come back later to this:

f. We end up with a graphical view of our (still empty) database:

Note: We can discover our database by adding key/value pairs clicking “+” in the above screen. We can also nest data, have objects, arrays, etc. If we export this database we’ll see that it is in fact a Javascript object described in JSON.

Note: With Firebasebase “Spark” pricing plan (the free one), we can store up to 1 GB in a Firebase Realtime Database. To know the currently used storage , we hit Project Overview in Firebase Console:

g. Let’s fix the read/write rules now by clicking on Rules tab.

{
  "rules": {
    "measures": {
        ".read": "false",
        ".write": "false"
    }
  }
}

What does this mean?

Those rules mean that a third party with no admin privileges, e.g. a script on a website, an Android app… trying to write to/read from measures node in the database will be denied. On the contrary, with admin privileges, all write to/read from any database node are permitted!

So, who has admin privileges?

• The owner of the Firebase Project (us!) manipulating the database from the Firebase Console.
• Some Cloud Function like the one we’ll write in Post 2 [link].
• An entity accessing the database with a “secret” type authentication. This is the case of an ESP8266 using FirebaseArduino library (described below).

h. At least, it’s time to get authentication credentials for our NodeMCU DEVKIT v1.0. We adapt here the (outdated) “configuration” in FirebaseDemo example from FirebaseArduino github repository [link].

• What we will call later in Arduino program FIREBASE_HOST corresponds to the Project Id appended by .firebaseio.com , for example esp8266-rocks.firebaseio.com. It’s the URL to the database.
• What we will call later in Arduino program FIREBASE_AUTH corresponds to the “secret” we obtain by the 5-step sequence below:

As indicated, getting authenticated by a Project ID and a secret is deprecated. The reason is that when a device gets authenticated that way, it has admin privleges on the database and this is not a good practice. Moreover, using the “secret” authentication technique deprives us from the subtility of the Firebase Security Rules feature (see [link]).

But at the time of writing, it seems that there is no other such ready-made solution for authenticating an ESP8266 Arduino to Firebase. On the FirebaseArduino API Reference [link], the only function to connect to a Firebase project is Firebase.begin(..., ...) and it requires two parameters like the ones we’ve seen before:FIREBASE_HOST (a URL) and FIREBASE_AUTH (a secret). It is said however that this latter can also be a token (and so we wouldn’t use anymore this deprecated solution) but its management is not so easy, given our project.

Class Documentation - firebase-arduino 1.0 documentation

Discussion about updating the authentication process in FirebaseArduino (i.e. no more use of a secret) is active on github [link]. Informations on this topic are welcomed.

Note: We can have several secrets for one database (thanks to “Add secret” button on the view above), for instance one per ESP8266 device connected. In that case, the database should then be managed to have a supplementary nesting level, just like in the view below. But, one more time, with “secret” authentication, device #1 has access to device #2 data and vice versa.

{
  "measures" : {
    "device01": {
      "-LH2fJPeouqr_JPE6XXZ" : 852,
      "-LH2gSuEplF4sAHWaGaE" : 801,
      "-LH2hbPTIfpIj0Ta6OsK" : 728
    },
    "device02": {
      "-LTOa64c9TKkr9s3_3Xn" : 16,
      "-LTObDyJrL_dsHw6mZ4k" : 13,
      "-LTOcLvAP49gBd6U9OhE" : 18,
      "-LTOdTsrDe_SEybwoXTI" : 14
    }
  }
}

E) 4) Setting up Arduino IDE to use FirebaseArduino library

We adapt here the (outdated) “software setup” documentation in FirebaseDemo example from FirebaseArduino github repository [link].

a. First we download FirebaseArduino library:

https://github.com/googlesamples/firebase-arduino/archive/master.zip

b. Then we install it in Arduino IDE: Arduino_IDE>Sketch>Include Library>Add .ZIP Library… and we choose the zip we just downloaded.

But Firebase Arduino has a dependency called ArduinoJson library [link]. We also need to install it in Arduino IDE.

c. First we download ArduinoJson library:

http://downloads.arduino.cc/libraries/github.com/bblanchon/ArduinoJson-5.13.2.zip

(Another possible link for this library is [link].)

d. Then we install it in Arduino IDE: Arduino_IDE>Sketch>Include Library>Add .ZIP Library… and we choose the zip we just downloaded.

E) 5) NodeMCU DEVKIT v1.0 program to push values to Firebase Realtime Database

In FirebaseArduino github page, there is several example and one of them is called FirebaseDemo [link]. It illustrates lot of functions of this FirebaseArduino library. We encourage you to explore theses functions (and see lively their effects in Firebase console) as we will only focus on the one we need in our project.

Indeed, among these functions we will just use the one called pushInt(). For example, to push the value of the int (integer) variable named sensorValue to the node "measures" of the Firebase Realtime Database, it is a simple as that:

Firebase.pushInt("measures", sensorValue);
if (Firebase.failed()) {
  // Do something if push went wrong
}

We can thus complete our luminosity measurement program:

WAAO! Look what we get in Firebase Realtime Database Console:

So, it seems that luminosity in North of France isn’t so bad, no ?

Troubleshooting

If things don’t work, you will get the message "pushing failed:..." in Arduino IDE Serial Monitor.

• In Arduino program, check your credentials FIREBASE_HOST and FIREBASE_AUTH.
• Try setting temporarily your database rules ."read" and .”write" to "true".
• UPDATE (November 27, 2018 - three months after this post was initially released) : Today we wanted to make our NodeMCU DEVKIT v1.0 work again. But, although everything was set up like before,we only got tons of "pushing failed:..."! Searching on the web, we managed to solve this issue. It was due to a Firebase certificate fingerprint that had changed. We can find this latter by typing our project URL appended by /.json in a browser and hit the lock icon to access certificate details. On the other side, in our FirebaseArduino library folder firebase-arduino-master/src, there is a file called FirebaseHttpClient.h. Inside it, there is a line where the fingerprint should be declared (hexadecimal digits by pairs). It has to match the one seen in the certificate details. That was no longer the case!

F) Let’s save power with ESP8266 deep sleep mode

When we need to push a new measure every 3 seconds (to be exact : measure then push then delay 3 seconds), it’s not worth putting ESP8266 Wi-Fi off after the push is done. ESP8266 would lose seconds to reconnect and we would miss some measures. But we have to be aware that Wi-Fi draws a lot of power. For instance, ESP-12E consumes 120 mA while transmitting via Wi-Fi N. In such a situation we can’t run in an autonomous way for days and days with a USB (solar) power bank.

Things are different when we need a new measure every 5 minutes (for instance). In this case it is worth putting ESP-12E in deep sleep mode (consumption: 10 µA) after the push to database is done:

// Measure analog value:
// ...
// Push value to database:
// ...
// Now let's go in deep sleep for 5 minutes:
ESP.deepSleep(300e6, WAKE_RF_DEFAULT); // 1st parameter is in µs!

ESP.deepSleep() instruction puts ESP-12E in deep sleep mode during a time indicated in µs (first parameter). The WAKE_RF_DEFAULT mode (second parameter) ensures that Wi-Fi will be restarted when the chip wakes from deep sleep.

Once the deep sleep duration is achieved, D0 (GPIO16) pin of NodeMCU DEVKIT v1.0 will issue a high to low transition. So D0 pin has to be tied to RST pin in order to provoke a chip reset and make the program start again:

The overall average current will drop to mA and supplying the NodeMCU DEVKIT v1.0 by a power bank is now feasible. If we have time, we will make measures and update this post.

Important note: It’s a shame but D0 (GPIO16) controls as well the built-in LED of NodeMCU DEVKIT v1.0. Due to D0 to RST connection, we must forget the idea of managing the built-in LED as we did until now, otherwise we will provoke a chip reset when we set the built-in LED on!

Important note: While uploading the program from Arduino IDE to NodeMCU DEVKIT v1.0, the D0 to RST connection has to be removed otherwise upload won’t work.

One last thing: The code has to be refactored because we regularly provoke a reset. Thus, all code takes place in void setup() function. The void loop() function is an empty one.

The code is below. Try it now:

Note: When concerned with power saving, Feather HUZZAH development board (cf. part A) 2)) could be a good choice because it has a 3.7 V supply input. Sadly, we didn’t have time to investigate this solution.

Conclusion

We’ve travelled a lot to discover different technologies (ESP8266, Arduino IDE, Firebase Realtime Database) and link them together to build the fundations of a low cost ans low power IoT project! And, even if we’re not fully satisfied with the ESP8266 “secret” authentication, it is a secure project… as long as a hacker doesn’t have physical access to the ESP8266 chip.

There is more to come:

• In a second post we’ll see how to add a timestamp to each measured value of luminosity thanks to a Firebase Cloud Function.
• And in a third one we will lively plot luminosity values as they arrive, thanks to a web app using plotly.js.

So stay tuned!

Links to the posts: Post 1, Post 2, Post 3, GitHub and Live Demo.

Check also this post: GCP-Cloud IoT Core with ESP32 and Mongoose OS.