Disclaimer: Before I dive into the details, let’s set some ground rules:

1. Educational Purpose Only: This blog post serves as an educational resource. I’m sharing my experiences, but I’m just a hobbyist, not a professional. Always consult experts for specific advice.
2. Assumption of Risk: By following my instructions, you assume full responsibility for any consequences. DIY projects involve risks, so proceed with caution.
3. Safety First: Handle electronic components carefully. Use appropriate tools and safety gear. Be aware of electrical hazards. Follow manufacturer guidelines for microcontrollers and sensors.

Objectives

There are many prior projects of creating HAL 9000 replicas, like the very precise 3D data, the affordable replica and the talking replica from Adafruit, and the unsuccessful but good-looking crowdfunding project. This time, I’m going to use Moebius Models 1:1 HAL 9000 kit, which I bought on Amazon.

It’s already almost complete and easy to assemble for a casual DIYer like me. It comes with an LED for its iconic eye, but nothing else. It does not have even a back panel. So, I’m going to achieve followings in this project.

• Adding environment monitoring: I’m going to add capabilities to measure all the environment metrics (Temperature, Relative Humidity, Illuminance, CO2, VOC Index, NOx Index, PM 0.5/1.0/2.5/4.0/10.0, Pressure and Presence) by using I2C and UART sensors.
• Adding a microphone and a speaker: Since HAL 9000 is supposed to have listening and speaking capabilities, I’m going to use an I2S microphone, an I2S amplifier and a small speaker to add at least Ambient Sound Level monitoring and sound streaming capabilities, and possibly Voice Assistant capability.
• Adding buttons: Although HAL 9000 is supposed to communicate with us through voice interface, it will be nice to have some physical buttons for some use cases. I’ve already built my DIY smart button in another post. If I add buttons, I can configure them later for any use case. They will be useful at least for speaker volume control.
• Adding LEDs: Although the HAL 9000 model already has an LED for its eye, it will be nice if it can change the light of the eye and have additional ambient light around it. So, I’m going to use addressable LEDs to add more lighting capabilities.
• Adding an emergency battery: Since HAL 9000 is supposed to work even in an emergency situation, I’m going to add a UPS module and a battery so that it could work even when there is a power outage. I’m not trying to make it completely battery powered. It’s just for an emergency battery so that HAL 9000 can work for a short period of time after a power outage.
• Adding an RTC (Real Time Clock): Since HAL 9000 is supposed to have some level of autonomy even in an emergency situation, I’m going to add an RTC module so that it could recognize time even when there is a network outage or a power outage.
• Creating a PCB board: I’m going to design a custom PCB board to connect all the components as I tried in my V2 sensor node.
• Creating a back panel: This is the most important part of this project. Since the HAL 9000 model does not have a back panel and I want to keep the HAL 9000 model intact, I’m going to design a custom back panel by using 3D CAD for the first time and 3D print it so that I can put all the components on my custom back panel without modifying anything of the HAL 9000 model. Designing the entire enclosure is still too much for me. But designing only a back panel will be relatively easier because the back panel is hidden behind the good-looking HAL 9000 model.

Environment Monitoring

Based on my experience in V1/V2 sensor nodes and comparison of sensor boards, I’m going to use SHT41 for Temperature and Relative Humidity, BH1750 for Illuminance, SCD40 for CO2, SGP41 for VOC Index and NOx Index, SPS30 for PM 0.5/1.0/2.5/4.0/10.0, BMP280 for Pressure and LD2410C for Presence.

Regarding SHT41 and BH1750, I’m going to use sensor chips and create custom sensor boards so that I can expose them to the outside of the HAL 9000 model. It also allows me to reduce redundant pull-up resistors. Especially, exposing SHT41 to the outside will be an improvement from V1/V2. But it might be still possible that there could be thermal conduction through the warmed air inside the HAL 9000 model or through the back panel. So, I’m going to try an exhaust fan again as I tried in V2. Based on my failure in V2, I’m going to use a bit larger blower fan this time. I bought an affordable 5V powered 30mm PWM 4 pins blower fan from AliExpress, which seems to be aimed at Raspberry Pi 5 Active Cooler.

In terms of SCD40, SGP41, SPS30, BMP280 and LD2410C, I decided to use affordable sensor boards or sensor modules which I bought from AliExpress before US-China trade war.

Microphone, Amplifier and Speaker

Like V2, I’m going to use an ICS-43434 sensor board from Adafruit for an I2S mic. When it comes to an I2S amp, MAX98357A seems to be popular in online communities. I’m going to use an affordable one from AliExpress. It looks exactly the same with one from Adafruit for some reasons. Regarding a speaker, there are some options for MAX98357A — 3W 4Ω or 8Ω. The HAL 9000 model has a kind of speaker grille on its bottom, but its size is limited. There will be only a limited space to put a speaker behind the speaker grille. I’m not sure if a small speaker will really work, and I’m not sure either one big mono speaker unit or a small stereo pair of speaker units will work better. I’m definitely not an expert of audio. I just decided to use 3W 4Ω stereo speaker units that I found on Amazon. And I bought two MAX98357A for stereo.

As I may try Voice Assistant with a mic, amps and speaker units, I’m going to use a powerful ESP microcontroller board this time — Lolin S3 which has more memory than D1 mini or S3 mini that I used in V1/V2. I have used Lolin S3 in another project to monitor car data and it has been working great since then.

Regarding connections between ICS-43434, a pair of MAX98357A and Lolin S3, there seem to be some options for I2S communication. Since I2S is a bus protocol, we can define just one I2S interface for both audio input and output among a mic and amps. Which means that we need only one pair of GPIO pins for LRCLK (Left/Right Clock) and BCLK (Bit Clock). On the other hand, we can also define multiple I2S interfaces by using ID. For example, one for audio input and another one for audio output. In that case, we need a pair of GPIO pins for LRCLK and BCLK of audio input and another pair of GPIO pins for LRCLK and BCLK of audio output. Which means that we need 4 GPIO pins. In my case, I may use an I2S mic for Ambient Sound Level monitoring and amps for sound streaming. This use case requires two I2S interfaces at the time of this writing. And I may use an I2S mic and amps together for Voice Assistant. Either one I2S interface or two I2S interfaces would work for this use case. Since I’m still not sure either one of use cases would fit in my daily life, I decided to define two I2S interfaces for now. In addition to GPIO pins for LRCLK/BCLK, we also need GPIO pins for data — data input for a mic and data output for amps. And it seems that we don’t need separated GPIO pins for stereo data output. So, I need to use 6 of GPIO pins for I2S in total — 2 pins for LRCLK/BCLK of a mic, 1 pin for data input for a mic, 2 pins for LRCLK/BCLK of amps and 1 pin for data output for amps. For amps, I use the same GPIO pins for a pair of MAX98357A amps. As I describe later, I can configure each MAX98357A amp to output left, right or mixed channel by adding resistors.

And I’m going to use a buzzer as well for just in case. Though it might be unnecessary since I use a speaker.

Buttons

In my DIY smart button project, I used arcade style buttons. Although it has been working well since I installed it in my house, arcade style buttons won’t fit in the appearance of HAL 9000. Based on the good-looking prior example, I decided to use a square LED button. And I found an affordable and small square LED button on AliExpress. It’s basically a tact switch with an LED and it has 6 pins (4 pins for a tact switch and +/- pins for an LED). According to its description, it is 12V 50mA rating. It will work with 3.3V output from Lolin S3 with a small resistor (like 100Ω).

Regarding connections between buttons and Lolin S3, there might be some options depending on how I want to turn their LEDs on — making them always on, turning them on only when I push them, controlling their LEDs with more flexibility (e.g., using LEDs for guidance). If we want flexibility, we need to use GPIO pins to control their LEDs as well as GPIO pins for their switches. Because I need many GPIO pins in this project, I decided to go with a simple approach for buttons — turning their LEDs on only when I push them. In this case, I need only one GPIO pin for each button and can turn its LED on with just circuit connection (connecting a positive pin of a button’s LED to 3.3V, a negative pin of a button’s LED to one end of a button’s switch pins, and another end of a button’s switch pins to GND).

LEDs

Regarding an LED for the eye of HAL 9000, the HAL 9000 model comes with a 5mm DIP type LED. I found an addressable LED in the shape of 5mm DIP (like this and this). Although it has only one LED, it allows me to control color, brightness, and effect. I bought an affordable one from AliExpress. In terms of ambient lighting, I’m going to use 5V SK6812 LED strip, especially 60 LEDs/m version which is denser than 30 LEDs/m version that I used in V2. It seems that it draws 60mA/LED at maximum. If I use the entire 1m, it consumes 3.6A in total.

Emergency Battery

I’ve found an affordable UPS module on AliExpress. It is 15W 3A rating and it has a USB-C input. It also comes with a holder for two 18650 batteries. I bought a 5V/3A version of the UPS module and two flat top 18650 Li-ion batteries. Because the UPS module is 5V/3A, I better avoid using the entire 1m LED string.

A USB-C cable goes to the input of the UPS module and output pins (pads, actually) of the UPS module provides power to other components. When it comes to connection between the UPS module and other components, there seem to be two options. I could either (A) provide power from the UPS module to Lolin S3 and then provide power from Lolin S3 to other components, or (B) provide power from the UPS module to a shared 5V line and then provide power from the 5V shared line to all the components. According to Lolin S3 schematic, it does not have an explicit VBUS pin, and it has a 1N5819W, which is a 40V/1A Schottky barrier diode, between VBUS (USB-C input) and 5V. Which means that 5V pin can provide 1A at maximum while the UPS module can provide 3A at maximum. So, I decided to go with (B). I’m going to create a shared 5V line on the PCB board and the 5V line provides power to 5V powered components like Lolin S3, LEDs, the exhaust fan, SPS30, LD2410C. And then, Lolin S3 provides power to 3.3V powered components from its 3.3V pin.

I need to monitor at least input power voltage of the UPS module so that I can detect a power outage. I’m going to use a simple voltage divider circuit and ESPHome Analog to Digital Sensor.

By the way, when I ordered UPS modules, one of them arrived with one of capacitors coming off. It might be better to order more than you need.

RTC

ESPHome already supports I2C RTC module (DS1307). And it seems that the latest version (DS3231) is also supported. It is relatively expensive on major online stores, but I found an affordable one on AliExpress (like this). It can work with a CR2032 coin battery.

PCB Board

Because the HAL 9000 model is large enough, I could design a large custom PCB board for it. But there will be just one HAL 9000 in my house while the minimum order of PCB board for JLCPCB is 5. So, I decided to make it smaller as much as I can so that I can use the rest of them in other projects in the future, hopefully. As I’m going to make it work with a UPS module, I decided to make it the same size as the UPS module for my convenience.

After many tries-and-errors, my PCB design became like the following. To be honest, I mistakenly connected components and fried a couple of ESP32 board and UPS module during prototyping of the PCB board design.

As you can see, I’ve used the same approach with V2 and put different boards in one design so that I can have the main board and sub boards for BH1750, SHT41, DIP LED and buttons all at once to reduce the PCB manufacturing cost.

The main board hosts through-holes for Lolin S3, through-holes for female headers to attach sensor boards (SCD40, SGP41, BMP280) and amps (MAX98357A). Regarding SGP41 and BMP280, I’m going to stack them onto one female header. It also hosts a JST SH header for the exhaust fan and JST ZH headers for SPS30, ICS-43434, a buzzer and sub boards, and a JST XH header for LED strip. And it has a JST XH header for power input from the UPS module and a JST ZH header for the input power monitoring of the UPS module. Basically, I selected XH over ZH in case I expect current more than 1A.

As I described above, a USB-C cable basically goes to the UPS module and the UPS module provides power to the 5V line on the PCB board. But sometimes, I need to plug a USB-C cable directly into Lolin S3 (e.g., flushing the firmware in case Wi-Fi is not available, etc.). Although I can see SS54, which is a 40V/5A Schottky barrier diode, on the UPS module, I placed diodes around power input from the UPS module and input power monitoring of the UPS module to prevent current from flowing to the UPS module from the PCB board.

I also placed through-holes for resistors for SD pin and GAIN pin of MAX98357A so that I can add necessary values of resistors later for the purpose of channel and gain selection.

Finally, I placed an order to JLCPCB. It costed me 74 USD + shipping fee 11 USD. The unit cost was 17 USD.

After 2 weeks, I received the PCB board from JLCPCB. It looks great. I could easily detach sub boards from a main board with a diagonal cutter.

Back Panel

As I described in V1 project, I have used FreeCAD to just cut an official Sensirion SEN55 holder. It was the only one experience for me to use 3D CAD. I have never designed anything from scratch with 3D CAD. Since I’m a completely newbie, I’m not going to compare or review 3D CAD software. I just decided to use Tinkercad this time because I don’t have to install anything on my PC and it looks easy to use for a complete beginner like me.

The HAL 9000 model is a kind of box without a back. The back of it is completely open. There is no mechanism to attach a back panel, like a latch, a screw hole, a slit and so forth. So, I’m going to create a similar but a little bit smaller box without a front panel so that it can tightly fit into the HAL 9000 model. It’s going to be like an inner box of the HAL 9000 model. But hereinafter I call it just a back panel anyway. I used a caliper and measured the inside dimensions of the HAL 9000 model to design the back panel. The measured dimensions were a little bit different from the precise 3D data by the way.

After many tries-and-errors, my 3D design became like the following. It looks like a ship.

As you might be able see, I placed threaded screw holes to attach components. I used ISO metric screw thread object in Tinkercad. And I used dimensions of M2 screw (0.4mm pitch). It seems that using a thread insert or using a tapping tool would be a better idea rather than trying to directly print a threaded screw hole especially for a small screw like M2. But I decided to give it a try.

Because I want to attach an LED strip on the back of the back panel, I added an angled step to the back instead of making the back completely flat so that an LED strip can create nice ambient lighting around the HAL 9000 model against the wall. It is an improvement from V2. When it comes to a DIP LED for the eye, I precisely measured the expected position of the DIP LED so that it can fit into the hole of the eye of the HAL 9000 model when I put the back panel into the HAL 9000 model.

I created holes or slits on both sides of the back panel so that components like SHT41, BH1750, SPS30, ICS-43434 and the exhaust fan can be directly exposed to the outside. Placing holes for BH1750 and ICS-43434 on sides is not ideal. They should face in front. But I had no choice because I didn’t want to modify the HAL 9000 model.

I placed speaker boxes for a pair of speaker units behind the speaker grille. I also placed the buzzer mount close to the speaker grille rather than creating a hole on the side.

I placed a screw hole on the right bottom side so that it is exposed through the small cutout on the HAL 9000 model, which is originally designed for the switch of the LED of the eye. It will be helpful if I can use a screw to hold the HAL 9000 model.

Finally, I placed an order of my 3D design. I used Craftcloud again, which I used in V1 project to create a SEN55 holder. I exported a STL file of the back panel from Tinkercad and uploaded it to Craftcloud. Because the back panel is hidden behind the HAL 9000 model, I selected an affordable PETG material with black color and without a smoothing option. And I have chosen a service provider in Oregon who offered me the most affordable price. It costed me 31 USD + shipping fee (12 USD). Although it was the most affordable option I could get, 43 USD was almost the same with the HAL 9000 model itself.

The 3D print service provider contacted me to inform the risk of print failures in some areas like buttons and air vents because the wall thickness was too thin. Because I’ve already ordered the PCB board, I asked them to proceed printing anyway. It will be a future improvement.

After 7 days, I received the 3D printed back panel.

It looks good overall. But there was relatively large distortion especially in the upper area. I should have designed the back panel differently or should have chosen a different material. Because of distortion, it is a bit tight to fit the back panel into the HAL 9000 model. But fortunately, it was not a big problem.

Although there are still some to improve, my 3D design is here.

Building HAL 9000 version of Sensor Node

The HAL 9000 Model

The HAL 9000 model comes with 4 gray parts (the faceplate, the lens frame, the lens mount/focusing ring and the speaker grille), 2 clear parts (the inner lens and the outer lens), the display base, the LED and decals. I won’t use the display base and the LED in this project. Assembling those parts is easy. But I need to overcome two things — making holes on the speaker grille and painting gray parts.

In terms of making holes on the speaker grille, I used a mini rotary tool and created 24 columns x 18 rows = 432 holes.

The most difficult part was painting. When I was a child, I tried painting Gundam models (Gunpla) and I always ended up in miserable and horrible results. Since then, I have never done any model building and painting. I saw instructions on the Internet (like this, this and this) and bought TAMIYA paint spray rattle cans

• FINE SURFACE PRIMER L for all parts.
• TS-29 SEMI GLOSS BLACK (I could not find Semi Matt Black and bought Semi Gloss Black instead) for the faceplate and the outside of the lens mount/focusing ring.
• TS-6 MATT BLACK for the inside of the lens mount/focusing ring.
• TS-17 GLOSS ALUMINUM for the frame of the faceplate and the lens frame.
• TS-42 LIGHT GUN METAL for the speaker grille.

I created a paint booth with some cardboards in my garage and painted all the parts. I forgot to wash them before painting as suggested in some instructions. And I didn’t use a box fan as well. Although my process might have not been ideal, I’ve done my paint job successfully, I think. My wife complained about paint smell though.

I’ve put decals on the faceplate and the lens mount/focusing ring. As suggested on the Internet (like this), I used Testers Parts Cement & Window Maker to glue the inner lens and the outer lens on the lens mount/focusing ring. Lastly, I glued the lens mount/focusing ring, the lens frame and the speaker grille to the faceplate.

Now, the HAL 9000 model has been completed. It looks really good.

PCB board and components

As I use a stereo pair of speaker units, I attach two MAX98357A amps to the main board — one for the left speaker unit and another one for the right speaker unit. We can configure MAX98357A to output only left channel, only right channel or mixed channels by soldering a resistor around SD pin. Honestly, it was confusing to me, but this article helped me a lot. I soldered 100K resistor between 5V and SD pin of MAX98357A for the left and 500K resistor between 5V and SD pin of MAX98357A for the right. I also soldered 100K resistor between GND and GAIN pin of MAX98357A for the left and 100K resistor between GND and GAIN pin of MAX98357A for the right to have the maximum 15dB gain.

And then, I soldered a Lolin S3 directly onto the main board with male pin headers. Although it is better to make it detachable, I prioritized on reducing the height.

I used a pre-crimpled cable kit of JST ZH connectors for sub boards (SHT41 and BH1750) and SPS30 to connect them to the main board. And I soldered female headers for BMP280, SGP41 and SCD40 sensor boards onto the main board. And I soldered male headers to them so that I can just attach them to the main board. Because I placed through holes for BMP280/SGP41 and SCD40 too close, there was a little bit conflict between them, but I managed to place them. This will be a future improvement. Regarding LD2410C, I cut one end of a pre-crimpled JST ZH cable and directly soldered it on a pin. In terms of the exhaust fan, I just connected it onto the JST SH header on the main board.

For ICS-43434, I’ve done the same approach with LD2410C by cutting one end of a pre-crimpled JST ZH cable and directly soldered it on a pin. And for MAX98357A, I’ve done the same approach with sensor boards by using female headers and male headers. Regarding a buzzer, I created conversion cables from JST ZH connectors to DuPont female connectors.

Regarding buttons, I placed them onto the sub board and soldered them. Although I designed through holes for buttons based on the official layout data, I broke two buttons’ LED pins when I placed them onto the sub board because LED pins are too thin. This will be yet another future improvement.

I used a pre-crimpled JST ZH cable for the DIP LED sub board. But when I tried to solder a DIP LED onto the sub board, I realized that I made a mistake in the sub board design for the DIP LED. I placed through-holes in the order of Vin, Data-in and GND on the sub board. But the order of legs of DIP LED was like Data-in, Vin and GND. So, I had to bend legs to solder them to through-holes on the sub board. I also had to use heat shrink tubes so that DIP LED legs would not touch each other. This will be another future improvement. In terms of the LED string, I used a pre-crimpled cable kit of JST XH connectors. I cut off the existing JST SM connector from the strip and soldered JST XH cables instead.

When it comes to connections between the main board and the UPS module, I used a pre-crimpled JST XH cable for the power input from the UPS module and a pre-crimpled JST ZH cable for input power monitoring.

Back Panel

I’ve put all the components onto the back panel. I used M2 screws for almost all the components except SPS30, LD2410C and speaker units which don’t have any screw holes.

3D printed M2 screw holes were not precisely accurate but mostly worked well. Although some screw holes didn’t work because of distortion or my design flaw, I could use self-tapping screws and hold components anyway. It was a bit difficult to use screws for some screw holes because of their positions and orientations. But I managed to put all the components anyway. If I have another opportunity to build the HAL 9000 version of the sensor node again, I may want to improve my design.

Regarding SPS30, it was a bit tight to place it on the back panel because my design was too tight and there was even distortion. But I managed to push it down into its place anyway.

In terms of LD2410C and speaker units, I used hot glue. Before I glued speaker units, I stuffed polyfill into speaker boxes expecting improvement of sound quality.

I’ve put a 5mm glass lens, which I bought from Amazon in my V2 project, to the hole for BH1750.

I attached the LED strip to the angled step surrounding the back panel. Although I could attach all of 60 LEDs, I attached only 23 of them and covered just the upper part of the back panel to reduce power usage for just in case.

HAL 9000 version of Sensor Node

Now, I just put the back panel, which is packed with the PCB board and all the components, into the HAL 9000 model. It is a bit tight because of distortion and the soldering position of the DIP LED. But I’ve successfully put them all together. Because there is just friction between the HAL 9000 model and the back panel, there might be a risk that the HAL 9000 model fall off from the back panel hooked on the wall. I may use a screw in the screw hole which I created on the right bottom of the back panel. Although it just increases friction and does not securely lock the HAL 9000 model, it might be better than nothing.

Software Configuration

Configuration of ESPHome was basically straightforward. But I realized that I could not change the speed of the fan with a simple configuration. I thought that my choice of GPIO40 for the PWM signal output was a mistake because the official document says that only GPIO0 to GPIO33 can be used. However, I found that a configuration example on ESPHome community would work even in my case, and I could change the speed of the fan. When it comes to the lighting of the eye, I decided to use the pulse effect with a little bit customizing for now after testing existing light effects.

esphome:
  name: s3-01
  friendly_name: s3-01
  platformio_options:
    board_build.flash_mode: dio
  on_boot: 
    - priority: 200
      then:
        - ds1307.read_time:
            id: time_rtc

esp32:
  board: lolin_s3
  framework:
    type: arduino

psram:
  mode: octal
  speed: 80MHz

logger:
  baud_rate: 0

api:
  encryption:
    key: !secret api_key
  actions:
    - action: rtttl_play
      variables:
        song_str: string
      then:
        - rtttl.play:
            rtttl: !lambda 'return song_str;'

ota:
  - platform: esphome
    password: !secret ota_password

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

  ap:
    ssid: !secret ap_ssid
    password: !secret ap_password

external_components:
  - source: github://stas-sl/esphome-sound-level-meter

captive_portal:

i2c:
  sda: GPIO42
  scl: GPIO41
  scan: True

uart:
  tx_pin: GPIO17
  rx_pin: GPIO18
  baud_rate: 256000
  parity: NONE
  stop_bits: 1

ld2410:

sensor:
  - platform: adc
    pin: GPIO8
    name: Battery Voltage
    unit_of_measurement: V
    update_interval: 10s
    attenuation: 11db
    filters:
      - multiply: 2.0
  - platform: adc
    pin: GPIO9
    name: Power Voltage
    unit_of_measurement: V
    update_interval: 10s
    attenuation: 11db
    filters:
      - multiply: 2.0
  - platform: bh1750
    name: Illuminance
    address: 0x23
    update_interval: 60s
  - platform: sht4x
    address: 0x44
    update_interval: 60s
    temperature:
      name: Temperature
      id: temperature_sht41
    humidity:
      name: Humidity
  - platform: sgp4x
    address: 0x59
    update_interval: 60s
    voc: 
      name: VOC Index
    nox: 
      name: NOx Index
  - platform: scd4x
    address: 0x62
    update_interval: 60s
    co2:
      name: CO2
    temperature:
      name: Temperature SCD40
    humidity:
      name: Humidity SCD40
  - platform: sps30
    pm_1_0:
      name: PM 1.0
      id: pm_1_0
    pm_2_5:
      name: PM 2.5
      id: pm_2_5
    pm_4_0:
      name: PM 4.0
      id: pm_4_0
    pm_10_0:
      name: PM 10.0
      id: pm_10_0
    pmc_0_5:
      name: PMC 0.5
      id: pmc_0_5
    pmc_1_0:
      name: PMC 1.0
      id: pmc_1_0
    pmc_2_5:
      name: PMC 2.5
      id: pmc_2_5
    pmc_4_0:
      name: PMC 4.0
      id: pmc_4_0
    pmc_10_0:
      name: PMC 10.0
      id: pmc_10_0
    pm_size:
      name: Typical Particle Size
      id: pm_size
    address: 0x69
    update_interval: 60s
  - platform: bmp280_i2c
    address: 0x76
    update_interval: 60s
    temperature: 
      name: Temperature BMP280
      id: temperature_bmp280_aliexpress
      oversampling: 16X
      accuracy_decimals: 2
    pressure:
      name: Pressure
  - platform: pulse_counter
    pin: GPIO39
    name: Fan RPM
    id: fan_rpm
    unit_of_measurement: "RPM"
    accuracy_decimals: 1
    filters:
      - multiply: 0.5
    count_mode:
      rising_edge: INCREMENT
      falling_edge: DISABLE
    update_interval: 10s
  - platform: ld2410
    light:
      name: light
    moving_distance:
      name : Moving Distance
    still_distance:
      name: Still Distance
    moving_energy:
      name: Move Energy
    still_energy:
      name: Still Energy
    detection_distance:
      name: Detection Distance

binary_sensor:
  - platform: ld2410
    has_target:
      name: Presence
    has_moving_target:
      name: Moving Target
    has_still_target:
      name: Still Target
    out_pin_presence_status:
      name: out pin presence status
  - platform: gpio
    name: Button 01
    pin:
      number: GPIO21
      inverted: True
      mode:
        input: True
        pullup: True
    internal: True
    on_multi_click:
    - timing:
      - ON for at most 1s
      - OFF for at least 0.5s
      then:
        - script.execute: volume_up
    - timing:
      - ON for at most 1s
      - OFF for at most 1s
      - ON for at most 1s
      - OFF for at least 0.2s
      then:
      - logger.log: "Double Clicked"
      - binary_sensor.template.publish:
          id: button_01_double_click
          state: ON
      - delay: 0.5s
      - binary_sensor.template.publish:
          id: button_01_double_click
          state: OFF
    - timing:
      - ON for 1s to 2s
      - OFF for at least 0.5s
      then:
      - logger.log: "Single Long Clicked"
      - binary_sensor.template.publish:
          id: button_01_single_long_click
          state: ON
      - delay: 0.5s
      - binary_sensor.template.publish:
          id: button_01_single_long_click
          state: OFF
  - platform: template
    id: button_01_double_click
    name: Button 01 Double Click
    publish_initial_state: True
  - platform: template
    id: button_01_single_long_click
    name: Button 01 Single Long Click
    publish_initial_state: True
  - platform: gpio
    name: Button 02
    pin:
      number: GPIO14
      inverted: True
      mode:
        input: True
        pullup: True
    internal: True
    on_multi_click:
    - timing:
      - ON for at most 1s
      - OFF for at least 0.5s
      then:
      - logger.log: "Single Short Clicked"
      - binary_sensor.template.publish:
          id: button_02_single_short_click
          state: ON
      - delay: 0.5s
      - binary_sensor.template.publish:
          id: button_02_single_short_click
          state: OFF
    - timing:
      - ON for at most 1s
      - OFF for at most 1s
      - ON for at most 1s
      - OFF for at least 0.2s
      then:
      - logger.log: "Double Clicked"
      - binary_sensor.template.publish:
          id: button_02_double_click
          state: ON
      - delay: 0.5s
      - binary_sensor.template.publish:
          id: button_02_double_click
          state: OFF
    - timing:
      - ON for 1s to 2s
      - OFF for at least 0.5s
      then:
      - logger.log: "Single Long Clicked"
      - binary_sensor.template.publish:
          id: button_02_single_long_click
          state: ON
      - delay: 0.5s
      - binary_sensor.template.publish:
          id: button_02_single_long_click
          state: OFF
  - platform: template
    id: button_02_single_short_click
    name: Button 02 Single Short Click
    publish_initial_state: True
  - platform: template
    id: button_02_double_click
    name: Button 02 Double Click
    publish_initial_state: True
  - platform: template
    id: button_02_single_long_click
    name: Button 02 Single Long Click
    publish_initial_state: True
  - platform: gpio
    name: Button 03
    pin:
      number: GPIO13
      inverted: True
      mode:
        input: True
        pullup: True
    internal: True
    on_multi_click:
    - timing:
      - ON for at most 1s
      - OFF for at least 0.5s
      then:
        - script.execute: volume_down
    - timing:
      - ON for at most 1s
      - OFF for at most 1s
      - ON for at most 1s
      - OFF for at least 0.2s
      then:
      - logger.log: "Double Clicked"
      - binary_sensor.template.publish:
          id: button_03_double_click
          state: ON
      - delay: 0.5s
      - binary_sensor.template.publish:
          id: button_03_double_click
          state: OFF
    - timing:
      - ON for 1s to 2s
      - OFF for at least 0.5s
      then:
      - logger.log: "Single Long Clicked"
      - binary_sensor.template.publish:
          id: button_03_single_long_click
          state: ON
      - delay: 0.5s
      - binary_sensor.template.publish:
          id: button_03_single_long_click
          state: OFF
  - platform: template
    id: button_03_double_click
    name: Button 03 Double Click
    publish_initial_state: True
  - platform: template
    id: button_03_single_long_click
    name: Button 03 Single Long Click
    publish_initial_state: True

text_sensor:
  - platform: ld2410
    version:
      name: Firmware Version
    mac_address:
      name: MAC Address

output:
  - platform: ledc
    pin: GPIO40
    id: fan_pwm
  - platform: ledc
    pin: GPIO4
    id: buzzer

fan:
  - platform: speed
    output: fan_pwm
    name: Fan Toggle
    id: fan_toggle
    on_turn_on:
     - output.set_level:
          id: fan_pwm
          level: !lambda |-
             return id(fan_speed_override).state/100.0;

select:
  - platform: ld2410
    light_function:
      name: light function
    out_pin_level:
      name: out pin level

light:
  - platform: esp32_rmt_led_strip
    id: onboard_led
    name: Onboard LED
    rgb_order: RGB
    chipset: WS2812
    num_leds: 1
    pin: 38
    rmt_channel: 0
  - platform: esp32_rmt_led_strip
    id: strip_led
    name: Strip LED
    rgb_order: GRB
    chipset: SK6812
    num_leds: 23
    pin: 47
    is_rgbw: True
    rmt_channel: 2
  - platform: esp32_rmt_led_strip
    id: dip_led
    name: DIP LED
    rgb_order: RGB
    chipset: WS2811
    num_leds: 1
    pin: 48
    rmt_channel: 1
    effects:
      - pulse:
      - pulse:
          name: Custom Pulse
          min_brightness: 50%
          max_brightness: 100%
          transition_length:
            on_length: 2s
            off_length: 1s
          update_interval: 3s

rtttl:
  output: buzzer
  id: my_rtttl
  gain: 60%
  on_finished_playback:
    - logger.log: 'Song ended!'

i2s_audio:
#  - id: i2s_in
#    i2s_lrclk_pin: GPIO5
#    i2s_bclk_pin: GPIO7
  - id: i2s_out
    i2s_lrclk_pin: GPIO10
    i2s_bclk_pin: GPIO11

#microphone:
#  - platform: i2s_audio
#    id: i2s_mic
#    i2s_audio_id: i2s_in
#    adc_type: external
#    pdm: False
#    channel: left
#    i2s_din_pin: GPIO6

media_player:
  - platform: i2s_audio
    id: i2s_speaker
    name: I2S Speaker
    i2s_audio_id: i2s_out
    dac_type: external
    i2s_dout_pin: GPIO12
    mode: stereo

#speaker:
#  - platform: i2s_audio
#    id: i2s_speaker
#    i2s_audio_id: i2s_out
#    dac_type: external
#    i2s_dout_pin: GPIO12

i2s:
  ws_pin: 05
  bck_pin: 07
  din_pin: 06
  sample_rate: 48000
  bits_per_sample: 32
  channel: left
  bits_shift: 8

sound_level_meter:
  update_interval: 150s
  buffer_size: 1024
  mic_sensitivity: -26dB
  mic_sensitivity_ref: 94dB
  groups:
    - filters:
        - type: sos
          coeffs:
            # A-weighting:
            #       b0           b1            b2             a1            a2
            - [0.16999495, 0.741029, 0.52548885, -0.11321865, -0.056549273]
            - [1., -2.00027, 1.0002706, -0.03433284, -0.79215795]
            - [1., -0.709303, -0.29071867, -1.9822421, 0.9822986]
      # nested groups
      sensors:
        - type: eq
          name: LAeq_1min
          id: LAeq_1min
          unit_of_measurement: dBA
        - type: max
          name: LAmax_125ms_1min
          id: LAmax_125ms_1min
          window_size: 125ms
          unit_of_measurement: dBA
        - type: min
          name: LAmin_125ms_1min
          id: LAmin_125ms_1min
          window_size: 125ms
          unit_of_measurement: dBA

script:
  - id: volume_up
    then:
      - homeassistant.service:
          service: media_player.volume_up
          data:
            entity_id: media_player.s3_02_i2s_speaker
      - logger.log: "vol up trigger"
  - id: volume_down
    then:
      - homeassistant.service:
          service: media_player.volume_down
          data:
            entity_id: media_player.s3_02_i2s_speaker
      - logger.log: "vol down trigger"

number:
  - platform: template
    name: Fan Speed Override
    id: fan_speed_override
    internal: false
    max_value: 100.0
    min_value: 0.0
    step: 10.0
    optimistic: true
    mode: slider
    on_value:
      then:
        if:
          condition:
            fan.is_on: fan_toggle
          then:
          - output.set_level:
              id: fan_pwm
              level: !lambda "return x/100;"

time:
  - platform: ds1307
    id: time_rtc
    address: 0x68
    timezone: America/Los_Angeles
    update_interval: never
  - platform: homeassistant
    id: time_ha
    update_interval: 60s
    on_time_sync: 
      then:
        - ds1307.write_time:
            id: time_rtc
        - delay: 1s
        - ds1307.read_time:
            id: time_rtc

Result

Overall, it was successful.

• Environment Monitoring: Regarding Temperature and Relative Humidity, the approach of using a sub board has worked well. Unlike V2, measurement is almost the same with my reference when the back panel is not closed. When the back panel is closed with the HAL 9000 model, measurement seems to be affected by thermal conduction probably through the warmed air inside or through the back panel (+2°C to +3°C). But once I turn on the exhaust fan in 30% speed (3000 to 3500 rpm), measurement becomes reasonable compared to my reference (less than +1°C). Although the exhaust fan makes some level of noise, I think it is effective this time. The position of BH1750 is not good as expected. Because it is on the side of the back panel, it does not capture enough ambient light. And because there is the LED strip surrounding the back panel, Illuminance measurement is affected by the light from the LED when the LED is turned on. Other metrics like CO2, VOC index, NOx index, PM, Pressure and Presence are measured as expected.
• Microphone, Amplifier and Speaker: Because the hole for the mic is on the side, it might not be ideal for capturing voice in Voice Assistant. But it has worked well at least for Ambient Sound Level monitoring. The sound of the speaker is cheap. I’m not sure if polyfill stuffing actually made any difference for affordable tiny speaker units and small PETG speaker boxes. And the sound is somewhat unclear because the speaker grille blocks the sound. I created a lot of holes on the speaker grille though. But anyway, it’s acceptable. When it comes to Voice Assistant, it has worked as well. But I’m not sure if it would fit in my daily life.
• Button: All the three buttons have worked well. When I push a button, a button LED turns on and a button click event is captured. I used single click events of two buttons as volume-up and volume-down of the sound. They have worked well.

• LED: The DIP LED for the eye has worked very well. Although the eye of HAL 9000 is supposed to be just “red”, adding capabilities of changing color, brightness and even having some effects is an upgrade from the original HAL 9000 model. The pulse effect with a little bit customization gives a great character. The LED strip on the back panel has also worked well. Because I created a gap and an angle between the LED strip and the wall, it creates great ambient lighting.

• Emergency Battery: The UPS module has actually worked well. When I plugged off a USB-C cable, Lolin S3 and other components continue working without any disruption. Based on the measurement of the output of the UPS module, the output voltage has fluctuation, and it is around 4.5 V or 4.6 V in average. The measurement of the input of the UPS module has also worked. When I plug in a USB-C cable, it shows voltage around 4.9 V to 5 V with a little bit fluctuation. And when I plug off a USB-C cable, it goes to 0 V. Which means that I can use the measurement as a trigger for some automations in an emergency situation.
• RTC: Although I still have not used it for any use case, it has worked well. I realized that I could use DS3231 for waking up ESP32 from deep sleep by connecting SQW pin of DS3231 to one of GPIO pins of ESP32. This will be an improvement for the future.

Environment metrics data measured by HAL 9000 are collected to Home Assistant, sent to AWS, stored in DynamoDB and Timestream and visualized with my serverless application. Click events of buttons are also captured, used for volume-up and volume-down and sent to AWS as well. Although it is not recommended, I could use HAL 9000 as a Player Provider through Home Assistant MediaPlayers on Music Assistant. It does not always work though.

Final Thoughts

I successfully transformed the HAL 9000 model with ESPHome and Home Assistant, and achieved All-in-One environment monitoring and even more. Now, HAL 9000 is in my house. Its retro-futuristic and somewhat dystopian appearance matches DIY smart home system in this old house. Honestly, this project went well more than expected. And through the entire process, I could even achieve some level of confidence about PCB board design and 3D design. Next time, it may finally be time to 3D design an entire enclosure.

Disclaimer: Before I dive into the details, let’s set some ground rules:

1. Educational Purpose Only: This blog post serves as an educational resource. I’m sharing my experiences, but I’m just a hobbyist, not a professional. Always consult experts for specific advice.
2. Assumption of Risk: By following my instructions, you assume full responsibility for any consequences. DIY projects involve risks, so proceed with caution.
3. Safety First: Handle electronic components carefully. Use appropriate tools and safety gear. Be aware of electrical hazards. Follow manufacturer guidelines for microcontrollers and sensors.

Objectives

I’m going to upgrade the V1 sensor node as follows while keeping all of its capabilities to measure environment metrics.

• Adding Exhaust Fan: In V1, I tried to carefully place components inside an enclosure so that the heat, which comes from components themselves (especially ESP8266 microcontroller), doesn’t affect temperature measurement and warmed air goes from bottom to top through vents. But the result was not successful. The temperature inside the enclosure went up, and I had to adjust measurements anyway. In V2, I’m going to try an exhaust fan to take warmed air out of the enclosure to minimize adjustment, hopefully.
• Adding Night Light: In V1, I chose a simple plastic case as an enclosure so that it can fit in my house from an aesthetic perspective. Although it is subtle and inconspicuous, my family members and I can still see it, and it doesn’t make anything explicit because it is just an environment monitor. So, I decided to assign an explicit role to a sensor node so that my family members and I can accept that it’s there. Since a sensor node is placed in each room, it’s good to have a role that is necessary or good to have in each room. In V2, I’m going to add a night light. I actually have been using simple night lights, which I bought on Amazon, in my house for more than a year and some of them have been broken recently. It’s a good timing to replace them.
• Adding Ambient Noise Monitoring: In V1, I decided not to include ambient noise monitoring which I used to measure when I was using WxBeacon2. In V2, I’m going to undo my decision to achieve all-in-one monitoring.
• Adding Notification: In V1, I collected measurement data from sensor nodes to Home Assistant. The dashboard of Home Assistant has been the only way to see measurement data. But my family members don’t necessarily want to see the dashboard. So, I’m going to add a feature to let a sensor node alert us when there is something to know (like CO2 level increase, etc.). I could blink or change color of the night light. But it could be difficult to recognize during daytime. In V2, I’m going to add a simple buzzer.
• Adding BLE: In V1, I used Lolin D1 mini for ESP8266 microcontroller. It didn’t have BLE capability. But when I used ESP32 like Lolin S3 mini and S3 in another project to collect data from my car and send them to AWS, I thought BLE capability might have some use cases on a sensor node as well by combining with human presence detection while still balancing privacy and home security. In V2, I’m going to switch from ESP8266 to ESP32, especially from Lolin D1 mini to Lolin S3 mini. S3 mini is a little bit expensive but still small and almost the same size with D1 mini.
• Designing PCB: In V1, I used a sensor board to measure environment metrics, which is a breakout board with an actual sensor chip on it and easy to use for a casual DIYer like me. I connected sensor boards and a microcontroller with wiring and soldering. Although a sensor board itself is easy to use, a process of wiring and soldering multiple sensor boards and a microcontroller was very challenging for me. In V2, I’m going to design a custom PCB board for the first time to reduce wiring. This is not about a feature of a sensor node, but it is very important for me because I have never had any knowledge about PCB. I’m completely a newbie. But I’m going to give it a try.

Requirements

In V1, I set three requirements — “Cost”, “Appearance” and “Easy to build” and I’m going to stick with them this time as well. Especially regarding “Cost”, I realized that most of the cost comes from expensive sensors to measure air quality like CO2, VOC, NOx and PM. I need to keep my eyes on the cost in this V2 project as well since I’m going to add more features and even create a custom PCB.

Exhaust Fan

In V1, I used a simple plastic case as an enclosure that comes in three sizes (like this, this, this and this). Given the size of the case and its vents on top and bottom, the size of a fan needs to be very small. Even the largest case has less than 30mm in height. I’m not sure if such a small fan will be actually effective. But I’m going to give it a try to see how it may or may not work.

I’d like to power and control a fan from a S3 mini for simplicity. A 5V powered fan will be a simple choice because a S3 mini has a 5V output (VBUS) pin when we power it from USB-C. And I’d like to use a PWM fan to monitor and control fan speed, which has 4 pins — Vin (5V), GND, RPM signal (tacho) and PWM signal. I found several affordable options on AliExpress. One option is just a tiny fan like 20mm or 25mm (like this and this). The air goes into one of large open side and goes out from an opposite open side. Another option is a blower fan (like this and this). The air goes into a large open side and goes out from a sleek outlet in right-angled direction. In either option, I need to decide whether I blow the air into a case or suck the air out of a case. It seems that blowing into the air is considered to be efficient for cooling purpose in the computer case area. But I also need to consider how the air flow could affect PM measurement inside a case. Based on an official document from Sensirion (because I’m using Sensirion PM sensor), I might better avoid high air flow around the inlet and the outlet of the PM sensor. I decided to try sucking the air this time. I’m going to place a tiny fan on the top vents to let it suck the air out of a case.

I bought a 5V powered, PWM, 4 pins, 25mm fan which I found on AliExpress. It seems to be intended for a raspberry pi. But unlike the official raspberry pi active cooler, which is a 30mm blower fan, it’s a little bit smaller 25mm non-blower tiny fan. It has a JST SH connector like the official raspberry pi active cooler. The pinout looks the same with the official one. If I place a JST SH header on a PCB board, I will be able to easily connect a fan.

Night Light

Night light does not have to be very bright. It can be just some level of indirect lighting. Adding a couple of LEDs on the back of the plastic case or around the case would be enough. For simplicity, I prefer to power and control the light from a S3 mini. So, I’m going to use a 5V addressable LED light strip because a S3 mini has a 5V output (VBUS) pin when we power it from USB-C. Because my main use case is a night light, I use SK6812, especially a RGBWW version to have warm white lighting.

I bought a SK6812 LED strip on AliExpress. I chose 30 LEDs/m density. The strip has 3 wires — Vin (5V), GND and Data. Because I will need only a short length of LED strip, it might be OK to just directly connect a S3 mini to a LED strip — VBUS to 5V input, GND to GND and one of GPIO to Data. Though, it seems to be better to use a level shifter between GPIO and Data. I found a great article on the Internet and decided to use 74LV1T34.

The LED strip came with a JST SM connector which already attached to 3 wires. And it came with another short cable which has a JST SM connector in one end. Since JST SM is for wire-to-wire connection, there is no JST SM header which I can place on a PCB board. I’m going to cut the attached 3 wires and solder wires with XH connector instead. I bought a pre-crimped cable set from Amazon.

Ambient Noise Monitoring

I need a microphone to measure noise level. I found that ESPHome supports I2S Audio and there are I2S Microphones. At the time of this writing, ICS-43434 seems to be a good option. I bought an ICS-43434 breakout board from Adafruit.

I also found a very nice external component to measure ambient sound level with I2S microphone.

Buzzer

Because it’s just for a notification, I’m going to use a simple one. I noticed that a passive piezo buzzer is already supported by ESPHome.

I found an affordable one on AliExpress, which looks like the same type with one supported by ESPHome. The buzzer came with 3 pins — Vin (5V), GND and Data.

BLE

Lolin S3 mini already has BLE capability. One concern is that it is resource intensive. Since I2S is also resource intensive, using both I2S and BLE might cause some issues as described on ESPHome web site. But I’m going to give it a try.

PCB

There are many CAD software and PCB manufacturing services. I’m not going to compare them and discuss which one is good or not because I don’t have enough knowledge anyway. I decided to use EasyEDA and JLCPCB this time because I don’t have to install anything on my computer and thus found them easy to try.

Using EasyEDA was lots of try-and-error for me. I used “Std” version. Although it is free and easy-to-start, it was not necessarily intuitive for a newbie like me. Many online articles and blogs helped me a lot. After lots of try-and-error processes, I needed to make some decisions like “PCB or PCBA” and “Sensor Chip or Sensor Board”.

PCB or PCBA

Once we design a PCB board on EasyEDA, we can place an order of either just manufacturing the PCB board or even assembling components on the PCB board as an option (PCBA). A component is anything we put on a PCB board, like a resistor, a capacitor, a pin header, a sensor chip and so forth. If we just order a PCB board, we need to solder necessary components on the PCB board by ourselves. If we order a PCB board + PCBA option, we can receive the PCB board with components soldered on it. For each component, we can choose either assembling it with PCBA option or not just by checking “Add into BOM” option on EasyEDA. And there are some components that we want to use but are not listed as “JLCPCB Assembled” in the parts library on EasyEDA, which means that we cannot add them into BOM for the PCBA option. In case of that, we can still design a PCB board thanks to the community feature of EasyEDA. We can find a wide variety of components listed as “User Contributed” in the parts library on EasyEDA. By using “User Contributed” components which have footprint data, we can design a PCB board. And we can buy those components somewhere else (AliExpress, Adafruit, and so forth) and solder them on the PCB board by ourselves after we receive the PCB board. Since I’m not good at soldering, PCBA will be helpful. But of course, it comes with the cost. The cost of PCBA depends on some factors like the followings.

• Assembly Type (“Economic” or “Standard”): Assembly requires “setup fee” and “Standard” assembly needs higher setup fee than “Economic” assembly. If we want to go cheap, we need to avoid some conditions that require “Standard” assembly. For example, some very tiny components need “Standard” assembly, placing a component on the back of the board (double-sided PCB) needs “Standard” assembly, and so forth.
• Component Cost: We need to make sure the unit cost of a component is acceptable by comparing it with the same one which we can buy somewhere else like AliExpress, Adafruit and so forth.
• Component Type (“Basic” or “Extended”): Resistors, capacitors and so forth, especially in ordinary sizes, tend to be categorized as “Basic” component. But sensor chips tend to be categorized as “Extended” component. “Extended” component requires an additional fee (3 USD at the time of this writing). For example, if we place 5 different “Extended” sensor chips, we need additional 15 USD. Considering affordable sensor boards we can buy from AliExpress, we cannot just ignore 3 USD.

If we order lots of PCB boards and components, these additional costs will be balanced. But these will be relatively huge for a weekend project like this. If we want to keep the cost lower, we need to consider factors above when we design a PCB board.

Sensor Chip or Sensor Board

As I described above, I prefer to use a sensor board in general. Soldering a tiny sensor chip is way beyond my poor soldering skill. But a sensor board tends to come with on-board pull-up resistors for I2C bus. And a sensor board is much larger than an actual sensor chip. If we use multiple sensor boards, we will have redundant pull-up resistors and we will need a larger enclosure. In this project, I might be able to use a sensor chip if I use PCBA without soldering by myself. But it depends on the cost as described. If the cost of PCBA is too expensive for a sensor chip, I need to use an equivalent sensor board instead which I can buy from AliExpress, Adafruit and so forth.

• Temperature and Relative Humidity: Based on my experience in V1, I stick with SHT41 in V2. I can find some SHT41 sensor chips as “JLCPCB Assembled” in the parts library on EasyEDA, like SHT41-AD1B, SHT41-AD1F (with membrane cover), SHT41I (for Industry), SHT41A (for Automotive) and some revisions of each (R1 or R2). At the time of this writing, they are all “Extended” components. But I found that at least SHT41-AD1B-R2 is eligible for “Economic” assembly and its unit cost is not that expensive compared to a sensor board I bought from AliExpress. So, I decided to use a sensor chip this time.
• Illuminance and UV index: In V1, I used BH1750 for illuminance and LTR390 for illuminance and UV index. Since I have not found a good use case for UV index, I’m going to use either one of them in V2 to measure at least illuminance. I can find a BH1750 sensor chip as “JLCPCB Assembled” in the parts library, and it is categorized as “Extended” but eligible for “Economic” assembly at the time of this writing. But as I described in another post about V1, having an optical window for a sensor chip and guiding the light to a sensor chip is important for measuring illuminance. If I place a sensor chip on a PCB board, that I will place on the back plate of the case, there will be distance from the cover of the case, and it will be challenging to guide the light onto the sensor chip. So, I need to use a sensor board so that I can place it closer to the cover. But it does not necessarily mean I cannot use a sensor chip. If I can design a custom sensor board for a BH1750 sensor chip by myself, I can use a sensor chip on it. Regarding LTR390, I couldn’t find any in the parts library at the time of this writing. So, I need to buy a sensor board somewhere if I use LTR390.
• CO2: In V1, I used SCD40. I’m going to use the same SCD40 or similar SCD4X for V2. I can find some SCD4X sensor chips as “JLCPCB Assembled” in the parts library. But at the time of this writing, the unit cost is relatively expensive compared to a sensor board I can buy from AliExpress. And they require “Standard” assembly. So, I decided to continue to use a sensor board from AliExpress.
• VOC and NOx: In V1, I used SGP41. I’m going to use the same SGP41 for V2. I can find a SGP41 sensor chip as “JLCPCB Assembled” in the parts library. But at the time of this writing, it’s out of stock. So, I decided to use a sensor board from AliExpress.
• PM: In V1, I used SEN55. But it’s large and could not fit in a plastic case with an official holder from Sensirion. I managed to fit it inside the case by thinning the official holder. But the PCB board design will be challenging this time. Newly released SEN6x looks relatively smaller than SEN5x, but it is still thick and expensive. Although SEN5x allows me to ditch other sensor chips or boards like temperature, relative humidity, VOC and NOx since it includes those within it, I decided to use SPS30 instead of SEN5x this time. SPS30 seems to be a little bit old compared to SEN5X, but it is much smaller. Like SEN5x, SPS30 is more like a sensor “module” rather than a sensor “chip” or a sensor “board”. I could not find SPS30 as “JLCPCB Assembled” in the part library. So, I decided to buy SPS30 somewhere and found one on AliExpress.
• Pressure: In V1, I used BMP280. There seem to be other options like BMP388 or BMP390 nowadays. I can find those sensor chips as “JLCPCB Assembled” in the parts library. But BMP280 is out of stock at the time of this writing and BMP388/BMP390 are relatively expensive compared to sensor boards I can buy from AliExpress. They require even “Standard” assembly. I also considered other options like MPL3115A2, MS5611, and DPS310, but they are relatively expensive and requires “Standard” assembly as well. I decided to use a sensor board of BMP280 from AliExpress.
• Presence: In V1, I used LD2420 because LD2410C, which I tried first, was somehow unstable. I’m going to use LD2410C this time again. I can find LD2410C sensor “board’ as “JLCPCB Assembled” in the parts library at the time of this writing. But since I already have LD2410C, I’m going to use the one I already have.
• Ambient Noise: As I described earlier, I’m going to use ICS-43434. I can find an ICS-43434 sensor chip as “JLCPCB Assembled” in the parts library. It is “Extended” component but eligible for “Economic” assembly. However, it seems that it requires an additional fee for X-ray soldering check. And most importantly, we need a mic hole on a PCB board. If I place a sensor chip on a PCB, a mic hole needs to be opened to the back of the case, which is not good. So, I decided to use a sensor board so that I can place a mic hole closer to the cover of the case.

As a result, I can use a sensor chip for SHT41 and possibly BH1750. If I use a sensor chip for BH1750, there will be 3 sets of pull-up resistors on I2C bus (SCD40, SGP41 and BMP280). If I buy a sensor board of BH1750 or LTR390, there will be 4 sets of pull-up resistors. Although combined resistance of 4 sets of pull-up resistors is still higher than the minimum requirement and it might be OK, it’s a little bit disappointing because I cannot reduce redundant pull-up resistors compared to V1. So, I’m going to figure out a way to use a sensor chip for BH1750 and create a custom sensor board.

Designing and Ordering PCB

Now, I’m going to design a PCB board to place necessary components and connect all of them.

In V1, I used three sizes of cases and built the Basic, the Plus and the Max sensor nodes where each of them has only necessary sensor boards. In V2, I’m going to design the only one PCB board for the large case that has only an on-board SHT41 sensor chip and female headers or connectors for other sensors. I can just attach necessary sensor boards or sensor modules for each sensor node. Though I’m not sure if it is a good idea to make it more like a custom breadboard and rely on pin headers for connections.

Regarding placement of components, I decided to place a S3 mini at the bottom center of the PCB board so that I can connect a USB-C cable from outside of the case. And I place SPS30 on the left side and the exhaust fan on the top right so that the air coming out from the outlet of SPS30 won’t go back to the inlet, hopefully. I also place a sensor chip and sensor boards on the right side to get benefit of the air flow, hopefully.

When it comes to actual connections between the PCB board and components, I decided as follows.

• Microcontroller (S3 mini): I placed through-holes and female headers for a S3 mini so that I can attach a S3 mini on those female headers. If I use male pin headers, I can reduce the height of a USB-C connector of a S3 mini. But it will be still higher than a cable hole on the bottom of the case and it cannot be accessible from outside of the case. So, I’m going to open or widen a hole on the bottom of the case somehow. And I decided to use PCBA option for female headers because it’s just a lot of soldering. I also placed additional through-holes and female headers for S3 mini breakout for future experiment. And I placed decoupling capacitors (ceramic, 0603 size, 10µF) near 3.3V and 5V (VBUS) pins and decided to use PCBA option for them.
• Sensor Chip (SHT41): I placed a sensor chip of SHT41 directly on the PCB board and decided to use PCBA option as I described. I didn’t place pull-up resistors for I2C because I will add one or more sensor boards which have already on-board pull-up resistors. According to the official design guide, I need to avoid thermal conduction through the PCB board. It seems that I better use “milled slits” around a sensor chip. So, I decided to place the sensor chip close to the bottom vents and create cutouts around the sensor chip. I also placed a small decoupling capacitor (ceramic, 0603 size, 0.1µF) near a sensor chip and decided to use PCBA option.
• Sensor Chip (BH1750): I decided to create a custom sensor board with a sensor chip of BH1750 as I described. But I didn’t want to order two different PCB designs because it would be costly. I realized that it would be possible to put different PCB designs in one design. So, I decided to put a PCB board for V2 and a custom sensor board for BH1750 together in one PCB design. I placed through-holes for female headers to attach a custom sensor board on the main PCB board and placed through-holes for male pin headers on the custom sensor board. I also placed a small decoupling capacitor (ceramic, 0603 size, 0.1µF) near a sensor chip and decided to use PCBA option.
• Sensor Board (SCD40, SGP41, BMP280, ICS-43434 and LD2410C): I placed through-holes for sensor boards and decided to use female headers so that I can attach necessary sensor boards by myself. I decided to use PCBA option for female headers if they are available in stock. And I’m not going to place decoupling capacitors for sensor boards because they seem to already have capacitors.
• Sensor Module (SPS30): I placed a JST ZH header so that I can connect a SPS30 with a cable that has JST ZH connectors in both ends. And I decided to use PCBA option for a JST ZH header. Regarding the cable, I bought a pre-crimped ZH cable set from Amazon. I also placed a small decoupling capacitor (ceramic, 0603 size, 0.1µF) near the JST ZH header and decided to use PCBA option.
• Exhaust Fan: I placed a JST SH header so that I can connect an exhaust fan with a cable that has a JST SH connector in one end. I also placed a 2K resistor as pull-up for RPM signal reading and decided to use PCBA option for the resistor. I also placed a small decoupling capacitor (ceramic, 0603 size, 0.1µF) near the JST SH header and decided to use PCBA option.
• LED Strip (SK6812): I decided to place a JST XH header so that I can connect a LED strip with a cable that has a JST XH connector in one end. And I decided to use PCBA option for a JST XH header. Regarding the cable, I bought a pre-crimped XH cable set as described earlier. I also placed a level shifter (74LV1T34) and decided to use PCBA option for it. I also placed a small (ceramic, 0603 size, 0.1µF) and a large decoupling capacitor near a the JST XH header and decided to use PCBA option. Regarding a large capacitor, it seems that the capacitance between 100µF and 1000µF is recommended. Considering the unit price and the size, I used a tantalum 470µF capacitor.
• Buzzer: I placed through-holes for a buzzer so that I can attach the buzzer by myself. I also placed a small decoupling capacitor (ceramic, 0603 size, 0.1µF) near the buzzer and decided to use PCBA option.

After many try-and-errors, my PCB design became like the following. Although I’m just a newbie and it would be just horrible, you can see my PCB design in the pictures below. Since I’ve put the main PCB board and a custom sensor board for BH1750 together in the same PCB design, you can see the BH1750 sensor board sticking out. It’s basically separated from the main PCB board and can be easily detached from the main board.

Once I had done my PCB design on EasyEDA, I could seamlessly order the PCB board on JLCPCB. To be honest, I went back and forth between EasyEDA and JLCPCB many times in my try-and-errors though. Since the minimum order is 5 for “Economic Assembly”, the total PCB cost was about 55 USD (PCB: 2 USD and PCBA: 53 USD). It means that the unit cost of the PCB board was 11 USD. Considering it includes SHT41 and BH1750 on-board and eliminates all the wiring tasks, it might be worth it. But the shipping fee was quite expensive and costed 13.4 USD for the cheapest shipping method (Global Standard Direct Line). I’m still not sure if the total cost + the shipping fee + tax would worth it or not, especially for my weekend project. Although EasyEDA gives us discount coupons like “10 USD off for the first time order”, I didn’t take them into account when I calculated the cost above because they are not what we always get.

This is what I got from JLCPCB after 16 days. It looks great.

As intended, I could easily detach a BH1750 sensor board from a main board with a diagonal cutter.

Building DIY Environment Monitor V2

The custom PCB board allowed me to make building process much easier. I still needed soldering, but just for pin headers.

• Microcontroller (S3 mini): I soldered male pins to a S3 mini and attached it on the PCB board. Because a USB-C port is way above the cable hole on the bottom of the case, I cut vents on the bottom of the case and created another cable hole so that I can access the USB-C port from outside the case.

• Sensor Board (BH1750, SCD40, SGP41, BMP280, ICS-43434 and LD2410C): I soldered male pins to them and attached them to the PCB board. Regarding LD2410C, because it came with male pins soldered, I could just attach them to the PCB board. Because I need to place BH1750, ICS-43434 and LD2410C closer to the cover of the case and avoid conflict with SPS30, I used short female headers and double-stacked them on female headers on the PCB board. I also drilled small holes for BH1750 and ICS-43434 on the cover of the case. Honestly, I regret that holes are not well aligned. To increase the light through the hole, I used a small glass lens that I bought from Amazon.
• Sensor Module (SPS30): I created a cable which has JST ZH connectors in both ends by using a pre-crimped cable kit and connected SPS30 to the JST ZH header on the PCB board. I used a tie-band to place SPS30 on the PCB board.
• Exhaust Fan: Because it came with a JST SH connector, I could just connect it to the JST SH header on the PCB board. I used hot glue to attach the fan to the backplate of the case. Because the clearance between the fan and the cover was too tight than I expected, I needed to tilt the fan so that I could manage to fit it inside the case.
• LED Strip: I created a cable which has JST XH connector on one end by using a pre-crimped cable kit and soldered on another end to the LED strip. I drilled a hole on the back plate of the case and routed the cable of the LED strip to the back of the case. Since the case is too small for a 30LEDs/m strip, I could use only two LEDs.

• Buzzer: Because it came with male pin headers, I could just solder it to the PCB board.

This is what I’ve put all together.

Configuration

Since I used only ESPHome-ready components, configuration was straightforward.

esphome:
  name: s3-mini
  friendly_name: s3-mini
  platformio_options:
    platform: https://github.com/platformio/platform-espressif32.git
    board: lolin_s3_mini
    framework: arduino
    board_build.flash_mode: dio

esp32:
  variant: esp32s3
  board: lolin_s3_mini
  framework: 
    type: arduino
    version: latest

logger:
  baud_rate: 0

api:
  encryption:
    key: !secret api_key

ota:
  - platform: esphome
    password: !secret ota_password

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

  ap:
    ssid: "S3-Mini Fallback Hotspot"
    password: !secret ap_password

captive_portal:

esp32_ble_tracker:

external_components:
  - source: github://stas-sl/esphome-sound-level-meter

i2c:
  sda: GPIO35
  scl: GPIO36
  scan: True

uart:
  tx_pin: GPIO43
  rx_pin: GPIO44
  baud_rate: 256000

ld2410:

sensor:
  - platform: bh1750
    name: Illuminance
    address: 0x23
    update_interval: 60s
  - platform: sht4x
    address: 0x44
    update_interval: 60s
    temperature:
      name: Temperature
      id: temperature_sht41
    humidity:
      name: Humidity
  - platform: sgp4x
    address: 0x59
    update_interval: 60s
    voc: 
      name: VOC Index
    nox: 
      name: NOx Index
  - platform: scd4x
    address: 0x62
    update_interval: 60s
    co2:
      name: CO2
    temperature:
      name: Temperature SCD40
    humidity:
      name: Humidity SCD40
  - platform: sps30
    pm_1_0:
      name: PM 1.0
      id: pm_1_0
    pm_2_5:
      name: PM 2.5
      id: pm_2_5
    pm_4_0:
      name: PM 4.0
      id: pm_4_0
    pm_10_0:
      name: PM 10.0
      id: pm_10_0
    pmc_0_5:
      name: PMC 0.5
      id: pmc_0_5
    pmc_1_0:
      name: PMC 1.0
      id: pmc_1_0
    pmc_2_5:
      name: PMC 2.5
      id: pmc_2_5
    pmc_4_0:
      name: PMC 4.0
      id: pmc_4_0
    pmc_10_0:
      name: PMC 10.0
      id: pmc_10_0
    pm_size:
      name: Typical Particle Size
      id: pm_size
    address: 0x69
    update_interval: 60s
  - platform: bmp280_i2c
    temperature:
      name: Temperature BMP280
      oversampling: 16x
    pressure:
      name: Pressure
    address: 0x76
    update_interval: 60s
  - platform: pulse_counter
    pin: GPIO4
    name: Fan RPM
    id: fan_rpm
    unit_of_measurement: "RPM"
    accuracy_decimals: 1
    filters:
      - multiply: 0.5
    count_mode:
      rising_edge: INCREMENT
      falling_edge: DISABLE
    update_interval: 10s

binary_sensor:
  - platform: ld2410
    has_target:
      name: Presence
    has_moving_target:
      name: Moving Target
    has_still_target:
      name: Still Target
    out_pin_presence_status:
      name: out pin presence status
  - platform: gpio
    pin: GPIO16
    name: gpio out pin presence
    device_class: presence

select:
  - platform: ld2410
    light_function:
      name: light function
    out_pin_level:
      name: out pin level

output:
  - platform: ledc
    pin: GPIO2
    id: fan_pwm
  - platform: ledc
    pin: GPIO10
    id: buzzer

fan:
  - platform: speed
    output: fan_pwm
    name: Fan Toggle
    id: fan_toggle

light:
  - platform: esp32_rmt_led_strip
    id: onboard_led
    name: Onboard LED
    rgb_order: RGB
    chipset: WS2812
    rmt_channel: 0
    num_leds: 1
    pin: 47
  - platform: neopixelbus
    type: GRBW
    variant: sk6812
    pin: GPIO18
    num_leds: 2
    name: "LED Light"

rtttl:
  output: buzzer
  id: my_rtttl
  gain: 60%

i2s:
  ws_pin: 01
  bck_pin: 05
  din_pin: 03
  sample_rate: 48000
  bits_per_sample: 32
  channel: left
  bits_shift: 8

sound_level_meter:
  update_interval: 150s
  buffer_size: 1024
  mic_sensitivity: -26dB
  mic_sensitivity_ref: 94dB
  groups:
    - filters:
        - type: sos
          coeffs:
            # A-weighting:
            #       b0           b1            b2             a1            a2
            - [0.16999495, 0.741029, 0.52548885, -0.11321865, -0.056549273]
            - [1., -2.00027, 1.0002706, -0.03433284, -0.79215795]
            - [1., -0.709303, -0.29071867, -1.9822421, 0.9822986]
      sensors:
        - type: eq
          name: LAeq_1min
          id: LAeq_1min
          unit_of_measurement: dBA
        - type: max
          name: LAmax_125ms_1min
          id: LAmax_125ms_1min
          window_size: 125ms
          unit_of_measurement: dBA
        - type: min
          name: LAmin_125ms_1min
          id: LAmin_125ms_1min
          window_size: 125ms
          unit_of_measurement: dBA

Result

Technically, my V2 sensor node worked as expected. It measured all the environment metrics — Temperature and Relative Humidity, Illuminance, CO2, VOC Index, NOx Index, PM 0.5/1.0/2.5/4.0/10.0, Pressure, Presence and Ambient Noise Level. The exhaust fan started spinning when I turned it on, the LED turned on, the buzzer worked, and the PCB board worked as well. All the components worked and ESPHome worked great.

But considering objectives, I have to admit that the result was completely failure this time.

• Although I placed cutouts around a SHT41 sensor on the PCB board, it measured very higher values (+3°C to +5°C) than my reference (WxBeacon2 and SwitchBot Meter) even when I didn’t close the cover. And when I closed the cover, it measured even way higher (+8°C to +9°C) than my reference. Probably because I packed too many components and features within the case, there would be more heat coming out from components and I couldn’t avoid thermal conduction enough. I probably might better separate a temperature and humidity sensor as a sensor board rather than placing it on the main PCB board.
• The exhaust fan was not effective at all. First, my choice of 25mm was too large for the case. As I described, I tilted the fan and managed to stuff it into the case. But once I turned on the fan, it made large rattling noise because it hit the cover and cables. And even if I ignored the noise and kept it running, measured temperature didn’t decrease at all. Probably because placing the fan on top vents to suck the air out of the case was not a good idea, attaching the fan only with hot glue was not a good idea, and the fan was too large for the size of the case but was too small for this purpose. I probably might better expose a temperature and humidity sensor to the air outside the case through a hole or something rather than trying to take the warm air out of the case.
• The LED strip worked. But when I placed a sensor node on the wall, it was not enough for even indirect lighting. I need more gap and angle between the LED and the wall. Attaching the LED strip on the back plate was not just a good idea.
• Although BH1750 can measure the ambient light and ICS-43434 can measure the ambient noise, holes on the cover were not well aligned with them. I might need to think about a better approach.

Final Thoughts

Although the result was completely failure and disappointing, I learned a lot this time. Especially, it was the first time for me to design a PCB board and it was a very great experience. Since the PCB board worked as expected, it gave me some level of confidence. Based on the failure this time, I’ll try to make improvements in the next version.

Disclaimer: I’m not affiliated with manufacturers, sellers and businesses of enclosures which I refer to in this post. I’m just a user and a hobbyist.

Requirements

• Affordable price: In my case, I cannot afford to spend much on a project box. For example, when I created my sensor nodes in DIY environment monitor project, I needed 9 project boxes. And some I2C sensors were relatively expensive. I had to figure out the way to reduce the total cost to measure environment metrics all over the house. It’s too much for me to spend tens or hundreds of dollars on a project box. I prefer to pick up an affordable project box which is under, let’s say, 5 USD (including shipping fee). The cost requirement depends on each project and each person’s budget though.
• Appearance: This depends on our preferences. At least I prefer to have simple and subtle appearance of a project box when I place it somewhere in a room, like a wall of a room. It doesn’t have to be either gadget-y with shiny LEDs or heavy-duty.
• Size: We need to make sure if a project box has enough space to fit all the necessary components — a microcontroller, sensors and wires.
• Mounting mechanism: We need to think about how to mount a project box somewhere in a house, like a wall. We don’t want to just place it on a desk like a prototyping. An adhesive tape could work in some cases. But in case a box and components have some weight, or they get warmed up and could weaken an adhesive tape, we might need at least a hole to hook it on a wall or a hole to screw it on a wall.
• Vents: This is not always the case, but we might need vents or holes for air inlets/outlets or lighting depending on what we want to measure with sensors in a box. If you need to have waterproof or weatherproof, you might prefer to have better sealing rather than vents or holes though.
• Cabling mechanism: We might need to think about cabling as well. Especially when we connect a project box to a certain power source, it needs to have a hole for cabling.
• Durability: Although a project box does not have to be heavy-duty since we are talking about our DIY smart home projects, it cannot be too fragile and easily broken. We need to balance between the durability and the cost.
• Waterproof/Weatherproof: In some use cases like some monitoring in an outdoor area, we might need waterproof/weatherproof. I myself don’t have many use cases though.

Options

We can find several plastic project boxes which have holes and/or vents on the Internet. We can even see almost or exact the same boxes from different sellers. I’m not sure which one is the original. And I’m not sure what purposes these boxes were originally designed for.

Option 1: This is my favorite. So far, I have found 3 sizes (small, medium and large) and 1 variant of a large one. I used them in DIY environment monitor and a variant in DIY smart button. We can find them from different sellers on AliExpress or other online shops (like this, this, this and this).

• Pros: They are very affordable and each cost just around 1 USD at the time of this writing. We can buy them from different sellers on AliExpress. Because they are likely to be “Free shipping over $10” on AliExpress, it won’t be difficult to get free shipping if we buy a microcontroller and sensors together with the box. They all look very simple and subtle. And they have vents on top and bottom, a cabling hole on bottom and a mounting hole on the back. They have even screw posts on corners of the back panel, which we can use to place a PCB board if we want. We don’t have to make them with some tools by ourselves. If we need to use several boxes, it will be helpful. Although they are not waterproof/weatherproof, they are relatively sturdy.
• Cons: If we use a large sensor like Sensirion SEN5x, even a large box is a bit small in height as I described in another post. And although a USB-C cable can fit in a cabling hole, the hole is a bit small for a USB-C connector as I describe in another post as well. Which means that we need to place a microcontroller in an upper area so that an entire USB-C connector can fit within a box. Otherwise, we cannot close a cover.

Option 2: I found this on AliExpress (like this). There are 3 sizes, and I purchased a medium one and a large one. I used a large one in another project.

• Pros: They are very affordable and a set of several pieces cost just around 1 USD at the time of this writing. They are likely to be “Free shipping over $10” on AliExpress. Appearance seems to be a bit gadget-y compared to Option 1, but it is still simple (just my opinion). And they have vents on a cover, a cabling hole on bottom and a mounting hole on two corners of a cover.
• Cons: They are basically too small. It could work if we put only a microcontroller without any sensors and wires. We need to widen a cabling hole because it is too small even for a USB-C cable. And they are relatively soft and a bit fragile and even bent a bit from the first. Considering the price, it should not be a big problem though.

Option 3: I found this on AliExpress (like this). I have purchased it to see how it actually looks like. But I have not yet used it.

• Pros: It is a bit expensive compared to Option 1 and 2, but still affordable and under 5 USD at the time of this writing. It’s also likely to be “Free shipping over $10” on AliExpress. It looks simple and subtle. It has vents and 4 mounting holes around a box. It also has a rectangle hole for some reasons that I don’t know.
• Cons: The size is between a medium one and a large one of Option 1. Which means that it will be too small for a large sensor. And there is no cabling hole. A USB-C cable can run through the rectangle hole, but it will not be practical as you can see a picture above. We might need to make a cabling hole if we need.

Option 4: There are other simple plastic project boxes with vents and/or holes which are similar to Option 1–3 (like this, this, this and this). Although they themselves are under 5 USD, the total cost exceeds 5 USD including shipping fees because they are not “Free shipping over $10” on AliExpress at the time of this writing. So, I’m going to separate them from Option 1–3. I have purchased one of them to see how it looks like. But I have not yet used it.

• Pros: It is larger than a large one of Option 1. Although it’s a bit noticeable in its size, it still looks simple and relatively subtle. Because of its intended usage “Text Display”, which I’m not exactly sure, it could be possible to put something on the cover. It has vents on top and bottom, and mounting holes on top and bottom as well. But I’m still not sure how I use this mounting mechanism though.
• Cons: It is a bit expensive compared to Option 1 and 2. Although the box itself is still under 5 USD, the total cost including the shipping fee will be over 5 USD. Although it’s a large enclosure, it is a bit thinner. Which means that it is still a bit small for a large sensor. If we use a SEN5x official holder, we cannot close a cover. A thinner holder, which I created in another post, could fit. But I have not yet tried. And there is on cabling hole. A USB-C cable cannot go through a mounting hole. We need to make a cabling hole somewhere.

Option 5: I found this on AliExpress (like this). It looks a bit different from Option 1–4 because it is designed for a specific use case like an environment monitor or a smoke detector. There are many variations from the same seller (like this, this and this), and I purchased one of them to see how it actually looks like. I have not yet used it.

• Pros: It is designed very well. It will look like a nice smoke detector or something. It has small holes on a cover which can be used as vents, a cabling hole on bottom and a nice mounting bracket (not just a hole). It also has a tiny hole on bottom, which we often see in some products for factory reset.
• Cons: Although it is under 5 USD, it is not “Free shipping over $10” on AliExpress and the shipping cost is more expensive than the price of the box at the time of this writing. We need shipping fee for each box. The total cost will exceed 5 USD. The size of the box is between a medium one and a large one of Option 1. But it is thinner. The biggest problem is the size of the cabling hole. Only a tip of USB-C connector can go through the hole. It is a bit short for plugging into a microcontroller through this hole. Probably it is designed to fit a custom PCB board.

Option 6: When we search “project box” on the Internet, we can find many general-purpose boxes in different sizes (like this or this). They are made with either metal or plastic. A plastic one will be easier to use if we think about making vents or holes. And a plastic one will be suitable if we want to use some sensors like mmWave presence sensor or need to have better Wi-fi or BLE signal.

• Pros: They tend to be cheap. If you buy them from AliExpress, they cost only a couple of USD or even less than 1 USD. They are likely to be “Free shipping over $10” on AliExpress. From appearance perspective, they look simple and subtle because they are just bare boxes. Because there are a wide variety of sizes, it will be easy to find a size that fits our needs. They tend to have some level of durability as well.
• Cons: We need to create necessary vents or holes with some tools by ourselves. Although they tend to claim waterproof/weatherproof, it does not mean it can actually have waterproof/weatherproof after you create vents or holes.

Option 7: When we search “project box” on the Internet, we can find many junction boxes, electrical boxes, outlet boxes and switch boxes as well (like this and this). They are intended for professional usage like housing electrical components or wires. But it could be possible to use them for DIY smart home projects if they fit our needs. Some of them seem to be for more like IoT projects (like this). They are made with either metal or plastic

• Pros: The price varies, but there could be ones under 5 USD. They tend to have some holes and bushing or glands for cabling. If you are looking for waterproof/weatherproof, they will be helpful. They tend to be heavy-duty.
• Cons: Since it is intended for professional work, it definitely fits in a circuit box area, a garage, an outside area and even inside a wall in a house. But it does not necessarily look like something you place on a wall of living room. We can find some of them with vents, but they usually don’t have vents.

Option 8: I found a Stevenson screen type of enclosure on AliExpress (like this). It is not just a project box though. It seems that it is originally created for a weather station product but sold separately on its own as spare parts. I have not purchased it.

• Pros: It will be good if you specifically need a Stevenson screen type of enclosure for your outdoor environment monitoring.
• Cons: It will be a bit too much for indoor usage.

Option 9: We could use a Raspberry Pi case as a project box. There are many choices from the official store and other brands. They are made with either plastic or metal.

• Pros: Although they are computer cases, some of them could look simple and subtle. They have holes for interface ports of Raspberry Pi which we can use as vents and a cabling hole.
• Cons: They are likely to be over 5 USD.

Option 10: I found this on AliExpress. It is a smoke detector “product”. I have purchased it to see if its enclosure can be re-purposed as a project box.

• Pros: It is under 5 USD and even cheaper than Option 5. Because it is an actual product, it will have a better quality and fit in a wall. It has vents and nice mounting bracket.
• Cons: Taking guts out of it will need some work. And it does not have a cabling hole because it is intended to be powered by a battery. Although the size is between a medium one and a large one of Option 1, it is a bit small for DIY projects.

Final Thoughts

At the time of this writing, I think Option 1 is still the best and easiest to use. But I wish I could have a bit larger one. Considering their prices, it is hard to ask for more though. Let me know if you find a better option.

Architecture

There are two options. The one is collecting data within the car and sending them directly to Home Assistant (“Direct” approach). Another one is collecting data within the car, sending them to Cloud and then collecting them from Cloud to Home Assistant (“Indirect” approach).

• “Direct” approach: In this approach, we need to send data from a car to Home Assistant at home over either Wi-fi at home or cellular network. If we use Wi-fi at home, we can use either ESPHome native API with Home Assistant or MQTT. In either case, there is still time limitation problem where a car is within Wi-fi signal range. And BLE Node will face another problem outside Wi-fi signal range because ESPHome relies on Wi-fi connectivity. If we use cellular network, we can use MQTT. But we need to set up MQTT broker at home and open a port for MQTT broker on an internet router. And of course, we need to have cellular network capability somehow.
• “Indirect” approach: In this approach, we send data from a car to Cloud first and collect data from Cloud to Home Assistant at home. We can still use Wi-fi at home when a car is within Wi-fi signal range but will have time limitation problem. If we use cellular network, we can use MQTT broker on Cloud, publish data from a car and subscribe data from Home Assistant at home. Although we need to prepare cellular network capability somehow, we can collect data anytime at anywhere while a car is running.

I decided to use “Indirect” approach this time and use cellular network. As I described in the previous post, there are already many commercial solutions (like this, this and this). But I prefer to DIY. So, I’m going to create an affordable version — I place a BLE Node in a car, collecting data with an OBD2 adapter over BLE, publishing MQTT data to AWS IoT Core over cellular network, and subscribing MQTT data from Home Assistant at home. In fact, there is already a best practice published from AWS, which features Arduino for connecting ESP32 to AWS IoT. In this project, I’m going to use ESPHome.

Material

BLE Node: Like the previous post, it’s just an ESP32 controller in an enclosure. I used a Lolin S3 mini in the previous post. Since I need to use not only BLE but MQTT this time, the size of the program becomes larger and S3 mini had limitation. So, I use a Lolin S3 this time. I bought it from the official store on AliExpress. Regarding an enclosure, I have been using the same simple and affordable case in my projects (like this, this, this and this). But this time, I tried another simple and affordable case, which I also found on AliExpress. I bought two sizes for just in case (70x35x17mm and 88x40x18mm). Each of them came with 5 pieces. I don’t need many though. One piece would cost under 0.5 USD. But it turned out that Lolin S3 would not fit in a smaller one. And even a larger one didn’t have enough space for a USB type-C connector of a cable. I had to cut a case to widen a hole. Since it was a cheap and even a bit soft plastic case, a box cutter was enough to widen a hole. I need to find a better case for future improvement. And as always, I attached Lolin S3 to the backplate of the case with hot glue.

Cellular Network: I always bring my iPhone to use CarPlay when I use my car. It’s reasonable for me to rely on cellular connectivity of my iPhone rather than buying another mobile router or adding cellular capability to ESP32 somehow. So, I’m going to use Personal Hotspot of my iPhone.

The additional cost of this project including a Lolin S3 and a case was 8 USD at the time of this writing. Since I use my iPhone for cellular connectivity, I can take care of its cost in my existing data plan.

Implementation

I already created an external component of ESPHome called “obd2_ble” that collects data from a car with an OBD2 adapter over BLE in the previous post. I used ESPHome native API to publish data to Home Assistant. This time, I’m going to use MQTT to publish data to AWS IoT Core. Thankfully, ESPHome already has MQTT Client Component. Like I used BLE Client of ESPHome for connecting the BLE Node to an OBD2 adapter, I can use MQTT Client Component for connecting the BLE Node to a MQTT broker on AWS IoT Core. Regarding Home Assistant, it has MQTT integration for connecting it to a MQTT broker as I explained in another post. Let’s go into each of them.

AWS IoT Core: As I described in another post about how to send data from Home Assistant to AWS IoT Core, we can define a Thing for connecting a device like the BLE Node to a MQTT broker on AWS IoT Core and define a Device Shadow to maintain state of it. Although a Device Shadow is not mandatory if we need only MQTT capability, it might help us manage IoT data. When I connected Home Assistant to AWS IoT Core in another post, I defined Home Assistant as a Thing and created multiple Named Shadows for my DIY Environment Monitors to manage data, which means that Home Assistant was only one MQTT client device directly connecting to AWS IoT Core. In this case, one option would be just adding another Named Shadow for the BLE Node under the existing Thing. Another option would be defining a new Thing (and defining a new Device Shadow as well). Since AWS IoT Core allows us to separate credentials among Things, it would be ideal if we define different Things for different devices with different credentials from security perspective. So, I decided to define a new Thing that represents my car and define a Named Shadow for the BLE Node because Home Assistant and the BLE Node are two different clients from AWS IoT Core perspective. When we define a Thing, we can let AWS IoT Core autogenerate certificates as below. We need “Device certificate” (…-certificate.pem.crt), “Private key file” (…-private.pem.key) and one of “Root CA certificates” (AmazonRootCA1.pem) later. And we also need to make sure AWS IoT policy attached to the existing certificates for the existing Thing allows access to subscribing the new MQTT topics of the new Thing. If it’s not allowed, Home Assistant cannot subscribe the MQTT topics for the BLE Node later.

BLE Node: Regarding connection to AWS IoT Core, we can just use MQTT Client Component and certificates autogenerated by AWS IoT Core. We already downloaded certificates from AWS IoT Core as above. All we need is just to copy and paste contents of those certificates on ESPHome yaml configuration — the contents of AmazonRootCA1.pem go to “certificate_authority”, the contents of Device certificate go to “client_certificate” and the contents of Private key file go to “client_certificate_key”. Just make sure you put indent as below. At the time of this writing, I also had to disable discovery features as follows. If I didn’t disable them, the BLE Node got disconnected from AWS IoT Core after connecting to it. As you can see below, we still see “api” and “ota” sections, and “wifi” section still expects Wi-fi information at home because we assume we use Wi-fi at home during try-and-error process. But once we make sure it will work, we remove “api” and “ota” sections and change information in “wifi” section from Wi-fi at home to iPhone Personal Hotspot.

esphome:
  name: s3
  friendly_name: s3
  platformio_options:
    board_build.flash_mode: dio

esp32:
  board: lolin_s3
  variant: esp32s3
  framework:
    type: esp-idf

logger:
  level: DEBUG

api:
  encryption:
    key: !secret api_key

ota:
  - platform: esphome
    password: !secret ota_password

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

mqtt:
  broker: xxxxxxxxxxxxxx-ats.iot.us-east-1.amazonaws.com
  port: 8883
  certificate_authority: |
    -----BEGIN CERTIFICATE-----
    ...
    ...
    -----END CERTIFICATE-----
  client_certificate: |
    -----BEGIN CERTIFICATE-----
    ...
    ...
    -----END CERTIFICATE-----
  client_certificate_key: |
    -----BEGIN CERTIFICATE-----
    ...
    ...
    -----END CERTIFICATE-----
  discover_ip: False
  discovery: False
  discovery_retain: False
  log_topic: null
  birth_message: 
    topic:
    payload:
  will_message: 
    topic:
    payload:

When it comes to publishing data with MQTT from the BLE Node, there seem to be several options. Since I’m creating an external component, I’d like to use something already developed in esphome/components/mqtt. One option would be MQTTClientComponent. But it looks like a foundation class of MQTT connection handling and too much for me trying to just publish data. Another option would be MQTTComponent. But it also looks too much for me. I would still need to override many functions. One more another option would be CustomMQTTDevice, which allows me to use just a few helper functions including publishing data. I decided to use CustomMQTTDevice this time. As I described in the previous post, my OBD2BLEClient class inherits BLEClientNode class and Component class. I added CustomMQTTDevice class as well. I had to implement sending and receiving OBD2 commands and responses over BLE communication in the previous post. But this time, I just check MQTT connection and publish JSON data by using is_connected and publish_json method of CustomMQTTDevice because MQTT Client Component already takes care of MQTT connection.

My example code is here.

iPhone: I don’t want to manually turn on Personal Hotspot on iPhone every time I use my car, and I don’t need to always turn on Personal Hotspot. So, I’m going to use Automation feature of iOS. I use CarPlay connection/disconnection as a trigger to automatically turn on and off Personal Hotspot. At first, I created shortcuts of turning on and off Personal Hotspot based on Set Personal Hotspot. And then, I created automations that trigger CarPlay connection/disconnection and run shortcuts. I’m still not sure but I might need to add disabling Auto-Lock as well.

Home Assistant: Like another post, I’m going to use MQTT integration, not Mosquitto add-on. In my case, I have already set up MQTT integration to send environment data from DIY Environment Monitor. Basically, I don’t need to add anything regarding MQTT connection to AWS IoT. Because MQTT integration connects to just one MQTT broker, I cannot add another MQTT connection anyway. If we need more than two MQTT brokers, we need MQTT bridge like Mosquitto add-on. Although I can use the existing MQTT connection, I just disabled discovery feature of MQTT integration (See the update of another post) at the time of this writing. When it comes to using data receiving from MQTT topics, there are many options on Home Assistant, like MQTT triggered Automation, MQTT sensor, and so forth. I created MQTT sensors this time like below in configuration.yaml of Home Assistant.

mqtt:
  sensor:
    - name: "atrv"
      state_topic: "$aws/things/car-20241210/shadow/name/s3-01/update"
      value_template: "{{ value_json.state.reported.atrv | round(2) }}"
      unit_of_measurement: "V"
    - name: "engine_coolant_temperature"
      state_topic: "$aws/things/car-20241210/shadow/name/s3-01/update"
      value_template: "{{ value_json.state.reported.engine_coolant_temperature | round(2) }}"
      unit_of_measurement: "°C"
    - name: "stft"
      state_topic: "$aws/things/car-20241210/shadow/name/s3-01/update"
      value_template: "{{ value_json.state.reported.stft | round(2) }}"
      unit_of_measurement: "%"
    - name: "ltft"
      state_topic: "$aws/things/car-20241210/shadow/name/s3-01/update"
      value_template: "{{ value_json.state.reported.ltft | round(2) }}"
      unit_of_measurement: "%"
    - name: "fuel_tank_level_input"
      state_topic: "$aws/things/car-20241210/shadow/name/s3-01/update"
      value_template: "{{ value_json.state.reported.fuel_tank_level_input | round(2) }}"
      unit_of_measurement: "%"
    - name: "absolute_barometric_pressure"
      state_topic: "$aws/things/car-20241210/shadow/name/s3-01/update"
      value_template: "{{ value_json.state.reported.absolute_barometric_pressure | round(2) }}"
      unit_of_measurement: "kPa"
    - name: "control_module_voltage"
      state_topic: "$aws/things/car-20241210/shadow/name/s3-01/update"
      value_template: "{{ value_json.state.reported.control_module_voltage | round(2) }}"
      unit_of_measurement: "V"
    - name: "ambient_air_temperature"
      state_topic: "$aws/things/car-20241210/shadow/name/s3-01/update"
      value_template: "{{ value_json.state.reported.ambient_air_temperature | round(2) }}"
      unit_of_measurement: "°C"
    - name: "engine_oil_temperature"
      state_topic: "$aws/things/car-20241210/shadow/name/s3-01/update"
      value_template: "{{ value_json.state.reported.engine_oil_temperature | round(2) }}"
      unit_of_measurement: "°C"

Test

After many try-and-error at home, I’ve finally got the BLE Node working. I commented out “api” and “ota” sections and changed “wifi” section so that the BLE Node connects to iPhone.

#api:
#  encryption:
#    key: !secret api_key

#ota:
#  - platform: esphome
#    password: !secret ota_password

wifi:
  ssid: !secret personal_hotspot_ssid
  password: !secret personal_hotspot_password

I plugged the BLE Node to a USB port in my car. Once I start an engine, it’s powered up and starts searching Wi-fi. When I connect my iPhone to another USB port to use CarPlay, my iPhone automatically turn on Personal Hotspot and the BLE Node connected to Wi-fi and AWS IoT Core. The BLE Node also finds the OBD2 adapter in my car over BLE and starts collecting data from my car and publishing them to AWS IoT Core. While I’m driving my car, the BLE Node continued to collect and publish data. And when I stopped the engine, my iPhone turned off Personal Hotspot and the BLE Node powered down. It was successful.

Although it was successful, I found some issues as well.

• When I tried to put all the supported PIDs as sensors, text sensors or binary sensors as I described in the previous post, MQTT connection could not be established. Probably because BLE and MQTT handling are memory intensive. When I reduced the number of those sensors, it worked as expected. This will be future improvement.
• It seems that MQTT over cellular network with iPhone Personal Hotspot does not always work. Especially when I start my car in an area where cell phone reception is weak, MQTT communication ends up with no success even if cell phone reception becomes better in the middle of the drive. But when I start my car in strong cell phone reception area, MQTT communication becomes successful. Probably because iPhone Personal Hotspot stops working due to network problem even when display is ON.

Result

Unlike the previous post, now I don’t need to worry about time limitation anymore. The BLE Node keeps collecting and publishing data while my car is running, and Home Assistant at home collects the latest data through Device Shadow. As I described above, it does not always work. But it is enough in my use case. It works fines at least once sometime during the day.

Final Thoughts

I successfully collected data from a car to the ESPHome based BLE Node with an OBD2 adapter, sent data from the BLE Node to AWS IoT Core over cellular network and collect data from AWS IoT Core to Home Assistant at home. Although it’s just a DIY project and does not have commercial grade reliability and convenience, I’m satisfied with the result.

My car has a connectivity feature called STARLINK (not the satellite internet service). If I pay subscription fee for the correspondent service, I can check some sensor data (e.g., tire pressure, mileage, etc.), diagnostic data and even remotely start my car on the smartphone app. And there is the Home Assistant custom component to integrate the service with Home Assistant. I actually paid and tried the service for one year. But after one year subscription, I didn’t renew it because I couldn’t get the most from it for the price in my environment. Luckily, I didn’t receive any notification about a problem in my car. Unfortunately, at the same time, it couldn’t find a battery problem, which was the biggest problem for me. And I didn’t have a use case to remotely start my car. After I stopped the service, I had a tire pressure warning on the car dashboard. But it didn’t give me any motivation to pay for the service again because the car let me know a problem on its dashboard. Although the service was not necessary for its price in my case, it was enjoyable to see data collected from my car on Home Assistant. Since then, I have been wondering if I could do cheaper somehow.

There are some comprehensive end-to-end car data logger and GPS tracking services that come with a device to connect to an OBD2 (On-Board Diagnostics) port in a car and collect data to Cloud service via cellular network (like this and this). But they need subscription fee as well. There is also an advanced OBD2 scanner that incorporates an ESP32 board and publish data with MQTT so that we can directly collect data from a car to Home Assistant. Although it’s a bit expensive, I want to give it a try at some point in the future. Eventually, I ended up with a simple and affordable OBD2 adapter to collect data from a car. We can use BLE or Wi-Fi to communicate with the adapter to collect data from a car. Although there is a Home Assistant integration, we basically need an app to communicate with the adapter (like this). When it comes to more DIY approach, there is already complete implementation, but it is for Arduino. After many searches, I decided to use my favorite ESPHome to automatically and directly communicate with an OBD2 adapter and collect data from my car to Home Assistant.

Disclaimer: Before I dive into the details, let’s set some ground rules:

1. Educational Purpose Only: This blog post serves as an educational resource. I’m sharing my experiences, but I’m just a hobbyist, not a professional. Always consult experts for specific advice.
2. Assumption of Risk: By following my instructions, you assume full responsibility for any consequences. DIY projects involve risks, so proceed with caution.
3. Safety First: Handle electronic components carefully. Use appropriate tools and safety gear. Be aware of electrical hazards. Follow manufacturer guidelines.

Objective

An OBD2 adapter can collect lots of detailed data from a car for a diagnostic purpose. It is a tool for a car enthusiast or an expert. It is intended to use only when we need to diagnose a car. It is not intended to keep connecting to a car all the time. Although it is powered up even while the car is OFF (engine is stopped) by using a car battery, it basically collects data while the car is ON (engine is running).

I’m neither a car enthusiast nor an expert. I won’t be able to understand most of data. Collecting some potential alerts or basic data to Home Assistant at my home once a day would be enough for me. And of course, I want to do that automatically. Since I usually start my car in the morning and go somewhere and come back to my home in the evening and stop my car, only chances to collect data from my car to Home Assistant at my home are when my car is ON and an OBD2 adapter in my car is in BLE or Wi-Fi signal range in the morning and the evening. So, I’m going to keep connecting the OBD2 adapter to my car and automatically collect data from my car at home when I start my car in the morning or when I come back to the home in the evening.

Disclaimer: This is not the intended usage. As I describe later, there might be some security and safety concerns.

Architecture

Although I’m not an expert and don’t fully understand OBD2, the communication with OBD2 adapter looks like simple command (AT command) and response on serial terminal. It shows a prompt (“>”) waiting for an input. If we type in a command, it shows a response and a prompt again. So, we could just automate typing in a command on a prompt and parsing an output. That is the basic idea.

There are already some discussions about getting data from a car through an OBD2 adapter over BLE (like this and this). I also found a prior work trying to use ble_client of ESPHome only with YAML configuration. It seems that all I need is sending AT commands and receiving responses over BLE.

Since I’m familiar with ESPHome, I decided to use ESPHome BLE feature as well. I’m going to create a “BLE Node”, which is just an ESP32 board based on ESPHome, place it in the garage in my house, let it connects to the OBD2 adapter on my car with BLE, send some commands to my car through the OBD2 adapter over BLE connection, and get some data from my car. Regarding software to actually communicate with the OBD2 adapter and my car, I’m going to create an external component of ESPHome.

Materials

• OBD2 Adapter: There are many OBD2 adapters with a wide variety of prices. Although there might be a quality issue, I decided to buy an affordable one on Amazon. I picked up the one with BLE4.0.

• ESP32 Board: Because I liked Lolin D1 mini (ESP8266) in my previous projects like DIY Environment Monitor and DIY Smart Button, I decided to use a ESP32 board from the same brand. I picked up a small one — Lolin S3 mini. I bought it from the official store.

• Case: Like my DIY Environment Monitor and DIY Smart Button, I want to put an ESP32 board in a simple and affordable enclosure. I decided to use the small case that I used in my DIY Environment Monitor (like this, this, this and this).

At the time of this writing, the total cost was $33.37 for the OBD2 adapter, Lolin S3 mini and the case without tax and shipping fee.

Hardware Implementation

• OBD2 Adapter: There is nothing to do. I just plug the adapter into the OBD2 port in my car.
• BLE Node: This time, there is nothing difficult to do. I just attached the Lolin S3 mini onto the back panel of the case with hot glue.

Software Implementation and Configuration

If I just receive broadcasted data, esp32_ble_tracker would be enough. But this time, I need to let the BLE Node “connect” to the OBD2 adapter with BLE and send and receive data. So, I use ble_client. Connecting to the OBD2 adapter with ble_client is very easy. I just need MAC address of the OBD2 adapter. Although there is no MAC address information on the adapter, I could find it out by specifying esp32_ble_tracker in the YAML configuration, installing the firmware, and waiting for a while until the adapter showed up on logs.

logger:
  level: VERBOSE

esp32_ble_tracker:

[11:29:57][D][esp32_ble_client:110]: [0] [8C:DE:52:DD:FC:19] Found device
[11:29:57][D][esp32_ble_tracker:671]: Found device XX:XX:XX:XX:XX:XX RSSI=-81
[11:29:57][D][esp32_ble_tracker:692]:   Address Type: PUBLIC
[11:29:57][D][esp32_ble_tracker:694]:   Name: 'VEEPEAK'
[11:29:57][D][esp32_ble_tracker:697]:   TX Power: 3

Once I get MAC address, I can specify it on ble_client configuration.

logger:
  level: VERBOSE

esp32_ble_tracker:

ble_client:
  - mac_address: XX:XX:XX:XX:XX:XX
    id: veepeak

To communicate with the BLE based adapter, we need Service UUID and Characteristic UUID. We can find them out by using some BLE scanner apps or searching on the Internet. It seems that typical OBD2 adapters have just one Service UUID and two Characteristic UUIDs (one for “write” and one for “notify”). We can use “write” characteristic to send AT commands, and “notify” characteristic to get a notification for receiving data. I also found a good insight about UUIDs of almost exact the same OBD2 adapter that I bought. My OBD2 adapter, Veepeak OBDCheck BLE, uses “0xFFF0” for Service UUID, “0xFFF2” for “write” Characteristic UUID, and “0xFFF1” for “notify” Characteristic UUID.

Regarding OBD2, we need to send a series of AT commands for initialization like “ATZ”, “ATE0”, and so forth to get expected behavior. After initialization, we can send an actual command with at least a Mode and a PID (Parameter ID) like “010C”, which means Mode 01 and PID 0C. In either initialization commands or other actual commands, what we do is the same — automating command sending and response parsing like the following example. We can parse a response based on “\r” (0x0D) and a prompt “>” (0x3E). Responses are all hex ASCII characters. For example, a response in the following example like “ELM327 v1.5\r\r>” actually comes in a list of bytes “0x45 0x4C 0x4D 0x33 0x32 0x37 0x20 0x76 0x31 0x2E 0x35 0x0D 0x0D 0x3E”. We can just convert them to ASCII characters to understand them. And we need to be aware that OBD2 responses also include hex ASCII characters like the following example — “410C1CB2” in “7E804410C1CB2”. So, we need to handle a pair of ASCII characters as a hex like “0x41 0x0C 0x1C 0xB2”.

And we might get multiple responses for one command. In the following example, two responses are returned for one command. One of them starts with “7E8” while another one starts with “7E9”. They are CAN (Controller Area Network) IDs of ECUs (Electronic Control Unit). Because there are two ECUs in a car in the following example, which is actually my car by the way, both of them responds to one command. Even one response from one ECU could become multiple lines if the response is long. We need to concatenate them to get a full response.

ATZ                <--- Send a first initialization command with "\r".
ATZ                <--- Echo with "\r"
                   <--- "\r"
ELM327 v1.5        <--- Something like welcome message.
                   <--- "\r"
>ATE0              <--- Prompt and then send another command with "\r".
ATE0               <--- Echo with "\r"
OK                 <--- OK message
                   <--- "\r"
>ATL0              <--- Prompt and then send another command with "\r".
OK                 <--- OK message
                   <--- "\r"
>ATS0              <--- Prompt and then send another command with "\r".
OK                 <--- OK message
                   <--- "\r"
>ATH1              <--- Prompt and then send another command with "\r".
OK                 <--- OK message
                   <--- "\r"
>ATSP0             <--- Prompt and then send another command with "\r".
OK                 <--- OK message
                   <--- "\r"
>ATDPN             <--- Prompt and then send another command with "\r".
A0                 <--- OK message
                   <--- "\r"
>010C              <--- Prompt and then send Mode and PID with "\r".
SEARCHING...       <--- SEARCHING message
7E804410C1CB2      <--- A response from a ECU
7E904410C1CB8      <--- A response from another ECU
                   <--- "\r"
>                  <--- Prompt

There are 10 Modes and lots of PIDs in OBD2 standards, depending on what kind of data we want. Since each car does not fully support them, there are even PIDs to check what PIDs are supported by a car. Based on my objective, I would want to use Mode 03, 07 or 0A for DTC (Diagnostic Trouble Codes) data to see if there is anything wrong in my car. I found a reference of DTC codes for my car manufacturer. Because DTC codes are returned from my car if there are problems, Binary Sensor type will be suitable for DTC codes. I may want to use Mode 01 for some sensor data and Mode 05 and 06 for some test results if necessary. Because they are basically numeric values, Sensor type will be suitable for most of them. I may want to use Mode 09 for vehicle information as well. Text Sensor type will be OK for them. Because there are a wide variety of commands, Modes and PIDs, and I’m still not sure what are supported, I want to make them configurable on YAML like this.

esphome:
  name: s3-mini
  friendly_name: s3-mini
  platformio_options:
    platform: https://github.com/platformio/platform-espressif32.git
    board: lolin_s3_mini
    framework: arduino
    board_build.flash_mode: dio

esp32:
  variant: esp32s3
  board: lolin_s3_mini
  framework: 
    type: arduino
    version: latest

logger:
  level: DEBUG

api:
  encryption:
    key: !secret api_key

ota:
  - platform: esphome
    password: !secret ota_password

wifi:
  ssid: !secret wifi_ssid
  password: !secret wifi_password

  ap:
    ssid: "Esp32-01 Fallback Hotspot"
    password: !secret ap_password

captive_portal:

esp32_ble_tracker:

ble_client:
  - mac_address: XX:XX:XX:XX:XX:XX # MAC address of OBD2 adapter
    id: veepeak

external_components:
  - source: my_components

obd2_ble:
  id: mycar
  service_uuid: "0xFFF0"
  characteristic_uuid:
    write: "0xFFF2"
    notify: "0xFFF1"
  init_commands: "ATZ ATE0 ATL0 ATH0 ATS0 ATSP0" # Optional
  command_delay: 1000 # Optional
  command_wait: 5000 #Optional

sensor:
  - platform: obd2_ble
    obd2_ble_id: mycar
    name: "Engine RPM"
    can_id: "7E8"
    mode: "01"
    pid: "0C"
  - platform: obd2_ble
    obd2_ble_id: mycar
    name: "Vehicle Speed"
    can_id: "7E8"
    mode: "01"
    pid: "0D"

text_sensor:
  - platform: obd2_ble
    obd2_ble_id: mycar
    name: "Diagnostic Trouble Codes"
    mode: "03"
  - platform: obd2_ble
    obd2_ble_id: mycar
    name: "Vehicle VIN"
    can_id: "7E8"
    mode: "09"
    pid: "02"

binary_sensor:
  - platform: obd2_ble
    obd2_ble_id: mycar
    name: "Stored PCM failure"
    mode: "03"
    code: "P0010"
  - platform: obd2_ble
    obd2_ble_id: mycar
    name: "Stored Camshaft variable timing solenoid failure"
    mode: "03"
    code: "P0011"

Now, I’m going to implement my external component called “obd2_ble” to achieve these communications and configurations. There is an official document explaining how to implement an external component. And source codes of existing components, especially which use ble_client, have a lot to learn. I create OBD2BLEClient class that inherits BLEClientNode class and Component class. BLEClientNode class corresponds to ble_client and takes care of BLE communication. By inheriting it, my OBD2BLEClient can do some actions when some BLE related events occur. And I decided to use Component class this time instead of PollingComponent class because BLE Node needs to connect to the OBD2 adapter as soon as it finds the adapter, and don’t have to repeat anything in a fixed interval. After many try and error and learning, I created 5 files as below and placed them in the directory structure like this.

homeassistant
└── esphome
    └── my_components
        └── obd2_ble
            ├── __init__.py
            ├── binary_sensor.py
            ├── obd2_ble.cpp
            ├── obd2_ble.h
            ├── sensor.py
            └── text_sensor.py

My example code is here. I’m not a professional software engineer though. There will be better, more reliable and sophisticated way to implement the same thing. Although I tried to make my implementation generic and configurable, it might not just work in other environments because different OBD2 adapter has different behavior, and different car has different behavior.

• __init__.py: This registers obd2_ble component. It takes UUIDs, a series of initial AT commands and other parameters from YAML and pass them to OBD2BLEClient class.
• sensor.py: This registers sensors of obd2_ble component. It takes a list of sensors specified in YAML and pass them to OBD2BLEClient class.
• text_sensor.py: This registers text sensors of obd2_ble component. It takes a list of text sensors specified in YAML and pass them to OBD2BLEClient class.
• binary_sensor.py: This registers binary sensors of obd2_ble component. It takes a list of binary sensors specified in YAML and pass them to OBD2BLEClient class.
• obd2_ble.h and obd2_ble.cpp: These implements OBD2BLEClient class. It inherits ble_client::BLEClientNode and Component classes. It uses UUIDs, a series of initial AT commands, Modes and PIDs specified in the YAML configuration, uses ble_client capabilities to handle BLE connection, sends commands, receives and parses data, and publish them to Home Assistant. I tried to separate BLE handling and OBD2 handling in my code. BLE handling is event driven, and gattc_event_handler() inherited from BLEClientNode takes care of events. OBD2 handling is a sequential command and response processing, and loop() inherited from Component takes care of those processing.

Installation and Test

I placed the BLE Node in my garage. It is right in front of my car when the car is parked in the garage. And I plugged the OBD2 adapter to the OBD2 port of my car. In my case, the OBD2 port is located under the dashboard, beneath the steering wheel. It was not covered. I could easily plug the OBD2 adapter vertically.

Once I started the engine and the BLE Node found the OBD2 adapter, it connected to the adapter, sent commands, received responses and published data to Home Assistant. So, it was technically successful. It took just less than a minute to complete a series of commands.

I tried several commands to see what kind of data I could get from my car. In my case, my findings were as follows. I tried only standard OBD2 commands. There might be more than that, like proprietary ones. But I don’t have any information about them.

• My car supports 11-bit CAN ID. It might not support 29-bit CAN ID because it returned “NO DATA” for any command after initialization with ATSP7 command.
• My car has two ECUs, one with CAN ID 7E8 (engine) and another one with CAN ID 7E9 (transmission).
• Both ECUs support several PIDs in Mode 01 and 09. The total number of supported PIDs in Mode 01 and 09 was 56.
• Both ECUs might not support any PID for Mode 05 because they returned “NO DATA” for inquiries to check supported PIDs. One of them supports some PIDs in Mode 06.
• Because my car currently has no problem, I got empty responses in Mode 03 (Stored DTC), Mode 07 (Pending DTC) and Mode 0A (Permanent DTC). So, I still have not had any data for testing my obd2_ble component. Based on a reference of DTC codes from the manufacturer of my car, the number of DTC codes is 129. I’m not sure how many of them could be relevant to my car though.
• Because there are many supported PIDs and potentially related DTC codes, the number of sensors, text sensors and binary sensors becomes a lot if I define all of them in YAML. When I actually put all of them in YAML, some problems like memory error or data communication error became more likely to occur (*1). Although it was not a good idea to define all the sensors, text sensors and binary sensors for all the supported PIDs and DTC codes, and I still don’t know how to decode all of them, I got data like below. I was overwhelmed by lots of data. I need to narrow them down to get only necessary data. I might need to avoid defining each binary sensor for each DTC code as well. I also need to improve reliability of my code.

• I have been leaving the OBD2 adapter plugged in all the time. Because I’m using Component class and loop() function, the BLE Node tries to connect to the OBD2 adapter as soon as it finds the adapter, which is expected and good for my use case. But I realized that the BLE Node connects to the OBD2 adapter even when the engine is OFF because the adapter is powered by the car battery. After it connects to the adapter, it can even send a series of initialization commands and receive responses. But when it tries to send an actual command with a Mode and a PID like “010C”, it gets a time out error and cannot receive any response. So, it repeats connecting to the adapter and sending initialization commands while my car is in my garage and the engine is OFF (e.g., during the night), which is just wasting electricity of my home and car battery. So far, I have not experienced a dead battery because of this. But it is not nice. To solve the issue, I initially thought changing from Component/loop() to PollingComponent/update() to reduce frequency of the connection between the BLE Node and the OBD2 adapter. But in my use case, the BLE Node needs to collect data from my car as soon as it finds the adapter in BLE signal range. If it tries to connection in a fixed interval, it might decrease a chance to find the adapter while the engine is ON. I also thought adding some timer capabilities while using Component/loop(). But it would be complicated. I eventually decided to use an affordable and simple smart plug and plug the BLE Node to the smart plug. By using the smart plug, I can just turn on the BLE Node only in a couple of hours in the morning and the evening (*2). I might want to have a better approach like combining presence detection for future improvements.

Results

I’ve been using the BLE Node and the OBD2 adapter for a while. Although the BLE Node is working and it was technically successful, the result has been mixed for me so far.

• In my use case, I wanted to check if there is anything wrong in my car every day. But so far, I have not had any data, fortunately.
• I also wanted to see basic data like tire pressure, mpg and so forth. But it turned out that there is no standard PIDs for them. I should have checked all the PIDs in advance. (*3)
• Although I can see lots of Mode 01 or Mode 09 data, they are just too much for me at this moment. I might need to have knowledge to extract some insights from these data.
• BLE device detection has not been reliable so far. In my use case, I want the BLE Node to find the OBD2 adapter as soon as I start the engine in the morning, or I come back to the garage in the evening. The time the OBD2 adapter is in BLE signal range, and the engine is running, is just a few minutes of time window. But I realized that it would sometimes take longer than I expected for the BLE Node to find the OBD2 adapter. Once it finds and connects to the adapter, sending commands and receiving responses take less than a minute though.

Security and Safety Consideration

As I mentioned above, I’m leaving the OBD2 adapter plugged in all the time. Although the adapter manufacturer says it would be OK from battery consumption perspective, there might be some security and safety concerns. Since anyone within BLE signal range can connect to the adapter anytime, they can send some commands and get data from my car, which is not nice. And there might be some electronic interference as well. This post is just my personal experimental project. I won’t recommend anyone to do the same thing.

If there is any way or any OBD2 adapter that has a capability to restrict connection with configurable password or source address, that would be nice.

Final Thoughts

From a technical perspective, I successfully created my external component for ESPHome to communicate with an affordable BLE OBD2 adapter. It took me many try and error. Since I could test my implementation only in the limited time of the morning and the evening, it took me a longer period than I expected. Although there are still many to improve and consider, and I have not fully completed my use case, I’m satisfied with the result because I finally feel my car has become a part of my smart home.

Update (*1) [11/26/2024]: The problem was solved when I add a delay in publishing sensor states. It resulted in a longer time to complete collecting data though.

Update (*2) [11/26/2024]: I realized that all the sensor values would become “Unavailable” if I just turn off the BLE Node during the night. So, I added a “wait” time before the BLE Node disconnects from the OBD2 adapter when it cannot receive responses because the engine is OFF after all.

Update (*3) [12/02/2024]: I happened to know that there would be additional modes like Mode 22 that contains a manufacturer or a vehicle specific data. Although the fuel type is different, it seems that the manufacturer of my car could support Mode 22. In fact, I got some responses for some PIDs in Mode 22 and added them to my sample code. But I still don’t know how to decode them… I also learned that some responses could be used as a sort of battery voltage and added them to my sample code as well.

Objective

I’m going to create a simple serverless application that retrieve data from database, calculate statistics of yesterday, this week and last week, and this month and last month, and send a report to us by email every morning.

Architecture

This time, the architecture is very simple. As I already explained in the previous post, Home Assistant publishes MQTT messages to AWS IoT Core. When I created my DIY smart button, I initially used Amazon Timestream to record click events. But based on the recent try and error, I decided to switch from Timestream to DynamoDB to record click events. Since I don’t plan to store click events for a long term, DynamoDB would be enough and affordable. I define AWS IoT Core rule to route click event data to DynamoDB. And I create a lambda function to retrieve data from DynamoDB, calculate statistics and send email with Amazon SES. I create an Amazon EventBridge Schedule event (CloudWatch Event) to run the lambda function as a sort of cron job.

Implementation

My example code is here.

ESPHome and Home Assistant: There is no change from my previous post. I defined GPIO Binary Sensor and Template Binary Sensor on ESPHome, and created an automation on Home Assistant to publish click events to a MQTT topic of a Named Shadow on AWS IoT Core.

AWS IoT Core: Because I defined a Named Shadow for my DIY smart button device, button click events are stored in the JSON document as click state like below. Each click state will be “on”, “off” or “unavailable” and they are all varchar data type. Every time one of button is clicked, a correspondent state change (“off” to “on” and “on” to “off”) is sent from Home Assistant to AWS IoT Core by MQTT and JSON document of the Named Shadow is updated.

# JSON document of button click
{
  "state": {
    "reported": {
      "momentary_04_single_short_click": "off",
      "momentary_01_single_short_click": "off",
      "momentary_01_double_click": "off",
      "momentary_02_single_short_click": "off",
      "momentary_03_single_short_click": "off",
      "momentary_05_single_short_click": "off",
      "momentary_04_single_long_click": "off",
      "momentary_01_single_long_click": "off",
      "momentary_03_double_click": "off",
      "momentary_04_double_click": "unavailable",
      "momentary_03_single_long_click": "unavailable",
      "momentary_05_double_click": "unavailable",
      "momentary_05_single_long_click": "unavailable",
      "momentary_02_single_long_click": "unavailable"
    }
  }
}

I defined the following AWS IoT Core Rule to subscribe the MQTT topic and route data to DynamoDB. As I did in another post, I added TTL so that old data will be automatically deleted from DynamoDB. I used SQL version 2016–03–23. I don’t have to worry about data type casting error because all the data are varchar this time.

# AWS IoT Core Rule
SELECT
  CAST(state.reported.momentary_01_single_short_click AS varchar) as momentary_01_single_short_click,
  CAST(state.reported.momentary_01_single_long_click AS varchar) as momentary_01_single_long_click,
  CAST(state.reported.momentary_01_double_click AS varchar) as momentary_01_double_click,
  CAST(state.reported.momentary_02_single_short_click AS varchar) as momentary_02_single_short_click,
  CAST(state.reported.momentary_02_single_long_click AS varchar) as momentary_02_single_long_click,
  CAST(state.reported.momentary_02_double_click AS varchar) as momentary_02_double_click,
  CAST(state.reported.momentary_03_single_short_click AS varchar) as momentary_03_single_short_click,
  CAST(state.reported.momentary_03_single_long_click AS varchar) as momentary_03_single_long_click,
  CAST(state.reported.momentary_03_double_click AS varchar) as momentary_03_double_click,
  CAST(state.reported.momentary_04_single_short_click AS varchar) as momentary_04_single_short_click,
  CAST(state.reported.momentary_04_single_long_click AS varchar) as momentary_04_single_long_click,
  CAST(state.reported.momentary_04_double_click AS varchar) as momentary_04_double_click,
  CAST(state.reported.momentary_05_single_short_click AS varchar) as momentary_05_single_short_click,
  CAST(state.reported.momentary_05_single_long_click AS varchar) as momentary_05_single_long_click,
  CAST(state.reported.momentary_05_double_click AS varchar) as momentary_05_double_click,
  (timestamp() / 1000) as ts,
  ((timestamp() / 1000) + (365 * 24 * 60 * 60)) as ttl
FROM '$aws/things/home-assistant/shadow/name/d1-mini-10/update'

Because I want to retrieve data in a specific period of time, I decided to use the same approach with time series environment data in another post and defined the Partition Key and the Sort Key format like below.

# DynamoDB Partition Ley
${parse_time("yyyy-MM-dd", timestamp())}

# DynamoDB Sort Key
${concat(parse_time("HH:mm:ss.SSS", timestamp()), '#', topic(6))}

Amazon DynamoDB: I created a DynamoDB standard table with On-demand capacity mode. Once I defined the IoT Core Rule above, DynamoDB started to store button click events as you can see in the image below. Since I used date information for the Partition Key and time information (and device name) for the Sort Key, I can easily retrieve data in the specific time period. Here, date and time are UTC.

Each record has the following data format. This example is a record that I single-short-clicked a button 01.

{
  "pk": {
    "S": "2024-10-19"
  },
  "sk": {
    "S": "19:59:04.104#d1-mini-10"
  },
  "payload": {
    "M": {
      "momentary_01_single_short_click": {
        "S": "on"
      },
      "ts": {
        "N": "1729367944"
      },
      "ttl": {
        "N": "1760903944"
      }
    }
  }
}

AWS Lambda: I created a python script to retrieve data from DynamoDB, calculate statistics with Pandas and send a report with email by Amazon SES. It was basically straightforward. Once I convert data from DynamoDB to Pandas datafame, I can calculate statistics just by filtering, grouping and aggregating data. But handling timestamp confused me a lot. Especially because I decided to calculate statistics in local timezone (PST/PDT in my case) and defined a day starting 6am, not 12am. Since we wanted to count events related to housework and childcare, events in the last night (even after 12am and until early in this morning) needed to be counted as yesterday’s events. So, I decided to count events from 6:00am to next day 5:59am as 1-day events. It added a bit of complexity. We also wanted to know average time of each event in local timezone. I needed to extract only time information (without date) in local timezone, convert it to seconds, calculate average with Pandas and convert it back to time.

Amazon SES: I just used gmail. Outlook mail didn’t work on SES for some reason.

Amazon EventBridge: I set a schedule event like a cron job so that the lambda function is called every 6am.

Results

Now, we receive email report every morning like below. The report contains statistics of yesterday, this week (not in Monday report), last week, this month (not in the 1st day of the month), last month. It’s easy for us to glance at how things go recently on email. So far, I don’t find any cost concern.

Final Thoughts

I successfully created statistics reporting feature for my DIY smart button. My wife and I can keep aware of how things go in our housework and childcare. Although creating these features from scratch are likely to be inefficient, I could focus on core function scripting thanks to ESPHome, Home Assistant and AWS.

Objective

The objective of my dashboard has not been changed from my previous project. But this time, I reduce the cost of my dashboard. My dashboard shows daily trends of environment metrics in graphs as its default mode and keeps updating graphs. If I select a specific metrics or a specific room, it shows weekly trend and comparison over different weeks. And if I select a longer time range, it can show long term trends as well. In this use case, to keep updating daily trends cost a lot, and to show weekly trends sometimes costs a little bit. I need to reduce the cost of these short-term and mid-term data retrieval.

Architecture

Before Timestream, there seemed to be many practices of using DynamoDB for storing time series data. There are even best practices published from AWS (like this and this). And now, we have Timestream, and we can find articles or discussion about “DynamoDB vs Timestream”. But this time, I’m going to use both of them — DynamoDB for short-term and mid-term data (a day to a week) and Timestream for long-term data (weeks, months or years).

The overall architecture and flows have not been changed much. I just added DynamoDB in front of Timestream. It’s not just cache. It’s more like a tiered architecture. But I don’t move data from DynamoDB to Timestream because it’s complicated. Although there is data redundancy, it’s acceptable for me because majority of cost comes from queries — reading data in my case. Instead, I store data in both. And I set TTL (Time-To-Live) on data stored in DynamoDB so that it automatically keeps only short-term and mid-term data. Since one of lambda function (Lambda #4 in the picture below) retrieves data from Timestream before, I modified the lambda function to retrieve either DynamoDB or Timestream based on the time period specified as parameters.

Implementation

My example code is here.

DynamoDB: I created a table with a Partition key named “pk” (String type) and a Sort key named “sk” (String type). The table class is DynamoDB Standard and Capacity mode is “On-demand”. When we store time series data on DynamoDB, there have been some best practices and discussions. In the past, separating tables was one of best practices (like this and this), but it does not necessarily seem to be always the case now. Some discussion suggests using location and metrics for a Partition key and timestamp for a Sort key. Because we can use a partial match for a Sort key, using timestamp for a Sort key makes data retrieval easier. In my case, my dashboard mostly needs to retrieve all the metrics of all the location (device) during the specific period of times. And I retrieve old data less often. So, I decided to use date information (yyyy-MM-dd) for a Partition key and a combination of time information (HH:mm:ss) and device information for a Sort key.

IoT Core: I added a IoT Core Rule to route data to DynamoDB. I used a similar SQL with the one I used for Timestream. When I created the rule for Timestream, I needed to use SQL version 2015–10–08 because Timestream sometimes misunderstands data types between Double and Int especially when metrics data becomes like “24.0”. But in this case, I need to use SQL version 2016–03–23 because I want to use “parse_time” function, which is not supported in SQL version 2015–10–08, for DynamoDB Partition key and Sort key as below. But luckily, DynamoDB simply uses just Number type (Decimal) for all the number data. So, I can use SQL version 2016–03–23 and “parse_time” function. Regarding SQL statement, I used similar SQL to the one I used for Timestream. The only difference is, I added a TTL value so that I can use it in DynamoDB.

SELECT
    CAST(state.reported.temperature AS Double) as temperature,
    CAST(state.reported.humidity AS Double) as humidity,
    CAST(state.reported.illuminance AS Double) as illuminance,
    CAST(state.reported.presence AS varchar) as presence,
    CAST(state.reported.co2 AS Int) as co2,
    CAST(state.reported.voc AS Int) as voc,
    CAST(state.reported.nox AS Int) as nox,
    CAST(state.reported.pm1_0 AS Double) as pm1_0,
    CAST(state.reported.pm2_5 AS Double) as pm2_5,
    CAST(state.reported.pm4_0 AS Double) as pm4_0,
    CAST(state.reported.pm10_0 AS Double) as pm10_0,
    CAST(state.reported.pm2_5_avg_24h AS Double) as pm2_5_avg_24h,
    CAST(state.reported.pm10_0_avg_24h AS Double) as pm10_0_avg_24h,
    CAST(state.reported.pressure AS Double) as pressure,
    CAST(state.reported.uv AS Double) as uv,
    (timestamp() / 1000) as ts,
    ((timestamp() / 1000) + (10 * 24 * 60 * 60)) as ttl
FROM '$aws/things/ThingName/shadow/name/+/update'

# DynamoDB Partition Key Value
${parse_time("yyyy-MM-dd", timestamp())}

# DynamoDB Sort Key Value
${concat(parse_time("HH:mm:ss", timestamp()), '#', topic(6))}

Now, data started to be stored in DynamoDB as below. As I described above, the Partition key is a date string, and the Sort key is a combination of a time string and device name (I used “#” as a delimiter).

Lambda: I modified the Lambda #4 function to retrieve data from either DynamoDB or Timestream. When a specified time period starts with a time before DynamoDB TTL, it retrieves data from Timestream. In case Timestream, I’m using “bin” function to aggregate data when a period of time is relatively long. Timestream has a wide variety of aggregation functions, and these are one of the advantages of Timestream. But when I retrieve data from DynamoDB, I need to do similar aggregation by myself. Since I needed only simple time-bin based aggregation in this case, I could add the code to the Lambda #4 function. However, if I need more complex aggregation, I will need a different approach.

Cost

According to DynamoDB On-demand pricing information, the cost becomes something like the following.

Read and write requests: Because I’m not using transactional writes and I’m using eventually consistent reads, DynamoDB charges one WRU (Write Request Unit) for each write (up to 1KB), and 0.5 RRU (Read Request Unit) for each eventually consistent read (up to 4KB). WRU price is $1.25 per million write request units and RRU price is $0.25 per million read request units. There are 9 sensor nodes measuring metrics in my house, and each of them sends data every 5 minutes. Each data has 4 to 15 metrics. The size of each data is under 1KB even it has 15 metrics. So, each write request consumes 1 WRU. And there are 12 x 24 x 9 = 2,592 write requests per day. The cost of writes will be (2,592/1,000,000) x 1.25 = $0.00324/day, $0.0972/month (30 days). Regarding read requests, my dashboard shows the last 24 hours data of all the metrics of all the sensor nodes, and update graphs periodically. Based on the actual size of data read from DynamoDB, the size of the last 24 hours data is about 20 KB (sometimes less than 20 KB and sometimes more than 20 KB). So, every time my dashboard updates graphs, it consumes (20 / 4) x 0.5 = 2.5 RRU. If I let my dashboard update every 15 minutes all day, it consumes 2.5 x 4 x 24 = 240 RRU per day. And if I show the last 7 days data sometimes (let’s assume twice a day), it consumes 2.5 x 7 x 2 = 35 RRU. The total is 275 RRU per day. The cost of reads will be (275/1,000,000) x 0.25 = $0.00006875/day, $0.0020625/month (30 days).

Data storage: Because I’m using DynamoDB Standard table, first 25 GB stored per month is free. As I described above, my data will be within 25 GB limit for 10 days TTL.

Data transfer: Because 100 GB of data transfer out to the internet free each month, my case will fit in this limit.

Compared to the cost of Timestream, the cost of DynamoDB is much cheaper in my use case. Especially, Timestream Queries costs 1.6576$/day, and 49.728$/month (30 days) for 15 minutes update. Now with DynamoDB, it becomes $0.00006875/day, $0.0020625/month (30 days) for the same 15 minutes update. I’m going to think about storing more data on DynamoDB.

Final Thoughts

I successfully reduced the cost of my dashboard by combining DynamoDB with Timestream. Based on this cost reduction, I will be able to hang my dashboard on a wall and let it keep updating graphs all day.

Disclaimer: Before I dive into the details, let’s set some ground rules:

1. Educational Purpose Only: This blog post serves as an educational resource. I’m sharing my experiences, but I’m just a hobbyist, not a professional. Always consult experts for specific advice.
2. Assumption of Risk: By following my instructions, you assume full responsibility for any consequences. DIY projects involve risks, so proceed with caution.
3. Safety First: Handle electronic components carefully. Use appropriate tools and safety gear. Be aware of electrical hazards. Follow manufacturer guidelines for microcontrollers and sensors.

Objective

Like my sensor node project, I’m going to build a smart button device that can be connected to an existing power outlet and Wi-Fi network. Because of my laziness and procrastination habit, changing batteries is challenging for me. And I’m going to let the device has multiple buttons — at least 4 and 5 for just in case.

Materials

• Microcontroller: I have already decided to use Wemos LOLIN D1 mini, which is ESP8266 microcontroller board, because it’s cheap (Official Store on AliExpress) and I’ve used it in my sensor node project. It has many GPIO pins, but it seems that 5 of them are safe to use according to an online article — GPIO4, GPIO5, GPIO12, GPIO13 and GPIO14. I can connect 5 buttons to them.
• Enclosure: Like my sensor node project, I want to use a simple case. It does not have to be heavy duty or weatherproof because we use buttons inside our house. And I realized that I still had a case left over from my sensor node project. It has a flat surface to place some buttons. Depending on the size of buttons, I can place 5 buttons on its surface.
• Button: There are many types of buttons and switches on the market. Considering my objective, I need a simple momentary switch which has an easy-to-push size of button. So, an arcade style button will work for my use case. I found a simple and affordable arcade-like button set which comes with 24 pieces of buttons in an affordable price. I don’t need 24 buttons in this project though. They don’t have fancy LED lights and nice arcade grade microswitches. But each of them has just 2 pins and the height of buttons looks relatively short. The button size is 16mm, which is not so small and not so large. Based on the size of the case, I might be able to place 5 of them.
• Misc: I need to drill a 16mm hole on the cover of the case to place a button. Because I already have a power drill, I need to buy a step drill bit, or a hole saw drill bit. I decided to buy a 16mm hole saw drill bit this time. And like my sensor node project, I’m going to use a 6-pin cable with connectors so that wiring will be easier.

Building Hardware

I thought it would be easy to drill a hole on the plastic cover of the case with the hole saw drill bit. But it turned out, it took a little bit try-and-error for me. Honestly, I ruined one of my cases. I should have practiced more. But eventually I managed to drill 5 holes and successfully place 5 buttons. From appearance perspective, I wanted to choose a white color for all the buttons because it matches the color of the case. But I discussed the role of each button with my wife, and we decided to choose different color for each button.

Because each button has 2 pins, I’m going to connect one of them to one of GPIO pins of D1 mini, and another one to GND pin of D1 mini. To reduce the number of wires to connect D1 mini, I daisy-chained pins for GND among buttons with jumper wires and solder. And I connected a 6-pin cable, which has a connector, to pins for GPIO and one of pins for GND.

I soldered a 6-pin cable, which has a matching connector, to GPIO4/5/12/13/14 and GND pins on D1 mini as well. Comparing my sensor node project, wiring is quite easy this time.

The cost of materials is roughly $9 including D1 mini, case, three buttons, 6-pin cable. It’s affordable. If I count the cost of entire 24 pieces of buttons and a hole saw drill bit, it might be too expensive just for buttons though. Because I bought D1 mini and case from AliExpress, prices change frequently. Sometimes, you might get them much cheaper.

Configuring Software

Thanks to the official document of ESPHome and a great tutorial on the web, configuration was straightforward. I just need GPIO Binary Sensor, “on_multi_click” automation of binary sensor and Template Binary Sensor. Each button requires the following YAML configuration. I defined 5 set of these configuration so that each button has “Single Short Click”, “Single Long Click” and “Double Click” capabilities.

binary_sensor:
  - platform: gpio
    name: Momentary-01
    pin:
      number: GPIO04
      inverted: True
      mode:
        input: true
        pullup: true
    internal: True
    on_multi_click:
    - timing:
      - ON for at most 1s
      - OFF for at most 1s
      - ON for at most 1s
      - OFF for at least 0.2s
      then:
      - logger.log: "Double Clicked"
      - binary_sensor.template.publish:
          id: momentary_01_double_click
          state: ON
      - delay: 0.5s
      - binary_sensor.template.publish:
          id: momentary_01_double_click
          state: OFF
    - timing:
      - ON for 1s to 2s
      - OFF for at least 0.5s
      then:
      - logger.log: "Single Long Clicked"
      - binary_sensor.template.publish:
          id: momentary_01_single_long_click
          state: ON
      - delay: 0.5s
      - binary_sensor.template.publish:
          id: momentary_01_single_long_click
          state: OFF
    - timing:
      - ON for at most 1s
      - OFF for at least 0.5s
      then:
      - logger.log: "Single Short Clicked"
      - binary_sensor.template.publish:
          id: momentary_01_single_short_click
          state: ON
      - delay: 0.5s
      - binary_sensor.template.publish:
          id: momentary_01_single_short_click
          state: OFF
  - platform: template
    id: momentary_01_double_click
    name: Momentary-01 Double Click
    publish_initial_state: True
  - platform: template
    id: momentary_01_single_long_click
    name: Momentary-01 Single Long Click
    publish_initial_state: True
  - platform: template
    id: momentary_01_single_short_click
    name: Momentary-01 Single Short Click
    publish_initial_state: True

Once I configured ESPHome, all the binary sensors showed up on Home Assistant. Because I have 5 buttons and each of them has 3 types of clicks, there are 15 binary sensors. Each of them has “on” or “off” state.

Now I define Automation. Depending on a use case, I can trigger whatever I want by using click events of these buttons. You can do anything to make them actually smart.

In my case, since my objective is just counting some things related to housework and childcare over some period of times and understand statistics, I don’t necessarily need to have automations this time. I can either use related features of Home Assistant (like History Stats, Statistics, etc.) or create my custom application on Cloud.

Because I have been using AWS in my different project, I decided to record click events on Amazon Timestream for now. I defined a Named Shadow for the button device on AWS IoT Core and defined Automation as follows on Home Assistant to send click events data to the Named Shadow’s MQTT topic.

alias: button-device
description: ""
trigger:
  - platform: state
    entity_id:
      - binary_sensor.button_device_momentary_01_single_short_click
      - binary_sensor.button_device_momentary_01_single_long_click
      - binary_sensor.button_device_momentary_01_double_click
      - binary_sensor.button_device_momentary_02_single_short_click
      - binary_sensor.button_device_momentary_02_single_long_click
      - binary_sensor.button_device_momentary_02_double_click
      - binary_sensor.button_device_momentary_03_single_short_click
      - binary_sensor.button_device_momentary_03_single_long_click
      - binary_sensor.button_device_momentary_03_double_click
      - binary_sensor.button_device_momentary_04_single_short_click
      - binary_sensor.button_device_momentary_04_single_long_click
      - binary_sensor.button_device_momentary_04_double_click
      - binary_sensor.button_device_momentary_05_single_short_click
      - binary_sensor.button_device_momentary_05_single_long_click
      - binary_sensor.button_device_momentary_05_double_click
condition: []
action:
  - metadata: {}
    data:
      qos: "0"
      retain: false
      topic: |
        {% if 'button_device' in trigger.to_state.entity_id %}
          $aws/things/home-assistant/shadow/name/button-device/update
        {% endif %}
      payload: |
        {
          "state": {
            "reported": {
              {% if 'momentary_01_single_short_click' in trigger.to_state.entity_id %}
                "momentary_01_single_short_click": {{ trigger.to_state.state | tojson }}
              {% elif 'momentary_01_single_long_click' in trigger.to_state.entity_id %}
                "momentary_01_single_long_click": {{ trigger.to_state.state | tojson }}
              {% elif 'momentary_01_double_click' in trigger.to_state.entity_id %}
                "momentary_01_double_click": {{ trigger.to_state.state | tojson }}
              {% elif 'momentary_02_single_short_click' in trigger.to_state.entity_id %}
                "momentary_02_single_short_click": {{ trigger.to_state.state | tojson }}
              {% elif 'momentary_02_single_long_click' in trigger.to_state.entity_id %}
                "momentary_02_single_long_click": {{ trigger.to_state.state | tojson }}
              {% elif 'momentary_02_double_click' in trigger.to_state.entity_id %}
                "momentary_02_double_click": {{ trigger.to_state.state | tojson }}
              {% elif 'momentary_03_single_short_click' in trigger.to_state.entity_id %}
                "momentary_03_single_short_click": {{ trigger.to_state.state | tojson }}
              {% elif 'momentary_03_single_long_click' in trigger.to_state.entity_id %}
                "momentary_03_single_long_click": {{ trigger.to_state.state | tojson }}
              {% elif 'momentary_03_double_click' in trigger.to_state.entity_id %}
                "momentary_03_double_click": {{ trigger.to_state.state | tojson }}
              {% elif 'momentary_04_single_short_click' in trigger.to_state.entity_id %}
                "momentary_04_single_short_click": {{ trigger.to_state.state | tojson }}
              {% elif 'momentary_04_single_long_click' in trigger.to_state.entity_id %}
                "momentary_04_single_long_click": {{ trigger.to_state.state | tojson }}
              {% elif 'momentary_04_double_click' in trigger.to_state.entity_id %}
                "momentary_04_double_click": {{ trigger.to_state.state | tojson }}
              {% elif 'momentary_05_single_short_click' in trigger.to_state.entity_id %}
                "momentary_05_single_short_click": {{ trigger.to_state.state | tojson }}
              {% elif 'momentary_05_single_long_click' in trigger.to_state.entity_id %}
                "momentary_05_single_long_click": {{ trigger.to_state.state | tojson }}
              {% elif 'momentary_05_double_click' in trigger.to_state.entity_id %}
                "momentary_05_double_click": {{ trigger.to_state.state | tojson }}
              {% endif %}
            }
          }
        }
    action: mqtt.publish
mode: single

After I define AWS IoT Core Rule, click events began to be routed to Timestream. Now, I can retrieve “on” state of all the buttons and count them to calculate statistics.

Final Thoughts

I successfully created my DIY smart buttons. Because my objective was just counting some things, it was more like just Wi-Fi connected buttons rather than smart buttons though. I placed my button device on a wall in my house so that my wife and I can push buttons anytime we need. Because it is not battery powered and not mobile, we cannot bring them anywhere. But we don’t have to find it instead. We both know that where it is. And we don’t have to worry about changing battery. Although it does not have sophisticated appearance like commercial products, I’m very satisfied with my affordable DIY smart buttons.

Objective

I’m going to create a simple dashboard to visualize data. Since I already have Home Assistant dashboard and History integration feature to see the latest environment data, I want to use this additional dashboard to see trends of metrics over a relatively longer period of time in graphs, and even compare them over multiple periods of time.

Architecture

Because I’m not a professional software engineer, I consulted Generative AI on how to achieve the goal. As a result, I decided the architecture and the flow of my dashboard as a picture below. In the picture, I already covered “Data Ingestion” part in another posts. I have 9 of ESPHome based sensor nodes to measure metrics data, and those data are once collected to Home Assistant, and Home Assistant publishes those data to AWS IoT Core, and they are routed to Timestream. This time, I create “Login” and “Data Retrieval” parts. It is a typical serverless architecture that requires several frontend HTML and JavaScript, backend Lambda functions and some additional AWS services. I store frontend HTML and JavaScript on S3 and access them from a browser through CloudFront. When it comes to visualizing data in graphs, I use chart.js to have nice graphs which are similar to Home Assistant History integration. And those JavaScript access backend Lambda functions through API Gateway. And then, backend Lambda functions access data. I protect my dashboard with a very simple email authentication to have a minimum level of security. I register users’ email addresses in DynamoDB in advance, a user inputs their email address and requests a one-time code, the dashboard sends the code with Amazon SES, and the user inputs the code to get authorized. I also protect data retrieval through API Gateway with JWT token. Once a user gets authorized, the dashboard returns a token, frontend JavaScript sends data retrieval requests with the token, and API Gateway verifies the token before transferring those requests.

Implementation

All of my example codes are here.

IoT Core and Timestream: As I explained in another post, I have been using IoT Core Rule to route data, which arrive at Named Shadows’ MQTT topics, to Timestream for LiveAnalytics. My SQL statement to route data is as follows. And I set “1 day” for Memory store retention and “10 years” for Magnetic store retention for now.

SELECT 
  CAST(state.reported.temperature AS Double) as temperature,
  CAST(state.reported.humidity AS Double) as humidity,
  CAST(state.reported.illuminance AS Double) as illuminance,
  CAST(state.reported.presence AS varchar) as presence,
  CAST(state.reported.co2 AS Int) as co2,
  CAST(state.reported.voc AS Int) as voc,
  CAST(state.reported.nox AS Int) as nox,
  CAST(state.reported.pm1_0 AS Double) as pm1_0,
  CAST(state.reported.pm2_5 AS Double) as pm2_5,
  CAST(state.reported.pm4_0 AS Double) as pm4_0,
  CAST(state.reported.pm10_0 AS Double) as pm10_0,
  CAST(state.reported.pm2_5_avg_24h AS Double) as pm2_5_avg_24h,
  CAST(state.reported.pm10_0_avg_24h AS Double) as pm10_0_avg_24h,
  CAST(state.reported.pressure AS Double) as pressure,
  CAST(state.reported.uv AS Double) as uv 
FROM '$aws/things/MyThingName/shadow/name/+/update'

CloudFront and S3: I stored following HTML and JavaScript on my S3 bucket. And I didn’t allow public access to the bucket. I allowed only my CloudFront distribution to access the bucket. I also set my CloudFront distribution URL in “AllowedOrigins” in CORS setting.

• login.html and login.js: These show the login page where a user submits their email address to receive one-time code and redirect a browser to the verify page.
• verify.html and verify.js: These show the verify page where a user submits their email address and received one-time code, get a JWT token, store the token in a local storage and redirect a browser to the dashboard page.
• dashboard.html and dashboard.js: These show the dashboard page where a user can see graphs of environment metrics. For now, I simply implemented three modes — the glance mode (default), the metrics mode and the device mode. In the default glance mode, the dashboard shows all the metrics data of all the devices during the last 24 hours at a glance and keeps updating graphs every 15 minutes. In the metrics mode, the dashboard shows only one metrics data of all the devices during two different time periods (the last 7 days and the previous week for now) in side-by-side. In the device mode, the dashboard shows all the metrics data of one device, which means one location, during two different time periods for each metrics. Regarding graphs, I used chart.js and tried to make them look like History integration feature of Home Assistant. I used a line chart for almost all the metrics and used a horizontal floating bar chart (like discussed here) only for human presence.
• styles.css and config.json: These are just CSS and the configuration file for the dashboard.

API Gateway: I created the following three resources on my API Gateway.

• “/send-code” (POST and OPTIONS): This has Lambda proxy integration with Lambda function #1 to send one-time code based on a request from login.js above.
• “/authorize” (POST and OPTIONS): This has Lambda proxy integration with Lambda function #2 to verify one-time code based on a request from verify.js above and return a JWT token back to verify.js.
• “/data” (GET and OPTIONS): This has Lambda proxy integration with Lambda function #4 to fetch data from Timestream based on a request from dashboard.js above. It is also associated with Lambda Authorizer based on Lambda function #3 in its Method request setting to authorize the JWT token attached in the Authorization header of a request from dashboard.js.

Lambda: I created the following four Lambda python functions. Because I used Lambda proxy integration as described above, lambda functions basically need to return CORS headers.

• #1 “SendCode”: This is a simple python script to check if a user’s email address exists in DynamoDB, generate a random one-time code, store the code in DynamoDB and send the code to the email address by using Amazon SES.
• #2 “VerifyCode”: This is also a simple python script to verify a one-time code by comparing it with a code stored in DynamoDB, generate a JWT token and return the token back to a browser. I needed to add a layer to the lambda function to use the PyJWT module.
• #3 “AuthorizeDataAccess”: This is another simple python script to verify a JWT token. I added a layer as well.
• #4 “RetrieveData”: This is a python script to fetch data from Timestream based on request parameters like metrics, devices and a time period which are selected on the dashboard. I decided to use a Timestream query like the following example so that I can retrieve data in a table like format where each row contains all the metrics columns. If a requested time period becomes long, the amount of data becomes large. Since my sensor nodes send data every 5 minutes and every time human presence state changes, the amount of all the metrics data from all the devices in 24 hours becomes roughly 3,000 rows. I used aggregation in Timestream query like the following example to keep the amount of data around 3,000 even in case a requested time period becomes longer than 24 hours.

-- Timestream query without aggregation
SELECT
  time,
  DEVICE_NAME,
  MAX(CASE WHEN measure_name = 'temperature' THEN measure_value::double END) AS temperature,
  MAX(CASE WHEN measure_name = 'humidity' THEN measure_value::double END) AS humidity,
  MAX(CASE WHEN measure_name = 'illuminance' THEN measure_value::double END) AS illuminance,
  MAX(CASE WHEN measure_name = 'presence' THEN measure_value::varchar END) AS presence,
  MAX(CASE WHEN measure_name = 'co2' THEN measure_value::bigint END) AS co2,
  MAX(CASE WHEN measure_name = 'voc' THEN measure_value::bigint END) AS voc,
  MAX(CASE WHEN measure_name = 'nox' THEN measure_value::bigint END) AS nox,
  MAX(CASE WHEN measure_name = 'pm1_0' THEN measure_value::double END) AS pm1_0,
  MAX(CASE WHEN measure_name = 'pm2_5' THEN measure_value::double END) AS pm2_5,
  MAX(CASE WHEN measure_name = 'pm4_0' THEN measure_value::double END) AS pm4_0,
  MAX(CASE WHEN measure_name = 'pm10_0' THEN measure_value::double END) AS pm10_0,
  MAX(CASE WHEN measure_name = 'pm2_5_avg_24h' THEN measure_value::double END) AS pm2_5_avg_24h,
  MAX(CASE WHEN measure_name = 'pm10_0_avg_24h' THEN measure_value::double END) AS pm10_0_avg_24h,
  MAX(CASE WHEN measure_name = 'pressure' THEN measure_value::double END) AS pressure,
  MAX(CASE WHEN measure_name = 'uv' THEN measure_value::double END) AS uv
FROM "MyTimestreamDbName"."MyTimestreamTableName"
WHERE
  DEVICE_NAME IN ('d1-mini-01', 'd1-mini-02', 'd1-mini-03', 'd1-mini-04', 'd1-mini-05', 'd1-mini-06', 'd1-mini-07', 'd1-mini-08', 'd1-mini-09')
  AND measure_name IN ('temperature', 'humidity', 'illuminance', 'presence', 'co2', 'voc', 'nox', 'pm1_0', 'pm2_5', 'pm4_0', 'pm10_0', 'pm2_5_avg_24h', 'pm10_0_avg_24h', 'pressure', 'uv')
  AND time BETWEEN '2024-MM-DD HH:mm:ss' AND '2024-MM-DD HH:mm:ss'
GROUP BY time, DEVICE_NAME
ORDER BY time

-- Timestream query with aggregation
SELECT
  BIN(time, XXXXs) AS binned_time,
  DEVICE_NAME,
  MAX(CASE WHEN measure_name = 'temperature' THEN measure_value::double END) AS temperature,
  MAX(CASE WHEN measure_name = 'humidity' THEN measure_value::double END) AS humidity,
  MAX(CASE WHEN measure_name = 'illuminance' THEN measure_value::double END) AS illuminance,
  MAX(CASE WHEN measure_name = 'presence' THEN measure_value::varchar END) AS presence,
  MAX(CASE WHEN measure_name = 'co2' THEN measure_value::bigint END) AS co2,
  MAX(CASE WHEN measure_name = 'voc' THEN measure_value::bigint END) AS voc,
  MAX(CASE WHEN measure_name = 'nox' THEN measure_value::bigint END) AS nox,
  MAX(CASE WHEN measure_name = 'pm1_0' THEN measure_value::double END) AS pm1_0,
  MAX(CASE WHEN measure_name = 'pm2_5' THEN measure_value::double END) AS pm2_5,
  MAX(CASE WHEN measure_name = 'pm4_0' THEN measure_value::double END) AS pm4_0,
  MAX(CASE WHEN measure_name = 'pm10_0' THEN measure_value::double END) AS pm10_0,
  MAX(CASE WHEN measure_name = 'pm2_5_avg_24h' THEN measure_value::double END) AS pm2_5_avg_24h,
  MAX(CASE WHEN measure_name = 'pm10_0_avg_24h' THEN measure_value::double END) AS pm10_0_avg_24h,
  MAX(CASE WHEN measure_name = 'pressure' THEN measure_value::double END) AS pressure,
  MAX(CASE WHEN measure_name = 'uv' THEN measure_value::double END) AS uv
FROM "MyTimestreamDbName"."MyTimestreamTableName"
WHERE
  DEVICE_NAME IN ('d1-mini-01', 'd1-mini-02', 'd1-mini-03', 'd1-mini-04', 'd1-mini-05', 'd1-mini-06', 'd1-mini-07', 'd1-mini-08', 'd1-mini-09')
  AND measure_name IN ('temperature', 'humidity', 'illuminance', 'presence', 'co2', 'voc', 'nox', 'pm1_0', 'pm2_5', 'pm4_0', 'pm10_0', 'pm2_5_avg_24h', 'pm10_0_avg_24h', 'pressure', 'uv')
  AND time BETWEEN '2024-MM-DD HH:mm:ss' AND '2024-MM-DD HH:mm:ss'
GROUP BY BIN(time, XXXXs), DEVICE_NAME
ORDER BY binned_time

DynamoDB and SES: There is nothing special here. I just used DynamoDB to store user data and used SES to send email.

Cost

After I started storing data in Timestream and developing my dashboard, I realized that majority of the cost comes from Timestream usage. According to the official pricing document, the cost of Timestream depends on “Writes”, “Queries”, “Memory store” and “Magnetic store”. In my case, I can estimate the cost as follows (correct me if I’m wrong).

• Writes: Home Assistant publishes metrics data of each sensor node to the correspondent Named Shadow’s MQTT topic in every 5 minutes. The number of metrics is 4 at minimum (Temperature, Humidity, Illuminance, Presence) and 15 at maximum (Temperature, Humidity, Illuminance, Presence, CO2, VOC, NOx, PM1.0, PM2.5, PM4.0, PM10.0, PM2.5 in 24h average, PM10.0 in 24h average, Pressure, UV). The write data size will fit in 1 KB even at maximum (timestamp, dimension, 15 set of measure names and measure values). The number of write requests for each sensor node is 12 writes/hour, 288 writes/day, 8,640 writes/month (30 days). And Home Assistant publishes presence metrics data of each sensor node every time the presence state changes (from “on” to “off” or from “off” to “on”). Based on actual measurements in my house, the average number of write requests triggered by the presence state change of each sensor node is about 90 writes/day, so 2,700 writes/month (30 days). Total write requests for 9 sensor nodes will be 9 x (8,640 + 2,700) = 102,060 writes/month. The cost will be 102,060 x 0.50$/1,000,000 = 0.05$/month.
• Queries: The cost is calculated in TCU-hours. The official document says, “Each TCU is billed on an hourly basis with per-second granularity and for a minimum of 30 seconds”. In my case, my Timestream query does not exceed 30 seconds. It always ends within 10 seconds. And my Max Query TCU is set to 4 (minimum value). If my dashboard in the glance mode updates graphs of all the metrics of all the devices every 15 minutes, query time will be 4 x 30 = 120 seconds/hour, 120 x 24 = 2,880 seconds/day, which is 0.8 hour/day. The cost will be 4 TCU x 0.8 hour/day x 0.518$/TCU-hour = 1.6576$/day, and 49.728$/month (30 days).
• Memory store: My memory store retention is set to 24 hours. Although the write data size in every 5 minutes is within 1 KB, each metrics is handled as 1 record in Timestream. So, 15 metrics data are considered to be 15 KB data. In my case, the total size becomes 65 KB for 9 sensor nodes. So, 65 KB x 12 x 24 = 18.72 MB/day. Regarding the presence state change for 9 sensor nodes, 1 KB x 90 x 9 = 0.81 MB/day. Total data size is 19.53 MB/day = 0.01953 GB/day. The cost will be 0.01953 GB x 24 hours x 0.036$/GB-hour = 0.0168739$/day, 0.5062176$/month (30 days).
• Magnetic store: My magnetic store retention is set to 10 years. Because the total data size is 0.01953 GB/day, the total data size will be 0.01953 x 365 = 7.12845 GB in 1 year and 7.12845 x 10 = 71.2845 GB in 10 years. As the total data size increases, the cost of magnetic store increases month by month. The cost will reach to 7.12845 GB x 0.03$ GB/month = 0.21$/month after 1 year and 2.1$/month after 10 years.

As estimated above, the cost of queries takes a lot. I definitely don’t want to update graphs quite often. Even if I reduce the update cycle to every 30 minutes or even every 1 hour, the dashboard still costs 24.864$/month or 12.432$/month. It’s a lot for me just for showing graphs. It’s definitely not something that I can put on a wall to show graphs all the time.

Final Thoughts

I successfully created my dashboard on AWS to see trends of environment metrics measured by ESPHome and Home Assistant. Because I’m not good at maintaining my local IT infrastructure, I tried to store long term data on Timestream and visualize them on Cloud. But now, I realized that it would be costly for me. Although I’m still not sure if it would work for me, it was a good opportunity for me to learn dashboard implementation from Generative AI and revisit the balance between the cost of my local IT and Cloud.

Objective

Currently, I’m measuring environment metrics like temperature, relative humidity, illuminance, human presence, CO2, VOC index, NOx index, PM, pressure and UV index with my sensor nodes placed in several areas in my house. Those data are collected to Home Assistant with Native API. I can see environment metrics on Home Assistant dashboard, which is great. It is reasonable to see and use the latest data within local IT infrastructure in my house without relying on the Cloud infrastructure. But sometimes, I want to see long-term data. Storing data in the long term is challenging for me because of my laziness about reliability, management and maintenance of local IT infrastructure. So, I decided to use environment data sent to AWS IoT Core for long-term data visualization while continuing to use the latest and short-term data on Home Assistant at home.

Architecture

I already shared how to send data from Home Assistant to AWS IoT Core. I’m using Named Shadow to manage data on AWS IoT Core. Each named shadow corresponds to each sensor node and each named shadow has each MQTT topic. Now, measured data from each sensor node are sent to each MQTT topic as follows.

Because those environment metrics data are long-term time series data, I decided to store them in Amazon Timestream database. Thankfully, AWS IoT rules can route data arriving in MQTT topics to Amazon Timestream database.

Once I start storing data in Amazon Timestream, I can retrieve those data and use them on some applications, a dashboard in my case, which I’ll talk about later.

Configuration of Amazon Timestream database

It was very straightforward. I decided to use “Amazon Timestream for LiveAnalytics”. I created just “Standard database”, and a table with “Default partitioning”, 24 hours “Memory store retention” and 10 years “Magnetic store retention”.

Configuration of AWS IoT rules

This was a little bit tricky part. Routing data itself is very easy — opening up “Message routing” on AWS IoT, defining SQL statement to select data from MQTT topics, choosing “Timestream table” for “Rule actions”. Since each MQTT topic name includes correspondent sensor node name, I can extract sensor node name from MQTT topic name and use it as dimension for Timestream. Which means that I don’t have to send sensor node name in MQTT payload. And I don’t have to send timestamp as well since Timestream database adds timestamp. But when it comes to SQL statement, I initially used default SQL V2 (2016–03–23) and defined a simple SQL like this.

SELECT state.reported.* FROM '$aws/things/ha-20240731/shadow/name/+/update'

Because data are sent to named shadows, each data has the following structure. I can extract data by selecting “state.reported.*” like above SQL. And I can use a wild card “+” in FROM part to extract data from all the MQTT topics in one SQL statement. In my case “+” contains sensor node name as you can see in the above list of named shadows. I can extract an actual string in “+” as dimension by referring it with substitution templates. I defined “DEVICE_NAME” for dimension name and “${topic(6)}” for dimension value in my rule.

{
  "state": {
    "reported": {
      "temperature": 26.48,
      "humidity": 53.3,
      "pressure": 1012.4,
      "co2": 609,
      "voc": 405,
      "nox": 100,
      "pm1_0": 7.8,
      "pm2_5": 8.1,
      "pm4_0": 8.1,
      "pm10_0": 8.1,
      "illuminance": 140.4,
      "uv": 0.0,
      "pm2_5_avg_24h": 5.75,
      "pm10_0_avg_24h": 5.78,
      "presence": "off"
    }
  }
}

At first, everything seemed to be good. But when I checked stored values on Timestream, some metrics were sometimes missing. And I realized that there were errors caused by data type. It seems that Amazon Timestream automatically choose data type best suited for data. In my case, temperature, relative humidity, illuminance, PM, UV index have accuracy decimals like “26.48” in case of temperature. They are likely to be handled as “measure_value::double” in Timestream. And CO2, VOC index, NOx index does not have decimals. They are likely to become “measure_value::bigint” in Timestream. And human presence is either “on”, “off” or “unknown”. It becomes “measure_value::varchar” in Timestream. And problem happened in measurement data which have decimals. When measurement data becomes like “26.0” or “26.00”, Timestream causes data type error because it converts those to integer like “26” and it is different from the data type of existing data. I need to explicitly cast data in SQL.

Before I change SQL statement, I need to create a new table if the existing table already has wrong data types. In my case, UV index had wrong data type (bigint) from the beginning because its measurement inside home was almost always “0.00000”. According to AWS official document, I need to use SQL V1 (2015–10–08) to correctly cast data. I actually did several try-and-error until I found out this document. Finally, I used SQL V1, and defined SQL statement as follows. There might be more efficient way though.

SELECT
    CAST(state.reported.temperature AS Double) as temperature,
    CAST(state.reported.humidity AS Double) as humidity,
    CAST(state.reported.illuminance AS Double) as illuminance,
    CAST(state.reported.presence AS varchar) as presence,
    CAST(state.reported.co2 AS Int) as co2,
    CAST(state.reported.voc AS Int) as voc,
    CAST(state.reported.nox AS Int) as nox,
    CAST(state.reported.pm1_0 AS Double) as pm1_0,
    CAST(state.reported.pm2_5 AS Double) as pm2_5,
    CAST(state.reported.pm4_0 AS Double) as pm4_0,
    CAST(state.reported.pm10_0 AS Double) as pm10_0,
    CAST(state.reported.pm2_5_avg_24h AS Double) as pm2_5_avg_24h,
    CAST(state.reported.pm10_0_avg_24h AS Double) as pm10_0_avg_24h,
    CAST(state.reported.pressure AS Double) as pressure,
    CAST(state.reported.uv AS Double) as uv
FROM '$aws/things/ha-20240731/shadow/name/+/update'

Query for Timestream

Now, I can retrieve data from Timestream. I can use Query editor on Amazon Timestream to see how I can get data from Timestream. If I use a simple query as follows,

SELECT
 *
FROM "timestream-20240819"."environment-20240826"
WHERE
 time BETWEEN ago(5m) AND now()
 AND DEVICE_NAME IN ('d1-mini-01', 'd1-mini-02', 'd1-mini-03', 'd1-mini-04', 'd1-mini-05', 'd1-mini-06', 'd1-mini-07', 'd1-mini-08', 'd1-mini-09')
ORDER BY
 time ASC

I can get data as follows.

But considering my objective — data visualization on dashboard, I might want to use something like the following query so that I can get data in a sort of table structure.

SELECT
 time,
    DEVICE_NAME,
    MAX(CASE WHEN measure_name = 'temperature' THEN measure_value::double END) AS temperature,
    MAX(CASE WHEN measure_name = 'humidity' THEN measure_value::double END) AS humidity,
    MAX(CASE WHEN measure_name = 'illuminance' THEN measure_value::double END) AS illuminance,
    MAX(CASE WHEN measure_name = 'presence' THEN measure_value::varchar END) AS presence,
    MAX(CASE WHEN measure_name = 'co2' THEN measure_value::bigint END) AS co2,
    MAX(CASE WHEN measure_name = 'voc' THEN measure_value::bigint END) AS voc,
    MAX(CASE WHEN measure_name = 'nox' THEN measure_value::bigint END) AS nox,
    MAX(CASE WHEN measure_name = 'pm1_0' THEN measure_value::double END) AS pm1_0,
    MAX(CASE WHEN measure_name = 'pm2_5' THEN measure_value::double END) AS pm2_5,
    MAX(CASE WHEN measure_name = 'pm4_0' THEN measure_value::double END) AS pm4_0,
    MAX(CASE WHEN measure_name = 'pm10_0' THEN measure_value::double END) AS pm10_0,
    MAX(CASE WHEN measure_name = 'pm2_5_avg_24h' THEN measure_value::double END) AS pm2_5_avg_24h,
    MAX(CASE WHEN measure_name = 'pm10_0_avg_24h' THEN measure_value::double END) AS pm10_0_avg_24h,
    MAX(CASE WHEN measure_name = 'pressure' THEN measure_value::double END) AS pressure,
    MAX(CASE WHEN measure_name = 'uv' THEN measure_value::double END) AS uv
FROM "timestream-20240819"."environment-20240826"
WHERE
 DEVICE_NAME IN ('d1-mini-01', 'd1-mini-02', 'd1-mini-03', 'd1-mini-04', 'd1-mini-05', 'd1-mini-06', 'd1-mini-07', 'd1-mini-08', 'd1-mini-09')
    AND measure_name IN ('temperature', 'humidity', 'illuminance', 'presence', 'co2', 'voc', 'nox', 'pm1_0', 'pm2_5', 'pm4_0', 'pm10_0', 'pm2_5_avg_24h', 'pm10_0_avg_24h', 'pressure', 'uv')
    AND time BETWEEN ago(5m) AND now()
GROUP BY time, DEVICE_NAME
ORDER BY time

The result is as follows. Because data are grouped by time and DEVICE_NAME, one row contains all the metrics from one sensor node at one timestamp. Since all the sensor nodes do not measure all the metrics, unavailable metrics become NULL.

Final Thoughts

After several try-and-error, I successfully start storing data on Timestream. Although it still relies on my local IT for sending data, I don’t have to care about management and maintenance of infrastructure for long-term data, which is great. I still need to observe cost for in my use case to decide if it’s really a good idea or not. Now, I’m trying to create my own dashboard to visualize long-term data. I’ll share details later.