The “software bill of materials” (SBOM) has emerged as a key building block in software security and software supply chain risk management. An SBOM is a comprehensive list of all the software components, dependencies, and metadata associated with an application.

Espressif believes that this information is a key step towards ensuring the security of the connected devices. And as such, we have now enabled easy to use tools and solutions to track and analyze this information.

In this blog post, we shall look through the Espressif’s SBOM generator and analysis tool, ESP-IDF-SBOM.

SBOM

According to National Telecommunications and Information Administration’s (NTIA) SBOM FAQ, the official definition of SBOM goes as follows:

A Software Bill of Materials (SBOM) is a complete, formally structured list of components, libraries, and modules that are required to build (i.e., compile and link) a given piece of software and the supply chain relationships between them. These components can be open source or proprietary, free or paid, and widely available or restricted access.

Key data fields for each component includes:

• Supplier Name
• Component Name
• Version of the Component
• Other Unique Identifiers
• Dependency Relationship
• Author of SBOM Data
• Timestamp

SBOM data needs to be presented in consistent and digestible formats. Some formats to handle the data representation are SPDX, CycloneDX etc.

Simple conceptual SBOM tree for wifi_provisioning component from ESP-IDF looks like below:

SPDX

Software Package Data Exchange (SPDX) is an open standard for communicating software bill of materials (SBOM) information that supports accurate identification of software components, explicit mapping of relationships between components, and the association of security and licensing information with each component.

The image below outlines what data can be found in an SPDX document.

ESP-IDF-SBOM

ESP-IDF-SBOM is a SBOM generator tool. It generates SPDX format SBOM for ESP-IDF based applications. This tool also allows to check the generated SBOM for known security vulnerabilities against National Vulnerabilitiy Database (NVD).

SBOM Generator Workflow

Brief tool workflow goes like:

• Collect sbom.yml (a manifest file) files from dependant components
• Parse project description file generated by the ESP-IDF build system
• Generate SPDX report
• Scan generated SPDX against the CVE databse to generate final report

Getting Started

Install the tool and checkout its help:

$ pip install esp-idf-sbom
$ esp-idf-sbom --help

usage: esp-idf-sbom [-h] [-q] [-n] [-f] [-v] [-d] [--no-progress] {create,check,manifest} ...

ESP-IDF SBOM tool

positional arguments:
  {create,check,manifest}
                        sub-command help
    create              Create SBOM file based on the ESP-IDF project_description.json file.
    check               Check components/submodules in the ESP-IDF SBOM file for possible
                        vulnerabilities reported in the National Vulnerability Database.
    manifest            Commands operating atop of manifest files.

options:
  -h, --help            show this help message and exit
  -q, --quiet           By default auxiliary messages like errors, warnings, debug messages
                        or progress are reported to the standard error stream. With this
                        option set, all such messages are suppressed.
  -n, --no-colors       Do not emit color codes. By default color codes are used when stdout
                        or stderr is connected to a terminal.
  -f, --force-colors    Emit color codes even when stdout or stderr is not connected to a
                        terminal.
  -v, --verbose         Be verbose. Messages are printed to standard error output.
  -d, --debug           Print debug information. Messages are printed to standard error
                        output.
  --no-progress         Disable progress bar.

Generate the SBOM file:

$ cd esp-idf/examples/provisioning/wifi_prov_mgr
$ idf.py build
$ esp-idf-sbom create <project description file> --output-file prj.spdx

The project description file is a path to the project_description.json file, which is by default created by the ESP-IDF build system in the project's build directory.

Check SBOM against the security vulnerabilities:

$ esp-idf-sbom check prj.spdx

                                        Report summary                                         
┌───────────────────────────────────┬─────────────────────────────────────────────────────────┐
│ Date:                             │ 2023-10-10T08:21:39Z                                    │
│ Project name:                     │ project-wifi_prov_mgr                                   │
│ Project version:                  │ v5.2-dev-3250-g7d8f015a4c                               │
│ Vulnerability database:           │ NATIONAL VULNERABILITY DATABASE (https://nvd.nist.gov)  │
│ Generated by tool:                │ esp-idf-sbom (0.8.0)                                    │
│ Generated with command:           │ esp-idf-sbom check prj.spdx                             │
│ Number of scanned packages:       │ 57                                                      │
├───────────────────────────────────┼─────────────────────────────────────────────────────────┤
│ CRITICAL CVEs found:              │                                                         │
│ Packages affect by CRITICAL CVEs: │                                                         │
│ Number of CRITICAL CVEs:          │ 0                                                       │
├───────────────────────────────────┼─────────────────────────────────────────────────────────┤
│ HIGH CVEs found:                  │                                                         │
│ Packages affect by HIGH CVEs:     │                                                         │
│ Number of HIGH CVEs:              │ 0                                                       │
├───────────────────────────────────┼─────────────────────────────────────────────────────────┤
│ MEDIUM CVEs found:                │                                                         │
│ Packages affect by MEDIUM CVEs:   │                                                         │
│ Number of MEDIUM CVEs:            │ 0                                                       │
├───────────────────────────────────┼─────────────────────────────────────────────────────────┤
│ LOW CVEs found:                   │                                                         │
│ Packages affect by LOW CVEs:      │                                                         │
│ Number of LOW CVEs:               │ 0                                                       │
├───────────────────────────────────┼─────────────────────────────────────────────────────────┤
│ UNKNOWN CVEs found:               │                                                         │
│ Packages affect by UNKNOWN CVEs:  │                                                         │
│ Number of UNKNOWN CVEs:           │ 0                                                       │
├───────────────────────────────────┼─────────────────────────────────────────────────────────┤
│ All CVEs found:                   │                                                         │
│ All packages affect by CVEs:      │                                                         │
│ Total number of CVEs:             │ 0                                                       │
└───────────────────────────────────┴─────────────────────────────────────────────────────────┘


Packages with No Identified Vulnerabilities
                        
┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃        Package        ┃          Version          ┃                   CPE                   ┃
┡━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ project-wifi_prov_mgr │ v5.2-dev-3250-g7d8f015a4c │ cpe:2.3:a:espressif:esp-idf:5.2-dev-325 │
│                       │                           │       0-g7d8f015a4c:*:*:*:*:*:*:*       │
├───────────────────────┼───────────────────────────┼─────────────────────────────────────────┤
│         lwip          │           2.1.3           │ cpe:2.3:a:lwip_project:lwip:2.1.3:*:*:* │
│                       │                           │                :*:*:*:*                 │
├───────────────────────┼───────────────────────────┼─────────────────────────────────────────┤
│         cjson         │          1.7.16           │ cpe:2.3:a:cjson_project:cjson:1.7.16:*: │
│                       │                           │               *:*:*:*:*:*               │
├───────────────────────┼───────────────────────────┼─────────────────────────────────────────┤
│      protobuf-c       │           1.4.1           │ cpe:2.3:a:protobuf-c_project:protobuf-c │
│                       │                           │          :1.4.1:*:*:*:*:*:*:*           │
├───────────────────────┼───────────────────────────┼─────────────────────────────────────────┤
│       mbed_tls        │           3.4.1           │ cpe:2.3:a:arm:mbed_tls:3.4.1:*:*:*:*:*: │
│                       │                           │                   *:*                   │
├───────────────────────┼───────────────────────────┼─────────────────────────────────────────┤
│       freertos        │          10.4.3           │ cpe:2.3:o:amazon:freertos:10.4.3:*:*:*: │
│                       │                           │                 *:*:*:*                 │
└───────────────────────┴───────────────────────────┴─────────────────────────────────────────┘

...snip

Some highlights from the SBOM vulnerability check:

• wifi_prov_mgr application from ESP-IDF v5.2-dev was used for the exercise
• It is possible to scan the components which are not linked into final executable by adding --check-all-packages in the check command shown above
• It is also possible to generate the SBOM report for the ESP-IDF bootloader. Bootloader specific build description file can be found in the build/bootloader directory inside the project
• Some key components like RTOS, Networking Stack, TLS Stack etc. having Common Platform Enumeration entries have been highlighted in the above output
• Tool also displays information about other component and libraries (not shown in the above log) which are part of the build process

Report Format

This tool supports multiple formats for the generated output report like JSON, CSV etc. Please refer to the tool help for more details.

Compatibility

SPDX being a standard format for SBOM, also offers compatibility with the external tools. SBOM files generated from esp-idf-sbom can also be scanned using tools like cve-bin-tool. This offers an added advantage if you already have any external tools integrated in your system and want to take advantage of its specific features (e.g., more extended scan against different CVE databases).

Summary

To summarize, some key advantages of SBOM are:

• Forms inventory of the components building the software artifacts
• Lists software licenses, component versions and helps to keep them up-to-date
• Helps to perform security vulnerability analysis
• Becoming a pre-requisite in many security related certifications

ESP-IDF-SBOM tool is useful to generate SBOM for ESP-IDF based projects. The tool can also further help to scan and setup process for continuous security vulnerability scanning of the project.

We are actively working on to enable wider SBOM coverage across our software solutions and looking for feedback on the ESP-IDF-SBOM tool.

Please do try and share your experience!

Software Bill of Materials was originally published in The ESP Journal on Medium, where people are continuing the conversation by highlighting and responding to this story.

ESP32 series of chips supports secure boot scheme to allow only trusted firmware to execute from its flash storage. However, this requires careful management of the secure signing key, it must be generated and stored in a secure manner by the signing entity.

External HSM (Hardware Security Module) is specially designed cryptographic device that safeguards and manages secrets. HSM also provides specialised cryptographic services like encryption, digital signing etc. Now a days, there are cloud services being offered that provide cloud HSM based solutions.

External HSM Signing

External HSM offers a standard PKCS #11 based cryptographic interface for availing various services like signing the firmware. With PKCS #11 interface, external HSM could also be a remote cloud HSM based service over the network.

• Espsecure.py script (part of Esptool project) has been integrated with standard PKCS #11 interface and thus ensures interoperability with any external HSM for getting the ESP32's firmware signed.
• Please note that, as shown in the above diagram path of the vendor specific PKCS #11 library must be provided in the Espsecure.py configuration to use HSM mode.
• The initial support allows to generate signed application as per our Secure Boot V2 scheme with either RSA-PSS 3072 or ESDSA NISTP256 algorithm.

Signing Using YubiKey

We will be using YubiKey 5 Series as an external HSM for demonstration here.

Installation

Please find detailed setup guide for YubiKey host tool and PKCS #11 library setup here.

Note: Following are verified steps for Ubuntu 22.10 OS

# Install esptool 4.5 release along with HSM dependencies
pip install esptool[hsm]==4.5.dev3

# Install tools and PKCS#11 interface library
sudo apt install yubico-piv-tool ykcs11

# Generate ECC P256 private key in the 9c (digital signature) slot
yubico-piv-tool -a generate -s 9c -A ECCP256

HSM config file

Following is a HSM config file that we shall pass to espsecure.py

$ cat hsm_cfg.ini

# Config file for the external YubiKey based Hardware Security Module
[hsm_config]
# PKCS11 shared object/library
pkcs11_lib = /usr/lib/x86_64-linux-gnu/libykcs11.so
# HSM login credentials (default YubiKey pin)
credentials = 123456
# Slot number to be used (default YubiKey slot)
slot = 0
# Label of the object used to store the private key (default)
label = Private key for Digital Signature
# Label of the object used to store corresponding public key (default)
label_pubkey = Public key for Digital Signature

Please check directory where libykcs11.so is installed on your system and update the path accordingly.

Generate signature

Following command helps to sign the binary using configuration supplied in the hsm_cfg.ini file

$ espsecure.py sign_data --version 2 --hsm --hsm-config hsm_cfg.ini --output signed.bin unsigned.bin 

espsecure.py v4.5-dev
Trying to establish a session with the HSM.
Session creation successful with HSM slot 0.
Trying to extract public key from the HSM.
Got public key with label Public key for Digital Signature.
Connection closed successfully
Trying to establish a session with the HSM.
Session creation successful with HSM slot 0.
Got private key metadata with label Private key for Digital Signature.
Signing payload using the HSM.
Signature generation successful.
Connection closed successfully
Pre-calculated signatures found
1 signing key(s) found.
Signed 65536 bytes of data from unsigned.bin. Signature sector now has 1 signature blocks.

Verify signature

For sanity purpose, we can verify the signature using public key from the external HSM

$ espsecure.py verify_signature --version 2 --hsm --hsm-config hsm_cfg.ini signed.bin 

espsecure.py v4.5-dev
Trying to establish a session with the HSM.
Session creation successful with HSM slot 0.
Trying to extract public key from the HSM.
Got public key with label Public key for Digital Signature.
Connection closed successfully
Signature block 0 is valid (ECDSA).
Signature block 0 verification successful using the supplied key (ECDSA).

Documentation

Please refer to the Esptool documentation here for more details.

Please note that this feature shall be available in Esptool v4.5 release. For now, you may use Esptool dev release using pip install esptool[hsm]==4.5.dev3to try out this feature.

Secure Signing Using External HSM was originally published in The ESP Journal on Medium, where people are continuing the conversation by highlighting and responding to this story.

We shall also cover some of the features from OTA updates, focused around reliability and security that can help to build better IoT products with ESP32.

Background Information

Lets cover some basic information essential for understanding the overall OTA update procedure.

Partition Table & Flash Layout

The Partition table defines the flash layout on ESP32. Applications can specify its desired flash layout using simple CSV files. A template looks like below:

# Name,   Type, SubType, Offset,   Size, Flags
# Note: if you have increased the bootloader size, make sure to update the offsets to avoid overlap
nvs,      data, nvs,     ,        0x4000,
otadata,  data, ota,     ,        0x2000,
phy_init, data, phy,     ,        0x1000,
ota_0,    app,  ota_0,   ,        1M,
ota_1,    app,  ota_1,   ,        1M,

Important points to note:

• A binary partition table image is generated using the above CSV file. This occupies 4KiB (single flash sector) and is located at offset 0x9000 in default configuration. First 8 sectors (32KiB), before this, are reserved for the bootloader image.
• If more space is desired for bootloader, then the partition table offset can be adjusted using configuration option CONFIG_PARTITION_TABLE_OFFSET.
• Partitions ota_0 and ota_1 denote active and passive versions for application firmware. Identical copies ensure power safety during OTA updates.
• Application image partitions (ota_0/ota_1) are always adjusted to align on 64 KiB boundary, as that is the page size for flash cache mapping.
• Even though factory partition is supported, we will not be considering it, as that partition is not supported in anti-rollback scheme. If required, that logic can be coupled in primary firmware itself.
• For simplicity we will also not consider multiple firmware partitions (total 16) even though they are supported in ESP-IDF.

Visual representation of above configuration for 4MiB flash part is shown below:

Application Image Format

A detailed documentation can be found here, whereas, here we will only consider interesting bits that are applicable for this discussion.

Following is the application image descriptor (present at offset 0x20 in application image):

typedef struct {
    uint32_t magic_word;        /*!< Magic word ESP_APP_DESC_MAGIC_WORD */
    uint32_t secure_version;    /*!< Secure version */
    uint32_t reserv1[2];        /*!< reserv1 */
    char version[32];           /*!< Application version */
    char project_name[32];      /*!< Project name */
    char time[16];              /*!< Compile time */
    char date[16];              /*!< Compile date*/
    char idf_ver[32];           /*!< Version IDF */
    uint8_t app_elf_sha256[32]; /*!< sha256 of elf file */
    uint32_t reserv2[20];       /*!< reserv2 */
} esp_app_desc_t;

Few important fields to note:

• version: Firmware version embedded in image. This can be embedded in firmware image using multiple methods including configuration option or version file. By default its retrieved using command git describe --always --tags command if application is hosted in a git repository.
• secure_version : Security version embedded in image. This can be embedded using configuration option only. This is desired as it groups firmware images based on fixes against security vulnerabilities (e.g. revoked CA certificate) and thus ensuring security. More on this in followup section on anti-rollback feature.

Software Architecture

Above is simplified diagram that highlights some key components in an application from ESP-IDF that deals with OTA updates. In most cases, application needs to interact with public interface of esp_https_ota and app_update components only.

• ESP_HTTPS_OTA: This software component provides simplified API for updating device securely over TLS channel. API accepts two mandatory parameters, HTTPS URL where firmware image is hosted and server certificate to validate server’s identity.
• APP_Update: This software component abstracts out the complexity of dealing with partition and flash layer operations. It exposes simplified API that can be directly used by an application for programming firmware to device flash. Additionally it also provides some convenience APIs for dealing with rollback and anti-rollback use-cases that we shall cover in the later part of this article.

HTTPS OTA API

API for secure OTA update is kept very simple and requires 2 mandatory parameters:

• Server (Firmware) URL
• Server Certificate

Code snippet is as shown below, assuming firmware image is hosted on github.com server:

esp_http_client_config_t config = {
     .url = "https://raw.githubusercontent.com/path/to/firmware",
     .cert_pem = (char *) github_server_start_pem,
};

esp_err_t ret = esp_https_ota(&config);
if (ret == ESP_OK) {
     esp_restart();
} 

There is also support for advanced APIs which provide more granular control to the application in terms of validation of firmware version, monitoring OTA update progress etc.

Detailed documentation for this can be found here.

OTA Update Workflow

Simplified workflow for OTA update looks like below:

• Server authentication ensures that firmware is downloaded from trusted entity.
• Based on version information in downloaded image header, device can decide to perform upgrade.
• Validation on device side includes header and integrity (SHA256 checksum) related checks. This happens after entire firmware image is written to flash partition.
• If hardware secure boot is enabled then firmware image has signature appended to it. For flash encryption cases, it is entirely handled on device level during flash write stage. More on this in later section.

Boot-loader & Application Flowchart

Two entities play an important role in OTA update process, the boot-loader and the application.

• The boot-loader and application both rely on the partition table to extract information on various partitions and their offsets in flash.
• Partition otadata is responsible for selection of active firmware (latest updated) based on sequence numbers stored within it.
• otadata partition is allocated with 2 flash sectors (similar to active/passive firmware, 8KiB size) and thus allows for power safety while updating sequence numbers and other parameters during OTA update.

Following is a simplified flowchart for decision making involved during OTA update in an application. Also it shows how boot-loader picks up newly updated firmware after OTA update.

Flash Layout Transition

For normal OTA update scenario, flash layout transition is rather simple. It goes through following stages:

Stage 1

Newly shipped smart light bulb would have functional ota_0 partition and empty ota_1 partition. Boot-loader will always give control to firmware in ota_0 partition.

Stage 2

Application after performing OTA update will make ota_1 partition as active by updating its sequence number in otadata partition. Thus from subsequent RESET, boot-loader will always give control to firmware in ota_1 partition (till next OTA update).

Limitations

In the above scenario, if there is some issue (e.g. early firmware bootup crash) in the newly updated firmware from partition ota_1, there is no way the device can fallback to previous known working firmware.

Consideration I — Rollback

Typically OTA updates are staged and well tested in actual deployment scenarios, e.g. friend-family devices, beta-users and then finally to larger device groups. However this still can not ensure 100% recovery if OTA update goes wrong due to many parameters like variation in hardware attributes of device, or some rare firmware crashes.

The firmware rollback feature provides an ability to rollback to the previous known working firmware.

Boot-loader & Application Flowchart

• Unlike normal OTA update workflow, rollback case involves few intermittent states before marking firmware as valid.
• Transition from new firmware to valid or invalid is handled by both application and boot-loader.

Flash Layout Transition

For rollback enabled OTA update scenario, flash layout transition goes through following stages:

Stage 1

Newly shipped smart light bulb would have functional ota_0 partition and empty ota_1 partition. Boot-loader will always give control to firmware in ota_0 partition.

Stage 2

First OTA update writes new firmware image in passive partition (ota_1) and marks its state as ESP_OTA_IMG_NEW. The state is stored in otadata partition along with its sequence number.

Stage 3

After RESET, boot-loader sees a new firmware with stateESP_OTA_IMG_NEWand turns its state to ESP_IMAGE_PENDING_VERIFY, as its functionality is yet to be validated.

Stage 4/4'

Once new firmware starts execution, it can set its state to either ESP_OTA_IMG_VALID or ESP_OTA_IMG_INVALID based on application logic (will come to that in next section). If it fails to update the state of new and first time executing firmware then on subsequent RESET, boot-loader updates its state to ESP_OTA_IMG_ABORTED and thus falls back to previous firmware. Following APIs are provided for state transitions:

// Mark firmware as valid and cancel rollback process
esp_ota_mark_app_valid_cancel_rollback();

// Mark firmware as invalid and rollback to previous firmware
esp_ota_mark_app_invalid_rollback_and_reboot();

Firmware Functionality

Lets discuss more on transition to “Stage 4” from above.

Firmware functionality can be decided based on various checkpoints within it.

• Successful initialisation
• Connection to WiFi network
• Connection to OTA server

Connection to OTA server being the best possible option, as it ensures possibility of next OTA update. However this may need some changes on cloud agent side as well, which can acknowledge communication from device firmware. Once firmware reaches to desired checkpoint successfully, it can invoke earlier mentioned API to cancel rollback process and continue further execution.

Note: Rollback feature can be enabled through config option, which is discussed in more detail in followup section through working example code.

Consideration II — Anti-Rollback

Rollback offers an ability to go back to previous working firmware but sometimes this may not be a desired operation. Many times, a device firmware update can also modify some secure credentials (e.g. server’s certificate) embedded within it. This implies that all older firmware images are pretty much obsolete and if executed may posses security risks. “Firmware Anti-rollback” feature can help to mitigate such issues.

Anti-rollback is a feature that allows to execute firmware only-if the security version embedded within it is greater than the one programmed in the device’s EFuse (one-time-programmable memory). Certain bits (up-to 32) can be reserved to store the security version in EFuse. It is ensured that this is kept in sync with the one embedded in the firmware image by both the application and boot-loader during startup.

Above image illustrates idea of grouping firmware based on their security versions.

• Each group (G1/G2/G3) has 3 firmware images with different firmware version but same security version.
• If device anti-rollback reserved EFuse has 2 bits set (which means security version 2) then only firmware from G2 and G3 groups will be allowed to execute on device. Firmware images from group G1 having lesser security version will not be allowed to boot.
• Anti-rollback is tightly coupled with rollback, as in version in EFuse is only updated after functionality of new firmware image is confirmed. This can be seen in “Simplified rollback OTA update flowchart” shown earlier.

Working Example in ESP-IDF

Working example which implements rollback and anti-rollback features can be found here. This is slightly modified version of advanced_https_ota example from ESP-IDF.

Configuration Changes

Updated sdkconfig.defaults file looks like below:

# Use custom partition table without factory partition
CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions_example.csv"

# Enable firmware versioning from config option CONFIG_APP_PROJECT_VER_FROM_CONFIG=y
CONFIG_APP_PROJECT_VER="1"

# Enable firmware rollback and anti-rollback features
CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE=y
CONFIG_BOOTLOADER_APP_ANTI_ROLLBACK=y

# Set initial secure version to 0 CONFIG_BOOTLOADER_APP_SECURE_VERSION=0

# Set secure version eFuse field size CONFIG_BOOTLOADER_APP_SEC_VER_SIZE_EFUSE_FIELD=32

# Emulate secure version through flash partition for testing CONFIG_BOOTLOADER_EFUSE_SECURE_VERSION_EMULATE=y

• Both version and secure_version fields in the firmware image are set through config options itself.
• For testing purpose, we have enabled an option to emulate anti-rollback EFuses through flash partition. For actual product deployment scenario this option should be disabled.

Code Changes

• We have used information from API esp_https_ota_get_img_desc() to decide if OTA update is really required or not. This API retrieves image descriptor part thus its useful to get metadata that involves firmware version and security version. This also prevent further image download from remote server and unnecessary flash program cycles.
• Important point to note is that we are considering successful Wi-Fi connection as a checkpoint within firmware to cancel rollback process and mark image as valid.
• Same API (esp_ota_mark_app_valid_cancel_rollback) is also responsible for updating security version in EFuse matching with current firmware header.

Consideration III — Secure Boot & Flash Encryption

In this section we will see how OTA updates can be coupled with platform security features (more information covered here) in ESP32.

• Secure boot allows only trusted firmware to execute by verifying its signature with public key on device.
• During an OTA update, the firmware image hosted on the server should be signed with ECDSA NIST256p (secure boot v1 scheme) or RSA-3072 (secure boot v2 scheme) based private key. Signature is calculated on SHA256 checksum of firmware image, offering constant time verification, and then it is appended to it.
• As mentioned earlier in the flowchart diagram, the OTA workflow on application side includes image validation stage. For non secure boot enabled cases, validation includes only integrity checks in the form of image header magic bytes and plain SHA256 checksum. For secure boot enabled cases, it involves verification of signature using public key stored in device.
• After the secure boot verification stage, the firmware is marked as valid (non-rollback case) or new (rollback case). If signature is found to be invalid, its marked as invalid right away aborting the OTA update procedure.
• For flash encryption case there are no additional steps are required on server or device side. Firmware image hosted on the server is still in plain text format with (optional) secure boot signature appended to it. During flash write stage on the device, if flash encryption is enabled then firmware gets written in encrypted manner using hardware AES engine and key from EFuse.

Important Notes

• Above discussed features like “rollback” and “anti-rollback” must be enabled before device ships out of factory. Since these features enable certain code path in bootloader, which in itself can not be updated on-field.
• It is recommended that any sort of power save modes (e.g. WiFi modem sleep) should be disabled during OTA update. This allows optimal throughput for data transfer. ESP-IDF provided examples take explicit care for this.
• During OTA update flash goes through programming operations (erase/write) and hence flash cache stays disabled. If there are any interrupts that are required to kept enabled, then they should be registered with their handler in internal code memory (IRAM). More information on this can be found here.

Getting robust OTA update for IoT product is very essential and some of the considerations mentioned above can be helpful in making right design decisions!

OTA Updates Framework was originally published in The ESP Journal on Medium, where people are continuing the conversation by highlighting and responding to this story.

Memory has significant impact on silicon cost as well as die size, hence from hardware perspective having optimal size is important and from software perspective being able to utilise it to fullest is crucial.

In this post we will discuss some upcoming features and commonly available configuration options (knobs) in ESP-IDF to allow end application to utilise various internal memory regions in most optimal way.

Important Notes

• We will focus on single core mode of ESP32 here, as that is where more number of memory optimisation features are applicable.
• We will be considering typical IoT use-cases here where gaining memory at the cost of performance is acceptable criteria.
• We will take typical cloud application as case study which requires TLS connection with mutual authentication support.
• ESP-IDF feature branch used here can be found at, https://github.com/mahavirj/esp-idf/tree/feature/memory_optimizations

ESP32: Internal Memory Breakup

• As can be seen from above memory layout, there are various memory regions internal to silicon and of different clock speed.
• For single core use-cases, we get additional 32K of instruction memory, which otherwise would have acted as cache for APP CPU core.
• Instruction RAM access should always be 32-bit address and size aligned.
• For end application business logic, it is always desirable to have more DRAM, which is fastest memory without any access restrictions.

Case Study — AWS IoT Sample Application

• We will be taking subscribe_publish example from ESP-AWS-IoT here as case study to analyse memory utilisation.
• ESP-IDF provides an API to get minimum free heap or dynamic memory available in system using heap_caps_get_minimum_free_size(). Our aim would be to maximise this number (for relative analysis) and thus increase the amount of memory available for end application specific business logic (DRAM region to be specific).

Default Memory Utilisation

We will be using following patch on top of subscribe_publish example to log dynamic memory statistics.

• First we will be logging system level minimum free heap numbers for DRAM and IRAM regions respectively as mentioned earlier.
• Second we will be using heap task tracking feature which provides information on dynamic memory usage on per task basis. This feature is modified to also log peak usage numbers for DRAM and IRAM regions on per task basis.
• We will be logging this information for aws_iot_task, tiT (tcpip) and wifi tasks respectively (as these tasks define data transfer path from application layer to physical layer and vice-versa). It is also to be noted that there will some variation in peak memory usage of networking tasks based on environmental (like WiFi connection, network latency) factors.

Note: Change in core-id during task creation (from below patch) is for single core configuration that we are using for this particular example.

With default configuration (and heap task tracking feature enabled) we get following heap utilisation statistics (all values in bytes):

Task Heap Utilisation Stats:
||  Task         |  Peak DRAM  |   Peak IRAM ||
||  aws_iot_task |   63124     |     0       || 
||  tiT          |   3840      |     0       || 
||  wifi         |   31064     |     0       ||

System Heap Utilisation Stats:
||   Minimum Free DRAM |   Minimum Free IRAM || 
||       152976        |        40276        ||

Single Core Config

As mentioned earlier we will be using single core configuration for all our experiments. Please note that even in single core mode there is enough processing power available in ESP32 (close to 300 DMIPS), more than sufficient for typical IoT use-cases.

Corresponding configuration to be enabled in application:

CONFIG_FREERTOS_UNICORE=y

Updated heap utilisation statistics from application re-run as following:

Task Heap Utilisation Stats:
||  Task         |  Peak DRAM  |   Peak IRAM ||
||  aws_iot_task |   63124     |     0       || 
||  tiT          |   3892      |     0       || 
||  wifi         |   31192     |     0       ||

System Heap Utilisation Stats:
||   Minimum Free DRAM |   Minimum Free IRAM || 
||       162980        |        76136        ||

As can be seen from above, we have gained ~10KB memory in DRAM, since some of services (e.g. idle, esp_timer tasks etc.) for second CPU core are not required anymore. In addition IPC service for inter processor communication is also no more required, so we gain from stacks and dynamic memory of that service. Increase in IRAM is due to the freeing up of 32KB cache memory of second CPU core and some of code savings due to disablement of above mentioned services.

TLS Specific

Asymmetric TLS Content Length

This feature has been part of ESP-IDF from v4.0 onwards. This feature allows to enable asymmetric content length for TLS IN/OUT buffers. Thus application has an ability to reduce TLS OUT buffer from its default value of 16KB (maximum TLS fragment length per specification) to as small as say 2KB, and thus allowing 14KB of dynamic memory saving.

Please note that, it is not possible to reduce TLS IN buffer length from its default 16KB, unless you have direct control over server configuration or sure about server behaviour that it will never send inbound data (during handshake or actual data-transfer phase) over certain threshold.

Corresponding configuration to be enabled in application:

# Enable TLS asymmetric in/out content length
CONFIG_MBEDTLS_ASYMMETRIC_CONTENT_LEN=y
CONFIG_MBEDTLS_SSL_OUT_CONTENT_LEN=2048

Updated heap utilisation statistics from application re-run as following:

Task Heap Utilisation Stats:
||  Task         |  Peak DRAM  |   Peak IRAM ||
||  aws_iot_task |   48784     |     0       || 
||  tiT          |   3892      |     0       || 
||  wifi         |   30724     |     0       ||

System Heap Utilisation Stats:
||   Minimum Free DRAM |   Minimum Free IRAM || 
||       177972        |        76136        ||

As can be seen from above, we have gained ~14KB memory from aws_iot_task and thus minimum free DRAM number has increased accordingly.

Dynamic Buffer Allocation Feature

During TLS connection, mbedTLS stack keeps dynamic allocations active during entire session starting from initial handshake phase. These allocations includes, TLS IN/OUT buffers, peer certificate, client certificate, private keys etc. In this feature (soon to be part of ESP-IDF), mbedTLS internal APIs have been glued (using SHIM layer) and thus it is ensured that whenever resource usage (including data buffers) is complete relevant dynamic memory is immediately freed up.

This greatly helps to reduce peak heap utilisation for TLS connection. This will have small performance impact due to frequent dynamic memory operations (on-demand resource usage strategy). Moreover since memory related to authentication credentials (certificate, keys etc.) has been freed up, during TLS reconnect attempt (if required), application needs to ensure that mbedTLS SSL context is populated again.

Corresponding configuration to be enabled in application:

# Allow to use dynamic buffer strategy for mbedTLS
CONFIG_MBEDTLS_DYNAMIC_BUFFER=y
CONFIG_MBEDTLS_DYNAMIC_FREE_PEER_CERT=y
CONFIG_MBEDTLS_DYNAMIC_FREE_CONFIG_DATA=y

Updated heap utilisation statistics from application re-run as following:

Task Heap Utilization Stats:
||  Task         |  Peak DRAM  |   Peak IRAM ||
||  aws_iot_task |   26268     |     0       || 
||  tiT          |   3648      |     0       || 
||  wifi         |   30724     |     0       ||

System Heap Utilization Stats:
||   Minumum Free DRAM |   Minimum Free IRAM || 
||       203648        |        76136        ||

As can be seen from above, we have gained ~22KB memory from aws_iot_task and thus minimum free DRAM number has increased accordingly.

Networking Specific

WiFi/LwIP Configuration

We can further optimise the WiFi and LwIP configuration to reduce memory usage, at the cost of giving away some performance. Primarily we will reduce WiFi TX and RX buffers and try to balance it out by moving some critical code path from networking subsystem to instruction memory (IRAM).

To give some ballpark on performance aspect, with default networking configuration average TCP throughput is close to ~20Mbps, but with below configuration it will be close to ~4.5Mbps, which is still sufficient to cover typical IoT use-cases.

Corresponding configuration to be enabled in application:

# Minimal WiFi/lwIP configuration
CONFIG_ESP32_WIFI_STATIC_RX_BUFFER_NUM=4
CONFIG_ESP32_WIFI_DYNAMIC_TX_BUFFER_NUM=16
CONFIG_ESP32_WIFI_DYNAMIC_RX_BUFFER_NUM=8
CONFIG_ESP32_WIFI_AMPDU_RX_ENABLED=
CONFIG_LWIP_TCPIP_RECVMBOX_SIZE=16
CONFIG_LWIP_TCP_SND_BUF_DEFAULT=6144
CONFIG_LWIP_TCP_WND_DEFAULT=6144
CONFIG_LWIP_TCP_RECVMBOX_SIZE=8
CONFIG_LWIP_UDP_RECVMBOX_SIZE=8
CONFIG_ESP32_WIFI_IRAM_OPT=y
CONFIG_ESP32_WIFI_RX_IRAM_OPT=y
CONFIG_LWIP_IRAM_OPTIMIZATION=y

Updated heap utilisation statistics from application re-run as following:

Task Heap Utilization Stats:
||  Task         |  Peak DRAM  |   Peak IRAM ||
||  aws_iot_task |   26272     |     0       || 
||  tiT          |   4108      |     0       || 
||  wifi         |   19816     |     0       ||

System Heap Utilization Stats:
||   Minumum Free DRAM |   Minimum Free IRAM || 
||       213712        |        62920        ||

As can be seen from above log, we have gained roughly ~9KB of additional DRAM for application usage. Impact (reduction) in total IRAM comes because we moved critical code path from networking subsystem to this region.

System Specific

Utilising RTC (Fast) Memory (single-core only)

As can be seen from earlier memory breakup there is one useful 8KB RTC Fast memory (and reasonably fast) which has been sitting idle and not fully utilised. ESP-IDF will soon have feature to enable RTC Fast memory for dynamic allocation purpose. This option exists in single core configuration, as the RTC Fast memory is accessible to PRO CPU only.

It has been ensured that RTC Fast memory region will be utilised as first dynamic memory range, and most of startup, pre-scheduler code/services will occupy this range. This will allow to not have any performance impact in application code due to clock speed (slightly on slower side) of this memory.

Since there are no access restrictions to this region, capability wise we will call it as DRAM henceforth.

Let’s re-run our application with this feature and gather memory numbers.

Corresponding configuration to be enabled in application:

# Add RTC memory to system heap
CONFIG_ESP32_ALLOW_RTC_FAST_MEM_AS_HEAP=y

Updated heap utilisation statistics from application re-run as following:

Task Heap Utilization Stats:
||  Task         |  Peak DRAM  |   Peak IRAM ||
||  aws_iot_task |   26272     |     0       || 
||  tiT          |   4096      |     0       || 
||  wifi         |   19536     |     0       ||

System Heap Utilization Stats:
||   Minumum Free DRAM |   Minimum Free IRAM || 
||       221792        |        62892        ||

As can be seen from above log, we have gained 8KB of additional DRAM for application usage.

Utilising Instruction Memory (IRAM, single-core only)

So far we have seen different configuration options to allow end application to have more memory from DRAM (data memory) region. To continue along similar lines, it should be noted that there is still sufficient IRAM (instruction memory) left but it can not be used as generic purpose due to 32-bit address and size alignment restrictions.

• If access (load or store) is from IRAM region and size is not word-aligned, then processor will generate LoadStoreError(3) exception
• If access (load or store) is from IRAM region and address is not word-aligned then processor will generate LoadStoreAlignmentError(9) exception

In this particular feature from ESP-IDF, above mentioned unaligned accesses have been fixed through corresponding exception handlers and thus resulting in correct program execution. However, these exception handlers can take up-to 167 CPU cycles for each (restricted) load or store operation. So there could be significant performance penalty (as compared with DRAM access) while using this feature.

This memory region can be used in following ways:

• First through heap allocator APIs using special capability field known as MALLOC_CAP_IRAM_8BIT
• Second by redirecting DATA/BSS to this region using provided linker attributes, IRAM_DATA_ATTR and IRAM_BSS_ATTR

Limitations wise:

• This memory region can be not be used for DMA purpose
• This memory region can not be used for allocating task stacks

While discussing usage of this memory region with understood performance penalty, TLS IN/OUT (per our configuration value of buffer 16KB/2KB) buffers were found to be one of the potential candidate to allocate from this region. In one of the experiments, for transfer of 1MB file over TLS connection, time increased from ~3 seconds to ~5.2 seconds with TLS IN/OUT buffers moved to IRAM.

It is also possible to redirect all TLS allocations to IRAM region but that may have larger performance impact and hence this feature redirects only buffers whose size is greater or equal than minimum of TLS IN or OUT buffer (in our case threshold would be 2KB).

Let’s re-run our application with this feature and gather memory numbers.

Corresponding configuration to be enabled in application:

# Allow usage of IRAM as 8bit accessible region
CONFIG_ESP32_IRAM_AS_8BIT_ACCESSIBLE_MEMORY=y
CONFIG_MBEDTLS_IRAM_8BIT_MEM_ALLOC=y

Updated heap utilisation statistics from application re-run as following:

Task Heap Utilization Stats:
||  Task         |  Peak DRAM  |   Peak IRAM ||
||  aws_iot_task |   17960     |     21216   || 
||  tiT          |   3640      |     0       || 
||  wifi         |   19536     |     0       ||

System Heap Utilization Stats:
||   Minumum Free DRAM |   Minimum Free IRAM || 
||       228252        |        40432        ||

As can be seen from above log, we have gained another ~7KB of additional DRAM for application usage. Please note that, even though we have redirected all allocations above 2KB threshold to IRAM, there are still many smaller (and simultaneous) allocations happening from DRAM region (another local maxima) and hence effective gain is lower than what actually should have been. If additional performance impact is acceptable then it is possible to redirect all TLS allocation to IRAM and gain further at-least ~10–12KB memory from DRAM region.

Summary

• Having complete application control over selection of various features through configuration options is one of the important features in ESP-IDF.
• Through above exercise we have systematically evaluated various features and configuration options within ESP-IDF to increase DRAM (fastest memory) budget for end application by 63KB (minimum free DRAM size increased from ~160KB to ~223KB).
• Some of these configuration options are available for single-core configuration only (have been marked as such in title itself) but even in dual-core configuration memory savings are possible with rest of the options.
• It is further possible to utilise instruction memory (IRAM) as 8-bit accessible region for non-performance critical modules like logging and diagnostics by end application.
• It is recommended to disable some of the debugging features that we used in above exercise like heap task tracking to reduce metadata overhead (and further increase memory budget) once required system characterisation is achieved.

References

• Modified subscribe_publish example along with final sdkconfig.defaults file can be found here.
• This application should be built against ESP-IDF fork and feature branch from here.

ESP32 Memory Analysis — Case Study was originally published in The ESP Journal on Medium, where people are continuing the conversation by highlighting and responding to this story.

• Deploy new firmware on ESP32 in secure manner (single or group of devices, along with dynamic addition of new/re-provisioned device)
• Verify authenticity and integrity of new firmware after its deployed
• Extend OTA update security scheme to take leverage of hardware security features in ESP32

Working

At a high level,

• The firmware image (or any other partition’s content: filesystem, configuration data), is initially uploaded to an S3 bucket on AWS.
• An AWS OTA Job with the required certificates (for demo purpose can be self-signed) and code-signing profile (security scheme for ESP32 is ECDSA + SHA256) is setup.
• On the device side, the OTA agent from Amazon FreeRTOS needs to be enabled in the firmware, along with the certificate that is responsible for verifying the firmware update image (essentially ECDSA public key).
• The AWS OTA Job then takes the firmware image from the S3 bucket, signs it, and sends it over MQTT+TLS channel in small chunks to the OTA agent on the device.
• The OTA agent on the device then writes the newly received firmware to its storage and manages the state.
• At the end, firmware signature gets validated on the device and it gets approved for boot-up.
• Post boot-up, the OTA agent again interacts with AWS OTA Job for verifying sanity of firmware, and finally the firmware image gets marked as legitimate one, notifying the boot-loader to erase all older firmware instances from the device storage (for not allowing forced rollback).

Procedure

Lets quickly walk over the steps for getting the OTA update demo functional on ESP32:

• Please follow the Getting Started Guide for some of the prerequisites as documented at, https://docs.aws.amazon.com/freertos/latest/userguide/ota-prereqs.html
• Refer to https://docs.aws.amazon.com/freertos/latest/userguide/ota-code-sign-cert-esp.html for creating code signing profile for the ESP32 platform
• For downloading firmware to ESP32, refer to https://docs.aws.amazon.com/freertos/latest/userguide/ota-download-freertos.html#download-freertos-to-port
• Once the device boots up, the log should look like the one mentioned at, https://docs.aws.amazon.com/freertos/latest/userguide/burn-initial-firmware-esp.html
• Create an Amazon FreeRTOS OTA Job (by navigating to IoT Core -> Manage -> Jobs -> Create),

• Select “Sign a new firmware image for me” option,

• Create code signing profile, please select ESP32 platform here and provide certificates created earlier,

Enabling Hardware Security

The ESP32 port is so structured that the same secure firmware verification mechanism can be used by the ESP32 chipset for enabling secure boot.

ESP32’s secure boot scheme uses the same ECDSA + SHA256 algorithm. Hence the same public key that is used for the OTA firmware image verification can also be used by the bootloader to validate the firmware image on boot-up.

It is highly recommended that you use secure boot in conjunction with the OTA firmware updates in your products.

ESP32 OTA Updates — Amazon FreeRTOS was originally published in The ESP Journal on Medium, where people are continuing the conversation by highlighting and responding to this story.

Preface

Amazon FreeRTOS is the latest offering (after acquisition of FreeRTOS) from Amazon that tries to simplify cloud connectivity with Amazon services (e.g. Shadow, Green-grass etc.) It has support for multiple micro-controller platforms from different vendors. Our ESP32 is now officially supported in Amazon FreeRTOS.

In this article we will discuss about —

• ESP32 integration in Amazon FreeRTOS (using existing ESP-IDF)
• Getting Started with ESP32 using Amazon FreeRTOS

Architecture

Let us dig deeper in architecture of this offering in the context of ESP32 -

• Upstream or vanilla FreeRTOS had no support for theESP32 architecture (Xtensa LX6 dual core CPU). Support for this architecture is now available in the FreeRTOS V10 kernel. As of now it only supports single core mode of operation.
• A slightly modified version of the ESP-IDF is part of the Amazon FreeRTOS SDK itself. This IDF has all the required components including, hardware abstraction layer, Wi-Fi driver and other components that do not conflict with the ones already available in Amazon FreeRTOS SDK (e.g. Networking Stack, TLS stack etc.). Such a self contained package allows easier certification process as well.
• Another major change is that this port uses FreeRTOS’ native TCP/IP stack, rather than the LwIP stack from ESP-IDF. This has been enabled using the network interfacing layer that glues the FreeRTOS TCP/IP stack with ESP32’s Wi-Fi driver. More details about ESP32's networking architecture have been covered in this post.
• The WiFi portability layer acts as an abstraction, providing consistent application interface between different vendor platforms. Mostly the API for this layer helps in Wi-Fi interface/state management (Station and SoftAP) and power management.
• Another abstration layer is the secure socket layer shown above. It provides consistent application interface between different vendor’s network platforms (some of the vendors use custom Networking Stack) and interfacing with the mbedTLS stack.

Qualification

The Amazon FreeRTOS Qualification Program (Amazon FQP) is for microcontroller (MCU) vendors for qualifying platforms (not SoC itself), and both ESP32-DevKitC and ESP-WROVER-KIT are certified in the latest release.

For more information please refer to, https://aws.amazon.com/freertos/partners/

Getting Started

Please refer to the official Getting Started Guide on the AWS website for latest instructions on working with Amazon FreeRTOS: https://docs.aws.amazon.com/freertos/latest/userguide/getting_started_espressif.html

Examples

Please refer to https://docs.aws.amazon.com/freertos/latest/userguide/freertos-mds-projects-struct.html for navigating to examples, primarily includes:

• MQTT — Client that both subscribes to and publishes to the same MQTT topic, as a result of which each time the MQTT client publishes a message to the remote MQTT broker, the broker sends the same message back to the client
• Shadow — Simple lightbulb example to illustrate how client application and things communicate with the Shadow service
• Greengrass — Simple example that performs discovery of AWS greengrass core device

Future Road-map (Near Term)

Support for secure OTA updates using AWS is in Beta phase, please refer to, https://docs.aws.amazon.com/freertos/latest/userguide/freertos-ota-dev.html

This is certainly very important feature and as it evolves, all qualified vendors will support this, including Espressif.

This is still at early stage, please stay tuned as expecting some exciting road-map (with potential of use-cases it can cover) here!

Amazon FreeRTOS — ESP32 Support was originally published in The ESP Journal on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

In this article we will explore WiFi Networking Architecture in ESP-IDF (an Internet of Things development framework from Espressif for ESP32 chip).

Primary Components

• TCPIP Stack (lwIP): Embedded networking stack, with POSIX/BSD socket interface to application
• TCPIP Adapter Layer: Interfacing layer that provides wrappers on TCPIP stack APIs, for default handlers from Event Loop Task
• Event Loop Task: Task responsible for event dispatching and co-ordination between WiFi driver, TCPIP stack and Application. This has default handlers registered for various system events
• WiFi Driver: Acts as MAC layer in networking, with interface for sending/receiving data and WiFi events to higher layers

Application workflow

We are taking one of ESP-IDF example to demonstrate workflow of WiFi networking. Having said that, best way to represent would be using plantuml based sequence diagrams, there we go…

Part I — Initialization Phase

Typical workflow involves initialization of TCPIP Adapter, Event Loop Task and WiFi driver. Some key points to note here are:

• Application registers its event handler with Event Loop Task, refer here
• WiFi driver also needs event handler to send events across to higher layer, that happens during call to esp_wifi_init()using default WiFi configuration from here
• Default event handlers for various system events gets registered during Event Loop initialization, refer here
• During last step in above sequence diagram, WiFi driver tx callback, i.e. esp_wifi_internal_tx() gets registered with TCPIP stack, refer here and here

Part II — Setup Phase

• On successful WiFi association with access point, TCPIP stack input callback gets registered in WiFi driver, refer here
• Once application receives SYSTEM_EVENT_STA_START, it basically starts WiFi connection process using esp_wifi_connect(), followed up by DHCP phase

Part III — Data Transfer

This is straight forward, once there is acknowledgement for IP connectivity, communication can start with TCPIP stack provided POSIX socket interface

Call Graph

If you are really interested (not encouraged :P), maybe following call-graph can get more details, basically stripped version that involves only components discussed in above application workflow use-case.

Generated using GCC -fdump-rtl-expand CFLAG and egypt call graph tool, also available at https://raw.githubusercontent.com/mahavirj/misc/master/callgraph.png

Remarks

• Architecture looks clean and elegant
• Offers modularity as different components are not tightly coupled
• Application interface is simple and neat

What Next

So far so good, but can we really do some complex things here?

Maybe yes, e.g. replacing the entire Networking Stack (marked in blue above), lets explore that in the context of different (and exciting?) project with ESP32…Stay tuned!