Summary

When you use an ESP8266 board that has 16MB flash instead of 4MB, system performance is non-existent, most notably the config page does not load 99% of the time.

Tests

Scenario:

1. Erase entire flash of fresh ESP8266 board that has 16MB memory chip
2. Flash with any version of NAMF (I believe the same may be the case for original firmware). I was doing most of my tests on NAMF-2020-46rc5-en.bin
3. Connect to the board in AP mode with any computer that's close to the board. In my environment, NAMF was reporting a stable -31 dBm signal to client device.
4. Perform time curl -v http://192.168.4.1/config -o /dev/null several times

Expected outcome (true on same board but with 4MB flash):

• config page on average is fetched nearly instantly (less than 0.5 seconds)
• config page loads entirely (expecting network issues or similar but very rarely)

Actual outcome:

• page takes several seconds to load (average 2-8 seconds)
• most of the time, the page does not load and cURL exits with curl: (56) Malformed encoding found in chunked-encoding on various places (i.e. it's not single chunk)

Other observations:

• it doesn't matter whether configuration is on initial phase or something was saved before
• when there's no config on device, / redirects to /config and web browser hangs
• OS in mobile devices typically use hotspot detection and are also victim of broken /config, which may lead to confusion on user side
• DHCP server in AP mode is also extremely slow
• ICMP ping is stable, I believe it's due to different nature of library (async) which handles it
• /config.json is always working
• same behaviour when NAMF is connected to normal Wi-Fi and acts as a client

Background on flash itself

ESP8266 usually has 4MB SPI flash. However, some boards marketed as "WeMos D1 Mini Pro" that are preferred for their external antenna are sold with 16MB SPI flash.

That flash chip can be of various quality, but all my boards use Zbit ZB25VQ. Additionally, I'm quite certain that this can be ruled out, as flashing and dumping speeds were constantly the same on all boards.

Moreover, I tested all combinations of below flashing parameters (in esptool.py):

• size: 4MB vs 16MB (shouldn't matter as the image is compiled for 4MB)
• flash mode (see Espressif docs: DIO vs DOUT vs QIO vs QOUT
• flash frequency: 20 vs 26 vs 40 vs 80 MHz

While all those impact actual runtime (NAMF reports it) and there are some minor performance differences, it never solves the issue. It could be too small testing sample, but best results were on 16MB/QOUT/40MHz combination, when loading time of config page that didn't break was 1.5-3 seconds and I had somewhere between 50% and 75% success rate.

SPIFFS is using flash to store configuration, and it seems like during config page rendering it's read many times.

The chip I have on 16MB board seems to relatively rare, but some people reported similar issues on completely different platforms using ESP8266. For reference, it's 25VQ128DSJG.

My suspicions

After some digging, I think the culprit is SPIFFS on 16MB flash in general.

NAMF uses 4m2m layout by default, which means 4MB total flash, 2MB for SPIFFS, leaving 2MB for compiled program. I tried to rule out some other aspects of that profile, which are implemented in Arduino, like block size etc. The motivation is that SPI flash chips actually have internal cache, and cache miss ratio could be high due to different sizes of data. To achieve that, I tried to build an image of NAMF with 16m2m and 16m1m. This yielded little to low improvements.

In other words, I tried both:

• profiles optimized for larger flash, but with huge SPIFFS (14 or 15MB) - probably slow due to huge flash area allocated to SPIFFS
• profiles optimized for original flash size (4MB) - probably slow due to flash chip internal cache

Unfortunately, there's no easy way to create a custom profile like 16m14m, as those are compiled into development tools and not parametrized.

esp8266/Arduino#7095 (comment) seems to confirm that SPIFFS is expected to struggle on 16MB chip, and it doesn't matter how much of memory is actually allocated.

I also tried changing code of config page render process to add chunk sending more often and to include free memory stats in HTML response, but it looks like it's not the core of the issue. However, afterr random poking around the code it looks like memory is not explicitly freed - it's reserved on #77 and probably should be freed here: #77

Proposed solution

At the moment, we can at least put huge warning in docs, that 16MB flash is problematic.

Ideally, SPIFFS should be replaced with LittleFS, as it's deprecated and expected to be dropped from ESP8266 Arduino SDK completely. On the other hand, it looks like it has own issues and may not solve 16MB variant issues alow.

Web UI rendering rewrite

I was trying to figure out what's the core of the problem, but this spaghetti code originating from the original project is a nightmare. A few months ago, when I was considering porting some feature from the original project (SSL for InfluxDB) I was scared by the number of places I'd need to change to add a simple option.

I hope to find time to work on my fork that would replace HTML and JSON rendering that's currently a series of functions with a more modern approach: static HTML and JS files hosted from SPIFFS and only dynamic content implemented by serving JSON (it can be easily rendered from structs).

This Web UI already uses plenty of JavaScript and XHR to load content (e.g. WiFi list), so I believe it wouldn't be a big deal (i.e. there are no hardcore users with NoScript). Moreover, all endpoints used in external integrations could be left as-is, so it'd stay compatible. Furthermore, it's a weird decision to perform heavy tasks like rendering templates and actual HTML tags on ESP8266, when client devices have several orders of magnitude more power and memory to render those on the frontend.

My fork is https://github.com/danielskowronski/namf/tree/rewrite_webui