Build a $40 Voice Assistant That Never Phones Home

xenotrek for neonmaxima · September 29, 2026

The ESP32-S3 runs warm in your hand. That’s the ceramic antenna converting 3.3 volts into a radio signal that reaches your router two rooms away. Some people buy this chip to blink LEDs and log humidity readings for a greenhouse. Some make it into cool digital pets. We’re going to make it listen, think, and talk back without a single packet leaving your network.

I have a complicated relationship with microphones. Spent enough years in rooms where everything was recorded to develop a permanent allergy to surveillance equipment I didn’t install myself. Every commercial voice assistant is a wire. Alexa, Siri, Google Assistant, the new ChatGPT Voice thing. Your words leave your house, get processed on hardware you don’t control, and come back alongside a metadata trail that includes when you talked, what you said, and what room the echo profile suggests you were standing in.

The acoustic fingerprinting is the part that gets me. They can tell which room you’re in by the reverb pattern. They know if you’re alone. They know if you’ve rearranged your furniture. This is the kind of surveillance that used to require a physical entry team and a court order.

This build eliminates all of it. A $40 board handles the ears and mouth. A server on your local network handles the brain. Nothing crosses your perimeter. No API keys. No monthly bill. No acoustic profile uploaded to a company that will get breached. Not “might get breached.” Will. It’s a matter of when, and how embarrassing the disclosure email reads.

The Hardware: One Board Does Almost Everything

The Waveshare ESP32-S3-AUDIO board is the piece that makes this build practical instead of theoretical. It packages an ES8311 codec, an ES7210 ADC with a dual-microphone array, hardware echo cancellation, a speaker amplifier, an RGB LED ring, and 8MB of PSRAM into a single board that costs around $40.

The dual-mic array matters more than it sounds on the spec sheet. Single-microphone voice assistants pick up their own speaker output and feed it right back into the recognition pipeline. The device hears itself and tries to parse the echo. I’ve seen interrogation rooms with better acoustic separation. The ES7210's hardware echo cancellation means the board can talk and listen simultaneously without getting confused by its own voice.

Your full bill of materials:

• A USB-C cable for power and initial flashing

• A server machine on your LAN (anything that runs Docker: a Pi 5, a retired laptop, a NAS)

• Wi-Fi that reaches both devices

That’s it. No external DAC, no I2S breakout board, no amplifier module. Everything lives on one PCB. The simplest builds are always the most dangerous ones. Fewer components means fewer points of failure means fewer excuses not to actually finish the project.

The Software Stack: Four Links in the Chain

The signal path looks like a phone tree from 1993. Audio goes in on the ESP32-S3, gets shipped over WebSocket to a server on your LAN, runs through three AI modules in sequence, and the synthesized voice response comes back down the same connection.

On the ESP32-S3: Xiaozhi firmware. Open source, handles wake word detection on-device so the board isn’t streaming audio constantly, manages the WebSocket connection, and plays back the synthesized response. The default wake word is a Chinese phrase. You can swap it to “Jarvis” or “Computer” or whatever satisfies your particular flavor of main character energy. I won’t judge. I named my first wardialer after a character from Neuromancer.

Speech-to-Text: SenseVoice running through FunASR. Turns your spoken words into text. About 900MB, runs on CPU, handles English well once you point the config at the right language model. The default ships configured for Chinese, which will confuse you for about twenty minutes until you find the language setting buried three levels deep in a YAML file. Consider it a character-building exercise.

Get Aeon Flex, Elriel Assoc. 2133 [NEON MAXIMA]’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Language Model: Any OpenAI-compatible API server works here. The original build used Gemma through LM Studio, but the architecture is API-shape agnostic. Ollama with Llama 3.1 8B works. So does llama.cpp server mode. So does a quantized Qwen 3.5 if you want multilingual responses. This is the most personal part of the build because it determines the personality of the thing you’re talking to, and I’ve learned the hard way that personality matters when you’re having a conversation with a box at 2 AM.

Text-to-Speech: Kokoro via FastAPI. Synthesizes natural-sounding voice from the LLM’s text response. The quality is surprisingly good for a local model. Not “I forgot this was a machine” good, but leagues past the robotic monotone that most open-source TTS ships with. It sounds like a person who hasn’t had enough coffee, which is honestly more relatable than Siri’s performative cheerfulness.

The Assembly

The ESP32-S3 side is a firmware flash. Connect USB-C, run esptool.py or use the Xiaozhi web flasher, select your board variant, flash. The board reboots with an LED pattern that tells you it’s looking for Wi-Fi. If you’ve ever flashed custom firmware onto a pager or a two-way radio, this is the same feeling. That moment where the device blinks and you know it’s running your code now, not theirs.

The server side is a Docker Compose stack. Three containers: SenseVoice for STT, your chosen LLM server, and Kokoro for TTS. A coordinator ties them together and exposes the WebSocket endpoint that the ESP32-S3 connects to.

# Simplified architecture — verify exact compose structure before publishing

services:

stt:

image: sensevoice-funasr:latest

# ~900MB model downloaded on first run

llm:

image: ollama/ollama:latest

# Pull your preferred model after startup

tts:

image: kokoro-fastapi:latest

# Natural voice synthesis

coordinator:

# Ties STT → LLM → TTS pipeline together

# Exposes WebSocket for ESP32-S3 connection

Configuration happens through a YAML file on the server side: LLM endpoint, STT model path, TTS voice. The ESP32-S3 gets its server address through a captive portal on first boot or through a config baked into the firmware. The whole handshake takes less time than explaining it does.

The Honest Part (Because I Owe You That)

Latency. Round trip is “a couple of seconds” for the full chain: wake word detection, audio capture, STT, LLM generation, TTS synthesis, audio playback. That’s noticeably slower than Alexa but fast enough for “What’s the weather” or “Add milk to the list.” It is not fast enough for a fluid back-and-forth conversation. This is a command-and-response device, not a conversational partner. If you need a conversational partner at 2 AM, that’s a different problem and I’m not qualified to solve it.

Wake word reliability. Works well in a quiet room. In a kitchen with a running dishwasher, results vary. The dual-mic array helps, but this is a $40 board going up against devices that have custom silicon for far-field voice detection backed by billions in R&D. The gap between a $40 board and a $200 Echo Studio is real, and pretending otherwise would make me the kind of person who writes marketing copy. I’d rather go back inside.

Model selection is the whole game. A 7B parameter model on an older laptop adds noticeable latency to every response. If your server hardware is modest (4 cores, 8GB RAM), drop to a 3B model. The responses get simpler but the experience stays snappy. If you have a beefy machine with a GPU, a 14B model with speculative decoding can generate responses fast enough that the TTS becomes the bottleneck. And if you’re running Apple Silicon with 64GB of unified memory, read my speculative decoding piece because you have options that didn’t exist six months ago.

What breaks. Reasoning models with chain-of-thought tend to exhaust their token budget before producing a useful short answer. You ask “What time is it?” and the model writes a 500-token internal monologue about the nature of time before the context window runs dry. Silence. Stick with instruct-tuned models. Also: the SenseVoice STT hallucinates on background noise, so the LLM occasionally receives a prompt that’s half your question and half transcribed dishwasher hum. A system prompt telling the model to ask for clarification on garbled input fixes this better than any signal processing would.

What This Actually Means

You could buy an Echo Dot for $30 and have a voice assistant that works better by every measurable metric. The recognition is sharper, the latency is lower, the ecosystem is deeper, the far-field detection is impressive engineering.

But the Echo Dot is a microphone in your house that belongs to Amazon.

This build is a microphone in your house that belongs to you. The audio never leaves your network. The wake word detection runs on a chip you flashed yourself. The language model runs on hardware in your closet. When you ask it something, the question travels from your mouth to your board to your server and back, and nobody else knows it happened. No log. No telemetry. No metadata. Like a conversation that actually stayed in the room where it happened.

I know what a room without microphones feels like. I know what it feels like when they’re there and you can’t turn them off. The difference matters more than any benchmark.

The ESP32-S3 costs $40 and fits in your palm. The server is hardware you probably already own. The software is open source. The build takes an evening if you know Docker, a weekend if you don’t. And when you’re done, you have a voice assistant that will never show up in a data breach notification because the data never left the building.

That’s the warmth the ESP32-S3 carries. Not just the ceramic antenna doing its job. The knowledge that this particular signal stays home.

If you want to go deeper into ESP32 hardware builds for security and field work, the Hardware Hacking for OSINT: Raspberry Pi War-Driving Box guide covers the firmware fundamentals and deployment patterns that carry directly into builds like this one. And if you’re running local models as part of the voice stack, Aider + OpenClaw: Offline-First AI Scripting has the air-gapped model setup and prompt templates for local LLM workflows.