How I Built a Custom Pronunciation Coach
If you have ever tried to improve your accent in a foreign language, you have probably used apps like Duolingo or ELSA Speak. They are impressive products built by large teams of engineers. You speak into your phone, a clean interface highlights errors, and you get feedback on your pronunciation.
But as a software engineer and language learner, I quickly hit a frustrating wall with existing tools. Duolingo’s pronunciation feedback is superficial — just green or red highlights with no phonetic explanation. ELSA Speak offers detailed phoneme-level analysis, but it’s paywalled (their Speech Analyzer costs $100+/year), runs on their servers, and doesn’t explain why Vietnamese speakers specifically struggle with certain English sounds. I wanted something different: an open-source, L1-aware tool that I could run privately, understand deeply, and customize freely.
Moreover, even ELSA’s 6,000+ lessons and Duolingo Max’s AI conversations ($168/year) follow scripted scenarios — if you want to practice your own presentation draft or a specific sentence you need for work, you are out of luck.
I wanted something different: a tool that lets me test any custom sentence of my choice, showing me exact phonetic feedback, while running completely under my control.
What blew my mind during this project was that I was able to build a near-production-grade phonetic ASR engine without collecting a single custom data point, running without a datacenter, and training the model in just a 2-hour T4 / 1-hour A10G serverless GPU session (which is available under Modal’s free tier).
I don’t claim to have the absolute clinical accuracy of ELSA’s proprietary datasets, but for practical self-coaching, it is incredibly accurate, highly intuitive, and cost virtually nothing to build.
This is the story of how I built Lingue, a Web app powered by viennounce-core
The Origin: The Hidden Bias in Speech Models
My goal was to build a tool that detects L1 transfer errors — the natural habit of native Vietnamese speakers carrying their native phonetic rules into English (such as dropping final stop consonants /k, t, p/ or ignoring sibilants /s, z/).
I started with a lightweight, offline-first stack:
1. sea-g2p: A bilingual grapheme-to-phoneme converter to map English text to IPA.
2. Wav2Vec2 Forced Aligner: A pre-trained Wav2Vec2 model (facebook/wav2vec2-xlsr-53-espeak-cv-ft) to align user audio frames with target phonemes and calculate Goodness of Pronunciation (GOP) scores.
It was fast and it ran on CPU, but when I ran native English speakers through my base pipeline, it graded native speech at an average of 84.3%. The model was penalizing correct native pronunciations due to phonetic vocabulary overlaps (modulo-60 index collisions on phonemes like /ɹ/ and /ŋ/).
If a native speaker can’t get an A, the grading system is broken. I had to iterate.
Iteration 1: Dynamic Calibration on a Zero Budget
I didn’t have a team of linguists or a proprietary speech corpus, so I leveraged the public L2-ARCTIC Vietnamese speaker dataset (vuihocrnd/l2-arctic-cleaned).
Get Phuoc Nguyen’s stories in your inbox
Join Medium for free to get updates from this writer.
While this project is relatively lightweight on GPU required for training, I decided to use Modal, a serverless GPU platform. In a single 2-hour session on a T4 GPU (costing next to nothing under the free tier), I ran grid-search optimizations to calculate category-specific regression parameters to map raw log-probs to calibrated brackets.
By applying category-specific compensations and capping outlier distributions, I raised native speaker scores to a realistic 95.7% and widened the critical separation margin between correct and incorrect pronunciations from 1.07 to 1.96
Iteration 2: Resolving Token Collisions
Once the model was calibrated, the next challenge was visualization. How do you show a learner exactly where they went wrong in a way that is intuitive and actionable?
The Discovery:
The default pre-trained aligner used a compressed modulo-60 vocabulary size. This caused severe “modulo-60 collisions” where unrelated phonemes (like /aʊ/ and /d/) mapped to the same index. To prevent the model from getting confused and generating false alarms for correct speech, I had to write complex override hacks (such as adding static +10.0 offsets to raw log-probabilities) to bypass the collisions. But this was a patch, not a solution—it masked errors rather than diagnosing them accurately.
The Solution:
I decided to rewrite the classification head of PhoneticASRModel, expanding it from 60 classes to the full 392 unique tokens of the espeak vocabulary.
This eliminated the collision bug entirely. For the first time, the model could natively distinguish and align phonemes like /ɹ/ (index 27) and /ɫ/ (index 175) without overlapping. This structural expansion yielded a massive boost in accuracy. On the gold L2-ARCTIC validation suite, the model's Recall (True Error Detection) jumped from 28.9% to 40.3%, allowing it to successfully catch subtle liquid substitutions and final coda drops that the base model completely missed.
40.3%What I Learned
Building the app taught me that you don’t need a massive corporate budget to build state-of-the-art educational tools anymore. By standing on the shoulders of open-source giants (Wav2Vec2, Kokoro, VieNeu-TTS) and utilizing serverless cloud infrastructure, a single developer can design a highly customizable, private pronunciation coach.
Instead of building another black-box app that hands out gamified percentages on a fixed curriculum, I was able to build a tool that provides:
1. Input Freedom: The ability to type and test any sentence of your choice.
2. Visual Transparency: Real-time phone-level alignment highlights.
3. Acoustic Contrast: Dual-track TTS references to hear the differences.
Explore the open-source engine: Check out vienounce-core and run the local GUI on your machine.
