<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Abhishek Pandey on Medium]]></title>
        <description><![CDATA[Stories by Abhishek Pandey on Medium]]></description>
        <link>https://medium.com/@abhi.pandey03?source=rss-f8075bb6503e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*zClK-nSDZdr-XN3tyxaZzw.jpeg</url>
            <title>Stories by Abhishek Pandey on Medium</title>
            <link>https://medium.com/@abhi.pandey03?source=rss-f8075bb6503e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 13:09:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@abhi.pandey03/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Barge-In and Full-Duplex: The Architecture That Makes Voice Agents Feel Human]]></title>
            <link>https://medium.com/@abhi.pandey03/barge-in-and-full-duplex-the-architecture-that-makes-voice-agents-feel-human-48ef1b7b7fa6?source=rss-f8075bb6503e------2</link>
            <guid isPermaLink="false">https://medium.com/p/48ef1b7b7fa6</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[conversational-ai]]></category>
            <category><![CDATA[voice-ai]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Abhishek Pandey]]></dc:creator>
            <pubDate>Sat, 06 Jun 2026 12:44:15 GMT</pubDate>
            <atom:updated>2026-06-06T12:44:15.176Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Why every component in the cascade must be designed to stop mid-sentence</em></p><p>The agent is mid-response, confirming an appointment: “Your appointment is scheduled for Thursday the 15th at …” and the user cuts in: “No, not that date, I want…”</p><p>What happens in the next 200–300 milliseconds determines whether the conversation recovers or falls apart. The system must detect that the user has started speaking, stop the TTS audio that is still playing, cancel the LLM generation that is still producing tokens from the previous turn, and re-enter listening mode to process the new input. If any of these steps is slow or missing, the agent talks over the user or worse, it continues processing the old response while the user has already moved on, and the conversation derails.</p><p>Voice assistants never had this problem. Bixby, Alexa, Siri; they were half-duplex by design. The user spoke, then the assistant spoke. No overlap, no interruption. The closest thing to barge-in was something like Alexa playing music and the user giving a command; the system needed to detect the wake word over its own audio output, but that is a far simpler problem than managing a live two-way conversation where either party can speak at any moment. Full-duplex is what makes voice agents feel like conversations instead of command interfaces. But it changes everything about how you build the pipeline.</p><p>In the previous posts, I covered the cascaded architecture and the gatekeeper: <a href="https://medium.com/@abhi.pandey03/vad-and-turn-detection-the-gate-that-shapes-the-entire-conversation-6e2db98e2258">VAD and turn detection</a>. This post is about what happens when the user does not wait for the gate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XrTuIJ_zD4D-xTEVevvsZw.png" /></figure><h3>1. Half-duplex vs. full-duplex: why this changes everything</h3><p>Voice assistants are half-duplex: strict turn-taking, one speaker at a time. The user pressed a button or waited for the wake word, spoke their command, and then waited for the assistant to respond. The pipeline was linear; listen → process → respond → listen. Clean and straightforward.</p><p>Voice agents need full-duplex: both parties can speak at any point. The agent must monitor the user’s audio even while it is speaking. It must be ready to yield at any moment. Listening never stops.</p><p>This is not a feature you can ignore or defer in a production setup. It is a design constraint that shapes the entire system from day one. In half-duplex, the pipeline processes one direction at a time. In full-duplex, the agent must simultaneously play TTS audio on the outgoing stream, monitor the incoming audio stream for user speech through VAD, and be ready to cancel everything in-flight, such as LLM generation, TTS playout, and pending tool calls, within a few milliseconds. Every module in the pipeline must be built with the assumption that it can be interrupted at any time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EURH9wJbWG4ZM5RdD_R2TA.png" /></figure><h3>2. Anatomy of a barge-in: what must happen in milliseconds</h3><p>When a user interrupts the agent mid-sentence, the system must execute a precise sequence. Each step has a latency budget, and the entire chain must complete in under 200–300ms. Here is what happens:</p><h3>Step 1: Detect user speech onset</h3><p>VAD flags speech on the incoming audio while TTS is playing on the outgoing channel. The challenge is distinguishing genuine user speech from three things that look similar: background noise, backchannels (“uh-huh”, “mm”, “ok”, coughing, etc.), and the agent’s own audio bleeding back through the user’s speaker. The VAD is running continuously on the incoming channel, even while the agent is speaking.</p><h3>Step 2: Classify the interruption</h3><p>Is this a barge-in: the user wants to take over, or a backchannel: the user is acknowledging, and the agent should continue? This decision must happen within milliseconds. Most production systems take a conservative approach: any sustained speech above a configured duration threshold is treated as barge-in. Short sounds below the threshold are ignored. It is an assumption that occasionally yields false positives while avoiding the worst outcome of talking over the user.</p><h3>Step 3: Stop TTS playout</h3><p>The TTS audio must be stopped as soon as the interruption is detected, and any audio chunks already queued for playback are dropped. Latency here matters; if TTS keeps playing for even 200–300ms after the user starts speaking, the overlap feels broken. The user perceives the agent as not listening. True streaming TTS makes this easier because audio is produced and played in small chunks. Batch-generated TTS that produces the entire utterance before playback is much harder to cancel mid-sentence.</p><h3>Step 4: Cancel LLM generation</h3><p>Any in-flight LLM generation must be aborted. With streaming responses, this means cancelling the HTTP stream or inference request; tokens already generated but not yet sent to TTS are discarded. This is where your LLM integration must be cancellation-aware. If you run speculative generation (starting a response before the user finishes), all those parallel requests need to be cancelled too. And if a tool call is in flight, you need to handle its result arriving late: either cancel it, or tag it to the interrupted turn and discard the response so stale data never enters the context.</p><h3>Step 5: Re-enter listening mode</h3><p>The pipeline resets to the listening state. ASR picks up the new user input from where the interruption started. Critically, the conversation context must be updated to reflect what happened: the agent’s previous response was interrupted, and the system must track how far the agent got before the interruption. This matters for the LLM’s next turn; if the agent was listing three appointment slots and got cut off after the first one, the LLM needs to know the user only heard one option.</p><p>The entire sequence that is detect, classify, stop, cancel, re-listen- must be completed in under a few milliseconds. Anything slower and the user perceives the agent as unresponsive or talking over them.</p><h3>Why does this have to be event-driven?</h3><p>A sequential pipeline, where each module calls the next directly, cannot handle mid-flight cancellation cleanly. You need a component to be able to signal “stop everything” the moment a barge-in is detected. This is why interrupt-aware pipelines are event-driven: components publish and subscribe to events like <em>agent_listening</em>, <em>user_interruption_detected,</em> <em>llm_cancelled</em>, and <em>tts_flushed</em>. The decoupling is what makes cancellation possible without tangling every module together.</p><h3>3. Production failure modes</h3><h3>The agent talks over the user</h3><p>The user hears the agent continuing to speak after they have started talking. This is the most frustrating failure mode from the user’s perspective; it signals that the agent is not listening. In production, this often traces back to one of two causes: TTS buffering, where too much audio is queued ahead of playback, or barge-in detection failed to fire in time.</p><h3>Premature cancellation</h3><p>The agent stops mid-sentence on a false barge-in due to a noise spike, a backchannel, or an echo. The user did not intend to interrupt, but the agent went silent. Now there is an awkward pause. The user waits, the agent waits, and someone has to break the silence. This is less harmful than talking over the user, but still degrades the conversation quality, especially if it repeats, the user restarts, the agent restarts, and both keep colliding.</p><h3>Context corruption after interruption</h3><p>The LLM does not know what the agent managed to say before being interrupted. Two failure modes follow: it repeats information the user has already heard, or it skips information the user never received. Both are confusing. If the agent was listing medication instructions and got cut off after the second item, the LLM needs to know the user heard items one and two but not three and four. Without accurate tracking of the interruption point, the follow-up response is unreliable.</p><h3>Orphaned tool calls and API requests</h3><p>The LLM triggered a tool call or API request before the barge-in. The pipeline cancels the LLM generation, but the external request is already in flight. The API response arrives after the conversation has moved on. If the system naively processes this response, it injects stale data into a conversation that has shifted context. You need a mechanism to either cancel external requests on barge-in or tag them with a turn identifier and discard responses that arrive for a turn that is no longer active.</p><h3>Cascading state confusion</h3><p>Rapid successive interruptions; user interrupts, agent tries to respond, user interrupts again, can leave the pipeline in an inconsistent state. The LLM has a partial context from the first interruption, the TTS buffer may contain audio from the cancelled response, and the ASR is processing overlapping speech fragments. If the state machine is not designed for rapid transitions, these edge cases produce unpredictable behaviour. This is where robust state management becomes essential. Every possible transition: speaking → interrupted → listening → processing → speaking → interrupted again, must be handled explicitly.</p><h3>4. How frameworks handle it today</h3><p>Most voice agent frameworks have converged on similar architectural patterns for barge-in, though the implementation details and configurability vary.</p><p>The common pattern: VAD runs continuously on the incoming audio stream, even while the agent is speaking. When speech is detected above the barge-in threshold, the framework emits an interruption event. This event triggers a cascade of cancellations: the LLM generation is aborted, the TTS buffer is flushed, and the pipeline transitions to listening mode.</p><p>The key configuration knobs exposed to developers typically include:</p><ul><li><strong>Barge-in sensitivity threshold</strong>: the VAD confidence score required to identify speech vs. non-speech frames.</li><li><strong>Minimum speech duration</strong> (min_duration): how long speech must persist before it counts as an interruption.</li><li><strong>Minimum word count</strong> (min_words): how many words the user must say before it registers as a barge-in. Set this higher, say, 2 to 3 words, and brief sounds or single-word backchannels will not interrupt the agent. This is a simple, effective guard against false interruptions.</li><li><strong>Phase-based control</strong>: whether interruptions are allowed during specific parts of the response. Where the information the agent is delivering is critical, you can disable interruptions for that phase entirely.</li></ul><p>Beyond these threshold knobs, the frameworks are now shipping model-based interruption handling that tries to solve the backchannel problem directly. LiveKit’s <a href="https://livekit.com/blog/adaptive-interruption-handling">adaptive interruption</a> handling uses a model trained on real conversational audio to separate genuine barge-ins from backchanneling like “uh-huh” or “okay”. It runs after VAD detects incoming speech, and instead of stopping the agent the moment any speech appears, it analyses the audio to decide whether the agent should actually yield the turn. The important design choice: the decision is based on acoustic signals rather than waiting for a transcript, so it stays fast and avoids unnecessary interruptions. The common thread is that the naive “any speech above a threshold equals barge-in” rule is being replaced by a small dedicated model that asks a better question; not “is the user making sound?” but “does the user actually want the floor?”</p><p>The architectural lesson: frameworks give you a working baseline for barge-in handling, but the tuning and edge case management is where production engineering lives. The defaults are optimised only for common cases. Teams building production voice agents frequently end up customising the barge-in pipeline for their specific use case rather than relying on framework defaults.</p><h3>5. Where full-duplex is heading</h3><p>Everything in this post so far treats full-duplex as an engineering problem solved with a harness: VAD monitoring the incoming channel, a barge-in detector, explicit cancellation system built into every module. Another perspective is that this harness is merely a temporary solution. If a single model processes both audio channels simultaneously, interruptions and backchannels will no longer be seen as additional events to work around; instead, the model learns about these events naturally and generates responses accordingly.</p><p><a href="https://arxiv.org/html/2410.00037v2">Moshi</a>, from Kyutai, is the first real-time full-duplex spoken LLM. It models the user’s audio and the agent’s audio as parallel streams, allowing for overlaps, interruptions, and backchannels to emerge as simultaneous sequences in the token space instead of being detected by a separate module. There is no VAD, no turn detector, no cancellation cascade; the architecture removes the explicit turn boundary entirely. <a href="https://research.nvidia.com/labs/adlr/personaplex/">NVIDIA’s PersonaPlex</a> builds on Moshi’s architecture, adding control over the agent’s voice and role through prompting. Google and OpenAI have their own end-to-end voice models: <a href="https://blog.google/innovation-and-ai/technology/developers-tools/build-with-gemini-3-1-flash-live/">Gemini Live</a> and <a href="https://openai.com/index/advancing-voice-intelligence-with-new-models-in-the-api/">GPT Realtime</a>, respectively. Dedicated benchmarks such as <a href="https://arxiv.org/abs/2503.04721">Full-Duplex-Bench</a> have emerged to measure turn-taking, interruption, and backchannel handling under reproducible streaming protocols.</p><p>Another recent approach, called <a href="https://thinkingmachines.ai/blog/interaction-models/">interaction models</a>, comes from Thinking Machines. They argue that interactivity must scale in tandem with intelligence rather than being bolted on as an external framework, so that people can collaborate with AI in real time. In terms of architecture, they employ time-aligned micro-turns, where the model continuously interleaves 200ms segments of input and output. This approach ensures that overlap, silence, and interruption are preserved within its context instead of being compressed into standard alternating turns. Furthermore, they divide the system into two components: a real-time interaction model and an asynchronous background model dedicated to complex reasoning and tool execution. This separation aims to achieve a balance of responsiveness and depth without one process obstructing the other.</p><p>But for production systems today, cascaded pipelines with explicit cancellation remain the default, and the harness everyone is trying to eliminate is also what gives you control. You can tune every threshold, disable interruptions during critical information delivery, debug each failure mode in isolation, and swap any module independently. End-to-end full-duplex models prioritize naturalness over control, leading to drawbacks such as weaker controllability and context bloat.</p><p>The shift toward model-native full-duplex is real, but it will be gradual. The barge-in engineering in this post is not obsolete; it is what runs in production now, and it is the baseline these model-native approaches are being measured against.</p><h3>6. Built to be interrupted</h3><p>The ability to handle interruptions cleanly is not something you bolt onto a working voice agent. It is a design constraint that shapes every module in the cascade. The moment a barge-in is detected, TTS playout must stop immediately and ongoing LLM calls must be cancelled. The context manager must track what was said and what was cut off. And the event system must allow any component to trigger a pipeline-wide stop.</p><p>The models get the attention, but the cancellation logic, state management, and event orchestration are what make a voice agent feel like a real conversation. When a user interrupts and the agent responds naturally, stopping mid-sentence, listening, and picking up the new thread, this is not the model being smart, but the pipeline being well-engineered.</p><p><em>How does your pipeline handle barge-in: event-driven cancellation, or something else?</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48ef1b7b7fa6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[VAD and Turn Detection: The Gate That Shapes the Entire Conversation]]></title>
            <link>https://medium.com/@abhi.pandey03/vad-and-turn-detection-the-gate-that-shapes-the-entire-conversation-6e2db98e2258?source=rss-f8075bb6503e------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e2db98e2258</guid>
            <category><![CDATA[speech-recognition]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[conversational-ai]]></category>
            <category><![CDATA[voice-ai]]></category>
            <dc:creator><![CDATA[Abhishek Pandey]]></dc:creator>
            <pubDate>Sun, 03 May 2026 17:58:19 GMT</pubDate>
            <atom:updated>2026-05-10T17:50:38.793Z</atom:updated>
            <content:encoded><![CDATA[<p>Picture two calls running in production right now.</p><p>In the first, the user is trying to reschedule an appointment. They say, “I need to reschedule my appointment to next …. “ and pause, still thinking about the date. The agent jumps in and says, “Your appointment is confirmed.” A 200ms silence gap in the middle of a sentence looked like an end of turn.</p><p>In the second, the user is on speakerphone in a noisy room; TV in the background, people talking nearby. They finish their question and wait. The agent says nothing. The background noise keeps the energy above the Voice Activity Detection (VAD) silence threshold, so the system thinks the user is still speaking. The user says, “Hello?” Three seconds of dead air.</p><p>Both failures start at the same place: the first module in the pipeline. In the <a href="https://medium.com/@abhi.pandey03/from-voice-assistant-to-conversational-voice-ai-agent-what-really-changed-b5120024f678">previous post</a>, I introduced the cascaded architecture: VAD → ASR → LLM → TTS and said VAD deserves its own deep dive. This is that post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V3Hau5hOkELJa2byI7LnaA.png" /></figure><h3>1. What VAD actually solves and what it doesn’t</h3><p>Two problems are constantly conflated in this space: detecting speech presence and detecting turn completion. These are fundamentally different tasks.</p><p>Voice Activity Detection answers a simple question: &quot;Is someone speaking right now?&quot; It classifies audio frames as speech or non-speech. That is all it does.</p><p>Endpointing, also called turn detection or end of utterance (EOU), answers a harder question: &quot;Has the user finished their turn?&quot; This is what actually matters for a voice agent, because this decision is the trigger that tells the rest of the pipeline to start processing.</p><p>The distinction matters because a pause mid-sentence and a pause at the end of a sentence look identical to a system that only measures silence duration. A user saying &quot;I want to book an appointment for ... let me check ... next Tuesday&quot; has two long pauses that are not turn boundaries. A VAD system cannot tell the difference.</p><p>In practice, most production systems still use VAD for endpointing; they measure how long the user has been silent and trigger when that duration crosses a configured threshold. That threshold becomes a critical hyperparameter: set it too low, and you cut people off mid-thought, set it too high, and the agent feels unresponsive. There is no single value that works across all conditions and use cases. This is the core limitation of VAD-based endpointing, and the reason the field has moved toward dedicated turn detection models.</p><h3>2. How VAD works: the signal processing foundation</h3><p>I first worked on VAD during my master’s, when I encountered speech processing for the first time. We implemented these algorithms in MATLAB, working through the speech processing textbook. For this post, I revisited that same textbook; the fundamentals have not changed.</p><p>Before getting into why VAD breaks, it helps to understand what it actually looks at. A basic VAD system operates on short audio frames, typically 10 to 25 milliseconds, and makes a binary decision on each: speech or no speech. The features it uses to make that decision have evolved over decades, but the core ones remain consistent:</p><p><strong>Energy / Amplitude</strong></p><p>The simplest signal. Speech is louder than silence, so you set a threshold; if the amplitude exceeds it, flag the frame as speech. This works well in a quiet room. In real-world environments, such as during phone calls with background noise, using a speakerphone in a living room, or when a user is in a car, increased noise raises the energy floor, making the threshold unreliable. What looked like a clean decision boundary in a clean recording falls apart the moment real audio is introduced.</p><p><strong>Zero-Crossing Rate (ZCR)</strong></p><p>How often the audio signal crosses the x-axis within a frame. Speech has a characteristic pattern here. Voiced sounds, where the vocal cords vibrate at a regular rate, like the /a/ in “father” or /ee/ in “see,” produce lower zero-crossing rates. Unvoiced sounds, produced by turbulent airflow without vocal cord vibration, such as the /sh/ in “she” or /f/ in “five”, produce higher rates. The same applies to a noisy signal, where the ZCR is high. ZCR is useful as a secondary feature alongside energy, but not reliable on its own.</p><p><strong>Spectral Features</strong></p><p>Speech energy concentrates in specific frequency bands, roughly 300Hz to 3400Hz for telephony, broader for wideband audio. Noise in general tends to spread more evenly across the frequency spectrum. Looking at how energy is distributed across frequency bands gives a much better signal than amplitude alone. This is what gives frequency-domain VAD approaches their edge over pure time-domain methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uENJQkr7nJQl0_SmqZMKCg.png" /></figure><p>The above figure provides a glimpse of these features calculated for one of the sample audio files for reference. In practice, a heuristic VAD chains these features together: compute energy and spectral features per frame → apply a threshold → track silence duration → trigger when silence exceeds a configured duration. You play with parameters such as minimum speech or silence duration and the threshold. But tuning these parameters for practical scenarios is tough. Every adjustment trades off premature cuts against indefinite holds, and no single configuration works across all conditions. The user in a quiet room needs different thresholds than the user on a noisy speakerphone, but you are serving both with the same system.</p><h3>3. VAD in production: what actually gets used</h3><h3>WebRTC VAD</h3><p>The industry standard for over a decade has been developed by Google as part of the WebRTC project. This uses hybrid approach of combining energy-based, statistical (Gaussian Mixture Model classifier), and frequency-based methods. It is everywhere; embedded in browsers, video conferencing, VoIP, telephony stacks, and many production systems. Its failure modes are well understood. It has a high false positive rate in noisy environments, and no adaptation capability to different speakers or acoustic conditions.</p><h3><a href="https://thegradient.pub/one-voice-detector-to-rule-them-all/">Silero VAD</a></h3><p>The current default is in most production voice agent stacks, a multi-head attention-based neural network, trained on over 6000 languages. The model size is 1.8MB and processes a 30ms audio chunk in about 1ms on a single CPU thread. It supports both 8kHz and 16kHz sampling rates, which matters for telephony, where <a href="https://en.wikipedia.org/wiki/G.711">G.711</a> provides 8kHz audio. Instead of a binary energy threshold, Silero outputs a speech probability per frame, and you configure a confidence threshold, typically 0.7, above which it flags speech. It ships under the MIT license with no vendor lock-in. You will find Silero VAD integrated into Pipecat, LiveKit, Deepgram, and most other voice agent frameworks.</p><h3><a href="https://huggingface.co/pyannote/voice-activity-detection">Pyannote.audio</a></h3><p>A neural audio framework that goes beyond pure VAD: it combines voice activity detection with speaker change detection, overlapped speech detection, and speaker embeddings. The standard choice for diarization-heavy use cases, where you need other modules along with VAD. It is heavier than what most real-time voice agents need for endpointing, but worth knowing about when your use case requires speaker separation alongside activity detection.</p><h3>4. Where VAD breaks: production failures</h3><p><strong>How does VAD power endpointing today?</strong></p><p>Before we talk about where VAD breaks, it is worth understanding how it is actually being used in production. Despite the availability of dedicated turn detection models, the majority of voice agent frameworks still rely on VAD-based endpointing as their default strategy.</p><p>The mechanism is straightforward. VAD classifies each incoming audio frame as speech or non-speech. Once speech is detected, the system enters an active listening state. When the VAD stops detecting speech, a silence counter starts. If that silence duration crosses a configured threshold, the system declares end-of-turn; the ASR buffer flushes its final transcript, and the downstream pipeline kicks off.</p><p>The entire endpointing decision comes down to a few important configuration parameters:</p><p><strong><em>activation_threshold</em></strong>: (typically 0.5–0.7) Confidence threshold used by the VAD model to decide speech vs non-speech. Higher values make the decision more conservative (fewer false positives but more false negatives); lower values make the detection more sensitive</p><p><strong><em>min_speech_duration</em></strong>: Minimum continuous speech length required to mark the start of a speech chunk. This helps the system to ignore very short bursts, clicks, or noises.</p><p><strong><em>min_silence_duration</em></strong>: Minimum continuous silence duration to mark speech ended. This is the core knob. Shorter values cut users off prematurely. Too long, and the agent feels sluggish due to increased delay in endpointing detection.</p><p>This is the default endpointing strategy in Pipecat, LiveKit, and most other voice agent frameworks. Along with the above configurable parameters, these frameworks add significant engineering guardrails and fallbacks to avoid premature responses and to keep the agent feeling responsive. It works reasonably well for simple, predictable conversations, short turns, clean audio, and cooperative users. The problems start when real-world conditions push beyond those assumptions.</p><h3>Premature triggering (truncation)</h3><p>The agent responds before the user has finished speaking due to an early endpointing trigger. The user is forced to rephrase, or worse, the agent acts on incomplete information. This is usually a parameter configuration problem, not an application logic problem. This issue is measurable via checking the rising clarification request rate in your call logs.</p><h3>Indefinite hold (never triggering)</h3><p>Background noise, speakerphone audio, or cross-talk keeps the energy above the silence boundary. The system never detects that the user has stopped speaking. The agent waits indefinitely. The call drops or the user hangs up. This is especially common in telephony environments where you have no control over the user’s acoustic conditions.</p><h3>Telephony audio mismatch</h3><p>G.711 8kHz compressed audio behaves differently from the 16kHz wideband audio that most models are trained on. VAD models can perform dramatically differently on telephony input. Silero VAD explicitly supports 8kHz; most other models do not, or perform worse at that sample rate.</p><h3>Backchannel misclassification</h3><p>The user says “uh-huh”, “hmm” or “right” as an acknowledgment while the agent is speaking. A VAD system with no linguistic context interprets this as the start of a new turn or as a turn boundary. The cascade follows: ASR transcribes a partial utterance, LLM generates a premature response, and the conversation drifts. This is one of the most common failure modes in production, and it is invisible in component-level testing.</p><p>In a cascaded system, every VAD failure is expensive.<strong> </strong>Wrong endpointing produces the incorrect ASR segment, which corrupts the LLM context, which generates a bad response. A single mistake at the gate propagates through every downstream module.</p><h3>5. From VAD to turn detection: why acoustic signals aren’t enough</h3><p>VAD answers “is there speech?” but a voice agent needs to answer “is the user done talking?” These questions require fundamentally different information.</p><p><strong>What acoustic signals can tell you?</strong></p><p>Pitch contour dropping at sentence endings. Energy envelope decreasing. Speaking rate deceleration. Falling intonation at phrase boundaries. Silence duration after specific speech patterns. The presence of filler words like “um”, “hmm”, or “so” signals that the speaker is still holding the floor.</p><p><strong>What acoustic signals cannot tell you?</strong></p><p>Whether the sentence is syntactically complete. Whether the user asked a question and is waiting for a response. Whether a discourse marker like “so anyway” signals a topic shift or “wait” signals the user correcting themselves mid-sentence. These are linguistic signals, and a system operating purely on audio features will never see them.</p><p>Neural turn detection models combine both acoustic and linguistic signals. Humans also do turn detection based on tone, prosody, pace of speech, semantics of language, and grammar. This is the architectural step up from VAD. You are no longer classifying audio frames as speech or silence. You are predicting conversational intent: is this a moment where the speaker expects a response?</p><h3>6. Turn detection models: what’s available now</h3><h3><a href="https://huggingface.co/pipecat-ai/smart-turn-v3">Pipecat SmartTurn v3</a></h3><p>Built on a Whisper-Tiny encoder with a linear classification layer added for turn prediction, 8M parameters with int8 quantization, resulting in an 8MB ONNX model file. Inference runs at 12.6ms on CPU (AWS c7a.2xlarge) and 3.3ms on GPU (NVIDIA L40S). Supports 23 languages. Open source under the BSD 2-clause license. A good baseline for teams that want turn detection without heavy infrastructure.</p><h3><a href="https://huggingface.co/livekit/turn-detector">LiveKit Turn-detector</a></h3><p>LiveKit’s latest turn detector is a 0.5B parameter model, knowledge-distilled from Qwen2.5–0.5B-Instruct. Interesting architectural choice: it is text-only. No audio features at all; it processes the text output from the speech-to-text system. It uses a sliding window of the last four conversation turns (both user and agent utterances) as context and predicts whether the current point is the end of utterance (EOU), outputting a confidence score. Supports 14 languages, CPU-based inference with per-turn latency in the range of 50–160ms, and 396MB on disk. This approach bets entirely on linguistic context; no prosody, no intonation, no acoustic cues. Intended to reduce false interruptions by the agent, but fundamentally limited by STT quality: if the transcript is wrong, the turn prediction is wrong, and there is no audio signal to fall back on.</p><h3><a href="https://blog.vogent.ai/posts/voturn-80m-state-of-the-art-turn-detection-for-voice-agents">VoTurn-80M (Vogent)</a></h3><p>Based on SmolLM2–135M, fine-tuned and reduced to 80M parameters. The key differentiator is that it is multimodal, combining acoustic and linguistic features: a Whisper encoder provides acoustic features while the ASR transcript is appended for linguistic context, with joint attention over both modalities. Inference at approximately 7ms on a T4 GPU in fp16. Trained on real conversational data: customer service, medical intake, travel booking, plus synthetic edge cases. Currently only supports English and is not optimised for CPU inference.</p><h3>Krisp Turn-Taking Model</h3><p>A smaller model at 6M parameters trained only on audio inputs, designed for CPU-only deployment (65MB). Krisp has published limited technical details, but what’s available suggests it’s targeting the edge deployment niche where GPU inference is not available.</p><h3>Comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FYTVcIKZ5lqbBITWVI9y8Q.png" /><figcaption><strong><em>A note on accuracy:</em></strong><em> Each model publishes performance metrics on its own evaluation dataset using different definitions, such as accuracy, TPR/TNR, F1, AUPRC, etc. These numbers are not directly comparable across models. Check each model’s documentation for its reported numbers and evaluation methodology.</em></figcaption></figure><p><strong>ASR-integrated endpointing: when turn detection lives inside the ASR</strong></p><p>A newer class of models sidesteps the VAD-plus-turn-detector stack entirely by folding end-of-utterance detection directly into the ASR model itself. <a href="https://huggingface.co/nvidia/parakeet_realtime_eou_120m-v1">NVIDIA’s Parakeet Realtime EOU</a> (120M parameters, FastConformer architecture) is the clearest example; it emits an EOU signal as a byproduct of the final transcript, with no separate turn detection stage in the pipeline.<a href="https://deepgram.com/learn/introducing-flux-conversational-speech-recognition"> Deepgram’s Flux</a> takes a similar approach, where they deliver the transcription with context-aware turn detection so the developer never has to implement gap detection themselves. <a href="https://www.assemblyai.com/blog/introducing-universal-streaming">AssemblyAI’s Universal-Streaming</a> goes further: its intelligent endpointing combines acoustic signals and semantic features along with silence detection to make end-of-turn decisions natively within the STT layer.</p><p>The latency argument for this architecture is real. One fewer model in the inference chain means one fewer round-trip, and the EOU signal arrives the moment the transcript finalizes rather than after an additional classification pass. But the tradeoff is tight coupling. You get turn detection that works well within the ASR provider’s own acoustic model and degrades the moment you swap out the transcription backend. For teams that don’t change ASR vendors often and want a simpler deployment surface, this is a compelling default. For anyone building a multi-provider or modular pipeline, the separation between transcription and turn logic is usually worth keeping. Put simply, joint ASR+EOU trades intelligence for simplicity, the right default for teams that want something working with minimal integration surface, but a ceiling you’ll hit quickly once your conversation patterns get complex: hesitations, multi-part answers, domain-specific cadences. That’s when a dedicated turn detection layer earns its place.</p><h3>7. Where turn detection is heading</h3><p>The direction is clear, even if the destination is not fully resolved.</p><p>Multimodal models combining acoustic and linguistic signals are outperforming single-modality approaches. Audio-only models miss linguistic context; text-only models miss prosodic cues. The winning architectures use both. But in practice, the models alone are not capable of solving many challenges. Voice agent frameworks like LiveKit and Pipecat have invested heavily in engineering around these models, such as combining Silero VAD with dedicated turn detection, adding configurable thresholds, fallback timers, and pipeline-level guardrails. Much of the progress in production turn detection today is happening at the orchestration layer, not just the model layer.</p><p>On the research side, <a href="https://arxiv.org/abs/2603.09180">DuplexCascade</a> proposes an approach that absorbs turn detection into the LLM itself; a VAD-free cascaded system where the language model emits control tokens like &lt;user_is_speaking&gt; and &lt;user_finish_speaking&gt; as part of its output stream. If the LLM can predict turn boundaries from its own context, a separate endpointing module becomes redundant.</p><p>Speech LLMs like Moshi take this further, handling full-duplex conversation natively, where both the user and the agent can speak simultaneously. How the agent handles interruptions, barge-in, and overlapping speech shapes the entire pipeline architecture and deserves its own detailed treatment. But for production systems today, cascaded pipelines with dedicated turn detection models remain the pragmatic choice. The accuracy and controllability gap is still significant.</p><p>Cross-lingual turn-taking, where pause norms, prosodic cues, and backchannel patterns differ across languages, remains largely unsolved and is worth watching as multilingual voice agents become more common.</p><h3>8. The gate matters more than you think</h3><p>In the previous <a href="https://medium.com/@abhi.pandey03/from-voice-assistant-to-conversational-voice-ai-agent-what-really-changed-b5120024f678">post</a>, the thesis was “the hard part isn’t the models.” VAD and turn detection are the clearest illustration of this. The endpointing decision is the first domino; get it right, and you give ASR clean segments, the LLM sees correct context, and TTS responds at the right moment. Get it wrong, and the error propagates through every downstream module.</p><p>The tooling has improved dramatically, from energy thresholds to neural turn detection models that combine acoustic and linguistic signals, running in single-digit milliseconds. But the integration is still the hard part. How the endpointing decision feeds into your ASR buffer, how your pipeline handles cancellation on barge-in, how you tune thresholds for the specific acoustic conditions your users actually experience; these are engineering problems, not model problems.</p><p><em>What’s your endpointing strategy: heuristic VAD, neural turn detection, or something else entirely?</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e2db98e2258" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Voice Assistant to Conversational Voice AI Agent: What Really Changed]]></title>
            <link>https://medium.com/@abhi.pandey03/from-voice-assistant-to-conversational-voice-ai-agent-what-really-changed-b5120024f678?source=rss-f8075bb6503e------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5120024f678</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[conversational-ai]]></category>
            <category><![CDATA[voice-ai]]></category>
            <category><![CDATA[speech-recognition]]></category>
            <dc:creator><![CDATA[Abhishek Pandey]]></dc:creator>
            <pubDate>Sat, 04 Apr 2026 17:24:10 GMT</pubDate>
            <atom:updated>2026-05-10T17:49:20.921Z</atom:updated>
            <content:encoded><![CDATA[<p>In 2016, I was building acoustic models and language models for Samsung’s Bixby - a voice assistant designed to respond to a user’s command and take appropriate action. A user would say, “Set a timer for 10 minutes,” and we had roughly 2-3 seconds to respond and act. Now, a decade later, I am designing a system that conducts end-to-end conversations lasting up to an hour; responding in under 1 second, handling interruptions, recovering from misunderstandings (mostly caused by Automatic Speech Recognition (ASR) mis-recognitions), and navigating open-ended dialogue naturally. Same words: Voice AI - completely different problem.</p><p>Why the jump? Large Language Models (LLMs). A decade ago, voice assistants (Siri, Alexa, Google Assistant, Bixby, etc.) worked because the problem was simple and narrow, recognise a command, execute it, done. You could ship that with a good intent classifier and a Natural Language Understanding (NLU) pipeline. The introduction of LLMs shifted the voice modality away from simple, command-driven interactions. Now you could actually hold a conversation, handle ambiguity, recover when things go sideways and operate without a script. That changed what people expected from voice, and it changed what we had to build.</p><p>What does this actually enable? A voice AI agent can handle customer support, follow up on insurance claims, conduct surveys or promotional calls, or manage appointment scheduling across thousands of users simultaneously. Beyond telephony, voice agents are becoming the interface at kiosks in airports and hotels where multilingual support matters, and they are also enabling accessibility for users who cannot interact with screens at all. Use cases are endless, but the pattern remains the same: a natural conversation with a clear goal, where the cost, availability, or physical limits of human interaction create the bottleneck. What separates these from the traditional IVR menus and chatbots we have tolerated for decades is that these agents actually listen; they hold context, handle the unexpected, and sound natural enough that many users do not realise they are speaking to a machine. But the difference between a compelling demo and a system managing thousands of conversations every day is significant. Every one of those conversations occurs under real pressure, such as millisecond latency limits, unfamiliar accents, lossy and noisy audio, and users who interrupt or change their minds. This post is about the architecture that makes this work reliably.</p><p>What follows is a look at the production system behind a modern conversational voice AI agent; the architecture I work with daily. It is a cascade of four modules: Voice Activity Detection (VAD), ASR, LLM, and Text-to-Speech (TTS). Each module deserves its own deep dive. Where it helps, I will draw a line from how we solved these problems in the voice assistant era to how they work and break differently today.</p><h3>1. Voice assistant vs. voice agent - what actually changed</h3><h4>1.1 The fundamental difference in the interaction model</h4><p>At Bixby, we knew exactly what commands to expect from the user. Voice assistants are built around pre-defined intents and match the spoken input to one of the known intents, execute the action, and done. The conversation ends there. A voice agent has no fixed intent list. The user can ask anything, go off-script, and the agent must follow, while still steering toward a goal. The interaction is expected to feel like talking to another human being, not navigating a menu.</p><p>Voice assistants were also stateless per utterance. Each command was independent, and the model had no memory of what came before. A voice agent is fundamentally different here. Because an LLM is the reasoning engine, the entire conversation history is maintained as context. Every turn matters for the next response. Lose the context, and you lose the conversation.</p><h4>1.2 The latency reality check</h4><p>With a voice assistant, high latency is forgivable. The user issues a command and waits for the task to complete: “set an alarm for 5 AM” is done when the alarm is set, and a 2-second confirmation is acceptable. In a live voice conversation, that same 2-second pause feels like the call has dropped, and this is what happens in production.</p><p>What actually happens is a domino effect: the agent pauses, the user gets anxious, they start repeating themselves, which triggers the endpointing logic prematurely, which produces a partial or incorrect transcription, which then propagates downstream. A latency problem becomes a conversation failure.</p><p>With an end-to-end target of 600–800ms, the latency budget is shared across all four modules simultaneously. As a rough guide: VAD needs around 50–80ms, ASR around 150–200ms, LLM first token around 300–500ms, and TTS first audio around 100–150ms. These numbers shift based on your use case, model architecture, and inference provider, but the constraint is fixed. Every module is under pressure at the same time, and there is no slack to borrow from one to give to another.</p><h4>1.3 New failure modes that didn’t exist before</h4><p>Voice assistants handled interruption gracefully because they didn’t need to. The assistant listened to a complete utterance, then responded — the user naturally waited. A voice agent requires full-duplex conversation: both parties can speak at any point, and the agent must detect when the user has started talking and stop mid-sentence if needed. This is not just a problem to handle at the speech level, but it shapes the entire architecture, from how audio is captured to how downstream pipeline cancellation is handled.</p><p>Noisy environments introduce a separate failure mode. A voice assistant on a device can lean on hardware noise suppression and push-to-talk. A telephony agent receives whatever audio arrives over the network. Background noise can hold the endpointing logic in an open state, causing the agent to wait indefinitely for a silence boundary that never comes.</p><p>ASR misrecognition is the failure mode with the largest impact. In a voice assistant, a bad transcription means the user repeats the command; clean slate. In a voice agent, an ASR error mid-conversation corrupts the dialogue context. The LLM receives the wrong information, generates a response based on it, and the conversation drifts. Recovery requires the agent to detect the confusion and correct course without restarting and which is a design problem as much as a modeling one.</p><p>Finally, telephony audio is completely different from device microphone audio. Narrow-band 8kHz audio is heavily compressed by codecs like G.711. A model trained on wideband 16kHz microphone data behaves differently, sometimes dramatically, when it receives telephony input. This mismatch between training conditions and deployment conditions is one of the most common sources of production degradation that benchmarks never reveal.</p><h4>1.4 From WER to call success</h4><p>Voice assistant quality was measured by component metrics such as Word Error Rate (WER) for ASR and Mean Opinion Score (MOS) for TTS. Optimise each module in isolation and you are done. Voice agent quality lives in outcomes. The question is no longer “did the model get the words right?” but “did the conversation achieve its goal?”</p><p>The shift matters because a system that looks good on component metrics can still fail in production. A low-WER ASR can still fail the call if the LLM loses context at a particular turn and gives or collects incorrect information. The outcome is a failure regardless of what the transcription accuracy report shows. Conversely, an agent with imperfect ASR can succeed if its dialogue design gracefully handles misrecognition.</p><p>This is still an evolving area; there is no universally agreed evaluation framework for voice agent quality. But the direction is clear: measure the cascade end-to-end, not each module in isolation. The evaluation framework that measures this end-to-end is a deep topic in its own right.</p><h3><strong>2. The cascaded architecture - how a modern agent is built</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VwRXDLedGjauF01AxhN20Q.png" /></figure><p>The diagram above shows the essential components of a cascaded voice AI agent pipeline. What it does not show is that each module’s output is the next module’s input and its errors. This is the fundamental characteristic of a cascaded system: mistakes compound downstream, and a failure at any stage propagates through every stage that follows.</p><p>The reason most production voice agents use a cascaded architecture is simple: text-based LLMs are the best reasoning engines we have, and they do not understand audio. So you build a pipeline around them, convert speech to text, let the LLM think, convert text back to speech. Each module is a specialist. Here is what each one does and why it is harder than it looks.</p><h4>2.1 VAD: the gatekeeper</h4><p>After capturing the user’s voice, the first decision the system must make is: has the user finished speaking? This is called endpointing or turn detection, and Voice Activity Detection is one of the simpler ways to approach it. VAD-based systems mostly work on heuristics such as energy thresholds, silence duration, pitch patterns, and making them robust across a wide range of scenarios is genuinely challenging. Get it wrong in one direction and the agent starts responding while the user is still speaking. Get it wrong in the other, and it waits indefinitely for a silence boundary that never arrives.</p><p>More sophisticated systems train a dedicated turn detection model that considers both the acoustic nature of speech (energy envelope, pitch drop at sentence endings, silence duration) and the linguistic nature of conversation, such as whether the utterance sounds syntactically complete. This is a significant step up in complexity from simple VAD, and the trade-offs here deserve a dedicated deep dive.</p><h4>2.2 ASR: speech to text, while you are still speaking</h4><p>ASR is where audio becomes text, and in a voice agent, everything downstream depends on what it produces. In a real-time agent, this must happen in streaming fashion; partial transcriptions are fed to the downstream pipeline as they arrive, which reduces end-to-end latency significantly.</p><p>The trade-off is accuracy. Streaming ASR models generally perform worse than offline models like <a href="https://openai.com/index/whisper/">Whisper</a> because of a fundamental architectural difference: an offline model is non-causal; it needs to know the end of your sentence before deciding how to transcribe the beginning, giving it full context during inference. A streaming model must transcribe as audio arrives, with no access to what comes next. This is a classic performance versus latency trade-off, but for a real-time voice agent, you cannot afford a latency overrun, so you live with it and engineer around the accuracy gap.</p><p>The gap between benchmark accuracy and real-world performance is where the interesting problems live. These problems cannot be solved by model architecture alone; they require engineering solutions, post-processing, and the kind of grunt work that never appears in a paper. Named entity recognition for non-native words, accent robustness, and far-field degradation under real acoustic conditions each demanded its own solutions, independent of whatever the latest architecture could offer.</p><h4>2.3 LLM: the reasoning brain</h4><p>The LLM is where the agent actually thinks. It takes the transcribed text, the full conversation history, your system prompt with business logic, and decides what to say or do next. In practice, a big chunk of the engineering here is not about the model itself, but rather it is about what you feed it and how you manage context. Long conversations push against context window limits. You end up making real trade-offs about what to retain, what to summarise, and what to drop, and those decisions directly affect both cost and response quality.</p><p>Choosing which LLM to use is less about benchmarks and more about latency. Two numbers matter most: TTFT (Time to First Token) - how quickly the model starts generating, and TPS (Tokens Per Second) - how fast the rest follows. A model can score well on both and still be wrong for your use case if the inference provider adds overhead, or if your prompt complexity pushes TTFT past your latency budget. I have seen teams pick a model based on accuracy benchmarks and then spend weeks trying to make it fast enough.</p><h4>2.4 TTS: the voice of the agent</h4><p>TTS is what the user actually hears and they judge the entire system by it. Everything upstream can work perfectly, but if the voice sounds robotic or the pacing is off, the user’s impression of the agent drops immediately. Prosody is harder than it looks from the outside. Pitch contours, duration modeling, and the boundaries between synthesis units are where naturalness breaks down and where listeners notice, even when they cannot articulate why. Users judge conversational quality by how the agent sounds, not by what accuracy metrics it achieved internally.</p><p>Equally important is speed. The first audio must start playing before the LLM has finished generating its full response, which means streaming TTS is not optional; it is a hard requirement. The key metric here is TTFB (Time to First Byte): the time from receiving text input to producing the first byte of audio output. A high TTFB breaks the perceived flow of conversation regardless of how good everything upstream was.</p><h4>2.5 Why cascaded works and where it falls short</h4><p>Integrating well-understood modules is where most production failures happen. How mis-recognized texts by ASR are consumed by the LLM, how LLM generation latency affects TTS buffer behavior, and these interfaces are where assumptions made in isolation break down. The cascaded approach works in production because it is practical: you control every component independently and can build strong baselines by combining best-in-class models. But optimizing each module in isolation does not optimize the system. The interfaces are what separate a polished agent from a frustrating one.</p><p>There are also things a cascaded system simply cannot do. The moment ASR converts audio to text, you lose everything that is not words, the hesitation, the frustration, the sarcasm in someone’s voice. The LLM never sees any of it. Also, running these complex models sequentially introduces compounding latency. And errors cascade: a single ASR misrecognition can corrupt both the LLM’s response and the synthesised speech that follows.</p><p>This is why Speech LLMs are gaining traction; models like <a href="https://arxiv.org/abs/2410.00037">Moshi</a> that process audio directly without the text bottleneck, though they still lag on performance compared to cascaded systems (a gap worth exploring separately).</p><h3>3. What does “this agent works” actually mean?</h3><p>You can have green dashboards across every module and still have a broken agent. Optimising WER for ASR, MOS for TTS, and response accuracy for the LLM may not give the full picture of overall system health. A simple ASR mistake can confuse the LLM about context and produce an incorrect response but at the same time, the LLM has the capability to ignore minor ASR mistakes and self-heal from them. These metrics in isolation do not tell you whether the system is actually working.</p><p>This is why outcomes matter. If the agent’s goal is appointment scheduling and it collects the correct information and books the desired slot, the goal is achieved even if there were minor ASR errors along the way. The evaluation unit shifts from module accuracy to conversation success. Beyond task outcomes, you should also track the quality of the experience itself, whether users are getting frustrated, repeating themselves, or dropping off. The evaluation framework that measures end-to-end is a deep topic of its own.</p><h3>4. The hard part isn’t the models</h3><p>Voice AI did not just get better over the last decade; it became a different problem altogether. The shift from command-based systems to real-time, open-ended conversation changed every assumption about latency, robustness, and what success even means. The cascaded pipeline assembles these components into a system, and it is the system as a whole that defines voice agent quality, not any individual module.</p><p>Each of these modules has its own depth - VAD, ASR, LLM, and TTS each deserves a proper treatment. But before diving deeper, I want to hear from you:</p><p><em>What’s the hardest module in your voice stack right now?</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5120024f678" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>