<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anjana Edirisinghe on Medium]]></title>
        <description><![CDATA[Stories by Anjana Edirisinghe on Medium]]></description>
        <link>https://medium.com/@anjanaape?source=rss-ec4c8f28f45c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*DLVjdC2pTkUbbX91</url>
            <title>Stories by Anjana Edirisinghe on Medium</title>
            <link>https://medium.com/@anjanaape?source=rss-ec4c8f28f45c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 11:57:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@anjanaape/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Architecting a Multi-Tenant Voice Agent Pipeline as a Cloud Service: A Technical Deep Dive]]></title>
            <link>https://medium.com/@anjanaape/architecting-a-multi-tenant-voice-agent-pipeline-as-a-cloud-service-a-technical-deep-dive-12dd61cf703a?source=rss-ec4c8f28f45c------2</link>
            <guid isPermaLink="false">https://medium.com/p/12dd61cf703a</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[livekit]]></category>
            <category><![CDATA[ai-voice-agent]]></category>
            <category><![CDATA[integration]]></category>
            <dc:creator><![CDATA[Anjana Edirisinghe]]></dc:creator>
            <pubDate>Sun, 02 Aug 2026 11:33:06 GMT</pubDate>
            <atom:updated>2026-08-02T11:37:23.780Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zt0AKBLAzzIAZB8w51w8XA.png" /></figure><p>Developing a single-user local voice agent is straightforward. However, architecting a multi-tenant cloud platform that processes real-time audio streams simultaneously demand robust systems engineering, particularly when targeting sub-800ms latency, seamless interruption management, and zero data leakage.</p><p>This deep dive strips away the high-level hype and pulls back the curtain on the actual production architecture. We’ll cover our journey from clunky request-response middleware to an event-driven WebRTC pipeline, how we scale a stateful audio infrastructure on Kubernetes without dropping calls, and the OAuth2 token-exchange mechanics that keep it all secure. No wrappers, no fluff, just raw system design for low-latency voice infrastructure.</p><h3><strong>1.</strong> <strong>Introduction</strong></h3><h4><strong>1.1 The Vision: Voice Agents as a Scalable Cloud Service</strong></h4><p>Imagine you’ve spent months building, debugging, and perfecting a robust text-based AI agent. It’s connected to your databases, handles customer queries perfectly, and runs flawlessly inside an integration ecosystem like the WSO2 platform. But then the product requirement changes overnight: <em>“Our users don’t want to type anymore. They want to talk to it. In real-time.”</em></p><p>Traditionally, this request would trigger collective groans across the engineering team. Converting a text agent into a voice agent usually means ripping up your existing codebase. You suddenly have to figure out how to ingest raw audio bytes, handle flaky network connections, manage complex Speech-to-Text (STT) and Text-to-Speech (TTS) pipelines, and somehow stitch them together without adding seconds of lag.</p><p>Our vision was to completely eliminate this friction.</p><p>We set out to build a multi-tenant cloud service that acts as an intelligent “voice layer” on top of existing infrastructure. The ultimate goal? To allow developers to transform their already-deployed text agents into fully conversational voice agents using a simple, few-step wizard configuration with zero additional coding required.</p><p>By abstracting away, the complex engineering of real-time audio transport, orchestration, and model syncing into a scalable cloud service, we wanted to make adding a voice interface as simple as flipping a digital switch. Instead of worrying about audio buffers and packet loss, engineers can focus on what they do best: building the core logic.</p><p>But as we quickly found out, making something look that simple on the outside requires solving some incredibly complex infrastructure puzzles on the inside. Let’s look at the strict engineering boundaries we had to play by to make this a reality.</p><h4><strong>1.2 The “Holy Trinity” of Real-Time Voice Engineering</strong></h4><p>When we type into a chat interface like ChatGPT, we don’t mind waiting two or three seconds for a response. We watch the tokens animate onto the screen, and our brains accept the delay. Voice is entirely different. In human conversation, the average gap between speakers is roughly 200 milliseconds. If a voice agent takes two seconds to respond, the silence feels awkward, broken, and completely robotic. To make our “no-code wizard” voice layer feel like talking to a real human, we had to conquer what we call the “Holy Trinity” of Real-Time Voice Engineering: the tight coupling of Speech-to-Text (STT), the Large Language Model (LLM), and Text-to-Speech (TTS), all operating under a brutal time budget.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-3RRXCVLRX9vnBtPxaJkyg.png" /></figure><p>Here is how each piece of the trinity stacks up, along with the extra real-time features required to make them work in harmony:</p><p><strong>1. The Speech-to-Text (STT) Layer: Instant Transcription</strong></p><p>The clock starts ticking the millisecond the user finishes speaking. The STT engine’s job is to take raw, incoming audio streams and translate them into text immediately. If this layer introduces 1s of latency just trying to guess what the user said, the rest of the pipeline is already dead in the water. It needs to be incredibly fast, highly accurate, and capable of handling diverse accents and background noise.</p><p><strong>2. The Intelligence Layer: Time-to-First-Token (TTFT)</strong></p><p>Once the text is ready, it’s sent to the LLM. For voice, we don’t care how long it takes the LLM to finish writing its entire paragraph response; we only care about Time-to-First Token (TTFT). How fast can the model generate the very first word? We have to stream these tokens out asynchronously, word by word, so the next layer can start processing them immediately.</p><p><strong>3. The Text-to-Speech (TTS) Layer: Adaptive Audio Generation</strong></p><p>The final piece takes the text stream from the LLM and turns it back into a natural human voice. Traditional TTS systems require a full sentence before they can generate audio (so they can get the inflection right). In real-time engineering, we can’t afford that luxury. The TTS engine must stream chunks of audio bytes on the fly, balancing human-like warmth with lightning-fast generation.</p><p>The Secret Sauce: Turn-Handling and Barge-In</p><p>Nailing the Holy Trinity is only half the battle. In real-life conversations, people don’t behave like walkie-talkies. We don’t say “Over” when we’re done speaking. This introduces two massive engineering challenges:</p><p>Smart Turn-Handling &amp; Voice Activity Detection (VAD): How does the system know the difference between a user pausing to catch their breath versus actually finishing their thought? If the VAD is too aggressive, the agent will rudely cut the user off mid-sentence. If it’s too slow, the lag introduces an uncomfortable silence.</p><p>Barge-In (Interruption Management): What happens if the agent is continuously speaking, and the user interrupts to say, “No, wait, stop”? In a naive setup, the agent will keep talking over the user until its pre-generated audio buffer finishes playing. True barge-in requires the system to listen while it speaks. The millisecond the user interrupts, the pipeline must instantly halt the LLM generation, clear the audio playback buffer on the client’s device, and pivot to listening mode. When you add up the budget, our entire pipeline has an absolute maximum threshold of 800 milliseconds end-to-end to execute this entire loop. If any of these component&#39;s lag, the illusion vanishes.</p><h3><strong>2 The Evolution of Voice Pipeline</strong></h3><p>Architecting a voice agent platform is a classic case of <em>“you don’t know what you don’t know.”</em> When you first set out, it looks deceptively simple: you just need to glue an audio recorder, an AI model, and a speaker component together. But as we scaled our vision of turning static text bots into dynamic voice agents, we hit structural brick walls at every iteration.</p><p>We didn’t arrive at our final, ultra-low-latency architecture overnight. It was a journey of trial, error, and massive bottlenecks across three distinct evolutionary phases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*0ArAe6aZNbm6f82nCp6orA.png" /></figure><h4><strong>2.1 The Cloud Model Wrapper Approach</strong></h4><p>Our starting point relied heavily on existing enterprise integration patterns and managed cloud APIs. The concept was straightforward: leverage traditional request-response middleware to act as a proxy between the user and foundational AI services. We built custom-coded wrappers that orchestrated the ingestion of audio, packaged it into standard payloads, and shipped it off to proprietary cloud APIs for transcription and generation.</p><p>While this allowed for fast initial prototyping and highly accurate model responses, it completely fell apart under the weight of real-time requirements. Traditional integration buses and HTTP/REST frameworks are inherently synchronous and blocking. Forcing them to handle persistent, bidirectional WebSockets or chunked HTTP audio data caused massive thread-pool exhaustion. The network overhead of jumping across multiple third-party cloud boundaries meant our latency hovered in the multi-second territory making natural human conversation impossible.</p><p>For a deep dive into how we initially attempted to wire these real-time flows using enterprise integrations, you can read the first part of this journey here:<a href="https://medium.com/@anjanaape/building-real-time-voice-agents-with-wso2-integrator-part-i-3d62ceaf074a"> Building Real-Time Voice Agents: Part I</a>.</p><h4><strong>2.2 Local Stack Approach</strong></h4><p>Frustrated by cloud latencies, third-party pricing, and privacy constraints, we pivoted to an entirely on-premises, open-source stack. The goal here was absolute control over the hardware and data. We swapped proprietary APIs for cutting-edge open-source alternatives: running optimized local variants of Whisper for Speech-to-Text (STT), lightweight Large Language Models via local inference engines, and custom open-source Text-to-Speech (TTS) models.</p><p>This phase taught us the reality of running AI “on the metal.” While we successfully eliminated external network transit time, we traded it for massive computational bottlenecks. Running unquantized speech and language models locally demands extreme CPU and VRAM performance. In low-resource environments, the pipeline choked; if the system memory spiked, the audio stream immediately started lagging or dropping packets. We realized that writing custom asynchronous event loops to manually shuffle audio buffers between models is an operational nightmare to maintain, let alone scale as a cloud service.</p><p>I detailed our exact technical configurations, local model selections, and the raw performance tradeoffs of this self-hosted approach in a dedicated deep dive:<a href="https://medium.com/@anjanaape/on-premises-voice-agents-speech-processing-with-open-source-models-e882103f048d"> On-Premises Voice Agents: Speech Processing with Open-Source Models</a>.</p><h4><strong>2.3 Event Driven WebRTC Framework Approach</strong></h4><p>The fundamental lesson of our first two iterations was clear: stop treating real-time audio like standard web data. We needed a system designed from the ground up for low-latency media transport, network jitter resilience, and instant event handling. That is what led us to our final architectural shift: integrating a WebRTC Selective Forwarding Unit (SFU) mesh topology, powered by LiveKit.</p><p>What is LiveKit, Livekit is a complete, distributed framework designed to bridge the gap between frontend clients and AI-native applications. At its core, LiveKit operates as a highly scalable Selective Forwarding Unit (SFU) written in Go. Instead of forcing client devices to connect directly to one another (which chokes bandwidth), everyone establishes a single, ultra-low-latency UDP connection directly to the LiveKit server.</p><p>By shifting our core engine to an event-driven WebRTC media server, we fundamentally transformed how audio moves through our platform. WebRTC operates over UDP, dropping the heavy TCP packet-retransmission overhead that plagues WebSockets under poor network conditions. The SFU treats our AI components (the STT, LLM, and TTS) as native “participants” inside a virtual media room.</p><p>Instead of writing complex, error-prone code to bridge audio streams, we can now hook directly into low-level media room events (track_published, track_subscribed). The moment a user speaks, the audio is routed with microsecond latency directly to the agent worker; the moment the user interrupts, a native data channel signal halts the system instantly. This paradigm shifts finally gave us the foundation we needed to build a blazing-fast, predictable pipeline capable of transitioning from a single-user prototype into a secured, multi-tenant cloud SaaS.</p><p>You can explore our transition into this setup and see how we mapped out the core components here:<a href="https://medium.com/@anjanaape/production-grade-voice-agents-orchestrating-pipelines-with-livekit-part-iii-3233d3f5cdd3"> Production-Grade Voice Agents: Orchestrating Pipelines with LiveKit</a>.</p><h3>3. The Finalized Architecture: Anatomy of the Voice Pipeline</h3><p>With LiveKit serving as our robust real-time media foundation, we engineered a pragmatic end-to-end voice-processing loop. Instead of over-engineering the media path with custom data-interception layers, we leveraged LiveKit’s native plugin ecosystem for raw audio conversions and built a custom LiveKit LLM adapter to interface directly with our backend core integration layer.</p><h4>3.1 Ingestion &amp; Real-Time Voice Activity Detection (VAD) Tuning</h4><p>The journey begins when raw audio is captured from the client’s microphone and streamed over WebRTC directly to the LiveKit SFU. LiveKit treats our backend voice agent worker as a first-class participant in the virtual room, allowing it to monitor incoming audio tracks seamlessly.</p><p>To detect when a user starts and stops speaking without incurring network latency or cloud processing costs, the pipeline handles turn detection entirely in-process. We utilize the industry-standard Silero VAD model, running locally within our Python agent process layout via the livekit-local-inference package. Because the model runs natively on the host machine, it evaluates incoming audio frames with sub-millisecond efficiency.</p><p>When the local VAD registers that the user has stopped speaking, it establishes the speech boundary and signals the session to proceed with processing the captured audio segment. Tuning this local VAD requires balancing specific temporal criteria:</p><p>● Speech Threshold: Set to a precise 0.45 probability(default) to capture initial word fragments without triggering on microphone static.</p><p>● Min Silence Duration: Tuned dynamically around 400ms to 500ms(default). If a user pauses longer than this threshold, the local VAD classifies the turn as complete and instantly allows the downstream components to proceed.</p><h4>3.2 The Intelligence Layer: The Custom LiveKit LLM Adapter</h4><p>Once speech boundaries are determined, the framework orchestrates the handoff between transcription, reasoning, and voice generation. LiveKit offers massive architectural flexibility here. Developers using the platform can leverage built-in LiveKit Inference cloud models out of the box or employ a BYOK (Bring Your Own Keys) strategy to connect straight to dedicated provider endpoints like Deepgram or ElevenLabs.</p><p>In our deployment, the audio track is fed directly into the preconfigured STT plugin within AgentSession, which handles the heavy lifting of transcribing audio into text.</p><p>To bridge this text into core integration layer, we engineered a custom LiveKit LLM adapter using LiveKit’s open protocol specifications. While LiveKit provides pre-designed ecosystem adapters for tools like LangChain, our underlying enterprise microservices fabric relies on Ballerina, a cloud-native language optimized for highly concurrent network integrations. Our custom adapter bypasses standard framework limitations and establishes a streamlined, persistent WebSocket communication channel directly to the Ballerina runtime engine.</p><p>The data orchestration follows a clean, turn-based sequence:</p><p>● The Payload Handoff: The fully transcribed text from the STT plugin is captured by our custom adapter and sent across the socket to the Ballerina server.</p><p>● The Processing Execution: On the Ballerina side, the service invokes a synchronous call: agent.run(userText, sessionId). Because the pipeline was built when native streaming library support for this layout was constrained on the integration engine side, Ballerina processes the turn as a full text block.</p><p>● The Sentence-Chunked Delivery: Once the full text response is processed and flushed back across the WebSocket to the Python server, the custom adapter utilizes a localized _split_sentences utility. It breaks the full block down into clean sentence strings and drips them sequentially into the TTS plugin, faking an incremental delivery model to achieve smooth, natural sentence-by-sentence audio playback for the user.</p><p><strong>Session Persistence &amp; History Handling</strong></p><p>A major architectural advantage of this setup is how context is managed. We designed the conversational history layer within our custom LLM adapter to be completely optional. Because we maintain a persistent WebSocket connection, the Ballerina integration layer handles all conversational context and state memory natively.</p><p>To track and isolate these sessions across the multi-tenant architecture without manual overhead, had to heavily utilize the Session ID. If a user experiences a brief network blip and the WebSocket disconnects for a second, the adapter automatically initiates a reconnection handshake using the same sessionId. The Ballerina engine instantly maps the re-established socket back to its active memory state, ensuring the conversation continues seamlessly without losing context or requiring the client to re-transmit past conversational history.</p><h4>3.3 Handling Interruption: Asyncio Task Cancellation</h4><p>Real-world human conversations are highly dynamic, and users will inevitably cut off an agent mid-sentence. LiveKit’s framework treats the agent as a first-class participant in the room, meaning it natively monitors incoming audio tracks while simultaneously publishing the agent’s outbound voice track. The moment our local Silero VAD registers user speech over the playback, an interruption sequence triggers instantly.</p><p>Rather than managing complex custom network signals to stop downstream threads, our interruption handling relies on Python’s native asynchronous execution model:</p><p>● <strong>Task Eviction via Asyncio:</strong> When LiveKit detects a barge-in event, it leverages its built-in task management to trigger an internal cancellation. This immediately raises an asyncio.CancelledError right inside our custom adapter’s streaming execution loop (IntegratorAgentStream._run ()).</p><p>● <strong>Socket Cleanup:</strong> The exception immediately breaks the streaming playback loop on the Python side and closes the local WebSocket connection, ensuring that any remaining audio chunks queued for the TTS engine are instantly dumped.</p><p>It is worth noting an important architectural trade-off here: because the pipeline is split across an isolated network boundary, this cancellation happens entirely within the user-facing Python server. The backend Ballerina server’s agent.run() process operates independently and will continue executing its core thread until completion, unaware of the local audio cancellation. This decoupled design allows the user-facing audio layer to feel incredibly responsive and reset in under 100 milliseconds, giving the illusion of instant human reflexes while keeping the underlying integration protocol dead simple.</p><h3>4. Transforming the Pipeline into a Multi-Tenant SaaS</h3><p>Building a voice pipeline is one thing, but establishing the foundations of a multi-tenant cloud service requires clean resource management and reliable session isolation. Instead of managing complex customer databases or rewriting core logic inside our media layer, our platform bridges the real-time audio network to external integration instances using environment-level routing and operating system-level runtime isolation.</p><h4>4.1 Platform Requirements: Environment-Driven Routing and Configuration</h4><p>To maintain a clean architectural separation between media transport and business logic, our pipeline treats voice strictly as an abstract interface layer. The system doesn’t manage prompts, user identities, or backend configurations internally; it maps incoming real-time audio streams to target integrations through a centralized configuration layout.</p><p>Our platform’s behavior is driven entirely by static environment variables evaluated at process startup. This configuration manages the global constraints of the media session:</p><p><strong>Audio Provider Mapping:</strong> Defining external provider identifiers (such as “deepgram”) to determine which vendor SDKs handle media conversion.</p><p><strong>Voice Dynamics Tuning:</strong> Locking in strict processing boundaries via activation thresholds for the local voice activity detection engine.</p><p><strong>Upstream Destination:</strong> Directing the text translation layer to the exact integration node.</p><p>When an agent session initiates, the platform automatically ties the communication layers together using the native session identifier generated by LiveKit’s room manager. This session ID is mirrored directly as the handshake identifier for the internal WebSocket connection to the integration backend, forging a dedicated, traceable data tunnel for that specific user interaction without requiring the media server to maintain persistent state records.</p><h4>4.2 Resource Isolation: Subprocess Forking and Session Boundaries</h4><p>In a cloud architecture, preventing the “noisy neighbor” effect and ensuring system reliability are critical challenges. If one user’s voice stream encounters an unhandled runtime exception, it must not degrade the audio processing quality of another active room.</p><p>Our codebase enforces this isolation boundary at the operating system level, using LiveKit’s native process management strategy rather than overhead-heavy infrastructure layer provisioning:</p><p><strong>1. Per-Room Subprocess Forking</strong></p><p>Rather than executing all active audio loops inside a single monolithic daemon thread, our architecture employs a distributed runner pattern configured to fork a completely independent OS-level subprocess to host each specific room’s agent session.</p><p>Because each room runs inside its own isolated process space, the platform achieves complete fault containment. While routine conversational interruptions (like user barge-in) are handled gracefully using asynchronous cancellation loops, unexpected runtime faults, such as an unhandled exception within an underlying media plugin or a script crash are securely confined.</p><p>A critical error in one room’s process cannot propagate into or destabilize a sibling process. If a subprocess terminates unexpectedly, the central runner detects it and cleans up that specific room’s resources, leaving parallel sessions on the host machine completely untouched.</p><p><strong>2. Logical Resource Management</strong></p><p>It is worth noting that this forking model provides logical process isolation rather than strict physical resource caps (like groups or memory limits). To prevent the system from getting overwhelmed by too many simultaneous calls drawing from the shared host memory pool, the central runner tracks a global load threshold. If the overall system load crosses a specific capacity limit, the runner throttles new job admissions to protect the active, running sessions from performance degradation.</p><p><strong>3. Upstream Context and Logic Separation</strong></p><p>Because our code focuses purely on media handling and delegates reasoning to the target integration backend, data segregation is naturally inherited from the underlying integration framework.</p><p>The custom LiveKit LLM adapter simply relays the transcribed text across the session-tracked WebSocket. The upstream integration engine captures the text, identifies the unique socket connection, routes it to its internal context maps, evaluates any necessary tool or database calls within its own secure infrastructure, and returns the finished text block.</p><p>By bounding real-time media manipulation to isolated, room-locked subprocesses and routing intelligence out to dedicated integration runtimes, the architecture achieves stable enterprise-grade isolation with remarkably low system overhead.</p><h3>5. Cloud-Native Infrastructure: Component Deconstruction &amp; Containerization</h3><p>Scaling a real-time voice pipeline into a cloud platform requires an infrastructure layout that balances simple code management with runtime adaptability. Instead of managing a complex fleet of fragmented microservices for each distinct voice element, the architecture relies on a highly modular core service engine that dynamically delegates logic at both the process level and the cloud edge.</p><h4>5.1 Deconstructing the Pipeline into Composable, Swappable Components</h4><p>From a deployment perspective, the voice agent platform operates as a unified, containerized engine. Rather than distributing speech-to-text, text-to-speech, and orchestration across distinct network hops, which would devastate sub-800ms latency budget, the system spins up an isolated, single-component runner. Runtime isolation is achieved by executing one distinct process per active livekit room.</p><p>The modularity of the pipeline is engineered internally through structural code separation:</p><p>● <strong>Abstract Media Interfaces:</strong> Speech-to-text (STT) and text-to-speech (TTS) engines are managed via provider-specific factories. The exact vendor is resolved purely through environmental configuration at startup by Livekit. This enables seamless switching of voice or transcription providers without introducing complex code branching or architectural re-engineering.</p><p>● <strong>The Intelligence Proxy:</strong> The language model abstraction layer does not query model endpoints directly. Instead, our custom llm adapter treats the upstream integration platform as a black-box reasoning engine, proxying the transcribed prompt over a persistent WebSocket session and handling text chunks as they return.</p><p>The true decomposition boundary sits at the network edge. The core service container runs a lightweight utility server as an asynchronous background thread. This background layer handles base HTTP traffic, such as minting Livekit WebRTC access tokens for incoming clients and exposing health check probes, completely independent of the active media pipeline’s lifecycle.</p><p>Furthermore, to prevent tight coupling between systems, the application uses an infrastructure abstraction engine. Rather than hardcoding the upstream destination, the platform resolves the integration target ID dynamically at runtime via an internal integration platform API. This ensures that the voice agent engine, the underlying media server, and the integration backend remain independently deployable systems that discover each other dynamically on the cloud fabric.</p><h4>5.2 CI/CD and Artifact Management: Automated Registry Pipelines</h4><p>To support a continuous delivery model without sacrificing platform stability, changes are governed by automated integration and delivery pipelines. This automation ensures strict compliance with security, version control, and performance standards before code ever hits production.</p><p>The software lifecycle operates across two continuous automated stages:</p><ol><li><strong>The Gatekeeper Stage:</strong> Pull requests are validated by automated format verification policies that enforce conventional semantic commit formatting. Simultaneously, software composition analysis (SCA) routines scan the codebase for license vulnerabilities and dependency compliance, gating code long before it reaches the main repository branch.</li><li><strong>The Automated Release Stage:</strong> Upon successful merge, a release automation script parses the Git history since the last release tag. It dynamically calculates semantic version increments based on the commit prefixes, automatically rolling out patch, minor, or major version updates without manual intervention.</li></ol><p>The underlying container image uses a multi-stage compilation pattern designed to enforce early validation checks. The builder stage installs required system dependencies and pre-loads the local voice activity detection (VAD) models. This sanity check ensures that native binary dependencies load successfully during the build phase, completely preventing broken native runtimes from reaching the final production environment.</p><p>The minimized production image is configured to run as a restricted, non-privileged user to limit the security exploit surface. Before the image is allowed to push to the cloud container registry, a vulnerability scanner inspects the layers for flaws. Once cleared, the container is pushed to the secure registry in lockstep with an auto-generated changelog and an official code release tag.</p><h4>5.3 Configuration &amp; Secret Management Across Multi-Tenant Environments</h4><p>To keep our image assets entirely generic, no configuration state or environment variables are baked into the container layers. The engine relies entirely on a fail-fast environmental loading pattern at process initialization.</p><p>Required credentials, such as media server keys and speech service tokens, are evaluated immediately at startup. If any keys are missing, the runtime throws a configuration exception and terminates instantly, preventing the engine from accepting active user calls in a broken or unauthenticated state. During local development, these keys reside in local environment files that are explicitly ignored by configuration tracking and stripped from the Docker build context, ensuring credentials never leak into the image layers.</p><p>By managing the deployment surface declaratively, exposing only the necessary public token endpoints while keeping everything else internal, the cloud-native infrastructure maintains a minimal footprint while adapting fluidly to whatever integration environment it is provisioned to support.</p><h3>6. Performance Monitoring &amp; Pipeline Configuration for Low-Resource Environments</h3><p>A real-time voice agent lives and dies by its latency budget. Every millisecond spent transcribing, routing, thinking, and synthesizing speech is dead air. What makes this class of system unusual is that it is simultaneously latency-sensitive and resource-constrained: it must hold a live, bidirectional audio stream open for the duration of a conversation, yet run comfortably on commodity, GPU-less containers.</p><p>The architecture makes this achievable by drawing a clear line between heavy and light work. The expensive computation, converting audio to text and text back to audio, is offloaded to specialized external cloud vendors. Locally, the process only executes small, purpose-built models for detecting speech boundaries. By building our pipeline on top of the LiveKit Agents framework and the livekit-local-inference package, the system minimizes initialization overhead and completely bypasses any reliance on a GPU accelerator.</p><h4>6.1 Profiling the Pipeline: Identifying CPU and Memory Bottlenecks</h4><p>In a hosted-speech pipeline, local profiling yields counterintuitive results. CPU cycles are not consumed by heavy neural processing; instead, they are dominated by continuous, always-on Voice Activity Detection (VAD) scoring incoming audio frames.</p><p>To scale this predictably under multi-tenant concurrency, the platform enforces specific resource boundaries:</p><p>● <strong>Bounded Audio Buffers:</strong> Rolling audio memory arrays managed by the LiveKit input queue are assigned a strict ceiling, preventing long speech turns from inflating per-call memory consumption.</p><h4>6.2 Low-Resource Configuration: Model Tiers, Buffering, and Chunking</h4><p>Because the pipeline behavior is externalized via configuration rather than hardcoded logic, tuning for constrained environments requires adjusting environmental parameters rather than rewriting code:</p><p>● <strong>Model Tier Selection:</strong> Since speech tasks are delegated externally, model size translates directly into vendor tier selection. Swapping our Deepgram or ElevenLabs settings to a faster, lighter tier collapses response latency at the edge without requiring a system rebuild.</p><p>● <strong>Temporal Buffering Vitals:</strong> Fine-tuning the confidence thresholds and silence duration windows inside the local Silero VAD plugin allows the pipeline to balance responsiveness against stability. Shorter silence thresholds accelerate turnaround time but require precise tuning to prevent premature cutting off.</p><p>● <strong>Sentence-Chunked Streaming:</strong> Perceived latency is minimized by avoiding full-block synthesis. The application intercept layer splits text responses on structural sentence boundaries and drips them to the speech generation engine sequentially, allowing audio playback to begin immediately on the user’s device while downstream sentences are still being compiled.</p><h4>6.3 Real-Time Telemetry: Observation Layers</h4><ol><li><strong>The Media-Transport Layer</strong></li></ol><p>Audio streams ride a real-time communication stack built on top of the <strong>LiveKit SFU</strong> server, designed for lossy, jittery networks. This underlying WebRTC transport layer inherently tracks connection-quality metrics like packet loss and audio jitter per participant. While the core Python application focuses on media transit rather than aggregating these stats directly, they remain readily accessible at the transport boundary to be surfaced by external infrastructure monitors.</p><p>To ensure connection health across long turns, the application runs automated, periodic keepalive ping checks over its persistent WebSocket link to the integration backend, dropping and cleanly resetting the connection if the far end times out.</p><p><strong>2. The Application Layer</strong></p><p>The application layer focuses heavily on logging foundational execution state changes as they happen. The system captures and surfaces core lifecycle markers: room connections, participant handshakes, sessions becoming ready, and explicit mid-stream user barge-in cancellations handled via native asyncio task cancellations.</p><p>When an interruption occurs, the application cleanly detects the turn cancellation, logs the state change, and triggers the cleanup sequence to clear the playback.</p><h3>7. Scaling Real-Time Audio Infrastructure on Kubernetes (k8s)</h3><p>Interactive voice applications present unique infrastructure challenges compared to traditional, stateless web APIs. When scaling web services that rely on REST or GraphQL, requests are short-lived and self-contained; if a container pod is terminated during a downscale event or a rolling update, the next incoming HTTP request is simply routed to an alternate replica with zero user impact.</p><p>Real-time voice agents break this stateless paradigm entirely. A voice session is a highly stateful, long-lived workflow that demands specialized orchestrations when moving to a production Kubernetes environment.</p><h4>7.1 The Stateful Challenge: Why Traditional Scaling Fails for Continuous Audio</h4><p>When a user connects to a voice assistant, they establish a continuous, low-latency WebRTC media stream that must maintain its connection state uninterrupted.</p><p>A running voice session relies on a dense layer of dynamic, in-memory state:</p><p>● <strong>Persistent Media Tracks:</strong> Continuous UDP or TCP packet streams carrying raw audio data that cannot be dropped or reroutes mid-conversation.</p><p>● <strong>Voice Activity Detection (VAD) History:</strong> In-process VAD engines require a sliding window buffer history of incoming audio frames to accurately compute silence and speech boundaries.</p><p>● <strong>Decoupled Communications:</strong> The system maintains a distinct layer of concurrent network links, linking the agent to the external WSO2 integration agent over a persistent WebSocket, alongside streaming connections managed under the hood by specialized transcription and speech synthesis services.</p><p>Because of this deep statefulness, traditional autoscaling mechanisms collapse. If a standard Kubernetes Horizontal Pod Autoscaler (HPA) terminates an application pod based on an overall drop in average CPU usage, any active conversations hosted on that pod are severed instantly. This results in dropped calls and broken user interactions.</p><h4>7.2 Our Scaling Mechanism: Custom Metrics and Smart Autoscaling Strategies</h4><p>To achieve zero-downtime scaling without dropping in-flight calls, the platform decouples the media layer from standard service routing. The system utilizes a centralized job dispatch architecture combined with automated worker lifecycle management powered by the LiveKit Agents framework.</p><p>Traffic orchestration is handled cleanly across dedicated operational boundaries:</p><ol><li><strong>The Token Gateway:</strong> A lightweight token server runs as a background thread inside the agent’s parent process, exposing a REST endpoint. The client requests an access token that securely embeds the target room configuration and agent properties. Because the execution model forks child processes that only execute the specific session logic, the parent process binds to this port exactly once on the container host.</li><li><strong>The LiveKit Orchestrator:</strong> A centralized media server manages virtual room memberships and maintains an active job dispatch queue. The moment a client connects to a room, the orchestrator dispatches a session request job to an available agent worker container.</li><li><strong>The Agent Worker Pool:</strong> Each worker container process connects back to the orchestrator to advertise its availability. To ensure immediate responsiveness, the worker properties are tuned for real-time performance limits:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*iuRibgAtmyTe8sEthWMWrw.png" /></figure><h4>Zero-Downtime Downscaling (The Draining Protocol)</h4><p>When the Kubernetes HPA triggers a downscale event or initiates a rolling deployment update, the cluster sends a standard SIGTERM signal to the pod. The parent agent process intercepts this signal and immediately places the pod into a DRAINING state.</p><p>The pod immediately stops accepting new room assignments from the orchestrator. However, any active conversations already running inside the pre-forked subprocesses continue completely uninterrupted. The container remains alive for up to the 30-minute timeout limit, allowing users to finish their calls naturally. The moment the last active room session terminates, the parent process exits cleanly, allowing Kubernetes to delete the pod with zero user downtime.</p><h4>7.3 Mitigating the “Noisy Neighbor” Effect via Kubernetes Resource Limits</h4><p>Real-time audio pipelines are extraordinarily sensitive to compute delays. Any thread starvation or CPU throttling directly manifests as audible stuttering, broken turn transitions, or robotic speech synthesis.</p><p><strong>The Threat of Thread Oversubscription</strong></p><p>Under the hood, local machine learning libraries (such as the native Silero VAD running inside the local inference framework) utilize native C++ tensor and matrix math libraries like OpenMP. By default, these numeric libraries inspect the underlying physical host node rather than the container boundaries. If a pod is deployed on a bare-metal Kubernetes node with 32 or 64 cores, the native library will attempt to spawn a matching thread pool of equal size.</p><p>If multiple pods run concurrently on that same host node, they will quickly trigger heavy CPU oversubscription. Dozens of competing execution threads will thrash the OS scheduler, introducing massive context-switching overhead and latency spikes that degrade the audio quality of every pod on the machine.</p><p><strong>The Solution: Constrained Execution Boundaries</strong></p><p>To prevent this “noisy neighbor” collision, the container engine explicitly restricts the native execution pool by hardcoding environmental math library constraints inside the image:</p><blockquote>Dockerfile</blockquote><blockquote>ENV OMP_NUM_THREADS=2</blockquote><p>By explicitly restricting OpenMP and the underlying native ONNX execution layers to a maximum of two execution threads, the pod’s internal thread usage is perfectly aligned with typical cloud CPU allocations. This eliminates scheduler thrashing and yields predictable execution speeds for the local speech boundary models.</p><p>Furthermore, because each room session is forked into its own distinct OS-level subprocess via the WorkerType.ROOM runner achieves robust fault containment. Because each room runs within its own isolated process space with its own instance of the Python interpreter, Python’s Global Interpreter Lock (GIL) cannot block neighboring sessions. A sudden computational bottleneck or a routine cleanup operation in one room’s event loop will leave all parallel voice calls running on the container completely unaffected.</p><h3>8. Authentication &amp; Authorization: Securing Client-Side WebRTC Connections</h3><p>Building a low-latency WebRTC media pipeline is only half the battle; the other half is ensuring that media transport endpoints cannot be abused, hijacked, or leveraged to breach tenant boundaries.</p><p>When deploying real-time voice applications to the cloud, system architects frequently conflate two distinct authentication layers:</p><ol><li>The Infrastructure Access Token (OAuth2 Bearer Token): Authenticates trusted platform components (such as Backend service) to internal API gateways and control-plane endpoints.</li><li>The Media Room Join Token (Signed WebRTC JWT): Authenticates an individual, untrusted client browser session directly to a specific WebRTC media room.</li></ol><p>The fundamental architectural constraint governing this design is simple: master signing keys can never be exposed to the client browser. However, the browser must present a valid, signed credential to negotiate a WebRTC connection.</p><p>To bridge this gap, our control plane issues short-lived, narrowly scoped, single-room tokens on demand. Furthermore, while the control plane (token issuance) flows through our API Gateway, the media plane (WebRTC signaling over Secure WebSockets and encrypted audio transport) connects directly from the browser to the Selective Forwarding Unit (SFU). The edge gateway cannot intercept or terminate an established media stream, making strict token scopes and tight Time-To-Live (TTL) constraints critical at the moment of issuance.</p><h4>8.1 Isolated User-Wise Token Servers for Dynamic LiveKit Room Join Tokens</h4><p>To sign WebRTC room join tokens without shipping root API credentials to the browser, we run a dedicated token service that operates directly alongside the voice worker process.</p><p><strong>Co-Locating the Token Server</strong></p><p>Rather than deploying a separate microservice, the token engine runs as an asynchronous background thread inside the main voice agent process. This co-located design ensures that the token server and worker scale together, share the same underlying secret distribution path, and maintain synchronized configurations.</p><p>Because the main worker framework takes over the process’s primary execution loop, the REST server initiates its own event loop inside a daemon thread, binding to its internal control port to serve token requests and orchestrator health probes.</p><p>In a multi-tenant runner topology, where each active call forks an independent OS-level subprocess, the token server is bound once in the parent process during startup before process forking occurs, preventing port binding collisions across child workers.</p><p><strong>Endpoint Contract &amp; Least-Privilege Scoping</strong></p><p>The token server exposes a lightweight REST endpoint that accepts the target room name, participant identity, and session parameters. When invoked, it returns both the signed JWT and the WebRTC target endpoint URL sourced dynamically from server environment settings.</p><p>To enforce strict security boundary limits, tokens are generated adhering strictly to the principle of least privilege:</p><ul><li>Explicit Resource Restrictions: Tokens are restricted exclusively to microphone audio publishing. Even if a compromised client attempts to negotiate video or screen-sharing tracks, the media SFU drops the stream negotiation at the protocol layer.</li><li>Single-Room Scoping: Each token is strictly bound to a single room identifier, preventing a user from hopping across room boundaries.</li><li>Administrative Isolation: Administrative capabilities (such as force-muting other participants, room creation, or recording controls) are omitted entirely from client-facing tokens.</li></ul><p><strong>Bounding the Blast Radius with Short TTLs</strong></p><p>By default, standard media SDKs issue tokens valid for extended durations (often up to 6 hours). In a real-time voice setting, an unexpired token sitting in browser memory represents an unnecessary exposure window. We override standard framework defaults downward to 1 hour, with options to shrink it further per session.</p><p>Note: The token TTL governs the join window, not active call duration. Once a WebRTC peer connection completes its initial signaling handshake, the expiration of the underlying JWT does not terminate the ongoing media session. The TTL merely restricts the timeframe during which a leaked token can be reused to initiate a new connection.</p><p><strong>Statistically Side-Effect Free Token Issuance</strong></p><p>In early iterations, requesting a token triggered a separate API call to explicitly instruct the media server to dispatch an agent worker to the room. This approach created significant architectural friction:</p><ul><li>/getToken was no longer a pure read operation; it mutated infrastructure state.</li><li>Abandoned user clicks triggered orphan agent worker processes that consumed idle memory.</li><li>Additional network round-trips introduced unnecessary connection latency.</li></ul><p>To resolve this, we embedded the agent dispatch rule directly inside the signed token payload itself. Now, token issuance operates as a pure, stateless cryptographic function. The moment the client submits the token to join the room, the SFU inspects the signed room configuration, instantiates the room, and dispatches the configured agent worker in a single atomic step.</p><p><strong>The Downstream Isolation Chain</strong></p><p>The room name embedded inside the signed JWT serves as the foundational root key for our entire multi-tenant isolation model:</p><p>Whoever controls the room identifier controls the context boundary. Consequently, ensuring that clients cannot arbitrarily select, or hijack room identifiers requires robust boundary defense at the edge.</p><h4>8.2 Configuring Secure Endpoint Delivery: Providing Token Access via the Edge</h4><p>Exposing raw internal application ports directly to the open internet is a critical vulnerability. To protect our internal runtime while ensuring seamless client connectivity, we route token issuance through an API Gateway edge.</p><p><strong>Declarative Edge Configuration</strong></p><p>To publish our token endpoint safely, we define declarative component descriptors alongside our application container definitions. This configuration exposes internal ports via external managed HTTPS endpoints, ensuring TLS termination at the cloud edge.</p><p>By attaching an explicit API specification (such as an OpenAPI schema manifest) rather than using a catch-all proxy route, the gateway enforces a strict route allow-list. Only paths explicitly documented in the spec (e.g., token minting and health probes) are accepted at the edge; all other arbitrary request paths are rejected before reaching our application container.</p><p>Transport Separation: Control Plane vs. Media Plane</p><p>It is critical to distinguish between REST control channels and WebRTC media paths at the gateway edge:</p><ul><li>The Token Control Path (REST): Managed by the API Gateway. It enforces rate limits, TLS termination, and credential validation for token issuance.</li><li>The WebRTC Media Path (UDP / Direct Media WebSockets): Bypasses the API Gateway entirely. High-frequency signaling and encrypted audio packets stream directly between the client browser and the media infrastructure.</li></ul><p>Because WebRTC media bypasses the edge, traditional API gateway controls cannot inspect or terminate active streams. The gateway’s security value is concentrated entirely at the moment of token generation.</p><p><strong>CORS Strategy and Container Hardening</strong></p><p>Cross-Origin Resource Sharing (CORS) is configured dynamically via environment parameters to restrict which web domains can request media tokens. While token endpoints are wrapped with strict CORS controls, internal health check endpoints are explicitly isolated, they are reserved strictly for cluster orchestrator liveness and readiness probes.</p><p>At the container level, strict security boundaries are enforced:</p><ul><li>Single Port Exposure: Only the designated HTTP control port is exposed; the underlying agent worker initiates outbound connections to the media server and exposes no public listening interfaces.</li><li>Non-Root Execution: Containers execute under unprivileged, non-root user spaces.</li><li>Secret Isolation: Environment files containing sensitive keys are excluded from container build layers and injected dynamically at runtime via secret management volumes.</li></ul><h4>8.3 Hardening the Token Endpoint: Implementing OAuth2 Exchange Mechanics</h4><p>The Vulnerability of Client-Side Exchanges</p><p>In initial iterations, the browser executed an OAuth2 client-credentials grant directly against the API Gateway token endpoint. However, in single-page web applications, build-time variables are statically inlined into the frontend bundle.</p><p>Shipping client secrets to a public browser context violates core OAuth2 specification standards. Any user inspecting browser network traffic or source bundles can extract the application’s consumer key and secret, granting them the ability to bypass our application interface and mint media tokens arbitrarily.</p><p><strong>The Backend Solution</strong></p><p>To resolve this exposure, we recommend clients to use a Backend service layer. The browser makes an unauthenticated request to our Backend, which handles the upstream OAuth2 machine-to-machine exchange securely within trusted server memory.</p><p>The Backend manages upstream authorization tokens using an in-memory cache featuring double-checked locking and monotonic clock evaluations to prevent performance degradation and token refresh storms:</p><ol><li>Double-Checked Locking: The initial cache read operates lock-free for ultra-fast response times. On a cache miss, an execution lock is acquired, followed by an immediate re-check. Under high concurrency, multiple simultaneous cache misses execute exactly one upstream token exchange, eliminating thundering-herd spikes against the identity provider.</li><li>Monotonic Clock Tracking: Systems use monotonic time sources rather than system wall-clock time. System clock jumps (such as NTP time adjustments) cannot cause valid tokens to prematurely expire or expired tokens to be served accidentally.</li><li>Expiry Skew Window: Access tokens are refreshed early (e.g., 60 seconds prior to their actual expiration timestamp), preventing edge-case race conditions on in-flight requests.</li><li>Bounded Retry &amp; Response Sanitization: If the cached gateway bearer token is revoked prematurely, the Backend executes a single invalidation-and-retry sequence. If the upstream service fails, internal gateway stack traces and error details are stripped, returning a sanitized error response to the client.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=12dd61cf703a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Production-Grade Voice Agents: Orchestrating Pipelines with LiveKit (Part III)]]></title>
            <link>https://medium.com/@anjanaape/production-grade-voice-agents-orchestrating-pipelines-with-livekit-part-iii-3233d3f5cdd3?source=rss-ec4c8f28f45c------2</link>
            <guid isPermaLink="false">https://medium.com/p/3233d3f5cdd3</guid>
            <category><![CDATA[livekit]]></category>
            <category><![CDATA[voice-ai]]></category>
            <category><![CDATA[wso2]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[integration]]></category>
            <dc:creator><![CDATA[Anjana Edirisinghe]]></dc:creator>
            <pubDate>Wed, 29 Apr 2026 16:29:23 GMT</pubDate>
            <atom:updated>2026-05-01T19:17:52.126Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vugenys90RQTvxHiL7NdfQ.png" /></figure><h3>The Road So Far: Evolution of the Voice Agent Architecture</h3><p>Before diving into the orchestration of our finalized system, let’s quickly recap the milestones from our previous entries:</p><ul><li><strong>Part I: Enterprise Connectivity:</strong> We established the foundation by integrating real-time voice capabilities with the WSO2 Integrator, focusing on bridging the gap between legacy enterprise systems and modern AI interfaces.</li><li><strong>Part II: Localized Intelligence:</strong> We shifted focus to On-Premises deployment, leveraging open-source powerhouses like Whisper for Speech-to-Text (STT) and Ollama for local LLM execution to prioritize data privacy and cost efficiency.</li></ul><p>This third and final part marks the transition from a sequential pipeline to a High-Performance Orchestrated Agent. We are moving beyond the static request-response loops of the past to a LiveKit-driven environment where the agent doesn’t just “process” audio, it “lives” in the room with the user. This architecture is designed for the high-stakes demands of production: handling non-linear conversations, instantaneous interruptions, and multi-model flexibility at scale.</p><h3>What is LiveKit?</h3><p>In previous explorations, we relied on standard WebSockets to move audio between the client and the server. While WebSockets are excellent for general-purpose bidirectional data, real-time voice interaction, especially at production scale, demands a protocol optimized for media: WebRTC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FqPbr3tJ8jYBey9PcDY3kg.png" /><figcaption>LiveKit Connection</figcaption></figure><p><a href="https://livekit.com/">LiveKit</a> is an open-source WebRTC framework that serves as the “backbone” of this new architecture. Unlike the TCP-based WebSockets we used previously, LiveKit utilizes UDP-based streaming. This is a game-changer for two reasons:</p><ol><li><strong>Latency &amp; Jitter:</strong> It handles network fluctuations gracefully, skipping lost audio packets rather than pausing the whole stream (eliminating the “stutter” effect).</li><li><strong>The Selective Forwarding Unit (SFU):</strong> LiveKit serves as a high-performance hub that efficiently routes audio tracks among the user, the processing worker, and the agent logic.</li></ol><h3>Moving to a Session-Based Architecture</h3><p>As voice agents grow in complexity, managing a stateless pipeline becomes a bottleneck. In our final architecture, we introduce the <strong>AgentSession</strong>. Think of the AgentSession not just as a state manager, but as a dedicated &quot;concierge&quot; for every user that enters a LiveKit room.</p><p>Unlike traditional setups that handle one request at a time, the AgentSession manages a continuous, multimodal loop. It orchestrates the handoff between Speech-to-Text (STT), the LLM logic, and Text-to-Speech (TTS) in a single, cohesive lifecycle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bPibEzbCNpkHPvgHKi7tRw.png" /><figcaption>System Architecture</figcaption></figure><p>This architecture represents a high-performance, distributed pipeline designed for real-time voice AI. It moves away from simple request-response cycles toward a continuous, full-duplex environment where audio and intelligence are synchronized in real-time through a stateful AgentSession.</p><h3>1. Transport Layer (Client &amp; LiveKit Room)</h3><ul><li><strong>The Client:</strong> Captures user audio and displays the agent’s responses. It connects to the LiveKit Room via WebRTC.</li><li><strong>LiveKit Room:</strong> Acts as the central hub for media. Unlike traditional HTTP, it uses UDP-based streaming, which is essential for reducing latency and ensuring the “stutter-free” delivery of audio.</li></ul><h3>2. The Python Backend (The AgentSession Worker)</h3><p>This is where the heavy lifting of audio processing happens. It utilizes the LiveKit SDK to join the room as a participant and manages the entire lifecycle of the interaction through an AgentSession.</p><ul><li><strong>AgentSession Orchestration:</strong> The AgentSession acts as the &quot;glue&quot; for the pipeline. It coordinates Speech-to-Text (STT), Text-to-Speech (TTS), and LLM logic in a unified, stateful loop. This ensures that media and text are always synchronized.</li><li><strong>VAD &amp; Adaptive Barge-in:</strong> WithinAgentSession, the system monitors Voice Activity Detection (VAD). This allows for human-like interaction by handling &quot;barge-ins&quot;, the user AgentSession can instantly flush the agent&#39;s current speech audio the moment it detects the user interrupting. <br>Please note that using LiveKit Cloud is required to access their advanced adaptive interruption model.</li></ul><h3>3. The LLM Adapter &amp; Agent</h3><p>While the Python AgentSession handles the &quot;ears&quot; and &quot;voice,&quot; the Agent acts as the &quot;brain.&quot;</p><ul><li><strong>LLM Adapter:</strong> A bridge within the Python backend that enables communication between LiveKit SDK and Ballerina Agent via a WebSocket. This allows the AgentSession to offload complex reasoning to Ballerina while maintaining a persistent, low-latency connection.</li><li><strong>Ballerina Agent:</strong> Using the ballerinax/openai.chat connector, it manages the conversation with GPT-4. Also, handle context management.</li><li><strong>High-Concurrency Execution:</strong> Ballerina is used here as a high-concurrency orchestrator. Its non-blocking I/O and isolated concurrency model allow the agent to efficiently manage backend logic and LLM calls without slowing down the audio pipeline managed by the livekit AgentSession.</li></ul><h3>The Advantage: Why This Architecture Wins</h3><p>By moving to a LiveKit-centric model with WSO2 Integrator, we gain several critical advantages that transform a “clunky bot” into a “human-like assistant”:</p><ul><li><strong>Sub-Second Latency via WebRTC:</strong> Traditional WebSocket or REST-based agents often suffer from high jitter. The AgentSession utilizes UDP-based WebRTC<strong> </strong>transport, bypassing the &quot;Head-of-Line Blocking&quot; found in TCP. This significantly reduces latency, enabling a natural, sub-500ms response loop.</li><li><strong>Plug-and-Play Modality:</strong> One of the framework&#39;s strongest features is its modularity. The pipeline is compatible with virtually any provider; you can swap Deepgram for Cartesia (STT) or move from ElevenLabs to Azure (TTS) without changing a single line of orchestration logic. While the orchestration is flexible, utilizing these high-performance models requires valid API keys from the respective providers.</li><li><strong>The Adaptive Interrupt Handler:</strong> Managing interruptions is notoriously difficult. The AgentSession features include an Adaptive Interrupt Handler that intelligently distinguishes between a user&#39;s definitive &quot;barge-in&quot; and mere background noise. It can instantly clear the agent&#39;s current speech buffer and reset the LLM state, making the interaction feel truly bi-directional.</li></ul><h3>What is LLM Adapter?</h3><p>To bridge the Python-based AgentSession with the agentic layer in WSO2 integrator, it’s required to implement a custom LLM Adapter. Within the LiveKit Agents framework, the LLM is treated as a pluggable component much like the existing ecosystem plugins for LangGraph or Anthropic. This Adapter serves as a specialized plugin that redirects orchestration to Ballerina via a persistent WebSocket connection keyed by a unique session_id. This allows the Python worker to send transcripts to Ballerina and receive structured responses without the latency overhead of repeated handshakes.</p><p>A key architectural decision here is offloading Context Management to the ballerina/ai module. By centralizing the conversational history and state within the Ballerina Agent, the Python adapter acts as a high-performance thin client: it captures the transcription, forwards the payload through the persistent tunnel, and triggers the next step in the media pipeline upon response. This separation ensures the Python layer stays focused on media &quot;physics&quot; latency and VAD, while Ballerina handles the cognition and enterprise logic.</p><h3>Implementation Spotlight: The Orchestration Entrypoint</h3><p>The beauty of this system is its simplicity. In the Python environment, it defines an entry point that triggers whenever a new participant joins, initializing a specialized session:</p><pre>@server.rtc_session(name=&quot;ballerina_voice_agent&quot;)<br>async def entrypoint(ctx: JobContext):<br>    # Initializing the multimodal session<br>    session = voice.AgentSession(<br>        stt=deepgram.STT(),       # transcription<br>        llm=BallerinaAdapter(),   # custom LLM bridge<br>        tts=elevenlabs.TTS(),     # Expressive voice synthesis<br>        vad=silero.VAD.load()     # Robust interruption detection<br>    )<br>    session.start(ctx.room)<br>    await session.say(&quot;Welcome! How can I assist you today?&quot;, allow_interruptions=True)</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3233d3f5cdd3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[On-Premises Voice Agents: Speech Processing with Open-Source Models]]></title>
            <link>https://medium.com/@anjanaape/on-premises-voice-agents-speech-processing-with-open-source-models-e882103f048d?source=rss-ec4c8f28f45c------2</link>
            <guid isPermaLink="false">https://medium.com/p/e882103f048d</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai-voice-agent]]></category>
            <category><![CDATA[wso2-integrator]]></category>
            <dc:creator><![CDATA[Anjana Edirisinghe]]></dc:creator>
            <pubDate>Fri, 10 Apr 2026 09:39:31 GMT</pubDate>
            <atom:updated>2026-04-10T09:50:39.527Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P5h6YoSpqvqjrhJJ1IoHWw.png" /></figure><p>The <a href="https://medium.com/@anjanaape/building-real-time-voice-agents-with-wso2-integrator-part-i-3d62ceaf074a">first article</a> covered building a real-time voice agent using OpenAI’s integrated model suite, Whisper for speech-to-text, GPT for conversational AI, and TTS-1 for speech synthesis. This architecture provides simplicity and quick deployment, but introduces dependencies on external APIs for every voice interaction.</p><p>This article explores an architecture that deploys open-source speech models and LLM locally while maintaining flexibility in language model selection. The purpose is to achieve technical control, the ability to customize models, manage data privacy, eliminate network dependencies for voice processing, and scale infrastructure independently.</p><h3>Why Local Deployment?</h3><h4>Technical Control</h4><p>Local Whisper deployment enables fine-tuning on domain-specific vocabulary, acoustic conditions, or languages. Organizations working with specialized terminology (medical, legal, technical) can train custom models to improve transcription accuracy. This customization is not possible through API-only access.</p><p>TTS engines support voice cloning, prosody control, and multi-language synthesis without per-request constraints. Organizations can create custom voices matching brand identity or specific tonal requirements.</p><h4>Data Privacy and Compliance</h4><p>Audio data contains biometric information. Local processing ensures that voice data never leaves the organization&#39;s infrastructure, meeting compliance requirements for HIPAA, GDPR, and industry-specific regulations. When combined with local LLM deployment via Ollama, the entire conversation pipeline operates on-premises or in air-gapped environments.</p><h4>Infrastructure Independence</h4><p>Network dependencies introduce failure points. Local speech processing eliminates external API calls for voice operations, reducing system complexity and improving reliability during network outages or API service degradation.</p><h4>Model Version Control</h4><p>Local deployment provides deterministic control over model versions. Organizations can test upgrades, roll back problematic versions, and maintain specific model versions for regulatory or quality assurance purposes.</p><h4>Cost Structure</h4><p>Operational costs shift from per-request API fees to fixed infrastructure costs. For applications processing thousands of conversations monthly, this transition becomes economically favorable.</p><h3>Architecture Differences</h3><p>The communication protocol remains identical between both architectures. WebSocket handles bidirectional audio streaming between the client frontend and agent. The architectural change occurs in the backend service layer.</p><h4><strong>Architecture</strong></h4><p>The frontend remains unchanged. Audio capture and WebSocket streaming operate identically. The differentiation lies in backend service composition. In this architecture, LLM calls are routed to a locally deployed Ollama model. However, the system is designed for high flexibility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rHzvZHgrq5qAZSo2uf20Uw.jpeg" /><figcaption>System Architecture</figcaption></figure><h3>Implementing Local Whisper (Speech to Text)</h3><h4>Installation and Configuration</h4><p>The <em>faster-whispe</em>r library provides optimized inference through CTranslate2, achieving 4x speedup compared to the reference implementation while maintaining identical accuracy.</p><pre>pip install faster-whisper</pre><h4>Service Implementation</h4><pre>from fastapi import FastAPI, File, UploadFile<br>from faster_whisper import WhisperModel<br>import tempfile<br>import os<br><br>app = FastAPI()<br><br># Initialize model at startup (persistent in GPU memory)<br>model = WhisperModel(<br>  &quot;medium&quot;,<br>  device=&quot;cuda&quot;,<br>  compute_type=&quot;float16&quot;<br>)<br><br>@app.post(&quot;/transcribe&quot;)<br>async def transcribe(audio: UploadFile = File(…)):<br>  with tempfile.NamedTemporaryFile(delete=False, suffix=&quot;.wav&quot;) as tmp:<br>    content = await audio.read()<br>    tmp.write(content)<br>    tmp_path = tmp.name<br>  try:<br>    segments, info = model.transcribe(tmp_path)<br>    text = &quot; &quot;.join([segment.text for segment in segments])<br><br>    return {<br>      &quot;text&quot;: text,<br>      &quot;language&quot;: info.language,<br>      &quot;duration&quot;: info.duration<br>    }<br>    finally:<br>      os.unlink(tmp_path)</pre><h4>Hardware Specifications</h4><p>Faster-whisper is available in several model configurations, each offering distinct performance characteristics.</p><p>Tip: If you have an NVIDIA GPU, use device=”cuda” for maximum performance. If you are on a Mac (Intel or Apple Silicon) or a standard laptop without a dedicated GPU, simply change the device=”cpu” and set compute type=”int8&quot; to maintain speed.</p><h3>Implementing Local Text to Speech Service</h3><h4>Kokoro TTS Setup</h4><p>Kokoro provides neural speech synthesis with configurable voice characteristics. Alternative engines include Piper, Coqui TTS, or StyleTTS2.</p><pre>from fastapi import FastAPI<br>from fastapi.responses import Response<br>import kokoro_tts <br><br>app = FastAPI()<br>tts_engine = kokoro_tts.TTS(voice=&quot;default&quot;, device=&quot;cuda&quot;)<br><br>@app.post(&quot;/synthesize&quot;)<br>async def synthesize(request: dict):<br>  text = request[&quot;text&quot;]<br>  voice_config = request.get(&quot;voice_config&quot;, {})<br><br>  audio_bytes = tts_engine.synthesize(<br>    text,<br>    sample_rate=24000,<br>    **voice_config<br>  )<br><br>  return Response(<br>    content=audio_bytes,<br>    media_type=&quot;audio/wav&quot;<br>    )</pre><h3>Integration with WSO2 Integrator</h3><p>The agent communicates with the local Whisper service via HTTP:</p><pre>Simport ballerina/ai;<br>import ballerina/http;<br>import ballerinax/ai.ollama;<br><br>// Clients for local Python services<br>final http:Client whisperClient = check new (&quot;http://localhost:8001&quot;);<br>final http:Client ttsClient = check new (&quot;http://localhost:8002&quot;);<br><br><br>// Initialize the Agent<br>final ai:Agent voiceAgent = check new ({<br>    systemPrompt: {<br>        role: &quot;Voice Assistant&quot;,<br>        instructions: &quot;You are a helpful, friendly voice assistant. <br>                        Keep responses concise and conversational (1-3 sentences). Return only the spoken response — no markdown, <br>                        no lists. You can also manage a to-do list by adding tasks, listing tasks, and checking today&#39;s date when the user asks.&quot;<br>    },<br>    tools: [addTask, listTasks, getCurrentDate],<br>    model: check new ollama:ModelProvider(ollamaModel, serviceUrl = ollamaServiceUrl)<br>});<br><br>function transcribeAudio(byte[] audioData) returns string|error {<br>    http:Request request = new;<br>    request.setBinaryPayload(audioData);<br>    request.setHeader(&quot;Content-Type&quot;, &quot;audio/wav&quot;);<br><br>    json response = check whisperClient-&gt;/transcribe.post(request);<br>    return (check response.text).toString();<br>}<br><br>function synthesizeSpeech(string text) returns byte[]|error {<br>    json payload = {<br>        text: text,<br>        voice_config: {<br>            speed: 1.0,<br>            pitch: 1.0<br>        }<br>    };<br><br>    http:Response response = check ttsClient-&gt;/synthesize.post(payload);<br>    return check response.getBinaryPayload();<br>}</pre><h4>Complete Processing Pipeline</h4><pre>import ballerina/websocket;<br>import ballerina/log;<br><br>// WebSocket listener for real-time bidirectional audio<br>service /ws on new websocket:Listener(8003) {<br>    <br>    resource function get .(http:Request req) returns websocket:Service|websocket:UpgradeError {<br>        return new VoiceAgentService();<br>    }<br>}<br><br>service class VoiceAgentService {<br>    *websocket:Service;<br><br>    remote function onBinaryMessage(websocket:Caller caller, byte[] audioData) returns error? {<br>        log:printInfo(&quot;Audio chunk received. Processing locally...&quot;);<br><br>        // 1. STT: Call Local Faster-Whisper Service<br>        string transcription = check transcribeAudio(audioData);<br>        log:printInfo(&quot;User said: &quot; + transcription);<br><br>        // 2. LLM: Process with ai:Agent (OpenAI or Local Ollama)<br>        string aiResponse = check voiceAgent-&gt;run(transcription, &quot;session-123&quot;);<br>        log:printInfo(&quot;Agent response: &quot; + aiResponse);<br><br>        // 3. TTS: Call Local Kokoro/Piper Service<br>        byte[] audioResponse = check synthesizeSpeech(aiResponse);<br><br>        check caller-&gt;writeBinaryMessage(audioResponse);<br>        <br>        log:printInfo(&quot;Audio response sent to client.&quot;);<br>    }<br><br>    remote function onError(websocket:Caller caller, error err) {<br>        log:printError(&quot;WebSocket Error&quot;, &#39;error = err);<br>    }<br>}</pre><p>Containerizing the FastAPI-based STT and TTS services guarantees environmental consistency across any deployment. In this setup, the agent interacts with these containers via a local bridge network.</p><h3>Try It Out: Deploying Your Own Local Voice Agent</h3><p>Ready to get hands-on? You can find the full implementation and source code on <a href="https://github.com/anjanaed/ballerina-voice-agent-local"><strong>GitHub</strong></a><strong>.</strong></p><h4>Quick Start</h4><p>To get the system running, you can spin up the required services using Docker or perform a manual installation. Detailed, step-by-step startup instructions for both the client and server are available in the README.md.</p><h4>Optimizing Performance</h4><p>The project is pre-configured with lightweight models to ensure it runs smoothly even on hardware with limited processing power. While this is great for a quick start, these smaller models can sometimes lead to inconsistencies or lower accuracy.</p><p>For a more robust experience, the system is fully extensible. You can swap the defaults for advanced models to achieve significantly better reasoning and more natural voice synthesis.</p><h4>Default Stack</h4><p>The current out-of-the-box configuration uses:</p><ul><li>Speech-to-Text: OpenAI Whisper (Base)</li><li>Large Language Model: qwen2.5:0.5b (via Ollama)</li><li>Text-to-Speech: Kokoro-tts</li></ul><p>If you have a dedicated GPU, upgrading the STT to the large-v3 model will drastically improve transcription accuracy for complex technical terms.</p><h3>Summary</h3><p>This Architecture decouples voice processing from proprietary APIs while maintaining protocol compatibility at the frontend. Local deployment of Whisper and TTS engines provides technical control, enables model customization, ensures data privacy, and supports infrastructure independence.</p><p>The WebSocket communication layer remains constant across both architectures, allowing experimentation and migration without frontend changes. The Ballerina agent orchestrates services, managing local speech processing, local LLM inference, and graceful fallback strategies.</p><p>Users can adopt this architecture incrementally, validating each component before migrating to it in full. The flexibility to choose between cloud and local LLM deployment provides additional strategic options based on specific requirements.</p><h3>Next in Series</h3><p>Next, we’ll explore Architecture that integrates LiveKit integration with WebRTC for production-scale deployments. Topics include WebRTC’s UDP-based transport, packet loss handling strategies, and when the additional complexity provides value.</p><h3>Resources</h3><ul><li><a href="https://ballerina.io/">https://ballerina.io/</a></li><li><a href="https://github.com/anjanaed/ballerina-voice-agent-local">https://github.com/anjanaed/ballerina-voice-agent-local</a></li><li><a href="https://central.ballerina.io/ballerinax/openai.audio/latest">ballerinax/openai.audio — Ballerina Central</a></li><li><a href="https://central.ballerina.io/ballerina/ai/latest">ballerina/ai Module Docs</a></li><li><a href="https://lib.ballerina.io/ballerina/websocket/latest">Ballerina WebSocket Module Docs</a></li><li><a href="https://ballerina.io/learn/by-example/">Ballerina by Example</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e882103f048d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Real-Time Voice Agents with WSO2 Integrator: Part I]]></title>
            <link>https://medium.com/@anjanaape/building-real-time-voice-agents-with-wso2-integrator-part-i-3d62ceaf074a?source=rss-ec4c8f28f45c------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d62ceaf074a</guid>
            <category><![CDATA[research]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ballerina]]></category>
            <category><![CDATA[voice-agent]]></category>
            <category><![CDATA[wso2]]></category>
            <dc:creator><![CDATA[Anjana Edirisinghe]]></dc:creator>
            <pubDate>Sat, 04 Apr 2026 09:39:39 GMT</pubDate>
            <atom:updated>2026-04-10T09:52:16.496Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XFd_0hqUfvlgBTWk9fvPpw.png" /></figure><h3>What Is WSO2 Integrator?</h3><p><a href="https://wso2.com/integration-platform/integrator/">WSO2 Integrator</a> is a paradigm-shifting integration development platform that embodies simplicity and power. It is designed with a unique low-code development experience powered by the open-source Ballerina integration programming language.</p><p>It provides out-of-the-box support to connect anything APIs, AI agents, systems, and databases across any environment and protocol, and is complemented by 200+ pre-built connectors. What makes it distinctive is its fluid low-code and pro-code parity: developers can seamlessly switch between graphical low-code and textual pro-code interfaces and even view them side-by-side without leaving the VS Code editor.</p><p><a href="https://ballerina.io/"><strong>Ballerina</strong></a>:</p><p>At its core, WSO2 Integrator is powered by Ballerina<strong>, </strong>an open-source, cloud-native programming language optimized for integration, where network primitives are built directly into the language, making it simpler to write services and run them in the cloud. Its dual-representation model automatically generates real-time sequence and flow diagrams from code, keeping architecture visuals and implementation permanently in sync.</p><h3>Modular Approach to Build Voice Agents</h3><p>A voice agent needs to do three things: listen, think, and speak. The modular pipeline approach assigns a dedicated model to each responsibility, wired together by the integration layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cnu5dNQ2PPfaGPYvH-yiZQ.jpeg" /><figcaption>Voice Agent Pipeline</figcaption></figure><p>This decomposition gives full control of the process. It enables swapping Whisper for a local STT model, replacing TTS-1 with an open-source alternative like Kokoro, or upgrading the LLM independently without touching the rest of the pipeline. Each stage is also independently observable; if transcription quality drops, isolate Whisper. If response latency spikes, benchmark the LLM. If the audio sounds unnatural, tune TTS alone.</p><p>Ballerina’s ballerina/ai module provides a native ai:Agent abstraction that handles the LLM stage, managing GPT-4o calls, maintaining per-session conversation memory viasessionId, and returning a plain string response. The ballerinax/openai.audio connector handles both audio ends, with a single audio:Client instance covering Whisper-1 and TTS-1. WSO2 Integrator wires all three stages together, and its low-code flow view renders the pipeline as a live sequence diagram, with no separate diagramming effort required.</p><h3>System Architecture</h3><p>This implementation connects a client frontend directly to a Ballerina backend over a persistent WebSocket. The backend acts as the orchestrator, coordinating Whisper-1, GPT-4o, and TTS-1 to complete each voice interaction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LOel0pCHM7rZnMUqbU_dCA.jpeg" /><figcaption>System Architecture</figcaption></figure><h4>How it works:</h4><ol><li><strong>Capture &amp; Stream:</strong> The client listens to the user’s voice and streams the audio data to the Ballerina backend in real-time via a WebSocket.</li><li><strong>Buffering:</strong> The backend buffers these audio chunks until it detects a mute, signalling that the user has finished speaking.</li><li><strong>Speech-to-Text (STT):</strong> The gathered audio is sent to OpenAI Whisper, which converts the speech into text.</li><li><strong>Processing (LLM):</strong> This text is then passed to the LLM to generate an intelligent response.</li><li><strong>Text-to-Speech (TTS):</strong> The LLM’s text response is sent to OpenAI TTS for conversion back into a natural-sounding voice.</li><li><strong>Playback:</strong> The final audio is streamed back to the client over the same WebSocket for immediate playback.</li></ol><p>By using Ballerina’s native support for WebSockets and HTTP, wiring these components together is straightforward. This approach ensures a reliable, bidirectional flow where the agent can listen and respond smoothly without complex networking overhead.</p><h3>Component Breakdown</h3><h4>The Client Frontend</h4><p>The frontend captures microphone audio via the Web Audio API and sends it to the Ballerina backend over WebSocket. It can be done in two modes: sending a complete WAV file (identified by its RIFF header) or streaming raw PCM-16 chunks in real time. When the user stops speaking, the frontend can send a <em>stream_end</em> control message to signal that the buffer is complete and ready for processing.</p><p>Checkout Client implementation: <a href="https://github.com/anjanaed/bal-voice-agent/tree/master/client">bal-voice-agent/client at master · anjanaed/bal-voice-agent</a></p><h4>The Ballerina Agent</h4><p>The agent sits in the middle and owns the integration logic. It receives raw audio from the frontend, forwards it to Whisper for transcription, passes the transcript to a language model for a response, then calls TTS-1 to synthesize speech, and finally ships that audio back to the frontend over the same WebSocket.</p><p>Implementation: <a href="https://github.com/anjanaed/bal-voice-agent/tree/master/server">bal-voice-agent/server at master · anjanaed/bal-voice-agent</a></p><h4>OpenAI Models</h4><ul><li><strong>Whisper-1</strong> handles speech-to-text, accepting WAV audio and returning a transcription</li><li><strong>GPT-4o</strong> handles language understanding via the ai:Agent, with conversation history maintained per session</li><li><strong>TTS-1</strong> handles text-to-speech, returning raw MP3 bytes</li></ul><h3>WSO2 Integrator Low-Code Implementations</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J6q2rBrqM7mnWgsLT_LwFQ.jpeg" /><figcaption>WSO2 Integrator-Low Code Environment</figcaption></figure><h3>Demonstration</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F1180058642%3Fapp_id%3D122963&amp;dntp=1&amp;display_name=Vimeo&amp;url=https%3A%2F%2Fvimeo.com%2F1180058642&amp;image=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F2142072308-56a2e5e9e2c6f0041ac472659a5964a03f5aa9f93c6be314699265625fa75b09-d_1280%3Fregion%3Dus&amp;type=text%2Fhtml&amp;schema=vimeo" width="1920" height="1080" frameborder="0" scrolling="no"><a href="https://medium.com/media/ff09121d7617d4be039dfdf6ddceeec9/href">https://medium.com/media/ff09121d7617d4be039dfdf6ddceeec9/href</a></iframe><h3>The Trade-offs</h3><p>This architecture is not the production endpoint; it’s the diagnostic baseline. It isolates AI model performance from transport-layer variables, letting developers benchmark each pipeline stage independently before introducing WebRTC complexity.</p><p><strong>Core Competencies:</strong></p><ul><li>The ai:Agent abstraction handles session memory cleanly, no manual message history management</li><li>TCP guarantees ordered delivery, no out-of-sequence audio frames</li><li>Both streaming PCM and full WAV paths are supported from day one</li><li>Per-stage error messages reach the client immediately, making debugging straightforward</li></ul><p><strong>Limitations:</strong></p><p>The sequential STT → LLM → TTS chain means the user waits for the full round trip before hearing any response. In measured tests, this architecture clocks WebSocket communication latency at around 42.5ms, fast for the transport layer, but the AI processing stages stack on top of that. TCP also buffers on packet loss rather than degrading gracefully, which becomes noticeable on weaker connections.</p><p>For push-to-talk prototypes and controlled demos, this is entirely acceptable. For production voice agents demanding sub-200ms end-to-end responsiveness, this architecture reveals its ceiling, which is precisely its value as a diagnostic starting point.</p><h3>Summary</h3><p>We’ve laid the foundation for a functional voice agent. By using Ballerina as the orchestrator and OpenAI for the intelligence, we’ve successfully built a system that can:</p><ul><li><strong>Stream live audio</strong> over a bidirectional WebSocket.</li><li><strong>Convert speech to text</strong> using Whisper.</li><li><strong>Process language</strong> and generate responses via an LLM.</li><li><strong>Synthesize voice</strong> using TTS-1 for a natural-sounding reply.</li></ul><p>We now have a working “Listen-Process-Speak” loop that handles the heavy lifting of voice AI with minimal boilerplate code.</p><h3>What’s Next</h3><p>While our current “buffered” approach is a great starting point, it still feels a bit like a walkie-talkie, the user speaks, the agent waits, and then it responds. To make this feel like a true human conversation, we need to push further.</p><p>In the next part of this series, we’ll be:</p><ul><li>Progressively move toward on-premises deployability, replacing cloud-hosted models with open-source alternatives that can run entirely within their own infrastructure.</li><li>Implement LiveKit + WebRTC, replacing TCP transport with UDP for adaptive quality and built-in noise cancellation, coordinated through a Ballerina runtime agent</li></ul><p>Stay tuned as we evolve this basic setup into a truly reactive, intelligent voice assistant!</p><p><strong>On-Prem Voice Agent Blog</strong>: <a href="https://medium.com/@anjanaape/on-premises-voice-agents-speech-processing-with-open-source-models-e882103f048d">https://medium.com/@anjanaape/on-premises-voice-agents-speech-processing-with-open-source-models-e882103f048d</a></p><h3>Resources</h3><ul><li><a href="https://ballerina.io/">https://ballerina.io/</a></li><li><a href="https://github.com/anjanaed/bal-voice-agent">anjanaed/bal-voice-agent</a></li><li><a href="https://central.ballerina.io/ballerinax/openai.audio/latest">ballerinax/openai.audio — Ballerina Central</a></li><li><a href="https://central.ballerina.io/ballerina/ai/latest">ballerina/ai Module Docs</a></li><li><a href="https://lib.ballerina.io/ballerina/websocket/latest">Ballerina WebSocket Module Docs</a></li><li><a href="https://ballerina.io/learn/by-example/">Ballerina by Example</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d62ceaf074a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Host Your PWA on AWS — Free, Fast & with SSL]]></title>
            <link>https://medium.com/@anjanaape/host-your-pwa-on-aws-free-fast-with-ssl-510b710a8ee3?source=rss-ec4c8f28f45c------2</link>
            <guid isPermaLink="false">https://medium.com/p/510b710a8ee3</guid>
            <dc:creator><![CDATA[Anjana Edirisinghe]]></dc:creator>
            <pubDate>Fri, 01 Aug 2025 14:29:16 GMT</pubDate>
            <atom:updated>2026-04-09T13:42:44.503Z</atom:updated>
            <content:encoded><![CDATA[<h3>Host Your PWA on AWS -Free, Fast &amp; with SSL</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*a8RUl-tCtj5LPq7Z.jpg" /><figcaption>Amazon Web Services</figcaption></figure><p>Looking to host your full-stack application <strong>for free</strong>, <strong>with low latency</strong>, and <strong>SSL support</strong>? This guide is perfect for developers who want to quickly spin up an app — especially for <strong>demos, hackathons, MVPs, or client previews</strong> — without relying on services like Vercel or Netlify.</p><h3>1. Get Started with AWS Free Tier</h3><p>First, create an <a href="https://aws.amazon.com/free/">AWS account</a>. New users get <strong>12 months of free compute hours</strong>, which you can use to run a virtual machine (EC2 instance). It’s ideal for basic Progressive Web Apps (PWA) and backend APIs.</p><h3>2. Launch a Virtual Machine (EC2)</h3><p>Use Ubuntu for a cost-effective setup. For better performance within the free tier, consider:<br>Instance Type: m7i-flex.large (2 vCPU, 8GiB RAM) — Best one in free tier <br>Region: Choose the nearest to your user base</p><p>Ensure HTTP (80), HTTPS (443), and SSH (22) ports are open in your security group. There are tons of tutorials available for AWS VM creation.</p><h3>3. Connect to Your Instance</h3><p>Use your terminal or SSH client:</p><pre>ssh -i &quot;your-key.pem&quot; ubuntu@your-ec2-public-ip</pre><blockquote><em>Tip: Run </em><em>chmod 400 your-key.pem before connecting to fix permissions.</em></blockquote><h3>4. Set Up the Runtime Environment</h3><p>Install whatever environment your app needs (Node.js, Python, etc.).<br>Installing the latest Node version in an Ubuntu environment is a bit tricky. To install the <strong>latest Node.js</strong>, follow this excellent guide:<br> 👉 <a href="https://medium.com/@nsidana123/before-the-birth-of-of-node-js-15ee9262110c">How to Install Node.js Properly</a></p><h3>5. Upload Your App Files</h3><p>You can either use Git to clone your repo or transfer files via FTP (like FileZilla or WinSCP).</p><pre>git clone https://github.com/your-user/your-app.git<br>cd your-app</pre><h3>6. Run the Backend with PM2</h3><p>Use <a href="https://pm2.keymetrics.io/">PM2</a> to keep your backend app alive in the background.</p><pre>npm install -g pm2<br>pm2 start index.js --name your-app<br>pm2 save &amp;&amp; pm2 startup</pre><h3>7. Configure Nginx as a Reverse Proxy</h3><p>Install and configure Nginx to route traffic to your backend server.</p><pre>sudo apt install nginx<br>sudo nano /etc/nginx/sites-available/default</pre><p>Minimal config example:</p><pre>server {<br>  listen 80;<br>  server_name yourdomain.com;<br><br>location / {<br>    proxy_pass http://localhost:3000;<br>    proxy_set_header Host $host;<br>  }<br>}</pre><p>Then, to restart nginx</p><pre>sudo systemctl restart nginx</pre><h3>8. Use a Domain</h3><p>If you don’t have a domain, use <a href="https://www.duckdns.org/">DuckDNS</a> or a similar DDNS provider to get a domain for free. Then point the domain DNS IP to your EC2 server IP</p><pre>yourapp.duckdns.org A → your-ec2-ip</pre><h3>9. Add Free SSL with Certbot</h3><p>Secure your app using <strong>Let’s Encrypt</strong> SSL via Certbot:</p><pre>sudo apt install certbot python3-certbot-nginx<br>sudo certbot --nginx</pre><p>It auto-renews and handles HTTPS redirection as well.</p><h3>10. Deploy Frontend</h3><p>Build your frontend locally or on the instance:</p><pre>npm run build</pre><p>Then copy the build output to the web root:</p><pre>sudo cp -r build/* /var/www/html/</pre><p>Set Nginx root to /var/www/html; If serving static files.</p><h3>🎉 You’re Live!</h3><p>That’s it. You now have a <strong>full-stack app deployed on AWS</strong>, with:</p><ul><li><strong>Free server hosting</strong></li><li><strong>Fast regional performance</strong></li><li><strong>Custom domain or subdomain</strong></li><li><strong>HTTPS/SSL secured traffic</strong></li></ul><p>Ideal for MVPs, demos, portfolios, or just to learn how hosting works behind the scenes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=510b710a8ee3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Secure Authentication System for Progressive Web Apps(PWA) Using Auth0]]></title>
            <link>https://medium.com/@anjanaape/building-a-secure-authentication-system-for-progressive-web-apps-pwa-using-auth0-d9745c4bf1a8?source=rss-ec4c8f28f45c------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9745c4bf1a8</guid>
            <dc:creator><![CDATA[Anjana Edirisinghe]]></dc:creator>
            <pubDate>Sun, 27 Jul 2025 10:43:07 GMT</pubDate>
            <atom:updated>2025-07-27T10:43:07.296Z</atom:updated>
            <content:encoded><![CDATA[<p>As part of my Level 2 Software Development project, I recently implemented a full-fledged authentication system for a Single Page Application (SPA) using <strong>Auth0</strong>. After diving deep into the process, I discovered just how powerful and secure Auth0 can be, especially for Progressive Web Apps (PWAs). In this article, I’ll walk you through how I integrated Auth0 using their APIs instead of the default login page, the challenges I faced, and some key insights for developers planning to use Auth0 in real-world projects.</p><h3>Why Auth0?</h3><p>In today’s web applications, security is non-negotiable. Auth0 provides a robust, scalable authentication solution with minimal setup required. It supports:</p><ul><li>Email or username-based login</li><li>OAuth 2.0, OpenID Connect</li><li>Social login (Google, Facebook, etc.)</li><li>Custom rules and actions</li><li>Multi-tenancy with custom databases</li></ul><p>While there are built-in role management tools, I opted to handle role assignments manually through my backend for greater flexibility in our project.</p><h3>Setting Up Auth0</h3><p>To get started, you need to:</p><ol><li>Create an <strong>Auth0 account</strong></li><li>Create a new <strong>Application</strong></li><li>Use the default <strong>Database Connection</strong> (or create your own)</li></ol><p>Auth0 gives you both a <em>default application</em> and a <em>default user store</em> (called a connection). Depending on your architecture, you can either stick with these or customize them.</p><h3>Two Ways to Implement Auth0</h3><p>During my research, I found two primary ways to integrate Auth0 into a web app:</p><ol><li><strong>Using Auth0’s Universal Login Page</strong> — Quick and secure with a ready-made UI.</li><li><strong>Using Auth0 APIs</strong> — Gives full control over the frontend experience.</li></ol><p>Since I already had a custom-designed login screen in our React SPA, I went with <strong>Option 2</strong>: using <strong>Auth0 APIs</strong> directly.</p><h3>Signup and Login Workflow</h3><p><strong>Sign-up</strong> was straightforward. The frontend sends a signup request to Auth0’s /dbconnections/signup endpoint. But <strong>login</strong> is a bit different. For security reasons, login requests using the <em>Resource Owner Password</em> flow must be made from a <strong>secure backend</strong>, not directly from the frontend.</p><p>In my <strong>NestJS backend</strong>, I used the following code to handle login:</p><pre>axios.post(`https://${this.url}/oauth/token`, {<br>  grant_type: &#39;password&#39;,<br>  username: username,<br>  password: password,<br>  audience: `https://${this.url}/api/v2/`,<br>  client_id: this.clientId,<br>  client_secret: this.clientSecret,<br>  scope: &#39;openid profile email&#39;,<br>})</pre><p>✅ On success, you’ll receive:</p><ul><li>access_tokenUsed for API authorization</li><li>id_token: Contains user identity claims</li></ul><p>These tokens can be decoded using JWT to extract user data securely.</p><h3>Custom Login Logic with Actions</h3><p>Auth0 offers <strong>Actions</strong>, which are scripts triggered during login, signup, or other events. I used a <strong>Post-Login Action</strong> to embed the employee ID into a custom claim in the token:</p><pre>api.idToken.setCustomClaim(&quot;https://yourdomain.com/employee_id&quot;, employeeId);</pre><p>This way, the frontend or backend can securely retrieve the employee ID without needing to store it elsewhere.</p><h3>Multi-App Setup with Realms</h3><p>In my system, I had two different user groups with separate authentication needs. To handle this, I created <strong>two Auth0 Applications</strong>, each with its own <strong>Database Connection</strong>.</p><p>By default, Auth0 checks credentials against the default database. If you want to authenticate against a specific database, you must include a realm In the login request:</p><pre>axios.post(`https://${this.url}/oauth/token`, {<br>  grant_type: &quot;http://auth0.com/oauth/grant-type/password-realm&quot;,<br>  username: email,<br>  password: password,<br>  audience: `https://${this.audience}`,<br>  client_id: this.clientId,<br>  client_secret: this.clientSecret,<br>  realm: &quot;SuperAdmin&quot;, // Your custom DB connection name<br>})</pre><p>Without the realmAuth0 will ignore your custom database and always fall back to the default one.</p><h3>Forgot Password Flow</h3><p>Auth0 makes password resets simple. With a small API request, users receive a secure reset link via email — no need to store or manage passwords on your own.</p><h3>What About Roles?</h3><p>While Auth0 supports <strong>role-based access control (RBAC)</strong> out of the box, I chose to manage user roles in my backend system to suit specific project requirements. Still, if you need centralized role management, Auth0 provides a user-friendly interface and APIs to assign and check roles.</p><h3>Final Thoughts</h3><p>Auth0 has been a game-changer for our team’s authentication needs. While some advanced features are locked behind paid plans (like passkeys, custom domains, and advanced RBAC), the <strong>free tier</strong> was more than enough for our use case.</p><p><strong>Key Takeaways:</strong></p><ul><li>Don’t send login requests directly from the frontend — use your backend for that.</li><li>Use <strong>realms</strong> to manage multiple databases.</li><li>Post-login <strong>Actions</strong> are powerful for adding custom logic securely.</li><li>Auth0 is a great fit for both small and large-scale projects.</li></ul><p>If you’re building an SPA or PWA and want secure, scalable authentication with minimal headaches, I highly recommend giving Auth0 a try.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9745c4bf1a8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Blockchain & Web 3.0]]></title>
            <link>https://medium.com/@anjanaape/what-is-blockchain-web-3-0-1b017a00d1ed?source=rss-ec4c8f28f45c------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b017a00d1ed</guid>
            <category><![CDATA[sinhala]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[etherium]]></category>
            <category><![CDATA[web3]]></category>
            <category><![CDATA[bitcoin]]></category>
            <dc:creator><![CDATA[Anjana Edirisinghe]]></dc:creator>
            <pubDate>Mon, 23 Sep 2024 09:14:16 GMT</pubDate>
            <atom:updated>2024-09-23T09:14:40.904Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qcQk62eowoSOSiNEjkSGdw.jpeg" /></figure><p>In the ever-evolving world of technology, one buzzword has stood out in recent years: <strong><em>Blockchain</em></strong>. It’s been labeled as revolutionary, transformative, and disruptive. But what is it really? And how does it relate to Web3? Let’s explore these concepts from the ground up.</p><p>Understanding Blockchain: A Simple Explanation</p><p>At its core, blockchain is a digital ledger. Imagine a notebook that records transactions, but instead of being kept in one place, it’s duplicated across thousands of computers around the world. These transactions are bundled into “blocks,” and each block is linked or “chained” to the one before it. This forms a continuous, tamper-proof record of data.</p><p>What makes blockchain special is that no single entity controls this ledger. Instead, it operates through a decentralized network of computers, making it almost impossible to alter or hack.</p><p>A key feature of blockchain is transparency. Every transaction is visible to all participants, making fraud more difficult. Additionally, once data is recorded on the blockchain, it cannot be modified or deleted without consensus from the network.</p><p>Let’s go back to Inception…</p><p>Blockchain technology has its roots in <strong>Bitcoin</strong>, the first and most famous cryptocurrency, introduced in 2009 by the mysterious figure <strong>Satoshi</strong> <strong>Nakamoto</strong>. The idea was simple yet revolutionary: creating a financial system free from intermediaries like banks, powered solely by cryptographic algorithms and peer-to-peer networks.</p><p>While Bitcoin popularized blockchain, its potential for innovation didn’t stop there. Soon, developers realized blockchain could be used for much more than just digital currency. In 2015, <strong>Ethereum</strong> was launched by <strong>Vitalik</strong> <strong>Buterin</strong>, expanding blockchain’s possibilities with smart contracts — self-executing contracts that run when pre-defined conditions are met. This opened the door to decentralized applications (dApps) and countless other use cases.</p><p>Mechanisms behind blockchains…</p><p>Blockchain operates on several consensus mechanisms — protocols that verify transactions and keep the network secure. I’ll write about how blockchains work deep down soon. For now, Some popular ones include:</p><p>- **Proof of Work (PoW)**: This is used by <strong>Bitcoin</strong>. In PoW, miners compete to solve complex mathematical puzzles, and the first to solve it gets to add a block to the chain. However, this method requires significant computational power and energy.<br> <br>- **Proof of Stake (PoS)**: This mechanism is more energy-efficient than PoW. Instead of miners, validators are chosen to add new blocks based on the amount of cryptocurrency they hold (stake) and are willing to lock up as collateral. <strong>Ethereum</strong> <strong>2.0</strong> is transitioning to PoS.</p><p>- **Delegated Proof of Stake (DPoS)**: In this system, users vote for a small number of trusted validators who manage the blockchain. <strong>EOS</strong> and <strong>Tron</strong> are examples of blockchains using DPoS.</p><p>The Evolution…</p><p>Blockchain is constantly evolving, and the next major frontier is <strong>Web3,</strong> often described as the “decentralized internet.” In Web3, users have direct control over their data, assets, and interactions, bypassing centralized platforms like Google, Facebook, or Amazon.</p><p>Central to Web3 are <strong>dApps</strong> — decentralized applications that run on blockchain technology. Unlike traditional apps, dApps are not controlled by any one company. They operate autonomously once deployed, powered by smart contracts on blockchains like Ethereum, Binance Smart Chain, and Solana.</p><p>These dApps cover everything from decentralized finance (DeFi) platforms, where users can lend and borrow cryptocurrency without a bank, to decentralized social media, where no single entity owns or controls the content.</p><p>Current Major Blockchains…</p><p>Several blockchains have emerged as key players in this space, each with unique features and ecosystems:</p><p>- **Ethereum**: The leader in smart contracts and dApps, Ethereum pioneered decentralized finance (DeFi) and non-fungible tokens (NFTs).<br> <br>- **Binance Smart Chain (BSC)**: Known for its speed and low transaction fees, BSC is an alternative to Ethereum for dApp developers.<br> <br>- **Solana**: Famous for its scalability and high throughput, Solana offers lightning-fast transactions, making it popular for dApps requiring speed.<br> <br>- **Polkadot**: With its focus on interoperability, Polkadot allows multiple blockchains to transfer data and assets across each other seamlessly.</p><p>The future of blockchain looks incredibly bright. The transition of Ethereum to Ethereum 2.0, with its move to Proof of Stake, is expected to reduce energy consumption and improve scalability. Additionally, cross-chain solutions like Polkadot are paving the way for a truly interconnected blockchain ecosystem.</p><p>As the technology matures, we will likely see more industries adopting blockchain beyond finance. Supply chain management, healthcare, and even voting systems could be revolutionized by its transparency and security.</p><p>At the heart of this transformation is Web3, where users reclaim control. Instead of relying on tech giants, individuals will own their data and digital assets, and interactions will be direct and peer-to-peer.</p><p>Finally…</p><p>Blockchain technology is no longer just about cryptocurrencies. It’s a groundbreaking tool that powers a new era of the internet — Web3. As we stand at the intersection of blockchain’s past and its limitless future, one thing is clear: technology is here to stay, and it’s reshaping the digital landscape as we know it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b017a00d1ed" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>