<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Harshvardhan Bhosale on Medium]]></title>
        <description><![CDATA[Stories by Harshvardhan Bhosale on Medium]]></description>
        <link>https://medium.com/@harshvardhanbhosale9?source=rss-1f1358360f78------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*LhYnpVR8622FzK3lMZgtRw.png</url>
            <title>Stories by Harshvardhan Bhosale on Medium</title>
            <link>https://medium.com/@harshvardhanbhosale9?source=rss-1f1358360f78------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 17:44:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@harshvardhanbhosale9/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Scaling realtime VoIP customer support agents]]></title>
            <link>https://medium.com/@harshvardhanbhosale9/scaling-realtime-voip-customer-support-agents-42eac1e3a78a?source=rss-1f1358360f78------2</link>
            <guid isPermaLink="false">https://medium.com/p/42eac1e3a78a</guid>
            <category><![CDATA[system-design-project]]></category>
            <category><![CDATA[agentic-workflow]]></category>
            <category><![CDATA[ai-voice-agent]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <category><![CDATA[real-time-streaming-data]]></category>
            <dc:creator><![CDATA[Harshvardhan Bhosale]]></dc:creator>
            <pubDate>Mon, 23 Feb 2026 10:31:59 GMT</pubDate>
            <atom:updated>2026-02-23T10:31:59.272Z</atom:updated>
            <content:encoded><![CDATA[<p>Well, you set the pipeline for your call agent. Twilio, OpenAI, MCPs and data stores. Great..but what after this? Can it handle 1000 concurrent calls? Here I will share my experience with scaling one such pipeline.</p><blockquote>Context: I built and scaled a simple call agent pipeline to handle 1000+ daily calls for multiple NGOs and SMBs.</blockquote><blockquote>Dev Stack : Twilio, FastAPI, OpenAI Realtime, composio (MCPs)</blockquote><blockquote>Infra : ECS Fargate, SQS, ElastiCache, Chroma, mem0</blockquote><h3><strong>Overview of the architecture that I wanted to scale</strong></h3><p>Here, Twilio handles the media stream once caller and conversation layer connect. By using the window between webhook and websocket, the conversation layer would initialise an OpenAI session with context about the caller through mem0 or DB query. The conversation layer produces an intent.json that captures caller’s intent. The backend layer executes tools to fullfil the intent and the eval layer evaluates the output before deciding on a human escalation or returning it to the conversation layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pPmSzWx99TSmb5kgh8-Hsw.png" /><figcaption>Dev architecture</figcaption></figure><h3>Flaws in the above architecture</h3><p>Once you write down a problem, consider it half done. So lets solve 50% of our issues here:</p><ol><li>Single points of failures since no persistence and statefulness.</li><li>No load management in place.</li><li>No provision to avoid 429s</li><li>No retries (related to the first point)</li></ol><h3>The solution</h3><p>Anyone with decent backend experience would have guessed it by now. The answer to the above problems are the two pillars of distributed systems — Queues and Redis. Below is the final architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w7-jf1Ez5mLRJpRpZ8SyYg.png" /><figcaption>Deployed Architecture</figcaption></figure><p><strong>Call Ingestion &amp; Warm-Up</strong></p><p>When a caller dials in, Twilio fires a webhook to /incoming-call (incoming call handler). Rather than waiting for the WebSocket to open before fetching caller context, we fire the warm-up <strong>asynchronously in that window</strong> (mem0 query for caller memory, MongoDB for account data). The combined context lands in Redis[context:{CallSid}] before the first audio frame arrives. The webhook returns TwiML immediately; Twilio doesn&#39;t wait on us.</p><p>This single pattern eliminates the most common source of cold-start latency in voice agents.</p><blockquote>Thre reason why incoming call handler won’t be a SPOF is because it is a <strong>stateless HTTP endpoint</strong>. It does three things: reads the From field, fires an async warm-up task, and returns a TwiML string. There&#39;s no state inside it. That means you can run <strong>as many instances of it as you want behind the ALB, </strong>it&#39;s just another Fargate task. If one dies, the ALB routes the next incoming webhook to a healthy one. AWS ALB health checks will stop routing to unhealthy tasks within seconds.</blockquote><p><strong>Conversation Layer (Fargate Workers)</strong></p><p>Twilio opens a WebSocket → ALB → a Conversation Layer Worker running on Fargate. The worker pulls pre-warmed context from Redis (ElastiCache) and subscribes to Redis[results:{CallSid}] via Pub/Sub — this is how it&#39;ll receive backend results later without polling.</p><p>It initializes an OpenAI Realtime session with the caller context injected upfront, then bridges audio bidirectionally between Twilio and the Realtime API. At turn completion, it extracts an intent.json and pushes it to SQS[intent-queue].</p><p>Multiple workers sit behind the ALB. Any worker can handle any call since state lives in Redis, not in-process.</p><p><strong>Backend Layer</strong></p><p>Backend Agent Workers pull from SQS[intent-queue]. Each worker executes the intent using MCP tools : Airtable, MongoDB, Twilio, Gmail, whatever the integration surface requires. Results go to SQS[eval-queue].</p><p>SQS gives us durability and natural backpressure. If a backend worker hits a 429 or a tool times out, the message stays in the queue and gets retried with backoff. No custom retry logic needed.</p><p><strong>Eval Layer</strong></p><p>Eval Workers pull from SQS[eval-queue] and compare intent vs result. Two paths:</p><ul><li><strong>Escalate</strong> → SNS fires an alert to Slack/SMS. The call routes to a human operator via the Human Queue.</li><li><strong>No escalate</strong> → result published to Redis[results:{CallSid}].</li></ul><p>The Conversation Worker receives the result instantly via the Redis Pub/Sub subscription it set up at call start, feeds it into the OpenAI Realtime session, and the caller hears the response.</p><blockquote>The reason for using a Redis pub/sub channel to push results to the conversation layer is to overcome the competing consumers problem created by conversation layer workers. Essentially, we want to make sure to return the result to the exact worker thats is talking to the target user.</blockquote><p><strong>Post-Call Cleanup</strong></p><p>On hangup: mem0 gets a call summary write, MongoDB gets the full call record, Redis keys are cleaned up, and a structured log is emitted with CallSid, latency breakdown, and outcome. Nothing lives in ephemeral worker memory past call end.</p><h3>How This Fixes the Dev Architecture’s Flaws</h3><p><strong>Single points of failure / no statefulness</strong> — Fixed by externalizing all state to Redis and MongoDB. Worker crashes don’t lose calls; the next worker picks up from Redis.</p><p><strong>No load management</strong> — Fixed by ALB in front of Conversation Workers and SQS decoupling the backend and eval layers. Spiky call volume buffers in the queues rather than overwhelming downstream workers.</p><p><strong>No provision to avoid 429s</strong> — Fixed at two levels: SQS visibility timeouts + exponential backoff handle transient rate limits from MCP tool calls; the async warm-up spreads API calls across the webhook/WebSocket window rather than hammering them at session start.</p><p><strong>No retries</strong> — Fixed by SQS. Messages are not deleted until a worker explicitly acknowledges success. Failed processing → message becomes visible again → retry. Dead-letter queues catch anything that exhausts retries.</p><h3>Conclusion</h3><p>There are many more ways of scaling a voice based agentic workflow, this is the one which I found to be the most intuitive. The only bottleneck here is the OpenAI API with its cost limitations. This architecture makes sure that the infra is simple with only queues and cache. Would love to hear more about this in the comments!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42eac1e3a78a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sortify : A scaleable cron housekeeper for your Liked Songs on Spotify]]></title>
            <link>https://medium.com/@harshvardhanbhosale9/sortify-a-scaleable-cron-housekeeper-for-your-liked-songs-on-spotify-42681c7b6463?source=rss-1f1358360f78------2</link>
            <guid isPermaLink="false">https://medium.com/p/42681c7b6463</guid>
            <category><![CDATA[backend]]></category>
            <category><![CDATA[projects]]></category>
            <category><![CDATA[fastapi]]></category>
            <category><![CDATA[scalable-applications]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Harshvardhan Bhosale]]></dc:creator>
            <pubDate>Thu, 15 Jan 2026 16:21:56 GMT</pubDate>
            <atom:updated>2026-01-15T16:32:45.985Z</atom:updated>
            <content:encoded><![CDATA[<h3>Sortify : A scaleable cron housekeeper for your Liked Songs on Spotify</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w94kjPui7GqOkUdM7ZOdMA.png" /><figcaption><a href="https://sortify.run/">https://sortify.run/</a></figcaption></figure><p>As software engineers, we also tend to create tools to address problems we ourselves have encountered. In my case, it was my girlfriend’s “Liked Songs” playlist on Spotify, which was a mess. I needed a system to organize songs automatically into playlists by genres and languages. What started as a simple script turned into a fully-fledged SaaS app, and during this journey, I encountered a number of interesting problems to solve on the backend.</p><p>In this blog post, I will take you through the technical choices and implementations that make this system robust, secure, and scalable. We will explore rate limiting, retry patterns, security protocols, OAuth flows, and Large Language Model (LLM) integrations for smart classification.</p><h3>1. Rate Limiting: Playing Nice with APIs</h3><p>When working with third-party APIs, such as the Spotify API, it’s essential to abide by the rate limits set forth in the API guidelines. This goes beyond the simple avoidance of the 429 status code, and it’s about being a good API citizen.</p><p><strong>The Sliding Window Algorithm</strong><br>I did not use a fixed window counter, where once the counter reaches zero, it is not possible to send further packets until it reaches zero again. I instead used a Sliding Window algorithm. This ensures that if there is a burst of traffic at the end of one window and the beginning of another, it does not exceed the limit. I differentiated between different types of endpoints because not all endpoints are created equal. It is cheap to read data, but it is expensive to create playlists (writes) and do batch analysis.</p><pre>class EndpointType(Enum):<br>    READ = &quot;read&quot;           # Higher limit (80 req/30s)<br>    WRITE = &quot;write&quot;         # Lower limit (30 req/30s)<br>    BATCH = &quot;batch&quot;         # Strict limit (20 req/30s)</pre><p>The core logic uses asyncio.Lock to ensure thread safety and a deque to track timestamps of recent calls.</p><pre>async def acquire(self, endpoint_type: EndpointType = EndpointType.READ) -&gt; None:<br>    async with self._locks[endpoint_type]:<br>        now = datetime.now(timezone.utc)<br>        call_times = self.call_times[endpoint_type]<br>        <br>        cutoff = now.timestamp() - self.window_seconds<br>        while call_times and call_times[0] &lt; cutoff:<br>            call_times.popleft()<br>        <br><br>        if len(call_times) &gt;= limit:<br>            oldest = call_times[0]<br>            wait_time = (oldest + self.window_seconds) - now.timestamp()<br>            if wait_time &gt; 0:<br>                await asyncio.sleep(wait_time + 0.1)</pre><p>This ensures we never proactively hit a rate limit. But what if we do?</p><h4>Adaptive Backoff</h4><p>We also handle 429s reactively. If Spotify tells us to slow down, we don’t just wait; we increase a backoff_multiplier</p><pre>async def report_429(self) -&gt; int:<br>    async with self._global_lock:<br>        self._consecutive_429s += 1<br>        #capped at 4x<br>        self._backoff_multiplier = min(4.0, 1.0 + (self._consecutive_429s * 0.5))<br>        return min(60, 5 * self._consecutive_429s)</pre><p>This multiplier dynamically reduces our internal limits, allowing the system to “cool down” automatically.</p><h3>2. Robustness with Retry Patterns</h3><p>Network glitches happen. APIs time out. To make the system resilient, I employed two layers of retry logic.</p><h4>2.1. Library-Level Retries (Tenacity)</h4><p>For the Gemini AI integration, I used the tenacity library. It provides a clean, decorative way to handle retries with exponential backoff.</p><pre>@retry(<br>    stop=stop_after_attempt(5),<br>    wait=wait_exponential(multiplier=1, min=2, max=60),<br>    retry=retry_if_exception_type((httpx.HTTPError, Exception)),<br>    reraise=True<br>)<br>async def detect_languages(self, tracks: List[Track]) -&gt; BatchLanguageResult:<br>    # We will handle our calls here</pre><p>This elegantly handles transient failures without cluttering the business logic.</p><h3>2.2. Explicit Logic Retries</h3><p>For Spotify, where we need tight integration with our rate limiter, I wrote explicit retry loops. This gives us fine-grained control to acquire a rate limit token before every retry attempt.</p><pre>for attempt in range(max_retries):<br>    await self.rate_limiter.acquire(EndpointType.READ)<br>    response = await self.client.get(...)<br>    <br>    if response.status_code == 429:<br>        retry_after = await self._handle_rate_limit(response)<br>        await asyncio.sleep(retry_after)<br>        continue<br>        <br>    if response.status_code == 200:<br>        return response.json()</pre><h3>3. Security First</h3><p>Security isn’t an afterthought; it’s baked into the architecture.</p><h4>OAuth State Management</h4><p>A common vulnerability in OAuth flows is CSRF (Cross-Site Request Forgery). To prevent this, I generate a unique state parameter for every login request.</p><p>Crucially, <strong>I don’t just rely on cookies.</strong> I store the state in Firestore with a strict TTL (Time To Live) and bind it to the user’s ID.</p><pre>state = f&quot;{current_user[&#39;uid&#39;]}:{secrets.token_urlsafe(32)}&quot;<br>await firebase.store_oauth_state(state, current_user[&#39;uid&#39;], ttl_minutes=10)</pre><p>When the callback comes back, we verify the state against the database. This ensures:</p><ol><li>The request originated from us.</li><li>It’s for the correct user.</li><li>It’s recent (prevention of replay attacks).</li></ol><h4>API Security &amp; CORS</h4><p>The FastAPI backend uses slowapi to rate limit API endpoints by IP address (@limiter.limit(&quot;20/minute&quot;)), protecting against DDoS and abuse.</p><p>CORS (Cross-Origin Resource Sharing) is configured strictly. In production, we only allow requests from our specific frontend domain, preventing malicious sites from calling our API on a user’s behalf.</p><pre>allowed_origins = [settings.frontend_url, &quot;https://spotify-organiser.web.app&quot;]</pre><h3>4. Lazy Loading &amp; Fallbacks with LLMs</h3><p>Using LLMs (like Gemini 2.0 Flash) for song classification is powerful but introduces latency and cost. To optimize this, I used a “Check, then Ask” approach.</p><h4>The Heuristic Fallback</h4><p>Before calling the LLM, or if the LLM fails/times out, the system falls back to a deterministic heuristic engine.</p><pre>def _fallback_language_detection(self, tracks: List[Track]):<br>    if any(ind in combined_text for ind in [&quot;arijit&quot;, &quot;badshah&quot;, &quot;t-series&quot;]):<br>        return &quot;Hindi&quot;<br>    <br>    if &quot;instrumental&quot; in track.name.lower():<br>        return &quot;Instrumental&quot;<br>        <br>    # Default to English if confident metrics are low<br>    return &quot;English&quot;</pre><p>Such a hybrid model greatly reduces the cost of the API and enhances the performance, as the most frequent scenarios are handled immediately without calling the API.</p><h3>Conclusion</h3><p>A “simple” application can often have complex engineering needs. By adding robust rate limiting, retry mechanisms, and security, I’ve been able to develop a backend that not only works, but is production-ready.</p><p>The incorporation of LLMs in classification brings in intelligence that would not have been possible with static rules, and the fallback options make sure that this system works even when it’s not possible to connect to the AI. If you’re interested in the code or would like to give it a shot, take a look at the project here:</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42681c7b6463" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>