<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ashiya Ajare on Medium]]></title>
        <description><![CDATA[Stories by Ashiya Ajare on Medium]]></description>
        <link>https://medium.com/@ashiya.ajare?source=rss-fdc527338328------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*V_A8Qr0tElsVLGbWzfvYRw.jpeg</url>
            <title>Stories by Ashiya Ajare on Medium</title>
            <link>https://medium.com/@ashiya.ajare?source=rss-fdc527338328------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 05:56:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ashiya.ajare/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Illusion of the Single Task: What Actually Goes Into a Finished Project]]></title>
            <link>https://medium.com/@ashiya.ajare/the-illusion-of-the-single-task-what-actually-goes-into-a-finished-project-216a7060d858?source=rss-fdc527338328------2</link>
            <guid isPermaLink="false">https://medium.com/p/216a7060d858</guid>
            <dc:creator><![CDATA[Ashiya Ajare]]></dc:creator>
            <pubDate>Tue, 27 Oct 2026 14:45:18 GMT</pubDate>
            <atom:updated>2026-10-27T14:45:18.215Z</atom:updated>
            <content:encoded><![CDATA[<p>Have you ever looked at a finished piece of work — a polished report, a functional web page, a design layout, or even a detailed spreadsheet — and wondered, <em>How long could that possibly have taken to make?</em></p><p>From the outside, the answer usually looks deceptively simple. You see the end result, you estimate the time it took to type the words or arrange the elements, and you assume the process was a straight line from start to finish. You sit down, you do the work, and you hand it over.</p><p>Except that is rarely how it actually happens.</p><p>If you ask anyone who spends their days building, writing, analyzing, or designing, they will tell you that the visible output is only the tip of the iceberg. The actual execution — the part where you type, draw, code, or calculate — is often just a fraction of the total effort.</p><p>Most of the real work happens in the messy, unglamorous spaces before you start typing, while you are figuring things out mid-stream, and long after you thought the job was finished. Let’s walk through what a project actually looks like from the moment a request lands on your desk to the day it finally goes out into the world, using a common workplace deliverable — say, putting together a comprehensive project overview report — as our guide.</p><h3>1. Figuring Out What Is Actually Being Asked</h3><p>The journey of any project begins with a conversation or a message that usually sounds deceptively straightforward: <em>“Can you put together a quick report on how our recent updates are performing?”</em></p><p>At first glance, this sounds like a clear instructions. But if you open a blank document and start writing right away, you are setting yourself up for trouble.</p><p>Words like “quick report,” “overview,” or “summary” mean entirely different things to different people. To your manager, it might mean a one-page bulleted list of high-level numbers. To a client, it might mean a twenty-page deep dive with historical comparisons, charts, and a breakdown of methodology.</p><p>This is where the invisible work begins. Before a single sentence is written, you have to decode the request. You start asking questions: Who is going to read this? What decision are they trying to make with these numbers? Do they want raw data, or do they want interpretations and recommendations?</p><p>Clarifying these details early feels like slowing down, but it is actually the only way to avoid spending three days building the wrong thing entirely.</p><h3>2. Breaking the Mountain Into Molehills</h3><p>Once you actually understand the goal, the blank page or empty canvas can feel a bit intimidating. A request to “write a report” or “build a presentation” is too big for your brain to process all at once.</p><p>If you try to tackle the whole thing in one giant block of effort, you will likely get overwhelmed or stall out. This is why breaking things down is essential. You start dividing the vague request into smaller, concrete pieces.</p><p>For our report example, that might look like:</p><ul><li>Gathering the raw performance metrics from last quarter.</li><li>Pulling together qualitative feedback notes from the team.</li><li>Structuring an outline with three main sections: wins, challenges, and next steps.</li><li>Drafting the individual sections one by one.</li></ul><p>By chopping the mountain into manageable molehills, you create a path forward. You also start spotting dependencies — the things you cannot do until someone else gives you information, or the data points you realize are currently missing.</p><h3>3. The Research Trap: Finding What You Need</h3><p>With a rough outline in place, you move on to gathering your materials. This is where you dig through old files, check past archives, run queries, or read up on background context.</p><p>Research is vital, but it also comes with a hidden hazard: the rabbit hole.</p><p>It is remarkably easy to spend four hours reading tangentially related articles, sorting through disorganized folders, or looking for the <em>perfect</em> data set, only to realize you have completely run out of time to actually write the report.</p><p>Good preparation means knowing when you have enough information to move forward. You do not need to know everything in the universe about a topic; you just need enough context to speak accurately and address the problem at hand. Finding that boundary — knowing when to stop researching and start building — is a skill learned mostly through trial and error.</p><h3>4. Doing the Actual Work (And Watching the Plan Change)</h3><p>Eventually, you arrive at the part everyone thinks of as “the work.” You open the file and start putting the pieces together.</p><p>You write the paragraphs, build the charts, format the tables, and weave the narrative. For a brief moment, everything feels like it is flowing.</p><p>Then reality hits.</p><p>About halfway through, you realize that the second section completely contradicts a point you made in the first section. Or you discover that one of your key data sources is outdated, meaning you have to recalculate a crucial metric. Or perhaps the narrative structure you planned just does not work when you see it written down.</p><p>This is completely normal. No project ever goes exactly according to the initial plan. Execution is not a matter of simply transcribing thoughts from your head onto the page; it is a constant process of discovery, problem-solving, and course-correction as the work reveals its own complexities.</p><h3>5. The First Draft Is Never the Final Draft</h3><p>When you finally type the last sentence or place the final graphic, there is a temptation to hit send and call it a day. After all, the task is done, right?</p><p>Not quite. What you are looking at is not a finished product; it is a first draft. And first drafts are notoriously messy.</p><p>This is where you switch hats from <em>creator</em> to <em>editor</em>. You put the document aside for an hour — or better yet, overnight — and come back with fresh eyes.</p><p>Now you start checking the things that easily slip through the cracks during the creative phase:</p><ul><li><strong>Accuracy:</strong> Did you type that number correctly, or did a decimal point wander off?</li><li><strong>Completeness:</strong> Did you actually answer the original question asked at the very beginning?</li><li><strong>Consistency:</strong> Are your headings styled the same way? Does the tone shift halfway through?</li><li><strong>Clarity:</strong> Can someone who has no idea what you were thinking understand this sentence?</li></ul><p>Catching your own errors saves everyone else a lot of time later. It is the invisible polish that turns a collection of thoughts into a professional deliverable.</p><h3>6. Feedback, Collaboration, and Revisions</h3><p>Once you are happy with your polished draft, you share it with your team, your manager, or your client.</p><p>For people early in their careers, getting feedback can sometimes feel personal. If someone suggests changing a section or points out an error, it is easy to feel like your first attempt was a failure.</p><p>In reality, feedback is simply the next phase of the process. No single person has all the context, and fresh perspectives almost always expose blind spots you couldn’t see because you were too close to the work.</p><p>Imagine you spent days crafting a detailed breakdown of quarterly metrics, feeling quite proud of the analytical depth. When you hand it over, the reviewer says, <em>“This is great data, but the executive team only has five minutes for this meeting. Can we pull the three main takeaways right to the top and turn the rest into an appendix?”</em></p><p>That doesn’t mean your work was bad. It means the context shifted slightly, and the deliverable needs to be shaped to fit the actual environment it is entering. Revisions are where good work often turns into great work.</p><h3>7. The Final Mile: Delivery and Packaging</h3><p>Let’s say the revisions are done, everyone is happy, and the green light has been given. Are you finished?</p><p>Almost. There is still the final mile of delivery.</p><p>How you hand something over matters just as much as what you created. If you dump a massive, unlabeled file into a shared folder with no context, you create unnecessary friction for the people receiving it.</p><p>Before final delivery, thoughtful professionals take care of the administrative wrap-up:</p><ul><li>Giving files clear, logical names (so no one has to guess what Report_Final_v3_reallyfinal.pdf means).</li><li>Organizing supporting documents or source files so someone else can find them if needed.</li><li>Writing a brief summary email explaining what is attached, where things stand, and what (if anything) needs to happen next.</li></ul><p>Good packaging ensures that your work lands smoothly and can actually be used effectively by whoever asked for it.</p><h3>8. What Happens After Delivery?</h3><p>The project is handed over, the meeting is done, and the file is saved. Now it’s truly over, right?</p><p>Usually, there is a quiet aftermath. A few days later, someone might ask a follow-up question about a specific data point. Or a new piece of information arrives that slightly alters the conclusions you drew.</p><p>Beyond those immediate ripples, there is also the internal retrospective — the mental note you take about what went well and what felt unnecessarily difficult. Did you spend too much time hunting down data? Was the initial brief too vague? Did you underestimate how long the formatting would take?</p><p>These reflections do not usually happen in a formal meeting. They happen quietly in the back of your mind, shaping how you approach the next request that lands on your desk.</p><h3>The Part People Usually Don’t See</h3><p>When we look at our own work or the work of others, we tend to focus exclusively on the visible output. We measure productivity by visible speed — how fast can we type, how quickly can we fill the page?</p><p>But meaningful work is rarely just about speed. It is about the judgment calls made in the quiet moments between receiving a request and delivering the result:</p><ul><li>The clarifying questions that prevented a week of wasted effort.</li><li>The boundaries drawn during research to keep the project on track.</li><li>The mid-stream pivots when an initial assumption turned out to be wrong.</li><li>The careful editing and formatting that makes the final piece easy for others to digest.</li><li>The willingness to take feedback constructively and refine the output.</li></ul><p>A finished project is not a single task checked off a list. It is an entire ecosystem of decisions, revisions, coordination, and problem-solving.</p><p>Next time you look at a polished report, a clean design, or a well-executed plan, remember the invisible journey it took to get there. It wasn’t just made; it was built, piece by thoughtful piece.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=216a7060d858" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Hidden Engineering Problem in AI ApplicationsManaging State in Asynchronous Systems]]></title>
            <link>https://medium.com/@ashiya.ajare/the-hidden-engineering-problem-in-ai-applicationsmanaging-state-in-asynchronous-systems-d739a1373e15?source=rss-fdc527338328------2</link>
            <guid isPermaLink="false">https://medium.com/p/d739a1373e15</guid>
            <dc:creator><![CDATA[Ashiya Ajare]]></dc:creator>
            <pubDate>Mon, 21 Sep 2026 12:17:47 GMT</pubDate>
            <atom:updated>2026-09-21T12:17:47.867Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why race conditions, stale state, and loosely coordinated components become the real complexity of modern AI applications.</h3><p>When building modern AI applications, prototype demos lie to us. A naive script wraps an API call: take user input, send it to a model, await the text stream, and return it. It feels simple.</p><p>In production, that single prompt transforms into a multi-tiered, asynchronous distributed system. A real request looks more like this:</p><p>Client $\rightarrow$ API Gateway $\rightarrow$ Auth $\rightarrow$ Task Queue $\rightarrow$ Worker $\rightarrow$ LLM Provider $\rightarrow$ Database/Cache $\rightarrow$ Vector DB $\rightarrow$ External Tools $\rightarrow$ Client Response</p><p>Suddenly, you aren’t writing a simple wrapper around an inference endpoint. You are building and operating a distributed pipeline.</p><p>The hardest problems in these systems rarely stem from model accuracy or prompt context windows. They stem from classic distributed engineering issues: <strong>state, concurrency, timing, coordination, and failure recovery.</strong></p><h3>1. A Deceptively Simple Scenario</h3><p>Consider a common workflow: an AI agent generates a summary report for a user workspace, pulls context from an external vector index, updates a database record, and sends a notification.</p><p>Here is how that request unfolds across services:</p><pre>Time  Client / API Gateway           Background Worker              Redis / DB<br>----  --------------------           -----------------              ----------<br>T1    POST /generate-report -------&gt; Enqueues Job (Job #42)<br>                                     Status: PENDING ---------------&gt; Save (Job #42 = PENDING)</pre><pre>T2    Receives 202 Accepted<br>      (Polls or opens WebSocket)</pre><pre>T3                                   Pulls Job #42<br>                                     Calls LLM (Latent period: 4s)</pre><pre>T4    User edits workspace context<br>      POST /workspace/update --------------------------------------&gt; Save (Context v2)</pre><pre>T5                                   LLM completes (using Context v1)<br>                                     Overwrites report output ------&gt; Save (Report v1, Job #42 = DONE)</pre><pre>T6    App displays outdated Report v1 based on pre-edit context, while DB context is v2.</pre><p>Look at each piece in isolation:</p><ul><li>The API successfully accepted the request and enqueued Job #42.</li><li>The background worker executed its steps without throwing an exception.</li><li>The user’s update at T4 executed a valid mutation.</li><li>The database accepted both writes.</li></ul><p>Every component logged HTTP 200s or clean execution traces. Yet the system state is wrong: the final generated report reflects Context v1, but the UI and backend claim Job #42 ran successfully against the current workspace state (Context v2).</p><p>The problem wasn’t a thrown exception. It was a failure of state synchronization across time boundaries.</p><h3>2. Why AI Applications Compound State Complexity</h3><p>Traditional web applications perform synchronous, transactional database mutations within short-lived HTTP request cycles. A request arrives, opens a database transaction, updates a row, commits, and returns a response within 100 milliseconds.</p><p>AI applications break this paradigm:</p><ul><li><strong>High-latency execution boundaries:</strong> Model inference calls routinely take anywhere from 1.5 to 30+ seconds.</li><li><strong>Non-deterministic processing times:</strong> Execution length scales with token output, dynamic tool usage, and provider rate-limiting backpressure.</li><li><strong>Multi-step, loosely coupled workflows:</strong> An agent execution isn’t a single call; it’s a loop that invokes vector retrievals, multiple model inferences, code execution sandboxes, and webhooks.</li><li><strong>Streaming constraints:</strong> Applications stream partial state (tokens) directly to clients while simultaneously managing durable state transitions on the backend.</li></ul><pre>[HTTP Client] <br>     │ (Streaming WebSocket / SSE connection)<br>     ▼<br>[API Server] ──(Enqueue)──► [Queue / Redis] ◄──(Fetch)── [Worker Service]<br>                                                               │<br>                                         ┌─────────────────────┼─────────────────────┐<br>                                         ▼                     ▼                     ▼<br>                                   [LLM Provider]        [Vector Index]      [External Tools]<br>                                   (Latent Stream)        (RAG Search)        (Webhooks/APIs)</pre><p>Adding an LLM introduces asynchronous boundaries. Each step in the diagram above introduces a point where local state can drift from global state, network connections can drop, or concurrent operations can collide.</p><h3>3. Data vs. State</h3><p>A common point of confusion in backend architectures is confusing <strong>data</strong> with <strong>state</strong>.</p><ul><li><strong>Data</strong> represents facts: raw information, payloads, records, and history. It tells you <em>what you know</em>.</li><li><strong>State</strong> represents execution status: operational position, phase, and flags. It tells you <em>what the system currently believes is happening</em>.</li></ul><pre># This is Data:<br>{<br>    &quot;user_id&quot;: &quot;usr_992&quot;,<br>    &quot;prompt&quot;: &quot;Analyze sales Q3&quot;,<br>    &quot;retrieved_documents&quot;: [&quot;doc_12&quot;, &quot;doc_88&quot;],<br>    &quot;created_at&quot;: &quot;2026-09-21T10:00:00Z&quot;<br>}</pre><pre># This is State:<br>{<br>    &quot;job_id&quot;: &quot;job_8831&quot;,<br>    &quot;session_id&quot;: &quot;sess_401&quot;,<br>    &quot;status&quot;: &quot;AWAITING_TOOL_OUTPUT&quot;,<br>    &quot;current_step&quot;: 3,<br>    &quot;last_processed_event_id&quot;: &quot;evt_1049&quot;,<br>    &quot;version&quot;: 4<br>}</pre><p>When systems fail to separate data from state, they tend to rely on implicit state — inferring status by inspecting data attributes:</p><pre># Anti-pattern: Implicit state derived from data attributes<br>if job.response_text is None and job.error_message is None:<br>    # &quot;We must be running!&quot; ...or the worker crashed before setting error_message.<br>    process_job(job)</pre><p>Explicitly modeling state forces you to formalize valid operational phases, state transitions, and state ownership.</p><h3>4. Race Conditions in Asynchronous Workflows</h3><p>Async engines (like Python’s asyncio or Node.js) allow single-threaded concurrency. They do not prevent state-level race conditions.</p><p>Consider three classic race scenarios common in AI applications:</p><h3>Scenario A: The Stale Overwrite (Lost Update)</h3><pre>T1: Request A reads Job state (version 1)<br>T2: Request B reads Job state (version 1)<br>T3: Request B updates metadata, increments to version 2, and commits.<br>T4: Request A completes long calculation, writes back state based on version 1 data.<br>Result: Request B&#39;s update is wiped out.</pre><h3>Scenario B: Out-of-Order Event Delivery</h3><p>In an event-driven system streaming token updates or agent step completion over a message broker (e.g., RabbitMQ, Kafka, or NATS), network jitter can cause event inversion:</p><pre>Worker sends Event 1 (Step 1 Complete: &quot;Thinking...&quot;)<br>Worker sends Event 2 (Step 2 Complete: &quot;Searching Vector DB...&quot;)</pre><pre>Client receives Event 2 FIRST -&gt; Updates UI to &quot;Searching Vector DB...&quot;<br>Client receives Event 1 SECOND -&gt; Overwrites UI to &quot;Thinking...&quot;<br>Result: The UI displays an earlier state permanently.</pre><h3>Scenario C: Concurrent Agent Execution</h3><p>A user clicks “Re-run Prompt” twice in rapid succession.</p><pre># Unsafe handler<br>async def handle_rerun(session_id: str):<br>    session = await db.get_session(session_id)<br>    # The worker starts processing prompt B while prompt A is still streaming context modifications<br>    asyncio.create_task(run_agent_pipeline(session))</pre><p>If run_agent_pipeline mutates shared session history array in-place without optimistic locking, the second pipeline read interleaves with the first pipeline write. The prompt history array becomes corrupted with duplicated or out-of-order messages.</p><h3>5. The “Everything Worked” Debugging Problem</h3><p>State bugs in asynchronous systems are notoriously tricky to debug because they rarely raise explicit exceptions.</p><p>Imagine investigating a bug report: <em>“The AI assistant lost track of the summary history midway through my session.”</em></p><p>You check the logs:</p><pre>2026-09-21 14:02:01.102 [INFO] API: Accepted generation request req_01<br>2026-09-21 14:02:01.105 [INFO] Queue: Task task_99 pushed successfully<br>2026-09-21 14:02:01.400 [INFO] Worker-02: Picked up task task_99<br>2026-09-21 14:02:05.811 [INFO] LLM Gateway: 200 OK (2048 tokens returned)<br>2026-09-21 14:02:05.850 [INFO] Worker-02: Task task_99 completed cleanly</pre><p>Every service reports a success status. There are no tracebacks, no 500 errors, and no queue dropouts.</p><p>Yet, when you look at the database, the conversation context contains corrupted data.</p><h3>The Observer Effect in Async Debugging</h3><p>When you try to reproduce these issues locally, you might attach a debugger or add logging statements:</p><pre>async def process_step(state):<br>    logger.info(f&quot;DEBUG: Current state before step: {state}&quot;)<br>    # Adding this logger call or setting a breakpoint adds a 5ms delay<br>    # That 5ms delay completely changes the timing window!<br>    result = await call_llm(state)<br>    await save_state(result)</pre><p>The bug vanishes. This happens because logging or stepping through code alters task scheduling timing, closing the narrow race condition window.</p><p>To fix these types of issues, avoid relying solely on step-by-step logs. Instead, check for structural patterns like missing atomic transactions, non-isolated reads, or implicit state checks.</p><h3>6. State Machines as an Engineering Tool</h3><p>Scatter-shot boolean flags are a recipe for bugs:</p><pre># Avoid this pattern<br>class AgentJob:<br>    is_processing: bool = False<br>    is_ready: bool = False<br>    is_completed: bool = False<br>    is_cancelled: bool = False<br>    has_failed: bool = False</pre><p>With five booleans, your system mathematically allows $2⁵ = 32$ possible states. What does is_processing = True combined with is_cancelled = True and is_completed = True mean?</p><p>Explicit Finite State Machines (FSMs) solve this by defining fixed states and allowed transitions:</p><pre>┌──────────────┐<br>                  │ INITIALIZING │<br>                  └──────┬───────┘<br>                         │<br>                         ▼<br>                  ┌──────────────┐<br>       ┌─────────►│    READY     ├──────────┐<br>       │          └──────┬───────┘          │<br>       │                 │                  │<br>       │                 ▼                  │<br>┌──────┴───────┐  ┌──────────────┐  ┌───────▼──────┐<br>│  RETRYING    │◄─┤  PROCESSING  ├──► CANCELLED   │<br>└──────────────┘  └──────┬───────┘  └──────────────┘<br>                         │<br>                         ▼<br>                  ┌──────────────┐<br>                  │  COMPLETED   │<br>                  └──────────────┘</pre><h3>Implementing a Strict FSM with Optimistic Locking</h3><p>Here is an example using Python and SQLAlchemy to enforce state transitions and avoid silent race conditions:</p><pre>from enum import Enum<br>import uuid<br>from sqlalchemy import Column, String, Integer, Enum as SQLEnum<br>from sqlalchemy.orm import declarative_base</pre><pre>Base = declarative_base()</pre><pre>class JobState(str, Enum):<br>    INITIALIZING = &quot;INITIALIZING&quot;<br>    READY = &quot;READY&quot;<br>    PROCESSING = &quot;PROCESSING&quot;<br>    COMPLETED = &quot;COMPLETED&quot;<br>    FAILED = &quot;FAILED&quot;<br>    CANCELLED = &quot;CANCELLED&quot;</pre><pre>class IllegalStateTransition(Exception):<br>    pass</pre><pre>class StaleStateError(Exception):<br>    pass</pre><pre># Map each state to its allowed next states<br>VALID_TRANSITIONS = {<br>    JobState.INITIALIZING: {JobState.READY, JobState.FAILED},<br>    JobState.READY: {JobState.PROCESSING, JobState.CANCELLED},<br>    JobState.PROCESSING: {JobState.COMPLETED, JobState.FAILED, JobState.CANCELLED},<br>    JobState.COMPLETED: set(),<br>    JobState.FAILED: {JobState.READY}, # Allowed if retrying<br>    JobState.CANCELLED: set(),<br>}</pre><pre>class JobModel(Base):<br>    __tablename__ = &quot;agent_jobs&quot;</pre><pre>    id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))<br>    state = Column(SQLEnum(JobState), nullable=False, default=JobState.INITIALIZING)<br>    version = Column(Integer, nullable=False, default=1)</pre><pre>    def transition_to(self, new_state: JobState):<br>        allowed = VALID_TRANSITIONS.get(self.state, set())<br>        if new_state not in allowed:<br>            raise IllegalStateTransition(<br>                f&quot;Cannot transition from {self.state} to {new_state}&quot;<br>            )<br>        self.state = new_state<br>        self.version += 1</pre><p>Python</p><pre>async def transition_job_db(db_session, job_id: str, current_version: int, target_state: JobState):<br>    # Atomic conditional update: update succeeds ONLY if version hasn&#39;t changed<br>    result = await db_session.execute(<br>        update(JobModel)<br>        .where(JobModel.id == job_id, JobModel.version == current_version)<br>        .values(<br>            state=target_state,<br>            version=JobModel.version + 1<br>        )<br>    )<br>    if result.rowcount == 0:<br>        # Another process updated this job first<br>        raise StaleStateError(f&quot;Job {job_id} was modified concurrently.&quot;)</pre><h3>7. Events vs. State</h3><p>A common source of system bugs is confusing <strong>events</strong> with <strong>state</strong>.</p><ul><li><strong>Event:</strong> An immutable record of an action that occurred in the past (“Something happened”).</li><li><strong>State:</strong> The current consolidated truth (“What is true right now”).</li></ul><pre>Event: &quot;TokenChunkGenerated&quot;  │<br>Event: &quot;ToolCallRequested&quot;    │ ─── (Processed by Engine) ───► State: job.status = AWAITING_TOOL<br>Event: &quot;ToolCallCompleted&quot;    │</pre><h3>The Ingestion Trap</h3><p>Relying on events directly as your application state can cause issues:</p><ul><li><strong>Events can arrive out of order.</strong></li><li><strong>Events can be delivered twice</strong> (at-least-once delivery semantics).</li><li><strong>Events can be delayed.</strong></li></ul><p>If an application component treats receiving an event as an immediate trigger to mutate state without checking order or state transitions, bugs will occur.</p><h3>Idempotency and Event Handling</h3><p>State mutations driven by events must be <strong>idempotent</strong>. Processing the exact same event multiple times should yield the same final state.</p><pre>async def handle_tool_completed_event(db, event: Event):<br>    async with db.begin():<br>        job = await db.get_for_update(event.job_id)<br>        <br>        # Guard 1: Idempotency check via sequence/version tracking<br>        if job.last_processed_event_sequence &gt;= event.sequence_number:<br>            logger.info(f&quot;Skipping duplicate/older event {event.id}&quot;)<br>            return</pre><pre>        # Guard 2: State transition validity<br>        if job.state != JobState.PROCESSING:<br>            logger.warning(f&quot;Ignored event {event.id}: Job is in state {job.state}&quot;)<br>            return</pre><pre>        # Apply state changes safely<br>        job.apply_tool_result(event.payload)<br>        job.last_processed_event_sequence = event.sequence_number<br>        await db.commit()</pre><h3>8. Conversation and Session State</h3><p>Storing chat context in an AI application involves more than appending JSON strings to a database array.</p><p>A realistic conversation state spans multiple layers, each with its own lifespan and mutation rules:</p><pre>┌─────────────────────────────────────┐<br>                  │            Global Context           │<br>                  │   (System Prompts, Organization)    │<br>                  └──────────────────┬──────────────────┘<br>                                     │<br>                                     ▼<br>                  ┌─────────────────────────────────────┐<br>                  │            Persistent State         │<br>                  │   (User Profile, Saved Preferences) │<br>                  └──────────────────┬──────────────────┘<br>                                     │<br>                                     ▼<br>                  ┌─────────────────────────────────────┐<br>                  │            Session State            │<br>                  │  (Active Workspace, Active Tools)   │<br>                  └──────────────────┬──────────────────┘<br>                                     │<br>                                     ▼<br>                  ┌─────────────────────────────────────┐<br>                  │           Short-Term Context        │<br>                  │ (Window of recent N messages, RAG)  │<br>                  └──────────────────┬──────────────────┘<br>                                     │<br>                                     ▼<br>                  ┌─────────────────────────────────────┐<br>                  │           Derived State             │<br>                  │  (Summaries, Token Counts, Embeds)  │<br>                  └─────────────────────────────────────┘</pre><h3>The Danger of Cache and Invalidation Inconsistency</h3><p>To minimize database overhead, engineers often keep short-term context in Redis, while persisting long-term messages in Postgres.</p><pre># Anti-pattern: Dual writes without synchronization<br>async def append_user_message(session_id: str, message: Message):<br>    # Write 1: Update Postgres<br>    await postgres.save_message(session_id, message)<br>    <br>    # What if the process crashes RIGHT HERE?<br>    <br>    # Write 2: Update Redis context cache<br>    cached_context = await redis.get(session_id)<br>    cached_context.append(message)<br>    await redis.set(session_id, cached_context)</pre><p>If the worker drops between Write 1 and Write 2, the cache and primary database drift apart. The LLM will operate on stale context, ignoring the user’s newest message.</p><p><strong>Core rule:</strong> Treat memory caches strictly as invalidatable read models. Write to your durable database first, and either invalidate the cache key or use atomic cache-rebuilding pipelines.</p><h3>9. Redis, Databases, and the Search for Source of Truth</h3><p>A standard infrastructure layout for asynchronous AI platforms looks like this:</p><ul><li><strong>PostgreSQL / MySQL:</strong> Durable Source of Truth.</li><li><strong>Redis:</strong> Fast transient state, pub/sub streaming, distributed locks, rate limiting.</li><li><strong>Celery / Temporal / Asynq / RQ workers:</strong> Off-thread processing tasks.</li><li><strong>API Framework:</strong> Incoming routing and coordination.</li></ul><h3>10. Cancellation is a Correctness Requirement</h3><p>Cancellation is often treated as a minor user experience detail. In async AI systems, it is critical for system stability and state correctness.</p><p>If a user aborts an execution or closes their browser tab, and the backend ignores the signal:</p><ul><li>The worker continues running expensive LLM calls, draining your API budget.</li><li>The worker completes its processing and writes stale updates back to the database.</li><li>The user starts a new task, but the background worker for the abandoned task overwrites the new data.</li></ul><pre>User                      API Gateway                   Worker                    LLM Provider<br> │                            │                           │                            │<br> │── (1) POST /agent/run ────►│                           │                            │<br> │                            │── (2) Enqueue Task ──────►│                            │<br> │                            │                           │── (3) Start Long Inference►│<br> │                            │                           │                            │<br> │── (4) DISCONNECT / ABORT ─►│                           │                            │<br> │                            │── (5) Propagation Signal? │                            │<br> │                            │   (Missing!)              │                            │<br> │                            │                           │◄─ (6) Returns expensive ───│<br> │                            │                           │       token completion     │<br> │                            │                           │                            │<br> │                            │                           │── (7) Overwrites state ────│<br> │                            │                           │       with stale data!     │</pre><p>Distributed systems will encounter infrastructure failures. When building asynchronous AI pipelines, plan for these scenarios:</p><ul><li><strong>Inference endpoints timeout</strong> or return HTTP 429/503 errors.</li><li><strong>Workers crash</strong> mid-execution (e.g., node eviction, out-of-memory errors).</li><li><strong>Network connections drop</strong> mid-stream.</li><li><strong>Queues deliver messages multiple times.</strong></li></ul><h3>Essential Failure Patterns</h3><ul><li><strong>Correlation IDs:</strong> Pass a unique correlation_id through every queue, event, tool call, and database transaction. This lets you trace execution across independent services.</li><li><strong>Dead-Letter Queues (DLQ):</strong> Poison-pill requests (prompts that consistently cause worker crashes or parsing errors) must be routed to a DLQ after $N$ failed attempts to avoid blocking execution queues.</li><li><strong>Exponential Backoff with Jitter:</strong> When retrying failed LLM calls, apply randomized delay jitter to prevent thundering herd spikes against rate-limited endpoints.</li></ul><pre>import random<br>import asyncio</pre><pre>async def call_llm_with_backoff(prompt: str, max_retries: int = 5):<br>    base_delay = 1.0  # seconds<br>    max_delay = 30.0</pre><pre>    for attempt in range(max_retries):<br>        try:<br>            return await llm_client.generate(prompt)<br>        except Exception as err: # Network error, Rate limit, 503<br>            if attempt == max_retries - 1:<br>                raise err<br>            <br>            # Calculate backoff with jitter<br>            delay = min(max_delay, base_delay * (2 ** attempt))<br>            jitter = random.uniform(0, delay * 0.1) # 10% jitter<br>            total_sleep = delay + jitter<br>            <br>            logger.warning(f&quot;Attempt {attempt + 1} failed. Retrying in {total_sleep:.2f}s...&quot;)<br>            await asyncio.sleep(total_sleep)</pre><h3>12. A Practical Debugging Methodology</h3><p>When you encounter an unexplained state issue in an asynchronous AI system, follow this systematic debugging process:</p><pre>┌────────────────────────────────────────────────────────┐<br>│ 1. Identify the State Owner                            │<br>│    (Which store is authoritative for this field?)     │<br>└───────────────────────────┬────────────────────────────┘<br>                            │<br>                            ▼<br>┌────────────────────────────────────────────────────────┐<br>│ 2. Audit Mutators                                      │<br>│    (List every background task/API that can write to it)│<br>└───────────────────────────┬────────────────────────────┘<br>                            │<br>                            ▼<br>┌────────────────────────────────────────────────────────┐<br>│ 3. Correlate Events                                    │<br>│    (Filter logs by correlation_id across services)     │<br>└───────────────────────────┬────────────────────────────┘<br>                            │<br>                            ▼<br>┌────────────────────────────────────────────────────────┐<br>│ 4. Reconstruct Timeline                                │<br>│    (Sort operations by microsecond timestamps)        │<br>└───────────────────────────┬────────────────────────────┘<br>                            │<br>                            ▼<br>┌────────────────────────────────────────────────────────┐<br>│ 5. Identify Anomaly                                    │<br>│    - Out-of-order writes? - Duplicate task delivery?   │<br>│    - Stale cache read?    - Unhandled cancellation?    │<br>└────────────────────────────────────────────────────────┘</pre><h3>Case Study: Reconstructing an Out-of-Order Execution Bug</h3><p><strong>Symptom:</strong> A conversation agent intermittently responds to old context, ignoring the user’s latest prompt message.</p><p><strong>Correlated Log Analysis (by </strong>correlation_id = corr_8832a<strong>):</strong></p><pre>[10:15:00.100] API        INFO  Received user prompt: &quot;No, change target color to Blue&quot; (Msg ID: msg_2)<br>[10:15:00.105] API        INFO  Enqueued task_worker_process (Msg ID: msg_2)<br>[10:15:00.110] WORKER-A   INFO  Picked up task_worker_process for msg_2<br>[10:15:00.112] WORKER-A   INFO  Fetching context history from Redis...<br>[10:15:00.115] WORKER-A   INFO  Retrieved 1 message from Redis (msg_1)<br>[10:15:00.120] API        INFO  Postgres write completed for msg_2<br>[10:15:00.122] API        INFO  Redis context cache updated with msg_2</pre><p><strong>The Bug Uncovered:</strong></p><p>Look closely at microsecond timestamps 10:15:00.112 through 10:15:00.122:</p><ul><li>WORKER-A read context from Redis at .115—<strong>before</strong> the API process finished updating Redis with msg_2 at .122.</li><li>WORKER-A initiated the model inference call using stale context ([msg_1]), omitting msg_2 entirely.</li></ul><p><strong>The Fix:</strong></p><p>Change the pipeline sequence. Have the worker read authoritative history directly from PostgreSQL using a transaction lock, or ensure context cache invalidation completes <em>before</em> the background task is enqueued.</p><h3>13. The Bigger Lesson</h3><p>As AI applications mature, building them involves far more than constructing complex prompts or managing model context windows.</p><p>The underlying models will continue to get faster, cheaper, and more capable. But model improvements won’t resolve race conditions, out-of-order events, stale cache reads, or dropped connections.</p><p>Reliability in modern AI platforms isn’t determined by the intelligence of your LLM provider. It’s determined by the surrounding backend engineering.</p><p>Stop thinking about AI applications as a linear pipe:</p><p>$$\text{Prompt} \longrightarrow \text{LLM} \longrightarrow \text{Response}$$</p><p>Start architecture design around state transitions:</p><pre>Events ──► State ──► Orchestration ──► Models ──► External Tools ──► Updated State<br>  ▲                                                                         │<br>  └─────────────────────────────────────────────────────────────────────────┘</pre><p>Treat AI platforms as distributed systems. Explicitly model your state, use strict state transitions, handle cancellation, enforce idempotency, and design for failure upfront. That is how you build reliable AI software that scales.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d739a1373e15" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Retries Can Take Your System Down]]></title>
            <link>https://medium.com/@ashiya.ajare/why-retries-can-take-your-system-down-93ff3921543b?source=rss-fdc527338328------2</link>
            <guid isPermaLink="false">https://medium.com/p/93ff3921543b</guid>
            <dc:creator><![CDATA[Ashiya Ajare]]></dc:creator>
            <pubDate>Thu, 27 Aug 2026 09:52:36 GMT</pubDate>
            <atom:updated>2026-08-27T09:52:36.962Z</atom:updated>
            <content:encoded><![CDATA[<p>It was 2:15 PM on a Tuesday when the alerts started firing.</p><p>At first, it looked like a routine hiccup. The P99 latency on our primary checkout-service spiked from its comfortable 120 milliseconds to nearly 3 seconds. Within two minutes, error rates hit 4%. Within five minutes, the dashboard was solid red, CPU utilization across the API gateway nodes pegged at 99%, and the entire e-commerce platform ground to a complete halt.</p><p>In the war room, the initial suspect was a sudden spike in user traffic. But when we looked at the edge metrics, incoming user requests were completely flat. We weren’t under a DDoS attack from the outside.</p><p>Our own infrastructure was attacking itself.</p><p>When we peeled back the layers, the origin of the disaster wasn’t a database crash, a memory leak, or a bad deployment. It was a brief, 300-millisecond latency blip in a minor downstream microservice — the inventory-service—and a seemingly innocent piece of code every backend engineer has written a hundred times:</p><p>Python</p><pre># What could possibly go wrong?<br>def call_inventory_service(item_id):<br>    for attempt in range(3):<br>        try:<br>            return http_client.get(f&quot;/items/{item_id}&quot;, timeout=1.0)<br>        except TimeoutError:<br>            continue<br>    raise ServiceUnavailableException()</pre><p>That three-line retry loop transformed a momentary flick of noise into a full-scale, multi-service collapse.</p><h3>The Illusion of the Free Retry</h3><p>To understand how a self-healing mechanism becomes an engine of destruction, we have to acknowledge why we put retries in our code in the first place.</p><p>Distributed systems are inherently untrustworthy. Networks drop packets, routers reboot, garbage collection cycles pause threads, and deployment rollouts temporarily drop connections. These are transient failures — blips that last milliseconds.</p><p>If an HTTP request fails because a TCP connection was dropped mid-handshake, failing the entire user operation is terrible engineering. Retrying the request immediately makes sense. It bridges the gap between unreliable hardware and reliable user experiences. Used correctly, retries are the single most effective tool for masking the messy reality of networked software.</p><p>The fundamental mistake we make as developers is assuming that failure modes stay transient.</p><p>We write retry logic under the implicit assumption that the destination service is healthy, and that the error was just a ghost in the network wire. But when a service fails because it is overloaded, a retry is no longer a gentle safety net.</p><p>It is a sledgehammer hitting an already cracking wall.</p><h3>The Math of the Retry Storm</h3><p>Let’s return to our incident and trace the physics of what happened.</p><p>The inventory-service handles 1,000 requests per second under normal load. Its database thread pool has a maximum capacity of 1,200 concurrent queries.</p><p>At 2:14 PM, a background database migration caused the inventory-service query latency to jump from 50ms to 1.1 seconds.</p><p>Because the client timeout in checkout-service was configured to exactly 1.0 second, every single in-flight request began timing out just 100 milliseconds before the database could actually answer.</p><p>Here is where the mathematics of retry amplification kicked in.</p><pre>Time T+0s:   1,000 incoming requests/sec<br>             ↓ (All time out at 1.0s)<br>Time T+1s:   1,000 NEW incoming requests <br>           + 1,000 RETRY attempt #1 requests<br>             = 2,000 requests/sec arriving at inventory-service<br>             ↓ (All time out at 1.0s)<br>Time T+2s:   1,000 NEW incoming requests <br>           + 1,000 RETRY attempt #1 requests<br>           + 1,000 RETRY attempt #2 requests<br>             = 3,000 requests/sec arriving at inventory-service</pre><p>Within three seconds, a service built to handle 1,000 requests per second was receiving 3,000 requests per second.</p><p>The inventory-service wasn&#39;t just struggling anymore; it was buried under a wall of work it had zero mathematical chance of clearing. CPU utilization hit 100%, memory usage surged as request queues filled up, and thread pools starved.</p><p>The retries created a feedback loop: <strong>the slower the service got, the more retries it generated, and the more retries it generated, the slower it got.</strong></p><h3>Cascading Failures: The Contagion Spreads</h3><p>A retry storm rarely stays contained within the two services where it starts. Like a traffic jam on a major highway, the congestion backs up until the whole city is gridlocked.</p><p>As checkout-service threads sat waiting for inventory-service retries to time out, checkout-service itself ran out of execution threads. It could no longer accept new connections from the upstream api-gateway.</p><p>Now, api-gateway started experiencing timeouts from checkout-service. And what was api-gateway configured to do on timeouts?</p><p>It retried.</p><p>If api-gateway retries 3 times, and for each of those calls checkout-service retries 3 times, a single incoming user request can result in $3 \times 3 = 9$ downstream calls.</p><p>This is <strong>retry amplification</strong>. The failure propagated backward through the call graph. Within minutes, services that had nothing to do with inventory — user profiles, recommendation engines, shipping calculators — found their connection pools exhausted by workers blocked on the cascading queue.</p><p>The entire platform collapsed because of a 300-millisecond database lock.</p><h3>Breaking the Wave: Exponential Backoff and Jitter</h3><p>When we finally recovered the system by restarting services and clearing queues, the post-mortem focused on a simple question: <em>How do we allow retries for transient blips without enabling self-inflicted denial-of-service attacks?</em></p><p>The first line of defense is introducing time between attempts.</p><p>If a service fails, retrying 1 millisecond later is foolish. The underlying condition has almost certainly not changed. Instead of tight loops, clients should use <strong>exponential backoff</strong>: doubling the wait duration after each consecutive failure.</p><p>Instead of:</p><ul><li>Attempt 1: immediate</li><li>Attempt 2: immediate</li><li>Attempt 3: immediate</li></ul><p>Exponential backoff shifts the timeline:</p><ul><li>Attempt 1: Wait 100ms</li><li>Attempt 2: Wait 200ms</li><li>Attempt 3: Wait 400ms</li><li>Attempt 4: Wait 800ms</li></ul><p>This gives the failing downstream service breathing room to clear its queues. But exponential backoff has a hidden flaw when operated at scale: <strong>thundering herds</strong>.</p><p>Imagine 5,000 clients simultaneously hit a network switch that blips for 50 milliseconds. All 5,000 requests fail at once. If all 5,000 clients use identical exponential backoff algorithms, they will all wait exactly 100ms, and then hit the downstream service in unison. Then they will all fail, wait exactly 200ms, and hit the service in unison again.</p><p>Instead of a smooth stream of traffic, the downstream service experiences massive, synchronized waves of requests.</p><p>The solution is <strong>jitter</strong> — adding randomness to the backoff interval.</p><p>Python:</p><pre>import random</pre><pre>def calculate_backoff(attempt, base_delay=0.1, max_delay=2.0):<br>    # Calculate exponential delay: 0.1, 0.2, 0.4, 0.8...<br>    exp_delay = min(max_delay, base_delay * (2 ** attempt))<br>    <br>    # Add Full Jitter: randomize between 0 and exp_delay<br>    sleep_time = random.uniform(0, exp_delay)<br>    return sleep_time</pre><p>By adding randomness, the synchronized waves are smoothed out into a manageable trickle.</p><p>Jitter decorrelates client attempts, ensuring that a recovering service isn’t repeatedly knocked back down the moment it tries to stand up.</p><h3>Token Buckets and Retry Budgets</h3><p>Exponential backoff with jitter helps spread out retries, but it doesn’t solve the fundamental volume issue: if 50% of your primary traffic is failing, doing retries still increases total cluster traffic by up to 50%.</p><p>When a system is in deep distress, retries should be abandoned entirely.</p><p>This is where <strong>retry budgets</strong> come in. A retry budget dictates that a client cluster may only allocate a fixed percentage of its total traffic to retries — typically around 10%.</p><p>This is commonly implemented using a token bucket algorithm:</p><ul><li>Every successful request adds $1$ token to a pool (up to a maximum cap, e.g., 100).</li><li>Every retry attempt consumes $10$ tokens from the pool.</li><li>If the pool has fewer than 10 tokens, <strong>retries are forbidden</strong>. The client must let the failure surface immediately.</li></ul><p>Under normal operating conditions with a 99.9% success rate, the token bucket stays full. If a minor blip occurs, the client has plenty of tokens to retry.</p><p>However, if the downstream service undergoes a sustained outage, the token bucket drains within seconds. Once empty, the system stops retrying altogether, dropping total request volume back down to baseline and allowing the downstream service to recover.</p><p>Retries must be treated as a scarce, expensive resource — not a free default.</p><h3>The Silent Danger: Non-Idempotent Retries</h3><p>Up to this point, we’ve treated retries as a purely performance and capacity problem. But retries carry another, far more dangerous risk: <strong>data corruption</strong>.</p><p>Consider what happens when a client sends a request to a payment gateway: POST /v1/charges.</p><pre>[ Client ] --- ( 1. POST /v1/charges ) ---&gt; [ Payment Service ]<br>                                                   |<br>                                            ( Processed! )<br>                                                   |<br>[ Client ] &lt;--- ( 2. Timeout / Drop ) ------ x [ Payment Service ]</pre><p>The server successfully processes the credit card charge. But right as it sends the 200 OK response back, a network router drops the TCP packet.</p><p>From the client’s perspective, the request timed out. The client has no idea whether:</p><ul><li>The server failed to receive the request.</li><li>The server received it and crashed while processing.</li><li>The server successfully processed it, but the response was lost.</li></ul><p>If the client blindly retries POST /v1/charges, the customer gets billed twice.</p><p>A request is only safe to retry if it is <strong>idempotent</strong> — meaning executing it once has the exact same side-effect as executing it ten times.</p><p>GET, PUT, and DELETE operations are typically idempotent by spec. POST operations generally are not.</p><p>To safely retry operations that modify state, you must pass a unique <strong>Idempotency Key</strong> in the request headers:</p><p>HTTP</p><pre>POST /v1/charges HTTP/1.1<br>Host: api.payments.com<br>Idempotency-Key: 7b9a2c3f-8e1d-4f2a-9a1b-3c4d5e6f7a8b<br>Content-Type: application/json</pre><pre>{<br>  &quot;amount&quot;: 4999,<br>  &quot;currency&quot;: &quot;usd&quot;<br>}</pre><p>The server records this key in a fast cache (like Redis) alongside the result of the operation. If a retried request arrives with the same key, the server skips processing entirely and simply returns the cached response.</p><p>Without idempotency keys, every retry on a state-modifying endpoint is a game of Russian roulette with your database.</p><h3>Stopping the Bleeding: Circuit Breakers</h3><p>Even with backoff, jitter, and retry budgets, there are moments when a service is simply dead. Continuously throwing requests — even spaced-out ones — at a dead service wastes CPU cycles, drains connection pools, and increases overall latency.</p><p>You need a mechanism to cut the wire entirely: a <strong>Circuit Breaker</strong>.</p><p>Borrowed directly from electrical engineering, a software circuit breaker wraps a remote call and monitors for failures. It operates in three distinct states:</p><ul><li><strong>Closed (Normal):</strong> Requests flow straight to the downstream service. The breaker monitors error rates over a rolling window (e.g., the last 60 seconds).</li><li><strong>Open (Trip):</strong> If the error rate crosses a set threshold (e.g., 50% failure rate over 100 requests), the circuit breaker trips <strong>Open</strong>. All subsequent requests fail <em>immediately</em> at the client side without ever touching the network. No retries are attempted.</li><li><strong>Half-Open (Testing):</strong> After a cooldown period (e.g., 30 seconds), the breaker enters <strong>Half-Open</strong> mode. It allows a small trickle of trial requests through. If those requests succeed, the breaker resets to <strong>Closed</strong>. If they fail, it trips back to <strong>Open</strong>.</li></ul><p>By failing fast while Open, circuit breakers instantly liberate worker threads and prevent the cascade from dragging down upstream components.</p><h3>The Innocent Line of Code</h3><p>Looking back at our 2:15 PM incident, the post-mortem forced us to fundamentally change our perspective on system resilience.</p><p>We removed generic retry loops from our HTTP clients. We established cluster-wide retry budgets capped at 10%. We mandated exponential backoff with full jitter across all microservices, enforced idempotency keys on every stateful transaction, and wrapped inter-service communication in circuit breakers.</p><p>The next time the inventory-service experienced a database migration blip, the system behaved completely differently.</p><p>Instead of an avalanche of retries overwhelming the cluster, the client services backed off gracefully. The circuit breaker tripped for 30 seconds, allowing the inventory database to clear its lock without pressure. The edge gateway degraded gracefully, serving stale cached inventory counts to users rather than crashing the checkout flow. Total platform impact: zero downtime.</p><p>As backend engineers, we are trained to write defensive code. When an operation fails, our default instinct is to try again.</p><p>But in a distributed system, an unthrottled retry is not a form of resilience — it is a localized amplification of force. Every time you write a retry loop, you are choosing to add load to a system precisely when it is telling you it cannot handle what it already has.</p><p>Before you write for attempt in range(3):, ask yourself a critical question:</p><p><em>If every client does what I’m about to do right now, will the system recover — or will it burn?</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93ff3921543b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Breaks When You Take RAG From Demo to Production]]></title>
            <link>https://medium.com/@ashiya.ajare/what-breaks-when-you-take-rag-from-demo-to-production-f3ce206fe8cb?source=rss-fdc527338328------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3ce206fe8cb</guid>
            <category><![CDATA[embedding]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[vector-database]]></category>
            <dc:creator><![CDATA[Ashiya Ajare]]></dc:creator>
            <pubDate>Mon, 06 Jul 2026 18:51:06 GMT</pubDate>
            <atom:updated>2026-07-06T18:51:06.154Z</atom:updated>
            <content:encoded><![CDATA[<h4>Production failures I didn’t see coming — and what fixed them</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kurpz0LzbMfTZrBady2mgg.png" /></figure><p>Building a demo RAG system is surprisingly easy. Pull in LangChain or LlamaIndex, chunk a PDF, throw the chunks into a vector database, embed a query, stuff the top matches into a prompt, and watch GPT-4 answer questions about your document. It takes an afternoon. It feels like magic.</p><blockquote>Building one that consistently retrieves the right information in production is a completely different engineering problem.</blockquote><p>The gap between those two things is where most teams lose weeks. Not because RAG is conceptually hard — it isn’t — but because every step in the pipeline has a failure mode that only shows up at scale, with real documents, real users, and real edit histories. A demo has ten clean PDFs and three test questions. Production has forty thousand documents written by different departments over six years, half of them updated last Tuesday, and users asking questions the documents don’t actually answer.</p><p>I spent the better part of a year building and operating a production RAG system for internal knowledge search at a mid-size company (think: several hundred thousand documents across Confluence, Google Docs, PDFs, and Slack exports). This post is the list of things that broke, why they broke, and what we changed. Some of it will be obvious in hindsight. Most of it wasn’t obvious to us at the time, and it cost us real weeks — not hours — to diagnose.</p><p>I’m going to skip the “what is RAG” preamble. If you’re reading this, you’ve already built one. This is about the second system: the one you build after the first one embarrasses you in front of a stakeholder.</p><blockquote><strong>A quick note on structure:</strong> some of these mistakes are deep rabbit holes that took us three weeks to fully understand. Others are one paragraph because the fix was obvious once we found it. I’m not going to pad the short ones to make the article look symmetric.</blockquote><h3>1. Chunking by Character Count Instead of Structure</h3><p>Every RAG tutorial ships with — RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)</p><p>It&#39;s the path of least resistance, and it works fine on the tutorial&#39;s example document because that document was probably chosen because it chunks cleanly.</p><p>Chunks that start mid-sentence. Chunks that split a table into two meaningless fragments. Chunks that separate a heading from the content it introduces, so the retrieved chunk is a paragraph of prose with no idea what topic it’s about.</p><blockquote>We had a security policy document structured as nested bullet points — “Data Retention” as a heading, with sub-bullets for “Customer PII,” “Logs,” and “Backups,” each with its own retention period. A pure character-count splitter cut straight through the middle of the “Logs” sub-bullet. A user asking “how long do we retain logs” got back a chunk that said “90 days from creation date” with zero indication of what that number referred to. The model, doing its best, guessed it was about backups and said so, confidently, incorrectly.</blockquote><p><strong>The eventual solution:</strong> We moved to structure-aware chunking: split on markdown/HTML headers first, then on paragraph boundaries, and only fall back to character-count splitting inside a paragraph that’s still too long. For Confluence and Google Docs specifically, we wrote custom parsers that walked the actual document tree (headings, lists, tables) rather than the exported plain text, because plain-text export throws away exactly the structural signal we needed.</p><p>Chunking is not “split text into pieces of size N.” It’s “split text at the boundaries a human would use to explain it to a colleague.” If your chunker doesn’t know what a heading is, it’s throwing away free information.</p><h3>2. Chunk Size: Chasing a Single “Right” Number</h3><h4><strong>Why it happens ?</strong></h4><p>Everyone wants a magic constant. 512 tokens? 1000? 256? There’s an intuition that smaller chunks give more precise retrieval and larger chunks give more context, and that’s true, but the “right” number is a property of your content, not a property of RAG in general.</p><p>Small chunks retrieve the right sentence but the LLM lacks the surrounding context to interpret it correctly. Large chunks give the LLM enough context but dilute the embedding — a 1500-token chunk covering five subtopics produces an embedding that’s a blurry average of all five, so it doesn’t score highly for any single one.</p><blockquote>Our API documentation had very dense, small semantic units (one chunk = one endpoint parameter). Our onboarding guides had long narrative sections where meaning only made sense across several paragraphs. Using one global chunk size, we were either fragmenting the onboarding docs into confetti or bloating the API docs into unfocused blobs, depending on which way we tuned the knob.</blockquote><p>We ran a retrieval quality eval (more on this in mistake #17) across a grid of chunk sizes — 256, 512, 1000, 1500 — separately for each content type, and plotted recall@5 against chunk size. The curves peaked at completely different points for the two document types.</p><p>Per-source-type chunking config, not a global default. We tagged each ingestion pipeline with a content-type profile that dictated chunk size, splitter strategy, and whether to chunk at all (tables and short reference cards were often better left whole).</p><p>Stop looking for the right chunk size. Look for the right chunk size <em>per content type</em>, and build your ingestion pipeline to support that from day one rather than retrofitting it later — retrofitting means re-embedding everything, which is expensive and slow at scale.</p><h3>3. Overlap as a Band-Aid for Bad Chunking</h3><h4><strong>Why it happens ?</strong></h4><p>Chunk overlap (e.g., 200 tokens repeated between consecutive chunks) is supposed to prevent context from being severed at a chunk boundary. Teams often respond to boundary-cutting problems by cranking overlap up, because it’s an easy knob to turn.</p><p>Near-duplicate chunks flooding the top-k results. If chunk overlap is 50% of chunk size, you can easily retrieve three overlapping chunks that are 80% identical, meaning your “top 5 results” is really “top 2 distinct pieces of information padded with copies.”</p><blockquote>We once pushed overlap up to 300 tokens on a 1000-token chunk size to fix a boundary-cutting issue in HR policy docs. Retrieval quality metrics barely moved, but our average prompt size for the LLM call went up by ~40%, because now four of our five retrieved chunks were substantially the same paragraph. We were paying more per query and getting less diverse context.</blockquote><p>We dropped overlap back down to 10–15% and fixed the actual boundary problem at the source (see mistake #1 — structure-aware splitting). Overlap is a mitigation for a symptom; structural chunking is the cure. We kept a small overlap (not zero) specifically to prevent single sentences from being orphaned across a boundary, but stopped treating it as the primary defense.</p><p>Overlap should be a small safety margin, not the mechanism by which you preserve context. If you need heavy overlap to make retrieval work, your chunking strategy is wrong, not your overlap parameter.</p><h3>4. Embedding Drift After Document Updates</h3><h4><strong>Why it happens ?</strong></h4><p>Documents change. Ingestion pipelines built for the “add a new document” case often don’t handle “this existing document was edited” correctly — they either skip re-embedding entirely (stale vectors pointing at content that no longer exists) or they append a new version without removing the old chunks (duplicate, contradictory vectors in the index).</p><p>The system confidently retrieves and cites information that used to be true. This is one of the worst failure modes because it doesn’t look like a retrieval bug — it looks like a hallucination, and engineers waste time tuning prompts to fix a data freshness problem.</p><blockquote>Our pricing policy doc was updated to reflect a new discount tier. The RAG system kept citing the old tier for another two weeks, because our Confluence sync job used a webhook that fired on page_updated events, but we&#39;d never tested what happened when a page was updated <em>twice in quick succession</em> — the second webhook arrived while the first ingestion job was still running, got deduplicated by our idempotency layer (which keyed only on page ID, not on revision/version), and was silently dropped. The index was frozen at revision N-1 indefinitely.</blockquote><p>A user reported “the bot is telling people the old discount.” We pulled the chunk that was retrieved, checked its last_updated metadata (thankfully we had this), and saw it was stamped with a date before the policy change — which told us this was an ingestion freshness bug, not a model or retrieval quality bug, within about ten minutes. Metadata timestamps are cheap insurance; they turn a &quot;why is the answer wrong&quot; mystery into a &quot;why didn&#39;t ingestion run&quot; mystery, which is a much smaller search space.</p><p>We keyed idempotency on (page_id, revision_id) instead of just page_id, added a periodic full reconciliation job (nightly diff between source-of-truth document list and what&#39;s in the vector store, re-ingesting anything with a mismatched revision) as a safety net on top of the webhook-driven real-time path, and deleted-then-inserted chunks for updated documents rather than upserting by chunk ID, since chunk boundaries can shift when content changes and stale chunk IDs from a previous chunking pass can linger.</p><p>Webhooks lie. Real-time event-driven ingestion needs a batch reconciliation job as a backstop, or you will eventually serve stale data with total confidence and no error in any log.</p><h3>5. Metadata Filtering: An Afterthought That Should Have Been a Foundation</h3><p>Metadata filtering (by document type, department, date range, access level) is usually bolted on after the retrieval pipeline already works on pure vector similarity, because in the demo phase there’s no need for it — you have ten documents and no access control requirements.</p><p>Retrieval works fine until you have more than one “kind” of document in the index, at which point results start getting cross-contaminated — a question about the US expense policy retrieves the UK expense policy because it’s semantically similar, or worse, a question from a contractor retrieves an internal-only document because nothing in the pipeline knows the requester’s access level.</p><blockquote>We had both “draft” and “published” versions of policy documents in the same index (drafts were used for an internal review workflow). Nothing distinguished them in the embedding space — a draft and its published counterpart are nearly identical text — so the system would sometimes retrieve and cite draft content that hadn’t been approved yet. This was a genuine compliance concern, not just a quality issue.</blockquote><p>We treated metadata filtering as a pre-condition on retrieval, not a post-processing step: build the vector search query with metadata filters applied at the database level (most vector DBs support this natively — Pinecone metadata filters, Weaviate where clauses, pgvector combined with a WHERE clause in the same SQL query) so that draft documents, wrong-region documents, and access-restricted documents are excluded <em>before</em> the similarity search runs, not filtered out of the results afterward. Filtering after the fact is both slower (you&#39;ve wasted a similarity computation) and dangerous (if you retrieve top-5 and filter down to top-2, you might have wanted top-5 relevant results and now have only 2).</p><p>Metadata isn’t an enhancement to retrieval. For any system with more than one document type or any access control requirement, metadata filtering is part of the correctness contract, and it needs to be designed alongside the schema, not added after a compliance incident.</p><h3>6. Retrieval Quality Problems: Confusing “It Returns Something” With “It Returns the Right Thing”</h3><p>Vector search always returns results. Cosine similarity always produces a ranked list, even when nothing in the corpus is actually relevant. There’s no natural “I don’t know” signal, so teams mistake “the pipeline runs without errors” for “the pipeline works.”</p><p>The system answers questions it has no business answering, using the “best available” (but not actually relevant) chunk, and answers them fluently and confidently, because the LLM doesn’t know the retrieved context is weak — it just does its best with what it’s given.</p><blockquote>A user asked about a benefits policy for a country we don’t operate in. There was no relevant document. The vector search still returned five results — the five <em>least dissimilar</em> chunks in the corpus, which happened to be from a vaguely related benefits document for a different country. The LLM synthesized a plausible-sounding answer from those chunks. It was confidently wrong, and worse, it read exactly like our correct answers in tone and structure, so users had no way to distinguish it from a good response.</blockquote><p>We started logging similarity scores, not just retrieved content, on every query, and built a dashboard of the score distribution. This revealed that a meaningful fraction of queries had a top score below a threshold we later determined correlated with “not actually relevant” (through manual review of ~200 low-scoring queries).</p><p>A minimum similarity threshold, below which we don’t even call the LLM — we return “I couldn’t find relevant information for this” directly. This is a blunt instrument (thresholds don’t generalize perfectly across query types) but it eliminated the worst class of confident-wrong-answers. We paired it with a lighter-weight “relevance check” step (see #14, reranking) for the middle ground of queries that were plausible but uncertain.</p><p>Vector similarity is not a relevance guarantee. Build an explicit “no good match” path into your system from the start; otherwise your system will always answer, even when it shouldn’t.</p><h3>Closing Thoughts</h3><p>The actual difficulty of production RAG isn’t any single mistake — it’s that these failure modes are quiet. They don’t throw exceptions. They don’t fail CI.</p><p>A mis-chunked document doesn’t crash; it just becomes slightly less findable, forever, until a user notices and by then you’ve forgotten what changed.</p><blockquote>That’s really the throughline here: <strong>production RAG systems fail silently, in the gap between “the code ran” and “the answer was right.”</strong></blockquote><p>Structure-aware chunking, per-content-type chunk sizing, metadata as a first-class schema decision, and a real reconciliation job for ingestion — all of it exists to convert silent failures into visible, measurable ones. That conversion is, in my opinion, the actual engineering work of building a production RAG system.</p><p>This isn’t the full list. We hit at least a dozen more of these over the year — reranking decisions, hallucination-despite-correct-retrieval, latency tuning, the eval pipeline that ended up being the thing that made everything else fixable. I’ll get into those in a follow-up post once I’ve written them up properly rather than rushing them here.</p><h4>In the meantime, I’d rather hear from you than talk past you</h4><blockquote>What’s broken in your RAG system that isn’t on this list? Did any of these six match something you hit, or did you find a different root cause for a similar-looking symptom?</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3ce206fe8cb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Human in the Loop: Why AI Still Needs Humans]]></title>
            <link>https://medium.com/@ashiya.ajare/human-in-the-loop-why-ai-still-needs-humans-aa819682b37d?source=rss-fdc527338328------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa819682b37d</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ashiya Ajare]]></dc:creator>
            <pubDate>Mon, 29 Jun 2026 06:49:33 GMT</pubDate>
            <atom:updated>2026-06-29T06:49:33.782Z</atom:updated>
            <content:encoded><![CDATA[<h4>The secret to building AI that actually works isn’t a bigger dataset — it’s knowing when to ask for a human perspective.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r3RlHKxZC15sUNV8fLftyg.png" /></figure><h3>Picture this:</h3><p>You are rushing to a crucial job interview. You’re slightly nervous, but you’re making good time. You plug the address into your GPS, trust the robotic voice blindly, and cruise down the highway. Suddenly, the navigation app cheerfully instructs you to “turn right now.” You look to your right, expecting a hidden side street, but all you see is a concrete barrier and a steep drop into a construction zone.</p><p>If you followed that instruction blindly, you’d be in a lot of trouble. Luckily, you have common sense. You ignore the digital panic, keep driving straight, and let the app recalculate.</p><p>We’ve all been there. Whether it’s ChatGPT confidently making up a historical fact, your phone’s face unlock failing because you’re wearing sunglasses, or Netflix recommending a dark true-crime documentary right after you watched a lighthearted sitcom.</p><p>These little hiccups point to a larger truth that often gets lost in the current tech hype: Artificial Intelligence is incredibly powerful, but it isn’t perfect. It lacks the grounded reality that we take for granted every single day.</p><p>What if AI didn’t always have to figure everything out on its own? What if the secret to building truly great technology wasn’t trying to make a flawless machine, but rather building a system where humans and machines work as a team?</p><h3>The Co-Pilot in the Passenger Seat</h3><p>In the tech world, we have a specific name for this teamwork: <strong>Human-in-the-Loop AI</strong>, or <strong>HITL AI</strong> for short. It sounds like fancy engineering jargon, but the core concept is beautifully simple.</p><p>Think back to when you were learning to ride a bicycle. Your parent or an older sibling didn’t just hand you the bike, wish you luck, and walk away. Instead, they held onto the back of the seat. They let you pedal and steer, but the moment you started to wobble toward a bush, they caught you. They gave you feedback — “keep your eyes forward,” “pedal faster” — until you could balance on your own.</p><p>That is exactly how Human-in-the-Loop works.</p><pre>[ User Query ] ──&gt; [ AI Model Processes ] ──&gt; [ Human Reviews/Corrects ] ──&gt; [ Better, Safer Output ]<br>                                                        │<br>                                                        └───&gt; [ Continuous Feedback Loop ]</pre><p>In a HITL system, the AI does the heavy lifting, like processing millions of data points or drafting text in seconds. However, a human sits in the loop to check the work, fix mistakes, and offer guidance. This creates a continuous <strong>AI feedback loop</strong>. Every time a human corrects the system, the AI learns from its mistakes, gradually becoming smarter, safer, and more reliable.</p><h3>Why AI Alone Flags on the Play</h3><p>To understand why we need this setup, we have to look at what happens when AI goes solo. Modern AI models are essentially prediction machines. They don’t “think” the way you and I do; they look at patterns in data and guess what should come next. Because of this, they run into a few classic roadblocks.</p><h3>The Hallucination Hazard</h3><p>First up is what engineers call “hallucination.” This happens when an AI doesn’t know the answer to a question but generates a completely fabricated response with absolute, unwavering confidence. Imagine a legal assistant AI inventing a non-existent court case to support a legal brief. Without a human lawyer reviewing that document before it hits a judge’s desk, a career could be ruined over a glitch.</p><h3>The Missing Context</h3><p>Then there’s the problem of context. AI is notoriously bad at reading between the lines. If a user posts a sarcastic comment online like, <em>“Oh great, another rainy day, absolutely love ruining my suede shoes,”</em> a basic content moderation AI might flag it as positive because of the word “love.” A human reader instantly spots the eye-roll. We understand nuance, culture, and humor in ways code simply cannot.</p><h3>The Blind Spots of Bias</h3><p>We also have to talk about bias. AI models learn from historical data created by humans. If that past data contains unfair biases — like a resume screening tool that historically favored male applicants for engineering roles — the AI will quietly learn and repeat those same mistakes. Human oversight is essential to spot these systemic blind spots and inject fairness into <strong>AI decision making</strong>.</p><h3>A Day in the Life of a Chatbot</h3><p>Let’s look at how this plays out in a practical, real-world scenario: a customer support chatbot for an airline.</p><p>Imagine an angry passenger named Sarah whose flight was canceled. She types a frantic message into the chat window. The AI gets to work first. It analyzes her text, pulls up her booking information, and drafts a standard response offering a flight voucher.</p><p>But the AI also detects high levels of frustration in Sarah’s language. Instead of sending the automated response directly to Sarah, the system pauses. It routes the conversation to a human customer service agent named Marcus.</p><p>Marcus looks at his screen. He sees the AI’s drafted response, but he also notices that Sarah is trying to get to a family wedding. Marcus tweaks the AI’s message, adds a personal touch of empathy, overrides the standard voucher policy to book her on a competitor’s flight, and hits send.</p><p>Sarah gets her problem solved by a compassionate human, and the AI logs Marcus’s edits. The system notes how Marcus handled the edge case, adjusting its internal logic for the next time a similar situation pops up. Everyone wins.</p><h3>Where Humans Hold the Reins</h3><p>This kind of <strong>Human-AI collaboration</strong> isn’t just for customer service. It’s happening in critical industries all around us.</p><ul><li><strong>Medical Diagnosis:</strong> In healthcare, AI can scan thousands of X-rays in seconds to flag potential tumors. But we don’t let the computer write the prescription. A radiologist reviews the highlighted scans, bringing years of clinical experience and medical intuition to make the final diagnosis.</li><li><strong>Self-Driving Cars:</strong> Autonomous vehicles are packed with AI, but they still rely on human safety drivers during testing. When the car encounters a chaotic construction site with confusing traffic cones, the human takes the wheel, teaching the system how to navigate the unexpected.</li><li><strong>AI Coding Assistants:</strong> Tools like GitHub Copilot or Cursor are fantastic for writing repetitive code blocks. However, any developer will tell you that if you copy and paste AI code without reviewing it, you’re bound to break your application. The developer acts as the architect, checking for security flaws and structural logic.</li></ul><p>Without human oversight in these areas, a single unverified AI decision could lead to medical misdiagnoses, financial fraud slipping through the cracks, or broken software deployment.</p><h3>The Upside of Staying Involved</h3><p>By keeping humans in the loop, we unlock huge advantages that code alone can’t deliver.</p><p>First, it builds deep <strong>trust</strong>. Customers and users feel significantly more secure knowing a human hand is guiding the technology.</p><p>Second, it dramatically reduces costly mistakes. Catching an error early in a feedback loop is much cheaper than clean-up duty after a public relations disaster.</p><p>Finally, it creates a culture of continuous learning. Instead of deployment being the end of a software project, the AI system evolves dynamically alongside the people using it every single day.</p><h3>The Reality Check: It’s Not All Smooth Sailing</h3><p>Of course, maintaining this balance isn’t a walk in the park. Building a true HITL system comes with practical trade-offs.</p><p>For one, it requires significant human effort and can quickly drive up operational costs. Hiring experts to constantly review, tag, and correct data is expensive.</p><p>It also slows things down. If an AI can process a loan application in milliseconds, but it has to wait three hours for a human manager to sign off, you lose that instant gratification users love. Scaling these systems globally becomes a complex puzzle of balancing speed against accuracy.</p><h3>Moving Toward a Balanced Future</h3><p>As we look ahead, tech conversations are dominated by talk of autonomous AI agents — systems that can chain tasks together and execute complex workflows entirely on their own. It sounds futuristic, and frankly, a bit intimidating.</p><p>But even as these systems grow more capable, the need for human guidance won’t disappear; it will simply shift. We will transition from being manual operators to acting as supervisors, strategists, and ethicists.</p><p>We need humans to define the boundaries of <strong>responsible AI</strong>. A machine can optimize for efficiency, but it cannot understand fairness, empathy, or accountability. If an AI system makes a mistake that harms someone, a line of code cannot apologize or stand in a courtroom. Humans must always hold the ultimate responsibility.</p><p>The future of technology isn’t a battle of Humans versus AI. It’s an alliance: <strong>Humans + AI</strong>.</p><h3>Wrapping Up</h3><p>The next time your navigation app tries to send you off a cliff, or a chatbot gives you a bizarre response, don’t despair about the future of technology. Just smile and remember that the digital world still needs a little humanity to find its way.</p><p>The smartest AI systems won’t be the ones that attempt to completely replace humans. They’ll be the ones designed with the humility to know exactly when to raise their digital hands and ask for help.</p><h3>Over to You</h3><p>Have you ever caught an AI model making a funny or frustrating mistake? How do you feel about humans reviewing AI decisions behind the scenes? Let’s chat in the comments below!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa819682b37d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The WebRTC Streaming Advantage]]></title>
            <link>https://medium.com/@ashiya.ajare/the-webrtc-streaming-advantage-019bb645ea70?source=rss-fdc527338328------2</link>
            <guid isPermaLink="false">https://medium.com/p/019bb645ea70</guid>
            <category><![CDATA[voice-recognition]]></category>
            <category><![CDATA[audio-pipeline]]></category>
            <category><![CDATA[webrtc]]></category>
            <dc:creator><![CDATA[Ashiya Ajare]]></dc:creator>
            <pubDate>Tue, 31 Mar 2026 08:16:08 GMT</pubDate>
            <atom:updated>2026-03-31T08:16:08.090Z</atom:updated>
            <content:encoded><![CDATA[<h4>Scaling speaker verification with SFUs and stateful buffer management.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/0*nkHWLelL14dgyxwv.jpg" /></figure><p>In the development of speaker verification systems, there is a common misconception that achieving a high-performing model is the final hurdle. For many of us, once the EER (Equal Error Rate) looks good on a static dataset, we feel the hard part is behind us.</p><p>However, as I quickly discovered, the real challenge begins the moment you attempt to transition from batch processing to real-time execution.</p><h3>The Gap Between Model and System</h3><p>In my previous post, <em>Speaker Verification Is a System, Not a Model</em>, I explored why decision reliability is the backbone of any production-grade biometric tool. But reliability is only one side of the coin.</p><p>A system that works perfectly on a pre-recorded .wav file often fails when introduced to a live stream. Our initial architecture was robust; <strong>it could make the right decision, but it couldn&#39;t make it at the right time.</strong></p><h3>The Latency vs. Reliability Trade-off</h3><p>When you move into the real-time domain, the engineering constraints shift from purely mathematical to structural. We found ourselves grappling with two primary issues:</p><ul><li><strong>Total System Latency:</strong> The delay between a user speaking and the system providing an authentication result.</li><li><strong>Real-Time Constraints:</strong> The necessity of processing audio chunks as they arrive without dropping packets or falling behind the stream.</li></ul><h3>The Naive Approach</h3><p>Initially, we relied on a file-based pipeline. It was a straightforward, linear process: <strong>Record → Upload → Process → Result</strong>.</p><p>While this worked for offline testing, it failed the “Liveness Test.” The latency of the API call — compounded by the time taken to finish recording and uploading the entire file — created a jarring user experience. To fix this, we had to shift our mental model from static files to <strong>temporal streams</strong>.</p><h3>Why WebRTC? The Industry Standard for Latency</h3><p>To achieve true real-time verification, we integrated <strong>WebRTC (Web Real-Time Communication)</strong>. While often associated with video conferencing, WebRTC is a powerhouse for high-fidelity audio, making it the ideal backbone for a voice biometric system.</p><p><strong>Key Advantages:</strong></p><ul><li><strong>Ultra-Low Latency:</strong> Typically sub-500ms, far faster than traditional protocols like HLS.</li><li><strong>Adaptive Quality:</strong> Automatically adjusts audio bitrate based on network conditions to prevent dropouts.</li><li><strong>Built-in Security:</strong> Enforces mandatory end-to-end encryption using SRTP.</li></ul><p><strong>Core Components:</strong></p><ul><li><strong>MediaStream API:</strong> Captures audio directly from the microphone.</li><li><strong>RTCPeerConnection:</strong> Manages the peer-to-peer data transmission.</li><li><strong>The Opus Codec:</strong> The “secret sauce” that supports everything from low-bitrate voice to high-fidelity stereo.</li></ul><h3>Scaling the Architecture: Moving to Media Servers</h3><p>Unlike a simple REST API, WebRTC requires a <strong>Signaling</strong> process to allow the client and server to “find” each other. For a scalable environment, we moved beyond simple Peer-to-Peer connections. By utilizing a <strong>Media Server (SFU)</strong>, we can redistribute the audio stream efficiently, ensuring our verification engine receives a consistent, high-quality stream regardless of concurrent user load.</p><h3>WebRTC Solved Transport — Not the System</h3><p>By integrating WebRTC, we conquered the “how” of moving audio. On paper, the transport problem was solved. However, we quickly realized that WebRTC is a delivery mechanism, not a logic engine. It exposed a deeper challenge: <strong>our verification logic was designed for files, but the world was now giving us a continuous, unpredictable stream.</strong></p><h4>1. From Stateless to Stateful</h4><p>In a file-based approach, the system is stateless. In a streaming environment, the system must maintain state across a series of continuous data chunks. We had to track the user’s identity and accumulated confidence scores over time.</p><blockquote><strong><em>The Insight:</em></strong><em> In a streaming system, there is no clear beginning or end — only continuity.</em></blockquote><h4>2. The Partial Audio Problem</h4><p>In batch processing, the model sees the entire utterance. In a stream, you deal with fragments.</p><ul><li><strong>The Dilemma:</strong> A 300ms audio chunk isn’t a voiceprint — it’s a guess.</li><li><strong>The Result:</strong> Isolated slices lead to noisy embeddings and unstable scores.</li></ul><h4>The Solution: In-Memory Stream Composition</h4><p>We solved this by rethinking how the backend consumes data. We built a <strong>Stateful Buffer Pipeline</strong>:</p><ol><li><strong>Continuous Frame Streaming:</strong> The frontend pushes raw audio frames.</li><li><strong>The Backend Collector:</strong> Frames are held in an active <strong>in-memory buffer</strong> tied to the session ID.</li><li><strong>The “Virtual File” Jump:</strong> When a threshold is met, we perform a zero-disk conversion, aggregating the buffer directly into a <strong>NumPy array</strong>.</li></ol><p>By bypassing .wav headers and physical disk I/O, we reduced final decision latency by orders of magnitude. The backend became an extension of the microphone.</p><h4>3. The Paradox of Decision Timing</h4><p>This is where engineering meets psychology. If we decide too early, we risk a False Rejection. If we wait too long, the “real-time” benefit vanishes. We realized that in real-time systems, <strong>timing is a core parameter of the model.</strong> We had to build logic that knew <em>when</em> it had heard enough to be certain.</p><h4>4. Designing for Stream Failures</h4><p>Batch systems fail at the start; <strong>streaming systems fail midway.</strong> We had to account for network jitter, packet loss, and abrupt disconnects. Our pipeline had to be resilient enough to handle a “broken” stream without crashing the verification session.</p><h3>Conclusion</h3><p>By shifting to this stream-aware architecture, we didn’t just improve our latency — we fundamentally changed how our system interacts with the world. We are no longer waiting for the audio to finish; we are verifying it as it happens.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=019bb645ea70" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Speaker Verification Is a System, Not a Model]]></title>
            <link>https://medium.com/@ashiya.ajare/speaker-verification-is-a-system-not-a-model-9de21feb67ee?source=rss-fdc527338328------2</link>
            <guid isPermaLink="false">https://medium.com/p/9de21feb67ee</guid>
            <category><![CDATA[speaker-recognition]]></category>
            <category><![CDATA[voice-recognition]]></category>
            <dc:creator><![CDATA[Ashiya Ajare]]></dc:creator>
            <pubDate>Tue, 24 Feb 2026 14:22:55 GMT</pubDate>
            <atom:updated>2026-02-25T06:08:42.038Z</atom:updated>
            <content:encoded><![CDATA[<h4>A practical look at building reliable voiceprints</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*9NZ8BFX7KWpNw7Zu-1pevg.png" /></figure><p>When I started building a speaker verification proof-of-concept, I assumed the hard part would be choosing the right deep learning model. It turned out the model was the easy part. The real challenge was turning good embeddings into reliable verification decisions.</p><h3>What is a Voiceprint?</h3><p>A voiceprint is a compact numerical representation of a person’s unique vocal characteristics. Speaker verification uses these prints to answer a simple question: <strong>Do these two voice samples belong to the same person?</strong></p><p>This capability is what underpins modern voice-based authentication, fraud detection, and personalized voice assistants.</p><h3>The Tech: From Raw Audio to Embeddings</h3><p>To create a voiceprint, we need a way to convert raw audio into a stable “identity” vector — an embedding. This requires a neural network trained to ignore <em>what</em> is being said (the content) and focus on <em>who</em> is saying it (the speaker).</p><p>For this system, I used the <strong>ECAPA-TDNN</strong> architecture as the backbone. It’s specifically designed to handle real-world audio variability and short clips.</p><blockquote><strong>Implementation Note:</strong></blockquote><blockquote>You can find the pre-trained version of this model on Hugging Face here: <a href="https://huggingface.co/speechbrain/spkrec-ecapa-voxceleb"><strong>SpeechBrain ECAPA-TDNN on Hugging Face</strong></a></blockquote><p>At this stage, ECAPA solved a major problem: it produced significantly more consistent voiceprints than simpler models. However, it also surfaced a critical realization: <strong>high-quality embeddings alone do not guarantee reliable verification.</strong></p><h3>The Hurdle: Comparison Problem</h3><p>The most straightforward way to verify a speaker is to compare two voiceprints using <strong>Cosine Similarity</strong>. If the similarity score is above a certain threshold (e.g., 0.85), you accept the speaker; otherwise, you reject them.</p><p>While this works in a controlled lab setting, it falls apart in the wild.</p><ul><li><strong>The Threshold Trap:</strong> A threshold that works perfectly for “Speaker A” in a quiet room might fail for “Speaker B” calling from a noisy cafe.</li><li><strong>Audio Variability:</strong> Short utterances or different microphones can cause the same speaker to produce wildly different scores.</li></ul><p>This is where the transition happens: moving from a “model” that outputs a number to a “system” that makes a decision.</p><blockquote><em>A common mistake is treating an embedding as a finished product. It isn’t. An embedding is just </em><strong><em>high-dimensional data</em></strong><em>. To turn that data into a reliable “Yes/No” decision, we need statistical layers that understand the probability of a match.</em></blockquote><h3>Choosing PLDA for Verification</h3><p>The real “magic” of PLDA isn’t just comparing two vectors; it’s calculating a <strong>Likelihood Ratio</strong>.</p><p>Instead of asking, <em>“How close are these two dots in space?”</em> (Cosine Similarity), PLDA asks:</p><blockquote><em>“What is the probability that these two recordings were generated by the same speaker (Hypothesis 1) versus two different speakers <br>( Hypothesis 2) ?”</em></blockquote><p>Standard Cosine Similarity is “identity-blind” — it treats every variation in the audio (like a noisy fan or a different microphone) as if it were a change in the speaker’s identity.</p><p><strong>PLDA</strong> solves this by decomposing the embedding into two distinct spaces: <strong>Within-class variance</strong> (how a single person’s voice changes between sessions) and <strong>Between-class variance</strong> (how different people sound compared to each other). By modeling these statistically, PLDA can ignore the “session noise” and focus purely on the “speaker signal.” It transforms a simple distance measurement into a probabilistic decision.</p><p>This is the final “boss” of speaker verification. While PLDA handles the math of identity, <strong>AS-Norm (Adaptive Symmetric Normalization)</strong> handles the reality of the environment.</p><h3>AS-Norm: The Final Polish</h3><p>Even with the probabilistic power of PLDA, we still face a frustrating reality: <strong>some speakers are simply harder to verify than others.</strong> In the research community, this is known as the “Biometric Menagerie.” Some speakers are “Sheep” (easy to verify), while others are “Goats” (whose voices are consistently difficult for models to match). If you use a single global threshold, you might accidentally lock out a “Goat” or let in an impostor who happens to sound like a “Sheep.”</p><h4>Why We Need Normalization ?</h4><p>A PLDA score of 15.0 might be a massive, certain match for one person, but a relatively weak, uncertain match for another. This happens because the “acoustic neighborhood” around each speaker is different.</p><p><strong>AS-Norm</strong> levels the playing field. It doesn’t just look at the two speakers in a vacuum; it compares them against a <strong>cohort</strong> — a background set of diverse, “impostor” voices.</p><h4>How It Works</h4><ol><li><strong>The Cohort Check:</strong> When the system compares a claim to a voiceprint, it also compares that claim against a “cohort” of 100–200 random background speakers.</li><li><strong>The Reality Check:</strong> It calculates the mean and standard deviation of these “impostor” scores.</li><li><strong>The Correction:</strong> If the current speaker’s score is high, but <em>everyone’s</em> score is high in that specific acoustic environment, AS-Norm “penalizes” the score. If the score is high despite the environment being difficult, the score is boosted.</li></ol><h3>The Result: A System, Not Just a Model</h3><p>By the time the audio has passed through <strong>ECAPA-TDNN</strong> (to get the features), <strong>PLDA</strong> (to understand the identity), and <strong>AS-Norm</strong> (to calibrate the score), we finally have a number we can trust.</p><p>We’ve moved from a raw embedding to a <strong>calibrated decision.</strong></p><h3>Conclusion: Systems Over Models</h3><p>If there is one thing I learned building this proof-of-concept, it’s that <strong>a model is just a component, but a system is a solution.</strong> Choosing a state-of-the-art model like ECAPA-TDNN gave me a great head start, but it was the statistical “glue” — PLDA for identity math and AS-Norm for score calibration — that actually made the system usable in the real world.</p><p>For engineers entering the world of voice biometrics, the lesson is clear: don’t get stuck on the embedding. Focus on the decision logic. Because at the end of the day, your users don’t care about your cosine similarity scores — they care about a system that is secure, reliable, and actually works when they’re calling from a noisy street corner.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9de21feb67ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Interning: The Key to Faster Python]]></title>
            <link>https://medium.com/@ashiya.ajare/interning-the-key-to-faster-python-5971153d4867?source=rss-fdc527338328------2</link>
            <guid isPermaLink="false">https://medium.com/p/5971153d4867</guid>
            <dc:creator><![CDATA[Ashiya Ajare]]></dc:creator>
            <pubDate>Fri, 31 Oct 2025 09:23:36 GMT</pubDate>
            <atom:updated>2025-10-31T09:23:36.440Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*Cc2sG87PyP2iftouAKFffA.png" /></figure><pre>a = &quot;hello&quot;<br>b = &quot;hello&quot;<br>print(a is b)  # True<br><br>x = &quot;hello world!&quot;<br>y = &quot;hello world!&quot;<br>print(x is y)  # False</pre><blockquote><em>If you’ve ever seen this and thought</em> “Wait, what?”<em>, you’re not alone.</em></blockquote><blockquote><em>Python sometimes decides that two identical-looking values are actually the same object — and sometimes it doesn’t.</em></blockquote><blockquote><em>This invisible optimization, called </em><strong><em>interning</em></strong><em>, is one of the quiet reasons Python can be both </em>slow and fast<em> at the same time.</em></blockquote><h3>So, what exactly <em>is</em> interning?</h3><p><strong>In technical terms</strong>: Interning is a memory optimization where Python reuses a single immutable object in memory for identical values, speeding up comparison (is check) by comparing memory addresses (pointers) instead of content.</p><p>At its core, <strong>interning</strong> is Python’s sneaky little performance trick. It means that for certain immutable objects — like small integers, short strings, or frequently reused identifiers — Python doesn’t keep making fresh copies every time.</p><p>Instead, it quietly reuses the same object in memory.</p><blockquote>It’s Python’s behind-the-scenes efficiency vault. If you assign a variable the value ‘secret’, and then another variable the same value, Python only ever places one ‘secret’ into the vault. Both of your variables get a key, and that key always opens the door to the exact same, single object.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*ZZYXxc6Em11TnWM2zXREtw.png" /><figcaption>Python’s efficiency vault: Multiple variables, one “secret” object, all sharing the same key to unlock it</figcaption></figure><p>This saves both <strong>memory</strong> and <strong>comparison time</strong> — because if two variables point to the same object, checking equality becomes lightning-fast.</p><p>Here’s a quick demo:</p><pre>a = &quot;secret&quot;<br>b = &quot;secret&quot;<br>print(a is b)  # True</pre><p>Both a and b are pointing to the same &quot;vault entry&quot; — one object, shared key.<br>But now, let’s make a tiny change:</p><pre>a = &quot;secret_code&quot;<br>b = &quot;secret_code&quot;<br>print(a is b)  # False</pre><p>Same letters. Same meaning.<br>Yet Python quietly decided these deserve <strong>two different vaults</strong>.</p><blockquote>Why?<br><em>Because interning has its own set of invisible rules — it doesn’t apply to every string, only the ones Python thinks are worth caching.<br>That’s interning doing its thing — but selectively.</em></blockquote><h3>When Does Python Decide to Intern?</h3><p>Okay, so here’s the deal with <strong>string interning</strong> in Python. If the interpreter isn’t just hoarding <em>everything</em>, how does it decide what to put in that little ‘efficiency box’ it keeps on the side?</p><p>It really comes down to what’s predictable and what it knows is going to pop up constantly.</p><p>Generally, you’ll see interning happen for:</p><ul><li><strong>Short strings</strong> that look exactly like identifiers — think simple variable names, keywords like for or while, or small, simple words.</li><li>Any string without a space or a special character. It’s like, if it’s a solid block of text, it’s a good candidate.</li><li><strong>Small integers</strong> — it’s pretty consistent with the range, usually from <strong>-5 to 256</strong>. Everything in that tiny window is pre-cached.</li><li>And honestly, sometimes the <strong>compiler</strong> just takes a gamble on what it suspects will be reused often enough to justify the memory cost.</li></ul><p>Let’s take a look at how this works under the hood:</p><pre># Identifier-like string (letters, numbers, underscores) - Interned<br>s1 = &quot;user_name&quot;<br>s2 = &quot;user_name&quot;<br>print(f&quot;s1 is s2: {s1 is s2}&quot;)  # True (They point to the same object)<br><br># String with a non-identifier character (the hyphen) - Not guaranteed to be Interned<br>t1 = &quot;user-name&quot;<br>t2 = &quot;user-name&quot;<br>print(f&quot;t1 is t2: {t1 is t2}&quot;)  # False (They are usually separate objects)</pre><blockquote><em>The key difference here is how Python classifies the strings. </em><strong><em>user_name</em></strong><em> is a simple, </em><strong><em>identifier-like</em></strong><em> string—it could be a variable name—so Python automatically interns it. The interpreter assumes it&#39;s going to be reused frequently.</em></blockquote><blockquote><em>But when you introduce a </em><strong><em>hyphen</em></strong><em> in </em><strong><em>user-name</em></strong><em>, it&#39;s no longer a valid identifier, so Python treats it like a generic block of text and won&#39;t always bother to intern it. This is the interpreter making an efficiency decision based on potential reuse.</em></blockquote><h3>So, Does This Matter for My Code?</h3><p>At the end of the day, why are we talking about this internal optimization? <strong>Memory, pure and simple.</strong></p><p>Imagine you have a massive application that defines the string literal &#39;REQUEST_PENDING&#39; five thousand times across various modules and functions. Without interning, you would literally be storing five thousand separate copies of that string in memory.</p><p>With interning, Python stores the string <strong>once</strong> and then simply points all five thousand variables to that single object. That’s a huge win for saving memory, reducing overall overhead, and keeping your application snappy.</p><p>It’s one of those <strong>silent, brilliant things</strong> Python does for us behind the scenes to optimize resource usage — you get the performance boost without having to think about the mechanism!</p><h3>The Final Takeaway</h3><p>Interning is like the <strong>main entrance to a huge building.</strong> Hundreds of people (your variables) might need to go inside, but they all use that <em>one</em> main door (the interned object)</p><p>So, the next time you see a tiny Python integer or a simple variable name, remember that <strong>Python is quietly doing the heavy lifting</strong> for you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5971153d4867" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[JWT Validation in Rails: The Security Mistake Most Devs Make]]></title>
            <link>https://medium.com/@ashiya.ajare/jwt-validation-in-rails-the-security-mistake-most-devs-make-498079d6e89b?source=rss-fdc527338328------2</link>
            <guid isPermaLink="false">https://medium.com/p/498079d6e89b</guid>
            <category><![CDATA[authentication]]></category>
            <category><![CDATA[rails]]></category>
            <category><![CDATA[ruby-on-rails]]></category>
            <category><![CDATA[jwt]]></category>
            <dc:creator><![CDATA[Ashiya Ajare]]></dc:creator>
            <pubDate>Thu, 24 Apr 2025 10:46:44 GMT</pubDate>
            <atom:updated>2025-04-24T10:46:44.784Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="JWT Authentication" src="https://cdn-images-1.medium.com/max/1024/0*ZtIvLQj-bGE9uEMn.jpg" /></figure><p><strong>Ever run into something weird while coding?</strong></p><p>While working on JWT authentication in a Ruby on Rails project, something unexpected came up. A small tweak — changing just the last character of a JWT token — didn’t break it. Instead, the token still decoded correctly.</p><p>Normally, even the slightest modification to a JWT would make it completely invalid. But interestingly, if the last character was a letter (say, ‘a’), swapping it with a few successive letters (‘b’, ‘c’, ‘d’, etc.) still worked. Same with numbers.</p><p>That raised an interesting question — why does this happen? Shouldn’t JWTs be stricter? Or is there some underlying logic at play?</p><blockquote>Quick disclaimer:</blockquote><blockquote>This behavior isn’t really a flaw in JWT itself. JWT is just a standard. The problem comes from <em>how</em> we implement it.</blockquote><p>Before we dive deeper, if you’re not familiar with JWT, you might want to check out “<a href="https://medium.com/@shubhadeepchat/explaining-json-web-tokens-jwt-with-real-life-examples-62a183889022">Explaining JSON Web Tokens (JWT) with Real-Life Examples</a>” for a quick primer.</p><h3>JWT Authentication:</h3><p>Here’s a JWT token generated during authentication in my Ruby on Rails project. When I decode it, the expected data is displayed correctly.</p><p>Token Generated:</p><pre>{<br>  &quot;token&quot;: &quot;eyJhbGciOiJIUzI1NiJ9.ejEyMjcyVyY2kijozLCJleHAiOjE3NDA1NzYxMTk9.JWinOOvHQWPYYi5N7-IXRMY3Pryipfkass11Wy000&quot;,<br>  &quot;user&quot;: {<br>    &quot;id&quot;: 3,<br>    &quot;first_name&quot;: &quot;ashiya&quot;,<br>    &quot;last_name&quot;: &quot;ajare&quot;,<br>    &quot;email&quot;: &quot;ashiyaa@gmail.com&quot;<br>  }<br>}</pre><p>Data decoded using correct JWT token:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TlvrFJHB9kBp6lw80T1hyg.png" /></figure><p>Now, I slightly modify the token — changing just the last character from ‘0’ to ‘2’. You’d expect this to break the decoding, right?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*BEsxsEvRRKMuIGUtHao_ig.png" /></figure><p>Surprisingly, the decoded data remains exactly the same !</p><p>This behavior is limited to only four successive character replacements. In the example above, the token will decode correctly if the last character ‘0’ is replaced with ‘1’, ‘2’, or ‘3’. However, replacing it with ‘4’ will result in the expected <em>‘invalid token’</em> error.</p><h3><strong>Base64 Encoding of JWT:</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*1cA65cDJKAONzfXY.jpg" /><figcaption>Base64 encoding used in JWT</figcaption></figure><p>Most of us assume that, with JWT, even the slightest change in the token would throw an error. So, why does it decode correctly ?<br>This behavior is due to the “Base64 Encoding” used in JWT.</p><p>The first two parts of a JWT token ( Header and Payload) are Base64 URL-encoded and the third part (Signature) is for ensuring integrity.</p><p>How does this encoding works in a token ? It can be broken down in three easy steps for simpler understanding of the mechanism —</p><ul><li>Base64 works by taking <strong>3 bytes of data</strong> at a time and turning them into <strong>4 characters</strong>.</li><li>But what if the total data isn’t a perfect multiple of 3?</li><li>That’s when Base64 adds a little extra padding at the end, using the = symbol.</li></ul><blockquote>This padding tells the decoder: “Hey, this part is just to fill in the gaps. Ignore it when reading the real data.”</blockquote><ul><li>JWT tokens strip out the = padding. That’s why the <strong>last character</strong> isn&#39;t always critical. You might even change it slightly, and it’ll still decode fine — which can confuse people and lead to mistakes in how we validate tokens.</li></ul><p>When the last character of the token is changed within a certain range, it might still produce a valid Base64 decoding because the structure remains valid. However, outside of this range, structure of the token breaks and the decoding fails leading to error “Invalid Token”.</p><p>However, this behavior can introduce potential security threats in the system.</p><h4>Where Things Go Wrong:</h4><p>Here’s the catch: decoding the token doesn’t mean its valid. The reason this feels like a loophole is because most JWT implementations (especially using HS256) don’t enforce strict signature verification. Developers often decode the token and assume it’s valid if no error is thrown — that’s where the problem lies.</p><p>This isn’t JWT’s fault. It’s all about <strong>how we use JWT</strong>.</p><h4>Looking at the code:</h4><p>The following code snippet represents the most common way, used by developers, to implement JWT validation in Rails.</p><pre>require &#39;jwt&#39;<br><br>SECRET_KEY = Rails.application.secrets.secret_key_base<br><br>def encode_token(payload)<br>  JWT.encode(payload, SECRET_KEY, &#39;HS256&#39;)<br>end<br><br>def decode_token(token)<br>  begin<br>    JWT.decode(token, SECRET_KEY, true, algorithm: &#39;HS256&#39;)<br>  rescue JWT::DecodeError<br>    nil<br>  end<br>end</pre><p>This implementation works well, but the function ‘decode_token’ raises an error only when the decoding of the token fails completely. However, given how Base64 encoding works, this approach does not seem favorable.</p><p>This problem can be solved in many ways and one of the ways is to implement any of the <strong>encryption algorithms</strong> and a <strong>conditional check to verify the signature</strong> to enforce strict validation of JWT tokens.</p><h4>A More Secure Approach: What I Did Differently</h4><p>The following implementation uses <strong>secure encoding and decoding of JWT tokens</strong> using <strong>RS256 (RSA Signature with SHA-256)</strong> instead of the commonly used HS256.</p><pre>module JsonWebToken<br>  PRIVATE_KEY = OpenSSL::PKey::RSA.new(File.read(&quot;private.pem&quot;))<br>  PUBLIC_KEY = OpenSSL::PKey::RSA.new(File.read(&quot;public.pem&quot;))<br><br>  def self.encode(payload, exp = 7.days.from_now)<br>    payload[:exp] = exp.to_i<br>    payload[:hash] = Digest::SHA256.hexdigest(payload.to_json)<br>    JWT.encode(payload, PRIVATE_KEY, &#39;RS256&#39;) # Use RS256<br>  end<br><br>  def self.decode(token)<br>    begin<br>      decoded = JWT.decode(token, PUBLIC_KEY, true, algorithm: &#39;RS256&#39;)[0]<br>      <br>      # Verify hash integrity<br>      expected_hash = Digest::SHA256.hexdigest(decoded.except(&quot;hash&quot;).to_json)<br>      if decoded[&quot;hash&quot;] != expected_hash<br>        raise JWT::VerificationError, &quot;Token payload tampered with!&quot;<br>      end<br><br>      unless JWT.encode(decoded, PRIVATE_KEY, &#39;RS256&#39;) == token<br>        raise JWT::VerificationError, &quot;Token signature mismatch!&quot;<br>      end<br>      HashWithIndifferentAccess.new(decoded)<br>    rescue JWT::VerificationError =&gt; e<br>      Rails.logger.error(&quot;JWT Error: Invalid token signature or payload&quot;)<br>      nil<br>    rescue JWT::DecodeError =&gt; e<br>      Rails.logger.error(&quot;JWT Decode Error: #{e.message}&quot;)<br>      nil<br>    end<br>  end<br>end</pre><p>This approach uses Asymmetric encryption for secure signing of the token. Here<strong>, RS256</strong>, which is an <strong>asymmetric algorithm</strong>. That means we have a <strong>private key to sign the token</strong> and a <strong>public key to verify it</strong>. It’s safer than HS256, which just uses one shared secret key for both.</p><p>But I didn’t stop there.</p><p>I also added a <strong>SHA256 hash of the payload</strong>. This hash acts like a fingerprint of the data inside the token. If even a single character inside the payload is changed, the hash won’t match — and we can instantly tell something was tampered with.</p><p><strong>Steps in decoding the token:</strong></p><ol><li>First, we decode the token using the <strong>public key</strong> — that gives us back the payload.</li><li>Then, we <strong>regenerate the hash</strong> from the decoded payload and check if it matches the original hash inside the token.</li><li>As an extra step, we also <strong>re-encode the decoded payload</strong> using the <strong>private key</strong>, and compare it with the original token.</li></ol><p>If either the hash check fails or the re-encoded token doesn’t match the original, we reject it — even if the token still technically “decodes”.</p><p>This way, even if someone tweaks a character and the token still looks valid, it won’t pass our integrity checks.</p><h3>Conclusion</h3><p>Most developers assume that using JWT means secure authentication. But, that’s not always the case.</p><p>This blog explores just one of the common oversights — there are more out there.</p><p>The real issue isn’t with JWT, but with how it’s implemented — especially when we overlook the importance of strict signature verification.</p><p><strong>Security is an ongoing process.</strong> Always validate JWTs properly, avoid relying on just decoding, and consider stronger algorithms like RS256 to ensure integrity.</p><p>Stay curious, stay secure. 🔐</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=498079d6e89b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>