<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vipul on Medium]]></title>
        <description><![CDATA[Stories by Vipul on Medium]]></description>
        <link>https://medium.com/@vipul319?source=rss-b1a4cdd865b7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*6MS5EG6PHmV9aKKW</url>
            <title>Stories by Vipul on Medium</title>
            <link>https://medium.com/@vipul319?source=rss-b1a4cdd865b7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 21:58:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vipul319/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Part 3: Agent Memory — Hot State, Cold State, and the Redis + Postgres Pattern]]></title>
            <link>https://medium.com/@vipul319/part-3-agent-memory-hot-state-cold-state-and-the-redis-postgres-pattern-d79fd779acc5?source=rss-b1a4cdd865b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/d79fd779acc5</guid>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[ai-agents-in-action]]></category>
            <category><![CDATA[state-management]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[redis]]></category>
            <dc:creator><![CDATA[Vipul]]></dc:creator>
            <pubDate>Tue, 14 Jul 2026 13:40:54 GMT</pubDate>
            <atom:updated>2026-07-31T11:08:27.410Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Part 3 of 5 in the series “Building Production-Grade AI Agents: From Prototype to Scale”</em></p><p>In <a href="https://medium.com/p/c4e07aba14d6">Part 2</a>, we made agents durable with Temporal — they survive crashes and restarts. But there’s a problem: on retry, the entire LangGraph re-runs from scratch. If your agent failed at step 7 of 10, you’re re-doing steps 1–6 needlessly.</p><p>We need <strong>memory</strong>. Not just “conversation history” memory — operational memory that lets agents resume mid-execution, cache expensive results, and build knowledge over time.</p><p>The solution: a two-tier system with <strong>Redis for hot state</strong> (fast, ephemeral) and <strong>Postgres for cold state</strong> (durable, queryable).</p><h3>Why Two Tiers?</h3><p>No single database handles everything an agent needs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0EsV_yaKH_n9Da4BAXuiEQ.png" /><figcaption>Two Tier Database Need ?</figcaption></figure><p>Trying to use Postgres for checkpoints means 20ms writes on every node transition. Using Redis for audit trails means losing data on restart. Each tier does what it’s good at.</p><h3>The State Flow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d9ig_EktWhfUfnqW-JyRZA.png" /><figcaption>State Flow Diagram</figcaption></figure><h3>Redis: The Hot Layer</h3><h3>What Lives in Redis</h3><pre>import redis.asyncio as redis<br>import json<br>import hashlib<br>from datetime import timedelta<br><br>class AgentHotState:<br>    &quot;&quot;&quot;Fast, ephemeral state for agent execution.&quot;&quot;&quot;<br>    def __init__(self, client: redis.Redis):<br>        self.redis = client<br>    # --- Checkpoints ---<br>    async def save_checkpoint(<br>        self, workflow_id: str, node: str, state: dict, ttl: int = 3600<br>    ):<br>        &quot;&quot;&quot;Save agent state at a node boundary. TTL: 1 hour.&quot;&quot;&quot;<br>        key = f&quot;ckpt:{workflow_id}:{node}&quot;<br>        await self.redis.setex(key, ttl, json.dumps(state, default=str))<br>    async def load_checkpoint(self, workflow_id: str, node: str) -&gt; dict | None:<br>        &quot;&quot;&quot;Load checkpoint - returns None if expired or missing.&quot;&quot;&quot;<br>        raw = await self.redis.get(f&quot;ckpt:{workflow_id}:{node}&quot;)<br>        return json.loads(raw) if raw else None<br>    async def get_latest_checkpoint(self, workflow_id: str) -&gt; tuple[str, dict] | None:<br>        &quot;&quot;&quot;Find the most recent checkpoint for a workflow.&quot;&quot;&quot;<br>        pattern = f&quot;ckpt:{workflow_id}:*&quot;<br>        keys = []<br>        async for key in self.redis.scan_iter(pattern):<br>            keys.append(key)<br>        <br>        if not keys:<br>            return None<br>        <br>        # Keys are ordered by node execution - last is most recent<br>        latest_key = sorted(keys)[-1]<br>        raw = await self.redis.get(latest_key)<br>        node_name = latest_key.decode().split(&quot;:&quot;)[-1]<br>        return (node_name, json.loads(raw))<br>    async def clear_checkpoints(self, workflow_id: str):<br>        &quot;&quot;&quot;Clean up after workflow completes.&quot;&quot;&quot;<br>        pattern = f&quot;ckpt:{workflow_id}:*&quot;<br>        keys = [key async for key in self.redis.scan_iter(pattern)]<br>        if keys:<br>            await self.redis.delete(*keys)<br>    # --- Retrieval Cache ---<br>    async def cache_retrieval(<br>        self, query: str, tenant_id: str, results: list[dict], ttl: int = 300<br>    ):<br>        &quot;&quot;&quot;Cache search results. TTL: 5 minutes.&quot;&quot;&quot;<br>        key = self._cache_key(query, tenant_id)<br>        await self.redis.setex(key, ttl, json.dumps(results))<br>    async def get_cached_retrieval(<br>        self, query: str, tenant_id: str<br>    ) -&gt; list[dict] | None:<br>        raw = await self.redis.get(self._cache_key(query, tenant_id))<br>        return json.loads(raw) if raw else None<br>    # --- Dedup Locks ---<br>    async def acquire_dedup(self, call_id: str, ttl: int = 60) -&gt; bool:<br>        &quot;&quot;&quot;Prevent duplicate LLM calls. Returns True if lock acquired.&quot;&quot;&quot;<br>        return await self.redis.set(f&quot;dedup:{call_id}&quot;, &quot;1&quot;, nx=True, ex=ttl)<br>    async def release_dedup(self, call_id: str):<br>        await self.redis.delete(f&quot;dedup:{call_id}&quot;)<br>    # --- Helpers ---<br>    def _cache_key(self, query: str, tenant_id: str) -&gt; str:<br>        h = hashlib.md5(f&quot;{tenant_id}:{query}&quot;.encode()).hexdigest()<br>        return f&quot;cache:retrieval:{h}&quot;</pre><h3>Key Design Decisions</h3><p><strong>Why TTL on checkpoints?</strong> If a workflow is abandoned (user never approves, workflow times out), checkpoints auto-expire. No garbage collection needed.</p><p><strong>Why hash the cache key?</strong> Queries can be long. A fixed-length hash keeps Redis keys uniform and avoids key-length limits.</p><p><strong>Why dedup locks?</strong> If Temporal retries an activity, you might re-run an LLM call that already completed. The dedup lock prevents wasting tokens:</p><pre>async def call_llm_with_dedup(prompt: str, call_id: str) -&gt; str:<br>    # Try to acquire lock<br>    if not await hot_state.acquire_dedup(call_id):<br>        # Another execution already running this call — wait for result<br>        await asyncio.sleep(1)<br>        cached = await hot_state.load_checkpoint(workflow_id, f&quot;llm:{call_id}&quot;)<br>        if cached:<br>            return cached[&quot;response&quot;]<br>    <br>    try:<br>        response = await llm.op(prompt)<br>        return response<br>    finally:<br>        await hot_state.release_dedup(call_id)</pre><h3>Postgres: The Cold Layer</h3><h3>Schema</h3><pre>from sqlalchemy import (<br>    Column, String, JSON, DateTime, Integer, Float, Text, Index<br>)<br>from sqlalchemy.dialects.postgresql import JSONB<br>from sqlalchemy.ext.declarative import declarative_base<br><br>Base = declarative_base()<br><br>class Conversation(Base):<br>    &quot;&quot;&quot;Complete conversation history - queryable, permanent.&quot;&quot;&quot;<br>    __tablename__ = &quot;conversations&quot;<br>    id = Column(String(64), primary_key=True)<br>    workflow_id = Column(String(64), index=True)<br>    tenant_id = Column(String(64), index=True)<br>    topic = Column(Text)<br>    messages = Column(JSONB)  # Full message array<br>    metadata = Column(JSONB)  # Model, tokens, etc.<br>    created_at = Column(DateTime, index=True)<br>    updated_at = Column(DateTime)<br>    __table_args__ = (<br>        Index(&quot;idx_conv_tenant_created&quot;, &quot;tenant_id&quot;, &quot;created_at&quot;),<br>    )<br><br>class AuditLog(Base):<br>    &quot;&quot;&quot;Every agent action - for debugging, billing, compliance.&quot;&quot;&quot;<br>    __tablename__ = &quot;audit_log&quot;<br>    id = Column(Integer, primary_key=True, autoincrement=True)<br>    workflow_id = Column(String(64), index=True)<br>    tenant_id = Column(String(64), index=True)<br>    event_type = Column(String(32))  # node_start, node_complete, error, retry<br>    node_name = Column(String(64))<br>    input_summary = Column(JSONB)    # Truncated input (not full state)<br>    output_summary = Column(JSONB)   # Truncated output<br>    duration_ms = Column(Integer)<br>    token_count = Column(Integer)<br>    cost_usd = Column(Float)<br>    error = Column(Text, nullable=True)<br>    created_at = Column(DateTime, index=True)<br>    __table_args__ = (<br>        Index(&quot;idx_audit_workflow&quot;, &quot;workflow_id&quot;, &quot;created_at&quot;),<br>        Index(&quot;idx_audit_tenant_type&quot;, &quot;tenant_id&quot;, &quot;event_type&quot;),<br>    )<br><br>class AgentMemory(Base):<br>    &quot;&quot;&quot;Long-term agent knowledge - learnings, preferences, facts.&quot;&quot;&quot;<br>    __tablename__ = &quot;agent_memory&quot;<br>    id = Column(String(64), primary_key=True)<br>    tenant_id = Column(String(64), index=True)<br>    agent_type = Column(String(64))<br>    memory_type = Column(String(32))  # fact, preference, learning, correction<br>    content = Column(Text)<br>    metadata = Column(JSONB)<br>    relevance_score = Column(Float, default=1.0)  # Decay over time<br>    created_at = Column(DateTime)<br>    last_accessed = Column(DateTime)<br>    access_count = Column(Integer, default=0)</pre><h3>The Cold State Store</h3><pre>from sqlalchemy.ext.asyncio import AsyncSession<br>from sqlalchemy import select, insert<br>from datetime import datetime, timezone<br><br><br>class AgentColdState:<br>    &quot;&quot;&quot;Durable, queryable state for agent history and memory.&quot;&quot;&quot;<br>    def __init__(self, session_factory):<br>        self.session_factory = session_factory<br>    # --- Conversations ---<br>    async def save_conversation(<br>        self, workflow_id: str, tenant_id: str, topic: str, messages: list[dict]<br>    ):<br>        async with self.session_factory() as session:<br>            conv = Conversation(<br>                id=f&quot;{workflow_id}:conv&quot;,<br>                workflow_id=workflow_id,<br>                tenant_id=tenant_id,<br>                topic=topic,<br>                messages=messages,<br>                metadata={&quot;message_count&quot;: len(messages)},<br>                created_at=datetime.now(timezone.utc),<br>                updated_at=datetime.now(timezone.utc),<br>            )<br>            session.add(conv)<br>            await session.commit()<br>    async def get_recent_conversations(<br>        self, tenant_id: str, limit: int = 10<br>    ) -&gt; list[dict]:<br>        async with self.session_factory() as session:<br>            result = await session.execute(<br>                select(Conversation)<br>                .where(Conversation.tenant_id == tenant_id)<br>                .order_by(Conversation.created_at.desc())<br>                .limit(limit)<br>            )<br>            return [row.to_dict() for row in result.scalars()]<br>    # --- Audit ---<br>    async def log_event(<br>        self,<br>        workflow_id: str,<br>        tenant_id: str,<br>        event_type: str,<br>        node_name: str,<br>        duration_ms: int = 0,<br>        token_count: int = 0,<br>        cost_usd: float = 0.0,<br>        error: str = None,<br>    ):<br>        async with self.session_factory() as session:<br>            entry = AuditLog(<br>                workflow_id=workflow_id,<br>                tenant_id=tenant_id,<br>                event_type=event_type,<br>                node_name=node_name,<br>                duration_ms=duration_ms,<br>                token_count=token_count,<br>                cost_usd=cost_usd,<br>                error=error,<br>                created_at=datetime.now(timezone.utc),<br>            )<br>            session.add(entry)<br>            await session.commit()<br>    async def get_workflow_cost(self, workflow_id: str) -&gt; float:<br>        &quot;&quot;&quot;Total cost for a workflow - useful for billing.&quot;&quot;&quot;<br>        async with self.session_factory() as session:<br>            from sqlalchemy import func<br>            result = await session.execute(<br>                select(func.sum(AuditLog.cost_usd))<br>                .where(AuditLog.workflow_id == workflow_id)<br>            )<br>            return result.scalar() or 0.0<br>    # --- Agent Memory ---<br>    async def remember(<br>        self, tenant_id: str, agent_type: str, memory_type: str, content: str<br>    ):<br>        &quot;&quot;&quot;Store a long-term memory.&quot;&quot;&quot;<br>        async with self.session_factory() as session:<br>            mem = AgentMemory(<br>                id=f&quot;{tenant_id}:{agent_type}:{hashlib.md5(content.encode()).hexdigest()[:12]}&quot;,<br>                tenant_id=tenant_id,<br>                agent_type=agent_type,<br>                memory_type=memory_type,<br>                content=content,<br>                metadata={},<br>                relevance_score=1.0,<br>                created_at=datetime.now(timezone.utc),<br>                last_accessed=datetime.now(timezone.utc),<br>                access_count=0,<br>            )<br>            session.add(mem)<br>            await session.commit()<br>    async def recall(<br>        self, tenant_id: str, agent_type: str, limit: int = 20<br>    ) -&gt; list[str]:<br>        &quot;&quot;&quot;Retrieve memories, ordered by relevance and recency.&quot;&quot;&quot;<br>        async with self.session_factory() as session:<br>            result = await session.execute(<br>                select(AgentMemory)<br>                .where(<br>                    AgentMemory.tenant_id == tenant_id,<br>                    AgentMemory.agent_type == agent_type,<br>                )<br>                .order_by(<br>                    AgentMemory.relevance_score.desc(),<br>                    AgentMemory.last_accessed.desc(),<br>                )<br>                .limit(limit)<br>            )<br>            memories = result.scalars().all()<br>            # Update access counts<br>            for mem in memories:<br>                mem.access_count += 1<br>                mem.last_accessed = datetime.now(timezone.utc)<br>            await session.commit()<br>            return [m.content for m in memories]</pre><h3>Wiring It Together: Checkpointed LangGraph Agent</h3><p>Now we modify the LangGraph nodes to checkpoint at each boundary:</p><pre>class CheckpointedAgent:<br>    &quot;&quot;&quot;<br>    LangGraph agent with Redis checkpointing.<br>    On retry, resumes from last successful node.<br>    &quot;&quot;&quot;<br>def __init__(self, workflow_id: str, hot: AgentHotState, cold: AgentColdState):<br>        self.workflow_id = workflow_id<br>        self.hot = hot<br>        self.cold = cold<br>        self._start_time = None<br>    async def run(self, initial_state: dict) -&gt; dict:<br>        graph = self._build_graph()<br>        # Check for existing checkpoint (retry scenario)<br>        checkpoint = await self.hot.get_latest_checkpoint(self.workflow_id)<br>        if checkpoint:<br>            node_name, saved_state = checkpoint<br>            # Resume from checkpoint<br>            state = saved_state<br>            # Skip to the node AFTER the checkpointed one<br>            result = await graph.ainvoke(state, config={&quot;start_at&quot;: node_name})<br>        else:<br>            # Fresh execution<br>            result = await graph.ainvoke(initial_state)<br>        # Persist final result to cold storage<br>        await self.cold.save_conversation(<br>            workflow_id=self.workflow_id,<br>            tenant_id=initial_state.get(&quot;tenant_id&quot;, &quot;default&quot;),<br>            topic=initial_state.get(&quot;query&quot;, &quot;&quot;),<br>            messages=result.get(&quot;messages&quot;, []),<br>        )<br>        # Cleanup hot state<br>        await self.hot.clear_checkpoints(self.workflow_id)<br>        return result<br>    def _build_graph(self):<br>        graph = StateGraph(ResearchState)<br>        # Wrap each node with checkpoint logic<br>        graph.add_node(&quot;retrieve&quot;, self._with_checkpoint(&quot;retrieve&quot;, retrieve_node))<br>        graph.add_node(&quot;analyze&quot;, self._with_checkpoint(&quot;analyze&quot;, analyze_node))<br>        graph.add_node(&quot;quality_check&quot;, self._with_checkpoint(&quot;quality_check&quot;, quality_check_node))<br>        graph.add_node(&quot;refine_query&quot;, self._with_checkpoint(&quot;refine_query&quot;, refine_query_node))<br>        graph.set_entry_point(&quot;retrieve&quot;)<br>        graph.add_edge(&quot;retrieve&quot;, &quot;analyze&quot;)<br>        graph.add_edge(&quot;analyze&quot;, &quot;quality_check&quot;)<br>        graph.add_conditional_edges(&quot;quality_check&quot;, should_continue, {&quot;done&quot;: END, &quot;retry&quot;: &quot;refine_query&quot;})<br>        graph.add_edge(&quot;refine_query&quot;, &quot;retrieve&quot;)<br>        return graph.compile()<br>    def _with_checkpoint(self, node_name: str, node_fn):<br>        &quot;&quot;&quot;Wrapper that checkpoints after each node completes.&quot;&quot;&quot;<br>        async def wrapped(state: dict) -&gt; dict:<br>            self._start_time = time.time()<br>            # Execute the actual node logic<br>            result = await node_fn(state)<br>            # Save checkpoint to Redis<br>            await self.hot.save_checkpoint(self.workflow_id, node_name, result)<br>            # Audit to Postgres (async, non-blocking)<br>            duration_ms = int((time.time() - self._start_time) * 1000)<br>            asyncio.create_task(<br>                self.cold.log_event(<br>                    workflow_id=self.workflow_id,<br>                    tenant_id=state.get(&quot;tenant_id&quot;, &quot;default&quot;),<br>                    event_type=&quot;node_complete&quot;,<br>                    node_name=node_name,<br>                    duration_ms=duration_ms,<br>                )<br>            )<br>            return result<br>        return wrapped</pre><h3>The Crash Recovery Sequence</h3><p>With checkpointing, here’s what happens when things fail:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*omRdgX5ZOV2DfemohGhc0A.png" /><figcaption>Crash State Sequence Diagram</figcaption></figure><p><strong>Cost savings:</strong> Without checkpointing, the retry re-runs retrieve + analyze (let’s say $0.50 in LLM calls + 2 minutes). With checkpointing, you skip straight to quality_check — saving both money and time.</p><h3>Pattern: Redis-First, Postgres-Async</h3><p>The golden rule: <strong>never let Postgres latency slow down agent execution.</strong></p><pre>async def complete_node(workflow_id, node_name, state, hot, cold):<br>    &quot;&quot;&quot;After a node completes: fast Redis write, async Postgres write.&quot;&quot;&quot;<br>    <br>    # SYNC: Save to Redis (&lt; 1ms) — agent continues immediately<br>    await hot.save_checkpoint(workflow_id, node_name, state)<br>    <br>    # ASYNC: Persist to Postgres (20-50ms) — doesn&#39;t block the agent<br>    asyncio.create_task(<br>        cold.log_event(<br>            workflow_id=workflow_id,<br>            event_type=&quot;node_complete&quot;,<br>            node_name=node_name,<br>            duration_ms=state.get(&quot;_duration_ms&quot;, 0),<br>        )<br>    )</pre><p><strong>What if Postgres write fails?</strong> That’s fine — the audit log is best-effort during execution. The critical state is in Redis. When the workflow completes, the final Postgres write is synchronous (within the Temporal activity, so it retries if needed).</p><h3>Pattern: Retrieval Caching</h3><p>The same query might hit the retrieval system multiple times (refine → re-retrieve loop). Don’t re-embed and re-search every time:</p><pre>async def retrieval_node_with_cache(state: dict) -&gt; dict:<br>    hot = get_hot_state()<br>    query = state[&quot;query&quot;]<br>    tenant_id = state.get(&quot;tenant_id&quot;, &quot;default&quot;)<br><br># Check cache first<br>    cached = await hot.get_cached_retrieval(query, tenant_id)<br>    if cached:<br>        return {**state, &quot;retrieved_docs&quot;: cached, &quot;cache_hit&quot;: True}<br>    # Cache miss - do the expensive search<br>    docs = await milvus_hybrid_search(query, tenant_id)<br>    # Cache for next time (or next iteration of the loop)<br>    await hot.cache_retrieval(query, tenant_id, docs, ttl=300)<br>    return {**state, &quot;retrieved_docs&quot;: docs, &quot;cache_hit&quot;: False}</pre><h3>Pattern: Agent Memory (Cross-Workflow Learning)</h3><p>Agents can learn from past interactions and apply that knowledge to future ones:</p><pre>async def analyze_with_memory(state: dict) -&gt; dict:<br>    cold = get_cold_state()<br>    tenant_id = state.get(&quot;tenant_id&quot;, &quot;default&quot;)<br><br># Recall relevant memories<br>    memories = await cold.recall(tenant_id=tenant_id, agent_type=&quot;researcher&quot;)<br>    # Include memories in the prompt<br>    memory_context = &quot;\n&quot;.join(f&quot;- {m}&quot; for m in memories[:5])<br>    prompt = f&quot;&quot;&quot;Previous learnings:<br>{memory_context}<br>Now analyze: {state[&#39;query&#39;]}<br>Documents: {format_docs(state[&#39;retrieved_docs&#39;])}&quot;&quot;&quot;<br>    analysis = await llm.ainvoke(prompt)<br>    # Learn from this interaction (if noteworthy)<br>    if state.get(&quot;quality_score&quot;, 0) &gt; 0.9:<br>        learning = f&quot;For queries like &#39;{state[&#39;query&#39;]}&#39;, &quot;<br>        learning += f&quot;the refined query &#39;{state.get(&#39;refined_query&#39;, &#39;&#39;)}&#39; worked well.&quot;<br>        await cold.remember(<br>            tenant_id=tenant_id,<br>            agent_type=&quot;researcher&quot;,<br>            memory_type=&quot;learning&quot;,<br>            content=learning,<br>        )<br>    return {**state, &quot;analysis&quot;: analysis}</pre><h3>Observability: What Did the Agent Do?</h3><p>With the audit log in Postgres, you can answer crucial questions:</p><pre>-- How much did workflow X cost?<br>SELECT SUM(cost_usd) FROM audit_log WHERE workflow_id = &#39;wf_123&#39;;<br><br>-- What&#39;s our average agent execution time?<br>SELECT <br>    node_name,<br>    AVG(duration_ms) as avg_ms,<br>    COUNT(*) as executions<br>FROM audit_log<br>WHERE event_type = &#39;node_complete&#39;<br>GROUP BY node_name<br>ORDER BY avg_ms DESC;<br>-- Which workflows had retries?<br>SELECT workflow_id, COUNT(*) as attempts<br>FROM audit_log<br>WHERE event_type = &#39;retry&#39;<br>GROUP BY workflow_id<br>HAVING COUNT(*) &gt; 1;<br>-- Cost per tenant this month<br>SELECT <br>    tenant_id,<br>    SUM(cost_usd) as total_cost,<br>    COUNT(DISTINCT workflow_id) as workflow_count<br>FROM audit_log<br>WHERE created_at &gt;= DATE_TRUNC(&#39;month&#39;, NOW())<br>GROUP BY tenant_id<br>ORDER BY total_cost DESC;</pre><h3>Summary: The Two-Tier Pattern</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cvdLyVyjMT70hNoT0idHKA.png" /><figcaption>Data Flow Architecture</figcaption></figure><p><strong>Rules:</strong></p><ol><li>Redis writes are synchronous (agent waits) — they’re &lt;1ms so it’s fine</li><li>Postgres writes are async during execution — don’t block the agent</li><li>Final Postgres write is synchronous — happens inside Temporal activity (retryable)</li><li>On crash: load from Redis first, fallback to Postgres</li><li>On workflow complete: flush to Postgres, delete from Redis</li></ol><p><em>Next: </em><a href="https://medium.com/p/fecb02272678"><em>Part 4 — Hybrid Retrieval with Milvus + BGE-M3 →</em> </a><br><em>Previous: </em><a href="https://medium.com/p/c4e07aba14d6"><em>← Part 2— Durable Agent Execution with LangGraph + Temporal</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d79fd779acc5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 2: Durable Agent Execution with LangGraph + Temporal]]></title>
            <link>https://medium.com/@vipul319/part-2-durable-agent-execution-with-langgraph-temporal-c4e07aba14d6?source=rss-b1a4cdd865b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4e07aba14d6</guid>
            <category><![CDATA[fault-tolerance]]></category>
            <category><![CDATA[temporal]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[design-systems]]></category>
            <dc:creator><![CDATA[Vipul]]></dc:creator>
            <pubDate>Fri, 10 Jul 2026 03:28:56 GMT</pubDate>
            <atom:updated>2026-07-10T03:28:56.617Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Part 2 of 5 in the series “Building Production-Grade AI Agents: From Prototype to Scale”</em></p><p>In <a href="https://medium.com/p/870021d3afb2">Part 1</a>, we identified the core problem: AI agents fail mid-execution and lose all progress. In this part, we solve it by combining LangGraph’s graph-based agent logic with Temporal’s durable execution guarantees.</p><p>The result: <strong>agents that finish what they start, no matter what goes wrong.</strong></p><h3>The Core Insight</h3><p>LangGraph defines <em>what</em> agents think. Temporal guarantees <em>that</em> they finish thinking.</p><pre>LangGraph = a directed graph of reasoning steps<br>Temporal  = a system that ensures every step completes (or retries until it does)</pre><p>The integration pattern is simple: <strong>each LangGraph graph runs inside a Temporal activity.</strong> Temporal handles retries, timeouts, and crash recovery. LangGraph handles the agent logic.</p><h3>LangGraph: Graph-Based Agent Logic</h3><p>If you’re coming from simple LLM chains, LangGraph’s key advantage is <strong>explicit control flow.</strong> You define nodes (what the agent does at each step) and edges (how it decides what to do next).</p><pre>from langgraph.graph import StateGraph, END<br>from typing import TypedDict<br><br>class ResearchState(TypedDict):<br>    query: str<br>    retrieved_docs: list[dict]<br>    analysis: str<br>    quality_score: float<br>    iteration: int<br><br>async def retrieve_node(state: ResearchState) -&gt; ResearchState:<br>    &quot;&quot;&quot;Search for relevant documents.&quot;&quot;&quot;<br>    docs = await retriever.search(state[&quot;query&quot;], top_k=10)<br>    return {**state, &quot;retrieved_docs&quot;: docs}<br><br>async def analyze_node(state: ResearchState) -&gt; ResearchState:<br>    &quot;&quot;&quot;Analyze retrieved documents with LLM.&quot;&quot;&quot;<br>    prompt = f&quot;Analyze these documents for: {state[&#39;query&#39;]}\n\n&quot;<br>    prompt += &quot;\n---\n&quot;.join(d[&quot;content&quot;] for d in state[&quot;retrieved_docs&quot;])<br>    <br>    analysis = await llm.ainvoke(prompt)<br>    return {**state, &quot;analysis&quot;: analysis}<br><br>async def quality_check_node(state: ResearchState) -&gt; ResearchState:<br>    &quot;&quot;&quot;Score the analysis quality.&quot;&quot;&quot;<br>    score = await evaluate_quality(state[&quot;analysis&quot;], state[&quot;query&quot;])<br>    return {**state, &quot;quality_score&quot;: score}<br><br>async def refine_query_node(state: ResearchState) -&gt; ResearchState:<br>    &quot;&quot;&quot;Generate a better query based on what we found.&quot;&quot;&quot;<br>    refined = await llm.ainvoke(<br>        f&quot;The query &#39;{state[&#39;query&#39;]}&#39; got low-quality results. &quot;<br>        f&quot;Generate a better search query.&quot;<br>    )<br>    return {<br>        **state, <br>        &quot;query&quot;: refined, <br>        &quot;iteration&quot;: state[&quot;iteration&quot;] + 1,<br>    }<br><br>def should_continue(state: ResearchState) -&gt; str:<br>    &quot;&quot;&quot;Conditional edge: continue or stop.&quot;&quot;&quot;<br>    if state[&quot;quality_score&quot;] &gt;= 0.8:<br>        return &quot;done&quot;<br>    if state[&quot;iteration&quot;] &gt;= 3:<br>        return &quot;done&quot;  # max retries<br>    return &quot;retry&quot;<br><br>def build_research_graph():<br>    graph = StateGraph(ResearchState)<br>    graph.add_node(&quot;retrieve&quot;, retrieve_node)<br>    graph.add_node(&quot;analyze&quot;, analyze_node)<br>    graph.add_node(&quot;quality_check&quot;, quality_check_node)<br>    graph.add_node(&quot;refine_query&quot;, refine_query_node)<br>    graph.set_entry_point(&quot;retrieve&quot;)<br>    graph.add_edge(&quot;retrieve&quot;, &quot;analyze&quot;)<br>    graph.add_edge(&quot;analyze&quot;, &quot;quality_check&quot;)<br>    graph.add_conditional_edges(<br>p        &quot;quality_check&quot;,<br>        should_continue,<br>        {&quot;done&quot;: END, &quot;retry&quot;: &quot;refine_query&quot;},<br>    )<br>    graph.add_edge(&quot;refine_query&quot;, &quot;retrieve&quot;)<br>    return graph.compile()</pre><p>This gives you:</p><ul><li><strong>Loops</strong>: Quality too low? Refine and retry.</li><li><strong>Conditionals</strong>: Branch based on state, not just the last LLM output.</li><li><strong>Explicit state</strong>: Every node receives and returns typed state — no hidden globals.</li></ul><p>But it doesn’t give you durability. If the process dies between “analyze” and “quality_check,” all work is lost.</p><h3>Temporal: Making It Durable</h3><p>Temporal is a workflow engine. It records every step as an event. If a worker dies, another worker picks up the workflow and replays it from the last completed step.</p><h3>Wrapping LangGraph as a Temporal Activity</h3><pre>from temporalio import activity, workflow<br>from temporalio.common import RetryPolicy<br>from datetime import timedelta<br><br>@activity.defn<br>async def run_research_agent(query: str) -&gt; dict:<br>    &quot;&quot;&quot;<br>    A complete LangGraph agent, wrapped as one Temporal activity.<br>    <br>    If this fails (LLM timeout, OOM, server crash):<br>    - Temporal retries automatically<br>    - Up to 3 attempts with exponential backoff<br>    - Heartbeat ensures Temporal knows we&#39;re alive<br>    &quot;&quot;&quot;<br>    graph = build_research_graph()<br>    <br>    initial_state = {<br>        &quot;query&quot;: query,<br>        &quot;retrieved_docs&quot;: [],<br>        &quot;analysis&quot;: &quot;&quot;,<br>        &quot;quality_score&quot;: 0.0,<br>        &quot;iteration&quot;: 0,<br>    }<br>    <br>    result = await graph.ainvoke(initial_state)<br>    <br>    # Heartbeat: tell Temporal we&#39;re still working<br>    activity.heartbeat(f&quot;completed with score={result[&#39;quality_score&#39;]}&quot;)<br>    <br>    return {<br>        &quot;analysis&quot;: result[&quot;analysis&quot;],<br>        &quot;quality_score&quot;: result[&quot;quality_score&quot;],<br>        &quot;iterations&quot;: result[&quot;iteration&quot;],<br>    }<br><br>@activity.defn<br>async def run_synthesis_agent(findings: list[dict]) -&gt; str:<br>    &quot;&quot;&quot;Synthesize multiple research findings into a report.&quot;&quot;&quot;<br>    graph = build_synthesis_graph()<br>    result = await graph.ainvoke({&quot;findings&quot;: findings})<br>    return result[&quot;report&quot;]</pre><h3>The Temporal Workflow</h3><pre>@workflow.defn<br>class ResearchPipeline:<br>    &quot;&quot;&quot;<br>    Durable research pipeline:<br>    1. Research (with retries)<br>    2. Human approval (waits indefinitely, survives restarts)<br>    3. Synthesis (with retries)<br>    &quot;&quot;&quot;<br><br>def __init__(self):<br>        self._approved = False<br>        self._status = &quot;starting&quot;<br>    @workflow.run<br>    async def run(self, topic: str) -&gt; dict:<br>        self._status = &quot;researching&quot;<br>        # Step 1: Research (retries on failure)<br>        findings = await workflow.execute_activity(<br>            run_research_agent,<br>            topic,<br>            start_to_close_timeout=timedelta(minutes=5),<br>            heartbeat_timeout=timedelta(seconds=60),<br>            retry_policy=RetryPolicy(<br>                maximum_attempts=3,<br>                backoff_coefficient=2.0,<br>                maximum_interval=timedelta(seconds=30),<br>            ),<br>        )<br>        self._status = &quot;awaiting_approval&quot;<br>        # Step 2: Wait for human (durable - survives crashes/deploys)<br>        await workflow.wait_condition(lambda: self._approved)<br>        self._status = &quot;synthesizing&quot;<br>        # Step 3: Synthesize<br>        report = await workflow.execute_activity(<br>            run_synthesis_agent,<br>            [findings],<br>            start_to_close_timeout=timedelta(minutes=3),<br>            retry_policy=RetryPolicy(maximum_attempts=2),<br>        )<br>        self._status = &quot;complete&quot;<br>        return {&quot;report&quot;: report, &quot;findings&quot;: findings}<br>    @workflow.signal<br>    async def approve(self):<br>        &quot;&quot;&quot;External signal: human approved the findings.&quot;&quot;&quot;<br>        self._approved = True<br>    @workflow.query<br>    def get_status(self) -&gt; str:<br>        &quot;&quot;&quot;Query current status without affecting workflow.&quot;&quot;&quot;<br>        return self._status</pre><h3>What Temporal Gives You: Crash Recovery</h3><p>Here’s the sequence when things go wrong:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o2Ao70qqrBYoxGvpoeEmxg.png" /><figcaption>Crash Recovery using Temporal</figcaption></figure><p><strong>Key point:</strong> In this basic version, the retry re-runs the entire LangGraph from scratch. In Part 3, we’ll add <strong>Redis checkpointing so retries resume</strong> <strong>from the last successful node — not from the beginning</strong>.</p><h3>What Temporal Gives You: Human-in-the-Loop</h3><p>This is where Temporal truly shines. A workflow can <strong>wait indefinitely</strong> for a human signal — and it costs nothing. No blocked threads, no polling, no state to manage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ChNZ1NByPcMAP0T2_nuG_g.jpeg" /><figcaption>Human in the loop (Agentic Ai)</figcaption></figure><p><strong>What happens during those 3 days:</strong></p><ul><li>The workflow exists only as events in Temporal’s database</li><li>No threads blocked, no memory consumed</li><li>Servers can restart, redeploy — doesn’t matter</li><li>When the signal arrives, a worker picks up the workflow and replays it</li></ul><h3>What Temporal Gives You: Multi-Agent Fan-Out</h3><p>Run multiple LangGraph agents in parallel as child workflows or concurrent activities:</p><pre>@workflow.defn<br>class MultiAgentResearch:<br>    @workflow.run<br>    async def run(self, topic: str) -&gt; dict:<br>        # Fan-out: research + critique in parallel<br>        research_task = workflow.execute_activity(<br>            run_research_agent, topic,<br>            start_to_close_timeout=timedelta(minutes=5),<br>            retry_policy=RetryPolicy(maximum_attempts=3),<br>        )<br>        critic_task = workflow.execute_activity(<br>            run_critic_agent, topic,<br>            start_to_close_timeout=timedelta(minutes=5),<br>            retry_policy=RetryPolicy(maximum_attempts=3),<br>        )<br>        # Fan-in: wait for both<br>        research, critique = await asyncio.gather(<br>            research_task, critic_task<br>        )<br>        # Synthesize using both perspectives<br>        report = await workflow.execute_activity(<br>            run_synthesis_agent,<br>            {&quot;research&quot;: research, &quot;critique&quot;: critique},<br>            start_to_close_timeout=timedelta(minutes=3),<br>        )<br>        return report</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MWCQKDY2kqMK2ROLRVPZbA.png" /></figure><h3>Running the Temporal Worker</h3><pre>import asyncio<br>from temporalio.client import Client<br>from temporalio.worker import Worker<br><br>async def main():<br>    # Connect to Temporal server<br>    client = await Client.connect(&quot;localhost:7233&quot;)<br>    # Start worker (processes activities + workflows)<br>    worker = Worker(<br>        client,<br>        task_queue=&quot;agent-tasks&quot;,<br>        workflows=[ResearchPipeline, MultiAgentResearch],<br>        activities=[run_research_agent, run_synthesis_agent, run_critic_agent],<br>    )<br>    await worker.run()<br><br># Start a workflow from your API<br>async def start_research(topic: str) -&gt; str:<br>    client = await Client.connect(&quot;localhost:7233&quot;)<br>    handle = await client.start_workflow(<br>        ResearchPipeline.run,<br>        topic,<br>        id=f&quot;research-{uuid4()}&quot;,<br>        task_queue=&quot;agent-tasks&quot;,<br>    )<br>    return handle.id  # Return workflow ID for status checks<br><br># Send approval signal<br>async def approve_research(workflow_id: str):<br>    client = await Client.connect(&quot;localhost:7233&quot;)<br>    handle = client.get_workflow_handle(workflow_id)<br>    await handle.signal(ResearchPipeline.approve)<br><br># Query status<br>async def get_status(workflow_id: str) -&gt; str:<br>    client = await Client.connect(&quot;localhost:7233&quot;)<br>    handle = client.get_workflow_handle(workflow_id)<br>    return await handle.query(ResearchPipeline.get_status)</pre><h3>Practical Tips</h3><h3>1. Set Heartbeat Timeouts</h3><p>If your LangGraph agent takes &gt;30 seconds (it will), use heartbeats so Temporal doesn’t kill it prematurely:</p><pre>@activity.defn<br>async def run_long_agent(query: str) -&gt; dict:<br>    graph = build_graph()<br>    <br>    # Heartbeat during long operations<br>    for i, step in enumerate(graph.stream({&quot;query&quot;: query})):<br>        activity.heartbeat(f&quot;step {i}: {step.get(&#39;node&#39;, &#39;unknown&#39;)}&quot;)<br>    <br>    return step</pre><h3>2. Make Activities Idempotent</h3><p>Temporal may run an activity twice (crash after completion but before acknowledgment). Design for it:</p><pre>@activity.defn<br>async def run_research_agent(query: str, idempotency_key: str) -&gt; dict:<br>    # Check if we already have results for this key<br>    cached = await redis.get(f&quot;result:{idempotency_key}&quot;)<br>    if cached:<br>        return json.loads(cached)<br>    <br>    # Run the agent<br>    result = await graph.ainvoke({&quot;query&quot;: query})<br>    <br>    # Cache result (idempotent on retry)<br>    await redis.setex(f&quot;result:{idempotency_key}&quot;, 3600, json.dumps(result))<br>    return result</pre><h3>3. Separate Task Queues by Cost</h3><p>Put expensive GPU-heavy activities on dedicated workers:</p><pre># Cheap activities (orchestration, caching)<br>worker_light = Worker(client, task_queue=&quot;agent-light&quot;, ...)<br><br># Expensive activities (LLM calls, embedding)<br>worker_heavy = Worker(client, task_queue=&quot;agent-heavy&quot;, ...)</pre><h3>4. Use Workflow Timeouts as Cost Guards</h3><p>Prevent runaway agents from burning your API budget:</p><pre>handle = await client.start_workflow(<br>    ResearchPipeline.run,<br>    topic,<br>    id=f&quot;research-{uuid4()}&quot;,<br>    task_queue=&quot;agent-tasks&quot;,<br>    execution_timeout=timedelta(minutes=30),  # Hard cap<br>)</pre><h3>What’s Missing</h3><p>This setup re-runs the entire LangGraph from scratch on retry. If your graph has 8 nodes and it fails at node 7, you waste the work from nodes 1–6.</p><p>In the next part, we’ll add <strong>Redis checkpointing</strong> — so retries resume from the last successful node, not from the beginning. Combined with Postgres for durable history, you get a complete two-tier memory system.</p><p><em>Next: </em><a href="https://medium.com/p/d79fd779acc5"><em>Part 3 — Agent Memory: The Redis + Postgres Pattern →</em> </a><br><em>Previous: ← </em><a href="https://medium.com/p/870021d3afb2/edit"><em>Part 1 — Why Your AI Agent Architecture Won’t Scale</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4e07aba14d6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 1: Why Your AI Agent Architecture Won’t Scale]]></title>
            <link>https://medium.com/@vipul319/part-1-why-your-ai-agent-architecture-wont-scale-870021d3afb2?source=rss-b1a4cdd865b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/870021d3afb2</guid>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[langgraph]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[agentic-rag]]></category>
            <dc:creator><![CDATA[Vipul]]></dc:creator>
            <pubDate>Fri, 10 Jul 2026 03:26:28 GMT</pubDate>
            <atom:updated>2026-07-10T03:26:28.555Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hhSpx7XIEubJ9LpCKQZT8Q.jpeg" /><figcaption>Image generated by AI</figcaption></figure><p><em>Part 1 of 5 in the series “Building Production-Grade AI Agents: From Prototype to Scale”</em></p><p>You built an AI agent. It works in your notebook. It demos well. Then you deploy it and everything falls apart.</p><p>The LLM times-out mid-workflow. The server restarts and 8 minutes of work vanishes. Your retrieval returns documents about “retry” but misses docs about “retries.” A human needs to approve something, and your entire pipeline blocks for three days — in memory.</p><p><strong>The prototype-to-production gap for AI agents is real.</strong> And it’s not a prompt engineering problem — it’s an infrastructure problem.</p><p>In this series, I’ll walk through a five-component architecture that solves these problems cleanly. Each piece has one job:</p><pre>LangGraph   → thinking<br>Temporal    → reliability<br>Redis       → hot state<br>Postgres    → cold state<br>Milvus      → knowledge retrieval</pre><p>This isn’t theoretical. It’s the architecture I use for multi-agent systems that need to be durable, observable, and scalable.</p><h3>The Problems Nobody Talks About</h3><h3>Problem 1: Agents Die Mid-Execution</h3><p>Your agent is 8 steps into a 10-step research workflow. The LLM provider returns a 500. Your server gets restarted for a deploy. The container hits its memory limit.</p><p><strong>Result:</strong> All work is lost. The user retries from scratch.</p><p>Most frameworks don’t have an answer for this beyond “wrap it in a try/catch and retry the whole thing.” That’s not good enough when step 7 cost $2 in API calls and took 4 minutes.</p><h3>Problem 2: No Durable Human-in-the-Loop</h3><p>Real workflows need human gates: approval before executing a plan, review before sending an email, sign-off before deploying code.</p><p>But your agent runs in a single process. If you need human approval, you either:</p><ul><li>Block a thread/worker for hours/days (wasteful, fragile)</li><li>Build your own queue + state machine + polling (reinventing Temporal badly)</li><li>Give up and make it synchronous (defeats the purpose)</li></ul><h3>Problem 3: Retrieval Fails Silently</h3><p>Your RAG pipeline returns “relevant” documents, but the agent gives a wrong or incomplete answer. Why?</p><ul><li><strong>Vocabulary mismatch</strong>: User searched “retry mechanism” but the doc says “automatic re-execution on failure”</li><li><strong>Bad chunking</strong>: The relevant paragraph got split across two chunks, neither retrieves well alone</li><li><strong>No context</strong>: A chunk says “set max_attempts to 3” but doesn’t mention it’s about Temporal — the embedding doesn’t know</li></ul><p>You can’t fix these with a better prompt. You need better retrieval infrastructure.</p><h3>Problem 4: State Is Everywhere and Nowhere</h3><p>Your agent needs:</p><ul><li>Fast checkpoint access mid-execution (milliseconds matter)</li><li>Queryable conversation history (for analytics, debugging)</li><li>Audit trail (who asked what, what did the agent do, how long, how much)</li><li>Semantic memory (what has this agent learned over time)</li></ul><p>No single database handles all of these well. You need a deliberate state architecture.</p><h3>The Architecture</h3><p>Here’s what I landed on after working on production agent systems:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FKW45ApKQ7UE4BErXN4bXA.jpeg" /><figcaption>Production AI Agent Architecture ( TEMPORAL + LANGRAPH)</figcaption></figure><h3>Why Not Just pgvector?</h3><p>pgvector works for &lt;1M vectors with moderate query load. But:</p><ul><li>It shares resources with your OLTP workload</li><li>No native hybrid (BM25 + vector) search</li><li>Performance degrades at scale — it’s an extension, not a purpose-built vector engine</li><li>No GPU acceleration</li></ul><p>If your vector index is a critical path for agent quality (it is), give it a dedicated system.</p><h3>Why Not Just LangChain’s Built-in Memory?</h3><p>LangChain’s memory is in-process. It doesn’t survive restarts, can’t be queried externally, doesn’t scale across workers, and isn’t observable. Fine for notebooks; not for production.</p><h3>Why Not Just a Queue (SQS, RabbitMQ)?</h3><p>Queues handle message delivery. They don’t handle:</p><ul><li>Workflow state (which step am I on?)</li><li>Conditional branching (if quality is low, retry)</li><li>Signals (human approved, resume)</li><li>Queries (what’s the current status?)</li><li>Replay (crash happened, resume from step 5)</li></ul><p>Temporal is a workflow engine, not a queue. The difference matters.</p><h3>What Each Part of This Series Covers</h3><p><strong>Part 2: Durable Agent Execution with LangGraph + Temporal</strong> How to wrap LangGraph agents in Temporal activities, handle retries, implement human-in-the-loop gates, and recover from crashes automatically.</p><p><strong>Part 3: Agent Memory — The Redis + Postgres Pattern</strong> Designing a two-tier state system: Redis for fast checkpoints and caching, Postgres for durable history and audit. How agents checkpoint mid-execution and resume after failures.</p><p><strong>Part 4: Hybrid Retrieval with Milvus + BGE-M3</strong> Chunking strategies that don’t destroy context, hybrid dense+sparse search with RRF fusion, cross-encoder re-ranking, and how to tune retrieval without a labeled dataset.</p><p><strong>Part 5: Putting It All Together</strong> The full end-to-end pipeline: request → Temporal → LangGraph → Milvus → Redis → Postgres → response. Production tips, observability, and common pitfalls.</p><h3>Before You Start: Is This For You?</h3><p><strong>This architecture is right if:</strong></p><ul><li>Your agent workflows take &gt;30 seconds</li><li>Failures during execution are unacceptable (you can’t just retry from scratch)</li><li>You need human approval gates</li><li>You’re serving multiple tenants</li><li>Retrieval quality directly impacts output quality</li><li>You need audit trails (compliance, debugging, cost tracking)</li></ul><p><strong>This is overkill if:</strong></p><ul><li>You’re building a chatbot with simple Q&amp;A</li><li>Workflows complete in &lt;10 seconds</li><li>You’re prototyping and don’t have users yet</li><li>A single LLM call with RAG is sufficient</li></ul><h3>What You’ll Be Able to Build</h3><p>By the end of this series, you’ll have the architecture for:</p><ul><li><strong>Research agents</strong> that survive crashes and resume from where they left off</li><li><strong>Multi-agent pipelines</strong> with fan-out/fan-in and shared memory</li><li><strong>Human-in-the-loop workflows</strong> that wait days for approval without blocking resources</li><li><strong>RAG systems</strong> that find what keyword search misses (and vice versa)</li><li><strong>Observable agent systems</strong> where you can answer “what did the agent do, why, and how much did it cost?”</li></ul><p>Let’s build it.</p><p><em>Next up: </em><a href="https://medium.com/p/c4e07aba14d6"><em>Part 2 — Durable Agent Execution with LangGraph + Temporal →</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=870021d3afb2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 4: Hybrid Retrieval with Milvus + BGE-M3]]></title>
            <link>https://medium.com/@vipul319/part-4-hybrid-retrieval-with-milvus-bge-m3-fecb02272678?source=rss-b1a4cdd865b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/fecb02272678</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[milvus]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[vector-embeddings]]></category>
            <category><![CDATA[hybrid-search]]></category>
            <dc:creator><![CDATA[Vipul]]></dc:creator>
            <pubDate>Wed, 08 Jul 2026 17:07:52 GMT</pubDate>
            <atom:updated>2026-07-09T04:46:16.749Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Part 4 of 5 in the series “Building Production-Grade AI Agents: From Prototype to Scale”</em></p><p>In <a href="https://medium.com/p/d79fd779acc5">Part 3</a>, we gave agents memory — Redis for fast checkpoints, Postgres for durable history. But memory without knowledge is useless. Your agents need to <strong>find relevant information</strong> from large document collections.</p><p>This is where most RAG systems silently fail. Not because the LLM is bad, but because the retrieval step returns the wrong documents. The agent confidently answers based on irrelevant context — and you don’t even know it’s wrong.</p><p>In this part, we’ll build a retrieval system that combines <strong>semantic understanding</strong> (dense vectors) with <strong>keyword precision</strong> (sparse vectors), fused through Reciprocal Rank Fusion. We’ll use Milvus for the vector search and BGE-M3 as the embedding model — one model that produces both dense and sparse vectors in a single pass.</p><h3>Why Hybrid Search?</h3><p>Pure semantic search has a fatal flaw: <strong>vocabulary mismatch.</strong></p><p>User searches: &quot;Temporal retry configuration&quot; Document says: &quot;Set maximum_attempts and backoff_coefficient in the RetryPolicy&quot;</p><p>A dense embedding model <em>might</em> connect these — or it might not. The words are completely different. If it misses, your agent gets no relevant context and hallucinates an answer.</p><p>Pure keyword search has the opposite problem: <strong>synonym blindness.</strong></p><p>User searches: &quot;how to make agents fault tolerant&quot; Document says: &quot;durable execution guarantees workflow completion despite failures&quot;</p><p>BM25 finds zero matching terms. The most relevant document in your corpus gets a score of zero.</p><p><strong>Hybrid search solves both:</strong></p><pre>Dense (semantic):  &quot;fault tolerant&quot; ≈ &quot;durable execution&quot; ✓<br>Sparse (keyword):  &quot;RetryPolicy&quot; = &quot;RetryPolicy&quot; ✓<br>Combined:          Best of both worlds</pre><h3>Why Milvus?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2ElKUgMcD1J5zi6_9ttF2g.png" /><figcaption>Why Milvus</figcaption></figure><h3>Why BGE-M3?</h3><p>The key decision for hybrid search: what model produces your sparse vectors?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*THQVu2iZ6fsEDurCf79aTQ.png" /><figcaption>Comparision analysis between BM25, SPLADE and BGE-M3</figcaption></figure><p><strong>One model for everything</strong></p><p>BGE-M3 wins on operational simplicity: <strong>one model, one forward pass, both vectors.</strong> Its sparse quality is within 2–3% of dedicated SPLADE, and you get a strong dense embedding for free.</p><h3>How BGE-M3’s Sparse Vectors Beat BM25</h3><pre># BM25 for &quot;temporal workflow retry&quot;<br>bm25_sparse = {<br>    &quot;temporal&quot;: 2.8,<br>    &quot;workflow&quot;: 1.9,<br>    &quot;retry&quot;: 2.4,<br>    # Only exact words. &quot;retries&quot; won&#39;t match.<br>}<br><br># BGE-M3 for &quot;temporal workflow retry&quot;  <br>bgem3_sparse = {<br>    &quot;temporal&quot;: 2.1,<br>    &quot;workflow&quot;: 1.8,<br>    &quot;retry&quot;: 2.3,<br>    # EXPANSION - related terms activated:<br>    &quot;retries&quot;: 1.4,<br>    &quot;orchestration&quot;: 0.9,<br>    &quot;execution&quot;: 0.7,<br>    &quot;failure&quot;: 0.6,<br>    &quot;backoff&quot;: 0.4,<br>}</pre><p>That expansion is the difference between finding the “RetryPolicy” document and missing it entirely.</p><h3>Chunking: Where Retrieval Actually Fails</h3><p>Before we index anything, we need to split documents into chunks. This is where most RAG systems break — and it’s invisible because retrieval doesn’t throw errors, it just returns the wrong results.</p><h3>The Fundamental Tension</h3><pre>Small chunks (128-256 tokens)     Large chunks (512-1024 tokens)<br>├── More precise retrieval         ├── More context per result<br>├── Less noise in results          ├── Fewer fragments<br>├── But: lose surrounding context  ├── But: dilute relevance signal<br>└── Dense search works better      └── Sparse search works better</pre><h3>Strategy: Structure-Aware Chunking with Context Prepending</h3><p>The best approach respects document structure and makes each chunk self-contained:</p><pre>import re<br>from dataclasses import dataclass, field<br><br>@dataclass<br>class Chunk:<br>    text: str<br>    heading_path: list[str]<br>    chunk_type: str  # prose, code, table, list<br>    source: str<br>    position: int<br><br>class StructureAwareChunker:<br>    &quot;&quot;&quot;<br>    Split by document structure (headings, code blocks).<br>    Prepend heading context so each chunk is self-contained.<br>    &quot;&quot;&quot;<br>    def __init__(self, max_tokens: int = 512, overlap_sentences: int = 1):<br>        self.max_tokens = max_tokens<br>        self.overlap_sentences = overlap_sentences<br>    def chunk(self, text: str, source: str) -&gt; list[Chunk]:<br>        sections = self._split_by_headings(text)<br>        chunks = []<br>        for section in sections:<br>            headings = section[&quot;headings&quot;]<br>            content = section[&quot;content&quot;].strip()<br>            if not content:<br>                continue<br>            chunk_type = self._detect_type(content)<br>            # Rule: never split code blocks or tables<br>            if chunk_type in (&quot;code&quot;, &quot;table&quot;):<br>                chunks.append(Chunk(<br>                    text=self._with_context(content, headings),<br>                    heading_path=headings,<br>                    chunk_type=chunk_type,<br>                    source=source,<br>                    position=len(chunks),<br>                ))<br>                continue<br>            # Split large prose sections at sentence boundaries<br>            if self._token_count(content) &lt;= self.max_tokens:<br>                chunks.append(Chunk(<br>                    text=self._with_context(content, headings),<br>                    heading_path=headings,<br>                    chunk_type=chunk_type,<br>                    source=source,<br>                    position=len(chunks),<br>                ))<br>            else:<br>                for sub in self._split_sentences(content, self.max_tokens):<br>                    chunks.append(Chunk(<br>                        text=self._with_context(sub, headings),<br>                        heading_path=headings,<br>                        chunk_type=&quot;prose&quot;,<br>                        source=source,<br>                        position=len(chunks),<br>                    ))<br>        return chunks<br>    def _with_context(self, text: str, headings: list[str]) -&gt; str:<br>        &quot;&quot;&quot;<br>        Prepend heading breadcrumb. This is CRITICAL.<br>        <br>        Without: &quot;Set max_attempts to 3 and backoff_coefficient to 2.0&quot;<br>        With:    &quot;[Temporal &gt; Configuration &gt; Retry Policy] Set max_attempts...&quot;<br>        <br>        The embedding now knows this chunk is about Temporal retry config.<br>        &quot;&quot;&quot;<br>        if headings:<br>            context = &quot; &gt; &quot;.join(headings)<br>            return f&quot;[{context}]\n{text}&quot;<br>        return text<br>    def _detect_type(self, text: str) -&gt; str:<br>        if &quot;```&quot; in text:<br>            return &quot;code&quot;<br>        if re.search(r&#39;^\|.*\|.*\|&#39;, text, re.MULTILINE):<br>            return &quot;table&quot;<br>        if re.search(r&#39;^\s*[-*•]\s&#39;, text, re.MULTILINE):<br>            return &quot;list&quot;<br>        return &quot;prose&quot;<br>    def _split_by_headings(self, text: str) -&gt; list[dict]:<br>        &quot;&quot;&quot;Split markdown by heading hierarchy.&quot;&quot;&quot;<br>        lines = text.split(&quot;\n&quot;)<br>        sections = []<br>        current_headings = []<br>        current_content = []<br>        for line in lines:<br>            heading_match = re.match(r&#39;^(#{1,4})\s+(.+)&#39;, line)<br>            if heading_match:<br>                # Save previous section<br>                if current_content:<br>                    sections.append({<br>                        &quot;headings&quot;: list(current_headings),<br>                        &quot;content&quot;: &quot;\n&quot;.join(current_content),<br>                    })<br>                    current_content = []<br>                level = len(heading_match.group(1))<br>                title = heading_match.group(2).strip()<br>                # Maintain heading hierarchy<br>                current_headings = current_headings[:level-1]<br>                current_headings.append(title)<br>            else:<br>                current_content.append(line)<br>        # Last section<br>        if current_content:<br>            sections.append({<br>                &quot;headings&quot;: list(current_headings),<br>                &quot;content&quot;: &quot;\n&quot;.join(current_content),<br>            })<br>        return sections<br>    def _split_sentences(self, text: str, max_tokens: int) -&gt; list[str]:<br>        &quot;&quot;&quot;Split at sentence boundaries, respecting max size.&quot;&quot;&quot;<br>        sentences = re.split(r&#39;(?&lt;=[.!?])\s+&#39;, text)<br>        chunks = []<br>        current = []<br>        current_len = 0<br>        for sentence in sentences:<br>            sent_len = self._token_count(sentence)<br>            if current_len + sent_len &gt; max_tokens and current:<br>                chunks.append(&quot; &quot;.join(current))<br>                # Overlap: keep last sentence for continuity<br>                current = current[-self.overlap_sentences:] if self.overlap_sentences else []<br>                current_len = sum(self._token_count(s) for s in current)<br>            current.append(sentence)<br>            current_len += sent_len<br>        if current:<br>            chunks.append(&quot; &quot;.join(current))<br>        return chunks<br>    def _token_count(self, text: str) -&gt; int:<br>        &quot;&quot;&quot;Approximate token count (words * 1.3).&quot;&quot;&quot;<br>        return int(len(text.split()) * 1.3)</pre><h3>The Five Golden Rules of Chunking</h3><ol><li><strong>A chunk must be understandable alone</strong> — prepend heading context</li><li><strong>Never split code blocks</strong> — a half-function is useless</li><li><strong>Never split tables</strong> — a row without headers is meaningless</li><li><strong>Target 256–512 tokens</strong> — sweet spot for BGE-M3</li><li><strong>Overlap 1–2 sentences</strong> — prevents losing info at boundaries</li></ol><h3>The Indexing Pipeline</h3><p>Durable indexing via Temporal (from Part 2):</p><pre>from pymilvus import Collection<br>from FlagEmbedding import BGEM3FlagModel<br>@activity.defn<br>async def chunk_documents(docs: list[dict]) -&gt; list[dict]:<br>    &quot;&quot;&quot;Structure-aware chunking.&quot;&quot;&quot;<br>    chunker = StructureAwareChunker(max_tokens=512)<br>    all_chunks = []<br>    for doc in docs:<br>        chunks = chunker.chunk(doc[&quot;text&quot;], source=doc[&quot;id&quot;])<br>        for chunk in chunks:<br>            all_chunks.append({<br>                &quot;id&quot;: f&quot;{doc[&#39;id&#39;]}:{chunk.position}&quot;,<br>                &quot;text&quot;: chunk.text,<br>                &quot;source&quot;: doc[&quot;id&quot;],<br>                &quot;doc_type&quot;: chunk.chunk_type,<br>                &quot;heading_path&quot;: &quot; &gt; &quot;.join(chunk.heading_path),<br>                &quot;tenant_id&quot;: doc.get(&quot;tenant_id&quot;, &quot;default&quot;),<br>            })<br>    activity.heartbeat(f&quot;chunked {len(docs)} docs → {len(all_chunks)} chunks&quot;)<br>    return all_chunks<br><br>@activity.defn<br>async def embed_chunks(chunks: list[dict]) -&gt; list[dict]:<br>    &quot;&quot;&quot;Generate dense + sparse embeddings with BGE-M3.&quot;&quot;&quot;<br>    model = BGEM3FlagModel(&quot;BAAI/bge-m3&quot;, use_fp16=True)<br>    texts = [c[&quot;text&quot;] for c in chunks]<br>    output = model.encode(texts, return_dense=True, return_sparse=True)<br>    for i, chunk in enumerate(chunks):<br>        chunk[&quot;dense_embedding&quot;] = output[&quot;dense_vecs&quot;][i].tolist()<br>        chunk[&quot;sparse_embedding&quot;] = output[&quot;lexical_weights&quot;][i]<br>    activity.heartbeat(f&quot;embedded {len(chunks)} chunks&quot;)<br>    return chunks<br><br>@activity.defn<br>async def upsert_to_milvus(chunks: list[dict]) -&gt; int:<br>    &quot;&quot;&quot;Upsert to Milvus. Idempotent - safe to retry.&quot;&quot;&quot;<br>    collection = Collection(&quot;knowledge_base&quot;)<br>    collection.upsert([<br>        {<br>            &quot;id&quot;: chunk[&quot;id&quot;],<br>            &quot;embedding&quot;: chunk[&quot;dense_embedding&quot;],<br>            &quot;sparse_embedding&quot;: chunk[&quot;sparse_embedding&quot;],<br>            &quot;chunk_text&quot;: chunk[&quot;text&quot;],<br>            &quot;source&quot;: chunk[&quot;source&quot;],<br>            &quot;doc_type&quot;: chunk[&quot;doc_type&quot;],<br>            &quot;heading_path&quot;: chunk[&quot;heading_path&quot;],<br>            &quot;tenant_id&quot;: chunk[&quot;tenant_id&quot;],<br>            &quot;created_at&quot;: int(time.time()),<br>        }<br>        for chunk in chunks<br>    ])<br>    return len(chunks)<br><br>@workflow.defn<br>class IndexingWorkflow:<br>    @workflow.run<br>    async def run(self, documents: list[dict]) -&gt; int:<br>        total = 0<br>        for batch in batched(documents, size=50):<br>            chunks = await workflow.execute_activity(<br>                chunk_documents, batch,<br>                start_to_close_timeout=timedelta(minutes=5),<br>                retry_policy=RetryPolicy(maximum_attempts=3),<br>            )<br>            embedded = await workflow.execute_activity(<br>                embed_chunks, chunks,<br>                start_to_close_timeout=timedelta(minutes=10),<br>                retry_policy=RetryPolicy(maximum_attempts=3, backoff_coefficient=2.0),<br>                heartbeat_timeout=timedelta(seconds=60),<br>            )<br>            count = await workflow.execute_activity(<br>                upsert_to_milvus, embedded,<br>                start_to_close_timeout=timedelta(minutes=2),<br>                retry_policy=RetryPolicy(maximum_attempts=5),<br>            )<br>            total += count<br>        return total</pre><h3>The Hybrid Search</h3><p>Using ANN is approximate but production-viable:</p><ul><li>Uses index structures (HNSW, IVF, DiskANN) to search a subset of vectors</li><li>O(log n) or sub-linear</li><li>At 1M vectors: might check ~1000–5000 candidates instead of 1M</li><li>Latency: &lt;5ms</li><li>Guarantees: returns ~95–99% of the true nearest neighbours (recall)</li></ul><pre>from pymilvus import AnnSearchRequest, RRFRanker, Collection<br>from FlagEmbedding import BGEM3FlagModel<br><br>class HybridRetriever:<br>    &quot;&quot;&quot;<br>    Dense + Sparse hybrid search with Milvus.<br>    Fused via Reciprocal Rank Fusion.<br>    &quot;&quot;&quot;<br>    def __init__(self):<br>        self.model = BGEM3FlagModel(&quot;BAAI/bge-m3&quot;, use_fp16=True)<br>        self.collection = Collection(&quot;knowledge_base&quot;)<br>        self.collection.load()<br>    def search(<br>        self,<br>        query: str,<br>        tenant_id: str,<br>        top_k: int = 10,<br>        k_rrf: int = 60,<br>    ) -&gt; list[dict]:<br>        # Encode query → dense + sparse (one pass)<br>        output = self.model.encode(<br>            [query], return_dense=True, return_sparse=True<br>        )<br>        dense_vec = output[&quot;dense_vecs&quot;][0].tolist()<br>        sparse_vec = output[&quot;lexical_weights&quot;][0]<br>        # Filter by tenant<br>        filter_expr = f&#39;tenant_id == &quot;{tenant_id}&quot;&#39;<br>        # Dense search request<br>        dense_req = AnnSearchRequest(<br>            data=[dense_vec],<br>            anns_field=&quot;embedding&quot;,<br>            param={&quot;metric_type&quot;: &quot;COSINE&quot;, &quot;params&quot;: {&quot;ef&quot;: 128}},<br>            limit=top_k,<br>            expr=filter_expr,<br>        )<br>        # Sparse search request<br>        sparse_req = AnnSearchRequest(<br>            data=[sparse_vec],<br>            anns_field=&quot;sparse_embedding&quot;,<br>            param={&quot;metric_type&quot;: &quot;IP&quot;},<br>            limit=top_k,<br>            expr=filter_expr,<br>        )<br>        # Hybrid search with RRF fusion<br>        results = self.collection.hybrid_search(<br>            reqs=[dense_req, sparse_req],<br>            ranker=RRFRanker(k=k_rrf),<br>            limit=top_k,<br>            output_fields=[&quot;chunk_text&quot;, &quot;source&quot;, &quot;heading_path&quot;, &quot;doc_type&quot;],<br>        )<br>        return [<br>            {<br>                &quot;id&quot;: hit.id,<br>                &quot;content&quot;: hit.entity.get(&quot;chunk_text&quot;),<br>                &quot;source&quot;: hit.entity.get(&quot;source&quot;),<br>                &quot;heading_path&quot;: hit.entity.get(&quot;heading_path&quot;),<br>                &quot;doc_type&quot;: hit.entity.get(&quot;doc_type&quot;),<br>                &quot;score&quot;: hit.score,<br>            }<br>            for hit in results[0]<br>        ]</pre><h3>Tuning RRF: The k Parameter</h3><p>Reciprocal Rank Fusion merges two ranked lists:</p><pre>RRF_score(doc) = 1/(k + rank_in_dense) + 1/(k + rank_in_sparse)</pre><p>The k parameter controls how much rank position matters:</p><pre>k=1:   rank1=0.500  rank2=0.333  rank5=0.167  (3x spread between #1 and #5)<br>k=60:  rank1=0.016  rank2=0.016  rank5=0.015  (1.07x spread — nearly flat)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gqOQnnW9nllOjrTUUlkIiA.png" /><figcaption>RRF K parameter</figcaption></figure><h3>Tuning Without Labeled Data</h3><p>No evaluation dataset? Use these heuristics:</p><pre>class AdaptiveRetriever(HybridRetriever):<br>    &quot;&quot;&quot;Adapt RRF k based on query characteristics.&quot;&quot;&quot;<br><br>  def search(self, query: str, tenant_id: str, top_k: int = 10):<br>          k_rrf = self._choose_k(query)<br>          return super().search(query, tenant_id, top_k, k_rrf)<br>      def _choose_k(self, query: str) -&gt; int:<br>          words = query.split()<br>          # Short queries with specific terms → favor sparse precision<br>          has_exact = any(<br>              w.isupper() or &quot;_&quot; in w or &quot;.&quot; in w  # API_KEY, config.retry<br>              for w in words<br>          )<br>          if len(words) &lt;= 3 or has_exact:<br>              return 40  # More rank-sensitive<br>          # Long natural language → favor dense understanding<br>          if len(words) &gt;= 10:<br>              return 80  # More presence-sensitive<br>          return 60  # Balanced default</pre><h3>Tuning With Labeled Data</h3><p>If you have query-relevance pairs (even 50–100), grid search:</p><pre>async def evaluate_config(k_rrf: int, eval_set: list[dict]) -&gt; dict:<br>    retriever = HybridRetriever()<br>    hit_at_5 = 0<br>    mrr = 0<br>    for sample in eval_set:<br>            results = retriever.search(sample[&quot;query&quot;], &quot;default&quot;, top_k=10, k_rrf=k_rrf)<br>            retrieved_ids = [r[&quot;id&quot;] for r in results]<br>            relevant = set(sample[&quot;relevant_ids&quot;])<br>            if relevant &amp; set(retrieved_ids[:5]):<br>                hit_at_5 += 1<br>            for rank, doc_id in enumerate(retrieved_ids, 1):<br>                if doc_id in relevant:<br>                    mrr += 1.0 / rank<br>                    break<br>        n = len(eval_set)<br>        return {&quot;k&quot;: k_rrf, &quot;hit@5&quot;: hit_at_5/n, &quot;mrr&quot;: mrr/n}<br>    # Grid search<br>    for k in [10, 20, 40, 60, 80, 100]:<br>        metrics = await evaluate_config(k, eval_set)<br>        print(f&quot;k={k:3d}  Hit@5={metrics[&#39;hit@5&#39;]:.3f}  MRR={metrics[&#39;mrr&#39;]:.3f}&quot;)</pre><h3>Adding Re-Ranking for Maximum Quality</h3><p>Bi-encoders (BGE-M3) encode query and document independently — fast but approximate. Cross-encoders see both together — slow but much more accurate.</p><p>The re-ranker (cross-encoder) compensates for ANN’s approximation error. The pipeline is:</p><ol><li>ANN → fast, returns Top 50 candidates (might miss 1–2 true top results)</li><li>Re-rank → precise, scores all 50 candidates with cross-encoder → returns Top 10</li></ol><p><strong>Pattern: retrieve broadly, re-rank precisely.</strong></p><pre>from sentence_transformers import CrossEncoder<br><br>class RerankedRetriever:<br>    &quot;&quot;&quot;<br>    Stage 1: Milvus hybrid search → top 50 (fast, ~10ms)<br>    Stage 2: Cross-encoder re-rank → top 10 (precise, ~200ms)<br>    &quot;&quot;&quot;<br>    def __init__(self):<br>        self.hybrid = HybridRetriever()<br>        self.reranker = CrossEncoder(&quot;BAAI/bge-reranker-v2-m3&quot;)<br>    def search(self, query: str, tenant_id: str, top_k: int = 10) -&gt; list[dict]:<br>        # Stage 1: Broad retrieval<br>        candidates = self.hybrid.search(query, tenant_id, top_k=50)<br>        if not candidates:<br>            return []<br>        # Stage 2: Re-rank<br>        pairs = [(query, doc[&quot;content&quot;]) for doc in candidates]<br>        scores = self.reranker.predict(pairs)<br>        for doc, score in zip(candidates, scores):<br>            doc[&quot;rerank_score&quot;] = float(score)<br>        reranked = sorted(candidates, key=lambda d: d[&quot;rerank_score&quot;], reverse=True)<br>        return reranked[:top_k]</pre><p><strong>Impact:</strong> In my experience, re-ranking improves Hit@5 by 8–15% on domain-specific corpora. The trade-off is ~200ms added latency — worth it for accuracy-sensitive use cases.</p><h3>Wiring Into LangGraph (from Part 2)</h3><pre>async def retrieval_node(state: dict) -&gt; dict:<br>    &quot;&quot;&quot;LangGraph node: retrieve with caching.&quot;&quot;&quot;<br>    hot = get_hot_state()  # Redis (from Part 3)<br>    retriever = RerankedRetriever()<br><br>    query = state[&quot;query&quot;]<br>        tenant_id = state.get(&quot;tenant_id&quot;, &quot;default&quot;)<br>        # Check Redis cache first<br>        cached = await hot.get_cached_retrieval(query, tenant_id)<br>        if cached:<br>            return {**state, &quot;retrieved_docs&quot;: cached, &quot;retrieval_cached&quot;: True}<br>        # Full hybrid search + re-rank<br>        docs = retriever.search(query, tenant_id, top_k=10)<br>        # Cache for 5 minutes<br>        await hot.cache_retrieval(query, tenant_id, docs, ttl=300)<br>        return {**state, &quot;retrieved_docs&quot;: docs, &quot;retrieval_cached&quot;: False}</pre><h3>The Full Retrieval Data Flow</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tmh-j5X8V2pRpXbrzzIHwQ.jpeg" /><figcaption>Sequence diagram for lang graph interaction with Redis for quick lookup and Milvus for vector store</figcaption></figure><h3>Summary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_aBn1XUZ3EX8CXi3Rfrb4Q.png" /><figcaption>Recollection and Reasoning of each step</figcaption></figure><p><em>Next: </em><a href="https://medium.com/p/e8c7c4961c2f"><em>Part 5 — Putting It All Together: The Complete Pipeline →</em></a> <br><em>Previous: </em><a href="https://medium.com/p/d79fd779acc5"><em>← Part 3 — Agent Memory: The Redis + Postgres Pattern</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fecb02272678" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 5: Putting It All Together — The Complete Pipeline]]></title>
            <link>https://medium.com/@vipul319/part-5-putting-it-all-together-the-complete-pipeline-e8c7c4961c2f?source=rss-b1a4cdd865b7------2</link>
            <guid isPermaLink="false">https://medium.com/p/e8c7c4961c2f</guid>
            <category><![CDATA[langgraph]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[temporal]]></category>
            <category><![CDATA[system-design-concepts]]></category>
            <dc:creator><![CDATA[Vipul]]></dc:creator>
            <pubDate>Wed, 08 Jul 2026 17:07:15 GMT</pubDate>
            <atom:updated>2026-07-08T17:07:15.956Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Part 5 of 5 in the series “Building Production-Grade AI Agents: From Prototype to Scale”</em></p><p>We’ve built the pieces:</p><ul><li><a href="https://medium.com/p/c4e07aba14d6">Part 2</a>: LangGraph + Temporal for durable agent execution</li><li><a href="https://medium.com/p/d79fd779acc5">Part 3</a>: Redis + Postgres for two-tier memory</li><li><a href="https://medium.com/p/fecb02272678">Part 4</a>: Milvus + BGE-M3 for hybrid retrieval</li></ul><p>Now let’s wire them into a <strong>single, end-to-end production pipeline</strong> — from user request to final response, with every failure mode handled.</p><h3>The Complete Request Lifecycle</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yZZWYmyzGcyr1n49726Lag.png" /><figcaption>Complete Request Lifecylce</figcaption></figure><h3>Project Structure</h3><pre>agent-platform/<br>├── src/<br>│   ├── agents/                    # LangGraph definitions<br>│   │   ├── research_agent.py     # Research graph<br>│   │   ├── synthesis_agent.py    # Synthesis graph<br>│   │   ├── critic_agent.py       # Critic graph<br>│   │   └── nodes/                # Shared node implementations<br>│   │       ├── retrieve.py<br>│   │       ├── analyze.py<br>│   │       └── quality.py<br>│   │<br>│   ├── workflows/                 # Temporal workflows<br>│   │   ├── research_pipeline.py  # Single-agent research<br>│   │   ├── multi_agent.py        # Fan-out/fan-in<br>│   │   └── indexing.py           # Document ingestion<br>│   │<br>│   ├── activities/                # Temporal activities<br>│   │   ├── agent_runner.py       # Wraps LangGraph as activity<br>│   │   ├── embedding.py          # BGE-M3 encoding<br>│   │   └── indexing.py           # Milvus upsert<br>│   │<br>│   ├── state/                     # State management<br>│   │   ├── hot.py                # Redis operations<br>│   │   ├── cold.py               # Postgres operations<br>│   │   └── models.py             # SQLAlchemy models<br>│   │<br>│   ├── retrieval/                 # Search infrastructure<br>│   │   ├── chunker.py            # Structure-aware chunking<br>│   │   ├── encoder.py            # BGE-M3 wrapper<br>│   │   ├── searcher.py           # Hybrid search<br>│   │   └── reranker.py           # Cross-encoder re-rank<br>│   │<br>│   ├── api/                       # FastAPI endpoints<br>│   │   ├── routes.py<br>│   │   └── schemas.py<br>│   │<br>│   └── workers/                   # Temporal workers<br>│       ├── agent_worker.py       # Heavy (LLM calls)<br>│       └── indexing_worker.py    # GPU (embeddings)<br>│<br>├── infra/<br>│   ├── docker-compose.yml        # Local dev: Redis, PG, Milvus, Temporal<br>│   └── k8s/                      # Production manifests<br>│<br>├── tests/<br>│   ├── test_chunker.py<br>│   ├── test_retrieval.py<br>│   └── test_workflows.py<br>│<br>└── eval/<br>    ├── eval_set.json             # Query-relevance pairs<br>    └── run_eval.py               # Retrieval quality benchmark</pre><h3>The Glue: Dependency Injection</h3><pre># src/config.py<br>from dataclasses import dataclass<br><br>@dataclass<br>class Config:<br>    # Redis<br>    redis_url: str = &quot;redis://localhost:6379&quot;<br>    <br>    # Postgres<br>    postgres_url: str = &quot;postgresql+asyncpg://user:pass@localhost:5432/agents&quot;<br>    <br>    # Milvus<br>    milvus_host: str = &quot;localhost&quot;<br>    milvus_port: int = 19530<br>    milvus_collection: str = &quot;knowledge_base&quot;<br>    <br>    # Temporal<br>    temporal_host: str = &quot;localhost:7233&quot;<br>    temporal_task_queue: str = &quot;agent-tasks&quot;<br>    <br>    # Models<br>    embedding_model: str = &quot;BAAI/bge-m3&quot;<br>    reranker_model: str = &quot;BAAI/bge-reranker-v2-m3&quot;<br>    llm_model: str = &quot;gpt-4o&quot;<br>    <br>    # Retrieval<br>    retrieval_top_k: int = 10<br>    rerank_candidates: int = 50<br>    rrf_k: int = 60<br>    cache_ttl_seconds: int = 300<br>    checkpoint_ttl_seconds: int = 3600<br><br># src/deps.py<br>from functools import lru_cache<br>import redis.asyncio as redis<br>from sqlalchemy.ext.asyncio import create_async_engine, async_sessionmaker<br>from pymilvus import connections, Collection<br>@lru_cache<br>def get_config() -&gt; Config:<br>    return Config()  # In production, load from env/secrets<br>@lru_cache<br>def get_redis_client() -&gt; redis.Redis:<br>    return redis.from_url(https://medium.com/get_config().redis_url)<br>@lru_cache<br>def get_db_session_factory():<br>    engine = create_async_engine(get_config().postgres_url)<br>    return async_sessionmaker(engine, expire_on_commit=False)<br>@lru_cache<br>def get_milvus_collection() -&gt; Collection:<br>    cfg = get_config()<br>    connections.connect(host=cfg.milvus_host, port=cfg.milvus_port)<br>    collection = Collection(cfg.milvus_collection)<br>    collection.load()<br>    return collection<br>def get_hot_state():<br>    from src.state.hot import AgentHotState<br>    return AgentHotState(get_redis_client())<br>def get_cold_state():<br>    from src.state.cold import AgentColdState<br>    return AgentColdState(get_db_session_factory())<br>def get_retriever():<br>    from src.retrieval.searcher import RerankedRetriever<br>    return RerankedRetriever(<br>        collection=get_milvus_collection(),<br>        config=get_config(),<br>    )</pre><h3>The API Layer</h3><pre># src/api/routes.py<br>from fastapi import FastAPI, HTTPException<br>from temporalio.client import Client<br>from uuid import uuid4<br><br>app = FastAPI()<br><br>@app.post(&quot;/agent/research&quot;)<br>async def start_research(request: ResearchRequest):<br>    &quot;&quot;&quot;Start a durable research workflow.&quot;&quot;&quot;<br>    client = await Client.connect(get_config().temporal_host)<br>    workflow_id = f&quot;research-{uuid4().hex[:12]}&quot;<br>    handle = await client.start_workflow(<br>        ResearchPipeline.run,<br>        request.topic,<br>        id=workflow_id,<br>        task_queue=get_config().temporal_task_queue,<br>        execution_timeout=timedelta(hours=24),  # Max lifetime<br>    )<br>    return {&quot;workflow_id&quot;: workflow_id, &quot;status&quot;: &quot;started&quot;}<br><br>@app.get(&quot;/agent/{workflow_id}/status&quot;)<br>async def get_status(workflow_id: str):<br>    &quot;&quot;&quot;Query workflow status (non-blocking).&quot;&quot;&quot;<br>    client = await Client.connect(get_config().temporal_host)<br>    handle = client.get_workflow_handle(workflow_id)<br>    try:<br>        status = await handle.query(ResearchPipeline.get_status)<br>        return {&quot;workflow_id&quot;: workflow_id, &quot;status&quot;: status}<br>    except Exception:<br>        raise HTTPException(404, &quot;Workflow not found&quot;)<br><br>@app.post(&quot;/agent/{workflow_id}/approve&quot;)<br>async def approve(workflow_id: str):<br>    &quot;&quot;&quot;Send human approval signal.&quot;&quot;&quot;<br>    client = await Client.connect(get_config().temporal_host)<br>    handle = client.get_workflow_handle(workflow_id)<br>    await handle.signal(ResearchPipeline.approve)<br>    return {&quot;workflow_id&quot;: workflow_id, &quot;status&quot;: &quot;approved&quot;}<br><br>@app.get(&quot;/agent/{workflow_id}/result&quot;)<br>async def get_result(workflow_id: str):<br>    &quot;&quot;&quot;Get workflow result (blocks until complete or timeout).&quot;&quot;&quot;<br>    client = await Client.connect(get_config().temporal_host)<br>    handle = client.get_workflow_handle(workflow_id)<br>    try:<br>        result = await handle.result(timeout=timedelta(seconds=30))<br>        return {&quot;workflow_id&quot;: workflow_id, &quot;result&quot;: result}<br>    except TimeoutError:<br>        return {&quot;workflow_id&quot;: workflow_id, &quot;status&quot;: &quot;still_running&quot;}<br><br>@app.post(&quot;/index&quot;)<br>async def index_documents(request: IndexRequest):<br>    &quot;&quot;&quot;Start a durable indexing workflow.&quot;&quot;&quot;<br>    client = await Client.connect(get_config().temporal_host)<br>    workflow_id = f&quot;index-{uuid4().hex[:12]}&quot;<br>    await client.start_workflow(<br>        IndexingWorkflow.run,<br>        request.documents,<br>        id=workflow_id,<br>        task_queue=&quot;indexing-tasks&quot;,<br>    )<br>    return {&quot;workflow_id&quot;: workflow_id, &quot;document_count&quot;: len(request.documents)}</pre><h3>Docker Compose for Local Development</h3><pre># infra/docker-compose.yml<br>version: &quot;3.8&quot;<br>services:<br>  # --- Data Layer ---<br>  redis:<br>    image: redis:7-alpine<br>    ports:<br>      - &quot;6379:6379&quot;<br>    volumes:<br>      - redis_data:/data<br>  postgres:<br>    image: postgres:16-alpine<br>    ports:<br>      - &quot;5432:5432&quot;<br>    environment:<br>      POSTGRES_DB: agents<br>      POSTGRES_USER: agent_user<br>      POSTGRES_PASSWORD: agent_pass<br>    volumes:<br>      - pg_data:/var/lib/postgresql/data<br>  # --- Vector Search ---<br>  milvus-etcd:<br>    image: quay.io/coreos/etcd:v3.5.5<br>    environment:<br>      ETCD_AUTO_COMPACTION_MODE: revision<br>      ETCD_AUTO_COMPACTION_RETENTION: &quot;1000&quot;<br>    volumes:<br>      - milvus_etcd:/etcd<br>  milvus-minio:<br>    image: minio/minio:RELEASE.2023-03-20T20-16-18Z<br>    environment:<br>      MINIO_ACCESS_KEY: minioadmin<br>      MINIO_SECRET_KEY: minioadmin<br>    command: minio server /minio_data<br>    volumes:<br>      - milvus_minio:/minio_data<br>  milvus:<br>    image: milvusdb/milvus:v2.4-latest<br>    ports:<br>      - &quot;19530:19530&quot;<br>      - &quot;9091:9091&quot;<br>    depends_on:<br>      - milvus-etcd<br>      - milvus-minio<br>    environment:<br>      ETCD_ENDPOINTS: milvus-etcd:2379<br>      MINIO_ADDRESS: milvus-minio:9000<br>    volumes:<br>      - milvus_data:/var/lib/milvus<br>  # --- Orchestration ---<br>  temporal:<br>    image: temporalio/auto-setup:latest<br>    ports:<br>      - &quot;7233:7233&quot;<br>    environment:<br>      DB: postgresql<br>      DB_PORT: 5432<br>      POSTGRES_USER: temporal_user<br>      POSTGRES_PWD: temporal_pass<br>      POSTGRES_SEEDS: temporal-db<br>  temporal-db:<br>    image: postgres:16-alpine<br>    environment:<br>      POSTGRES_USER: temporal_user<br>      POSTGRES_PASSWORD: temporal_pass<br>  temporal-ui:<br>    image: temporalio/ui:latest<br>    ports:<br>      - &quot;8080:8080&quot;<br>    environment:<br>      TEMPORAL_ADDRESS: temporal:7233<br>volumes:<br>  redis_data:<br>  pg_data:<br>  milvus_etcd:<br>  milvus_minio:<br>  milvus_data:</pre><h3>Running the Workers</h3><pre># src/workers/agent_worker.py<br>&quot;&quot;&quot;<br>Worker for LLM-heavy activities.<br>Run on machines with network access to LLM providers.<br>&quot;&quot;&quot;<br>import asyncio<br>from temporalio.client import Client<br>from temporalio.worker import Worker<br><br>from src.activities.agent_runner import (<br>    run_research_agent,<br>    run_synthesis_agent,<br>    run_critic_agent,<br>)<br>from src.workflows.research_pipeline import ResearchPipeline<br>from src.workflows.multi_agent import MultiAgentResearch<br><br>async def main():<br>    client = await Client.connect(&quot;localhost:7233&quot;)<br>    worker = Worker(<br>        client,<br>        task_queue=&quot;agent-tasks&quot;,<br>        workflows=[ResearchPipeline, MultiAgentResearch],<br>        activities=[<br>            run_research_agent,<br>            run_synthesis_agent,<br>            run_critic_agent,<br>        ],<br>    )<br>    print(&quot;Agent worker started. Listening on &#39;agent-tasks&#39;...&quot;)<br>    await worker.run()<br><br>if __name__ == &quot;__main__&quot;:<br>    asyncio.run(main())</pre><pre># src/workers/indexing_worker.py<br>&quot;&quot;&quot;<br>Worker for GPU-heavy activities (embedding).<br>Run on machines with GPU access.<br>&quot;&quot;&quot;<br>import asyncio<br>from temporalio.client import Client<br>from temporalio.worker import Worker<br><br>from src.activities.embedding import embed_chunks<br>from src.activities.indexing import chunk_documents, upsert_to_milvus<br>from src.workflows.indexing import IndexingWorkflow<br><br>async def main():<br>    client = await Client.connect(&quot;localhost:7233&quot;)<br>    worker = Worker(<br>        client,<br>        task_queue=&quot;indexing-tasks&quot;,<br>        workflows=[IndexingWorkflow],<br>        activities=[chunk_documents, embed_chunks, upsert_to_milvus],<br>    )<br>    print(&quot;Indexing worker started. Listening on &#39;indexing-tasks&#39;...&quot;)<br>    await worker.run()<br><br>if __name__ == &quot;__main__&quot;:<br>    asyncio.run(main())</pre><h3>Production Checklist</h3><h4>Reliability</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5xfdzbtDPuOYzD8-_iI1DA.jpeg" /></figure><h4>Observability</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6MQ2B88pL3fBk2QwMVuLQw.jpeg" /></figure><h4>Security</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5euj_5AJU5DsSPa2-RNcCQ.jpeg" /></figure><h4>Performance</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zz7o1vFBVVnWTByo6LcIYg.png" /></figure><h3>Common Pitfalls</h3><h3>1. Storing Too Much in Temporal</h3><p>Temporal event history has size limits. Don’t pass full documents through workflow state:</p><pre># ❌ Bad: passes 50 documents through Temporal<br>findings = await workflow.execute_activity(run_research, topic)<br>report = await workflow.execute_activity(run_synthesis, findings[&quot;full_documents&quot;])</pre><pre># ✅ Good: store in Postgres, pass reference<br>findings = await workflow.execute_activity(run_research, topic)<br># Research activity stores docs in Postgres, returns IDs<br>report = await workflow.execute_activity(run_synthesis, findings[&quot;doc_ids&quot;])</pre><h3>2. Not Setting Heartbeat Timeouts</h3><p>Without heartbeats, Temporal waits for start_to_close_timeout before detecting a dead worker. That could be 5 minutes of wasted time:</p><pre># ❌ Bad: 5 minute timeout, no heartbeat<br>await workflow.execute_activity(<br>    run_research, topic,<br>    start_to_close_timeout=timedelta(minutes=5),<br>)</pre><pre># ✅ Good: heartbeat every 30s, detect failures fast<br>await workflow.execute_activity(<br>    run_research, topic,<br>    start_to_close_timeout=timedelta(minutes=5),<br>    heartbeat_timeout=timedelta(seconds=30),<br>)</pre><h3>3. Caching Without Tenant Isolation</h3><pre># ❌ Bad: shared cache across tenants<br>cache_key = f&quot;cache:{hash(query)}&quot;</pre><pre># ✅ Good: tenant-scoped<br>cache_key = f&quot;cache:{tenant_id}:{hash(query)}&quot;</pre><h3>4. No Checkpoint Expiry</h3><pre># ❌ Bad: checkpoints live forever<br>await redis.set(f&quot;ckpt:{wf_id}&quot;, state)</pre><pre># ✅ Good: auto-expire after 1 hour<br>await redis.setex(f&quot;ckpt:{wf_id}&quot;, 3600, state)</pre><h3>5. Splitting Code Blocks During Chunking</h3><pre># ❌ Bad: code block split at token boundary<br>chunk_1 = &quot;```python\ndef retry(max_attempts):\n    for i in range(&quot;<br>chunk_2 = &quot;max_attempts):\n        try:\n            return func()\n```&quot;</pre><pre># ✅ Good: keep code blocks whole<br>if chunk_type == &quot;code&quot;:<br>    # Never split — store as one chunk even if over max_tokens<br>    pass</pre><h3>Scaling Considerations</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KjooMF7z0Jn7Pt3_Irz1qQ.jpeg" /></figure><h3>The Final Architecture (One Last Time)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iGTI1ntC8sNJZlhZn9YwZg.png" /></figure><h3>Series Recap</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ln5kGpxH3anVdFpKKzUFZA.jpeg" /></figure><p>The stack:</p><pre>Think    → LangGraph<br>Finish   → Temporal<br>Cache    → Redis<br>Remember → Postgres<br>Retrieve → Milvus</pre><p>Start simple: one LangGraph agent in one Temporal workflow, searching one Milvus collection, with Redis caching and Postgres audit. Scale each piece independently as your load grows.</p><p>The infrastructure is boring. The agents are interesting. Build the boring parts right so you can focus on the interesting parts.</p><p><em>This concludes the series. If you found it useful, follow me for more on production architecture.</em></p><p><a href="https://medium.com/p/fecb02272678"><em>← Part 4 — Hybrid Retrieval with Milvus + BGE-M3</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e8c7c4961c2f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>