<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Viren Baraiya on Medium]]></title>
        <description><![CDATA[Stories by Viren Baraiya on Medium]]></description>
        <link>https://medium.com/@v1r3n?source=rss-dc3485d163fc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*rcjJeM7BO_aIgvcx.</url>
            <title>Stories by Viren Baraiya on Medium</title>
            <link>https://medium.com/@v1r3n?source=rss-dc3485d163fc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:37:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@v1r3n/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Agents on Conductor — Architecture for Production AI]]></title>
            <link>https://medium.com/orkes/agents-on-conductor-architecture-for-production-ai-da5b55cc99b7?source=rss-dc3485d163fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/da5b55cc99b7</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[workflow]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[durable-execution]]></category>
            <dc:creator><![CDATA[Viren Baraiya]]></dc:creator>
            <pubDate>Tue, 25 Aug 2026 18:27:46 GMT</pubDate>
            <atom:updated>2026-08-25T18:27:46.065Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Separating Brain from Execution.</strong></h4><p><strong><em>Here is a question for every team building AI agents: when your model emits a tool call, what actually stands between that call and execution?</em></strong></p><p>In most agent stacks, the honest answer is: the model’s good behavior. The tool call goes straight to a dispatcher whose job is “do what the model said.” Whether a guardrail runs, whether an approval is required, whether the tool is even in scope — all of it depends on the model having behaved.</p><p>A guardrail that runs because the model chose to invoke it is not a guardrail — <em>It is a suggestion</em>.</p><p>Distributed systems solved this shape of problem a long time ago, and the pattern is one every infrastructure engineer already lives with: separate the control plane from the data plane. Agents need the same split. The model belongs in a reasoning plane — it looks at state, weighs the goal, and decides what should happen. The execution plane — deterministic, policy-enforcing, durable — decides what actually runs. Nothing crosses from one plane to the other without being validated.</p><p>We think that missing split is the single biggest thing keeping agents out of production. So we built it in, and today we are shipping it.</p><p>Conductor has been orchestrating production workflows since its Netflix days, and for years the community’s favorite question was how far it scales (<a href="https://medium.com/orkes/running-a-billion-workflows-a-month-with-netflix-conductor-5fd358d9b6c1">a billion workflows a month</a>, it turns out). The question has changed. Today it is: how do we put agents in production?</p><blockquote>This is our answer.</blockquote><p><a href="https://github.com/conductor-oss/conductor">Conductor</a> (<a href="https://github.com/conductor-oss/conductor">https://github.com/conductor-oss/conductor</a>) now runs AI agents as first class citizens. You can write agents using any of the supported <a href="https://docs.conductor-oss.org/documentation/clientsdks/index.html">SDKs</a> in Conductor OR bring the agent with the framework you already use — LangChain, LangGraph, OpenAI, Vercel, Google ADK — Conductor <strong>compiles</strong> it into a <strong>durable workflow </strong>and executes it under one rule that never breaks:</p><blockquote><strong>the LLM reasons; it does not execute.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8YJ4LoFyUlg5dBtMF-X--Q.png" /><figcaption>Agents on Conductor</figcaption></figure><h3>The LLM reasons, Conductor executes</h3><p>The LLM is used purely as a reasoning engine: it looks at the current state, the goal, and the tools it has, and proposes what should happen next. That proposal is data. The engine decides whether it becomes an action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PDGfDPe2SVPNnisbAu0pKw.png" /><figcaption>Agent run on Conductor</figcaption></figure><p>Concretely: if an agent declares tools A, B and C, the LLM <em>cannot</em> decide to execute tool X — and when it emits a call to X anyway, the engine rejects it before anything is scheduled and returns a recorded error to the conversation:</p><pre>Unknown tool &#39;str_replace&#39;. <br>Available tools: check_inventory, create_ticket, notify_oncall</pre><p>That example comes straight from a comment in our compiler — models really do emit tool names nobody ever offered them. Because the check is compiled into the workflow graph rather than written into the prompt, it holds no matter what the model outputs.</p><p><strong>Remember, none of this makes the LLM deterministic</strong> — it won’t be, and it doesn’t need to be. Determinism is the execution plane’s job, and every rule of engagement is a property of the compiled graph, enforced no matter what the model returns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vptERPt21ZH5VZiYObs-tA.png" /><figcaption>Conductor Agents — Rules of engagement</figcaption></figure><h3>How does an agent become a workflow?</h3><p>The agent definition compiles into a workflow definition with a ReAct loop. Session state is maintained using workflow variables and reasoning is done by an LLM using LLM_CHAT_COMPLETE task. Every task in Conductor has built-in retry policies and durability built-in — they come for free. Tool execution happens via FORK_JOIN_DYNAMIC fan-out or SUB_WORKFLOW task that can run entire graphs.</p><p>Guardrails are compiled so if the tool has a guardrail it gets scheduled and executed before the actual tool is called.</p><p>A tool is just a task that the platform already has: your worker function, an HTTP endpoint, an MCP tool, a person, another agent, or an entire workflow.</p><p>Every execution also carries a principal, and credentials resolve against it at poll time — your agent code never embeds a key.</p><p><strong>LLM-written plans compile or they don’t run.</strong> When the agent produces a multi-step plan, the plan JSON is compiled into a validated workflow definition before any of it runs: unknown tools, cycles, dangling references and out-of-grammar conditions all fail compilation — success conditions go through a real AST whitelist parser, because a regex denylist in front of a JS engine is not defense in depth — and each step’s arguments are validated against the tool’s input schema. An invalid plan fails as a unit and routes to a fallback path; it never executes halfway.</p><h3>Bring the agent you already wrote</h3><p>Here is the full lifecycle with LangChain:</p><pre>from conductor.ai.agents import AgentRuntime<br>from langchain.agents import create_agent<br>from langchain_core.tools import tool<br><br>@tool<br>def check_token() -&gt; str:<br>    &quot;&quot;&quot;Check a token.&quot;&quot;&quot;<br>    return &quot;available&quot;<br><br>agent = create_agent(&quot;openai:gpt-4o-mini&quot;, tools=[check_token],<br>                     system_prompt=&quot;You are a helpful assistant.&quot;)<br><br>with AgentRuntime() as runtime:<br>    runtime.run(agent, &quot;Is the token set?&quot;)  # develop: compile and execute once<br>    runtime.deploy(agent)                    # release: register without executing<br>    runtime.serve(agent)                     # operate: run tool workers and block</pre><p>run() executes the agent durably — the whole execution is visible in the UI from the first try.</p><p>deploy() registers the agent on the server under a name and a version, after which anyone can invoke it without importing your framework or its dependencies.</p><p>serve() runs the worker process for your tool functions.</p><h3>Long Horizon Agents</h3><p>Real agent work is long-horizon: it waits for approvals, rate limits, people, arrival of end of day closing data etc. Most runtimes hold the loop in memory, so the agent lives exactly as long as its process. On Conductor the loop is a workflow — every transition is persisted before the next task is scheduled, so a restart, a deploy or a crash resumes the run from where the work actually was.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BRQm-W_d_CWel72OjuQY8A.png" /></figure><p>That durability changes the horizon:</p><ul><li><strong>Waiting is free.</strong> A HUMAN approval or a message wait is durable — no thread parked, no worker polling, nothing burned while nothing happens.</li><li><strong>You can talk to a running agent.</strong> A signal lands in the next model turn as context, so new facts reach a mid-flight agent without a restart.</li><li><strong>You can stop one gracefully.</strong> The agent finishes its current turn and completes with its last output instead of dying mid-tool-call.</li><li><strong>Agents run on events and cadences.</strong> A deployed agent is a workflow, so Conductor’s <a href="https://docs.conductor-oss.org/devguide/cookbook/event-driven.html">event-driven recipes</a> and <a href="https://docs.conductor-oss.org/devguide/ai/scheduling-agents.html">scheduler</a> apply unchanged: a Kafka, SQS, NATS or AMQP event can start one, a <a href="https://docs.conductor-oss.org/devguide/cookbook/sending-signals.html">signal</a> can resume one, and a nightly agent is one cron expression.</li><li><strong>Retries and replay are built-in</strong>. These are not specialized scripts or manual checkpoints you have to write.</li></ul><h3>The single-responsibility agent</h3><p>The teams that succeed in production stop building <em>an</em> agent and start building many small ones: one agent, single responsibility, a well defined set of tools and guardrails, a clear contract at the boundary — the same discipline that took us from monoliths to microservices, for the same reason.</p><p>A small agent’s failure modes are enumerable: you can list everything it touches, write the eval set, hand it the narrowest credentials. An agent with forty tools and a system prompt describing your entire business is not a powerful agent. <em>It is a distributed system with no interfaces</em>.</p><blockquote>How do you build something big out of small agents? The answer: composition.</blockquote><h3>Agents and workflows, living together in harmony</h3><p>Complex systems — agent harnesses — are essentially workflows: deterministic tools and processes interwoven with agent execution. The wiring goes both directions: a workflow calls an agent as an ordinary step, and an agent calls an entire workflow as a tool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I4JWJdJ2OEnLCF1KRjeASQ.png" /></figure><p>An agent monitors your Kubernetes cluster; an alert event starts it — nothing sat polling. The agent reads the logs, applies your policies, and decides the cluster needs a restart. That is reasoning, exactly what you want an LLM for. But restarting a cluster is not reasoning: check the cluster type, notify the owners, get an approval if it is production, drain, restart, verify health, roll back on failure. Those are deterministic paths that must not deviate from run to run — a workflow.</p><p><em>So the agent decides that the cluster restarts; the workflow determines how, </em>the same way every time. Reasoning where judgment is needed, deterministic execution everywhere else — that is the shape we believe production agentic systems converge on.</p><h3>Durability, explainability, determinism</h3><p>Strip away the mechanics and this release stands on three pillars — the properties the execution plane exists to provide, each answering a different question in an agent’s life.</p><p><strong>Determinism</strong> answers the question before the run: what <em>can</em> this agent do? Tools, guardrails and bounds are compiled and enforced, so the answer is a finite list your security team can read, not a prediction about model behavior.</p><p><strong>Durability</strong> answers the question during the run: will the work survive? The loop is a workflow, not a process — restarts, deploys and three-day approvals do not lose it.</p><p><strong>Explainability</strong> answers the question after the run: what did it actually do? Because the engine can only act by recording, the execution history is not telemetry <em>about</em> the run — it <em>is</em> the run, stored for months and walkable in the UI, turn by turn.</p><p>Miss any one of the three and you have a demo. Deliver all three and you have an agent you can put in production.</p><h3>Try it</h3><p><a href="https://docs.conductor-oss.org/quickstart/first-agent.html">Your first agent</a> — a working agent on a local server in minutes</p><p><a href="https://docs.conductor-oss.org/devguide/ai/agent-guardrails.html">Agent guardrails</a> — enforcement points and failure outcomes</p><p><a href="https://docs.conductor-oss.org/devguide/ai/multi-agent-architecture.html">Multi-agent architecture</a> — supervisor, swarm and pipeline shapes</p><p>Can you run agents in production the way you run services in production — bounded, governed, observable, durable? With this release, the answer is a resounding yes.</p><blockquote>Build the brain with whatever framework you like. Let Conductor be the hands.</blockquote><p>Don’t forget to give us a ⭐️ <a href="https://github.com/conductor-oss/conductor">https://github.com/conductor-oss/conductor</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da5b55cc99b7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/orkes/agents-on-conductor-architecture-for-production-ai-da5b55cc99b7">Agents on Conductor — Architecture for Production AI</a> was originally published in <a href="https://medium.com/orkes">Orkes Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Rebuilding Workflow Engines Inside Your Agent Harness]]></title>
            <link>https://medium.com/orkes/agent-harness-workflow-engine-2b5651edd0f7?source=rss-dc3485d163fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b5651edd0f7</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[netflix-conductor]]></category>
            <category><![CDATA[workflow]]></category>
            <dc:creator><![CDATA[Viren Baraiya]]></dc:creator>
            <pubDate>Thu, 11 Jun 2026 16:22:15 GMT</pubDate>
            <atom:updated>2026-06-11T16:22:15.833Z</atom:updated>
            <content:encoded><![CDATA[<p><em>An </em><strong><em>agent harness</em></strong><em> is the software that surrounds a model and lets it act as an agent: it supplies context, exposes tools, manages state and permissions, runs the reasoning loop, evaluates results, and feeds observations back. Both reports land on the same point — model capability alone does not decide whether an agent succeeds; the environment around it matters at least as much.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8H9cr4ItCtkTpUjWJ9la6Q.png" /></figure><p>But <em>harness</em> has become a single word for two very different kinds of problem. Some have taken a genuinely new form now that the decision-maker is a probabilistic model: choosing what belongs in its context, helping it plan toward an ambiguous goal, designing tools it can use correctly, judging output with no clean pass/fail test.</p><p>The other kind is old: persisting state across failures, retrying safely, waiting for approval, enforcing what an action may do, coordinating recovery, and reconstructing what happened afterward. That second set is not a model problem. It is a workflow problem, and we have been solving it in production for years with platforms like <a href="https://github.com/conductor-oss/conductor">Conductor</a> that we originally built at Netflix.</p><p>A workflow engine is not the whole harness. But once an agent starts taking consequential actions across real systems, a durable workflow runtime should own their execution. The agent can decide what happens next. It should not also be the thing that remembers what already happened, retries it safely, enforces production policy, and recovers after the process dies.</p><h3>How an agent harness works</h3><p>Take an example I’ll use throughout: an on-call agent watching a service after a deploy. Error rates climb, and it has to decide how to respond — gather more evidence, compare the canary against the previous version, wait, roll back, or page a human.</p><p>The harness has <em>two distinct responsibilities:</em></p><p><strong>Deciding what should happen next.</strong> The agent reads the world — metrics, alerts, recent deploys, what it has already tried — and chooses. This is where a model earns its place: it can reason across incomplete, contradictory evidence, decide it needs one more diagnostic step before acting, and change its mind when new information arrives. Call this the planning problem.</p><p><strong>Executing the decision reliably.</strong> Suppose it decides to roll back. The rollback is not open-ended reasoning; it is an operational process with explicit steps:</p><ol><li>Confirm the new version correlates with the error spike.</li><li>Compare canary and baseline metrics.</li><li>Request approval if production policy requires it.</li><li>Shift traffic away from the new version.</li><li>Roll back, and verify recovery against the defined SLOs.</li><li>Reopen traffic, or escalate if recovery fails.</li><li>Produce an incident record.</li></ol><p>The agent decides <em>whether and when</em> to start the rollback. The workflow defines <em>how</em> it proceeds and <em>what happens when a step fails</em>. The runtime executes that definition durably. Retries, approvals, recovery, policy enforcement, state, and history are not reasoning problems — they are workflow-runtime responsibilities. The harness connects the two.</p><h3>Why teams keep rebuilding the execution layer</h3><p>If durable execution is mature infrastructure, why do agent teams keep reimplementing pieces of it inside prompts, progress files, and hand-written loops?</p><p>Partly because the first failures of long-running agents look like memory problems, not orchestration problems. Anthropic’s progress files and initializer agents exist to fix context discontinuity, incomplete handoffs, and premature declarations of success. That is real work, but a progress file is not a saga runtime: it does not give you safe retries, durable timers, human approvals, idempotency, or compensation. What these patterns reveal is a single requirement — an agent’s work cannot live only inside the model’s context window. The moment its actions touch external systems, you are in the territory workflow engines were built for.</p><p><em>The other reason is a misconception about what workflow engines can do</em>. The common picture is a rigid, pre-drawn graph written in code where every branch is known before the run begins, which makes an engine look incompatible with an agent that decides as it goes. That picture is out of date. A modern workflow engine can branch on runtime data, loop, wait for signals, pause for human input, select tasks dynamically, fan out parallel work whose width is decided mid-run, and invoke sub-workflows chosen during execution. The path through the workflow need not be known before it starts.</p><p>But agents introduce a requirement beyond ordinary runtime branching, and this is where the real difference lies. An agent does not only choose among branches you authored in advance — it may propose a new step, dependency, parallel branch, success condition, or recovery action, or revise the plan wholesale. The problem is no longer choosing a path through a known graph. It is safely admitting new structure into the graph while it runs.</p><p>In case of Conductor, the workflow itself can be fully <a href="https://docs.conductor-oss.org/devguide/ai/dynamic-workflows.html">dynamic</a>. This is a superpower of Conductor which lands itself to agentic use cases quite well.</p><h3>Agent harness vs. workflow engine: two planes</h3><p>I call this model <strong>open-world execution</strong>: the workflow does not need the complete task graph before the run starts; the runtime provides a durable substrate inside which the structure can be extended or revised as new information arrives. The next step is chosen while the workflow runs. Historically the chooser was a service reacting to data, a condition, a human approval, or a rules engine. An agent is simply a new kind of chooser. The durability primitives underneath do not need reinventing — but the interface through which plans are proposed, validated, admitted, and revised does.</p><p>This resembles what I think of as a <a href="https://medium.com/agentspan/late-bound-sagas-why-your-agent-is-not-an-llm-in-a-loop-a8c50731c551"><strong>late-bound saga</strong>.</a> A conventional saga coordinates a sequence of operations and compensates when later ones fail. In a late-bound saga the runtime may not know the full sequence up front — but every action admitted into the plan still has to carry explicit semantics: the capability that runs it, the permissions it needs, how success is judged, its retry and failure policy, and whether a compensation exists. A workflow engine does not promise that every action can be undone — some effects are irreversible, and compensations can fail too. What it promises is durable coordination: defined recovery actions get recorded, invoked, retried by policy, and escalated when they cannot complete.</p><p>That gives a clean way to picture the harness’s control path, as two planes. The <strong>planning plane</strong> is agentic, emergent, and probabilistic — the agent. The <strong>execution plane</strong> is durable, auditable, policy-controlled orchestration around work that may itself be non-deterministic — the workflow engine. The planning plane may be probabilistic. The execution record cannot be.</p><h3>For consequential agents, a workflow engine is necessary</h3><p>The foundational pieces of the execution plane are already solved, and having built these systems, I can tell you the durability primitives are the easy, finished part. Back to the rollback: surviving a crash midway needs durable state and resumption; not shifting traffic twice after a retry needs idempotency; pausing for approval and resuming later needs durable waiting; recovering from a partial failure needs compensation; and a queryable history records which steps ran and with what result. None of this is frontier research. Engines differ in how they get there — some replay deterministic code, others reconstruct state from persisted records — but the requirement is the same: execution must outlive the process performing it.</p><p>What a workflow engine does not do is the harder half. It will not tell the agent what belongs in its context or whether its diagnosis is right; it will not write a tool description the model uses correctly, judge whether an answer is good, or decide whether a proposed plan is sensible. Those are planning, context, and evaluation problems, and a workflow engine asked to solve them is the wrong tool.</p><p>So the decomposition is not that workflows solve agents; it is that an agent system pairs a mature execution problem with a new planning problem, and the mistake is rebuilding the mature half by hand because it arrived alongside the new one. The primitives are mature; the agent-native contract that composes them safely is not. On policy: enforcing a known policy at execution time is the old, solved part; deciding whether to admit a model’s proposed plan is the new part.</p><h3>What the architecture looks like</h3><p>The two planes meet at a controlled boundary. The agent proposes an action or plan; the runtime validates it against policy, turns whatever it accepts into durable execution, runs it through approved capabilities, records the outcome, and hands the result back for the agent to observe.</p><pre>                  Goal + Context<br>                       |<br>                       v<br>                 Agent / Planner<br>                       |<br>              Proposed action or plan<br>                       |<br>                       v<br>            Validation + Policy Gate<br>                       |<br>                 Admitted tasks<br>                       |<br>                       v<br>           Durable Workflow Runtime<br>               |       |       |<br>               |       |       +--&gt; Human approvals<br>               |       +----------&gt; External systems<br>               +------------------&gt; Tool workers<br>                       |<br>                       +-----------&gt; Recovery actions<br>                       |<br>                       v<br>                Recorded outcomes<br>                       |<br>                       v<br>           Agent observes and replans</pre><p>The boundary should be an explicit contract, not an instruction buried in a prompt:</p><pre>{<br>  &quot;action&quot;: &quot;rollback_deployment&quot;,<br>  &quot;inputs&quot;: {<br>    &quot;service&quot;: &quot;checkout&quot;,<br>    &quot;targetVersion&quot;: &quot;2026.06.04&quot;<br>  },<br>  &quot;successCriteria&quot;: [<br>    { &quot;metric&quot;: &quot;error_rate&quot;, &quot;operator&quot;: &quot;lt&quot;, &quot;value&quot;: 0.01, &quot;window&quot;: &quot;5m&quot; },<br>    { &quot;metric&quot;: &quot;p99_latency_ms&quot;, &quot;operator&quot;: &quot;lt&quot;, &quot;value&quot;: 500, &quot;window&quot;: &quot;5m&quot; }<br>  ],<br>  &quot;approvalPolicy&quot;: &quot;production-change&quot;,<br>  &quot;onFailure&quot;: &quot;escalate_to_on_call&quot;<br>}</pre><p>This is why open-world execution does not mean unrestricted execution: the agent may propose new structure, but every admitted action must resolve to an approved capability with a typed interface, bounded permissions, runtime-enforced policy, and an observable outcome. The plan can take shape mid-run; execution does not become informal just because the planner is a model.</p><h3>Three ways an agent can participate in a workflow</h3><p>Not every agent needs fully open-world execution. There is a progression, and most teams should start at the top:</p><ol><li><strong>Invoke a predefined workflow</strong> — the agent picks from trusted, pre-authored operations (roll back a deploy, refund an order, rotate a credential). Simplest and safest.</li><li><strong>Choose the next action dynamically</strong> — a durable workflow owns the loop, and the agent picks each next step from an approved set. The path isn’t fixed, but the capabilities are.</li><li><strong>Generate or revise the plan</strong> — the agent proposes new steps or recovery behavior mid-run. The most powerful model, and the one that most needs the admission boundary above.</li></ol><p>These are not competing definitions of an agent; they are increasing degrees of authority handed to the planning plane.</p><h3>Conclusion</h3><p>An agent harness is much larger than a workflow engine — context, tools, permissions, evaluations, memory, and the reasoning loop itself, and getting those right is genuinely hard. A capable model does not make that work, or the systems problem around it, disappear.</p><p>But the execution layer should not be rebuilt inside prompts, progress files, or an in-memory loop. When an agent’s decisions become real actions, those actions need durable state, policy, safe retries, human checkpoints, recovery, and a queryable history — every one a workflow-runtime responsibility. The primitives are mature; what is new is applying them to plans a model proposes, the runtime validates, and the agent reshapes as it learns.</p><p>The boundary I’d draw is simple: let the agent decide what should happen next, and let the workflow runtime remember, enforce, execute, and recover. The next generation of agent infrastructure won’t replace workflows with agents — it will make agents first-class participants in workflows, and let those workflows take shape while they run.</p><h3>References</h3><ol><li>Late Bound Sagas</li></ol><p><a href="https://medium.com/agentspan/late-bound-sagas-why-your-agent-is-not-an-llm-in-a-loop-a8c50731c551">https://medium.com/agentspan/late-bound-sagas-why-your-agent-is-not-an-llm-in-a-loop-a8c50731c551</a></p><p>2. Dynamic Workflows in Conductor</p><p><a href="https://docs.conductor-oss.org/devguide/ai/dynamic-workflows.html">https://docs.conductor-oss.org/devguide/ai/dynamic-workflows.html</a></p><p>3. Conductor OSS</p><p><a href="https://github.com/conductor-oss/conductor">https://github.com/conductor-oss/conductor</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b5651edd0f7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/orkes/agent-harness-workflow-engine-2b5651edd0f7">Stop Rebuilding Workflow Engines Inside Your Agent Harness</a> was originally published in <a href="https://medium.com/orkes">Orkes Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Workflow Engine Is Durable. Your Agent’s Decisions Aren’t.]]></title>
            <link>https://medium.com/agentspan/your-workflow-engine-is-durable-your-agents-decisions-aren-t-a99fa9a74e3a?source=rss-dc3485d163fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/a99fa9a74e3a</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[netflix-conductor]]></category>
            <category><![CDATA[agentic-workflow]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Viren Baraiya]]></dc:creator>
            <pubDate>Fri, 08 May 2026 07:27:30 GMT</pubDate>
            <atom:updated>2026-05-08T07:27:30.919Z</atom:updated>
            <content:encoded><![CDATA[<p>We built <a href="https://netflixtechblog.com/netflix-conductor-a-microservices-orchestrator-2e8d4771bf40">Conductor at Netflix</a> because microservice workflows would not stay still.</p><p>Teams shipped daily. Workflow shapes changed constantly. A runtime that required the entire graph to be frozen before execution could not keep up. So Conductor took a different path: workflow structure could be influenced by data produced during execution. The engine could materialize new work, persist it as state, schedule it, retry it, and record it in the same execution history as everything authored up front.</p><p>Today, we call this property <strong>dynamic synthesis</strong>.</p><p>At the time, it was just a practical answer to a microservices problem. Now it looks like the missing primitive for agents.</p><p>Nearly ten years later, at Orkes — the company commercializing Conductor — we hear the same question from almost every customer running a serious AI initiative:</p><blockquote><em>Can we run our agents on Conductor?</em></blockquote><blockquote><strong>Yes.</strong></blockquote><p>But the more important question is why so many agent systems start to break when they meet the durability model of traditional workflow-as-code platforms. After watching teams try to run serious agents in production, we have reached one conclusion:</p><h4><strong>Agents do not just call tools. Agents synthesize control flow.</strong></h4><p>The model decides what to do next, which tools to call, which branches to create, which work to fan out, when to ask a human, what to retry, and what compensation should exist if an action has to be unwound later.</p><p>If that control flow is forced into pre-authored workflow code, the agent becomes narrower. If it is generated outside the workflow engine, the audit trail splits. If it is left in application logs, the system is not truly durable.</p><p>The runtime under an agent has to record the agent’s decisions as state — not merely record that a model call happened.</p><p><strong><em>That is the gap.</em></strong></p><h4>What we keep seeing</h4><p>A team picks a durable execution platform — usually a workflow-as-code one, because that is what their platform team operates. They build the agent. It works in development. They demo it. Leadership is happy.</p><p>Then the agent meets a security review, a compliance review, an audit, or a real production incident. The audit story is the one that breaks first.</p><p>When someone asks “what did the agent do, and why?”, the team reconstructs the answer from application logs, model gateway traces, tool logs, and workflow event history. Four systems. Four clocks. Four versions of the truth.</p><p>The workflow engine can tell you that a model call happened and what it returned. It cannot tell you what the agent actually planned — the structure it synthesized, the branches it introduced, the compensations it paired with actions, the validation gates it passed through. That work lives outside the durable state machine.</p><p>That is not an audit trail. That is forensics.</p><p>The teams that hit this wall do one of two things. They build a planning service next to the workflow engine — which doubles the surface area without making the agent’s decisions native to either runtime. Or they make the agent narrower, wrapping the model in an activity and branching on its return value from a pre-coded set of options. The second approach is fine when the branches are known ahead of time. It is not the same as letting the agent compose the execution graph.</p><p>The team ships something. It is not the agent they set out to build.</p><h4>The replay contract is not a detail</h4><p>Workflow-as-code platforms are built on a powerful contract: the workflow function is deterministic, and the runtime achieves durability by replaying that function over a persisted event history.</p><p>Side effects — clock reads, network calls, model calls, database writes — are pushed into activities. The first execution records the result. Later replays reuse the recorded result.</p><p>This model has real strengths. It gives durable recovery from failures, a strong programming model, type-checked workflows, IDE ergonomics, and a clean way to reason about long-running code. For workflows whose shape is known ahead of time, it is excellent.</p><p>But the replay contract is not an implementation detail. It is the foundation of the durability model. And that foundation creates a hard boundary for agents.</p><p>To be clear: replay-based systems can call models from activities. They can persist the returned value. They can branch on that value. They can build useful agentic systems when the possible branches are known ahead of time.</p><p>The limitation appears when the model is not merely selecting among pre-authored branches, but synthesizing new workflow structure during execution.</p><p>New tasks. New dependencies. New fan-outs. New human gates. New compensations. New follow-up steps based on context the developer did not know when the workflow was authored.</p><p>That synthesized structure is the agent’s real control flow.</p><p>If the workflow function must remain deterministic, that control flow cannot simply emerge inside the function as arbitrary new structure. It has to be fenced into pre-authored branches, interpreted by application code, or generated into a separate workflow outside the runtime.</p><p>That is where durability stops.</p><p>The platform may durably record that a model was called. It may durably record the model’s returned value. But unless the runtime can accept the agent’s newly synthesized execution graph as a first-class state transition, the agent’s decisions are not native to the durable record.</p><p>The system is durable around the agent. Not through it.</p><h4>Open-world execution</h4><p>The runtime under an agent has to do one specific thing:</p><p><em>Accept new workflow structure as a legitimate state transition during execution.</em></p><p>Borrowing from the open-world / closed-world distinction in logic and databases, I call this property <strong>open-world execution</strong>.</p><p>Open-world execution means the runtime does not require the full workflow graph to be known before execution begins. New tasks, dependencies, branches, fan-outs, human approvals, and compensations can be introduced during execution, validated by policy, persisted as state, and scheduled by the same engine that runs pre-authored workflows.</p><p>The runtime’s answer to “what can happen next?” is not limited to what was committed in advance. It is defined by what has been safely recorded.</p><p>In a closed-world runtime, the set of possible futures is bounded by the workflow code deployed before the execution started.</p><p>In an open-world runtime, the execution record is allowed to grow. A worker — ordinary code or an LLM-driven worker — can return a payload that says:</p><blockquote><em>“Here are the next tasks under this workflow.”</em></blockquote><p>The runtime validates that payload, persists the tasks into the same execution record, schedules them, records the results, and advances the workflow.</p><p>There is no second code path for dynamically synthesized work. Once the new structure is recorded, it is just another durable state transition.</p><p>Replay does not need to re-derive the agent’s decision from a model. It does not need to ask the model to be deterministic. It does not need to reconstruct intent from logs.</p><p>The decision has already become state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cnmj8TtXbbShT1DvKsewrA.png" /></figure><h4>Why Conductor fits</h4><p>Conductor was not created for agents. It was created because microservices needed a workflow runtime where execution could evolve without freezing the world every time a team shipped.</p><p>That led to primitives like FORK_JOIN_DYNAMIC, SUB_WORKFLOW, HUMAN, EVENT, WAIT, DECISION, and INLINE tasks.</p><p>The important part is not any one task type. The important part is the state model: Conductor can accept runtime-generated structure, persist it under the execution, schedule it through the same coordinator, and record it in the same ledger as the rest of the workflow.</p><p>Human-authored workflows and runtime-authored agent plans feed the same underlying primitive — a workflow definition with a tasks array, where each task carries a name, type, reference, and inputs, and ordering follows from the array plus task types like FORK_JOIN_DYNAMIC or SUB_WORKFLOW for runtime fan-out:</p><pre>{<br>  &quot;name&quot;: &quot;agent_investigation&quot;,<br>  &quot;version&quot;: 1,<br>  &quot;tasks&quot;: [<br>    {<br>      &quot;name&quot;: &quot;fan_out_lookups&quot;,<br>      &quot;taskReferenceName&quot;: &quot;fan_out_lookups_ref&quot;,<br>      &quot;type&quot;: &quot;FORK_JOIN_DYNAMIC&quot;,<br>      &quot;inputParameters&quot;: {<br>        &quot;dynamicTasks&quot;: &quot;${agent_proposal.output.tasks}&quot;,<br>        &quot;dynamicTasksInput&quot;: &quot;${agent_proposal.output.inputs}&quot;<br>      }<br>    }<br>  ]<br>}</pre><p>Whether that structure was authored by a developer at deploy time or synthesized by an agent at runtime, the engine treats it as workflow state.</p><p>That is what makes Conductor interesting for agents. Not because it has an AI feature bolted onto the side. Because the core execution model already fits the agent workload.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9S28M5YaY86AOR3g-ZQWJA.png" /><figcaption>Conductor + Agentspan — Durable production agents</figcaption></figure><h4>Agentspan: reasoning plane and execution plane</h4><p>Agentspan is our attempt to package this primitive for agents. It separates the system into two planes.</p><p><strong>Reasoning plane: the model.</strong> The model receives a context projection: session state, available tools, prior results, conversation history, policy hints, and relevant memory. It proposes intent.</p><p>It does not execute tools directly. It does not mutate production systems directly. It does not become the runtime.</p><p>It proposes what should happen next.</p><p><strong>Execution plane: Conductor.</strong> Conductor validates the proposal against the agent’s tool catalog and policy. It materializes the proposal as workflow tasks under the agent’s parent execution. It schedules the work, retries failures, records results, and enforces the same RBAC, traceability, and recovery semantics used for ordinary workflow tasks.</p><p>In that split, the model proposes structure, Conductor executes it under policy, and the durable ledger preserves the proposal, materialization, and results together.</p><p>That is the boundary we want in production AI systems. The LLM should be allowed to propose structure. It should not be allowed to become the invisible place where production control flow lives.</p><h4>On-call: an agent investigating an incident</h4><p>Suppose an on-call agent is investigating latency on the checkout service.</p><p>The agent has tools for metrics, deploy history, alert correlation, incident updates, rollback, roll-forward, and paging a human.</p><p>The model sees the page, reads the current context, and proposes a parallel investigation:</p><pre>{<br>  &quot;intent&quot;: &quot;FORK_JOIN_DYNAMIC&quot;,<br>  &quot;tasks&quot;: [<br>    {<br>      &quot;type&quot;: &quot;metrics_lookup&quot;,<br>      &quot;input&quot;: { &quot;service&quot;: &quot;checkout&quot;, &quot;window&quot;: &quot;30m&quot; }<br>    },<br>    {<br>      &quot;type&quot;: &quot;deploy_history&quot;,<br>      &quot;input&quot;: { &quot;service&quot;: &quot;checkout&quot;, &quot;window&quot;: &quot;2h&quot; }<br>    },<br>    {<br>      &quot;type&quot;: &quot;alert_correlate&quot;,<br>      &quot;input&quot;: { &quot;service&quot;: &quot;checkout&quot;, &quot;window&quot;: &quot;30m&quot; }<br>    }<br>  ]<br>}</pre><p><em>Conductor validates the tools, checks the policy, materializes the fan-out as tasks, schedules them, records each result, and resumes the agent with a new context projection.</em></p><p>The model did not execute anything; the runtime did. The execution ledger now contains the proposal, the materialized tasks, the tool results, and the next decision point in one durable record.</p><p>That is the difference between tool calling and durable agent execution.</p><p>Tool calling says: the model asked for tools.</p><p>Durable agent execution says: the model proposed a state transition, the runtime validated it, the engine materialized it, and the ledger recorded it.</p><p>The agent’s plan is not living in application logs. It is part of the durable execution record, alongside the workflow steps that run beside it. Whatever the agent decides to do next — fan out further, escalate to a human, retry under a different policy, hand off to another agent — flows through the same primitive and lands in the same ledger.</p><p>That is what production requires.</p><h4>Pick the runtime whose durability reaches the work</h4><p>This is not an argument for rip-and-replace. If you operate a workflow-as-code platform today for code-defined workflows, keep it. Those platforms are good at what they were built for.</p><p>When the workflow is a program and the program is known ahead of time, replay-based durability is a strong model. It gives developers a clean abstraction for long-running code.</p><p>But agents are not ordinary long-running code. An agent’s control flow is produced at runtime. The model is not merely computing a value inside a fixed graph. It is proposing the graph.</p><p>That changes the durability requirement.</p><p>The question is no longer:</p><blockquote><em>Can my workflow engine call an LLM?</em></blockquote><p>Every serious platform can call an LLM. The real question is:</p><blockquote><em>Can my workflow engine durably record the control flow the LLM synthesizes?</em></blockquote><p>If the answer is no, the agent’s decisions live somewhere else: in logs, in a planning service, in generated workflow definitions, or inside application code that interprets model output. That may be acceptable for simple systems. It is not enough for agents that need governance, auditability, replay, compensation, human review, and production recovery.</p><p>The question is not whether your workflow engine is durable. The question is where that durability stops.</p><p>For code-defined workflows, closed-world workflow platforms can be excellent. The workflow is the program, and the program is known ahead of time.</p><p>For agents, the workflow is not fully known until the model begins reasoning. The agent’s decisions are the control flow.</p><p>If those decisions live outside the durable state machine, your agent is not durable. It is surrounded by durable infrastructure.</p><p>That distinction will matter more as agents move from demos to production systems that touch money, infrastructure, customer data, and security policy.</p><p>Conductor had open-world execution because microservices needed it first. Agentspan builds on that primitive for agents.</p><blockquote><strong>The model reasons. The runtime governs. The ledger records. The system recovers.</strong></blockquote><p>Pick the runtime whose durability extends to where the work actually happens.</p><p><strong>Conductor</strong> — <a href="https://github.com/conductor-oss/conductor">github.com/conductor-oss/conductor</a></p><p><strong>Agentspan</strong> — <a href="https://github.com/agentspan-ai/agentspan">github.com/agentspan-ai/agentspan</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a99fa9a74e3a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/agentspan/your-workflow-engine-is-durable-your-agents-decisions-aren-t-a99fa9a74e3a">Your Workflow Engine Is Durable. Your Agent’s Decisions Aren’t.</a> was originally published in <a href="https://medium.com/agentspan">Agentspan</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The anatomy of an agent runtime]]></title>
            <link>https://medium.com/agentspan/the-anatomy-of-an-agent-runtime-e6eb10bdfcdd?source=rss-dc3485d163fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/e6eb10bdfcdd</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Viren Baraiya]]></dc:creator>
            <pubDate>Wed, 29 Apr 2026 20:38:13 GMT</pubDate>
            <atom:updated>2026-04-29T20:38:13.225Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A companion to </em><a href="https://medium.com/agentspan/late-bound-sagas-why-your-agent-is-not-an-llm-in-a-loop-a8c50731c551"><em>Late-Bound Sagas: Why Your Agent Is Not an LLM in a Loop</em></a><em>. That post answered why your agent is not an LLM in a loop. This one answers what you build instead.</em></p><p>This post dives deeper into the architecture of an agent runtime — one built around a clean separation between planning and execution, not just an LLM in a loop.</p><p>The LLM is the brain. The tools are the hands. The runtime is the nervous system that keeps them from killing you. Most teams build the brain and the hands and then wonder why the thing falls over in production. This post is about the nervous system.</p><h3>Requirements for an agent runtime</h3><p>Before the design, the shape of the problem. Any runtime worth shipping has to meet all of these under adversarial operational conditions — process crashes, cluster reboots, network partitions, API outages, humans who wander off for a weekend:</p><ul><li><strong>R1 — Mid-step crash safety.</strong> An agent that terminates mid-tool-call must not leave the external system in an ambiguous state. On recovery, every side effect has run exactly once or is detectable as a duplicate.</li><li><strong>R2 — Long-suspension survival.</strong> A saga blocked on a human, a webhook, or an external asynchronous event must consume zero compute during the wait and survive any number of infrastructure events while suspended.</li><li><strong>R3 — Out-of-band interruption.</strong> Supervisors, humans, budget caps, and timeouts must be able to interrupt a running or suspended saga by ID, without shared memory, without racing the saga’s own progress.</li><li><strong>R4 — Dynamic graph.</strong> The execution graph is synthesized at runtime by a non-deterministic planner. The runtime cannot require the graph to be known at compile time.</li><li><strong>R5 — Polyglot tools.</strong> Tools written in any language, running on any host, must participate as first-class workers. The planner’s language does not constrain the executor’s language.</li><li><strong>R6 — Production-grade observability.</strong> Every decision the agent made, every tool it called, every result it saw, must be recoverable for post-hoc inspection — for at least 13 months after the fact, across model version changes.</li><li><strong>R7 — Bounded-context projection.</strong> The planner’s context is bounded; the saga’s history is not. The runtime must project one onto the other without losing the information the planner needs to decide.</li></ul><h3>Non-goals</h3><p>Five things this design explicitly does <strong>not</strong> attempt:</p><ul><li><strong>Deterministic replay of the LLM.</strong> Temperature=0 is not bit-exact across infrastructure. Model versions change. We make the <em>ledger</em> replayable, not the model.</li><li><strong>Preventing a poorly-implemented tool from leaking its own credentials.</strong> The runtime scopes credentials to tool subprocesses; a tool that prints its credential to stdout is outside our threat model.</li><li><strong>Guarantees about LLM output quality.</strong> Guardrails exist; correctness of reasoning does not. The runtime makes a wrong agent debuggable, not right.</li><li><strong>Replacement of the model provider’s SDK.</strong> The planner calls models through whatever SDK ships. We wrap the call; we do not reimplement it.</li><li><strong>Unbounded agent execution.</strong> Every saga must have enforceable bounds — on wall-clock time, on recursion depth, on token budget, or on step count. Long-running and always-on agents are supported; <em>un</em>bounded agents are not. The runtime relies on the operator to configure at least one bound; a saga with no configured bounds will be refused at compile time.</li></ul><h3>Four properties (of the runtime) that survive a crash</h3><p>Build an agent runtime wrong and a badly-timed kill -9 charges your customer twice. Build it right and four things are true at the same time, even under a process that dies mid-step. Everything else in the design falls out of keeping all four true simultaneously.</p><p><strong>Proposal before effect.</strong> Every action is written to the ledger before it runs, with enough information to identify the effect on replay.</p><ul><li><em>Failure if absent:</em> a recovering process re-runs an action that already completed. You charge the customer twice.</li><li><em>Enforcement:</em> the runtime refuses to dispatch any task until its proposal row is committed.</li></ul><p><strong>Effectively-once side effects.</strong> A side effect runs exactly once per logical proposal, no matter how many times the process dies between proposing and recording the result.</p><ul><li><em>Failure if absent:</em> duplicate Stripe charges, duplicate emails, duplicate database rows, no way to tell which one is “real.”</li><li><em>Enforcement:</em> idempotency keys derived from the step’s stable ledger address, passed to external systems and deduped on re-dispatch.</li></ul><p><strong>Monotonic, durable graph.</strong> The execution graph is append-only. Committed nodes do not change shape. Recovery reads the graph; it does not reconstruct it.</p><ul><li><em>Failure if absent:</em> recovery produces a different graph than the pre-crash state. Debugging is impossible.</li><li><em>Enforcement:</em> the ledger is append-only; status transitions are monotonic; no node is ever deleted or mutated after commit.</li></ul><p><strong>External signal delivery.</strong> Signals from outside the saga’s process — humans, webhooks, timeouts, budget caps, other agents — are durable and deliverable by saga ID.</p><ul><li><em>Failure if absent:</em> “cancel this saga” has no mechanism. A suspended agent cannot be reached. HITL is impossible for real wall-clock durations.</li><li><em>Enforcement:</em> signals are ledger rows addressed to the saga’s ID; delivery wakes a suspended saga on the next scheduler tick.</li></ul><p>The properties are not independent. Effectively-once depends on proposal-before-effect — you need a written proposal to identify the effect on replay. External signals depend on the durable graph — signals are nodes on the graph, and they have to survive restarts. You can skip any one of the four and build a demo. You cannot skip any one of the four and survive a production Tuesday.</p><p>The rest of this document is how each property is enforced, what fails if it isn’t, and what the cost of enforcement is.</p><h3>The two planes: planner and executor</h3><figure><img alt="Agent Runtime — The Brain and the Hands" src="https://cdn-images-1.medium.com/max/1024/1*_dAHqZmSbex4eirr01jikw.png" /></figure><p>The planner plane is the LLM. The executor plane is everything else. The boundary between them is the runtime.</p><p>The planner has one job: given the trajectory so far, emit the next intent. It is a pure function — f(history) -&gt; intent — in the mathematical sense. It does not touch the network. It does not open files. It does not read the clock. If it needs the current time, the current time is in history, because the runtime put it there on a previous step. The planner is stateless beyond its prompt.</p><p>The executor has one job: make the intent real. It takes the intent off a queue, runs the corresponding code, and reports a result. The executor is where your code lives. It’s allowed to be dirty — open sockets, call Stripe, mutate disk. The executor’s relationship to the planner is mediated entirely by the ledger.</p><p>This boundary is not aesthetic. It is the thing that makes proposal-before-effect possible. If the LLM can both decide and act, there is a moment where it has decided and the runtime doesn’t know yet. That moment is the race in which you charge the customer twice.</p><p>A fair objection: modern models emit tool calls directly in a single generation. Doesn’t that collapse the two planes? No. When Claude or GPT emits call(transfer_funds, 5000), the model has produced a <em>token sequence</em> that describes a tool call. Nothing has been called. The runtime parses the sequence, writes the proposal to the ledger, dispatches the worker, writes the result back, and hands a new history to the model on the next call. The model&#39;s &quot;tool use&quot; is the planner proposing. The runtime&#39;s dispatch is the executor acting. The fact that modern models emit multiple parallel tool calls in a single generation just means one planner step produces N children in the graph — a fan-out node, not a linear one. The runtime commits all N proposals atomically before dispatching any worker.</p><p>The cleanest user-facing shape for the planner is the yield pattern:</p><pre>@agent<br>def hire_for_role(role, recruiter_id):<br>    intent = yield plan(history)<br>    result = yield execute(intent)<br>    ...</pre><p>Generators give you the right semantics for free. The user writes straight-line code; the runtime controls the schedule. Every yield is a durable pause. Between two yields, nothing the user wrote runs on the server — the runtime decides when to resume and with what. Activities in Temporal approximate this; generators in Python make it idiomatic. The decorator compiles the function into a workflow definition; the runtime steps through it, calling the LLM to fill in the plan(...) blanks and dispatching workers to fulfill the execute(...) ones. In Agentspan&#39;s case the decorator becomes a Conductor workflow whose LLM-decided steps live as loop nodes that the planner extends at runtime.</p><figure><img alt="two planes of agent runtime — planner and executor" src="https://cdn-images-1.medium.com/max/1024/1*WNJo9UUUvMEaQwIHAKtozw.png" /></figure><h3>What’s in the ledger</h3><p>The ledger is Conductor’s task model — Netflix’s open-source workflow engine, running at scale since 2016.</p><p>Four properties of the ledger are what the rest of this design rests on.</p><p><strong>Identity.</strong> Every run has a stable ID; every step within it has a stable address that survives retries. That addressability is what turns a flat task list into a graph the runtime can navigate — extending it as the agent runs, resuming into it after a crash, unwinding it for compensation.</p><p><strong>History is append-only.</strong> Status transitions are monotonic. Committed rows do not mutate. Recovery reads the graph; it does not reconstruct it.</p><p><strong>Payload.</strong> Every step carries its input and its output as durable data, including token counts on LLM calls. The runtime rolls those counts into a workflow-level total as tasks complete, which is what makes budget caps a single read rather than a scan.</p><p><strong>Graph.</strong> Parent-child relationships come from the workflow definition the compiler emitted — the fan-out node and its children, the sub-workflow call and its body — not from a parent-pointer column. Compensation walks this structure backward.</p><p>One consequence deserves a closer look. Effectively-once against an external system depends on the worker passing a stable idempotency key on every retry. The ledger’s per-step address is that key: unique across the saga’s history, stable across retries, already available. Workers hash it into the Idempotency-Key header when calling Stripe or anything else that honors one. The ledger guarantees effectively-once inside the runtime. The idempotency key extends it one hop outward — and only one hop, because the external system has to cooperate.</p><figure><img alt="agent runtime ledger for durability" src="https://cdn-images-1.medium.com/max/1024/1*muQ17qlAzKhC4px5vuLNVw.png" /></figure><h3>Tools MUST run somewhere else</h3><p>SDK-embedded workers — where your “tool” is a Python function that the agent library calls directly in-process — collapse the dispatcher into your code. They feel simpler at the hello-world stage. They become a cage the moment one of three things is true: you want to scale the tool independently of the planner (one high-throughput scraper, one high-memory reasoning node); you want to write the tool in a language other than the planner’s (your underwriting model has lived in Java for fifteen years, and nobody is rewriting it); you want to bound the blast radius of a tool crash (the tool segfaulted on a pathological PDF; your planner should not go down with it).</p><p>A workers-as-protocol-clients model — where a worker is any process that speaks the runtime’s task protocol and polls for work — addresses all three. The planner doesn’t know or care where get_weather lives. It proposes get_weather(&quot;NYC&quot;); the runtime writes the proposal and enqueues a task on a queue named get_weather; any worker subscribed to that queue picks it up, runs, and acks. Python, Go, Java, a serverless endpoint, an HTTP API — the runtime doesn&#39;t distinguish. Agentspan ships sugar variants (http_tool, api_tool, mcp_tool) that let the server itself act as the worker for pure-HTTP tools, so you don&#39;t need to run anything separate to call Stripe. The substrate is the same regardless: the planner proposes, the dispatcher queues, a worker — somewhere — handles the queue. Conductor has been shaping this protocol since 2016.</p><h4>Credentials</h4><p>One consequence worth dwelling on: tools run in their own process by default. A @tool with credentials=[&quot;STRIPE_KEY&quot;] gets that key injected into its subprocess environment, scoped to the execution, and the key never enters the planner&#39;s process. The planner can&#39;t leak a credential it was never handed. Agents that reason about Stripe don&#39;t have Stripe&#39;s API key in their prompt context. This is not a security feature bolted on; it&#39;s what falls out of separating the planes and dispatching across a boundary.</p><h3>Compilation: the agent becomes a workflow</h3><p>When you write an agent in one of the Agentspan SDKs — Python, TypeScript, Java, Go, .NET, Ruby, or Rust — you are not writing an agent. You are writing a <em>specification</em>: which model to use, which tools it has, which strategies apply (handoff, sequential, parallel, swarm), which guardrails fire on which events, which memory and context-management policies apply, which credentials bind where, which evals run when. Callables get replaced with _worker_ref markers at serialization. The spec lands on the server as a structured JSON DTO. Two stages run before the agent ever executes.</p><p>The <strong>normalizer</strong> converts framework-specific agent shapes into a canonical AgentConfig. An OpenAI Agents SDK agent, a Vercel AI SDK agent, a LangGraph StateGraph, a LangChain AgentExecutor, a Claude Agent SDK agent, a Google ADK LlmAgent, and a native Agentspan agent all land on the same AgentConfig type. A Spring-based NormalizerRegistry auto-discovers AgentConfigNormalizer beans and dispatches on frameworkId(), so supporting a new framework is a new normalizer — nothing else downstream changes. Everything downstream operates on AgentConfig; nothing downstream knows which framework the user wrote in.</p><p>The depth of that lowering varies by framework. Three strategies exist in the codebase:</p><ul><li><strong>Full extraction</strong> (OpenAI, Vercel AI, LangGraph-with-model). Model config, tools, guardrails, handoffs, and output schemas are fully extracted. The LLM call runs as a server-side Conductor task; each tool becomes an independent worker task. This is the tightest integration and gets the most value out of the runtime.</li><li><strong>Graph-structure preservation</strong> (LangGraph with an emitted _graph). The LangGraph DAG — nodes, edges, conditional edges, reducers, retry policies, recursion limits — carries through into a corresponding Conductor workflow. LLM nodes, human nodes, and subgraph nodes each compile to their matching primitive. Conditional edges become router workers. Recursion limits map to the loop&#39;s iteration cap.</li><li><strong>Passthrough</strong> (LangChain, Claude Agent SDK, LangGraph fallback). The entire framework-specific agent runs inside a single SIMPLE worker task. Agentspan wraps it without introspection. You still get durable execution, crash recovery, HITL, and observability — just not the fine-grained step-level tracing that full extraction provides.</li></ul><p>One canonical IR, multiple lowering depths, the same runtime properties regardless of how deep the lowering went. You can port incrementally — start passthrough, move to graph-structure, move to full extraction — without rewriting the agent.</p><p>The <strong>compiler</strong> takes the AgentConfig and emits a Conductor workflow definition. Stripped of implementation detail, the dispatch looks like this:</p><pre># Given a canonical AgentConfig, emit a Conductor workflow.<br>def compile(agent: AgentConfig) -&gt; WorkflowDef:<br>    match agent.strategy:<br><br>        case SingleAgentWithTools:<br>            return do_while(<br>                body=[<br>                    llm_chat_complete(agent.model, agent.tools),<br>                    switch_on_llm_response({<br>                        &quot;tool_call&quot;: simple_task(&quot;${plan.tool}&quot;),<br>                        &quot;fan_out&quot;:   fork_join(&quot;${plan.fan_out}&quot;),<br>                        &quot;handoff&quot;:   sub_workflow(&quot;${plan.agent}&quot;),<br>                    }),<br>                ],<br>                until=agent_done,<br>            )<br><br>        case Parallel:               return fork_join(agent.children)<br>        case Handoff | AgentAsTool:  return sub_workflow(agent.child)<br>        case HumanInTheLoop:         return human_task(agent.form)<br>        case Guardrail:              return check_task(agent.guardrail,<br>                                                       on_fail=agent.policy)<br>        case InlineGate:             return inline_js(agent.script)</pre><p>Two things worth naming. First, the HumanInTheLoop case emits Conductor&#39;s native HUMAN system task — no worker, no poll loop; the task sits in the ledger until a signal arrives. Second, the InlineGate case compiles simple sentinel logic (a text-contains check between sequential stages, for instance) into an INLINE task that runs server-side via graaljs: no worker round-trip for the things that do not need one.</p><p>Here is what it looks like end-to-end. A three-line agent definition:</p><pre>from agentspan.agents import Agent, tool<br><br>@tool<br>def search_candidates(role: str, limit: int) -&gt; list[str]: ...<br><br>@tool<br>def request_interview(candidate_id: str) -&gt; str: ...<br><br>agent = Agent(<br>    name=&quot;hiring_agent&quot;,<br>    model=&quot;anthropic/claude-sonnet-4-6&quot;,<br>    tools=[search_candidates, request_interview],<br>)</pre><p>Compiles to this Conductor workflow (abbreviated):</p><pre>{<br>  &quot;name&quot;: &quot;hiring_agent&quot;,<br>  &quot;tasks&quot;: [{<br>    &quot;name&quot;: &quot;agent_loop&quot;,<br>    &quot;type&quot;: &quot;DO_WHILE&quot;,<br>    &quot;loopCondition&quot;: &quot;$.iteration.done == false&quot;,<br>    &quot;loopOver&quot;: [<br>      { &quot;name&quot;: &quot;plan&quot;,<br>        &quot;type&quot;: &quot;LLM_CHAT_COMPLETE&quot;,<br>        &quot;inputParameters&quot;: {<br>          &quot;model&quot;: &quot;anthropic/claude-sonnet-4-6&quot;,<br>          &quot;messages&quot;: &quot;${workflow.input.history}&quot;,<br>          &quot;tools&quot;: [&quot;search_candidates&quot;,<br>                    &quot;request_interview&quot;]<br>        }<br>      },<br>      { &quot;name&quot;: &quot;route_on_response&quot;,<br>        &quot;type&quot;: &quot;SWITCH&quot;,<br>        &quot;evaluatorType&quot;: &quot;value-param&quot;,<br>        &quot;expression&quot;: &quot;$.plan.output.action_type&quot;,<br>        &quot;decisionCases&quot;: {<br>          &quot;tool_call&quot;: [<br>            { &quot;type&quot;: &quot;SIMPLE&quot;,<br>              &quot;name&quot;: &quot;${plan.output.tool}&quot; }<br>          ],<br>          &quot;fan_out&quot;: [<br>            { &quot;type&quot;: &quot;FORK_JOIN&quot;,<br>              &quot;forkTasks&quot;: &quot;${plan.output.fan_out}&quot; }<br>          ],<br>          &quot;handoff&quot;: [<br>            { &quot;type&quot;: &quot;SUB_WORKFLOW&quot;,<br>              &quot;name&quot;: &quot;${plan.output.target_agent}&quot; }<br>          ]<br>        },<br>        &quot;defaultCase&quot;: [{ &quot;type&quot;: &quot;TERMINATE&quot; }]<br>      }<br>    ]<br>  }]<br>}</pre><p>Nothing in that JSON is novel. Conductor has been running workflows shaped like this for a decade. What’s new is that the LLM_CHAT_COMPLETE task exists, and that the SWITCH on the LLM’s output is the <em>only</em> dynamic part of the graph. Everything else — retry policies, rate limits, worker dispatch, crash recovery, the visual UI that renders this workflow when it runs — is what Conductor does every day orchestrating video pipelines at Netflix. Temporal would be an equally valid substrate to build this on; Conductor is the one I know, and the one Agentspan uses.</p><p>The workflow that comes out is <strong>deterministic in Conductor’s sense</strong>. Conductor does not know or care that an LLM_CHAT_COMPLETE task calls an LLM, that a SUB_WORKFLOW is another agent, or that a HUMAN task is waiting on a recruiter’s click. It sees a graph of tasks, schedules them, records their results, retries per policy, applies rate limits, emits metrics, persists state. Durable workflow execution, distributed priority task queues, crash recovery, replay from a checkpoint, pause and resume — everything Conductor has done for a decade orchestrating Netflix’s pipelines now runs under the agent.</p><p>What the LLM contributes — the part that is <em>not</em> deterministic — is the <em>values</em> it produces: which tool to call, which handoff to take, which branch of a conditional to follow, when the DO_WHILE exits. The LLM is the <strong>brain</strong>, selecting edges in a graph that Conductor walks. The tools are the <strong>hands</strong>, executing what the brain selected. The workflow is the skeleton. The compiler produces the skeleton from your spec.</p><p>The late-bound saga from earlier sections is what this looks like at runtime: the specific path the agent’s execution takes through that skeleton, recorded in the ledger node by node. The skeleton is structurally bounded; the trajectory through it is decided by the model.</p><figure><img alt="Agent runtime — compiling an agent into deterministic workflow" src="https://cdn-images-1.medium.com/max/1024/1*8gI8wygQdwCGXQN3GQbiZA.png" /></figure><p>Because the agent compiles to a Conductor workflow, it inherits Conductor’s retries, rate limits, replay, observability, restart-from-step-N, and visual UI — rather than reimplementing them. Retry on tool failure is the same retry Conductor does on any task. Rate limits on the model API are the same rate-limit primitives Conductor ships with. The execution UI is Conductor’s execution UI, rendering a graph whose nodes happen to include LLM calls. Replay is Conductor’s replay. Restart is Conductor’s restart. Everything that is production-grade for a workflow engine is now production-grade for an agent, because the agent <em>is</em> a workflow.</p><p>And because the emitted artifact is a Conductor workflow definition, every agent in Agentspan is, in a literal sense, a workflow you can inspect, diff, version, export, and run by hand without the SDK. The compiler’s output is not an implementation detail. It is a first-class artifact.</p><h3>How the graph grows</h3><p>The compiled workflow is the skeleton. The saga’s <em>trajectory</em> through it — which iterations of the main loop fire, which tools get called in which order, which branches of which guardrail trigger — does not exist when the saga starts. Here is how it fills in.</p><p>The runtime begins with a single root node: the task. It asks the planner for the next intent, passing the root’s input as the history. The planner returns an intent — call it plan_0. The runtime writes plan_0 as a plan step. Its output contains a list of N proposed effects — possibly one, possibly many if the model emitted parallel tool calls. The runtime materializes those N effects as scheduled steps, all wired together as children of the plan, in one transactional commit. That commit is the monotonic durable graph doing its job: either all the children are scheduled or none of them are.</p><p>Now the runtime dispatches the N effects. Each effect travels to a worker queue; each worker pulls, executes, and acks with a result. Each result is a ledger write that flips the step’s status to a terminal state. The runtime watches for the fan-in: once all N children have completed, it asks the planner for the next intent, handing it the updated history — which now includes plan_0 and all its children. The planner emits plan_1. The graph extends.</p><p>That is the whole algorithm. <em>Plan, commit, dispatch, join, plan.</em> Every one of those verbs is a durable operation. The loop lives in the runtime, not in your code, which means the loop survives when your code doesn’t.</p><p>Recovery is straightforward. The runtime starts a saga supervisor from the ledger: it reads the saga’s graph, finds the frontier (the set of steps not yet in a terminal state, and the most recent plan without a successor), and resumes. If a step was scheduled but never picked up by a worker, it re-enqueues it. If a step was in flight — dispatched, no result yet — the worker either reports back (it survived) or the step times out and the runtime decides whether to redispatch (it didn’t). If a plan is waiting on its children, the runtime waits. If all children are done, it calls the planner. Recovery doesn’t need a special code path; it’s the same code path as forward progress, started from a different place in the graph.</p><h3>Signals: how the outside reaches a suspended saga</h3><p>A signal is a message addressed to saga_id, written to the ledger as a signal kind step, and delivered to the planner on its next call as part of the history. That&#39;s the whole mechanism. The things that fall out of it:</p><p><strong>A human clicking “approve”</strong> on a tool that requires approval is a signal. The approval-required tool proposed the effect; the runtime suspended the saga waiting for the signal; the UI wrote a signal row when the human clicked; the runtime woke the saga; the planner saw the approval in its history on the next call. Days of wall-clock time, zero process-resident state.</p><figure><img alt="Agent runtime — durability across long running agents surviving crashes" src="https://cdn-images-1.medium.com/max/1024/1*CFXb20hwjh7tK3a1lcJDsQ.png" /><figcaption>Agent runtime — durability across long running agents surviving crashes</figcaption></figure><p><strong>A timeout, a cancellation, or a supervisor telling a sub-saga to abort</strong> are all the same mechanism: a signal written to the target saga’s ledger. Timers are the runtime writing to itself when a deadline passes. Cancellations are the runtime writing on behalf of a user. Agent-to-agent messaging is one planner writing to another saga. If the writer dies mid-signal, the write either committed (and the target sees it) or did not (and the writer’s recovery will retry). There is no RPC, no shared memory, no in-process delivery. The ledger is the inbox.</p><p><strong>A budget cap is a signal</strong> the runtime writes when sum(tokens_in + tokens_out) * price crosses a threshold recorded at saga start. The planner sees budget_exceeded and can compensate, summarize, or escalate. Because cost is a ledger column, this is a SQL trigger, not a framework feature.</p><p>At-least-once is the right semantics for signals; deduplication happens at the saga level by letting the planner see duplicate signals in its history and filter them. Making the runtime dedupe is tempting and wrong: the runtime doesn’t know what “two signals mean the same thing” means.</p><h3>Compensation for a graph the model wrote</h3><p>Traditional sagas have static compensations: every local transaction has a predeclared inverse. An agent’s graph is synthesized at runtime; the inverse has to be synthesized too. Three strategies, composed by the runtime in priority order.</p><p><strong>Tool-declared inverse.</strong> The @tool author declared a compensation — another @tool or inline function — at definition time.</p><ul><li><em>How the runtime executes it:</em> on unwind, the runtime walks the graph backward, finds every effect with a registered compensation, and dispatches the inverses in reverse order, each through its own proposal-before-effect bracket.</li><li><em>Failure mode:</em> the compensation itself can fail (e.g., the refund API is down). The runtime retries per policy; on persistent failure, the saga escalates with partial-unwind state recorded in the ledger.</li></ul><p><strong>Model-authored sequence.</strong> The saga receives a cancel signal and at least one effect has no declared inverse.</p><ul><li><em>How the runtime executes it:</em> the planner sees the cancel signal in its history and emits a propose_compensation_sequence intent. This is a plan node; its children are effect nodes; all go through the same four-property bracket. The compensation is a sub-saga of the main saga.</li><li><em>Failure mode:</em> the model proposes an incorrect compensation. The ledger records the attempt; an operator can intervene. This is not a runtime failure; it is the correct answer to a fundamentally semantic problem.</li></ul><p><strong>Escalation.</strong> An effect is flagged irreversible: true on the tool definition (e.g., wire_transfer, send_email).</p><ul><li><em>How the runtime executes it:</em> the runtime stops unwinding at the irreversible boundary. The planner receives the halt condition in its history and can propose a forward-compensating action (wire money back, send correction email) or emit a human_decision_required signal.</li><li><em>Failure mode:</em> none at the runtime level — the runtime correctly refuses to pretend the irreversible action can be undone. Semantic resolution is the operator’s job.</li></ul><p>Irreversibility is a property of the tool, recorded as a flag on the effect. wire_transfer is irreversible. send_email is irreversible. write_to_db is typically not. The runtime never pretends otherwise. When the compensation planner asks the model to unwind, the history contains the irreversibility flags; the model is instructed to stop at the boundary and escalate, or propose a forward-compensating action. Neither the runtime nor the model can undo an ACH transfer; the runtime&#39;s job is to make sure nobody pretends otherwise.</p><h3>Context is the hard problem</h3><p>The ledger is unbounded. A saga can have tens of thousands of steps. The planner’s context window is bounded, and the bound is not growing fast enough to outrun the growth in trajectory length. Every runtime has to reconcile these two, and no projection strategy does it cleanly — they’re all lossy in different directions.</p><p>The design move is to refuse to pick one. The ledger stays the source of truth; the planner sees a projection of it; and the projection is a first-class, pluggable function the operator chooses per agent:</p><pre># The projection is the contract between ledger and planner.<br># Stable signature; operator-chosen implementation.<br>def project(ledger: Ledger, at_step: int) -&gt; History: ...</pre><p>That signature is load-bearing. The function is pure, deterministic given the ledger, and invoked at every planner step. Agentspan ships three implementations and lets you register your own:</p><ul><li><strong>Full replay.</strong> Feed the entire ledger in. Works for short sagas, breaks on long ones, wastes money on every step because the prefix doesn’t change.</li><li><strong>Windowed.</strong> Feed the root plus the last K steps. Simple, cheap, and catastrophic the moment the model needs context from step 14 of a 2000-step saga. Pair with a recall(step) tool so the planner can pull an old step&#39;s output back into context when it notices the gap.</li><li><strong>Summarized.</strong> Periodically, the runtime inserts a summarize plan step whose output replaces a chunk of the ledger in future views. The summaries themselves live in the ledger, which means they replay, which means a debug session months later sees exactly what the planner saw.</li></ul><p>There is no best projection. Summarization is the least lossy but costs an LLM call to produce and can bury a detail the model needed on step 847. Windowing is cheapest but blind to anything old. Retrieval splits the difference but puts load on the planner to know what to ask for. Which is right depends on the agent, the average saga length, the cost budget, and what the agent is expected to remember. Making the choice explicit — a configured projection per agent, recorded in the workflow definition, visible in the ledger — is what turns “memory” from a black box into a debuggable surface.</p><p>Two properties of this design matter more than which strategy you pick.</p><p><strong>The projection is recorded, not inferred.</strong> Every planner step in the ledger references the projection function and its parameters. When you open a saga in the UI six months later and ask “why did the agent not use the invoice from step 40?”, the answer is visible: the projection at step 312 was windowed(k=50), and step 40 wasn&#39;t in the window. Without that recording, the question has no answer.</p><p><strong>Projections compose.</strong> A real production agent uses more than one. The typical shape is windowed recent history plus summarized older history plus a retrieval escape hatch — three projections stacked into one History. Because the projection is a function, composition is function composition. Because the composition is registered per agent, the operator can change the mix without touching the agent&#39;s code.</p><p>This is also where the illusion of “memory” ends and the mechanics begin. An agent “remembers” a fact from two hours ago if, and only if, the projection at step N includes that fact. The ledger never forgot; the projection did. Being precise about this is the difference between a runtime you can reason about and a system that occasionally surprises you.</p><h3>Replay and the model-version problem</h3><p>Temporal’s promise is deterministic replay: given the same code and the same activity outputs, the workflow produces the same decisions. That property is load-bearing for correctness on top of Temporal.</p><p>You can’t have it for agents. Temperature=0 is not bit-exact across infrastructure changes. Model versions change underneath you. Injecting the current date alone introduces non-determinism in the LLM’s output. Asking for deterministic replay in the Temporal sense is asking the model to be reproducible, and it isn’t.</p><p>So the runtime has to distinguish two kinds of replay.</p><p><strong>Recovery replay</strong> is what happens when a process dies and a saga resumes. This is deterministic in the sense that matters: committed results are treated as final, uncommitted effects are re-dispatched with the same idempotency key, no LLM call is re-issued for a committed plan step, no tool is re-invoked for a committed effect. The saga moves forward from where it was. This is sound because proposal-before-effect plus the durable graph make side effects witnessable; you don’t need the model to be bit-exact, you need the ledger to be.</p><p><strong>Cognitive replay</strong> is what happens when you want to ask “what would this agent have done if I changed prompt X?” at debugging time. This is not deterministic; it’s a new run against a recorded history. You run the planner with the ledger’s input at step K and see what comes out. You compare. The answer is rarely the same tokens, but it’s often the same shape, and the shape is what you want when you’re debugging.</p><p>The cost of getting these confused is high. Teams who expect cognitive replay to be bit-exact build flaky test suites. Teams who expect recovery replay to “just work” without the proposal-before-effect bracket double-charge customers. Separating them — in the design and in the UI — is worth the effort.</p><h3>Testing agents without shipping them to production</h3><p>The ledger doubles as the substrate for every testing strategy an agent runtime actually needs. There are three patterns, each targeting a different failure mode, each built on the same artifact: a recorded ledger.</p><p><strong>Unit-testing the planner.</strong> The planner is a pure function — f(history) -&gt; intent — with no side effects. That makes it unit-testable the way any pure function is: construct a history, call the planner, assert on the intent. In practice, &quot;constructing a history&quot; means either writing ledger rows by hand (tedious but precise) or loading a recorded saga&#39;s ledger and feeding it to the planner at an arbitrary step. The runtime exposes the second pattern as a first-class operation:</p><pre>from agentspan.testing import load_saga, run_planner<br><br>ledger = load_saga(&quot;s_42&quot;)<br>intent = run_planner(agent, ledger, at_step=5)<br>assert intent.tool == &quot;enrich_candidate&quot;<br>assert intent.args[&quot;limit&quot;] == 20</pre><p>This tests the planner’s decision at step 5 of a real saga without running any tools, without hitting any external system, and without spending any tokens beyond the one planner call. A CI run of 100 such tests costs less than one production saga.</p><p><strong>Integration-testing the saga.</strong> The runtime’s end-to-end test is: start a saga with a fixed seed (deterministic tool-mocks, recorded LLM responses), let it run to completion, compare the resulting ledger to a golden ledger. Every row in the golden ledger must match: plans, effects, signals, compensations, terminal status. This is the kind of test that catches “the agent used to call the search tool three times; now it calls it four” — a regression the LLM alone would not surface.</p><pre>from agentspan.testing import replay_saga, assert_ledger_matches<br><br>result = replay_saga(<br>    agent,<br>    task=&quot;hire for senior backend&quot;,<br>    llm_responses=fixture(&quot;recorded_responses.json&quot;),<br>    tool_mocks={&quot;search_candidates&quot;: lambda role, limit: [...], ...},<br>)<br>assert_ledger_matches(result.ledger, fixture(&quot;golden_ledger.json&quot;))</pre><p><strong>Regression-testing against model version changes.</strong> When the model provider ships claude-sonnet-4-7 and you want to know whether it behaves the same as claude-sonnet-4-6 on your production workloads: cognitive replay becomes CI. Point the planner at a corpus of real ledgers, rerun each at a specific step with the new model, diff the trajectories. The question isn&#39;t &quot;does the new model produce the same tokens&quot; — it won&#39;t — but &quot;does the new model make the same <em>decisions</em> at the same branch points.&quot; This is measurable: for each plan step, the old model proposed tool X with args A; the new model proposed tool Y with args B. You score the divergence. Anything over a threshold triggers human review before the model version rolls forward.</p><p>Crucially, none of these patterns require the runtime to support a special “test mode.” They all operate on the ledger — the same artifact the runtime produces in production. A test is just a saga you run against recorded inputs and assert on the result. A regression check is just a saga you replay. An outage debug is just a saga you load in the UI. One substrate, three use cases, one mental model.</p><h3>Observability is not a feature</h3><p>Distributed tracing for agents is often sold as a checkbox. In a ledger-based runtime, it’s what you already have. Every step has timing data, a position in the graph, its input, its output, and a token count where relevant. A trace is a graph query. A waterfall is a topological sort. <em>Why did the agent do this</em> is a scan of the ledger ordered by time.</p><p>What you add on top is cosmetics: a visual UI over the graph, Prometheus metrics aggregated from the running totals, OpenTelemetry spans that correspond one-to-one with steps. The hard work was done when you decided the ledger had stable per-step addressing, durable inputs and outputs, and a workflow-level rollup. Don’t re-do it in a separate tracing layer.</p><h3>The cost of the ledger</h3><p>On a single-node Conductor setup with Redis-backed queues and Postgres-backed indexing (wrk2 sustained load, batched long-poll at 10 tasks/batch, ~80 MB/s sustained workload data), four writes per step is ~20ms of ledger overhead per state transition at p99. An LLM call is 500ms at the aggressive end — short output, warm model — and several seconds in the common case. The ledger is 1–3% of the planner round-trip. You do not feel it on the single-saga path. You feel the LLM.</p><p>What you feel under load is throughput, not latency. The same setup sustains ~1,450 task executions per second and ~210 moderately-complex workflows per second — a rate that projects to roughly 540M workflows per month on one commodity node. Netflix runs Conductor at multiple orders of magnitude beyond that using Cassandra with a Redis-based queuing layer. Servers are stateless; horizontal scale is linear until Redis or Postgres becomes the bottleneck, at which point the knobs are Redis Cluster and provisioned IOPS. Full methodology and numbers are in <a href="https://medium.com/orkes/running-a-billion-workflows-a-month-with-netflix-conductor-5fd358d9b6c1">Running a Billion Workflows a Month with Conductor</a>.</p><p>The real failure mode to watch for is ledger contention under fan-out. A saga that proposes 1,000 parallel effects in one plan step will commit 1,000 rows in one transaction, then drive 1,000 workers through 1,000 result writes. The ledger handles it. The bottleneck is the downstream worker queue, and more often, the external APIs the workers are calling. A runtime that has thought about this — partitioned queues per tool type, backpressure on fan-out width, per-saga write batching — absorbs the spike. A runtime that has not will throttle the wrong thing and the symptom will look like a database problem when it is a queue problem.</p><h3>Failure modes</h3><p>A design is only as good as its worst-case behavior. Five failure modes are what a production operator will actually see; this section is the runtime’s contract for how each is handled.</p><p><strong>Worker crash mid-tool-call.</strong> The worker process dies after the external side effect ran, but before acking the result back to the ledger.</p><ul><li><em>Detection:</em> task heartbeat timeout; the worker does not respond within the configured dispatch window.</li><li><em>Runtime response:</em> redispatch the task to any available worker, carrying the <strong>same idempotency key</strong>. The external system returns the existing result or re-executes idempotently. The ledger records the result exactly once.</li><li><em>Operator sees:</em> brief uptick in the task-retry metric. Saga completes without intervention.</li></ul><p><strong>LLM provider outage.</strong> The planner’s model API returns 5xx or times out on every request.</p><ul><li><em>Detection:</em> LLM_CHAT_COMPLETE task failure rate exceeds threshold over a rolling window.</li><li><em>Runtime response:</em> per-saga exponential backoff up to a bounded ceiling. Sagas remain alive but paused. If the operator has configured a fallback model, the runtime fails over per policy.</li><li><em>Operator sees:</em> saga progress rate drops; no sagas fail. The backoff metric exposes the upstream issue directly.</li></ul><p><strong>Worker pool exhaustion.</strong> Task dispatch rate exceeds worker capacity for a particular tool; queue depth grows unboundedly.</p><ul><li><em>Detection:</em> queue depth metric exceeds threshold; task-wait-time p99 climbs.</li><li><em>Runtime response:</em> backpressure on new fan-outs for that tool type. Sagas that do not use the exhausted tool continue unaffected. Tasks are never silently dropped.</li><li><em>Operator sees:</em> “Workers for tool_x saturated&quot; alert. Sagas depending on tool_x slow; others unaffected.</li></ul><p><strong>Context window overflow.</strong> A saga’s history exceeds the planner’s context bound.</p><ul><li><em>Detection:</em> the projection function detects token count over the configured model limit before dispatching the LLM_CHAT_COMPLETE task.</li><li><em>Runtime response:</em> apply the configured context-management policy — summarization, windowed truncation, or explicit retrieval-on-demand. If no policy is configured and hard overflow is reached, the saga fails with CONTEXT_OVERFLOW and the ledger is preserved for inspection.</li><li><em>Operator sees:</em> either degraded saga quality (summarization was lossy) or explicit saga failure with a full ledger to debug. Never a silent truncation.</li></ul><p><strong>HITL signal delayed indefinitely.</strong> A saga suspends on a HUMAN task; the human takes a weekend, a vacation, or never responds at all.</p><ul><li><em>Detection:</em> the HUMAN task is a durable row in the ledger; its signal is delivered through a queue-backed eventing layer. Signals are not lost — they are received whenever the human eventually acts, or never at all if they don’t. The failure mode here is semantic (did the human respond?), not delivery (did the message arrive?).</li><li><em>Runtime response:</em> the saga stays suspended, consuming zero compute, until a signal arrives. If a per-HUMAN-task timeout is configured, the runtime writes a timeout signal at the deadline and the planner decides how to proceed (retry, compensate, escalate). Without a configured timeout, the saga remains suspended indefinitely, which is sometimes the correct behavior — a pending approval is a pending approval.</li><li><em>Operator sees:</em> the saga surfaces as “awaiting human” in the UI with full ledger context and the age of the wait. Suspended sagas are first-class inventory, not zombie processes.</li></ul><p>The unifying principle: <strong>the runtime’s worst behavior is to stop making progress, never to make incorrect progress.</strong> A stopped saga is debuggable. A saga that silently did the wrong thing is not. Every failure mode above chooses visibility and pause over silent continuation.</p><p>Three classes of issue are explicitly out of scope for the runtime to handle and belong to the operator or the application:</p><ul><li><strong>Logic bugs in tools.</strong> A tool that computes the wrong answer is the tool author’s problem. The runtime records the call and the result; it does not validate correctness.</li><li><strong>Prompt injection via tool outputs.</strong> A malicious tool output that manipulates the planner is a content-layer problem. Guardrails help; the runtime does not detect injection.</li><li><strong>Semantic compensation.</strong> The runtime walks the graph backward on cancel; it does not know that refunding a payment also requires sending an apology email. The compensation planner decides; the runtime executes.</li></ul><h3>Tradeoffs accepted</h3><p>No design is free. Five costs we pay on purpose, each chosen because the alternative was worse.</p><p><strong>No deterministic replay of the planner.</strong> Temperature=0 does not give bit-exact replay across infrastructure; model versions change; even the current date in context produces different outputs.</p><ul><li><em>What we get:</em> recovery replay still works — committed results are final, uncommitted effects re-dispatched with the same idempotency key. Cognitive replay becomes a design-time tool, not a correctness guarantee.</li><li><em>Why it’s the right call here:</em> LLMs are non-deterministic. A runtime that pretends otherwise builds flaky tests and misleads its operators. Separating the two kinds of replay is the honest abstraction; collapsing them to sell “Temporal for agents” is not.</li></ul><p><strong>More operational surface area.</strong> A runtime server, a queue, a database, a worker pool, a UI. Not a single import.</p><ul><li><em>What we get:</em> durable execution, horizontal scaling, operator visibility, the ability to reason about one saga across pods, the ability to kill a saga by ID from outside its process.</li><li><em>Why it’s the right call here:</em> the alternative — an in-process agent library — is a single point of failure that cannot be inspected, interrupted, or scaled independently of the application it’s embedded in. When the process dies, the saga dies with it, and nothing that happened during the run is recoverable. An operator staring at a production incident will take more components over a black box every time.</li></ul><p><strong>The agent definition is not Python — it’s a workflow.</strong> What you write in Python (or TypeScript, or Java) compiles to a Conductor workflow definition. The workflow is the artifact that runs.</p><ul><li><em>What we get:</em> production-grade retries, rate limits, observability, replay, pause/resume, a visual UI, horizontal scale, thirteen months of operational history — all inherited, not built.</li><li><em>Why it’s the right call here:</em> building these from scratch for agents is a decade of work that has already been done for workflows. Pretending that an agent is sufficiently different from a workflow to justify starting over is the mistake most agent frameworks have made. An agent is a workflow whose graph is synthesized at runtime by a non-deterministic planner. That’s the only new thing. Everything else is solved.</li></ul><p><strong>Signals at-least-once, not exactly-once.</strong> The planner may see the same signal twice in its history.</p><ul><li><em>What we get:</em> simpler delivery semantics. No distributed consensus required at the signal boundary. No “did my signal actually arrive” ambiguity because the ledger is the inbox.</li><li><em>Why it’s the right call here:</em> deduplication at the planner level is cheap — the LLM sees “accept” and “accept” in the history and treats them as the same event. Runtime-level exactly-once is distributed-systems folklore and does not exist without coordination overhead that would dominate the ledger’s latency budget. Pay the cheap dedup at the planner; skip the expensive dedup at the runtime.</li></ul><p><strong>No guarantee that the agent’s answer is correct.</strong> Guardrails catch policy violations; they do not validate truth.</p><ul><li><em>What we get:</em> the runtime is a substrate, not a referee. Correctness is a product-layer concern, handled by evals, human review, and domain-specific validators.</li><li><em>Why it’s the right call here:</em> correctness is not decidable in the general case. A runtime that claims to guarantee correct agent output is lying, and a runtime that tries to enforce correctness ends up either refusing valid outputs or waving through invalid ones. The honest move is to make the wrong answer <em>visible and debuggable</em> — through the ledger, the trace, the recorded projection at each step — rather than pretending to prevent it.</li></ul><p>The theme across all five: we prefer <strong>visible, bounded, operator-controllable cost</strong> over hidden, unbounded, operator-invisible cost. Every tradeoff above trades a marginal cost for the ability to reason about the system from outside it. For production software, that is the correct direction.</p><h3>The irreducible core</h3><p>A runtime for agents — the nervous system between the brain and the hands — does six things. Everything else is sugar or consequence.</p><p>It <strong>owns the program counter</strong>, so your code dying does not kill the saga. It <strong>witnesses every intent</strong> before it becomes an effect, so “did this happen?” is always answerable. It <strong>gates every effect</strong> until the ledger can account for it, so you never charge the customer twice. It <strong>delivers every external signal</strong> as a first-class ledger event, so the outside world can interrupt a running agent without racing it. It <strong>projects the growing history</strong> into the bounded context of a non-deterministic planner, so the model sees exactly what the runtime decides it should see. It <strong>refuses to execute any code the user wrote</strong> except as a dispatched task in a worker it does not share a process with, so one bad tool cannot take down the brain.</p><p>That is the core. Drop any one of the six and you have a framework that demos well and fails quietly. Keep all six and you have a runtime.</p><p>The decorator, the UI, the provider integrations, the multi-agent strategies, the guardrails, the compiler — all of it is either sugar over those six properties or a feature built by taking them as given.</p><p>An agent is not an LLM in a loop. It’s a saga the model writes as it runs, and the thing that makes it real is the runtime underneath.</p><h3>Appendix: key types</h3><p>The eight terms that do the most work in this document.</p><ul><li><strong>Saga.</strong> A long-running, durable agent execution. A single agent invocation produces a single saga, identified at the substrate level by a Conductor workflowInstanceId. Survives process death and cluster restarts.</li><li><strong>Step.</strong> A single task in the saga’s Conductor task list. Every plan, every effect, every signal, every compensation is a task. Identified by taskId (unique per attempt) and referenceTaskName (stable across retries).</li><li><strong>Plan.</strong> A task of type LLM_CHAT_COMPLETE in which the planner proposes what to do next. Output lands in outputData.toolCalls as one or more intents.</li><li><strong>Intent.</strong> The planner’s proposed next action: a tool call, a fan-out, a handoff, or a final answer. An intent is <em>data</em>, not an action. It becomes an action only when the runtime materializes it as a scheduled task.</li><li><strong>Effect.</strong> A task the runtime dispatches to execute an intent — typically SIMPLE, SUB_WORKFLOW, FORK_JOIN, or INLINE. Produces a result in outputData.</li><li><strong>Signal.</strong> A message addressed to a saga from outside the saga’s process. Signals from humans (delivered to a waiting HUMAN task), the runtime itself (timeouts, budget caps), or other sagas (supervisor abort). Delivered through Conductor&#39;s eventing layer; observed by the planner on its next call.</li><li><strong>Projection.</strong> The function that maps the full task history into the bounded context the planner sees on each call. Implemented as windowed truncation, summarization, retrieval-on-demand, or a mix.</li><li><strong>Lowering.</strong> The compiler’s job: transforming a canonical AgentConfig into a Conductor workflow definition. Called &quot;compilation&quot; in this document even though no code generation occurs, because the transformation has every property a compiler has except a target ISA.</li></ul><p>Code: <a href="https://github.com/agentspan-ai/agentspan">github.com/agentspan-ai/agentspan</a>. Agentspan is open source and looking for builders. PRs welcome.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e6eb10bdfcdd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/agentspan/the-anatomy-of-an-agent-runtime-e6eb10bdfcdd">The anatomy of an agent runtime</a> was originally published in <a href="https://medium.com/agentspan">Agentspan</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Late-Bound Sagas: Why Your Agent Is Not an LLM in a Loop]]></title>
            <link>https://medium.com/agentspan/late-bound-sagas-why-your-agent-is-not-an-llm-in-a-loop-a8c50731c551?source=rss-dc3485d163fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8c50731c551</guid>
            <category><![CDATA[temporal]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[llm-agent]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Viren Baraiya]]></dc:creator>
            <pubDate>Mon, 13 Apr 2026 20:15:52 GMT</pubDate>
            <atom:updated>2026-04-13T20:39:12.905Z</atom:updated>
            <content:encoded><![CDATA[<p>Here is how a lot of agents are still written today:</p><pre>def run_agent(task):<br>  state = State(task)<br>  while not state.is_terminal():<br>    intent = llm.plan(state)<br>    result = execute_tool(intent.tool)<br>    state.append(result)<br>  return state.final_answer()</pre><p>Here is how it <em>should</em> be written:</p><pre>@agent<br>def run_agent(task):<br>  while not done(task):<br>    intent = yield plan(task)<br>    result = yield execute(intent)<br>    task = yield append(task, result)</pre><blockquote><em>The syntax change is small. The runtime boundary is not. In the first version, your process owns execution state. In the second, the runtime does.</em></blockquote><h3>An agent is a saga that the model writes as it runs.</h3><p>Not a loop the model sits inside. That distinction — saga vs. loop — is the difference between a demo and a system you can run a business on. It has a name: the <strong>Late-Bound Saga</strong>. The rest of this post is what it is, why you need it, and what your code looks like on the other side.</p><p><em>I’ve spent the last decade building </em><a href="https://github.com/conductor-oss/conductor"><em>Conductor</em></a><em>, the durable-execution runtime we open-sourced in 2016 at Netflix, which today runs long-lived workflows at a lot of companies you’ve heard of. This post is what I’ve learned about how that kind of runtime has to change for agents — and why the shape of the answer is not what the durable-execution lineage has been building for the last ten years.</em></p><h3>Durability is not continuity</h3><blockquote><em>My agent runs in a pod. I checkpoint to Postgres after every tool call. If the process dies, Kubernetes brings it back. I have durability. What else is there?</em></blockquote><p>Your agent calls Stripe to charge a customer $4,000. The call succeeds. Before your code can write the row that says “Stripe call complete,” the pod gets evicted — node pressure, a rolling deploy, an OOM from a context window that grew larger than you planned. Kubernetes restarts the pod. Your agent reads the last checkpoint, which says “about to call Stripe.” It calls Stripe again. You’ve just charged the customer $8,000.</p><p>Postgres did its job. Kubernetes did its job. Your checkpoint did its job. You still double-charged the customer.</p><p>The bug isn’t that anything failed. Durability means your data survives. Continuity means your program counter survives. The runtime knows which instruction you were about to execute, determines whether the side effect already ran, and avoids executing it again — no matter how many times the process dies between attempting it and recording it.</p><p>A pod plus a database gives you the first. To get the second, something other than your application code has to own the instruction pointer. That something is a runtime — a real one, not a library you import from inside a loop you wrote yourself.</p><h3><strong>Your demo worked because it was twelve steps long</strong></h3><p><strong>A 200-step agent run succeeds 13% of the time.</strong> Not because the model is wrong. Because of math.</p><p><em>The exact number depends on independence, retries, and step shape. The point is not the precise percentage. The point is that long runs punish naïve control flow.</em></p><p>Suppose every step succeeds with 99% reliability — optimistic, since it assumes your model never times out, your tools never rate-limit, and your network never blips. A 20-step workflow succeeds 81% of the time. A 50-step workflow, 60%. A 100-step workflow, 36%. A 200-step workflow — what a serious multi-agent task looks like once you count sub-agent calls — succeeds 13% of the time.</p><p>Seven out of eight runs fail. Because you stacked enough 99%s on top of each other that the floor fell out from under them. The only tool the loop has to respond is to start over from the beginning, which doubles the token bill and reintroduces the same probability distribution on the retry.</p><h3>The LLM should not execute anything</h3><p>The fix is the strictest possible separation between two planes.</p><p><strong>The LLM plans.</strong> It is a pure function from state to intent, no file handles, no network, no ability to mutate anything. It looks at the world and emits a description of what it would like to happen next. That description is data, not action. The LLM cannot make Stripe calls because the LLM cannot make any calls at all (not that the LLM cannot, but it <em>must not</em>).</p><p><strong>The runtime executes.</strong> It receives intents from the planning plane, writes them to a persistent ledger <em>before</em> doing anything with them, performs the side effect against the real world, writes the result back, and only then asks the LLM what to do next. The ledger — not process memory — is the source of truth.</p><p>When an agent needs to delegate to another agent, it doesn’t call a function. It yields an intent and returns to the pool. The runtime catches the intent, dispatches the work, and wakes the LLM back up only when there’s a result to look at.</p><blockquote><strong>The LLM proposes; the runtime disposes.</strong></blockquote><p><strong>An agent runtime has to provide at least four things a loop cannot:</strong></p><h4><strong>a. Intent ledger</strong></h4><p>Records every proposed action before it executes, so “did this happen?” is answerable from outside the agent’s process.</p><h4><strong>b. Effectively-once execution</strong> of side effects</h4><p>A crash between proposing an action and recording its result never causes the action to run twice.</p><h4><strong>c. Suspension and resumption across process boundaries</strong></h4><p>An agent waiting on a human, a webhook, or another agent consumes zero compute and survives infrastructure churn.</p><h4><strong>d. Out-of-band signal delivery</strong></h4><p>Signaling a running or suspended agent, so a supervisor, a human, or another agent can inject new context without killing the workflow.</p><p>A Kubernetes pod gives you none of these. A Postgres checkpoint gives you a weak version of the first one and nothing else.</p><p>A durable-execution runtime such as Conductor, Temporal, and the rest of the lineage gives you all four. If you’re building a long-lived workflow today, one of these is almost certainly the right substrate. The question this post is about is what changes when the thing producing the steps is not a programmer at commit time but a language model at runtime. That is not a critique of durable execution. It is an argument that durable execution is the right <em>substrate</em> for agents, and that agents need a layer on top of it that the existing frameworks weren’t designed to provide.</p><p>The interesting question is whether you can have all four properties <em>and</em> keep the LLM’s freedom to decide, at runtime, what the workflow even looks like. The answer is the <strong>Late-Bound Saga</strong>.</p><h3>The Late-Bound Saga: a workflow the LLM writes one edge at a time</h3><p>A Late-Bound Saga is a workflow whose execution graph does not exist when the workflow starts. The LLM synthesizes it edge by edge at runtime, and each edge is committed to a durable ledger at the moment it is proposed — <em>before</em> it executes. The graph is append-only. The LLM doesn’t traverse a pre-built graph; it extends the graph by one node at a time, with the runtime as witness and executor.</p><p>The order matters. Not: <em>LLM decides → tool executes → write down what happened.</em> Instead: <em>LLM decides → write the intent → tool executes → write the result → LLM sees the result.</em> Four ledger writes per step, bracketing the side effect on both sides. The bracket is the entire game, because it is the only structure that lets a recovering process answer “did this already happen?” without asking the outside world.</p><p>The <em>late-bound</em> half of the name gives you the dynamism — the LLM decides the workflow as it goes. The <em>saga</em> half gives you the durability — every decision is a transaction with a compensation, written down before it runs. <strong>You don’t have to choose between the two. The whole point of the pattern is that you don’t.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xdel0GVjzZTsAAD6g9TF5g.png" /></figure><h3>What it looks like in motion</h3><p>The agent’s task: <em>find the three best candidates for the senior backend role, and schedule first-round interviews with each.</em></p><p><strong>Step 1 — the LLM proposes a search.</strong> The runtime asks the LLM for an intent. It proposes:</p><p><em>search_candidates(role=”senior backend”, limit=20).</em></p><p>The runtime writes the intent, executes the search in a worker, writes the twenty results back. One node in the ledger. One edge in the graph.</p><p><strong>Step 2 — the fan-out.</strong> The runtime asks again. The LLM proposes twenty parallel enrich_candidate calls. The runtime writes twenty pending intents, dispatches twenty workers — possibly in twenty languages on twenty machines — and waits for each to report back. The graph now has twenty-one nodes and twenty edges, none of which existed sixty seconds ago.</p><p><strong>Step 3 — the saga goes to sleep for three days.</strong> The LLM ranks, selects three, and proposes three request_interview_slot calls. The runtime dispatches the emails, writes the three pending nodes, and <strong>suspends the entire saga to disk.</strong> The agent’s process exits. There is nothing running. No pod sitting idle burning compute. The saga is a row in a database and a handful of pending webhook subscriptions.</p><p>Three days later, a recruiter clicks “accept.”</p><p>The webhook fires.</p><p>The runtime looks up the saga, writes the result, and asks the LLM for the next intent.</p><p>The LLM, which has no idea three days have passed because it has no concept of time, proposes a calendar invite. The saga continues. To the LLM, no time has passed. To your infrastructure bill, no compute was consumed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*23M6az2xavpF1YLZ07ZAlQ.png" /></figure><h3>Signals are cache invalidation for intent</h3><p><strong>Now suppose on day two, a supervisor agent tells the saga to abort.</strong> In the loop model, there is no agent to tell — the process exited days ago.</p><p>In the Late-Bound Saga model, the runtime <em>is</em> the inbox. You send a signal to the saga’s ID. The runtime writes the signal, wakes the saga, and asks the LLM what to do. The LLM sees the cancellation, proposes a compensation sequence — 1) withdraw the pending requests, 2) send polite declines, 3) terminate — and the runtime executes each step.</p><p>Think of a signal as cache invalidation for intent. The agent’s plan is a cache. The world is the source of truth. When the world changes, the cache has to be invalidated and rebuilt. In a volatile loop you can’t, because the cache <em>is</em> the process and the process is blocked on a <em>requests.get</em>. In a Late-Bound Saga, the cache is the ledger, the invalidation is a ledger write, and the rebuild is the next call to the LLM.</p><p>Same mechanism, every other out-of-band event: a rate limit, a hallucination flagged by an evaluator, a budget cap, a human typing “stop, use staging instead.”</p><p>None of these are tool calls. All of them are signals, and a runtime that doesn’t have a signal primitive isn’t a runtime — it’s a job queue with extra steps.</p><h3>The runtime model, built into Agentspan</h3><p><a href="https://github.com/agentspan-ai/agentspan">Agentspan</a> sits on top of Conductor, the durable-execution runtime I open-sourced at Netflix in 2016. Conductor treats workers as protocol clients rather than SDK-bound processes, which means your planner can run in Python where the LLM ecosystem lives, your high-throughput data tool can run in Go, your legacy underwriting model can stay in the JVM where it has lived for fifteen years, and none of them have to share a process or a language. <a href="https://github.com/agentspan-ai/agentspan">Agentspan</a> is the layer that turns that substrate into an agent runtime.</p><p>Here is the candidate-search saga, in Agentspan, in the smallest form I can write it:</p><pre>from agentspan import agent, tool<br><br>@tool<br>def search_candidates(role: str, limit: int): ...<br><br>@tool<br>def enrich_candidate(candidate_id: str): ...<br><br>@tool<br>def req_interview_slot(candidate_id: str, recruiter_id: str): ...<br><br>@agent(model=&quot;claude-sonnet-4-6&quot;)<br>def hire_for_role(role: str, recruiter_id: str):<br>    &quot;&quot;&quot;<br>    Find the three best candidates for the given role and<br>    schedule first-round interviews with each of them.<br>    &quot;&quot;&quot;</pre><p>That is the user-level program. The rest of the machinery lives where it belongs: in the runtime. No loop. No state machine. No explicit call to the LLM. When you invoke <em>hire_for_role(“senior backend”, “rec_42”)</em>, the runtime starts a saga, calls the LLM with the task and the available tools, writes the proposed intent to the ledger, dispatches the worker, writes the result back, and calls the LLM again. If the process dies, the saga survives. If it waits three days for a recruiter, the runtime suspends it to disk and the wait costs nothing. If a supervisor sends cancel(saga_id), the runtime delivers it as a signal and the LLM gets a chance to compensate.</p><blockquote><strong>None of these are features bolted on top of the decorator. They are what the decorator means.</strong></blockquote><p>And because every intent and every result is in the ledger, you also get the thing every senior engineer who has ever debugged a non-deterministic system at 3am actually wants: <strong>distributed tracing for cognition.</strong> Point-in-time state inspection. Deterministic replay from any node. The hardest question in debugging an agent — <em>what did it know, and when did it know it? — </em>has an answer, and the answer is a database query.</p><h3>The Monday morning experiment</h3><p>Open the agent you’re building at work. Find the loop. Every agent has one. Answer three questions, on paper, honestly.</p><p><strong>One.</strong> If the process dies in the middle of the third iteration, kill -9, no warning — what happens to any side effect that iteration had already started? Not what <em>should</em> happen. What actually happens.</p><p><strong>Two.</strong> If a human or another agent needs to tell this loop to stop and reconsider — “send a message from outside” — how does the message get in? If the answer is “it can’t,” you don’t have signals at all.</p><p><strong>Three.</strong> Six months from now, your agent does something weird at 3:14am on a Tuesday. Where do you go to find out exactly what it saw and exactly what it decided? If the answer is “the logs,” check whether your logs contain the LLM’s full input and output for every step in a form you can replay against a different model. They don’t.</p><p>If any of the three made you flinch, the flinch is the thing this post is about. The flinch is the gap between durability and continuity, between a loop and a saga, between an LLM that executes and an LLM that proposes. The runtime is the thing on the other side of the flinch.</p><p>An agent is not an LLM in a loop. An agent is a saga the model writes as it runs. The loop was a hack we built while we were waiting to figure out what agents actually were. We’ve figured it out. It’s time to write the runtime.</p><p><em>Code: </em><a href="https://github.com/agentspan-ai/agentspan"><em>github.com/agentspan-ai/agentspan</em></a><em>.</em></p><p><em>Agentspan is open-source and is looking for builders and hackers. Send your PR!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8c50731c551" width="1" height="1" alt=""><hr><p><a href="https://medium.com/agentspan/late-bound-sagas-why-your-agent-is-not-an-llm-in-a-loop-a8c50731c551">Late-Bound Sagas: Why Your Agent Is Not an LLM in a Loop</a> was originally published in <a href="https://medium.com/agentspan">Agentspan</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Workflows as a Distributed Transactional Backend]]></title>
            <link>https://medium.com/orkes/workflows-as-a-distributed-transactional-backend-f25d6565c3df?source=rss-dc3485d163fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/f25d6565c3df</guid>
            <category><![CDATA[app-development]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[workflow]]></category>
            <dc:creator><![CDATA[Viren Baraiya]]></dc:creator>
            <pubDate>Wed, 26 Apr 2023 17:34:20 GMT</pubDate>
            <atom:updated>2024-01-04T11:04:32.669Z</atom:updated>
            <content:encoded><![CDATA[<p>In a recent <a href="https://a16z.com/2023/04/14/the-modern-transactional-stack/">blog</a>, a16z discussed the state of the modern transactional stack and called out Orkes Conductor as one of the important frameworks of the modern transactional backend . That article discusses the current state and landscape of application development platforms that provide transaction management capabilities out of the box. In this article, we look at the two main approaches — database oriented and workflow oriented transactional backends — and discuss the architecture of an application leveraging each of them and weigh the pros and cons of the two approaches.</p><h3>State of Modern App Development</h3><p>All modern apps are distributed systems. In modern applications, different components or services often need to work together to provide a seamless user experience. For example, a mobile app might need to communicate with a back-end service to retrieve data or process a user’s request. This requires the app and the service to exchange information over a network, which is a key characteristic of a distributed system.</p><p>Moreover, modern applications often rely on cloud computing and other infrastructure services, such as databases, message queues, and caching systems, which are distributed by nature. These services provide key features like scalability, fault tolerance, and high availability, all of which are essential for modern applications to meet the demands of a global audience.</p><p>The distributed nature of modern applications allows them to be more flexible, scalable, and resilient, which is why they have become the preferred architecture for modern day enterprise software development.</p><p>Compare this to a traditional enterprise application where all the requests from clients are executed within a database transaction and all of the data required to serve the need of the application resides in a single application specific database. This works great but creates a pattern of monolith that is harder to scale, change and difficult to maintain longer term. Also, there is a question of long running workflows which could span over days if not months or even years, where the question of maintaining application transactions still requires a solution that is not offered by databases.</p><p>This brings an important question — if you are adopting an application architecture that is running distributed services, how do you handle transactions that span across multiple services?</p><h3>Transactions, guarantees, and modern apps (<strong><em>are all transactions created equal?</em></strong>)</h3><p>Consider an order management system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3pUD8BZALkSeHyxLGrulKA@2x.png" /><figcaption>Order Management System</figcaption></figure><p>An application with multiple services, has two types of transactions:</p><ol><li><strong>Local to the service</strong>. The <em>Generate Order</em> step above creates a new order in the system and inserts details into the order table.</li><li><strong>Application level transaction</strong>: The transaction that controls the entire application flow. E.g. In the above example, we have a transaction that starts when the order is placed and completes when the flow terminates. Any persistent failures in between, have to be reversed in service specific databases.</li></ol><h3>Consistency in a distributed system</h3><p>We want the systems to be consistent in their state of the world. However, this is often harder to achieve in a distributed system given multiple actors and need to coordinate a single transaction across them that could be running for days.</p><p>In a traditional database oriented system, the database acts as a transaction coordinator providing the level of consistency that is offered by the database. However, that still does not solve the cases of long running and asynchronous flows where the transaction needs to be updated out of band and managed according to the business requirements.</p><h3>Workflow Engines as a distributed transactional backend</h3><p>Workflows in distributed event driven applications, allow services to communicate with each other via events, making it a source of truth for the application state. An orchestrator such as Orkes Conductor acts as a centralized coordinator that maintains the application state and handles the distributed transactions. This is often referred to as <strong>saga pattern</strong>.</p><p>The core idea of the Saga pattern is that long transactions are split into short transactions coordinated by a transaction coordinator such as Orkes Conductor. If each short transaction operation successfully completes, then the global transaction completes normally, and if a step fails, the compensating operations are invoked one at a time in reverse order.</p><p>This approach massively simplifies management of long running distributed transactions and the problem of managing application state across multiple services.</p><h3>Distributed systems and transactions in a long running process</h3><p>Managing long running transactions often require different strategies to handle the patterns of “rollback” or compensation. The rollback is typically done by reversing the local transactions via compensation. What gets rolled back and the associated compensation often depends on the current state of the transaction.</p><p>For example, in our order management system, if the delivery fails, the compensation logic could be to issue a refund on the credit card, send an email to the user notifying the situation and updating the status of the order in the database.</p><p>Failure to process credit card transactions could mean canceling the order and sending email to the user.</p><p>With a platform like Orkes Conductor this is handled via compensation workflows that are triggered when a workflow fails due to an error in the system or a business case validation failure (insufficient inventory). Workflow engines serve two purposes here:</p><ol><li>A workflow is the source of truth for the application state. Workflow systems can be queried to gain the status of the transaction at any point and it would act as an aggregator across multiple services and their local backend. In the order management example, we could ask the system the status of an order ID and it will tell us exactly what the order status is and which service it is with.</li><li>A workflow acts as a distributed transaction coordinator. If a particular step has to fail in the process the system takes care of running compensation logic as specified in the blueprint.</li></ol><h3>Application state vs Business State</h3><p>A typical application will have two kinds of states:</p><ol><li><strong>Application state:</strong> This defines the state of the application flow as defined by the program to achieve a specific goal — e.g. process order.</li><li><strong>Business state:</strong> Typically manages the state of the business entity — e.g. the delivery status of an order. You could have <strong>multiple</strong> application flows that are updating or reading the business state.</li></ol><p>These two states are often decoupled. The business state is often updated by an application flow as a step in its workflow as part of the local transaction.</p><h3>Database Centric Transactional Stack</h3><p>One approach is to use a database centric stack that allows you to manage application and business state from a database transaction. This approach has its merits in the simplicity as you can write your entire application with the database managing the overall state.</p><h4>Where does this approach start to break down?</h4><p>While simple, it breaks down as the application logic starts to get more complex. However, there are several downsides to this approach:</p><ul><li>Most of the platform today focuses on typescript/javascript as the language for writing the logic, which limits the libraries ecosystem you can use.</li><li>This architecture lands very well with UI driven applications and simple CRUD applications, however for application integrations and long running processes you typically have to integrate with other systems.</li><li>Handling of long running transactions still requires implementing strategies to handle compensation and application specific logic to rollback transactions.</li><li>Harder to implement many of the enterprise integration patterns with external systems</li><li>Single source of truth ends up creating a monolithic dependency on a single database system making it harder to scale with business complexity. e.g. A single DB to manage a company’s core order management system may not be an ideal solution for enterprises.</li><li>Managing entire business flow in the code makes it harder to visualize the progress and complicates the debug flow by increasing mean time to detect and repair when there are issues in the system.</li></ul><h3>Orkes Conductor + Business State Manager</h3><p>The architecture where the workflow engine works alongside business state management workers is not new, we leveraged this architecture at Netflix successfully across multiple applications where Conductor maintained the application state and dedicated workers managed the business state in a database. (However, that part was not open sourced).</p><p>In this architecture, a set of dedicated workers responsible for maintaining business state are “listening” on the application workflow state and creating the necessary database transactions to reflect the business state. e.g. when the workflow starts, the <strong>onStart</strong> event inserts a new order with the order id in the database and when the workflow completes, the onComplete event updates the status. In this case, the status listeners are the source of writes to the database table, which anyone can read outside of the workflow.</p><h4>Reasons why workflow + business state manager works better:</h4><p>Though it might seem complex at first, the approach scales well with business complexity and growth.</p><ul><li>Greater choice of development ecosystem with polyglot languages/framework choices</li><li>Decoupling of business logic and state management from the database and how/where the business state is stored.</li><li>Being able to scale components independently and ability to create greater re-usability of the services / components with domain ownerships. For example, a platform team could own and scale customer contact services (email, SMS) independent of the use cases they are used with (order management, marketing etc.)</li><li>Databases that handle business state can be multi-model ranging from traditional relational databases to NoSQL and specialized databases such as graph or vector databases for ML workloads.</li></ul><h3>Conclusion</h3><p>Combining the semantics of a database centric approach into the workflow, creates a powerful combination that allows building very complex and scalable distributed applications with ease. We believe that this approach provides the best of both worlds and scales much better in terms of handling business complexity as well as application usage growth.</p><p>Check us out at <a href="http://orkes.io/">orkes.io</a> and the open source Conductor at <a href="https://github.com/conductor-oss/conductor">https://github.com/conductor-oss/conductor.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f25d6565c3df" width="1" height="1" alt=""><hr><p><a href="https://medium.com/orkes/workflows-as-a-distributed-transactional-backend-f25d6565c3df">Workflows as a Distributed Transactional Backend</a> was originally published in <a href="https://medium.com/orkes">Orkes Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dynamic Workflows using Code in Netflix Conductor]]></title>
            <link>https://medium.com/orkes/dynamic-workflows-using-code-in-netflix-conductor-579498b56ab8?source=rss-dc3485d163fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/579498b56ab8</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[workflow-as-code]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[workflow-engine]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <dc:creator><![CDATA[Viren Baraiya]]></dc:creator>
            <pubDate>Fri, 03 Mar 2023 09:50:40 GMT</pubDate>
            <atom:updated>2024-11-25T12:45:35.042Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://github.com/conductor-oss/conductor">Conductor</a> is a popular platform for building resilient stateful applications by creating workflows that span across services. You can try out the workflows from this article at <a href="https://play.orkes.io/">Playground</a>, a free hosted version of Conductor.</p><figure><img alt="Dynamic workflows as Code in Conductor" src="https://cdn-images-1.medium.com/max/1024/0*kfEnQcPknK0_6MCu.jpg" /></figure><h3>What is a Workflow?<a href="https://orkes.io/content/blog/dynamic-workflows-using-code-in-netflix-conductor#what-is-a-workflow">​</a></h3><p>Before diving into how to create workflows using Conductor, let’s first define what a workflow is. In simple terms, a workflow is a series of tasks or steps executed in a specific order to accomplish a goal. Workflows are used to automate complex processes and ensure that all the necessary steps are completed in a logical sequence. Conductor workflows are composed of Tasks and Operators.</p><blockquote>Workflow = {Tasks + Operators}</blockquote><p>Tasks are services encapsulating the business logic that runs outside the Conductor server and are implemented as a Microservice, Lambda, or Worker. Workers run outside the Conductor server and can be implemented in any supported language. A workflow can contain multiple workers written in different languages.</p><p>Operators are primitives from programming languages that are used to control the flow of execution inside a workflow. Conductor supports operators such as switch, loop, fork/join, and sub-workflows, allowing you to define complex workflows.</p><h3>Separation of Workflows from Re-usable Services</h3><p>Conductor promotes clear separation between application workflow and services that are used as building blocks of the workflow as ‘Tasks’. This separation ensures that the tasks follow the single responsibility principle and are generally stateless in nature.</p><p>This model ensures two things:</p><ol><li>Re-usability of services across many workflows promoting greater collaboration across teams.</li><li>Stateless Task Workers can quickly scale up/down based on the volume while maintaining the state only at the Conductor server.</li></ol><h3>Workflow definition — JSON or Code?</h3><p><em>Why not both?</em></p><p>Conductor server stores the workflow definitions as JSON on the server side. However, this does not restrict users from expressing their workflows as JSON alone. Conductor supports creating workflows using Code and executing both pre-registered as well as dynamic workflows expressed using code.</p><h4>Simple Example<a href="https://orkes.io/content/blog/dynamic-workflows-using-code-in-netflix-conductor#simple-example">​</a></h4><p>Let’s take an example of a simple two-task workflow:</p><figure><img alt="Two-Task workflow in Conductor" src="https://cdn-images-1.medium.com/max/726/0*rjtRw64S-0CetJUK.png" /></figure><h4>JSON</h4><pre>{<br>  &quot;name&quot;: &quot;simple_two_task_workflow&quot;,<br>  &quot;version&quot;: 1,<br>  &quot;schemaVersion&quot;: 2,<br>  &quot;tasks&quot;: [<br>    {<br>      &quot;name&quot;: &quot;task1&quot;,<br>      &quot;taskReferenceName&quot;: &quot;task1&quot;,<br>      &quot;inputParameters&quot;: {},<br>      &quot;type&quot;: &quot;SIMPLE&quot;<br>    },<br>    {<br>      &quot;name&quot;: &quot;task2&quot;,<br>      &quot;taskReferenceName&quot;: &quot;task2&quot;,<br>      &quot;inputParameters&quot;: {},<br>      &quot;type&quot;: &quot;SIMPLE&quot;<br>    }<br>  ]<br>}</pre><p>The same workflow in different languages:</p><h3>Java</h3><pre><br>ConductorWorkflow workflow = new ConductorWorkflow(workflowExecutor);<br>workflow.setName(&quot;simple_two_task_workflow&quot;);<br>workflow.setVersion(1);<br><br>//Add tasks<br>workflow.add(new SimpleTask(&quot;task1&quot;, &quot;task1&quot;));<br>workflow.add(new SimpleTask(&quot;task2&quot;, &quot;task2&quot;));<br><br>//Execute<br>workflow.executeDynamic(//input);</pre><h3>Go</h3><pre>conductorWorkflow := workflow.NewConductorWorkflow(executor.NewWorkflowExecutor(client.NewAPIClient(nil, nil)))<br>conductorWorkflow.Name(&quot;simple_two_task_workflow&quot;).Version(1)<br><br>//Add Tasks<br>conductorWorkflow.<br>  Add(workflow.NewSimpleTask(&quot;task1&quot;, &quot;task1&quot;)).<br>  Add(workflow.NewSimpleTask(&quot;task2&quot;, &quot;task2&quot;))<br><br>//Execute<br>conductorWorkflow.StartWorkflow(//input)</pre><h3>Python</h3><pre>workflow = ConductorWorkflow(<br>  executor=workflow_executor,<br>  name=simple_two_task_workflow,<br>  version=1,<br>)<br><br>#Two Tasks<br>task1 = SimpleTask(&#39;task1&#39;, &#39;task1&#39;)<br>task2 = SimpleTask(&#39;task2&#39;, &#39;task2&#39;)<br><br>#Add tasks to the workflow using &gt;&gt; operator<br>workflow = workflow &gt;&gt; task1 &gt;&gt; task2<br><br># Execute the workflow<br>workflow.start_workflow(#input)</pre><p>Creating workflows using code opens up use cases where it might be impossible to define workflows using static definitions — this could be when the number of tasks and their flow depends on the data that is dependent on the other factors.</p><h3>Complex Example</h3><p>Let’s take an example of a hypothetical workflow that is created dynamically based on the user data:</p><pre>ConductorWorkflow workflow = new ConductorWorkflow(workflowExecutor);<br>workflow.setName(&quot;complex_dynamic_workflow&quot;);<br>workflow.setVersion(1);<br><br>//Get the list of users to send notification to<br>List&lt;UserInfo&gt; users = getUsers();<br>Task&lt;?&gt;[] tasks = new Task[users.size()];<br><br>int counter = 0;<br>for (UserInfo user : users) {<br>  if(user.sendEmail) {<br>    SimpleTask task = new SimpleTask(&quot;send_email&quot;, &quot;send_email_&quot; + counter);<br>    task.input(&quot;email&quot;, user.email);<br>    tasks[counter++] = task;<br>  } else {<br>    SimpleTask task = new SimpleTask(&quot;send_sms&quot;, &quot;send_sms_&quot; + counter);<br>    task.input(&quot;phone&quot;, user.phone);<br>  tasks[counter++] = task;<br>  }<br>}<br><br>//Run all the tasks in parallel<br>workflow.add(new ForkJoin(&quot;run_in_parallel&quot;, tasks));<br><br>//Execute workflow and get the future to wait for completion.<br>CompletableFuture executionFuture = workflow.execute(new HashMap&lt;&gt;());<br><br>//Alternatively, kick off the workflow if it&#39;s going to be a long-running workflow<br>String workflowId = workflowClient.startWorkflow(new StartWorkflowRequest().withWorkflowDef(workflow.toWorkflowDef()));</pre><p>In the above example, we load up the user list from a backend store, and for each user, create a task to send sms or email. A workflow is created by adding appropriate notification tasks for each user here and is then executed.</p><p>Depending on how long the workflow takes to complete, Conductor provides a way to wait for the workflow completion using ‘Futures’ or kick off a workflow that returns the workflow execution id that can be used by workflowClient to monitor the execution. (or can be searched and viewed in the UI)</p><p>You can write workflows using code in Java, Golang, Python, CSharp, Javascript, and even Clojure with Conductor.</p><p>Visit <a href="https://github.com/conductor-sdk">Conductor SDK on GitHub</a> for the latest SDKs with fully working example apps.</p><h3>Dynamic Workflows and Conductor</h3><p>The ability to dynamically create workflows using code allows developers to address very complex use cases where it is impossible to pre-define workflows. With Conductor, you can still do this, with the full power of Conductor visualization that allows you to visualize the entire execution in the UI. It’s like having your cake and eating it too!</p><h3>Summary<a href="https://orkes.io/content/blog/dynamic-workflows-using-code-in-netflix-conductor#summary">​</a></h3><p>Netflix Conductor is a powerful platform that lets you create the most complex workflows while making it very easy to handle runtime scenarios with powerful debugging and visualization tools, reducing the mean time to detect and resolve issues in the production environment.</p><p>Be sure to check out <a href="https://github.com/conductor-oss/conductor">Conductor</a> on GitHub. Our Orkes team do provide <a href="https://play.orkes.io/">Conductor Playground — a free version of Conductor</a>.</p><p><a href="https://join.slack.com/t/orkes-conductor/shared_invite/zt-2vdbx239s-Eacdyqya9giNLHfrCavfaA">Join our Slack community</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=579498b56ab8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/orkes/dynamic-workflows-using-code-in-netflix-conductor-579498b56ab8">Dynamic Workflows using Code in Netflix Conductor</a> was originally published in <a href="https://medium.com/orkes">Orkes Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running a Billion Workflows a month with Conductor]]></title>
            <link>https://medium.com/orkes/running-a-billion-workflows-a-month-with-netflix-conductor-5fd358d9b6c1?source=rss-dc3485d163fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fd358d9b6c1</guid>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[high-performance]]></category>
            <category><![CDATA[microservices]]></category>
            <category><![CDATA[workflow]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Viren Baraiya]]></dc:creator>
            <pubDate>Tue, 20 Sep 2022 21:38:19 GMT</pubDate>
            <atom:updated>2026-03-14T08:30:38.596Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*dqJtSrtrZ2SzcwvbAGiPiA.png" /></figure><p><a href="https://github.com/conductor-oss/">Conductor</a> is a well known platform for service orchestration allowing developers to build stateful applications in the cloud without having to worry about resiliency, fault tolerance and scale. Conductor allows building stateful applications — automation as well as human actor driven flows by composing services that are mostly stateless.</p><p>At <a href="https://orkes.io/">Orkes</a>, we regularly scale Conductor deployments to scale to handle millions of workflows per day and we often get questions from the community on how to scale the Conductor and if the system can be scaled to handle billions of workflows</p><h3>TL;DR</h3><p>We ran a benchmark to test the limits of Conductor’s scalability using a simple single node setup on a laptop. The results were not so surprising, given our experience with Conductor, — with the right configuration, Conductor can be scaled to handle the demands of really large workloads.</p><p><strong>Throughput</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ea1VdAegEwXftJlkFDkH4Q.png" /></figure><p><strong>Latency</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cyzkTSQ0T_ghWHZWpv4wRg.png" /></figure><h3>Benchmarks</h3><p><strong>Throughput</strong></p><p>We measured the peak stable throughput Conductor can achieve on the following parameters:</p><ul><li>Number of workflow start request per second</li><li>Number of task updates per second — translates to no. of state transitions happening per second</li><li>Number of workflows completing per second with moderate complexity</li><li>Amount of data processed per second</li></ul><p><strong>Latency</strong></p><p>We measured <em>p50</em>, <em>p95</em> and <em>p99</em> latencies for the critical APIs in Conductor:</p><ul><li>Workflow Start</li><li>Task Update</li></ul><h3>Benchmarking Tools</h3><p>We used <a href="https://github.com/giltene/wrk2">wrk2</a>, a fantastic tool to generate stable load on the server. Wrk2 improves on <a href="https://github.com/wg/wrk">wrk</a> and adds the ability to generate sustained load at a specific rate (-R parameter).</p><p>We created a load testing workflow which is complex enough with a total of 13 steps.</p><p>For the experiment a set of load testing workers were created that were used to poll for the tasks in the workflow, and produce a dummy output.</p><h4>Monitoring</h4><p>Conductor has a pluggable metrics system and we used Prometheus to capture and Grafana to visualize various metrics.</p><h3>The Setup</h3><p>The experiment was designed to send a sustained workload of 200+ workflow executions / sec. The test workflow also embedded a sub-workflow with a single step inside, so during the experiment, we were starting approximately 400+ workflow/sec.</p><p>For the Conductor version, we used Orkes’ build of Conductor that is tuned and customized to handle large workflows with minimal latency possible and optimize the operational costs.</p><blockquote>Get Conductor releases : <a href="https://github.com/conductor-oss/conductor">https://github.com/conductor-oss/conductor</a></blockquote><p>Conductor servers are a <em>horizontally scalable system</em>, for this test we ran a single node server — of course your production setup should have at-least 3 nodes in multiple availability zones / racks for higher availability.</p><h4>Hardware</h4><p>The Conductor server was running on a Macbook Pro with M1 Max CPU and 64GB of RAM.</p><p>The same machine also ran a single node redis server.</p><p>Postgres database was running on another Macbook Pro with M1 Max CPU and 32GB of RAM. The same machine also ran task workers.</p><p>A Core i9 Macbook pro with 32 GB RAM ran Prometheus, Grafana, a load generator docker container and task workers.</p><p>The systems communicated over WiFi 6 network — no wired connectivity, which potentially could have improved latencies a bit. The machines were roughly equivalent to a <strong>c7g.2xlarge </strong>instance type of AWS (memory consumption in the tests was below 16GB offered by this instance type).</p><h3>Peak Workflow Start Requests</h3><p>Testing the limits of how many workflows can be started in a burst. For this experiment we wanted to remove any network latencies so we ran the <em>wrk</em> on the same host as the Conductor server running thereby removing any network latencies out of equation. This gives us a theoretical max request/sec assuming the network is limitless (which it isn’t’).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*U1o122s7MZI32Nr_" /><figcaption><em>Workflow Start Requests / sec. Under the normal load the no. of start requests averages at about 1.8K/sec</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*GIXqewAkvCRkCIpY" /><figcaption>Latencies under high load with ~2K request/sec</figcaption></figure><h3>Workflow Completion Metrics</h3><p>For this experiment, we asked a question:</p><blockquote>If all<em> the tasks in a workflow were instantaneous, how many workflows can we start and complete in a second with a sustained load such that there is minimal backlog of the tasks.</em></blockquote><p>To test this, we used <em>wrk2</em> to send a sustained load of 210 workflow execution requests, where each workflow contains a sub workflow and worker tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/0*-7ImvhIj4aXru8av" /><figcaption>Workflow Execution Graph</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*B38z48V4w4FwUEWu" /><figcaption><em>Workflows getting completed per second</em></figcaption></figure><h3>Task Level Metrics</h3><p>Conductor publishes the depth of the pending task queue, this is useful when deciding when to scale up/down your worker clusters. Here is the snapshot of the worker queue depth. High numbers for a given task indicates the worker resource starvation and need to scale the worker cluster to fulfill the demand.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*mrZIon0vNRdid9Fg" /><figcaption><em>Pending queue size of tasks at a given point in time. Sustained high numbers indicate worker starvation and a need to scale out workers.</em></figcaption></figure><p>We achieved a consistent throughput of <strong>1450+ task executions / sec</strong>. This included polling for the task, executing the business logic (in our case producing randomized test output) and updating the task status on the server by workers. Each successful task completion initiates the state transition of the workflow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*SFuxrO7uwhUpPIdB" /><figcaption><em>Number of worker tasks getting updated per second.</em></figcaption></figure><p>Critical to the throughput is the update of the task execution back to the Conductor server. This is <strong>the most</strong> critical API operation we found that is directly responsible for the throughput of the server. We optimized the server to limit the tail latencies ensuring p99 stays well under the check.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*a17xwwp1grTkCkPL" /><figcaption><em>Task update from worker latencies in milliseconds.</em></figcaption></figure><h4>Task Poll</h4><p>Conductor uses a long-poll for the task polling, and the request waits until there is a task available for the worker or the timeout, with the timeout set to 100 ms in this experiment. Polling implements batching for more efficient use of the network and connection. In the test, the batch size was set to 10 for the tasks by workers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*sZE_bTatVu5IZA2G" /><figcaption><em>Latencies for task poll</em></figcaption></figure><h3>Data Processed by Workflow</h3><p>We generated fake data for the test to simulate the real world scenarios in terms of data transfer. The amount of data being processed by workflows impacts the requirements for provisioned IOPS (on cloud environments) and Network throughput requirements.</p><p>The experiment averaged a sustained rate of ~80 MB/sec data processed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*PxtvJK8CYU7gtIw6" /><figcaption><em>Amount of data (task inputs and outputs) being processed at a given point in time.</em></figcaption></figure><h3>Scaling to handle billions of workflows / month</h3><p>The experiment used a total of 3 commodity hardware (roughly equivalent to <strong>c7g.2xlarge </strong>instance types on AWS). Throughout the experiment JVM heap size consistently remained below the 2GB mark.</p><p>The experiment created a workload of 210 moderately complex workflows per second, which if run constantly for a month will generate about 540M workflows.</p><h4>Conductor Servers</h4><p>Conductor servers themselves are stateless and can be scaled out to handle larger workload demands. Each of the server nodes can handle the workload based on the available CPU and Network and can be scaled out to handle larger workloads.</p><p><strong>Redis</strong></p><p>Redis serves as the backbone for state management and queues used by servers to communicate. Scaling higher workload requires either scaling up redis or using redis cluster to better distribute workload.</p><p><strong>Postgres</strong></p><p>Postgres is used for indexing of workflow data. Beyond the disk storage requirements, scaling postgres requires two factors <em>1) adequate CPU and 2) IOPS required </em>(especially on cloud environment) to handle the writes under heavy workloads. Writes to postgres are asynchronous done using durable queues (check out <a href="https://github.com/orkes-io/orkes-queues">orkes-queues</a>) but longer delays means completed workflows remain in Redis for longer periods of time, requiring larger Redis memory.</p><h3>Conclusion</h3><p>We often get asked, can Conductor be scaled to handle <strong>billions</strong> of workflows per month? The answer is a resounding <strong>YES</strong> it can. If you would like to give it a try, check out our developer playground at <a href="https://play.orkes.io/">https://play.orkes.io/</a></p><p><a href="https://orkes.io/">Orkes,</a> founded by the founding engineers of Netflix Conductor, is a fully managed service offering Conductor as a hosted service in the cloud and on-prem. Checkout <a href="https://github.com/conductor-oss/conductor">Conductor OSS.</a></p><p>If you are an enterprise and have a use case that you would like to run a PoC, please <a href="https://www.orkes.io/talk-to-an-expert">reach out to us</a>.</p><p><em>Don’t forget to give us a</em> ⭐️<a href="https://github.com/conductor-oss/"> https://github.com/conductor-oss/conductor</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fd358d9b6c1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/orkes/running-a-billion-workflows-a-month-with-netflix-conductor-5fd358d9b6c1">Running a Billion Workflows a month with Conductor</a> was originally published in <a href="https://medium.com/orkes">Orkes Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>