You type a plain-English sentence — “build me a workflow that books a flight, a hotel, and a rental car” — and a moment later, Copilot has registered an executable multi-agent workflow in your session. Very cute. Very magical. Very “look, Mom, the agents planned my vacation.”

Except I don’t like magic in my runtimes. I wanted to know what was actually happening under the hood between my prompt and that running workflow. How does the model even know the workflow API? Is it baked into the model weights? Stuffed into a giant hidden system prompt? Loaded from a secret skill? Where does the generated code live on disk, and what actually executes it?

So I did what I always do when a feature feels like a black box: I started pulling threads. I traced the tool schemas through the CLI binary, inspected the session logs, and opened Procmon to watch which files the process actually touched on disk.

And of course, once I saw how the runtime builds and journals a workflow, the distributed-systems part of my brain (if you read my earlier post on Azure Durable Task Scheduler, you already know where this is going) couldn’t resist asking the next question: what happens if I kill the process right after the hotel is booked, before the rental car agent finishes?

Stick with me. What started as a peek behind the curtain turned into a great lesson in the difference between having a log of what happened and having a durable workflow you can actually resume.

Peeking Behind the Curtain: How Copilot Learns to Write a Workflow

When you ask Copilot to create a dynamic workflow, there’s no magic Markdown compiler turning your English into a state machine. It’s a very practical, three-step handshake between the native CLI runtime and the bundled Copilot SDK.

First, the native runtime exposes the workflow tools (dynamic_workflows_manage and run_dynamic_workflow) to the model through standard tool descriptions and schemas. If you trace the strings in the CLI (1.0.94-3 in my initial trace), they lead straight through the JavaScript bridge into the bundled prebuilds\win32-x64\runtime.node.

And right inside the tool description for dynamic_workflows_manage, the runtime leaves the model a breadcrumb: call operation: "guide" before authoring any workflow code.

So that’s step one. The model calls dynamic_workflows_manage({ operation: "guide" }), and the runtime hands back the local paths to the bundled SDK documentation:

%LOCALAPPDATA%\Programs\GitHub Copilot\copilot-sdk\docs\workflows.md
%LOCALAPPDATA%\Programs\GitHub Copilot\copilot-sdk\index.d.ts

(Depending on how you installed Copilot CLI, that path might live under %LOCALAPPDATA%\copilot\pkg\win32-x64\<version>\copilot-sdk\docs\workflows.md instead — which is why the guide operation returns the exact path for your active install rather than hardcoding it.)

Step two is refreshingly low-tech: the model simply calls view on that workflows.md path and reads the manual. Literally. The SDK workflow guide enters the context window as a normal tool result, teaching the model the exact shape of defineWorkflow, ctx.step, ctx.agent, and ctx.parallel.

Step three is turning that knowledge into a runnable extension. In the author flow, the model calls dynamic_workflows_manage({ operation: "author", meta, run }), passing the workflow metadata and the JavaScript run function body. The host validates it, wraps it in a small ES module scaffold under the session’s extensions\<name>\extension.mjs folder, drops an .authored-workflow marker next to it, and reloads the extension into the session:

import { defineWorkflow, joinSession } from "@github/copilot-sdk/extension";

const workflow = defineWorkflow({
    /* generated meta and run omitted */
});

await joinSession({ workflows: [workflow] });

Notice the clean division of labor: the model writes the meta declaration and the run body; the host writes the import and joinSession boilerplate; and @github/copilot-sdk/extension resolves to the bundled SDK inside the extension host. Creating and registering the workflow doesn’t run it — execution is a separate step via run_dynamic_workflow.

Watching It Happen Live in the CLI

Reading traces is one thing; watching it happen live is much more fun. I spun up a fresh Copilot CLI session (running 1.0.99-2 through cli-tunnel) and asked it to build a Seattle trip-planning workflow — register only, don’t execute yet.

Right on cue, the first thing Copilot did was ask permission to read the bundled workflows.md guide from disk:

In this newer CLI version (1.0.99-2), instead of the single-shot author call, Copilot used the extension lifecycle directly — guide → scaffold → edit → reload → inspect — and registered seattle-trip-planning with five ctx.step operations (three parallel simulated bookings, a review step, and a guarded confirmation) without executing it:

Because I like to verify what happened behind the curtain, I checked the creation turn from two angles. First, I queried the session’s recorded tool execution log to see the exact sequence of tool calls (guide returning the path → view reading workflows.md → scaffolding the extension → reading and patching extension.mjs):

And second, because nothing beats OS-level ground truth, I opened Sysinternals Process Monitor (Procmon) and watched the process touch copilot-sdk\docs\workflows.md on disk:

When you run that workflow on the happy path — say, a simulated trip from SFO to Seattle for November 10–14, 2026 — it flies through in one shot and prints a clean booking table:

What Copilot Actually Writes When You Don’t Coach It

Before we break things, it’s worth looking at the code Copilot generated when I gave it an open-ended prompt in a fresh desktop session:

Create a Copilot dynamic workflow that plans a trip: given a destination, book a flight, a hotel and a rental car, review booking results, then confirm. Create/register only, do not execute or make reservations.

I deliberately didn’t mention Durable Task Scheduler, idempotency keys, or replay boundaries. I wanted to see what “natural” Copilot workflow code looks like out of the box.

To its credit, it didn’t write a naive “call three things and print success” script. It set up three parallel booking agents, followed by a fourth review agent, followed by a guarded confirmation step:

const services = ["flight", "hotel", "rental car"];
const results = await ctx.parallel(services.map((service) => () => ctx.agent(
    `Book the ${service} for a trip to ${JSON.stringify(tripDestination)}. ` +
    "Use available booking tools, actual service responses, and only travel details, constraints, and authorization already supplied by the user. " +
    "If essential details, authorization, or booking tools are missing, return needs_input and explain what is needed. " +
    "Return booked only after the travel service confirms the reservation, with its actual confirmation number. " +
    "If the booking fails, return failed with the reason. Never invent a reservation or confirmation number.",
    { label: `Book ${service}`, schema: bookingSchema },
)));

It even added structured schema checks (booked, needs_input, failed), verified that every booking returned a non-empty confirmation number, and had the review agent validate consistency before allowing the final ctx.step("confirm-trip", ...) to confirm the itinerary:

if (!allBooked || !review.approved || review.issues.length > 0) {
    return { status: "not_confirmed", destination: tripDestination, bookings, review };
}

return await ctx.step("confirm-trip", () => ({
    status: "confirmed",
    destination: tripDestination,
    bookings,
    review,
}));

As application logic goes, that’s genuinely thoughtful code.

As a durable distributed workflow, though, look closely at what’s missing. There are no provider idempotency keys. There’s no external orchestration state mapping a logical trip to a resumable run. And wrapping the final return object in ctx.step("confirm-trip", ...) checkpoints the final summary — it doesn’t magically wrap the external tool calls happening inside those three parallel ctx.agent(...) calls in durable, idempotent boundaries.

Which brings us to the fun part: pulling the plug.

Pulling the Plug Between “Booked” and “Finished”

Let’s run the crash test. (And before anyone worries about my corporate Amex: all bookings in these tests hit local fake booking providers. The Copilot workflows and agents were real; the airlines were not.)

I ran that exact generated trip workflow on the default local executor (1.0.95-3) and engineered a very specific crash point:

1. Flight agent calls the flight tool, gets a simulated reservation, and finishes (flight = 1).
2. Hotel agent calls the hotel tool, gets a simulated reservation, and finishes (hotel = 1).
3. Car agent reaches the rental car tool — and I hold the request right before it creates the reservation (car = 0).
4. While the workflow is mid-flight, I kill the CLI, extension host, and provider processes.

Then I reopened the exact same Copilot session.

Here’s the part that surprised me at first. This wasn’t a “the process died and wiped its memory” failure. The local journal had saved the completed flight and hotel records. The fake provider receipts were sitting right there on disk. Copilot remembered the bookings.

So I asked it to resume the interrupted run. And the runtime came back with a flat refusal:

was interrupted before it settled durably and cannot be resumed

The run was marked error (reason: interrupted, canResume: false). Even worse, the old rental car agent row was still stuck reporting running in storage after the process had died, leaving a phantom live-agent count of 1.

How Copilot Actually Maintains Its Native Ledger (and Why It Still Wasn’t Enough)

To understand why Copilot remembered the bookings but still refused to continue, I looked at how the SDK and native CLI actually maintain workflow state under the hood (anchored in nodejs/src/session.ts and src\runtime\src\workflows\store.rs in the native runtime).

Copilot’s default local workflow runner is not stateless. When a workflow starts, the native runtime mints a runId and an active executionToken for that attempt, and every method on ctx talks back to the native session store over JSON-RPC:

• ctx.step(key, producer) is a read-through, write-through memoization cache backed by the native ledger (session.workflow.journal.get and session.workflow.journal.put).
• ctx.agent(prompt, options) calls session.workflow.agent, which spawns the sub-agent and records a full lifecycle row (agentId, phaseId, status, resolvedModel, startedAt, completedAt, activeMs, plus consumed sub-agent and token accounting on the run envelope).
• ctx.log and ctx.phase batch ordered progress lines through WorkflowProgressBuffer and flush them via session.workflow.log.

Look at the ctx.step implementation in nodejs/src/session.ts — it’s super clean:

const cached = await awaitWorkflowOperation(
    () =>
        self.rpc.workflow.journal.get({
            runId: params.runId,
            executionToken: params.executionToken,
            key,
        }),
    controller.signal
);
if (cached.hit) {
    return cached.resultJson;
}

const result = await producer();
await awaitWorkflowOperation(
    () =>
        self.rpc.workflow.journal.put({
            runId: params.runId,
            executionToken: params.executionToken,
            key,
            resultJson: result,
        }),
    controller.signal
);
return result;

So why couldn’t it resume if journal.put and workflow.agent had already written the completed outputs to disk?

Because in the native store, having journal entries on disk is not what makes a run resumable. A run is only marked canResume: true when the attempt settles cleanly at a controlled checkpoint (for example, pauseAtCheckpoint). When you kill the process mid-flight while an executionToken is still active, the runtime sees an ungracefully severed attempt. On rejoin, it marks the run envelope as status: "error", reason: "interrupted", and canResume: false — and any call to session.workflow.resume (resumeFromRunId) gets rejected before it ever touches the step journal.

To verify that directly against journal.get / journal.put, I reproduced the exact same crash boundary in the 1.0.99-2 CLI with a deterministic step-only version (seattle-trip-interruption-demo). All three parallel bookings and the review step finished and were written to the native step journal via session.workflow.journal.put, and the workflow paused at a file gate right before final confirmation (WAITING_FOR_OPERATOR_RELEASE):

I killed the CLI and its workers, reopened the same conversation, and called run_dynamic_workflow with resumeFromRunId for that run. Same result:

All four completed step outputs were still sitting in the database byte-for-byte intact. The ledger remembered the data, but the crashed attempt lost execution authority.

That’s the distinction that’s so easy to miss when you only test the happy path:

Remembering completed work and being able to resume a workflow are two completely different capabilities.

A ledger tells you what happened before the crash. A durable orchestrator knows how to replay that history, re-establish execution authority, and pick up at the exact next step without re-running completed work.

Wiring in Durable Task Scheduler (and Where the Native Boundary Pushes Back)

This is the exact problem space Azure Durable Task Scheduler (DTS) was built for. So I built an experimental fork of copilot-sdk to see what it takes to plug DTS into joinSession.

I added an optional WorkflowExecutor seam so an extension can bring its own durable execution engine while defaulting to normal local behavior if omitted:

export interface WorkflowExecutor {
    execute(
        definition: WorkflowDefinition,
        invocation: WorkflowInvocation,
        host: WorkflowExecutionHost,
        signal: AbortSignal
    ): Promise<{ result?: JsonValue }>;
}

Under the hood, the architecture splits cleanly into three pieces:

• The client schedules or attaches to a deterministic DTS orchestration instance.
• The worker pulls work items and runs the JavaScript workflow code.
• The scheduler (DTS) persists the orchestration history across process crashes.

When a worker restarts after a crash, the replay adapter (workflow-replay.ts) walks the workflow code against the recorded JSON history in DTS. Every Agent, Step, and Progress call becomes a tracked activity. If DTS already has a recorded completion for step 1, replay returns the cached JSON immediately without re-invoking the agent or step, and advances until it hits the first unseen operation.

How Recovery Actually Works: The Carrier Run Pattern

Here’s the catch, though: WorkflowExecutor lives in the JavaScript SDK host, while resumeFromRunId is gated by the native CLI runtime. Teaching the JavaScript layer about DTS doesn’t magically rewrite the native binary’s rule that an interrupted native run ID is non_resumable.

So how do you recover an interrupted DTS-backed workflow when the native CLI refuses resumeFromRunId?

You use a new native run as a carrier for the existing DTS orchestration instance.

I tested this end-to-end using the official changed-file code review workflow (review-changed.mjs) running against a local DTS emulator. The workflow reviews two files in sequence: a cart-total calculation (which has a bug: reduce missing its initial 0 accumulator) and a discount-price function (which accidentally subtracts a percentage instead of multiplying).

1. On Run 1, the first review agent finished analyzing the cart fixture (492 bytes of structured review output) and DTS checkpointed the completion.
2. Before the second review agent could finish, I killed the process.
3. After reopening, instead of asking the native CLI to resume the dead native run ID, I launched a fresh native carrier run configured via workflow-dts.ts to attach to the original DTS instance ID (verifying that the workflow scope, arguments, and code hash matched).

It worked like a charm.

The carrier run attached to the existing DTS orchestration, replayed past the first step by reusing the exact 492-byte cart review from DTS history, and spawned only one new agent to finish the discount-file review. Across the crash and recovery, only two workflow agents ever ran — one per file.

Is it seamless UI magic yet? Not quite. Because the carrier is a new native run, the CLI UI doesn’t automatically rehydrate the old Review phase progress bars from the dead run, and you still have to manage your logical run mapping and budgets across attempts (plus make sure your external tools carry idempotency keys if a worker dies right after an API call succeeds). Honest engineering trade-offs — but the core replay mechanics are rock solid.

Where This Leaves Us

I started this whole rabbit hole just wanting to know how Copilot turns a plain-English sentence into a runnable workflow.

What I love about looking under the hood here is how pragmatic the design actually is. There’s no mystery compiler — just a clean guide → view → extension.mjs loop that teaches the model the SDK on demand and registers normal JavaScript into the session. And the local runtime already does the work of journaling step outputs, agent runs, and progress lines as it goes.

Where things get interesting is the gap between keeping a ledger and recovering a live workflow. When a process dies mid-flight, having the previous step results sitting on disk isn’t enough if the runtime can’t safely re-establish execution authority and replay past completed work.

Plugging a WorkflowExecutor seam into copilot-sdk showed just how naturally Azure Durable Task Scheduler fits that gap: you keep the exact same defineWorkflow developer experience, let Copilot author the workflow the way it already wants to, and let

DTS handle the replay when reality inevitably pulls the plug.

Originally published at https://www.tamirdresher.com on October 15, 2026.

The thing that surprised me most is how small the core idea is. A graph, a ReAct agent and a Deep Agent are not three different machines. They are one machine (Pregel) fed three different amounts of configuration.

Everything below was run offline: no network, no live model, a scripted fake chat model. Versions are pinned to langgraph 1.2.12, langchain-core 1.6.6, langchain 1.4.3, deepagents 0.7.20; internals move between releases, so treat the class names as a snapshot. The code is in the samples repo: langgraph_primitives_demo.py produced every printed output in sections 1–6, and structured_cases.py covers section 5c.

Side notes before we start: Python pieces and C# equivalents

If you come from C#, a few Python/LangGraph words show up immediately. They’re plain Python or plain LangGraph.

You’ll see What it is Closest C# idea TypedDict A dictionary whose keys and value types are declared for type checkers; at run time it's an ordinary dict. A DTO / record, except it stays a dictionary. Annotated[list, operator.add] A type with attached metadata: "this is a list, and here's extra info." Python itself ignores the extra part; LangGraph reads it. A type plus an attribute, e.g. [Reducer(Add)] List<string> visited. operator.add A plain function that does a + b (for lists, concatenation). A method group such as (a, b) => a + b, passed as a Func<T,T,T>. lambda x: x + 1 An anonymous function. That's what I meant by "unnamed" before: not a keyword of this project, just Python's lambda. A C# lambda x => x + 1; a Func<int,int> delegate. a function passed as a value Functions are first-class objects, so add_node("a", a) just hands over the function. Passing a delegate or method group. invoke, stream Methods on a runnable. Execute / IAsyncEnumerable<T> on an interface.

Two LangGraph words are worth fixing in your head now.

State is the dictionary that flows through the graph. Every node receives it and returns a partial update to it, not a whole new state. If you know C#, think of an immutable record plus with { ... } expressions, applied by the engine for you.

Channel is one named slot of that state, and it owns the rule for how new values are merged in. A state key total becomes a channel called total. The rule matters when several nodes write in the same round: should the new value replace the old one, or be combined with it? In C# terms a channel is roughly a field paired with its own merge function, like Func<T, T, T> merge.

The name is misleading if you know C#. It is not System.Threading.Channels: nothing is queued for a consumer to await, and nothing blocks. It is not Rx's IObservable: nothing pushes values to subscribers. It is not a TPL Dataflow block either, though that is the closest family, because "a node runs when something it depends on was updated" is the same idea as dataflow triggering. The nearest literal picture is a versioned cell in a shared state object: a field whose Update(IEnumerable<T> writes) method decides how writes combine, plus a version number that goes up when it changes. Nodes are woken by comparing those versions, not by receiving messages. (Two exceptions behave a bit more like queues: Topic, used for Send, collects several values; EphemeralValue holds a value for one round only.)

Reducer is that merge function: reducer(old, new) -> combined. You attach one with Annotated. If you've used LINQ's Aggregate, it's the same shape: fold each new value into a running one. operator.add as a reducer on a list means "append the new items to the old items".

With those in hand, here is the smallest interesting graph.

1. The only contract: Runnable

A Runnable is anything with invoke, ainvoke, stream, astream, batch and with_config. That's it. In C# you'd call this an interface that everything implements, an IRunnable<TIn, TOut>. The | operator composes two of them (Python lets a class overload |, like a C# operator |):

chain = RunnableLambda(lambda x: x + 1) | RunnableLambda(lambda x: x * 10)
type(chain).__name__   # RunnableSequence
chain.invoke(1)        # 20

Why this matters: a compiled LangGraph is also a Runnable. Its MRO starts CompiledStateGraph → Pregel → PregelProtocol. (MRO is the method resolution order, Python's inheritance chain: CompiledStateGraph : Pregel : PregelProtocol.) So a whole graph can sit inside a chain, or inside a node of another graph. That is how subagents work later.

2. What compile() produces

Take the smallest interesting graph: two nodes with a state that has an overwrite field and an append field. A node is just a function that takes the state and returns a partial update:

import operator
from typing import Annotated, TypedDict
from langgraph.graph import StateGraph, START, END

class S(TypedDict):
    total: int                              # bare key: one write per superstep
    visited: Annotated[list, operator.add]  # reducer: appends

def a(s):                               # node "a": doubles n, logs "a"
    return {"total": s["total"] * 2, "visited": ["a"]}

def b(s):                               # node "b": adds 1 to n, logs "b"
    return {"total": s["total"] + 1, "visited": ["b"]}

g = StateGraph(S)
g.add_node("a", a)
g.add_node("b", b)
g.add_edge(START, "a")
g.add_edge("a", "b")
g.add_edge("b", END)
app = g.compile()
app.invoke({"total": 10, "visited": []})        # {'total': 21, 'visited': ['a', 'b']}

StateGraph is only a builder (like a C# ServiceCollection before BuildServiceProvider()). compile() lowers it into primitives. Here is what the demo prints for the compiled object:

Those lines are not part of the graph code above. They come from a small inspection helper I wrote (it’s show() in the demo script). It reads the public attributes of the compiled object, app.channels and app.nodes:

# `app` is the compiled graph from above (a Pregel object)
print("channels:", {name: type(ch).__name__ for name, ch in app.channels.items()
                    if not name.startswith("branch:")})    # hide the per-edge channels for brevity
for name, node in app.nodes.items():
    if name == "__start__":
        continue                                            # internal input node, skipped for brevity
    print(f"node {name!r}: triggers={node.triggers} bound={type(node.bound).__name__}"
          f" writers={len(node.writers)}")

which prints:

channels: {'total': 'LastValue', 'visited': 'BinaryOperatorAggregate',
           '__start__': 'EphemeralValue', '__pregel_tasks': 'Topic'}
node 'a': triggers=['branch:to:a'] bound=RunnableCallable writers=2
node 'b': triggers=['branch:to:b'] bound=RunnableCallable writers=1

The lines starting with node describe PregelNode objects, so here is what one is. This is the class from langgraph/pregel/_read.py, trimmed to the fields that matter here (I left out error-handler and subgraph fields, and the docstrings are mine, condensed from the source's):

class PregelNode:
    # A container, not a Runnable. The engine uses it to build a runnable task each time the node fires.

    channels: str | list[str]      # which channels to READ as input. For node "a": ['total', 'visited'],
                                   # so your function receives a dict with those two keys (the `s` in `def a(s)`)
    triggers: list[str]            # which channels WAKE the node: if any is written, the node runs next step.
                                   # For node "a": ['branch:to:a']
    bound: Runnable                # the actual logic: a Runnable wrapping your function, called with the input
    writers: list[Runnable]        # run AFTER `bound`; each takes its output and writes it to channels
                                   # (one stores your return value in state, one wakes the next node)
    mapper: Callable | None        # optional transform of the input before `bound` (None for node "a")

    retry_policy: Sequence[RetryPolicy] | None   # what happens if the node raises (None here)
    cache_policy: CachePolicy | None             # reuse results for identical input (None here)
    timeout: TimeoutPolicy | None                # per-invocation limit (async nodes only)
    tags: Sequence[str] | None                   # tracing labels
    metadata: Mapping[str, Any] | None           # tracing metadata

Note the split between channels (what it reads) and triggers (what wakes it). They are separate on purpose: a node can read many channels but be woken by just one. In C# terms a PregelNode is a record of "my inputs, my wake-up conditions, my handler, my post-processing", much like a message handler registration with a filter, except the messages are channel updates.

How to read the printout, line by line:

• 'total': 'LastValue': the total field became a channel of type LastValue. It holds one value, and a write replaces it. That is what the plain total: int in S meant.
• 'visited': 'BinaryOperatorAggregate': the visited field became a channel that combines old and new with your function, here operator.add. That is what Annotated[list, operator.add] meant, and why the list grows.
• '__start__': 'EphemeralValue': an internal channel that carries your invoke(...) input into the graph. EphemeralValue means it only lives for one round.
• '__pregel_tasks': 'Topic': an internal queue-like channel. It is where Send packets go (section 5c); our graph doesn't use it, but every graph has it.
• node 'a': triggers=['branch:to:a']: node a runs when the channel branch:to:a is updated. That channel is what the edge START → a turned into, so there is no edge object, only "wake a when this channel changes".
• bound=RunnableCallable: the thing that actually runs. It is a Runnable wrapper around your plain function a.
• writers=2 vs writers=1: after a runs, two things write: one stores its returned update into total/visited, and one writes to branch:to:b to wake node b. b is last, so it only has the first writer; its edge goes to END, which wakes nothing.

Note the channel list has no branch:to:a or branch:to:b, because my filter hid them. They exist, one per edge, and the nodes' triggers point at them.

Here is the same transformation as a picture. The first diagram is what you drew with add_edge; the second is what compile() actually builds (I added the __start__ input channel and the two writers per node to the picture, as the printout showed them):

What you wrote (builder):

flowchart LR
   S([START]) --> a --> b --> E([END])

What compile() built (runtime):

flowchart TD
   I["__start__ channel<br/>(your input)"] -->|wakes| CA["channel branch:to:a"]
   CA -->|wakes| NA["node a<br/>reads: total, visited<br/>bound = a(s)<br/>writer1 → total, visited<br/>writer2 → channel branch:to:b"]
   NA -->|wakes| NB["node b<br/>reads: total, visited<br/>bound = b(s)<br/>writer1 → total, visited<br/>(no writer2: its edge goes to END)"]

The arrows between nodes are gone in the second diagram. What remains are boxes (nodes) and named mailboxes (channels): “a” does not know “b” exists, it just drops a message in branch:to:b.

So the mental model is:

• Channels hold state (see the side notes above). Each state key becomes a channel whose type is its merge rule: LastValue keeps the latest value across steps but accepts at most one write per superstep (two parallel writers raise InvalidUpdateError, shown in 5c); BinaryOperatorAggregate applies your reducer (operator.add), which is why visited appends instead of replacing. Extra channels carry control flow: __start__ holds the input, and an edge a → b becomes a channel named branch:to:b. The double underscores are only a naming convention for internal channels (Python's __name__ style, not a keyword).
• Nodes are PregelNodes. Each has triggers (which channels wake it), channels (which it reads), bound (the actual Runnable, here a wrapper around your function) and writers (what it writes after running).
• Edges disappear. There is no edge object at run time. a → b is "a's writer writes to branch:to:b, and b triggers on branch:to:b". get_graph() reconstructs the pretty edge list for drawing, but the engine never walks it.

Conditional edges and Send use the same mechanism: a router Runnable runs as an extra writer and writes to whichever branch:to:X channels it chooses (or sends a task with its own input for map-reduce).

To see it, add a router to the same graph: a now picks b or c with g.add_conditional_edges("a", route, ["b", "c"]), where route(s) returns "b" if s["total"] > 5 else "c". As code (it reuses S, a and b from above, and adds node c and the router):

def c(s):                                # node "c": the alternative path, resets total
    return {"total": 0, "visited": ["c"]}

def route(s):                            # the router: returns the NAME of the next node
    return "b" if s["total"] > 5 else "c"

g = StateGraph(S)
g.add_node("a", a)
g.add_node("b", b)
g.add_node("c", c)
g.add_edge(START, "a")
g.add_conditional_edges("a", route, ["b", "c"])   # replaces add_edge("a", "b")
g.add_edge("b", END)
g.add_edge("c", END)
app = g.compile()
app.invoke({"total": 10, "visited": []})   # {'total': 21, 'visited': ['a', 'b']}  (a: 20 > 5 -> b)
app.invoke({"total": 1,  "visited": []})   # {'total': 0,  'visited': ['a', 'c']}  (a: 2 -> c)

I compiled and ran this, and inspected it the same way:

channels: total=LastValue, visited=BinaryOperatorAggregate, __start__, __pregel_tasks=Topic,
          branch:to:a, branch:to:b, branch:to:c   (the last three are EphemeralValue)
node a: triggers=['branch:to:a']  writers=[state write, _route]
node b: triggers=['branch:to:b']  writers=[state write]
node c: triggers=['branch:to:c']  writers=[state write]

What you wrote:

flowchart LR
   S([START]) --> a
   a --> b --> E1([END])
   a --> c --> E2([END])

What compile() built:

flowchart TD
   NA["node a<br/>writer1: state write → total, visited<br/>writer2: _route(s) ← your router function"]
   NA -->|"router returns 'b'"| CB["writes branch:to:b"] -->|wakes| NB[node b]
   NA -->|"router returns 'c'"| CC["writes branch:to:c"] -->|wakes| NC[node c]

The difference from the linear case is only the second writer of a. Instead of a fixed write to branch:to:b, it is _route, which calls your function and writes to whichever channel the answer names. The other channel is never written, so that node simply doesn't run. With total starting at 10, a makes it 20, the router picks b, and invoke returns {'total': 21, 'visited': ['a', 'b']}; c never fires. Start at 1 and the router picks c instead. (_route is the internal writer's name as I saw it in this version; treat it as an implementation detail.)

3. Pregel: the superstep loop

Pregel is a model for computing over graphs, described in a 2010 Google paper (“Pregel: A System for Large-Scale Graph Processing”, SIGMOD 2010). It builds on the older “bulk synchronous parallel” idea. Open-source systems such as Apache Giraph, Spark GraphX and Flink Gelly offer the same vertex-centric model. LangGraph borrows the idea to run your graph (it also names its compiled graph class Pregel, and that is why the module is called pregel). In one sentence: work proceeds in rounds, where every ready node runs in parallel on the same snapshot of the state, and only when all of them finish are their results merged and the next round decided. In C# terms, picture a round of Task.WhenAll(...) over the ready nodes, where every task reads the same snapshot and a barrier merges their results before the next round. (LangGraph's version is an adaptation: Google's Pregel passes messages between graph vertices, while LangGraph's "vertices" are your nodes and the messages are channel writes.) If you work in .NET, you may already have met this algorithm: Microsoft Agent Framework's workflow engine, in both its C# and Python versions, uses what its docs call "a modified Pregel execution model — a Bulk Synchronous Parallel (BSP) approach with superstep-based processing", so its executors run in the same supersteps. See Workflow Builder & Execution, "Execution Model: Supersteps" for details. In langgraph/pregel/_loop.py each round is:

1. tick: check the stop conditions, then prepare_next_tasks finds every node whose trigger channel was updated since it last ran.
2. Run those tasks (in parallel if there is more than one). A task reads the channel values from the start of the round and returns writes; nothing it writes is visible to its siblings.
3. after_tick: apply_writes merges all the writes into the channels through each channel's rule, bumps versions, emits stream output, and (with a checkpointer) saves a checkpoint.

Repeat until no node is triggered. Running the two-node graph with invoke gives {'total': 21, 'visited': ['a', 'b']}, and stream_mode="updates" shows one superstep per node:

[{'a': {'total': 20, 'visited': ['a']}}, {'b': {'total': 21, 'visited': ['b']}}]

The “write then apply at the barrier” split is the single most important idea. It’s why parallel branches are safe (they can’t see each other), why reducers exist (two branches writing the same key in the same round need an Annotated reducer, otherwise the graph fails), and why a checkpoint between rounds is a consistent snapshot of ordinary channels (the Deep Agents DeltaChannel is the exception, see 5b).

Send: when one node must become many tasks

A normal edge says “after a, run b", once, with the shared state as input. Sometimes you don't know how many tasks you need until runtime: the model asked for three tools, or you have N documents to process. Send(node, arg) covers that. It is a small packet meaning "run this node once, with this private input", and a router can return a whole list of them:

import operator
from typing import Annotated, TypedDict
from langgraph.graph import START, END, StateGraph
from langgraph.types import Send

class S(TypedDict):
    items: list
    out: Annotated[list, operator.add]       # reducer, so parallel writes are appended

def fan(s):                                  # router: one Send per item
    return [Send("work", {"x": i}) for i in s["items"]]

def work(a):                                 # 'a' is the Send's argument, NOT the graph state
    return {"out": [a["x"] * 10]}

g = StateGraph(S)
g.add_node("work", work)
g.add_conditional_edges(START, fan, ["work"])
g.add_edge("work", END)
app = g.compile()

print(app.invoke({"items": [1, 2, 3], "out": []}))

Streaming with stream_mode="debug" and printing each task event shows what the loop scheduled (I ran this):

task work {'x': 1} step 1
task work {'x': 2} step 1
task work {'x': 3} step 1
{'items': [1, 2, 3], 'out': [10, 20, 30]}

One node, three tasks, all in the same superstep, each with its own input. Who does what:

• You (or create_agent) write the router. It only returns the Send objects; it doesn't run anything.
• The Pregel loop owns the tasks. The router’s return value is written to the internal __pregel_tasks channel (the Topic from section 2). At the next barrier the loop reads that channel and, for each packet, builds a task for the named node with the packet's argument as its input (internally prepare_push_task_send, as opposed to the ordinary "triggered by a channel" tasks). The runner then executes those tasks together, like any other superstep.
• When: the router runs at the end of one superstep; the Send tasks run in the very next one. They then write their results through the normal channels, which is why the example needs the operator.add reducer: three tasks wrote out in the same round.

The C# picture is await Task.WhenAll(items.Select(i => Work(i))), except the list is decided by a router in the middle of the graph, and each of those calls is a first-class task that gets its own checkpoint writes. This is exactly what create_agent does for parallel tool calls, as the next section shows.

In one sentence: the Pregel loop runs your graph in supersteps, where a superstep is one round in which every ready node runs on the same snapshot of state and all their writes are merged at a barrier before the next round starts.

4. Example: an AI agent is just a graph

Before the harder parts (bubble-up, reducers, durability), here is the payoff of everything so far: an AI agent is expressed as a graph. It sounds like a different species, but create_agent(model, tools=[...]) returns a CompiledStateGraph, the same kind of object as app in section 2. So we can read how it is built, then run it, then look at the graph it produces.

4.1 The code of create_agent

create_agent is not magic; it is a function that makes a StateGraph using the same builder calls as section 2 and compiles it. Below is the relevant core of langchain/agents/factory.py (langchain 1.4.3), condensed by me to the path with no middleware and no structured output; the real file is about 2,000 lines, and bodies marked ... are omitted. The node and router functions are shown in full, simplified from model_node, _make_model_to_tools_edge and _make_tools_to_model_edge:

def create_agent(model, tools, ...):
    graph = StateGraph(state_schema=AgentState, ...)   # 'messages' etc. become channels

    # ---- node 1: "model" -- a plain function node ----
    def model_node(state, runtime):
        messages = state["messages"]                    # read the channel
        if system_message:
            messages = [system_message, *messages]
        ai_message = model.bind_tools(tools).invoke(messages)   # call the chat model
        return {"messages": [ai_message]}               # write: append to 'messages'
    graph.add_node("model", RunnableCallable(model_node))

    # ---- node 2: "tools" -- a ToolNode, itself a Runnable ----
    tool_node = ToolNode(tools)    # reads the last AIMessage's tool_calls, runs each tool,
    graph.add_node("tools", tool_node)                  # and writes one ToolMessage per call

    graph.add_edge(START, "model")                      # entry point

    # ---- router after "model": tool calls -> "tools", otherwise finish ----
    def model_to_tools(state):
        last_ai = last_ai_message(state["messages"])
        if not last_ai.tool_calls:                      # classic exit condition of an agent loop
            return END
        pending = [c for c in last_ai.tool_calls if c["id"] not in answered_ids(state)]
        return [Send("tools", [call]) for call in pending]   # one task per pending tool call
    graph.add_conditional_edges("model", model_to_tools, ["tools", END])

    # ---- router after "tools": loop back, unless a return_direct tool ran ----
    def tools_to_model(state):
        if all_executed_tools_are_return_direct(state):
            return END
        return "model"                                  # let the model read the results
    graph.add_conditional_edges("tools", tools_to_model, ["model"])

    return graph.compile(checkpointer=checkpointer, ...)   # same compile() as section 2

Three things are worth noticing, because they connect back to what we already saw:

• Nodes are ordinary functions of state. model_node reads the messages channel and returns a partial update that the reducer appends. That is exactly what node a did in section 2. ToolNode is just another Runnable.
• The loop is two conditional edges. model_to_tools and tools_to_model are the same kind of router as route in section 2. The cycle model → tools → model is the whole agent loop.
• Parallel tool calls are Send. When the model asks for two tools at once, the router returns one Send("tools", [call]) per call, so each runs as its own task in the next superstep (the Send mechanism explained at the end of section 3). In our one-tool run there is just one pending call, so there is one task.

The real function adds more nodes and edges when you pass middleware or a response format (that is what the extra ...middleware.before_agent nodes in the Deep Agents section are), but the skeleton is this.

4.2 Using it

Here is the code from the demo that calls create_agent. To stay offline and deterministic it uses a scripted model, a fake chat model that first asks to call the tool double(21) and then, once it sees the tool's result, answers. (A real agent would pass a real chat model in its place.)

from langchain.agents import create_agent
from langchain_core.language_models import BaseChatModel
from langchain_core.messages import AIMessage, HumanMessage, ToolMessage
from langchain_core.outputs import ChatGeneration, ChatResult
from langchain_core.tools import tool

@tool                                    # turns a plain function into a tool the model can call
def double(n: int) -> str:
    """Double n."""                      # the docstring is the description the model sees
    return str(n * 2)

class Scripted(BaseChatModel):           # a fake model with a fixed script
    @property
    def _llm_type(self): return "scripted"
    def bind_tools(self, tools, **kw): return self.bind(tools=tools, **kw)
    def _generate(self, messages, stop=None, run_manager=None, **kw):
        done = [m for m in messages if isinstance(m, ToolMessage)]
        if not done:                     # turn 1: no tool result yet, so ask for the tool
            msg = AIMessage(content="", id="ai-1", tool_calls=[
                {"name": "double", "args": {"n": 21}, "id": "call-1", "type": "tool_call"}])
        else:                            # turn 2: tool result is in the history, so answer
            msg = AIMessage(content=f"answer {done[-1].content}", id="ai-2")
        return ChatResult(generations=[ChatGeneration(message=msg)])

agent = create_agent(Scripted(), tools=[double])      # one call builds the whole graph
out = agent.invoke({"messages": [HumanMessage("double 21")]})

Inspecting agent the same way as before prints:

nodes   : ['model', 'tools']
channels: {'messages': 'BinaryOperatorAggregate', 'jump_to': 'EphemeralValue',
           'structured_response': 'LastValue', '__start__': ..., '__pregel_tasks': ...}
node 'model': bound=RunnableCallable
node 'tools': bound=ToolNode
edges   : [('__start__','model'), ('model','__end__',cond), ('model','tools',cond), ('tools','model',cond)]
messages: HumanMessage 'double 21' → AIMessage[tool_call double] → ToolMessage '42' → AIMessage 'answer 42'

4.3 The graph it produced

The graph that create_agent built is small. Asking the compiled object to draw itself (agent.get_graph().draw_mermaid()) gives this, where dashed arrows are conditional edges:

graph TD;
    __start__([__start__]) --> model(model)
    model -.->|tool calls present| tools(tools)
    model -.->|no tool calls| __end__([__end__])
    tools -.-> model

This is what get_graph().draw_mermaid() produced (I ran it), with the two edge labels added by me to explain the routing. Walking it with our run: __start__ puts the HumanMessage into messages and wakes model. The scripted model replies with a tool call, so the router sends the run to tools. tools executes double(21) and appends a ToolMessage('42'), which wakes model again. This time the reply has no tool calls, so the router goes to __end__. That is the messages: line above: two trips through model, one through tools.

That is the whole ReAct loop:

• State is a messages channel with an append reducer (so the conversation accumulates, just like visited above).
• model is a node that calls the chat model. tools is a ToolNode: a Runnable that reads the last AIMessage's tool_calls, runs each tool and writes ToolMessages.
• The loop is a conditional edge out of model: tool calls present → tools, otherwise → __end__. And tools → model is a plain back-edge. A cycle in the graph is the agent loop; the Pregel loop just keeps ticking while something is triggered.

Nothing agent-specific exists in the engine. “Agent” is a graph shape.

5. Graph bubble-up

A quick way to place this next to Send from the previous section: both are things a node or router hands to the loop, but in opposite directions. Send is data going down: a request to create tasks, written to a channel and picked up at the next barrier. Bubble-up is a control signal going up: an exception that stops work and travels from inside a node to the code that owns the run. They are separate mechanisms. They meet in one place: a task created by Send is an ordinary task, so if it calls interrupt() or raises a GraphBubbleUp, that signal bubbles up through the loop like any other (for example, an agent whose tool asks a human for approval).

Some things a node does are not return values. interrupt("approve?") has to stop the whole run, not just the node. LangGraph implements that as an exception, much like throwing a custom exception in C# that a catch further up the stack deliberately handles:

GraphBubbleUp(Exception)
  ├─ GraphDrained        (cooperative drain)
  ├─ GraphInterrupt      (interrupt(), human-in-the-loop)
  │    └─ NodeInterrupt
  └─ ParentCommand       (a subgraph telling its parent what to do)

“Bubble up” means exactly that: the exception is raised inside the node and travels up through the runner. Per _retry.py and _runner.py:

• the retry wrapper (run_with_retry) deliberately does not retry a GraphBubbleUp; ordinary exceptions go through your RetryPolicy;
• a GraphInterrupt has its pending writes saved to the checkpointer, so the pause survives;
• a ParentCommand is handled by the enclosing graph, which is how a Command(graph=Command.PARENT, ...) from a subgraph reaches its parent; a root graph suppresses a GraphInterrupt and returns the interrupt in its result rather than raising.

Command is just a small dataclass of four optional fields: update, goto, resume, graph. You use it in two places, and a different pair of fields matters in each:

• Returned from a node, to do “update state and route” in one step: Command(update={"total": 1}, goto="b"). update is applied like a normal node return value, and goto (a node name, a list of names, or Send packets) picks what runs next without needing an edge. graph=Command.PARENT aims it at the enclosing graph instead of the current one (that is the ParentCommand above).
• Passed as the input to invoke, to continue a paused run: Command(resume="yes"). Normally invoke takes new state; here it is told "don't start over, the saved run is waiting for an answer, and this is it". resume is the value that the pending interrupt() call returns.

The demo below uses the second form. The first form (update/goto) is shown in section 5c.

The demo shows the whole cycle with an in-memory saver. First the graph, with one node that calls interrupt(). A checkpointer is required, because the pause has to be saved somewhere:

from langgraph.checkpoint.memory import InMemorySaver
from langgraph.types import Command, interrupt

def ask(s):
    answer = interrupt("approve?")       # pauses the whole run here; returns the resume value next time
    return {"visited": [f"answer={answer}"]}

h = StateGraph(S)                        # same state class S as before
h.add_node("ask", ask)
h.add_edge(START, "ask")
h.add_edge("ask", END)
happ = h.compile(checkpointer=InMemorySaver())
cfg = {"configurable": {"thread_id": "t"}}   # identifies the saved conversation/run

print("first call :", happ.invoke({"total": 0, "visited": []}, cfg))   # runs until interrupt(), returns
print("next       :", happ.get_state(cfg).next)                        # which node is waiting
print("resumed    :", happ.invoke(Command(resume="yes"), cfg))         # continue, answering "yes"

which prints (the interrupt id is shortened):

first call : {'total': 0, 'visited': [], '__interrupt__': [Interrupt(value='approve?', id='014f…')]}
next       : ('ask',)
resumed    : {'total': 0, 'visited': ['answer=yes']}

Reading it: the first invoke does not raise. It returns normally, with the pending question under __interrupt__. get_state(cfg).next says node ask is the one waiting. The second invoke is given a Command(resume="yes") instead of new input; it finds the saved checkpoint for the same thread_id and continues.

Note what “resume” really is: the node runs again from the top, and this time interrupt() returns the resume value instead of raising. That re-execution is why side effects before an interrupt() must be idempotent.

6. And Deep Agents

Where it comes from. create_deep_agent lives in the deepagents package, a separate open-source (MIT) project from LangChain, published at langchain-ai/deepagents. Its package metadata describes it as an "agent harness with a built-in filesystem and context management, sub-agent delegation, skills, and long-term memory", and marks it Beta (I used 0.7.20). It is not part of langgraph or langchain; it depends on langchain. Its source imports create_agent from langchain.agents, and its docstring says several arguments are "passed through to create_agent".

Where it sits in the stack. Think of three layers, each built on the one below:

deepagents   create_deep_agent   a ready-made "harness": files, subagents, summarization, skills, memory
   |  calls
langchain    create_agent        the model <-> tools loop, plus the middleware system
   |  builds
langgraph    StateGraph / Pregel the runtime: channels, supersteps, checkpoints

In C# terms: LangGraph is the runtime (like ASP.NET’s host and pipeline), create_agent is a minimal app template, and Deep Agents is the opinionated starter kit that pre-installs a set of middleware on that template. Which is why the graph below looks the same as section 4's.

create_deep_agent(model=..., tools=[...]) also returns a CompiledStateGraph, with the same model and tools nodes, plus a middleware node. What the demo printed:

nodes   : ['model', 'tools', 'PatchToolCallsMiddleware.before_agent']
channels: messages: DeltaChannel, files: DeltaChannel,
          _summarization_event, _summarization_session_id, ...
tools the model sees: ['delete','double','edit_file','execute','glob','grep','ls','read_file','task','write_file']
edges   : ... ('model','model',cond), ('model','tools',cond), ('tools','model',cond) ...

So Deep Agents is the agent graph with extra state and extra tools, added by middleware:

• Extra channels: files (a virtual file system kept in graph state) and summarization bookkeeping for long conversations. messages is a DeltaChannel here rather than a plain reducer channel, so I wouldn't assume the agent's channel types equal the plain agent's.
• Extra tools: ls, read_file, write_file, edit_file, glob, grep, execute from the filesystem middleware, and task. execute is listed to the model, but the default StateBackend has no execute (I checked: hasattr(StateBackend, "execute") is False); it only runs when the backend implements SandboxBackendProtocol, otherwise it returns an error. So no shell access by default, and files is ephemeral graph state.
• A middleware hook as a node: PatchToolCallsMiddleware.before_agent runs before the first model call (it repairs dangling tool calls). Only middleware that overrides before_agent, before_model, after_model or after_agent adds nodes (I confirmed M.before_model and M.after_model appear in nodes); middleware that wraps model or tool calls does not.

What is middleware?

Middleware is how create_agent is extended without editing the loop. It is the same idea as ASP.NET middleware or DelegatingHandler: you write a class, hand a list of them to create_agent(..., middleware=[...]), and the loop calls your hooks at defined points. AgentMiddleware has two families of hooks (from langchain/agents/middleware/types.py):

• Node hooks, which run between steps: before_agent, before_model, after_model, after_agent. They receive the state and return a state update (or None), and can set jump_to to redirect the loop. Each one you override becomes a real node in the graph.
• Wrap hooks, which run around a call: wrap_model_call(request, handler) and wrap_tool_call(request, handler). You call handler(request) to continue, so you can change the request, retry, short-circuit or post-process the response. These live inside the existing model and tools nodes, so they add no nodes.

A middleware can also contribute tools and extra state keys. I ran this to check:

class Audit(AgentMiddleware):
    def before_model(self, state, runtime):             # a node hook
        log.append(f"before_model: {len(state['messages'])} msgs"); return None
    def wrap_model_call(self, request, handler):        # a wrap hook
        log.append("wrap_model_call: enter")
        resp = handler(request)                         # the next layer, ending in the model
        log.append("wrap_model_call: exit"); return resp

agent = create_agent(fake_model, middleware=[Audit()])

nodes: ['Audit.before_model', '__end__', '__start__', 'model']
log  : ['before_model: 1 msgs', 'wrap_model_call: enter', 'wrap_model_call: exit']

Audit.before_model is a new node; wrap_model_call added nothing to the graph but ran around the model call. That is exactly the PatchToolCallsMiddleware.before_agent node in the Deep Agents output above. Deep Agents is mostly a curated middleware list (filesystem, summarization, tool-call patching, optional skills and memory).

What is the state backend?

Deep Agents’ file tools (ls, read_file, write_file, edit_file, glob, grep) do not touch the disk directly. They call a backend, an object implementing BackendProtocol (ls, read, write, edit, grep, glob, delete, upload/download). The backend decides where files actually live, so the same agent can run against different storage. It is a repository/strategy interface, like IFileProvider in .NET. The backends in the package:

Backend Where files live StateBackend (the default) In the graph state's files channel. Ephemeral: persists within a thread through checkpoints, not across threads StoreBackend A LangGraph BaseStore, so persistent across conversations FilesystemBackend The real disk LocalShellBackend The real disk plus unrestricted local shell (execute) BaseSandbox (and subclasses) A sandbox that implements execute() CompositeBackend Routes by path prefix to other backends (for example /memories/ to a store, the rest to state)

Note: this is not the same thing as a LangGraph checkpointer, which is also sometimes called a state backend (section 5b). The checkpointer stores the graph’s snapshots; the Deep Agents backend stores the agent’s files. With the default StateBackend the two meet: its docstring says reads and writes go through CONFIG_KEY_READ and CONFIG_KEY_SEND, so a file write is just an ordinary channel write to files, applied at the barrier and checkpointed like any other state. That is why files showed up as a channel in the output above, and why the default needs no disk. StateBackend raises if used outside a graph run; to pre-seed files, pass {"files": {...}} to invoke.

Side note: the same pattern in the Squad SDK. Squad’s SDK (bradygaster/squad, packages/squad-sdk) has the matching idea in its StorageProvider interface (read, write, append, list, delete, mkdir, rename, stat, createIfAbsent...), with FSStorageProvider (default), InMemoryStorageProvider and SQLiteStorageProvider: a narrow interface, several implementations, one default. It was added by Dina Berry (diberry; #567, #640). Squad also has a separate, higher-level StateBackend (working-tree files, an orphan git branch, or that plus git notes), which is not the same thing: an adapter wraps it as a StorageProvider. Source, pinned to the commit I read: storage-provider.ts, state-backend.ts, State Backends docs. Back to Deep Agents.

What is DeltaChannel?

Both messages and files in the Deep Agents graph are DeltaChannels instead of the plain reducer channel. The problem it solves: a normal reducer channel checkpoints its full value every step, so a long conversation or a large virtual file system is copied into every checkpoint. DeltaChannel (beta, langgraph/channels/delta.py) instead stores only a sentinel in ordinary checkpoints and rebuilds the value by replaying the earlier writes through the reducer, writing a full snapshot every so often (every 1000 updates by default). It is event sourcing: the log of writes is the truth, snapshots are an optimization.

The price is in its own docstring: the reducer must be deterministic and batching-invariant (reducer(reducer(s, xs), ys) == reducer(s, xs + ys)), and the checkpoint saver must keep the ancestor writes (get_delta_channel_history). Section 5b covers what this means for checkpoints.

Deep Agents capability inventory

From the installed package layout (deepagents/middleware and deepagents/backends), and what I observed above:

• Middleware modules: filesystem, subagents, async subagents, summarization, memory, skills, permissions, rubric, patch-tool-calls, prompt caching, message eviction, overflow clipping, unsupported-content, tool exclusion.
• Backends (where files/execute actually go): state (graph state, the default seen in the demo), store, composite, filesystem, local shell, sandbox, LangSmith, context hub.
• Only middleware that defines before_*/after_* hooks becomes a graph node (we saw PatchToolCallsMiddleware.before_agent); the others wrap the model or tool call in place, so they don't appear in nodes. I inferred that rule from this one observed node and the module layout, not from reading the middleware loader.
• In this default build the model’s tools were the seven filesystem/shell tools, task, and my own double. I did not see a todo-list tool, so I won't claim one.

task is a graph calling a graph

The subagent mechanism is the most interesting part, and it is ordinary LangGraph. In deepagents/middleware/subagents.py, task is a StructuredTool. Its body prepares a fresh state (for a normal subagent just {"messages": [HumanMessage(description)]} plus non-private state keys), then calls:

result = subagent.invoke(subagent_state, subagent_config)
return _return_command_with_state_update(result, runtime.tool_call_id)

Because a compiled graph is a Runnable (section 1), the subagent is simply another compiled agent invoked from inside a tool call. Its result comes back as a Command(update={..., "messages": [ToolMessage(...)]}), which the parent's ToolNode applies to the parent's channels. The subagent's context window is isolated; only its final message and selected state keys return.

5b. Reducers, channel types and checkpoints, for real

Checkpoints. With a checkpointer, each superstep boundary writes a row. The demo saves the two-node graph in an InMemorySaver and lists it:

step=-1 source=input values={'__start__': {'total': 10, 'visited': []}}
step= 0 source=loop  values={'total': 10, 'visited': []}
step= 1 source=loop  values={'total': 20, 'visited': ['a']}
step= 2 source=loop  values={'total': 21, 'visited': ['a', 'b']}
versions_seen keys: ['__input__', '__start__', 'a', 'b']
channel_versions: {'__start__': '2', 'total': '4', 'visited': '4', 'branch:to:a': '3', 'branch:to:b': '4'}

A checkpoint is a small dict: channel_values, channel_versions, versions_seen, updated_channels, an id, a timestamp and v. Two fields do the scheduling:

• channel_versions is a monotonically increasing version per channel (the real strings are zero-padded counters plus a random suffix; I shortened them).
• versions_seen[node] records which trigger version each node last consumed. prepare_next_tasks runs a node when a trigger channel's version is newer than what that node has seen. That is the whole "what runs next" rule, and it is why resuming from a checkpoint needs no extra bookkeeping.

Step -1 stores the raw input; steps 0..N are loop supersteps. Writes made by tasks that have run but not yet been applied are stored separately as pending writes against their parent checkpoint (put_writes). That is how a crash in the middle of a parallel superstep doesn't lose the branches that already finished, and how interrupt state survives.

The saver (state backend) contract. BaseCheckpointSaver is the storage seam. The core methods are put (a checkpoint), put_writes (pending writes for a task), get_tuple and list (read), with async twins (aput, aget_tuple, ...), plus thread deletion/pruning helpers and a serde (here JsonPlusSerializer). Rows are addressed by thread_id, checkpoint_ns (the subgraph namespace) and checkpoint_id; a new thread id is a new empty history. I only ran InMemorySaver; I haven't exercised the SQLite or Postgres savers, so I make no claim about them.

Caveat: DeltaChannel. Deep Agents' messages and files use DeltaChannel (beta in langgraph/channels/delta.py). Per my reading of that source, not a run, it checkpoints a sentinel on ordinary steps and rebuilds the value from ancestor writes, with periodic full snapshots, so it needs deterministic reducers and savers that keep ancestor history. "Each checkpoint is a full snapshot" holds only for ordinary channels.

5c. When things get structured or go wrong: conflicts, routing, subgraphs, retry

The earlier sections showed the happy path. This one covers what happens when things go wrong or get more structured: two nodes fight over a key, a node needs to both update state and choose where to go, a graph contains another graph, a node throws, and you want to control how often state is saved or watch it live. I ran every snippet below offline (structured_cases.py in the samples repo); outputs are quoted as seen.

Two parallel nodes write the same key

Remember the superstep rule from section 3: nodes in the same superstep all read the old state, and their writes are applied together at the barrier. So what if two of them write the same key? For a plain field like total: int, the channel is a LastValue, which can hold only one value. "Which of the two writes wins?" has no sane answer, so LangGraph refuses to guess:

class S(TypedDict):
    total: int

g = StateGraph(S)
g.add_node("a", lambda s: {"total": 1})
g.add_node("b", lambda s: {"total": 1})
g.add_edge(START, "a")                           # a and b both start from START,
g.add_edge(START, "b")                           # so they run in the same superstep
g.add_edge("a", END)
g.add_edge("b", END)
g.compile().invoke({"total": 0})

InvalidUpdateError: At key 'total': Can receive only one value per step. Use an Annotated key to handle multiple values.

The error message already contains the fix. If you declare the key as Annotated[list, operator.add], the channel becomes a BinaryOperatorAggregate, which knows how to combine several writes, and the same graph returns {'total': ['a', 'b']}. In C# terms: it is the difference between two threads assigning the same field (a race you must forbid) and two threads calling list.AddRange through a lock (a merge you defined). The reducer is the lock.

Command: update state and choose the next node in one return

Normally a node returns a state update, and the edges decide what runs next. Sometimes the node itself knows where to go (a router, an agent deciding to hand off). Returning a Command lets it do both at once, as defined in section 5:

def router(s):
    return Command(update={"out": ["r"]}, goto="t")   # write state AND pick the next node

g = StateGraph(C)                       # C has: total: int, out: Annotated[list, operator.add]
g.add_node("r", router)
g.add_node("t", lambda s: {"out": ["t"]})
g.add_edge(START, "r")
g.add_edge("t", END)                                   # note: no edge from r to t
g.compile().invoke({"total": 0, "out": []})

{'total': 0, 'out': ['r', 't']}

There is no r → t edge in the graph. The goto created that routing at run time, and out shows both nodes ran, in order. The other two uses of Command you already met: resume= to continue after an interrupt (section 5), and graph=Command.PARENT to send the update to the enclosing graph when you are inside a subgraph (next subsection). Send (section 3) is the third routing tool: use goto to pick one next node, Send to start many tasks with different inputs.

A graph inside a graph (subgraphs)

Because a compiled graph is a Runnable (section 1), you can pass it to add_node like any function. The parent treats it as one node; inside, it runs its own supersteps. Its checkpoints live in their own namespace so they don't collide with the parent's, and stream(..., subgraphs=True) lets you see inside:

child = StateGraph(C)                        # C is the same state schema as above
child.add_node("t", lambda s: {"out": ["t"]})
child.add_edge(START, "t")
child.add_edge("t", END)

parent = StateGraph(C)
parent.add_node("child", child.compile())    # a compiled graph used as a node
parent.add_edge(START, "child")
parent.add_edge("child", END)
app = parent.compile(checkpointer=InMemorySaver())

cfg = {"configurable": {"thread_id": "1"}}
for event in app.stream({"total": 0, "out": []}, cfg, subgraphs=True):
    print(event)

(('child:512f5449-103f-f070-0e96-928c949f2c95',), {'t': {'out': ['t']}})
((), {'child': {'total': 0, 'out': ['t']}})

Each event is (path, update). The first has a non-empty path: node t finished inside the child. The second has the empty path (): the parent sees the whole child node finish. The path is the namespace. This is also why the task tool from section 6 is "a graph calling a graph".

Retry: what happens when a node throws

A node that raises normally fails the whole run. Attach a RetryPolicy and the loop reruns just that node instead. One gotcha I hit: the default retry_on deliberately excludes ValueError (programming-error exceptions are not retried by default), so my flaky node needed retry_on=ValueError explicitly:

def flaky(s):
    n["i"] += 1
    if n["i"] < 3: raise ValueError("boom")        # fails twice, then works
    return {"out": ["ok"]}

g.add_node("f", flaky, retry_policy=RetryPolicy(retry_on=ValueError, initial_interval=0.01))

{'total': 0, 'out': ['ok']}      # n["i"] == 3: two failures, third attempt succeeded

The retry is around the node, not the graph, so other nodes’ finished work in the same superstep is not repeated. And the control-flow exceptions from section 5 (GraphBubbleUp: interrupts, parent commands) are never retried, because they are not failures. In C# this is Polly's Retry policy wrapped around one call, with the exception filter being retry_on.

Saving less often, and watching live

Two knobs control the cost and visibility of all this machinery:

• durability on invoke/stream decides when checkpoints are written: 'sync' (before the next step starts; safest, slowest), 'async' (in the background while the next step runs), or 'exit' (only when the run ends; fastest, but a crash loses the in-flight run). These descriptions are from the parameter's documented meaning, not something I measured. The signature default is None; I did not verify what that resolves to, so I don't state an effective default.
• Stream modes choose what stream emits: values (full state after each step), updates (just each node's return, which is what you saw above), checkpoints, tasks, debug, messages (LLM tokens), and custom (whatever your node writes via the stream writer). Same run, different views.

Deep Agents extras (listed, not exercised)

From the module inventory only, I did not run these: skills, memory (prompt injection or a BaseStore), permissions, and interrupt_on human-in-the-loop. Subagents can be any compatible runnable, and task isolates context by default; an experimental fork mode inherits parent state.

Reading state back, and time travel

get_state(cfg) returns a StateSnapshot (values, next, config, metadata, created_at, parent_config, tasks, interrupts), and get_state_history(cfg) walks the parent chain. Time travel is just picking an older snapshot and invoking from it:

state before b = {'total': 20, 'visited': ['a']}  ->  invoke(None, that config) = {'total': 21, 'visited': ['a', 'b']}

The Store is a different thing. A checkpointer is per-thread run state. BaseStore (demo: InMemoryStore) is cross-thread long-term memory addressed by a namespace tuple and a key: store.put(("users","tamir"), "pref", {...}). Deep Agents' memory and store-backed file backends build on this idea; I only verified the plain InMemoryStore here.

A final thought

What stayed with me after reading all this code is how little magic there is. A graph is a set of channels and the nodes that wake when those channels change. Edges are just writes to channels, supersteps give you a clean read, write, merge rhythm, and the occasional special exception or Command is how a node steers the loop from the inside. An agent is that same machine with a model node and a tool node in a cycle, and a Deep Agent is the same machine again, with middleware adding channels, tools and hook nodes. Once I saw it that way, the docs, the stack traces and the checkpoints all started to make sense, and I stopped treating LangGraph as a black box.

If you want to poke at it yourself, everything in this post runs offline, with no API key, in the samples repo. Pin the versions from the top of the post, because the internals move between releases.

After my last couple of posts about Durable Task, I got a comment that made me realize I had skipped an important question:

This all looks interesting, but what are the actual use cases?

Not how Durable Task works. Not what happens during replay or how the scheduler persists state. But the more practical question: what problems does it actually solve?

When should I reach for durable execution instead of regular C#, a queue, a database, or a background worker? And when would adding it just make things more complicated?

Those are fair questions. Rather than walk through another hello world orchestration, I want to look at real situations where things get messy, especially when work takes a long time, depends on external systems, or gets interrupted halfway through. I also want to be clear about what Durable Task does not solve for you.

Imagine a background worker creating a customer’s Azure environment. It provisions a network, starts a database deployment, creates storage, and deploys an application. Then the worker disappears. Azure might still be working, but your local variables and continuations are gone. Restarting the method is easy. Figuring out which operations completed and which are safe to repeat is the hard part.

Of course, you could build your own solution with a database, a queue, an outbox, scheduled jobs, and reconciliation logic. Sometimes that’s exactly what you should do. But once you find yourself building a whole system just to remember what happens next and how to resume it, durable orchestration starts to get interesting.

I’ll walk through five examples: invoice approvals, Azure tenant provisioning, subscription payments, breaking news articles arriving through out of order webhooks, and an AI agent investigating an incident. We’ll use the standalone Durable Task .NET SDK v1.26.0 with DTS, and I’ll show both the useful parts and the responsibilities that still belong to our application.

If you want the background first, I covered what happens at a durable await and three less obvious Durable Task capabilities in the previous two posts.

A note on the examples: The snippets below are deliberately focused on orchestration boundaries. Application adapters, DTO definitions, registrations, and some helper implementations are omitted. They illustrate the design rather than constituting a compiled, tested sample. In particular, do not infer exactly-once external effects from durable history.

1. An invoice needs approval. Why should the worker remember the wait?

Consider an invoice-processing service. It extracts a document, validates the result, and either posts the invoice, rejects it, or asks a human to review it. A batch may contain eight invoices, only one of which needs approval. The other seven shouldn’t wait for it.

Here’s an ordinary C# implementation of the individual processing path:

public static async Task<InvoiceResult> ProcessAsync(
    InvoiceJob invoice, IInvoiceApplication app, CancellationToken ct)
{
    InvoiceCandidate candidate = await app.ExtractInvoiceAsync(invoice, ct);
    InvoiceAssessment assessment = await app.ValidateInvoiceAsync(candidate, ct);

    if (assessment.Check == InvoiceCheck.NeedsReview)
    {
        await app.OpenInvoiceReviewAsync(
            new InvoiceReview(invoice, assessment.PayloadReference), ct);
        return new(invoice.InvoiceKey, "AwaitingReview");
    }

    if (assessment.Check == InvoiceCheck.Rejected)
        return new(invoice.InvoiceKey, "Rejected");

    if (assessment.Check != InvoiceCheck.Valid)
        throw new InvalidOperationException("Unrecognized invoice assessment.");

    AccountingReceipt receipt = await app.SubmitApprovedInvoiceAsync(
        new InvoiceCandidate(invoice, assessment.PayloadReference), ct);
    return new(invoice.InvoiceKey, "Posted", receipt.EntryReference);
}

There’s nothing inherently unreliable about this method. OpenInvoiceReviewAsync can persist a review ticket and notification outbox. But when someone approves the invoice tomorrow, another handler must load the right document generation, check the deadline, recover the approved payload, and continue. If the worker dies between extraction and validation, the application must decide whether to extract again or retrieve the saved result.

That continuation protocol is what Durable Task can express directly.

Give each invoice its own durable history

A bounded batch can start independent child orchestrations:

public override async Task<InvoiceResult[]> RunAsync(
    TaskOrchestrationContext context, InvoiceBatch batch)
{
    if (batch.Invoices is null || batch.Invoices.Length is < 1 or > 8
        || batch.Invoices.Select(x => x.InvoiceKey).Distinct().Count()
           != batch.Invoices.Length)
        throw new ArgumentException("Supply 1 to 8 distinct invoices.");

    return await Task.WhenAll(batch.Invoices.Select(invoice =>
        context.CallSubOrchestratorAsync<InvoiceResult>(
            nameof(ProcessInvoice), invoice)));
}

Eight is an application admission limit, not a DTS quota. Larger batches can be paged and subject to tenant/global concurrency limits. A child that completes can post its invoice while another child remains suspended awaiting review. Task.WhenAll waits for every child; it does not undo a sibling's accounting entry if another child fails.

Inside the invoice child, extraction and validation are activities. Once their successful results are recorded in orchestration history, a compatible replay follows the same code path using those recorded results instead of repeating the activities. This is re-execution of orchestrator code against history, not restoration of a suspended CLR stack.

When validation requires review, the child can express the wait:

var review = new InvoiceReview(invoice, assessment.PayloadReference);
await context.CallActivityAsync("OpenInvoiceReview", review);

TimeSpan remaining =
    invoice.ReviewDeadlineUtc - context.CurrentUtcDateTime;

if (remaining > TimeSpan.Zero)
{
    try
    {
        InvoiceReviewHint hint =
            await context.WaitForExternalEvent<InvoiceReviewHint>(
                $"invoice-review:{invoice.ReviewId}", remaining);

        if (hint.ProcessingId != invoice.ProcessingId
            || hint.ReviewId != invoice.ReviewId)
            throw new InvalidOperationException("Review identity mismatch.");
    }
    catch (OperationCanceledException)
    {
        // The durable wait timed out. Read the committed decision below.
    }
}

InvoiceDecision decision =
    await context.CallActivityAsync<InvoiceDecision>(
        "ResolveInvoiceReview", review);

The timer is durable. No thread is occupied while the invoice waits, and a replacement worker can replay the child to the same pending wait. But the event is only a hint. An authenticated endpoint must atomically commit a single terminal review decision to the application’s review store before signaling the child. ResolveInvoiceReview reads or closes that authoritative ticket, including when the timer fires. If approval was committed just before the deadline but the hint arrived just after it, the stored decision, not notification timing, determines the outcome.

Only a valid automatic assessment or a committed human approval may proceed to the accounting activity. That activity uses a stable posting key and immutable request fingerprint. If the accounting system accepted the post but the activity’s acknowledgement vanished, the integration must reconcile the same posting identity. Durable history cannot promise exactly-once accounting.

This is the part I like. Instead of building separate handlers to remember extraction, validation, the review wait, and posting, I can describe the whole process in one place and let Durable Task remember where it got to. I still need to authorize reviewers, persist their decisions, and make accounting submissions safe to repeat. If my existing continuation handlers already do all of that cleanly, I might not need another runtime.

2. Tenant provisioning: the cloud already has a deployment engine

Now consider onboarding a SaaS tenant. A TenantCreated event means the tenant record was accepted, not that its environment is ready. This design provisions a dedicated network, PostgreSQL server, storage with private access, and Container App; then it configures DNS, checks readiness, and activates the tenant.

The ordinary C# method is easy to read:

public async Task<ProvisionedEnvironment> ProvisionAsync(
    EnvironmentRequest request,
    CancellationToken cancellationToken = default)
{
    NetworkResources network = await azure.CreateVirtualNetworkAsync(
        request, cancellationToken);

    Task<PostgreSqlServerReference> databaseTask =
        azure.CreatePostgreSqlServerAsync(
            request, network.PostgreSqlSubnet, cancellationToken);
    Task<StorageAccountReference> storageTask =
        azure.CreateStorageAccountAsync(
            request, network.StoragePrivateEndpointSubnet,
            cancellationToken);

    await Task.WhenAll(databaseTask, storageTask);

    ContainerAppReference application = await azure.DeployContainerAppAsync(
        request, network.ContainerAppsSubnet,
        await databaseTask, await storageTask, cancellationToken);

    await azure.CreateDnsRecordAsync(
        request, application.Endpoint, cancellationToken);
    await azure.WaitForApplicationReadyAsync(
        application.Endpoint, cancellationToken);

    return new ProvisionedEnvironment(
        request.EnvironmentId, application.Endpoint);
}

The helpers above wrap Bicep/ARM operations and wait for terminal completion or readiness. They’re application abstractions, not Azure SDK methods. The parallel modules own disjoint resources and don’t compete over subnet updates.

Before adding DTS, ask an uncomfortable question: does ARM already solve this? For a single well-defined deployment, it may. ARM/Bicep already supports dependency ordering, parallel deployment, idempotent reapplication with stable identities, and server-side deployment state. A worker crash doesn’t erase an ARM deployment. Querying it and reapplying the same desired state may be sufficient.

The wider onboarding process is different. The application must acknowledge the tenant request, report progress, coordinate readiness with activation, survive external approval steps if any, and decide what to do with a partly provisioned environment. That’s where a business-level workflow can add value.

Separate resource readiness from tenant activation

First, the API authorizes and durably admits an immutable onboarding request, then returns 202 Accepted, a request ID, and a status URL. Accepted does not mean ready. The application records the tenant, environment, immutable specification, and ownership generation. Reusing the same idempotency key with the same specification returns the existing request; changing the specification is rejected.

The scheduler handoff is not magically atomic with the application’s database. Save the intent to start in an outbox alongside admission; dispatch it and reconcile the workflow identity. Repeatedly scheduling a new instance is not the same as resuming an existing one.

The outer orchestration can then express the business dependency:

public override async Task<TenantActivation> RunAsync(
    TaskOrchestrationContext context, EnvironmentPlan plan)
{
    EnvironmentResult resources =
        await context.CallSubOrchestratorAsync<EnvironmentResult>(
            nameof(ProvisionEnvironment), plan);

    return await context.CallActivityAsync<TenantActivation>(
        "ActivateTenant", new TenantActivationRequest(plan, resources));
}

The child finishes at resource readiness. Activation is a separate, idempotent application activity that revalidates tenant ownership and actual resource readiness. If activation fails, the outer workflow fails visibly without automatically deleting the resources the child provisioned. Repairing activation is a different problem from rolling back provisioning.

Inside the child, stage dependencies form waves: network; database and storage in parallel; application; DNS; readiness. Each stage activity records an intent and starts or reconciles a provider operation. A separate read activity performs one bounded provider status query; the orchestrator schedules a durable timer between polls using context.CurrentUtcDateTime and context.CreateTimer.

DateTime now = context.CurrentUtcDateTime;
if (now >= deadline)
    return Failed("PollingDeadlineExceeded");

TimeSpan delay = TimeSpan.FromSeconds(progress.PollAfterSeconds);
DateTime wakeAt = delay >= deadline - now
    ? deadline
    : now.Add(delay);

await context.CreateTimer(wakeAt, CancellationToken.None);

No HTTP request or Task.Delay runs inside the orchestrator. The activity does the I/O; the orchestration history records the activity result and timer decision. A 45-minute polling budget limits additional polling. It does not cancel an Azure operation that was already accepted.

Follow one failure all the way through

Imagine the database deployment was accepted by Azure, but the worker died before the activity completion reached DTS:

1. DTS does not have a recorded successful activity result, so activity redelivery is possible.
2. The redelivered adapter consults the application’s operation journal and the stable Azure deployment/resource identity.
3. If the deployment is still running, it resumes inquiry rather than starting an unrelated database.
4. Once the provider returns terminal completion, the activity records the result and the child continues toward application deployment.
5. The tenant is activated only after the resource child returns validated readiness evidence.

Contrast that with a different failure: the activity completion was recorded, and only then the worker crashed. On compatible replay, the orchestrator consumes that result from history without invoking the completed activity again. These are different failure windows; durable execution history solves one, and your adapter’s reconciliation protocol solves the other.

Compensation isn’t an undo button

A failed workflow doesn’t mean Azure stopped working. Imagine that we’ve started creating a database and a storage account in parallel. Storage provisioning fails, but Azure is still creating the database. If we immediately delete everything we can see, that database might finish provisioning after cleanup has already checked for it. We’ve now reported failure and cleanup while an orphaned database continues generating charges.

Before cleaning up, the application needs to:

1. Prevent this failed onboarding attempt from submitting any new provisioning operations.
2. Check previously submitted operations, including requests whose responses were lost, and reconcile their actual state. Don’t assume a timeout or a failed worker stopped Azure.
3. Delete only resources created exclusively for this attempt, and only after establishing that outstanding operations cannot create them again after deletion.

Automatic cleanup also has a strict boundary: this must be a failed initial onboarding attempt, before tenant activation or customer-data admission. Existing shared resources and active customer data are never cleanup targets. Recheck current ownership and provider identity before each deletion, and delete eligible new resources in reverse dependency order: DNS, application, database/storage, then network.

If we cannot establish what is still running or what we safely own, retain the resources and surface NeedsIntervention for reconciliation rather than guessing. Record both the original provisioning failure and any cleanup failures. Durable Task can coordinate this recovery process, but it cannot decide which Azure resources are safe to delete.

Going deeper: fencing and draining in-flight operations
Two related ideas explain why cleanup is harder than reversing completed steps:

• prevents outdated attempts from making new changes, provided the system receiving those changes enforces the restriction. For example, our application can reject submissions from an older provisioning generation. This does not revoke requests Azure already accepted.
• Draining and reconciling in-flight operations means accounting for previously submitted work, including requests whose acknowledgements were lost, before declaring cleanup safe. Some distributed-systems discussions call reaching a state with no relevant outstanding work quiescence (from quiescent, meaning quiet or inactive). Here we’re describing a practical cleanup requirement, not the formal concurrency property quiescent consistency and not a guarantee Azure provides automatically.

For background, see Martin Kleppmann’s explanation of fencing tokens and Microsoft’s ARM deployment cancellation API. Even cancellation may leave resources partially deployed. If we cannot establish that earlier requests can no longer create resources, we retain the environment for reconciliation rather than declaring cleanup complete.

There is one thing to keep in mind here. Task.WhenAll waits for the whole provisioning wave, but if one of those activities throws, execution will not continue to the code that checks the results. In this example I treat expected failures as results and let unexpected failures fail the orchestration normally.

This also matters for cleanup. Waiting for all the provisioning tasks to finish tells me what succeeded and what failed, so I know what needs to be cleaned up. It does not automatically perform that cleanup. I still need to explicitly run the compensation steps for the resources that were already created.

Here, Durable Task is useful for connecting resource provisioning, the long waits, and tenant activation without writing my own continuation state machine. ARM still provisions the resources. My application still needs an outbox, operation identities, safe cleanup, and a way to repair uncertain outcomes. I would not add Durable Task just to rebuild the dependency graph ARM already gives me.

3. A payment timeout is not a decline

Suppose a subscription charge times out. The provider may have accepted the payment before the response disappeared. A naive retry risks charging twice; a naive decline path risks sending a misleading reminder.

A conventional implementation can persist NextCheckUtc, finish its handler, and let a scheduled job inquire later. That's a reasonable architecture. A durable orchestration becomes interesting when the recovery policy also includes due dates, callbacks, bounded reminders, a fixed grace deadline, and transition to the next billing cycle.

Before going further, a billing cycle means one billing period and its associated payment obligation, not one charge attempt or the entire subscription. For a €20 monthly subscription, September and October are separate cycles. Each has its own CycleId, due date, and initial PaymentKey; both belong to the same SubscriptionId. GraceUntilUtc sets the deadline for this cycle's automated recovery. If a payment times out, the cycle remains the same while the service investigates whether that attempt actually succeeded.

The distinction matters for ContinueAsNew. Once a paid receipt is confirmed and the billing service authoritatively plans the next cycle, the orchestrator can continue with new cycle input while retaining its instance ID and starting a fresh execution history:

This is a conceptual lifecycle, not a promise that each month will be charged successfully. An unresolved cycle does not automatically advance, and the billing service, not the orchestrator, decides whether a next cycle is permitted.

Here the billing ledger, not DTS, is authoritative. The billing store binds a cycle ID and stable payment key to the amount and currency. Payment state distinguishes Unknown, Pending, Declined, Paid, and NotCharged, with cancellation and receipt information. The workflow coordinates inquiries and reminders. A separate payment service owns any newly authorized charge after card correction.

The orchestration waits until the cycle’s due date with a durable timer, then calls an activity that either initiates the initial payment only when authorized and definitively not submitted, or reconciles an existing paid, pending, or unknown attempt. Its recovery loop checks state and waits for a callback hint or the next six-hour check, whichever comes first:

DateTime now = context.CurrentUtcDateTime;
if (now >= cycle.GraceUntilUtc || check == 15)
    break;

if (!reminderSent && !payment.SubscriptionCancelled
    && payment.State == PaymentState.Declined
    && now >= cycle.DueUtc.AddHours(6))
{
    await context.CallActivityAsync(
        "SendPaymentReminder",
        new PaymentReminder(cycle, $"{cycle.CycleId}:reminder-1"));
    reminderSent = true;
}

now = context.CurrentUtcDateTime;
if (now >= cycle.GraceUntilUtc)
    break;

DateTime wakeAt = now.AddHours(6);
if (wakeAt > cycle.GraceUntilUtc)
    wakeAt = cycle.GraceUntilUtc;

try
{
    BillingHint hint = await context.WaitForExternalEvent<BillingHint>(
        $"billing:{cycle.CycleId}:v{cycle.EventSchema}", wakeAt - now);

    if (hint.CycleId != cycle.CycleId
        || hint.EventSchema != cycle.EventSchema
        || string.IsNullOrWhiteSpace(hint.EventId))
        throw new InvalidOperationException("Billing event identity mismatch.");
}
catch (OperationCanceledException)
{
    // Timer elapsed. Query the billing ledger.
}

payment = await context.CallActivityAsync<PaymentSnapshot>(
    "ReadCyclePayment", cycle);

This excerpt assumes the surrounding orchestrator has validated the cycle, UTC deadlines, grace interval, snapshot identity, and terminal paid/cancellation branches. It also bounds the loop to at most 16 inspected snapshots. Callback handlers authenticate, deduplicate, and persist events in the billing inbox/ledger before sending a hint. Neither a callback nor a payment-method update proves that a charge succeeded.

When a paid receipt is validated, an idempotent activity closes the paid cycle and obtains the authoritative next cycle. Only then does the orchestrator call context.ContinueAsNew(next, preserveUnprocessedEvents: false) with that next-cycle input. That preserves the instance identity while starting a fresh execution history. Here we deliberately discard buffered old hints because the billing facts were persisted before notification; old settlements remain tied to their original billing cycle.

If the grace or check budget expires with an unresolved payment, the application outcome is NeedsReconciliation, tracked by a durable recovery queue. Runtime completion isn't proof of payment. Cancellation stops future collection when recorded but doesn't erase an already accepted charge.

I like being able to read the payment recovery policy as one flow, with its due date, callbacks, reminders, and transition to the next cycle. But the billing ledger still decides whether a payment actually happened. The workflow does not get to guess, charge again, or replace the existing payment safeguards. And if one scheduled job already handles this without much ceremony, I would keep that instead.

4. Webhook bursts: an example where durable state may not be enough

This example is a little different from the previous ones because it is also a good example of where Durable Task does not solve the whole problem.

Imagine a breaking-news publisher. Editors keep updating a developing story as new information arrives, and readers expect the publisher’s search results to show the latest published version. Every time an editor publishes a change, the content-management system (CMS) emits a webhook. Our service receives it and updates a separate search index.

For example, an article titled Major storm hits the coast has three published revisions:

• Revision 11: the initial report.
• Revision 12: updated casualty figures and evacuation information.
• Revision 13: a corrected headline and the latest official statement.

The webhook for revision 12 arrives before the delayed notification for 11; another webhook for 12 arrives twice. While our search index is still processing revision 12, revision 13 is published. Worse, two indexing operations can finish in the opposite order from when they started. Without protection, a slow revision-12 update could overwrite revision 13, leaving readers with stale breaking-news results.

Here’s the flow. The arrows show possible timing, not a promise of ordering:

The key distinction: the durable entity remembers the newest revision we’ve heard about. It does not lock the search index or guarantee that only one indexing activity runs. The search destination must atomically reject an older revision if a newer one has already been committed. In the diagram, the delayed revision-12 operation is rejected rather than overwriting revision 13. A separate reliable inbox and follow-up dispatcher ensure that newly published revisions aren’t lost between checks.

For this example, assume the CMS supplies strictly increasing positive revision numbers per article generation, immutable content for each published revision, and versioned tombstones for deletion. A recreated article gets a new generation. If the CMS supplies only opaque ETags or webhook arrival timestamps, this particular revision-ordering algorithm does not apply.

A conventional handler can already be correct:

public static async Task HandleAsync(
    DocumentNotice notice, IDocumentProjection projection,
    IProjectionCheckpointStore checkpoints, CancellationToken ct)
{
    long applied = await checkpoints.ReadAppliedAsync(
        notice.DocumentKey, ct);
    if (notice.Revision <= applied)
        return;

    DocumentSnapshot snapshot =
        await projection.ReadDocumentSnapshotAsync(
            new DocumentRead(notice.DocumentKey, notice.Revision), ct);

    ProjectionReceipt receipt =
        await projection.ApplyDocumentRevisionAsync(snapshot, ct);

    await checkpoints.AdvanceAppliedAsync(receipt, ct);
}

This assumes the destination atomically rejects stale revisions and handles duplicates idempotently, while the checkpoint advances monotonically. A lost acknowledgement can be reconciled against the destination. Add a reliable inbox/dispatcher and periodic source reconciliation, and you may already have everything you need.

What does a durable entity change? It can persist the highest observed and applied revisions and serialize updates to those counters:

public sealed class DocumentRevision : TaskEntity<DocumentRevisionState>
{
    public void Observe(long revision)
    {
        if (revision <= 0)
            throw new ArgumentOutOfRangeException(nameof(revision));
        State.LastObserved = Math.Max(State.LastObserved, revision);
    }

    public RevisionProgress Read() =>
        new(State.LastObserved, State.LastApplied);

    public void ConfirmApplied(ProjectionReceipt receipt)
    {
        if (receipt.DocumentKey != Context.Id.Key
            || receipt.ConfirmedRevision <= 0
            || string.IsNullOrWhiteSpace(receipt.ReceiptReference))
            throw new InvalidOperationException("Invalid projection receipt.");

        State.LastApplied = Math.Max(
            State.LastApplied, receipt.ConfirmedRevision);
    }
}

A separate SynchronizeDocument orchestration calls Observe, waits two seconds with a durable timer, reads the entity, fetches an authoritative immutable snapshot through a read activity, applies it through a write activity, and calls ConfirmApplied only after obtaining a valid external receipt. It can make up to four passes, then request follow-up if work remains.

Two different identities, two different guarantees. A durable entity is addressed by its entity name and key. In this example, DocumentRevision plus the article's generation-qualified key identifies one entity. Durable Task processes operations for that same entity instance serially, so simultaneous Observe and ConfirmApplied calls cannot corrupt its counters. This does not mean Durable Task automatically runs only one SynchronizeDocument orchestration per article: by default, separately scheduled orchestrations have separate instance IDs. See Microsoft: Durable entities and orchestration instance identity.

If we want one active synchronization controller per article, we can deliberately adopt the singleton orchestrator pattern: assign a stable orchestration instance ID derived from the article’s generation-qualified key, rather than creating a new instance for each webhook. The inbox records every notice; the entity keeps the highest observed revision; the dispatcher starts the controller if needed or signals the existing controller to check again. The inbox must also cover the race between a controller’s final read and its completion. A check-then-start request can itself race, so the dispatcher must handle duplicate-start outcomes and reconcile admission rather than assume both callers’ success responses mean two controllers started.

A singleton controller avoids concurrent controllers for the same article when admission is implemented correctly, but it is not an exactly-once guarantee for external effects. A timed-out or redelivered activity may have left an indexing request running, and an earlier execution may have submitted work before it stopped. Keep the destination’s atomic revision check and reconcile uncertain writes even with a singleton controller. The sequence diagram above intentionally depicts the multiple-controller variant, to show the race that the entity’s serialization alone does not prevent.

But here’s the catch: serializing entity operations does not serialize the external index writes. Two orchestrations can both read LastObserved = 13 and LastApplied = 12, then fetch and attempt revision 13 concurrently. The two-second pause may even align them. This design does not establish fewer fetches, fewer writes, or true debounce behavior.

Its correctness still depends on a destination-enforced atomic generation/revision fence. A delayed revision-12 write cannot overwrite an already committed revision 13. If the destination cannot enforce that rule, introduce a separately enforced single writer or fenced-writer protocol instead. The entity alone is insufficient.

A final read also races with a newly arriving revision. The application inbox/outbox must durably admit every validated notice and dispatch follow-up work; periodic reconciliation handles missing webhooks. Resetting an entity cannot reset the destination’s revision protection.

The entity gives us a durable place to remember the latest article revision, and its operations are serialized for that entity key. That is useful, but it does not magically serialize the external indexing requests. If our existing checkpoint store and dispatcher already coordinate revisions reliably, adding an entity might not buy us much. I would measure actual indexing calls before claiming this approach reduces work.

5. An AI-assisted incident workflow: preserving the exact decision across a crash

I have written before about this kind of AI workflow and why deterministic orchestration matters when models and tools are involved. In that post, I focused mostly on making the workflow predictable. I did not spend nearly enough time on what happens when the process disappears halfway through. That is what I want to explore here.

Consider a production incident assistant: it gathers telemetry, asks a model to propose a remediation, waits for an engineer to approve the specific proposal, runs the approved operation, and verifies the result. The investigation might take minutes; approval might arrive hours later. During that time, the process hosting the assistant may be recycled or deployed.

We need parallel work, durable waits, replay, and external-operation reconciliation. We also face a new problem: an LLM response is not a deterministic function we should casually recompute after recovery. The next response might recommend a different action, use different tool arguments, or refer to different evidence.

Before: an ordinary asynchronous agent pipeline

Suppose the application already has services for telemetry, model inference, approval, remediation, and verification. A simplified ordinary implementation might look like this:

public static async Task<IncidentReport> InvestigateAsync(
    IncidentRequest incident,
    IIncidentApplication app,
    CancellationToken ct)
{
    Task<EvidenceRef> logs = app.CollectLogsAsync(incident, ct);
    Task<EvidenceRef> metrics = app.CollectMetricsAsync(incident, ct);
    await Task.WhenAll(logs, metrics);
    EvidenceBundle evidence = new(await logs, await metrics);

    ProposedRemediation proposal =
        await app.GenerateRemediationAsync(incident, evidence, ct);
    await app.OpenApprovalAsync(proposal, ct);

    // A separate callback must resume the right proposal after approval.
    return new IncidentReport(incident.IncidentId, "AwaitingApproval");
}

This method is not inherently wrong. The application could save a state machine in SQL and continue from an approval callback. But that continuation handler must retrieve the exact evidence and proposal version, check authorization and expiry, and decide whether remediation was already submitted. If the worker crashes after the model generates its proposal but before saving it, generating again is a new decision, not recovery of the same one.

The crucial design decision is to separate the nondeterministic generation step from the deterministic coordination step. The model call belongs inside an activity. The orchestrator receives a stable reference to an immutable proposal, not a mutable chat transcript or a promise that the model will answer identically next time.

After: durable coordination around an agent

For this example, assume the application defines IncidentPlan with a stable incident ID, immutable request generation, approval ID and UTC deadline. EvidenceBundle contains references to saved telemetry, not raw logs. ProposedRemediation contains an immutable proposal ID, evidence references, canonical action parameters, a content fingerprint and policy version. ApprovalDecision names the approved proposal ID and fingerprint. These are application contracts, not types supplied by Durable Task.

The request records below name the payloads passed to activities. ApprovalKind distinguishes Pending, Approved, Rejected, and Expired; only approval can lead to remediation. The activities and remaining DTOs are application code, not a complete executable sample:

public sealed record GenerateProposalRequest(
    IncidentPlan Plan, EvidenceBundle Evidence);
public sealed record RemediationApprovalRequest(
    IncidentPlan Plan, ProposedRemediation Proposal);
public sealed record VerifyRemediationRequest(
    IncidentPlan Plan, RemediationReceipt Receipt);

public sealed class InvestigateIncident
    : TaskOrchestrator<IncidentPlan, IncidentOutcome>
{
    public override async Task<IncidentOutcome> RunAsync(
        TaskOrchestrationContext context, IncidentPlan plan)
    {
        Task<EvidenceRef> logs = context.CallActivityAsync<EvidenceRef>(
            "CollectLogs", plan);
        Task<EvidenceRef> metrics = context.CallActivityAsync<EvidenceRef>(
            "CollectMetrics", plan);

        await Task.WhenAll(logs, metrics);
        EvidenceBundle evidence = new(await logs, await metrics);
        ProposedRemediation proposal =
            await context.CallActivityAsync<ProposedRemediation>(
                "GenerateAndSaveProposal",
                new GenerateProposalRequest(plan, evidence));

        await context.CallActivityAsync(
            "OpenRemediationApproval",
            new RemediationApprovalRequest(plan, proposal));

        TimeSpan remaining =
            plan.ApprovalDeadlineUtc - context.CurrentUtcDateTime;
        if (remaining > TimeSpan.Zero)
        {
            try
            {
                // This event is only a wake-up hint, not authorization.
                ApprovalHint hint =
                    await context.WaitForExternalEvent<ApprovalHint>(
                        $"incident-approval:{plan.ApprovalId}", remaining);
                if (hint.IncidentId != plan.IncidentId ||
                    hint.ApprovalId != plan.ApprovalId)
                    throw new InvalidOperationException(
                        "Approval identity mismatch.");
            }
            catch (OperationCanceledException)
            {
                // Deadline elapsed. Resolve authoritative approval state.
            }
        }

        ApprovalDecision decision =
            await context.CallActivityAsync<ApprovalDecision>(
                "ResolveApproval",
                new RemediationApprovalRequest(plan, proposal));
        if (decision.Kind is ApprovalKind.Rejected or ApprovalKind.Expired)
            return new(plan.IncidentId, decision.Kind.ToString());
        if (decision.Kind != ApprovalKind.Approved)
            throw new InvalidOperationException(
                "Approval is unresolved or unrecognized; reconcile the ticket.");

        // The activity must independently revalidate authorization,
        // proposal fingerprint, current policy and execution eligibility.
        RemediationReceipt receipt =
            await context.CallActivityAsync<RemediationReceipt>(
                "ExecuteApprovedRemediation",
                new ApprovedRemediation(plan, proposal, decision));

        VerificationResult verification =
            await context.CallActivityAsync<VerificationResult>(
                "VerifyRemediation",
                new VerifyRemediationRequest(plan, receipt));
        return new(plan.IncidentId, verification.Status,
            receipt.ReceiptReference);
    }
}

The snippet expresses dependencies, not a security boundary. The authenticated approval endpoint must atomically commit one terminal decision against the immutable proposal ID, fingerprint and deadline before sending its event hint. ResolveApproval reads that committed record or atomically expires a still-pending ticket at its deadline. An unresolved or unrecognized result fails visibly for reconciliation. A hint cannot authorize a tool call, and a late or duplicate hint cannot overwrite a terminal decision. The remediation activity checks the stored decision again and applies any current safety policy before acting. An approved plan that is no longer safe should stop for fresh review, not be executed merely because it appears in orchestration history.

Notice that GenerateAndSaveProposal must use a stable generation identity and reconcile its own saved result if the model responded but the activity acknowledgement was lost. Durable Task will reuse the activity result only after that result was recorded in orchestration history. It cannot guarantee a model is called exactly once in every failure window. Likewise, ExecuteApprovedRemediation needs a stable execution key, an operation journal and a provider-specific read/reconciliation path; blindly retrying a potentially destructive tool is unacceptable.

Follow the failure, not just the happy path

Consider four points at which the worker might disappear:

Interruption Safe recovery behavior

After telemetry collection was recorded

Replay consumes recorded activity results; it does not collect those results again merely because the worker restarted.

After model generation but before its activity result was recorded

The activity uses the stable generation identity to retrieve the saved proposal or reconcile an uncertain generation; it must not silently substitute a new plan for one already presented.

While waiting for an engineer

The durable wait resumes after restart. Approval authority remains in the application approval store, and the fixed deadline does not move.

After remediation succeeded but before its activity completion was recorded

Redelivery reconciles the same execution key and external receipt. It does not assume the action failed or execute it a second time without evidence.

A useful test deliberately kills the worker at each of those points. Then inspect orchestration history, the proposal and approval records, and the remediation operation journal. The invariant is not “the orchestrator completed.” It is “only the specifically authorized proposal was executed, and uncertain side effects were reconciled before further action.”

So what did we gain?

We now have a workflow that can collect evidence, generate a proposal, wait for approval, execute the approved action, and continue after a worker restart. That is already pretty useful. But notice the boundary: we made the workflow around the AI durable. We did not make the model deterministic, and we did not make every bit of state inside an agent durable. The proposal and approval still need stable identities, and an uncertain tool execution still needs reconciliation.

This is really the foundation of what I want from durable AI systems. It is also why we went further and built Durable Agents and Durable Workflows as extensions to Microsoft Agent Framework (MAF). Instead of having every application build this coordination around its agents, we wanted durability to be part of the way agents and their workflows run.

That deserves its own article. In the next part, I will use this same scenario to show what changes when durability moves from the outer orchestration into the agent and workflow programming model itself.

A final thought

The common thread across these examples is simple. Durable Task can remember what completed, what we are waiting for, and where execution should continue. It cannot tell us whether an external payment really happened, whether an Azure operation is still running, or whether an AI generated action is still safe to execute. Our applications still need to answer those questions.

I would use Durable Task when the code for remembering and resuming a business process starts becoming a system of its own. Not just because I have a few await calls.

Originally published at https://www.tamirdresher.com on September 27, 2026.

Well, I’m not the guy who breaks promises :-)

Once you understand how a workflow can keep going without its original worker, a few practical questions follow. Can different parts run on different kinds of workers? Can a workflow that is already running move to a newer implementation? And when something fails after several successful steps, can we repair it without starting the whole business operation again?

I’ll use the standalone Durable Task .NET SDK v1.26.0 with manual registration, without the optional source generator. The code below is based on public API definitions and samples. These are teaching illustrations, not compiled or executed demonstrations. Real activity implementations and connection values are omitted; the shipping example includes host setup and startup. The clients and workers are assumed to be configured for the intended DTS task hub.

Keep the workflow together, split the workers

Suppose order validation is inexpensive, but shipping needs a large integration SDK, different compute, or its own release schedule. I don’t particularly want every validation worker carrying the shipping dependencies just because both activities belong to the same workflow.

The workflow can stay one workflow without making every worker carry every activity.

The distinction is between coordination and where work runs. There are three pieces to keep separate. The registry maps task names and versions to code a worker can run. Filters describe the work that worker asks DTS to send it. An optional worker version policy adds acceptance checks. They need to agree: a filter neither registers a missing implementation nor translates its payload.

Here’s where those pieces live. Solid arrows show interactions between roles; dashed links show configuration used by the worker SDK, not an order of checks. Client and worker roles can share a process. Click to enlarge.

An orchestrator still requests activities by name. Work item filtering lets DTS deliver those work items to eligible workers, without putting worker addresses into the workflow.

Here is a small, independently defined orchestration:

using System.Threading.Tasks;
using Microsoft.DurableTask;

public sealed class OrderRouting : TaskOrchestrator<string, string>
{
    public override async Task<string> RunAsync(
        TaskOrchestrationContext context, string orderId)
    {
        string validation = await context.CallActivityAsync<string>(
            "ValidateOrder", orderId);
        string shipping = await context.CallActivityAsync<string>(
            "ShipOrder", orderId);
        return $"{validation}; {shipping}";
    }
}

The validation activity throws if the order is invalid. Its returned string is informational, not an approval flag. The activity implementations are outside this example.

The orchestration knows the logical task name ShipOrder, not the shipping worker's server address. CallActivityAsync creates an action and a local result task, not a direct call to that host or an immediate durable record. DTS later delivers eligible work; the receiving worker finds its registered implementation and reports the result, which the orchestration uses through history. Shipping workers can move or scale without changing server addresses in the workflow. The activity's own external connections are still ordinary application networking.

Configure the worker, then start the host

For an ordinary specialist worker, register only its own tasks and let the SDK generate the filters. The shipping project’s Program.cs can look like this. ShipOrderActivity is the project's existing unversioned TaskActivity<string, string> implementation; register its dependencies in builder.Services as usual before building the host. Set ConnectionStrings:Dts to the shared scheduler and task hub connection value. Its environment variable form is ConnectionStrings__Dts.

using System;
using Microsoft.DurableTask.Worker;
using Microsoft.DurableTask.Worker.AzureManaged;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

HostApplicationBuilder builder = Host.CreateApplicationBuilder(args);
string connectionString = builder.Configuration.GetConnectionString("Dts")
    ?? throw new InvalidOperationException("Configure ConnectionStrings:Dts.");

builder.Services.AddDurableTaskWorker()
    // Map the durable activity name to this worker's local handler.
    .AddTasks(registry =>
        registry.AddActivity<ShipOrderActivity>("ShipOrder"))
    // Request work for the registered tasks.
    .UseWorkItemFilters()
    .UseDurableTaskScheduler(connectionString);

using IHost host = builder.Build();
// Start the host and keep the worker running until shutdown.
await host.RunAsync();

The worker configuration block registers a worker; it does not start one. Build() creates the host, and RunAsync starts it and keeps it running until shutdown.

ShipOrderActivity still needs to do the real shipping work. Returning a cheerful string is not a shipping integration, however much easier that would make the example.

The other deployments use the same Program.cs pattern, with their own registrations inside AddTasks:

The validation row likewise assumes an existing ValidateOrderActivity : TaskActivity<string, string> implementation and its registered dependencies, with the invalid order behavior described above. Every participating worker explicitly calls .UseWorkItemFilters(), connects to the same scheduler and task hub, and starts its host. For this SDK configuration, that explicit builder call is the filtering setup I rely on, not an assumed default.

The parameterless call generates filters from the completed registry and worker options for that worker when its options are resolved. It does not freeze a list of whichever registrations happen to appear above the call in the file. Finish configuring the host before building and starting it.

What can you select?

The custom filter object is DurableTaskWorkerWorkItemFilters, in Microsoft.DurableTask.Worker. These are properties on that object. Each property holds a collection; the entry types are nested inside DurableTaskWorkerWorkItemFilters.

These next three are calls on the worker builder returned by AddDurableTaskWorker(). They are not members of the filter object.

UseWorkItemFilters()

Generate selections from the completed registry and worker options.

UseWorkItemFilters(filters)

Use the supplied custom object instead of generating selections. This configures whole collections, not an incremental filter addition.

UseWorkItemFilters(null)

Disable filtering. This does not pause the worker or mean “receive nothing.”

You can select multiple names and concrete versions for orchestration and activity work. An entity filter has no version property. This is not a predicate engine for payloads, tenants, instance IDs, hardware, or priorities. There is no documented name glob, regex, or prefix syntax here: use the actual registered name, not Ship*.

Creating new DurableTaskWorkerWorkItemFilters() without populating Orchestrations, Activities or Entities leaves all three collections empty. That is not a documented way to request no work, so don't rely on it as a pause switch. To stop processing, stop the worker. The .UseWorkItemFilters(null) method call above explicitly disables filtering; it is not a pause switch either.

Optionally advertise less than you register

Sometimes the code a worker contains and the work it should receive are intentionally different. Here, the registry contains ShipOrder and CancelShipment, both at version "1", but the filter advertises only ShipOrder version "1".

The optional example below replaces the worker configuration block in Program.cs: the block that starts with builder.Services.AddDurableTaskWorker() and ends with .UseDurableTaskScheduler(connectionString). Put ShippingSubset.Configure(builder.Services, connectionString, shipOrder, cancelShipment); in that position, after reading the connection string and before builder.Build(). The application supplies shipOrder and cancelShipment as its existing activity instances, with the types shown below. Keep the configuration reading, Build() and await host.RunAsync() in place; do not append a second default worker registration.

using Microsoft.DurableTask;
using Microsoft.DurableTask.Worker;
using Microsoft.DurableTask.Worker.AzureManaged;
using Microsoft.Extensions.DependencyInjection;

public static class ShippingSubset
{
    public static void Configure(
        IServiceCollection services,
        string connectionString,
        TaskActivity<string, string> shipOrder,
        TaskActivity<string, string> cancelShipment)
    {
        var filters = new DurableTaskWorkerWorkItemFilters
        {
            Activities = new[]
            {
                new DurableTaskWorkerWorkItemFilters.ActivityFilter(
                    "ShipOrder", new[] { "1" }),
            },
        };

        services.AddDurableTaskWorker()
            .AddTasks(registry =>
            {
                registry.AddActivity(
                    "ShipOrder", new TaskVersion("1"), () => shipOrder);
                registry.AddActivity(
                    "CancelShipment", new TaskVersion("1"),
                    () => cancelShipment);
            })
            .UseWorkItemFilters(filters)
            .UseDurableTaskScheduler(connectionString);
    }
}

The versioned registrations supply implementations; the ActivityFilter supplies a name and version selection. This worker can instantiate both activities, but asks to receive only the advertised one. If the workflow needs CancelShipment, an eligible worker elsewhere must serve it.

The caller also needs to request the version you advertised. For this optional example, use this separate orchestration rather than assuming the earlier unversioned OrderRouting call selects version 1:

using System.Threading.Tasks;
using Microsoft.DurableTask;

public sealed class ShippingRequest : TaskOrchestrator<string, string>
{
    public override Task<string> RunAsync(
        TaskOrchestrationContext context, string orderId)
        => context.CallActivityAsync<string>(
            "ShipOrder", orderId,
            options: new TaskOptions
            {
                Version = new TaskVersion("1"),
            });
}

Register ShippingRequest on an orchestration worker with registry.AddOrchestrator<ShippingRequest>("ShippingRequest"), that worker's own parameterless filters, and the same task hub and host startup pattern. TaskOptions.Version here requests activity version "1"; it does not register the activity on the caller.

Keep the exact name and version values aligned across the call, filter, and implementation. The SDK’s custom filter validator checks that the selected names are registered in the corresponding work kind. It does not check that every advertised version has an implementation. If you also configure UseVersioning(...), its acceptance policy must agree with those selections. A broader advertisement cannot supply missing code.

For most specialist services, the first pattern is simpler: register only what the role needs and generate its filters. Use the custom subset when that separation is actually useful, not just because another configuration object exists.

What happens between this worker and DTS?

The public SDK and documented service contract give us the useful sequence without needing to know the scheduler’s internal implementation:

1. Configure the host. Register the local implementations and choose generated or custom filters. In the GrpcDurableTaskWorker constructor, the SDK reads this worker's named filter configuration. This single line omits the surrounding constructor; name is the worker configuration name, not an activity name.

this.workItemFilters = workItemFiltersMonitor?.Get(name);

1. Request work. The SDK includes the configured eligibility in its work request. This excerpt from GrpcDurableTaskWorker.Processor.cs shows the request construction, with the omitted concurrency and capability fields marked:

return this.client!.GetWorkItems(
    new P.GetWorkItemsRequest
    {
        // Concurrency and capability fields omitted.
        WorkItemFilters = this.worker.workItemFilters?.ToGrpcWorkItemFilters(),
    },
    cancellationToken: cancellation);

1. This is worker SDK code using the public request contract, not the service’s matching implementation. The worker is asking DTS for work, not the orchestrator calling the shipping host.
2. Receive matching work. DTS delivers work according to the documented matching contract. If no eligible worker is available, that work item stays pending.
3. Run the registered implementation. In the activity handling path of GrpcDurableTaskWorker.Processor.cs, the SDK looks up a registered activity using the name and requested version from the received work. scope.ServiceProvider supplies local dependency injection. Here is the lookup excerpt, with the surrounding handler omitted:

bool found = this.worker.Factory is IVersionedTaskFactory versionedFactory
    ? versionedFactory.TryCreateActivity(
        name,
        requestedVersion,
        scope.ServiceProvider,
        out ITaskActivity? activity)
    : this.worker.Factory.TryCreateActivity(name, scope.ServiceProvider, out activity);

1. The final branch supports the other local factory interface; it is not fallback delivery to another worker. Execution happens only when found is true. Configured worker version checks still apply, and the worker reports the activity's outcome through the normal protocol.

Configure the registry and filters before starting the host. For a change, rebuild or restart the worker with the new configuration; mutating a running worker’s filter object is not a documented hot update mechanism. Set the intended filters on every worker in a mixed pool. Enabling them only on the shipper does not restrict an unfiltered worker beside it.

Here’s why that separation matters when a worker is missing. Follow the shipping step of our primary unversioned example, after validation has completed: the validation worker cannot receive ShipOrder, so that work waits until an eligible shipping worker requests it. Its result becomes history for a later orchestration pass. The columns are roles, not required separate machines; arrows show conceptual order, not measured timing. Click to enlarge.

What happens if the shipping deployment is missing?

With this filtered setup, the shipping work stays pending. The orchestration can still be Running, waiting for that result. There is no fallback that asks the validation worker to improvise a shipping implementation.

That makes deployment order important: deploy the handler before callers begin requesting it, keep task names and versions compatible, and monitor the backlog. Worker specialization gives you independent deployment and scaling choices. It does not configure an autoscaler for you.

One more boundary worth stating: filtering is routing, not authorization or tenant isolation. Worker identities and access controls still matter, as does access to whatever external systems the activities use. Sharing a task hub is not permission to skip those checks.

Keep the instance, change the next execution

Now suppose the workflow lives longer than a release cycle. We have a better implementation, but waiting for every existing instance to finish would be a fairly optimistic deployment plan.

Do we need to create a different workflow instance and somehow transfer the business identity to it?

Not necessarily. First, separate two terms. The instance is the logical workflow addressed by its instance ID. An execution is one generation of that instance. We can keep the instance while deliberately beginning another execution with a new version and a fresh execution history.

The API for that boundary is ContinueAsNew with ContinueAsNewOptions.NewVersion. It is not an assignment to context.Version, and it is not permission to change arbitrary code halfway through replay.

Version and backend scope. The released .NET v1.26.0 SDK exposes this API, and the public standalone migration sample shows the intended routing model. A compatible backend must support the version change, and the standalone worker must forward it. That is not a claim that every backend or every managed DTS deployment has the same feature rollout. Verify support in the environment you intend to use.

The next version can accept a different input shape. Here v1 waits for a count it already computes and puts that value into optional fields on v2’s input. V2 reuses it only when the count still means the same thing for the same immutable batch snapshot. Otherwise it calls its own counting activity. These fields are our chosen handoff, not a way for v2 to reach back into v1’s completed history.

In this example, v1 already calls CountRecordsV1 and ends with a continuation. We change what it passes at that existing boundary. Both counting activities are existing read only TaskActivity<string, long> implementations supplied by the application.

using System;
using System.Threading.Tasks;
using Microsoft.DurableTask;

public sealed record BatchInputV1(
    string SnapshotReference,
    bool MigrationRequested = false);

public sealed record BatchInputV2(
    string SnapshotReference,
    long? CarriedRecordCount = null,
    string? CountContract = null,
    bool MigrationRequested = false);

public sealed class BatchLoopV1 : TaskOrchestrator<BatchInputV1, long>
{
    public override async Task<long> RunAsync(
        TaskOrchestrationContext context, BatchInputV1 input)
    {
        if (input.MigrationRequested)
        {
            throw new NotSupportedException(
                "The migration payload returned to v1. Check version support.");
        }

        // This activity call was already part of the old execution's sequence.
        long recordCount = await context.CallActivityAsync<long>(
            "CountRecordsV1", input.SnapshotReference);

        context.ContinueAsNew(new ContinueAsNewOptions
        {
            NewVersion = "2",
            NewInput = new BatchInputV2(
                SnapshotReference: input.SnapshotReference,
                CarriedRecordCount: recordCount,
                CountContract: "record-count/v1",
                MigrationRequested: true),
            PreserveUnprocessedEvents = true,
        });

        return recordCount;
    }
}

public sealed class BatchLoopV2 : TaskOrchestrator<BatchInputV2, long>
{
    public override async Task<long> RunAsync(
        TaskOrchestrationContext context, BatchInputV2 input)
    {
        if (input.CarriedRecordCount is long carried
            && carried >= 0
            && input.CountContract == "record-count/v1")
        {
            return carried;
        }

        long recordCount = await context.CallActivityAsync<long>(
            "CountRecordsV2", input.SnapshotReference);

        if (recordCount < 0)
        {
            throw new InvalidOperationException(
                "The counting activity returned an invalid negative count.");
        }

        return recordCount;
    }
}

long? makes missing different from zero. With the matching contract, 0 is a valid carried count and is reused. A missing or negative count, or a different contract, sends v2 to CountRecordsV2. A negative result from that activity fails instead of being accepted as a count.

CountContract is an application compatibility rule: the value must describe this same immutable snapshot using our record-count/v1 definition. The tag is not an SDK check or proof that the input is trustworthy. If the meaning of the count, schema, or business rules changes, validate or translate the carried data, or change the accepted contract and recompute. Presence alone is not enough.

The payload in NewInput must match v2's serialized input contract. The SDK adapter deserializes the input into the selected implementation's input type. It is not a CLR cast from BatchInputV1 to BatchInputV2, or an automatic conversion between arbitrary schemas. V1 constructs the new shape, and its fields, value types and serializer settings must agree with v2.

V2 can take different activity paths because it starts another execution rather than replaying v1’s completed history. Its code must still be compatible with its own history as it accumulates. The old local variables, tasks and continuation do not cross the boundary.

Both input types retain SnapshotReference and MigrationRequested. The guard catches a migrated payload routed back to v1 only if the payload and marker arrive and the converter can read them as BatchInputV1. A stricter converter can fail before the guard. The public migration example highlights the backend dependency; this check is not proof of backend or serializer compatibility.

Accepting a version is not the same as implementing it

We need both the worker’s version acceptance policy and the right registered implementation. A worker accepting older work does not magically contain the older code.

Here is the configuration inside an existing Generic Host setup. builder is the host builder, and connectionString identifies the same task hub. countRecordsV1 and countRecordsV2 are the application's existing TaskActivity<string, long> instances, with their real dependencies supplied by the application:

using Microsoft.DurableTask;
using Microsoft.DurableTask.Client;
using Microsoft.DurableTask.Client.AzureManaged;
using Microsoft.DurableTask.Worker;
using Microsoft.DurableTask.Worker.AzureManaged;
using Microsoft.Extensions.DependencyInjection;

builder.Services.AddDurableTaskWorker(worker =>
{
    worker.AddTasks(registry =>
    {
        registry.AddOrchestrator("BatchLoop", new TaskVersion("1"),
            () => new BatchLoopV1());
        registry.AddOrchestrator("BatchLoop", new TaskVersion("2"),
            () => new BatchLoopV2());
        registry.AddActivity("CountRecordsV1", new TaskVersion("1"),
            () => countRecordsV1);
        registry.AddActivity("CountRecordsV2", new TaskVersion("2"),
            () => countRecordsV2);
    });
    worker.UseVersioning(new DurableTaskWorkerOptions.VersioningOptions
    {
        Version = "2",
        DefaultVersion = "2",
        MatchStrategy =
            DurableTaskWorkerOptions.VersionMatchStrategy.CurrentOrOlder,
        FailureStrategy =
            DurableTaskWorkerOptions.VersionFailureStrategy.Reject,
    });
    worker.UseWorkItemFilters();
    worker.UseDurableTaskScheduler(connectionString);
});
builder.Services.AddDurableTaskClient(
    clientBuilder => clientBuilder.UseDurableTaskScheduler(connectionString));

Both classes use the same logical name, BatchLoop, with separate version registrations. This worker's explicit CurrentOrOlder policy permits version 1 work alongside version 2, and the registry supplies both implementations. Those are two separate requirements.

In this SDK path, an activity call without an explicit TaskOptions.Version inherits the orchestration execution's version. The two count activities are therefore registered at versions "1" and "2" to match their callers.

sends mismatched work back for another attempt. It does not upgrade the work or guarantee that an eligible worker exists. Keep support for version 1 while executions still need it.

After starting the host, an already resolved DurableTaskClient client can schedule the illustrative version 1 instance:

string instanceId = await client.ScheduleNewOrchestrationInstanceAsync(
    "BatchLoop",
    input: new BatchInputV1("snapshot-42"),
    options: new StartOrchestrationOptions
    {
        Version = new TaskVersion("1"),
    });

The starter’s await returns the instance ID after scheduling, not after the workflow finishes. "snapshot-42" is an illustrative reference to immutable data, not a claimed count. The continuation keeps that instance ID and creates a new execution at version 2, using the BatchInputV2 payload v1 constructed.

A separate instance can start directly at version "2" with input: new BatchInputV2("snapshot-42") and Version = new TaskVersion("2") in its start options. With no carried count or contract, it calls CountRecordsV2. That is a direct start, not the migration of the instance above.

Read the next diagram as data flow between two executions of one instance. Follow the count from the awaited v1 activity into NewInput, then follow v2's reuse check. The fork is application logic: use a compatible carried value, including zero, or run CountRecordsV2. No completed v1 history or live CLR objects move along the arrows. Click to enlarge.

Put the boundary where the workflow already has one

V1 already counts the snapshot before reaching its existing final continuation call. The migration changes the input it carries across that boundary and the requested next version. Preserve the earlier durable calls, their ordering, names, versions, inputs and compatible serialization behavior. Do not insert new work into an earlier replay just because v2 will use a different flow.

Finish and await work whose results matter before continuing. Outstanding operation results can be discarded at the boundary. That does not mean an activity already affecting an external system has been cancelled or rolled back.

PreserveUnprocessedEvents = true is explicit here and is also the default for this .NET option. It preserves unprocessed external events for the next execution; false discards them. The new code must understand both the carried input and any preserved event payloads. Mapping NewInput does not convert those event payloads. Fresh history does not mean an empty workflow with no starting records, and it does not promise that the service purges every diagnostic record.

Finally, return recordCount immediately after ContinueAsNew exits v1's method. On the supported continuation path, the continuation action and NewInput, not this return value, define the next execution. V1 does not call v2 directly or wait for it to finish. V2's later return completes this finite example.

This is an action from a running orchestration, not an operator command for reviving a failed one. Putting it in finally is not a reliable plan for recovering an uncaught failure.

Which brings us to the third capability.

Repair a failed instance without rerunning every successful step

Consider an import workflow: extract a batch, transform it, then load it into a destination. Extraction and transformation succeed and their results are recorded. Loading then fails because of a configuration problem.

You fix the configuration. Do you have to request the whole import again?

For an eligible failed instance, rewind provides a more targeted recovery path. It is an explicit operator request after a failure, not ordinary replay, not a retry policy, and not a version migration.

Rewind is more than setting Failed back to Running. The .NET client contract describes a new execution ID for the same instance, with recovery history that lets failed work run again. Compatible orchestration code runs against that history; its old stack is not resumed.

Here is the workflow we’ll follow. The activity results are references to the extracted and transformed data, not large datasets carried through the example:

using System.Threading.Tasks;
using Microsoft.DurableTask;

public sealed record LoadRequest(string BatchId, string ManifestReference);

public sealed class ImportBatch : TaskOrchestrator<string, string>
{
    public override async Task<string> RunAsync(
        TaskOrchestrationContext context, string batchId)
    {
        string extracted = await context.CallActivityAsync<string>(
            "ExtractBatch", batchId);
        string transformed = await context.CallActivityAsync<string>(
            "TransformBatch", extracted);
        return await context.CallActivityAsync<string>(
            "LoadBatch", new LoadRequest(batchId, transformed));
    }
}

Register ImportBatch and the named activity implementations on compatible workers in the same hub. Their actual I/O is omitted. This illustration uses no durable timers and no retry policy that creates durable timers: the current management documentation excludes orchestrations that use durable timers from rewind.

Suppose the loading failure makes this instance Failed. Before requesting recovery, fix the cause and reconcile any partial writes. The loader may have changed the destination before reporting failure. The destination has not agreed to forget that just because we found a useful SDK method.

For this import, recovery looks like this:

1. ExtractBatch's successful recorded data reference is reused by replay, without another extraction.
2. TransformBatch's successful recorded manifest reference is reused, without another transformation.
3. The failed LoadBatch work can be attempted again. Partial destination writes are still there unless the application has reconciled or compensated for them.

The SDK describes replacement history excluding failed activities and sub orchestrations, not clearing every successful result. Relevant failed sub orchestrations can be rewound recursively, and new records are appended as work runs again. This describes the recovery history contract, not physical database deletion or audit retention.

The caller chooses which eligible failed instance to recover, when to request it, and a diagnostic reason. RewindInstanceAsync has no parameter for selecting activities, an event range, an arbitrary checkpoint, replacement input or output, or individual successful results to discard. It is not a history editing API.

An administrative application or recovery automation can use this helper:

using System;
using System.Threading;
using System.Threading.Tasks;
using Microsoft.DurableTask.Client;

public static class ImportRecovery
{
    public static async Task RequestRewindAsync(
        DurableTaskClient client,
        string instanceId,
        string reason,
        CancellationToken cancellation = default)
    {
        OrchestrationMetadata? instance = await client.GetInstanceAsync(
            instanceId, getInputsAndOutputs: false,
            cancellation: cancellation);

        if (instance?.RuntimeStatus != OrchestrationRuntimeStatus.Failed)
        {
            throw new InvalidOperationException(
                "Rewind requires an existing Failed instance.");
        }

        await client.RewindInstanceAsync(
            instanceId, reason, cancellation);
    }
}

reason should describe the repair or reconciliation that actually happened. The state check is a useful preflight, not an atomic lock: another operator can act between the read and the request. The backend still validates eligibility. Rewind is for Failed instances, not Running, Pending, Terminated, or Completed ones.

That does not restore a saved instruction pointer, and it does not automatically change the workflow’s version. Keep the earlier action sequence replay compatible, and keep the referenced data and serialization contracts available.

The next diagram separates the failed execution from the operator’s repair and rewind request. The initial orchestration passes are condensed; the recorded successes are shown so it is clear what the later execution can reuse. There is no arrow that undoes the destination’s writes. Click to enlarge.

The important operational detail is what the management await means. RewindInstanceAsync completes when the request is enqueued, not when the import is finished. Query the instance afterward. Cancelling that call cancels enqueueing; it does not undo a rewind that was already accepted. If the request's outcome is uncertain, inspect the instance rather than blindly repeating the operator action.

This example is scoped to the public .NET API and a backend that supports rewind. Other clients, hosts, or providers should not be assumed to have identical support just because they share a backend name. An unsupported backend can reject the operation.

And please keep a business key for the loading operation. BatchId gives the destination something stable to deduplicate on, if the integration actually implements that contract. Merely passing the field does not create idempotency. An execution ID is a poor replacement here because rewind creates another execution.

Fix what broke, reconcile what may already have happened, then ask the failed instance to continue. Rewind is not external rollback, and a new attempt still has to handle its effects correctly.

The workflow is not the deployment

These capabilities solve different problems. Filtering chooses which workers can execute work. A continuation boundary deliberately moves the same instance into another execution and implementation. Rewind lets an operator request recovery after fixing an eligible failure, using the successful history that is still useful.

None of them removes the need for compatible code and data contracts, or careful handling of external effects. They give us better ways to manage those boundaries without building another workflow coordinator around the workflow coordinator.

That is what I meant by hidden gems. Once history and actions stop looking like magic, the interesting part is what they let us change safely around a running system.

Next, I’ll look at AI agents calling models and tools, and why durability matters when several agents coordinate their work. We’ll look at keeping progress and handling repeated effects in that setting too, with multiagent systems like Squad in mind :-)

Originally published at https://www.tamirdresher.com on September 24, 2026.

I’ve written about Durable Task and Durable Task Scheduler (DTS) before, including my DTS introduction. It still surprises me how many developers don’t really know them, let alone use them. They tackle failure and recovery problems every distributed system has to deal with. Yes, even yours with one microservice and a database.

Plenty of us can quote the fallacies of distributed computing. Quoting them, though, isn’t the same as handling the failures they describe in our code. Knowing that the network can fail doesn’t tell a workflow halfway through an order what to do when it does. Durable Task helps recover workflow progress. It doesn’t solve every problem in a distributed system.

I find the Durable Task Framework fascinating, but I think the apparent “magic” also puts people off. How can an orchestration make progress after its original execution in memory is gone? In this post, I want to go under the hood and show you the secret sauce.

And we can actually look. The Durable Task Framework/Core and modern .NET SDK are open source. We’ll follow their worker implementation, not the private internals of the managed DTS backend.

I’ll follow the standalone .NET gRPC worker in Durable Task SDK v1.26.0, which depends on DurableTask.Core 3.9.0. That’s the implementation path for this walkthrough, not a claim about every deployed package combination or Azure Functions configuration.

A small orchestration to follow

Here is the complete orchestration declaration we’ll follow, based on the public source. [DurableTask] marks the class for the optional generator described below. Activity implementations and registration are omitted; this is a teaching example, not an executed sample.

using System.Threading.Tasks;
using Microsoft.DurableTask;

[DurableTask(nameof(OrderFlow))]
public sealed class OrderFlow : TaskOrchestrator<string, string>
{
    public override async Task<string> RunAsync(TaskOrchestrationContext context, string input)
    {
        /* L1 */ string orderId = input;
        /* L2 */ decimal subtotal = await context.CallActivityAsync<decimal>("ReadSubtotal", orderId);
        /* L3 */ decimal total = subtotal + 10m;
        /* L4 */ string receipt = await context.CallActivityAsync<string>("ChargeOrder", new { orderId, total });
        /* L5 */ return receipt;
    }
}

If you’d rather avoid strings, the optional Microsoft.DurableTask.Generators package generates strongly typed helpers for activities and orchestrations defined as classes, including calling sub orchestrations and starting orchestrations. The SDK's typed example uses CallSayHelloTypedAsync and ScheduleNewHelloCitiesTypedInstanceAsync. That example uses Functions, but the generator also supports standalone workers.

The generated methods still delegate to CallActivityAsync, CallSubOrchestratorAsync, or ScheduleNewOrchestrationInstanceAsync. They add convenience and type safety at compile time, not a different replay or history model. The generator is versioned separately and marked as preview in the code we're following; that is not a claim about tested package compatibility or the latest release. I'm keeping the strings here so we can see the scheduling identities.

For the walkthrough, the input is "order-42", ReadSubtotal produces 100m, and ChargeOrder produces "receipt-7" after receiving a total of 110m. Those values are illustrative, not measured results.

The extra 10m gives us a local calculation to track. When a later invocation reaches L3, does it recover a saved total variable, or calculate the value again?

It calculates it again. Let’s see why.

Who calls RunAsync, and what happens at await?

Before L1 can run, some application code has to request an orchestration instance. Here, client is an already configured DurableTaskClient, connected to the same DTS task hub as a running worker. That worker has OrderFlow, ReadSubtotal, and ChargeOrder registered.

string instanceId = await client.ScheduleNewOrchestrationInstanceAsync(
    nameof(OrderFlow), input: "order-42");

The await in this starter waits for successful scheduling and returns the instance ID. It does not wait for the order to finish, and it does not call RunAsync on the starter's call stack. The [DurableTask] attribute on our class is a generator marker, not something that starts the method.

DTS schedules the work. Your worker runs the C#. The worker receives orchestration work and the available history from DTS. It looks up the registered orchestrator to find the code for OrderFlow. The starter and worker can be hosted in the same process; these are different roles, not a requirement for separate machines.

Inside the worker, a library runner reads the available history and drives the orchestration forward. Its .NET name is TaskOrchestrationExecutor. This is library code inside the worker, not DTS itself or a class you have to write. When it processes the starting ExecutionStarted record in the history, the SDK adapter deserializes the input and calls RunAsync with the context and "order-42". Now L1 runs.

The runner takes the recorded history and runs or replays our code as far as that history allows, collecting instructions for what should happen next. It returns the remaining instructions to the worker to send to DTS.

Core calls these instructions actions (the orchestration’s decisions about what should happen next) and returns them in OrchestratorExecutionResult.Actions.

The runner has just read a record saying this instance started and used it to enter RunAsync. Records like this are history events. They are the incoming history the runner uses to reconstruct progress, not C# events that application code subscribes to.

When RunAsync requests ReadSubtotal for "order-42", the call adds a request to the orchestration context. This local action describes work to ask for next. Creating it locally does not durably record the request. When a new activity is scheduled, a history event records that fact; another records its successful result, 100.

RunAsync itself returns the receipt, not those actions. The runner executes or reconstructs the method against the incoming history and returns a result object containing the actions still outstanding. During replay, matching recorded schedules removes candidate actions, so they are not sent again.

At L2, CallActivityAsync creates a scheduling instruction and a local result task. The instruction, called an action, says: ask DTS to run ReadSubtotal with "order-42" as input. The ordinary .NET task is what L2 awaits to get the subtotal. Both exist in worker memory. Creating them does not mean the activity has been dispatched or that DTS has durably recorded the request.

If that result task is incomplete, ordinary C# await yields without blocking its thread. This pauses the method, not the runner's history processing. An already completed task need not make the method pause.

The runner keeps reading any available history. A completion event later in this same pass can supply a result and let the method continue. A previously recorded schedule can be matched to the reconstructed instruction instead of sent again.

After processing the pass’s available history, the runner returns the remaining actions in OrchestratorExecutionResult.Actions, and the worker sends them to DTS. For new activity work, those actions request scheduling. Finishing this response is not finishing the order workflow. One pass can return several actions; an await is not itself a durable checkpoint.

Here is the actual return statement from Durable Task Core’s TaskOrchestrationExecutor.ExecuteCore, with the surrounding method omitted. The Actions property contains the remaining actions collected in the orchestration context:

return new OrchestratorExecutionResult
{
    Actions = this.context.OrchestratorActions,
    CustomStatus = this.taskOrchestration.GetStatus(),
};

Once a new activity is scheduled, a separate activity handler in a worker runs it and reports its outcome. A recorded outcome can then be supplied as history for another orchestration pass. In the reconstruction path below, RunAsync starts again and history resolves the new local task at L2. The activity is not calling back into the old suspended method.

Here’s the same flow as a conventional sequence diagram, read from top to bottom. The client gets an instance ID, the first worker pass asks for ReadSubtotal, and its recorded result lets a later pass recompute 110m and ask for ChargeOrder.

The columns are roles, not necessarily separate processes. Activation bars on the orchestration worker’s lifeline mark separate passes; a later pass may use the same worker process or a different one. Solid arrows show messages or calls, and dashed arrows show replies. This is a conceptual sequence for the reconstruction path, not a network capture or timing trace. The result reaches the later pass through history, not through a callback to the original waiting task.

What survives is history, not the suspended method

DTS keeps each instance’s history as part of the task hub’s managed state, and a worker receives those records to reconstruct execution. We are describing that contract, not the service’s database or storage layout.

For ReadSubtotal, two records in our example say:

1. Activity 0, named ReadSubtotal, was scheduled with input "order-42".
2. Activity 0 completed successfully with result 100.

These are conceptual descriptions, not an exact data format or the whole history. Starting and housekeeping records are omitted. The 0 identifies the activity, not the record’s position in history.

What about loops? Calling the same activity in a for or foreach is fine if, for the same inputs and recorded history, the loop reproduces the same ordered durable calls. Each call gets its own sequence ID from the runtime counter, and replay recreates those IDs at the same calls. Core matches the ID, action type, and activity name, not the name alone. These IDs are not loop indices or global counters for an activity: timers, sub orchestrations, and event sends share the sequence. Changing iteration order or which durable calls occur can break matching; the same number of calls is not enough.

The first record tells the runner that this activity was already scheduled. During replay, L2 creates a candidate action again. The runner matches the record against that action and removes the candidate from its outgoing actions, rather than scheduling ReadSubtotal anew. This record does not say the activity finished, and matching it does not complete the await. Core represents this scheduling record with the .NET class TaskScheduledEvent.

The second record supplies the successful outcome. Processing it provides the result to the reconstructed local task, so RunAsync can continue. Core represents this completion record with TaskCompletedEvent. Its TaskScheduledId is 0 here, connecting the result 100 to the activity that was scheduled.

For this call, the action requests ReadSubtotal, the history events record its scheduling and result, and the Task is the local object L2 awaits. Those event classes belong to the runtime's bookkeeping. Application code calls the orchestration and activity APIs; it normally does not construct history objects or raise C# events. DTS preserves history data, while the worker interprets it using local .NET objects.

On reconstruction, L2 assigns the recorded result to a fresh subtotal variable as 100m. L3 calculates 110m again. The old variable slots and Task objects are not restored. That does not mean 110m can never appear in history: L4 sends it as part of ChargeOrder 's input. It can be recorded there as serialized input data, not as a snapshot of every local variable or the suspended CLR stack.

No orchestration thread is parked waiting for the activity. The worker process doesn’t have to stop at every await either. Ending a pass through history isn’t the same as ending a process.

Does Worker 1 keep all those waiting tasks in memory?

Now, you’ve all heard something I didn’t say. Metaphorically, since you’re reading this. You’re probably thinking:

“Worker 1 ran the first pass. If Worker 2 resumes the orchestration after the activity finishes, what happens to Worker 1 and the TaskCompletionSource it created?"

“If Worker 1 handles 10,000 orchestration instances, each waiting for an activity that takes a week, does it keep 10,000 pending tasks and state machines in memory for that whole week?”

No. A waiting workflow does not require its original invocation or local tasks to stay alive. An ordinary async await does not park a thread with an OS stack either. Durable execution lets the workflow make progress without its original continuation in memory.

Worker 1 requests ReadSubtotal by returning a scheduling action. Once new actions and history have been committed, the runtime can unload that execution. When the activity reports 100 and the result is durably recorded, DTS keeps it in the instance history. The next pass needs that history, not Worker 1's memory.

An eligible Worker 2 connected to the same task hub, with compatible code registered, can receive that work. It reruns RunAsync and creates a new local Task and TaskCompletionSource. The TaskScheduled record matches the reconstructed request; the TaskCompleted record, through TaskScheduledId, supplies 100 to Worker 2's new task. Nothing needs to find Worker 1's original helper or call its old continuation. The durable activity ID is not a .NET Task.Id or a memory address. Switching workers alone does not rerun an activity whose successful result is already recorded.

The standalone gRPC pass can finish while the workflow waits. It does not require those 10,000 original orchestration task objects to be retained for a week. Once unreferenced, they can be collected. That does not promise immediate garbage collection, zero overhead, free waiting, or scaling to zero. Some hosts cache execution state, but recovery does not depend on the original worker or cache surviving.

There is one important distinction: a waiting orchestration is not the same as an activity that keeps running for a week. An activity executing, awaiting I/O, or sleeping for that long can retain activity worker resources. Its execution is not checkpointed in the middle, and failure can require another attempt with the usual side effect concerns. If the work is only waiting for time to pass or an outside answer, use a durable timer or wait for an external event instead.

Replay, line by line

Let’s walk the five lines through three logical activations. Here, an activation means one pass in which the worker processes orchestration history and produces actions.

This simplified trace follows the public source. It is not three measured deliveries or a promise about physical service batches, and housekeeping events are omitted. With no other durable operations in this example, the inspected Core counter assigns activity IDs 0 and 1. Those IDs correlate actions with scheduled tasks; they are not positions in the event history.

Read each strip from top to bottom: code position on the left, history cursor on the right, and pending actions, open result tasks, and locals below. A TaskCompletionSource is the helper supplying the awaited task's result. These are settled teaching checkpoints based on the public source, not a live debugger recording. Click any strip to open the full SVG and enlarge it.

The runner reads Earlier history first to reconstruct this execution. New events for this step are the events newly delivered for the current step; “new” does not mean their data is unrecorded. The read counts track events, not unfinished activities, and records already read stay stored.

A: ask for the subtotal

The starting event supplies the input. L1 reads "order-42". L2 calls ReadSubtotal, creating action 0 and an open local task. There is no result yet, so the method yields at L2. L3 through L5 have not executed.

The executor returns the new action to schedule ReadSubtotal with ID 0. Once that action is accepted, scheduling history records the operation. The activity runs separately and reports its outcome.

Notice what is missing: no durable record of a local variable called subtotal with an instruction pointer beside it.

B: use the subtotal, then ask for the charge

Now the available history includes the scheduled operation and a completion representing 100m.

The orchestration starts at L1 again. At L2, its code creates a candidate action 0 and another local task. Replay is not simply “look in a dictionary and instantly return a completed task.”

Core’s HandleTaskScheduledEvent matches the recorded scheduling event against the reconstructed action using the sequence ID, action kind, and activity name. It then removes that candidate from the outbound action map.

Matching the schedule removes the recreated action from the outgoing list; it does not, on its own, supply the result.

The completion handler uses the recorded activity ID to find its open local task, supplies the serialized result, and then removes the entry. This excerpt from Durable Task Core’s TaskOrchestrationContext.HandleTaskCompletedEvent keeps the existence check; the method wrapper and duplicate event branch are omitted:

int taskId = completedEvent.TaskScheduledId;
if (this.openTasks.ContainsKey(taskId))
{
    OpenTaskInfo info = this.openTasks[taskId];
    info.Result.SetResult(completedEvent.Result);

    this.openTasks.Remove(taskId);
}

The order matters: SetResult runs before removal, and the continuation can run inside that call. The snapshots show the settled state afterward.

When the executor processes the completion for ID 0, HandleTaskCompletedEvent sets the reconstructed completion source's result. The async chain can then continue within this same activation, deserialize the value, and assign subtotal = 100m. Core's orchestration synchronization context and synchronous task scheduler keep those continuations inside this pass through history.

Why the context matters. When an ordinary await needs to suspend, a captured SynchronizationContext controls how its continuation, the code after that await, is scheduled. The runtime installs its own context and uses its scheduler to keep that code in the runner's controlled history processing. ConfigureAwait(false) opts out of that capture and can escape that path, so don't add it inside orchestrator code. It does not necessarily switch threads on every await.

Keeping this context does not promise the same OS thread or process across activations. Each reconstruction has its own objects and context. Neither normal capture nor ConfigureAwait(true) makes Task.Delay or HTTP I/O durable. Activities follow ordinary .NET async guidance separately.

L3 calculates 100m + 10m, producing 110m. L4 calls ChargeOrder, creating the new action 1 with { orderId, total }. Its result is not available yet, so the method yields again.

The outstanding action is now Schedule 1: ChargeOrder. There is no new Schedule 0 merely because L2 executed again.

C: use both recorded operations, then return

With the second activity’s result available, the method again starts at L1.

L2 reconstructs action 0; history matches its schedule and supplies 100m. L3 calculates 110m again. That is a fresh calculation, not a saved local restored from a checkpoint.

L4 reconstructs action 1. Its recorded scheduling event removes the candidate from the outbound map, and its completion supplies "receipt-7". L5 returns that receipt. The executor can now emit orchestration completion, rather than a new activity schedule.

In this pass, result 0 comes from earlier history, while result 1 can be newly delivered for this step. Neither activity is newly scheduled just because both call sites ran again.

Zoom in on C. Read the Earlier history cards from oldest to newest. Earlier events left counts the earlier records still to read, moving from 4 to 0 as the cursor advances. It is not a count of unfinished activities, and reading a card does not delete its stored record. Matching a schedule removes a candidate action, while processing its completion resolves the open task. One completion can advance several source lines, and an await can span several events.

Only after the last record in the earlier history ( TaskScheduled for activity 1) has been processed does the executor set IsReplaying to false, before processing new events. L4 is still waiting on task 1. The newly delivered completion then resolves it, and L5 returns the receipt.

Reading all the earlier history does not mean every task has finished or every newly delivered event has been processed. It isn’t a general signal of workflow completion, and another activation can replay again.

Now suppose history contains TaskScheduled(0) but no completion or failure for it. Replay still matches and removes the candidate scheduling action, but the reconstructed task stays incomplete. Pending does not mean "schedule it again because we replayed." Redelivery of an outstanding activity is a separate delivery concern.

No crash is needed for any of this. In the reconstruction path, normal activity results drive the same processing of history. Replay is part of normal progress, not just recovery.

Can the runtime keep the local execution around?

The available caching path depends on the host and its configuration.

The standalone gRPC path used above constructs a new TaskOrchestrationExecutor. A separate runner used primarily by Azure Functions isolated hosting has an extended sessions path that can reuse an executor and process new events. Functions also documents instance caching that varies by provider.

That tells us what those paths can do, not which options a particular deployment enables. The diagrams show reconstruction, not a promise that every host unloads after every await or replays everything for each notification.

The code restrictions stop looking arbitrary

These mechanics explain why orchestrator code has to be deterministic.

For a given recorded history, the orchestration must reconstruct compatible durable operations in the required order. If a read of the wall clock, an unseeded random value, or an external response changes which operation comes next, the old history may no longer fit the new execution. Core performs useful matching checks, but that doesn’t guarantee it will detect every possible nondeterministic change for you.

For time, use context.CurrentUtcDateTime rather than reading the wall clock directly. For a GUID that is safe to replay, use context.NewGuid(). Replay safety is the point: NewGuid() is not a source of fresh, unpredictable randomness on every replay. If you need an external or independently random value, obtain it through an activity so its result can participate in the recorded history.

The same distinction applies to I/O. Awaiting HttpClient directly inside an orchestrator does not turn the HTTP request into a durable activity or record its response for replay. Put that I/O inside an activity. An HttpClient used by the activity implementation is a different case from an orchestrator awaiting it directly.

For a durable wait, the modern API here is context.CreateTimer(...), with TimeSpan or DateTime overloads and a cancellation token. Ordinary Task.Delay is normally nonblocking, but it is not a durable timer. The orchestrator code constraints follow from this boundary: ordinary async work does not become durable simply because it appears inside an orchestration method.

And then there are logs. L3 can execute more than once, so an ordinary log beside it can appear more than once. context.CreateReplaySafeLogger("OrderFlow") returns a logger that suppresses writes while IsReplaying is true. That cuts replay noise. It doesn't guarantee that each audit event is recorded exactly once.

Do not “fix” repeated execution by placing durable actions inside if (!context.IsReplaying). Replay needs to reconstruct those actions to match the history. Suppressing the log is useful. Suppressing the operation changes the program.

Replay is not retry

There are three mechanisms worth keeping separate:

1. Replay of a recorded outcome: orchestration code executes again and history resolves its reconstructed task. That does not itself request another execution of a successfully recorded activity.
2. A configured retry after failure: a recorded failure faults the reconstructed task. An explicitly configured retry policy or handler can deliberately arrange another attempt; the SDK’s retry branches are explicit.
3. Redelivery of unacknowledged activity work: the activity execution contract is at least once. An activity may execute again when its completion was not durably recorded, even if some external work already happened.

The boundary to watch is inside ChargeOrder.

Suppose the payment system accepts the charge for 110m. Before the activity's completion is durably recorded, its worker fails. The payment system has done something real. The orchestration history does not yet contain a successful result it can replay.

When the activity runs again, it can reach that payment system again. History can recover orchestration progress; it cannot close the gap between the external effect succeeding and the activity outcome being durably recorded. Using a durable framework doesn’t put the payment system and the scheduler in one atomic transaction.

The charge operation still needs idempotency or deduplication at the business level. For example, use a stable key for this particular operation, honored by an integration that supports it. The key must distinguish a duplicate attempt from another legitimate charge.

Now move the failure to after the successful completion has been durably recorded. If worker memory is lost then, replay can supply "receipt-7" and advance to L5 without scheduling that completed logical activity again. That recovers a recorded outcome. It doesn't promise every external effect happened exactly once.

Back to the debugger

The await is still ordinary C#. The framework call creates an action to perform a durable operation and a local task. History can later match that operation and complete a reconstructed task, letting compatible code calculate its locals and continue.

So when you see L2 or L3 execute again, ask two separate questions: which source lines are being replayed, and which actions are actually being emitted?

That is the secret sauce: the method can keep making progress without keeping its original stack alive. Its activities still have to handle external effects correctly.

Once you can follow the history and the actions, the apparent magic becomes something you can reason about in your own code. In upcoming posts, I’ll dig into more of the overlooked capabilities and hidden gems in the Durable Task ecosystem.

Originally published at https://www.tamirdresher.com on September 20, 2026.

What happens from the moment a payment is initiated until it reaches its destination? Why is the traditional way to make a cross-border payment so expensive and what is the solution? Payoneer architects Yonatan and Tamir were hosted in the Israeli podcast “reversim” and talked about the journey a payment goes through under-the-hood of the Payoneer system.

Spotify: https://open.spotify.com/episode/2UAZvxBuI37TzCASPhfVvB

Reversim: https://www.reversim.com/2023/01/453-payment-journey-in-fintech-company.html

A payment journey in a FinTech company — Reversim podcast (Hebrew) was originally published in Payoneer Engineering, Product & AI Transformation on Medium, where people are continuing the conversation by highlighting and responding to this story.

Story 1 was originally published in tamirtest on Medium, where people are continuing the conversation by highlighting and responding to this story.