<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Long Quanzheng on Medium]]></title>
        <description><![CDATA[Stories by Long Quanzheng on Medium]]></description>
        <link>https://medium.com/@qlong?source=rss-56b397853e0d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*4sQ6--DaNZpDzNF4.jpg</url>
            <title>Stories by Long Quanzheng on Medium</title>
            <link>https://medium.com/@qlong?source=rss-56b397853e0d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 16:22:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@qlong/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Temporal on Aurora Limitless: Postgres That Scales Out]]></title>
            <link>https://medium.com/@qlong/temporal-on-aurora-limitless-postgres-that-scales-out-6c033642b5c1?source=rss-56b397853e0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c033642b5c1</guid>
            <dc:creator><![CDATA[Long Quanzheng]]></dc:creator>
            <pubDate>Fri, 16 Oct 2026 01:35:35 GMT</pubDate>
            <atom:updated>2026-10-16T01:35:35.243Z</atom:updated>
            <content:encoded><![CDATA[<p><em>We </em><a href="https://github.com/superdurable/xtemporal"><em>ported</em></a><em> Temporal’s SQL persistence to Aurora PostgreSQL Limitless, tested every database behavior Temporal relies on, and worked out what it takes to run 100,000 actions per second.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xvhkpel-KTOesGxazyqIVA.png" /></figure><p>Every Temporal cluster that grows long enough ends up having the same conversation about the database.</p><p>Temporal writes every workflow state transition to its persistence store. On PostgreSQL, all of those writes go to one primary. You scale that primary up, then up again, and at some point there is no larger instance to move to. The standard answer at that point is Cassandra, which scales out but comes with its own operational world. Plenty of teams chose Postgres specifically so they wouldn’t have to live in that world.</p><p>Aurora PostgreSQL Limitless gives you a different option. To the application it looks like Postgres: same wire protocol, same SQL, same drivers. Underneath, tables are hash-partitioned across shards that each have their own compute, and a tier of routers plans queries and coordinates transactions. When you need more write capacity, you add shards.</p><p>We wanted to know if Temporal could run on it with every corrnectness guarantee. Turned out it can. The data models line up well, the correctness properties hold under adversarial testing, and the scaling path from a two-shard cluster to a hundred shards is straightforward. This post covers why we expected it to work, how we checked, and how we size a deployment for 100K actions per second.</p><h3>Temporal is already sharded</h3><p>Temporal splits all workflow state into a fixed number of history shards (numHistoryShards). Each history shard is owned by exactly one history host at a time, and every hot-path table, including executions, history_node, timer_tasks, transfer_tasks and buffered_events, has shard_id in its primary key. A normal workflow update touches one shard_id and nothing else.</p><p>That maps directly onto Limitless. We shard every history table on shard_id and put all of them in one collocation group. All rows for a given Temporal shard then live on the same Limitless shard, and a workflow transaction becomes a single-shard transaction: the router forwards it, the shard executes and commits it locally, and there is no two-phase commit anywhere on the hot path.</p><p>One Temporal history shard maps to one Limitless shard <em>History tables are collocated on </em><em>shard_id, so the write path for a workflow never crosses shards.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a9kRxnexn7dLL2g6Zgas1g.png" /></figure><p>This is the part that makes horizontal scaling real. Total write capacity becomes roughly “what one Limitless shard can do” times “number of Limitless shards”, because Temporal never asks two shards to agree on anything during normal operation.</p><p>The rest of the schema follows the same idea:</p><ul><li>Matching task tables are sharded on range_hash, the same key Temporal already partitions task queues by.</li><li>Task-queue user data and build-ID mappings are collocated on task_queue_name, so updating both stays single-shard.</li><li>Queue v2 is sharded on (queue_type, queue_name).</li><li>Small, read-mostly tables (namespaces, cluster_metadata_info, nexus_*, queue v1) are reference tables, replicated to every shard.</li></ul><p>Only two column-level changes were needed compared to the upstream PostgreSQL v12 schema. buffered_events.id used to be a BIGSERIAL; the plugin now assigns it per workflow while holding the execution row lock, which gives the ordering Temporal expects regardless of which router a connection lands on. And the unique index on cluster_membership now includes its shard key. Everything else is upstream SQL, unchanged.</p><h3>What has to be true for correctness</h3><p>Scaling was the easy argument. The harder question was whether a distributed database preserves the exact locking behavior Temporal’s SQL plugin depends on. We wrote those behaviors down and tested each one.</p><p>The most important is shard fencing. When a history shard moves to a new host, the new owner increments range_id on the shard&#39;s row. Every write from the old owner starts with SELECT range_id ... FOR SHARE inside the same transaction. If the old owner was blocked waiting on that lock while the takeover committed, it has to see the new range_id once the lock is released. Otherwise it would keep writing to a shard it no longer owns.</p><p>Single-node Postgres gets this right because READ COMMITTED re-reads a row after waiting on its lock (the EvalPlanQual mechanism). Limitless implements its own distributed READ COMMITTED, so this needed proof rather than assumption.</p><p>The full list we tested:</p><ol><li>A blocked FOR SHARE or FOR UPDATE returns the newest row version after the other transaction commits.</li><li>REPEATABLE READ raises 40001 when a row changed after the snapshot.</li><li>A conditional UPDATE ... WHERE range_id = $old re-evaluates its predicate after waiting.</li><li>The lock compatibility matrix, NOWAIT and lock_timeout match PostgreSQL.</li><li>A write committed through one router is immediately visible through another.</li><li>History transactions run on the single-shard path.</li><li>Distributed deadlocks are detected and one side gets 40P01.</li></ol><h3>How we tested it</h3><p>Everything ran against a real 16.13-limitless cluster, and every test also ran against stock PostgreSQL 16 as a control group. Expected results came from what PG16 actually did, not from what we remembered it doing.</p><p><strong>Deterministic cases.</strong> Each case uses two connections and channel barriers to force an exact interleaving. The SQL is copied from Temporal’s postgres plugin so we are testing the statements Temporal really runs. Each case ran 50 times with both connections on the same router and 50 times with them on different routers, under READ COMMITTED and REPEATABLE READ. The shard group’s DNS returns one router per lookup, so for the cross-router runs we dialed each router’s address directly and kept the TLS server name set to the endpoint.</p><p>The cross-router read-after-write check ran 100,000 times across 64 shard IDs in parallel. Every read saw the write that preceded it.</p><p><strong>Fencing under chaos.</strong> This is the test we cared about most. For 64 shard IDs, we ran a current owner and a stale owner at the same time, each with four writer goroutines spread randomly across routers. A takeover loop bumped range_id on each shard every 50 to 500 ms. The stale owner was never told to stop; it kept writing until its own FOR SHARE read revealed the new range. Every successful write was logged, and afterwards a checker verified four invariants:</p><ul><li><strong>Fencing:</strong> no write tagged with an old range_id committed after the takeover that retired it.</li><li><strong>Monotonic ranges:</strong> each shard’s range_id history strictly increases.</li><li><strong>Per-workflow CAS:</strong> each workflow’s versions run exactly 1..n with no duplicates and no gaps.</li><li><strong>Coverage:</strong> every shard went through enough takeovers that the first three checks mean something.</li></ul><p>A 60-minute run produced 812,786 commits and 58,020 takeovers. All four invariants held.</p><p>Then we did it again and rebooted the entire shard group in the middle. Twenty minutes, aws rds reboot-db-shard-group at minute eight. The shard group was back in about two minutes with the same router endpoints, writers reconnected on their own, and the run continued: 414,681 commits, 18,714 takeovers, 85,097 stale-owner writes rejected by the fence, 6,166 reconnects and 15,788 client-side cancellations injected on purpose. Fencing, monotonic ranges and per-workflow CAS held before the reboot, during it and after it.</p><p><strong>Temporal’s own test suites.</strong> With the semantics settled, we built the plugin (postgres12_limitless) and pointed Temporal&#39;s upstream persistence and functional suites at a real cluster. A single script provisions a fresh shard group, loads the schema with temporal-sql-tool, runs the suites and tears everything down. The persistence suites passed 483 subtests with zero failures. The functional workflow, signal, activity, child-workflow, continue-as-new and shard-acquisition suites passed 86 subtests with zero failures.</p><p>Verification results <em>Every property Temporal depends on, checked on Limitless and on PostgreSQL 16.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MJyWFov9ApcIxyt1WT2FuA.png" /></figure><h3>Throughput</h3><p>We measured throughput on the smallest Limitless topology: two routers and two shards, with the shard group capped at 64 ACU (about 16 ACU per node). Driving Temporal load until CPU sat at 70 to 80 percent, that cluster sustains <strong>500 to 900 actions per second</strong>.</p><p>The number we scale from is throughput per ACU: 500 to 900 actions per second on 64 ACU works out to 7.8 to 14 actions per second for every ACU in the shard group. Because each Temporal shard’s transactions stay on one Limitless shard, adding shards adds independent capacity, and we hold that per-ACU figure constant as the cluster grows.</p><p>The load scripts are <a href="https://github.com/superdurable/xtemporal/tree/main/x/limitless">here</a></p><h3>From 1 shard to 100</h3><p>Two levers grow a Limitless cluster. Each node can scale up to a larger ACU size, and you can add shards and routers with rds_aurora.limitless_split_shard and rds_aurora.limitless_add_router. The chart shows both: the blue band is the measured node size scaled out, the orange band uses 64 ACU nodes. Routers are added one-for-one with shards, the same ratio as the measured cluster.</p><p>Scaling Temporal on Aurora Limitless from 1 to 100 shards <em>Each shard with 64 ACU nodes contributes 1,000 to 1,800 actions per second. One hundred shards land at 100K to 180K.</em></p><p>At 64 ACU per node, each shard (with its router) adds 1,000 to 1,800 actions per second. The high estimate crosses 100K at 56 shards. The conservative estimate reaches it at 100 shards, so 100 is the number we plan around. Since the baseline was taken at 70 to 80 percent CPU, that headroom is already built into every row.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wlfcpCz2puyTS8lpJvK29w.png" /></figure><h3>Running it</h3><p>The plugin ships in xtemporal, our build of the Temporal server. Swap the official image for superdurable/xtemporal:1.32.1-x.2 and point the default store at your shard group:</p><pre>persistence:<br>  defaultStore: limitless-default<br>  visibilityStore: postgres-visibility<br>  numHistoryShards: 1024<br>  datastores:<br>    limitless-default:<br>      sql:<br>        pluginName: postgres12_limitless<br>        databaseName: postgres_limitless<br>        connectAddr: &lt;shard-group-endpoint&gt;:5432<br>        connectProtocol: tcp<br>        user: temporal<br>        connectAttributes:<br>          sslmode: require<br>        maxConns: 20<br>        maxIdleConns: 20<br>        maxConnLifetime: &quot;1h&quot;<br>    postgres-visibility:<br>      sql:<br>        pluginName: postgres12_pgx<br>        databaseName: temporal_visibility<br>        connectAddr: &lt;visibility-endpoint&gt;:5432<br>        connectProtocol: tcp<br>        user: temporal<br>        connectAttributes:<br>          sslmode: require</pre><p>Load the schema with the bundled SQL tool:</p><pre>temporal-sql-tool --endpoint &quot;$LIMITLESS_ENDPOINT&quot; --port 5432 \<br>  --user &quot;$DB_USER&quot; --database postgres_limitless \<br>  --connect-attributes &#39;sslmode=require&#39; --plugin postgres12_limitless \<br>  setup-schema --schema-name limitless/temporal --version 1.19</pre><p>maxConnLifetime recycles connections so new ones resolve the endpoint again and spread across routers. rds_aurora.limitless_stat_activity shows how sessions are distributed if you want to watch it.</p><h3>Wrapping up</h3><p>Temporal’s persistence layer was designed around shards from the start. Aurora Limitless gives Postgres the same shape. Put the two together with collocated tables and you get a Temporal backend that keeps every fencing and consistency guarantee and grows by adding shards, from a two-shard cluster doing hundreds of actions per second to a hundred-shard cluster doing a hundred thousand.</p><p>We are building an efficient &amp; low cost durable execution platform based on <a href="https://github.com/superdurable/xtemporal">Temporal</a> and <a href="https://github.com/superdurable/dex">Dex</a>. If you are interested, let me know!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c033642b5c1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fault tolerance in LangGraph: retries, timeouts, and error handlers]]></title>
            <link>https://medium.com/@qlong/fault-tolerance-in-langgraph-retries-timeouts-and-error-handlers-53ad12aecf68?source=rss-56b397853e0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/53ad12aecf68</guid>
            <category><![CDATA[workflow]]></category>
            <category><![CDATA[distributed-transaction]]></category>
            <category><![CDATA[langgraph]]></category>
            <dc:creator><![CDATA[Long Quanzheng]]></dc:creator>
            <pubDate>Thu, 04 Jun 2026 19:17:42 GMT</pubDate>
            <atom:updated>2026-06-04T19:17:42.428Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A clone from LangChain </em><a href="https://www.langchain.com/blog/fault-tolerance-in-langgraph"><em>blog</em></a><em>, co-authored by Sydney Runkle.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*otfrEWF73r4Taxgn.png" /></figure><p>In the real world, agents hit errors that prototypes never see: network failures, tool call errors, LLM rate limits.</p><p>Imagine you have a task that’s been running for hours or days that hits an unrecoverable error halfway through. What do you do? Abandon the run and completely start over? This isn’t a sustainable way to run production agents.</p><p>Writing the happy path is usually the easy part. The error handling boilerplate that makes it survive in production (retries, timeouts, fallbacks) is often longer than the business logic itself.</p><p><a href="https://docs.langchain.com/oss/python/langgraph/overview">LangGraph</a> treats this as a first-class concern. It’s a low-level orchestration framework that models your agent as a set of discrete steps (nodes), organized as a graph. For a typical agent, that’s a node that calls the model and a node that runs any tool calls it returns, plus any deterministic logic you want to wrap around that loop. Because LangGraph controls execution, it’s also the right place to handle what happens when any of those steps fail.</p><p>This post walks through the three primitives LangGraph helps you with that boilerplate. We cover how they compose and why having them inside the workflow engine matters, especially once you start thinking about cleanup / compensation logic. (See the docs: <a href="https://docs.langchain.com/oss/python/langgraph/fault-tolerance">Fault tolerance</a>.)</p><p>The three primitives are:</p><ol><li><a href="https://docs.langchain.com/oss/python/langgraph/fault-tolerance#retries">RetryPolicy</a>: automatic retries with backoff/jitter for transient errors.</li><li><a href="https://docs.langchain.com/oss/python/langgraph/fault-tolerance#timeouts">TimeoutPolicy</a>: a wall-clock or progress-based cap on a node attempt.</li><li><a href="https://docs.langchain.com/oss/python/langgraph/fault-tolerance#error-handling">error_handler</a>: a node that runs <strong>after retries are exhausted</strong>, with the failure context attached.</li></ol><p>In LangGraph, you define your agent by adding nodes and edges to a StateGraph. All three primitives attach directly to a node via <a href="https://docs.langchain.com/oss/python/langgraph/fault-tolerance#retries">add_node</a>, so your fault tolerance config lives right next to the logic it protects. (If you want to configure defaults once, see <a href="https://docs.langchain.com/oss/python/langgraph/fault-tolerance#graph-defaults">set_node_defaults</a>.)</p><pre>from langgraph.graph import StateGraph<br>from langgraph.types import RetryPolicy, TimeoutPolicy<br>from langgraph.errors import NodeError<br><br>(<br>    StateGraph(State)<br>    .add_node(<br>        &quot;call_llm&quot;,<br>        call_llm,<br>        retry_policy=RetryPolicy(max_attempts=4, backoff_factor=2.0),<br>        timeout=TimeoutPolicy(run_timeout=30, idle_timeout=5),<br>        error_handler=handle_model_failure,<br>    )<br>    ...<br>)</pre><h3>Starting from retries</h3><p>Transient failures are the most common kind of failure in any non-trivial graph: an LLM provider returns a 5xx, a vector store hits a connection reset, a downstream HTTP service is briefly unavailable. Every one of these is fundamentally a “try again in a moment and it’ll probably work” kind of error.</p><p>Without first-class support you end up writing the same wrapper inside every node:</p><pre>def call_llm(state):<br>    # ~25 lines of &quot;retry with backoff, but only on 5xx,<br>    # don&#39;t retry on 4xx, log each attempt, sleep with jitter&quot;<br>    ...</pre><p>LangGraph’s <em>RetryPolicy</em> removes that boilerplate. It applies <em>per node attempt</em>, with exponential backoff, optional jitter, and a configurable predicate for which exceptions count as retryable:</p><pre>from langgraph.types import RetryPolicy<br><br>policy = RetryPolicy(<br>    initial_interval=0.5,<br>    backoff_factor=2.0,<br>    max_interval=128.0,<br>    max_attempts=3,<br>    jitter=True,<br>    retry_on=(ConnectionError, TimeoutError),  # or a callable <br>)</pre><p>The default retry_on is intentionally conservative: it retries ConnectionError, 5xx responses from httpx/requests, and a few generic transient categories.</p><p>By default it <strong>does not</strong> retry ValueError, TypeError, RuntimeError, etc., which are almost always programming bugs.</p><p>The retry_on spec can be a collection of error types <strong>or</strong> a callable that checks an error at runtime to see if it matches retry criteria.</p><h3>Timeout: a special case of “transient failure”</h3><p>A timeout is really just “the attempt is treated as a transient failure because it’s been hanging too long.” Without an explicit timeout, a stuck HTTP call or a frozen subprocess can hang a graph run indefinitely.</p><p>LangGraph’s <a href="https://docs.langchain.com/oss/python/langgraph/fault-tolerance#timeouts">TimeoutPolicy</a> supports two types of timeouts:</p><pre>from langgraph.types import TimeoutPolicy<br><br>TimeoutPolicy(<br>    run_timeout=30.0,    # hard wall-clock cap on a single attempt<br>    idle_timeout=5.0,    # max time without observable progress<br>    refresh_on=&quot;auto&quot;,   # or &quot;heartbeat&quot;<br>)</pre><ul><li>run_timeout is a hard wall-clock cap on a single attempt. Useful when you simply do not care to ever wait more than N seconds for a node.</li><li>idle_timeout resets on every “progress” signal: channel writes, streamed chunks (automatically emitted from LangChain LLM models), child task events, LangChain callback events. Long-running but actively-streaming work doesn’t trip it, but a <em>truly</em> hung call does.</li><li>idle_timeout internally relies on “heartbeat” for every signal. If you control the work and emit your own progress beats, you can switch to refresh_on=&quot;heartbeat&quot; and explicitly call runtime.heartbeat() from inside the node.</li></ul><p>When a timeout fires, the node attempt is cancelled and a <a href="https://docs.langchain.com/oss/python/langgraph/fault-tolerance#nodetimeouterror">NodeTimeoutError</a> is raised.</p><h3>Error handlers: when retries aren’t enough</h3><p>Retries handle “this will probably work in 5 seconds.” However, they don’t handle the cases where retry exhaustion and you need to run some logic. For example, “we’ve tried six times, the payment provider is still down, and now you need to:</p><ul><li>mark the order as failed and notify the customer, or</li><li>roll back the partial side effects we already committed, or</li><li>publish a payment.failed event for the rest of the system to react to.”</li></ul><p>There are a lot of use cases for error handlers after retry exhaustion. This includes cleanup, alerting, dead-letter writes, fallback paths to a cheaper model, or just routing to a “we apologize” message.</p><p>In LangGraph, this is now supported naturally (docs: <a href="https://docs.langchain.com/oss/python/langgraph/fault-tolerance#error-handling">Error handling</a>):</p><pre>from langgraph.errors import NodeError<br><br>def on_call_llm_failed(state: State, error: NodeError) -&gt; State:<br>    log.error(&quot;call_llm failed after retries:%s&quot;, error.error)<br>    return {&quot;status&quot;: &quot;llm_unavailable&quot;}<br><br>  StateGraph(State)<br>  .add_node(<br>      &quot;call_llm&quot;,<br>      call_llm,<br>      retry_policy=RetryPolicy(max_attempts=4),<br>      error_handler=on_call_llm_failed,<br>  )</pre><p>A few things to notice about how this is wired:</p><p><strong>It only fires after retries are exhausted.</strong> This is the property that makes the feature actually useful. If you want to run on every exception, you’d just need to write a try/except inside the node.</p><p><strong>The failure context is injected.</strong> The handler can use parameter as NodeError to get the failing node’s name plus the exception (error.node, error.error).</p><p><strong>The transition is atomic.</strong> When the original node fails, its ERROR write is committed to the checkpoint, and the handler task is scheduled as a new task in the same step. This is crucial in some critical processes where you can’t go back to the regular steps after entering the error-handler steps. If the host process crashes mid-handler, next time it will resume the run re-schedules the handler, not the original failing node</p><p><em>The error handler runs in the same execution cycle.</em>* When a node fails, the error handler is scheduled immediately alongside any other nodes that were already running in that step. It doesn’t wait for them to finish, and they don’t wait for it.</p><p>*in LangGraph, we call an “execution cycle” a “superstep”, if you’re familiar with the runtime.</p><p><strong>You can set a default handler for every node.</strong> <a href="https://docs.langchain.com/oss/python/langgraph/fault-tolerance#graph-defaults">set_node_defaults</a> applies to every regular node that doesn’t specify its own, but a per-node error_handler= always wins.</p><p><strong>You can’t set another error handler for an error handler</strong>. So you don’t get infinite-recursion behavior.</p><h3><strong>Putting it together: fault tolerant flight booking</strong></h3><p>The three primitives above compose naturally, but their real power shows up in workflows that involve side effects: operations that change real-world state. Consider a flight booking: it’s not one action, it’s a sequence. Reserve a seat, process payment, issue a ticket. Each step talks to an external system. Any of them can fail.</p><p>The naive approach (just retry the whole thing) breaks down fast. If the reserving seat went through but the payment or issuing ticket fails, the reservation is stuck in a bad state . What you actually need is to retry each step individually, and if a step exhausts its retries, undo only the steps that already ran(including failed one because it’s unknown).</p><p>This is called the SAGA pattern, and it’s a standard way to handle failures in distributed systems where you can’t wrap everything in a single database transaction.</p><p>Here’s what that looks like in LangGraph:</p><pre>from typing import TypedDict, Annotated, Literal<br>import operator<br>from langgraph.graph import StateGraph, START, END<br>from langgraph.types import Command, RetryPolicy<br>from langgraph.errors import NodeError<br><br>class BookingState(TypedDict, total=False):<br>    booking_id: str<br>    passenger: str<br>    flight: str<br>    seat: str  # assigned once the seat is reserved<br>    amount: int  # fare to charge, in minor units<br>    payment_ref: str  # set once payment is captured<br>    ticket_no: str  # set once the ticket is issued<br>    completed: Annotated[list[str], operator.add]  # accumulates per-step <br><br>def to_compensate(state: BookingState, error: NodeError) -&gt; Command:<br>    &quot;&quot;&quot;Route any retry-exhausted step to the compensation node.&quot;&quot;&quot;<br>    return Command(<br>        &quot;&quot;&quot;include the failed node&quot;&quot;&quot;<br>        update={&quot;completed&quot;: [f&quot;FAILED:{error.node}&quot;]},<br>        goto=&quot;compensate&quot;,<br>    )<br><br>def reserve_seat(state) -&gt; BookingState:<br>    # Call the seat-inventory service to hold a seat for this itinerary.<br>    ...<br>    return {&quot;seat&quot;: &quot;12A&quot;, &quot;completed&quot;: [&quot;reserve_seat&quot;]}<br><br>def process_payment(state) -&gt; BookingState:<br>    # Charge the fare via the payment processor while the seat is held.<br>    ...<br>    return {&quot;payment_ref&quot;: &quot;pay_abc123&quot;, &quot;completed&quot;: [&quot;process_payment&quot;]}<br><br>def issue_ticket(state) -&gt; BookingState:<br>    # Confirm the seat and issue the ticket once payment is captured.<br>    ...<br>    return {&quot;ticket_no&quot;: &quot;TKT-7788&quot;, &quot;completed&quot;: [&quot;issue_ticket&quot;]}<br><br>def compensate(state) -&gt; Command[Literal[&quot;__end__&quot;]]:<br>    # Inspect `state[&quot;completed&quot;]` and undo only the steps that actually ran,<br>    # in reverse order, to keep the booking all-or-nothing.<br>    if &quot;issue_ticket&quot; in state[&quot;completed&quot;]:<br>        void_ticket(state)<br>    if &quot;process_payment&quot; in state[&quot;completed&quot;]:<br>        refund_payment(state)<br>    if &quot;reserve_seat&quot; in state[&quot;completed&quot;]:<br>        release_seat(state)<br>    return Command(goto=END)<br><br><br>graph = (<br>    StateGraph(BookingState)<br>    # All steps share the same retry policy and the same fallback target;<br>    # per-step overrides are still possible.<br>    .set_node_defaults(retry_policy=RETRYABLE, error_handler=to_compensate)<br>    .add_node(&quot;reserve_seat&quot;, reserve_seat)<br>    .add_node(&quot;process_payment&quot;, process_payment)<br>    .add_node(&quot;issue_ticket&quot;, issue_ticket)<br>    .add_node(&quot;compensate&quot;, compensate)<br>    .add_edge(START, &quot;reserve_seat&quot;)<br>    .add_edge(&quot;reserve_seat&quot;, &quot;process_payment&quot;)<br>    .add_edge(&quot;process_payment&quot;, &quot;issue_ticket&quot;)<br>    .add_edge(&quot;issue_ticket&quot;, END)<br>    .compile(checkpointer=checkpointer)<br>)</pre><p>What this gives you:</p><ul><li>Per-step backoff retries with the configured policy</li><li>An atomic transition into compensate once any step&#39;s retries are exhausted</li><li>Persistent state tracking which steps actually completed, so compensate only undoes what needs to revert</li></ul><h3>Final words</h3><p>Agents are taking on more autonomy, and with that comes more power to act. They’re booking flights, filing tickets, executing payments, calling internal services. The actions they take are increasingly high-consequence and difficult to reverse.</p><p>That raises the bar for reliability. A 1% transient failure rate is a minor inconvenience in a demo. In a production agent with dozens of steps and real-world consequences, it compounds quickly.</p><p>RetryPolicy, TimeoutPolicy, and error_handler are built into LangGraph so that it’s easy to build an agent that’s resilient to all sorts of errors. All you have to do is define policies that make sense for your use case, and the LangGraph agent runtime handles the rest.</p><p><strong>Get started:</strong> configure per-node retries, timeouts, and error handlers with the official <a href="https://docs.langchain.com/oss/python/langgraph/fault-tolerance">Fault tolerance docs</a>.</p><p>. All three primitives attach directly to a node via <a href="https://docs.langchain.com/oss/python/langgraph/fault-tolerance#retries">add_node</a>, so your fault tolerance config lives right next to the logic it protects. (If you want to configure defaults once, see <a href="https://docs.langchain.com/oss/python/langgraph/fault-tolerance#graph-defaults">set_node_defaults</a>.)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53ad12aecf68" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Workflow Should be Code, but Replay Based Durable Execution is NOT the ONLY way]]></title>
            <link>https://medium.com/@qlong/workflow-should-be-code-but-durable-execution-is-not-the-only-way-519f7682360c?source=rss-56b397853e0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/519f7682360c</guid>
            <dc:creator><![CDATA[Long Quanzheng]]></dc:creator>
            <pubDate>Wed, 20 Aug 2025 20:59:47 GMT</pubDate>
            <atom:updated>2026-08-28T21:00:23.415Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>NOTE</strong>: iWF is now forked as <a href="https://github.com/superdurable/dex">Dex</a> (Durable-Execution).</p><p>Today I read an article by Maxim Fateev, <em>“</em><a href="https://temporal.io/blog/the-fallacy-of-the-graph-why-your-next-workflow-should-be-code-not-a-diagram"><em>The Fallacy of the Graph: Why Your Next Workflow Should Be Code, Not a Diagram</em></a><em>.”</em> Many of his points resonated with me. They are close to what I learned in the past five years at Indeed. During that time, we built a DSL workflow engine, but failed and learned the limitations of DSL/graph/JSON based approach. With the learnings, we created iWF.</p><p>Our conclusions are similar to Maxim’s, but not the same. So I want to write this post as a response.I will share what my team explored at Indeed, what we learned, and what we built.</p><h3>Problems with a JSON-Based DSL</h3><p>They were almost the same as what Maxim described. When we first built this Graph-Based DSL workflow engine, it worked well for simple use cases.<br> But as the use cases became more complex, the challenges grew.In our case:</p><ol><li>We had to support more and more features for complex if-else branching and loops. The conditions became closer to boolean conditions in a programming language — multiple levels of parentheses, different operators, and so on. Eventually, we let customers write a JavaScript snippet.This was very flexible, but it provides bad user experience.</li><li>Readability of workflow implementation. Part of the logic lived in JSON, another part in Java code. To maintain it, we always had to switch between both.</li><li>Reusability of workflow logic was poor. It was hard to make one JSON a base and let others extend it like a Java class. This is a problem because workflows sometimes needed changes, and we have to keep many versions (because sometimes we didn’t want to change already started executions). The only solution was copy-paste. This became a nightmare for maintenance.</li><li>Unit testing was impossible. We could only do integration testing for the whole workflow.But sSometimes we wanted to isolate an issue and test part of the JSON DSL logic. That was not possible.</li><li>Weekly typing programming. There was no strong typing in many places. We had to do a lot of casting from an Object to a type which is error-prone.</li></ol><p>Because of these reasons, we decided to shutdown the engine after running for ~2 years in production with millions of workflows everyday, and successfully migrated to iWF.</p><h3>The Pitfalls of replay based Durable Execution</h3><p>At the beginning of our journey, we built several projects with Cadence/Temporal (first self-hosted Cadence, later migrated to Temporal Cloud). Then we decided to build our own workflow engine on top of it.</p><p>One of the reasons was to simplify the developer experience. Durable Execution is very powerful, but also very complex. At the start, it felt fine for a proof of concept, or when the business logic was still simple. Engineers were motivated to learn durable execution because it felt “cool.”<br>But as time passed, we faced more and more problems, which mostly summarized this <a href="https://blog.cgillum.tech/common-pitfalls-with-durable-execution-frameworks-like-durable-functions-or-temporal-eaf635d4a8bb">blog</a> that I wished I have read much earlier.</p><p><a href="https://blog.cgillum.tech/common-pitfalls-with-durable-execution-frameworks-like-durable-functions-or-temporal-eaf635d4a8bb">Common Pitfalls with Durable Execution Frameworks, like Durable Functions or Temporal</a></p><ol><li><strong>Versioning.</strong><br>Workflows are usually long-running (hours or more).<br>Every time we changed workflow code, durable execution required versioning.If not, replay would fail, leading to severe incidents.<br>CPU, memory, and network usage spiked immediately. Even if only one or two workflow executions had issues, the endless replays could degrade the whole fleet.</li><li><strong>Programming paradigm shift.</strong><br>Durable execution looked exciting at first, but after the novelty was gone, many problems remained. It introduced a completely new programming model with many implicit contracts and foot guns.<br>Exception handlings in query/signal/activity behaved very differently from normal development.<br>Workflow components cannot be regular components(e.g. Spring beans in Java). It means the dependencies, config, logging, and many patterns were not available. All these details, if an engineer didn’t use it for a few months, they forgot easily.</li><li><strong>Operation overhead.</strong><br>Workflow runs were based on workflow tasks.<br>This was very different from the familiar request→response model (like microservices). Monitoring, dashboards, and runbooks all had to be in a different. The application is a semi-stateful that heavily relys on memory as caching, which also creates a lots of complexity when cache misses for CPU and memory usage.</li><li><strong>Unit tests.</strong><br>Unit tests were possible, but very hard to write.<br>All APIs were static, so we could not mock them.We had to use testEnv to simulate execution.</li><li><strong>Multi-threading paradigm.</strong><br><a href="https://stackoverflow.com/questions/71356668/how-does-multi-threading-works-in-cadence-temporal-workflow">Multi-threading</a> was also very different than regular programming or mental model. It used cooperative thread scheduling. Most people could not understand or debug complex multi-threading cases.<br>When non-deterministic errors happened, debugging became almost impossible.Even for experts like me, sometimes I could not debug them.</li><li><strong>Continue-as-new.</strong><br> Not always needed, but when business logic grew complex, sometimes it was unavoidable.<br> And when it happened, it was a pitfall with extreme difficulty.</li></ol><h3>A novel programming model that we created</h3><p>We created iWF, a new programming model.<br> Its main goals are:</p><p><strong>Workflow-as-code.</strong><br> All workflow logic — branches, loops, data read/write — is written in the native programming language.</p><p><strong>Structured programming.</strong><br> Workflow logic is cleanly organized.<br> Developers can understand and maintain the code at a glance.</p><p><strong>Keep the advantages of durable execution.</strong><br> Backoff retry, durable timers, persisted data/state, parallel execution, multi-threading with locking, and external interactions are all supported.</p><p><strong>Standard paradigm for programming and operation.</strong><br>Workflow code is a regular microservice component, running as a normal microservice (REST).<br>Developers can reuse all they know about dependency management, error handling, and operations.</p><p><strong>Simplified and explicit APIs, better programming experience.</strong><br>We used as few concepts as possible.<br>APIs are explicit, easy to understand, and easy to remember.<br>Coding is strongly typed whenever possible (for example, based on Java reflection like in the Temporal Java SDK).</p><h3>Workflow-As-Code</h3><p>Let’s look at an example of sending notifications to inactive users — if a user don’t open the App for a week, send out an email.</p><p>It’s called “updatable timer” design pattern, and the production code is like below.</p><pre>class UpdatableTimerState implements WorkflowState&lt;Void&gt; {<br>@Override<br>    public CommandRequest waitUntil(Context context, ...) {<br>        return CommandRequest.forAnyCommandCompleted(<br>                TimerCommand.createByDuration(TIMEOUT_DURATION)),<br>                InternalChannelCommand.create(CHANNEL_RESET_TIMER)<br>        );<br>    }<br>    @Override<br>    public StateDecision execute(Context context, Void input, CommandResults commandResults,...) {<br>        TimerCommandResult timer = commandResults.getTimerResult(0);<br>        if (timer.getStatus() == FIRED) {<br>              // timer fires, call API to send an email<br>              svc.sendEmail(...);<br>              // assuming we want to complete workflow here<br>              return StateDecision.completeWorkflow();<br>        }<br>        // otherwise the channel receives a message to reset the timer<br>        return StateDecision.singleNextState(UpdatableTimerState.class);<br>    }<br>    // optionally control the backoff retry, default is infinite retry<br>    @Override<br>    public WorkflowStateOptions getStateOptions() {<br>        return new WorkflowStateOptions()<br>                .setExecuteApiRetryPolicy(<br>                        new RetryPolicy()<br>                                .backoffCoefficient(2f)<br>                                .maximumAttempts(5)<br>                                .maximumAttemptsDurationSeconds(3600)<br>                );<br>    }<br>}</pre><p>It uses native programing language to branch and loop. The logic is clear and clean, whereas it’s very error-prone in durable execution.</p><p><a href="https://medium.com/@qlong/iwf-deep-dive-workflowstate-durable-timer-1-0bb89e6d6fd4">Durable timers in Indeed Workflow Engine vs Cadence/Temporal</a></p><p>Workflow-as-code enable a lot of code inheritance/reuse, e.g., <a href="https://github.com/indeedeng/iwf-java-sdk/issues/196">we have user asking</a> for reusing some error handling logic, and they just need to refactor the code in Java inheritance.</p><pre>// A base class for share logic<br>class BaseStateWithExceptionHandler : WorkflowState {<br><br>  override StateDecision execute(...){<br>      try{<br>         return doExecute(...);<br>      }catch (e: NonRetryableException){ <br>         return StateDecision.singleNextState(ErrorHandleState.class, ...);<br>      }<br>  }<br><br>  abstract StateDecision doExecute(...); // each state must implement this <br>}<br><br>// Then each state will just:<br>class MyState:BaseStateWithExceptionHandler {<br>  override  StateDecision doExecute(...){<br>       // your state logic<br>  }<br>}</pre><h3>Structural Programming</h3><p>We have found that the structural way of iWF letting developers to organize workflow code works very well. It gives people a clean view of the workflow at a first glance:</p><ul><li>The declarative workflow states that executed on the background, and the first state as “starting state”</li><li>The declarative persistence data/search attributes as schema</li><li>The explicit RPC as mechanism of external interactions with the workflow</li><li>The declarative data channel that being used as FIFO queue throughout the workflow logic (between states and RPC).</li></ul><p>This is an <a href="https://github.com/indeedeng/iwf-python-samples/blob/main/ai-agent-email/ai_agent_workflow.py">example</a> of the production grade to show the structureness.</p><pre>class EmailAgentWorkflow(ObjectWorkflow):<br>    def get_persistence_schema(self) -&gt; PersistenceSchema:<br>        return PersistenceSchema.create(<br>            PersistenceField.data_attribute_def(DA_STATUS, str),<br>            PersistenceField.data_attribute_def(DA_CURRENT_REQUEST, str),<br>...<br>        )<br>def get_communication_schema(self) -&gt; CommunicationSchema:<br>        return CommunicationSchema.create(<br>            CommunicationMethod.internal_channel_def(CH_USER_INPUT, str),<br>        )<br>    def get_workflow_states(self) -&gt; StateSchema:<br>        return StateSchema.with_starting_state(<br>            InitState(),<br>            AgentState(),<br>            ScheduleState(),<br>            SendingState()<br>        )<br>    @rpc()<br>    def send_request(self, input: str, persistence: Persistence, communication: Communication) -&gt; bool:<br>        status = persistence.get_data_attribute(DA_STATUS)<br>        if status == STATUS_WAITING:<br>            persistence.set_data_attribute(DA_CURRENT_REQUEST_DRAFT, &quot;&quot;)<br>            communication.publish_to_internal_channel(CH_USER_INPUT, input)<br>            persistence.set_data_attribute(DA_STATUS, STATUS_PROCESSING)<br>            return True<br>        else:<br>            return False</pre><h3>Strongly typing</h3><p>iWF SDK strives to provide the best developer experience. There are lots of strongly typing experience that we have provide in iWF SDKs:</p><p><strong>State input</strong></p><p><a href="https://github.com/indeedeng/iwf-java-sdk/blob/main/src/main/java/io/iworkflow/core/WorkflowState.java#L12">A workflow state</a> has a generic parameter as input. The <a href="https://github.com/indeedeng/iwf-java-sdk/blob/main/src/main/java/io/iworkflow/core/StateDecision.java#L161">StateDecision</a> will enforce the input passing in is compatible at compile time (and run-time for GoSDK)</p><p><strong>RPC invocation</strong></p><p>In Java, the invocation of RPC uses a “<a href="https://github.com/longquanzheng/iwf-samples-internal/blob/main/src/main/java/com/indeed/iwf/patterns/controller/DesignPatternController.java#L76">Stub</a>” like ActivityStub in Temporal JavaSDK. This ensures the strongly type checking in compile time for input &amp; output.</p><p><strong>Get and set data attributes and channel</strong></p><p>This only works for run-time. iWF lets users define the data attributes and internal/signal channels in workflow definition, and the SDK will uses check on the type in the run-time.</p><p><strong>SearchAttribute</strong></p><p>The search attributes APIs are strongly typed.</p><h3>External interactions</h3><p>iWF provides very simple &amp; powerful mechanism to interact with a running workflow via RPC.</p><p>RPC can have access to not only persistence read/write API, but also interact with WorkflowStates using InternalChannel, or trigger a new WorkflowState execution in a new thread.</p><p>A RPC is defined in pure code. It belongs to a workflow definition(an implementation of Workflow interface) as a method of the instance.</p><p>You can define an unlimited number of RPCs for a workflow definition.</p><p>For example, in Java you can use the <a href="https://github.com/indeedeng/iwf-java-sdk/blob/main/src/main/java/io/iworkflow/core/RPC.java">RPC annotation</a> can make a method an RPC</p><pre>public class UserSignupWorkflow implements ObjectWorkflow {<br>    @RPC<br>    public String verify(Context context, String input, Persistence persistence, Communication communication) {<br>        String status = persistence.getDataAttribute(DA_Status, String.class);<br>        if (status == &quot;verified&quot;) {<br>            return &quot;already verified&quot;;<br>        }<br>        persistence.setDataAttribute(DA_Status, &quot;verified&quot;);<br>        communication.publishInternalChannel(VERIFY_CHANNEL, null);<br>        return &quot;done&quot;;<br>    }<br>}</pre><h3>Data read/write, multi-threading and locking</h3><p>Workflow can pass data via state input, or through “data/search attributes” by ONLY declaring the types in workflow definitions, without mataining any database schema.</p><p>iWF allow running hundreds of states in parallel, and they can all read/write to the same data/search attributes, and RPC can also read/write to them. This creates the potential of racing conditions for overriding each other, or dirty reads.</p><p>In iWF, it provides l<a href="https://github.com/indeedeng/iwf/wiki/Persistence#persistence-loading-policy">ocking mechanism</a> to prevent the racing conditions, the granularity is to a single data/search attribute. You only set the keys to lock in RPC annotations, or the StateOptions to enable it.</p><h3>Operations and Testing</h3><p>There is no NDEs at all in the iWF workflow. This is a game changer for any complex workflow use cases. Whatever user defined in the code, just deploy, and it will take effect immediately.</p><p>Though there are still breaking changes like data model changes (which are much, much easier than NDEs already). To future improve the experience, iWF provides a feature to <a href="https://github.com/indeedeng/iwf/wiki/%5BVersioning%5DHow-to-modify-workflow-code-without-breaking-changes#technical-in-compatible-changes">manage in-flight states using a system search attribute</a>.</p><p>Because of the architecture, the metrics, monitors and runbook of an iWF application is ALL standard as a microservice.</p><p>Scaling up an iWF application is extremely simple. Just add instances as needed. No need to understand metrics like scheduled_to_start latency, or worry about CPU/memory spike due to caching for replay.</p><p>The executions are all distributed by the standard LoadBalancer, without the needs of <a href="https://docs.temporal.io/task-queue">“TaskQueue” or TaskQueuePartitions</a>.</p><p>Writing unit tests is now extremely straightforward. You just need to use any standard unit test libraries. For example, in Java, just use Junit and mockito, like this <a href="https://github.com/indeedeng/iwf-java-samples/blob/main/src/test/java/io/iworkflow/workflow/subscription/SubscriptionWorkflowTest.java#L35">example</a>.</p><p>This eliminates A LOT, A LOT of headache that I have seen for SWEs at Indeed that they need to figure out how to use TestEnv to write a unit tests. Most of the time, for a lightly complex project, writing a test is harder than modifying a Temporal workflow, based on what I have seen.</p><p>iWF also empower the team to build integration tests easily. Because of the explicitness of “TimerCommand” and “WorkflowState”, users <a href="https://github.com/indeedeng/iwf-java-sdk/blob/main/src/main/java/io/iworkflow/core/Client.java#L1147">can skip a timer</a>(meaning firing it immediately) to let a integration test run without waiting for the real time timer.</p><h3>Summary with more comparison and design patterns</h3><p>So, all I want to show you is a different way that we have found/created to provide a great “workflow as code” experience. So far, we are so confident that this works well, based on the feedback of after running it for 4 years with millions of critical workflows everyday, and 30+ teams using it and still growing.</p><p>There are many more to show, if you are interested, see the below links for how it works compared to Cadence/Temporal.</p><p><a href="https://medium.com/@qlong/iwf-overview-for-temporal-users-part1-programming-model-difference-9f58e4793cfa">iWF Overview for Temporal Users-Part1: Programming Model Difference</a></p><p>Also, if you have doubt that iWF can build sophisticated business logic, check out the design patterns that we have adopted at Indeed.</p><p><a href="https://medium.com/@qlong/iwf-design-patterns-936a48336766">14 “Modern” Backend Software Design Patterns with Indeed Workflow Framework(iWF) on Temporal</a></p><p>And finally the post about why I created iWF</p><p><a href="https://medium.com/@qlong/a-letter-to-cadence-temporal-and-workflow-tech-community-b32e9fa97a0c">Why I created Indeed Workflow Framework(iWF)</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=519f7682360c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Modern Workflow Frameworks Are Essential for Today’s Backend Engineering]]></title>
            <link>https://medium.com/@qlong/why-modern-workflow-frameworks-are-essential-for-todays-backend-engineering-93b0f32cdfcb?source=rss-56b397853e0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/93b0f32cdfcb</guid>
            <dc:creator><![CDATA[Long Quanzheng]]></dc:creator>
            <pubDate>Fri, 25 Apr 2025 22:41:05 GMT</pubDate>
            <atom:updated>2026-08-28T20:48:24.198Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>NOTE</strong>: iWF is now forked as <a href="https://github.com/superdurable/dex">Dex</a> (Durable-Execution).</p><p>In the rapidly evolving landscape of software development, workflow frameworks have emerged as crucial tools for engineering teams building complex, resilient systems. But what exactly makes these frameworks so valuable, and why should engineering teams adopt them? Having spent nearly a decade building and supporting workflow frameworks, I’d like to share my perspective on their transformative impact on backend engineering.</p><h3>The Evolution of Backend Complexity</h3><p>Twenty years ago, most applications followed a simple pattern: receive a request, perform some business logic (often with a database transaction), and return a response. The application lived on a single server, and if that server failed, users simply experienced downtime until it was restored.</p><p>Today’s reality is dramatically different:</p><ul><li><strong>Distributed Systems</strong>: Applications now run across dozens or hundreds of machines</li><li><strong>Microservice Architecture</strong>: Business logic is spread across numerous services</li><li><strong>Cloud Infrastructure</strong>: Resources dynamically scale up and down</li><li><strong>Complex Workflows</strong>: Business processes span days, weeks, or months</li><li><strong>Intricate Coordination</strong>: Operations require orchestrating many internal and external dependencies</li></ul><p>This evolution brings considerable challenges. Any machine can fail at any time. Network partitions happen regularly. External services experience outages. And business processes have grown increasingly complex, often involving conditional logic, parallel execution paths, retry mechanisms, and long waiting periods.</p><h3>The Reliability Challenge</h3><p>Building reliable software in this environment is exceedingly difficult without the right tools. Consider what’s required for a seemingly simple process like customer onboarding:</p><ol><li>Collect and validate customer information</li><li>Run background checks that may take days to complete</li><li>Conditionally request additional documentation based on verification results</li><li>Process payment information</li><li>Create accounts across multiple systems</li><li>Send welcome communications through various channels</li><li>Schedule follow-up activities</li></ol><p>Now imagine implementing this reliably, ensuring that:</p><ul><li>The process can survive server failures at any point</li><li>Each step is retried appropriately when external services have outages</li><li>The overall state is maintained durably for weeks or months</li><li>Progress can be monitored and troubleshooted easily</li><li>Steps can be executed in parallel when possible</li><li>The process can be updated or versioned without disrupting in-flight executions</li></ul><p>Without a workflow framework, engineering teams typically resort to building complex state machines with database tables, conditional logic spread across services, and custom monitoring systems. This approach is not only time-consuming but also error-prone and difficult to maintain.</p><h3>The Productivity Imperative</h3><p>The complexity of modern backend systems has a direct impact on engineering productivity. Without appropriate tools, teams spend an increasing proportion of their time on infrastructure concerns rather than business logic.</p><p>I’ve observed this evolution firsthand throughout my career:</p><p>At Amazon (2015–2017), I witnessed the transition from XML-based workflow definitions to AWS Simple Workflow, which evolved from Java annotations to a code-first approach before becoming a public AWS offering. Amazon later developed Step Functions to simplify workflow programming, though it faced challenges with JSON-based workflow expressions.</p><p>In 2017, I had the opportunity to work with the Uber team that previously built AWS Simple Workflow as they continued their innovation journey. Their work on Cadence (later becoming Temporal) addressed many of the limitations of previous systems and was quickly adopted by hundreds of teams at Uber.</p><p>These investments in workflow technology weren’t made lightly. They were direct responses to the growing complexity of distributed systems and the need to maintain engineering productivity in the face of that complexity.</p><h3>The Modern Workflow Framework Advantage</h3><p>Modern workflow frameworks like <a href="https://github.com/indeedeng/iwf">iWF (Indeed Workflow Framework)</a> on top of <a href="https://temporal.io/">Temporal</a> provide several game-changing capabilities:</p><h3>1. Durable Execution</h3><p>Workflow frameworks ensure that your business processes survive machine failures, network partitions, and other infrastructure issues. The execution state is persisted durably, allowing processes to continue exactly where they left off after disruptions.</p><h3>2. Explicit Failure Handling</h3><p>Modern frameworks provide built-in mechanisms for retries, timeouts, and compensating actions. This allows developers to express precisely how their processes should behave when things go wrong — a critical capability in distributed systems.</p><h3>3. Long-Running Processes</h3><p>Unlike traditional request-response patterns, workflow frameworks natively support processes that run for extended periods, from minutes to months. Developers can express waiting conditions, timeouts, and reactivation triggers without complex custom infrastructure.</p><h3>4. Visibility and Debugging</h3><p>One of the most underappreciated benefits is the ability to observe and debug long-running processes. Modern workflow frameworks provide visibility into execution history, current state, and progress — transforming what would otherwise be opaque background processes into transparent, debuggable operations.</p><h3>5. Versioning and Updates</h3><p>As business requirements evolve, workflows need to change. Modern frameworks provide mechanisms for versioning and updating workflows without disrupting in-flight executions, a critical capability for maintaining business continuity.</p><h3>Real-World Impact</h3><p>The impact of adopting workflow frameworks can be transformative. In my experience supporting various teams across different companies, I’ve consistently observed:</p><ul><li><strong>Reduced Development Time</strong>: Projects that would take months are completed in weeks or even days</li><li><strong>Improved Reliability</strong>: Systems built on workflow frameworks handle failure gracefully without custom infrastructure</li><li><strong>Simplified Architecture</strong>: Complex coordination logic is expressed clearly rather than scattered across services</li><li><strong>Enhanced Maintainability</strong>: New team members can understand and contribute to workflow code more quickly</li><li><strong>Operational Excellence</strong>: Production issues are identified and resolved faster with built-in observability</li></ul><h3>When to Consider a Workflow Framework</h3><p>Not every application needs a workflow framework, but they become increasingly valuable when your system exhibits certain characteristics:</p><ul><li>Business processes that span multiple services</li><li>Operations that require coordination with external systems</li><li>Processes with conditional paths and complex branching logic</li><li>Long-running operations that may take hours, days, or longer</li><li>Requirements for high reliability even during infrastructure failures</li><li>Complex retry and failure-handling requirements</li></ul><h3>Conclusion: The Right Tool for Modern Challenges</h3><p>From a certain angle, the complexity of modern backend systems stems from not having the right tools for the job. Just as higher-level programming languages freed developers from assembly code and frameworks like Spring simplified Java development, modern workflow frameworks like iWF and Temporal address the fundamental challenges of building reliable, maintainable distributed systems.</p><p>In the rapidly evolving landscape of software development, these tools are becoming as essential as databases or message queues. If your team is building complex backend systems and isn’t utilizing a modern workflow framework, you’re likely reinventing wheels that could be better spent on delivering business value.</p><p>As the pace of change accelerates with innovations like AI and LLMs, having a solid foundation for orchestrating your business processes becomes even more critical. Modern workflow frameworks provide exactly that foundation, transforming how we build complex, resilient systems.</p><p>For a deeper dive into specific patterns enabled by workflow frameworks, check out my article on <a href="https://medium.com/@qlong/iwf-design-patterns-936a48336766">14 “Modern” Backend Software Design Patterns with Indeed Workflow Framework(iWF) on Temporal</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93b0f32cdfcb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[iWF Overview for Temporal Users-Part1: Programming Model Difference]]></title>
            <link>https://medium.com/@qlong/iwf-overview-for-temporal-users-part1-programming-model-difference-9f58e4793cfa?source=rss-56b397853e0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f58e4793cfa</guid>
            <dc:creator><![CDATA[Long Quanzheng]]></dc:creator>
            <pubDate>Fri, 25 Apr 2025 19:44:20 GMT</pubDate>
            <atom:updated>2026-08-28T20:48:16.584Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>NOTE</strong>: iWF is now forked as <a href="https://github.com/superdurable/dex">Dex</a> (Durable-Execution).</p><p>This blog post series is for developers who are familiar with Temporal, to have a quick overview of what Indeed Workflow Framework (iWF) provides an alternative to use Temporal — the most powerful duration execution engine on earth.</p><h3>iWF: A New Approach to Workflow Programming</h3><p>Programming model can make or break developer productivity and code maintainability. iWF (Indeed Workflow Framework) brings a fresh perspective to workflow programming.</p><p>This part1 explores how iWF’s programming model differs from Temporal’s, offering insights into its unique features and expressive capabilities.</p><p><a href="https://medium.com/@qlong/iwf-overview-for-temporal-users-part2-architectural-difference-f91a0dbfd782?postPublishedType=initial">Part2</a> will describe the architecture differences.</p><h3>The Power of Code-First Workflows</h3><p>iWF embraces a code-first approach, allowing developers to define workflows using native programming constructs. Unlike DSL-based workflow engines that create their own primitives for branching, waiting, and looping, iWF lets you use your language’s native control structures. This approach provides immediate familiarity and reduces the learning curve for developers.</p><p>Similar to Temporal, iWF workflow applications are fully dynamic in native code, your code changes go affected immediately after deploying. It also allows you to make higher level abstractions, etc, which will be described in part2.</p><h3>Core Concepts of iWF</h3><h4>WorkflowState: The Two-Phase Execution Model</h4><p>The most distinctive feature of iWF is the WorkflowState concept with its two-phase execution model:</p><ol><li><strong>waitUntil phase</strong> (optional): Returns “CommandRequest” specifying what to wait for</li><li><strong>execute phase</strong>: Runs when commands complete, returning a StateDecision for next steps</li></ol><pre>public class UserVerificationState implements WorkflowState {<br>    @Override<br>    public CommandRequest waitUntil(Context context, StateInput input, Persistence persistence) {<br>        // Wait for either a verification signal or a 7-day timeout<br>        return CommandRequest.forAnyOf(<br>            TimerCommand.createByDuration(TimeUnit.DAYS.toSeconds(7)),<br>            SignalChannelCommand.create(VERIFICATION_CHANNEL)<br>        );<br>    }<br>    <br>    @Override<br>    public StateDecision execute(Context context, StateInput input, CommandResults commandResults, Persistence persistence) {<br>        if (commandResults.isTimerFired()) {<br>            return StateDecision.singleNextState(StateMovement.create(REMINDER_STATE));<br>        } else {<br>            persistence.setDataAttribute(USER_STATUS, &quot;verified&quot;);<br>            return StateDecision.singleNextState(StateMovement.create(WELCOME_STATE));<br>        }<br>    }<br>}</pre><p>This two-phase approach enables expressive patterns that are difficult to implement in other workflow engines.</p><h4>Channels: Communication Pathways</h4><p>iWF provides two types of channels, both implemented as FIFO queues:</p><ul><li><strong>SignalChannel</strong>: External communication, similar to Temporal signals</li><li><strong>InternalChannel</strong>: Communication between workflow states and RPCs</li></ul><p>These channels exist naturally in your code without explicit creation or deletion, simplifying workflow programming.</p><h4>Persistence: State Management</h4><p>iWF’s persistence model provides a clean interface for data management:</p><pre>@Override<br>public List&lt;PersistenceFieldDef&gt; getPersistenceSchema() {<br>    return Arrays.asList(<br>        DataAttributeDef.create(SignupForm.class, &quot;Form&quot;),<br>        DataAttributeDef.create(String.class, &quot;Status&quot;)<br>    );<br>}</pre><p>You can then read and write data from any workflow state or RPC:</p><pre>String status = persistence.getDataAttribute(&quot;Status&quot;, String.class);<br>persistence.setDataAttribute(&quot;Status&quot;, &quot;verified&quot;);</pre><h4>RPC: Interactive Workflows</h4><p>Remote Procedure Calls in iWF provide a powerful way for external systems to interact with running workflows:</p><pre>@RPC<br>public String verify(Context context, String input, Persistence persistence, Communication communication) {<br>    String status = persistence.getDataAttribute(&quot;Status&quot;, String.class);<br>    if (status.equals(&quot;verified&quot;)) {<br>        return &quot;already verified&quot;;<br>    }<br>    persistence.setDataAttribute(&quot;Status&quot;, &quot;verified&quot;);<br>    communication.publishInternalChannel(&quot;verify_channel&quot;, null);<br>    return &quot;done&quot;;<br>}</pre><p>An iWF RPC combines the capabilities of <strong>Temporal query, signal, and synchronous update, offering a more intuitive programming model</strong>.</p><h3>Design Patterns Made Simple</h3><p>iWF’s programming model shines when implementing sophisticated business logic. <a href="https://medium.com/@qlong/iwf-design-patterns-936a48336766">The 14 “Modern” Backend Software Design Patterns with Indeed Workflow Framework(iWF) on Temporal</a> summarize all the <a href="https://github.com/indeedeng/iwf">iWF</a> design patterns that Indeed engineering teams have been using to power millions of workflow executions every single day, on top of <a href="https://temporal.io/cloud">Temporal Cloud</a>.</p><p>Compared to using traditional approaches, iWF/Temporal have reduced the engineering time from months to weeks or even days from development to operation and maintenance, simplified the software architecture significantly for any developers to understand.</p><h3>Polling: Simplified Service Status Checks</h3><p>Polling external services or resources is one of the most common workflow patterns. While conceptually simple, it requires careful design to handle properly.</p><h3>In Temporal:</h3><pre>@Override<br>public void execute(MyWorkflowInput input) {<br>    boolean conditionMet = false;<br>    while (!conditionMet) {<br>        Workflow.sleep(Duration.ofSeconds(30));<br>        conditionMet = activities.checkCondition(input.getResourceId());<br>    }<br>    // Continue when condition is met<br>}</pre><h3>In iWF:</h3><pre>public class PollingState implements WorkflowState&lt;Input&gt;{<br>  @Override<br>  public CommandRequest waitUntil(Context context, Input input, Persistence persistence) {<br>      return CommandRequest.forTimer(TimerCommand.createByDuration(30)); // 30 seconds<br>  }<br>  <br>  @Override<br>  public StateDecision execute(Context context, Input input, CommandResults results, Persistence persistence) {<br>      boolean condition = checkExternalService();<br>      if (condition) {<br>          return StateDecision.singleNextState(StateMovement.create(COMPLETION_STATE));<br>      }<br>      return StateDecision.singleNextState(StateMovement.create(PollingState.class)); // Loop back<br>  }<br>}</pre><h3>Parallel Execution: Intuitive Multi-Threading</h3><h3>In Temporal:</h3><pre>// Java SDK<br>Promise&lt;String&gt; promise1 = Async.function(() -&gt; activity1.doSomething());<br>Promise&lt;String&gt; promise2 = Async.function(() -&gt; activity2.doSomethingElse());<br>// Wait for both to complete<br>Promise.allOf(promise1, promise2).get();</pre><pre>// Go SDK<br>future1 := workflow.GoNamed(&quot;task1&quot;, func() { activity1.DoSomething() })<br>future2 := workflow.GoNamed(&quot;task2&quot;, func() { activity2.DoSomethingElse() })<br>// Wait for both to complete<br>future1.Get(ctx, nil)<br>future2.Get(ctx, nil)</pre><h3>In iWF:</h3><pre>// Start parallel execution<br>@Override<br>public StateDecision execute(Context context, Input input, CommandResults results, Persistence persistence) {<br>    return StateDecision.multiNextStates(<br>        StateMovement.create(PROCESS_A_STATE), <br>        StateMovement.create(PROCESS_B_STATE)<br>    );<br>}<br><br>// In the completion state, wait for both processes to finish<br>@Override<br>public CommandRequest waitUntil(Context context, Input input, Persistence persistence) {<br>    return CommandRequest.forAllOf(<br>        InternalChannelCommand.create(PROCESS_A_COMPLETE_CHANNEL),<br>        InternalChannelCommand.create(PROCESS_B_COMPLETE_CHANNEL)<br>    );<br>}</pre><p>Note that iWF provides a<a href="https://github.com/indeedeng/iwf/wiki/Basic-concepts-overview"> consistent approach across all SDKs</a>, with explicit multi-state definitions and clear synchronization using InternalChannels.</p><h3>Failure Recovery &amp; SAGA Patterns: Resilient by Design</h3><p>Error handling and compensation logic are critical for robust workflow applications, and both Temporal and iWF excel at these capabilities.</p><h3>In Temporal:</h3><pre>try {<br>    activities.bookFlight(flightDetails);<br>    activities.bookHotel(hotelDetails);<br>    activities.bookCar(carDetails);<br>} catch (ActivityFailure e) {<br>    // Compensation logic<br>    if (flightBooked) activities.cancelFlight(flightDetails);<br>    if (hotelBooked) activities.cancelHotel(hotelDetails);<br>}</pre><h3>In iWF:</h3><pre>// In workflow definition<br>@Override<br>public StateOptions getStateOptions() {<br>    return new StateOptions()<br>                // `ProceedToStateWhenExecuteRetryExhausted` controls the state that the workflow will transition to if this<br>                // state should fail more times than allowed by the configuration. By default, a state can retry until the<br>                // workflow timeout has passed. To ensure that a state can fail and move to a recovery state, there needs to be<br>                // a retry policy that tells iWF when to allow the state to fail.<br>                .setProceedToStateWhenExecuteRetryExhausted(RecoveryState.class)<br>                // `ExecuteApiRetryPolicy` tells iWF for how long or how many times a state is allowed to fail and retry. Here,<br>                // a new retry policy is set with its maximum number of attempts set to 1, so this state will only ever run<br>                // once. If `MaximumAttempts` is set to 0, the state can retry as many times as can fit within the workflow<br>                // timeout. The retry policy also includes `MaximumAttemptsDurationSeconds`, which will allow the state to retry<br>                // as many times as can fit in the number of seconds set. Like `MaximumAttempts`, if<br>                // `MaximumAttemptsDurationSeconds` is set to 0, the state can retry until the workflow timeout is reached. If<br>                // both `MaximumAttempts` and `MaximumAttemptsDurationSeconds` are set, whichever threshold is reached first<br>                // will determine when the state has finished retrying.<br>                .setExecuteApiRetryPolicy(new RetryPolicy()<br>                        .maximumAttempts(5));<br>}<br><br><br>// In compensation state<br>@Override<br>public StateDecision execute(Context context, Input input, CommandResults results, Persistence persistence) {<br>    // Same input as the failed state, contains all necessary context<br>    // Implement compensation logic here<br>    return StateDecision.gracefulCompletingWorkflow();<br>}</pre><h3>Resettable Timers: Avoiding Common Pitfalls</h3><p>Implementing durable timers that can be reset by external events is one of the most unintuitive patterns in Temporal due to its execution model.</p><h3>In Temporal (common mistake):</h3><pre>public class UpdatableTimerWorkflowImpl implements UpdatableTimerWorkflow {<br>    private DateTime timerFireTime;<br>    <br>    @Override<br>    public void start() {<br>        // This doesn&#39;t work as expected!<br>        Workflow.await(Duration.between(now(), timerFireTime));<br>        activities.sendEmail();<br>    }<br>    <br>    @Override<br>    public void updateTimer() {<br>        timerFireTime = Workflow.now().plusDays(7);<br>    }<br>}</pre><h3>In Temporal (correct implementation):</h3><pre>public class UpdatableTimerWorkflowImpl implements UpdatableTimerWorkflow {<br>    private boolean operationalSignal;<br>    private boolean shouldStayOpen = true;<br>    <br>    @Override<br>    public void start() {<br>        while (shouldStayOpen) {<br>            boolean timerFired = !Workflow.await(<br>                Duration.ofDays(7), <br>                () -&gt; operationalSignal<br>            );<br>            <br>            if (!timerFired) {<br>                // Signal was received<br>                operationalSignal = false;<br>            } else {<br>                // Timer fired<br>                shouldStayOpen = false;<br>            }<br>        }<br>        activities.sendEmail();<br>    }<br>    <br>    @Override<br>    public void updateTimer() {<br>        operationalSignal = true;<br>    }<br>}</pre><h3>In iWF:</h3><pre>class UpdatableTimerState implements WorkflowState&lt;Void&gt; {<br><br>    @Override<br>    public CommandRequest waitUntil(Context context, ...) {<br>        return CommandRequest.forAnyCommandCompleted(<br>                TimerCommand.createByDuration(TIMEOUT_DURATION)),<br>                InternalChannelCommand.create(CHANNEL_RESET_TIMER)<br>        );<br>    }<br><br>    @Override<br>    public StateDecision execute(Context context, Void input, CommandResults commandResults,...) {<br>        TimerCommandResult timer = commandResults.getTimerResult(0);<br>        if (timer.getStatus() == FIRED) {<br>              // timer fires, call API to send an email<br>              svc.sendEmail(...);<br>              // assuming we want to complete workflow here<br>              return StateDecision.completeWorkflow();<br>        }<br>        // otherwise the channel receives a message to reset the timer<br>        return StateDecision.singleNextState(UpdatableTimerState.class);<br>    }<br>}</pre><h3>Interactive Workflows: Seamless External Communication</h3><p>Both Temporal and iWF excel at enabling interactive workflows that can respond to external events and queries.</p><h3>In Temporal:</h3><pre>// Query method for reading data<br>@QueryMethod<br>public String getStatus() {<br>    return currentStatus;<br>}<br><br>// Signal method for updating data<br>@SignalMethod<br>public void updateStatus(String newStatus) {<br>    this.currentStatus = newStatus;<br>    this.statusUpdated = true;<br>}<br>// Main workflow method<br>@WorkflowMethod<br>public void execute() {<br>    // ... workflow logic ...<br>    Workflow.await(() -&gt; statusUpdated);<br>    // ... continue execution ...<br>}</pre><h3>In iWF:</h3><p>iWF’s RPC model combines Temporal’s query, signal, and synchronous update capabilities into a single, coherent interface, simplifying interaction with running workflows.</p><pre>// RPC method combining query and signal capabilities<br>@RPC<br>public String updateStatus(Context context, String newStatus, Persistence persistence, Communication communication) {<br>    String oldStatus = persistence.getDataAttribute(&quot;Status&quot;, String.class);<br>    persistence.setDataAttribute(&quot;Status&quot;, newStatus);<br>    communication.publishInternalChannel(STATUS_UPDATED_CHANNEL, null);<br>    return &quot;Status updated from &quot; + oldStatus + &quot; to &quot; + newStatus;<br>}<br></pre><h4>WaitForStateCompletion: Synchronous Operations for Asynchronous Processes</h4><p>Waiting for specific workflow states to complete is a common pattern for external systems that need to synchronize with workflow progress.</p><h3>In Temporal:</h3><pre>// Using synchronous update with polling<br>public class MyWorkflowImpl implements MyWorkflow {<br>    private boolean state1Completed = false;<br>    <br>    @WorkflowMethod<br>    public void execute() {<br>        // ... workflow logic ...<br>        Workflow.sleep(10s);<br>        state1Completed = true;<br>        // ... more workflow logic ...<br>    }<br>    <br>    @UpdateMethod<br>    public void waitForState1Completed() {<br>        return Workflow.await( ()-&gt; {state1Completed} );<br>    }<br>}<br><br>// Client code<br>WorkflowClient client = WorkflowClient.newInstance(service)) <br>MyWorkflow workflow = client.newWorkflowStub(MyWorkflow.class, workflowId);<br>workflow.waitForState1Completed();<br></pre><h3>In iWF:</h3><p>iWF provides a simple client API to wait for a specific state to complete.</p><pre>// Client code<br>client.waitForStateExecutionCompletion(wfId, State1.class);</pre><h3>Handling Edge Cases: Channel Draining</h3><p>Both Temporal and iWF provide mechanisms for handling edge cases like draining signals and managing concurrent access to workflow state.</p><p>See design patterns of <a href="https://medium.com/@qlong/draining-external-messages-channel-design-pattern-in-iwf-7bcb2d66b971">Draining External Messages(Channel)</a> and <a href="https://medium.com/@qlong/draining-internal-messages-channel-design-pattern-in-iwf-9e4bc7f0ec40">Draining Internal Messages(Channel)</a> for more details.</p><p>Especially for draining messages sent from external — because workflows may have continuously input/interaction from external, and it’s not safe to simply stop the workflow without draining the signal/messages that Temporal server has received.</p><p>In Temporal, this includes “continue-as-new”. And there are some helpers created recently but developers still need to write code to properly handling it. See “Drain all received signals in one workflow task” in the <a href="https://medium.com/@qlong/guide-to-continueasnew-in-cadence-temporal-workflow-using-iwf-as-an-example-part-1-c24ae5266f07">Guide to ContinueAsNew in Temporal Workflow</a>.</p><p>In iWF, there is no continue-as-new exposed to users. But they still need to handle “completing/failing workflow with draining signal/internal channels. This is a <a href="https://github.com/longquanzheng/iwf-samples-internal/blob/main/src/main/java/com/indeed/iwf/patterns/workflow/drainchannels/signal/DrainSignalChannelsWorkflow.java#L129">special feature </a>in IWF to do an atomic check on the channel emptiness and make a decision to go to next or complete the workflow.</p><h3>Handling Edge Cases: Concurrency with Persistence Locking in iWF</h3><p>iWF provides explicit mechanisms for handling edge cases concurrently read+write operations on data attributes to avoid any racing conditions.</p><pre>@RPC(dataAttributesLocking = {&quot;item1&quot;, &quot;item2&quot;})<br>public void updateItems(Context context, ItemsInput input, Persistence persistence) {<br>    // Safe atomic updates to multiple attributes<br>    Item item1 = persistence.getDataAttribute(&quot;item1&quot;, Item.class);<br>    Item item2 = persistence.getDataAttribute(&quot;item2&quot;, Item.class);<br>    <br>    // Modify items...<br>    <br>    persistence.setDataAttribute(&quot;item1&quot;, item1);<br>    persistence.setDataAttribute(&quot;item2&quot;, item2);<br>}</pre><h3>Building Sophisticated DSLs on iWF</h3><p>One of the most impressive capabilities of iWF is its ability to serve as a foundation for building domain-specific workflow languages. Using just four workflow states, iWF can power a sophisticated JSON/YAML-based DSL with features like. I will provide an example soon on this.</p><h3>Conclusion: The iWF Difference</h3><p>iWF’s design patterns demonstrate a reimagined approach to workflow programming that maintains the power and resilience of Temporal while offering:</p><ol><li><strong>Intuitive Programming Models</strong>: The two-phase execution model and clear separation of concerns make complex patterns more accessible</li><li><strong>Consistent APIs</strong>: Similar patterns and approaches across all language SDKs</li><li><strong>Reduced Complexity</strong>: Fewer options and more declarative approaches to common patterns</li><li><strong>Explicit Edge Case Handling</strong>: Built-in mechanisms for handling complex synchronization scenarios</li></ol><p>For Temporal users looking for a more streamlined workflow programming experience, iWF offers a compelling alternative that simplifies development while preserving the core benefits of durable execution.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f58e4793cfa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[iWF Overview for Temporal Users-Part2: Architectural Difference]]></title>
            <link>https://medium.com/@qlong/iwf-overview-for-temporal-users-part2-architectural-difference-f91a0dbfd782?source=rss-56b397853e0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/f91a0dbfd782</guid>
            <dc:creator><![CDATA[Long Quanzheng]]></dc:creator>
            <pubDate>Fri, 25 Apr 2025 19:44:10 GMT</pubDate>
            <atom:updated>2026-11-03T17:21:01.722Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>NOTE</strong>: iWF is now forked as <a href="https://github.com/superdurable/dex">Dex</a> (Durable-Execution).</p><p><em>This blog continues on </em><a href="https://medium.com/@qlong/iwf-overview-for-temporal-users-part1-programming-model-difference-9f58e4793cfa"><em>Part1 Programming Model difference</em></a><em> to summarize the differences in high level.</em></p><p>When developers first encounter iWF (Indeed Workflow Framework) with Temporal experience, they often wonder how an iWF application differs architecturally.</p><p>While iWF leverages Temporal’s powerful foundation, it introduces significant architectural innovations that simplify development, deployment, and operations. This blog post explores these key architectural differences and why they matter for building production workflow systems.</p><p>Below is the <a href="https://github.com/indeedeng/iwf/wiki/iWF-Design">architecture</a> diagram of iWF server and application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/0*st49KOS0RGo7VtqM" /></figure><h4>Poll vs Push: Communication Paradigms</h4><p>Temporal uses a polling-based model</p><pre>┌─────────────┐                ┌─────────────┐<br>│ Temporal    │ ◄─── POLL ───► │ Workflow    │<br>│ Server      │                │ Applications│<br>└─────────────┘                └─────────────┘<br>      |                              |<br>      | (polling)                    | (in-memory state)<br>      ▼                              ▼<br>┌─────────────┐                ┌─────────────┐<br>│ Task Queues │                │ Dependency  │<br>│             │                │ Services    │<br>└─────────────┘                └─────────────┘</pre><p>In contrast, iWF uses a push-based model:</p><pre>┌─────────────┐                ┌─────────────┐<br>│ iWF         │ ──── PUSH ───► │ Workflow    │<br>│ Server      │                │ Applications│<br>└─────────────┘                └─────────────┘<br>      |                              |<br>      | (push model)                 | (stateless)<br>      ▼                              ▼<br>┌─────────────┐                ┌─────────────┐<br>│ Temporal    │                │ Dependency  │<br>│ (internal)  │                │ Services    │<br>└─────────────┘                └─────────────┘</pre><p>This architectural difference has profound implications for metrics, monitoring, and operations. With iWF’s push model, workflow applications are standard micro-services that receive direct task requests, making them immediately familiar to developers and operations teams. There’s no need to monitor polling rates, tune sticky queues, or manage complex task queue partitioning — just standard HTTP metrics and conventional scaling approaches familiar to any microservice architecture.</p><p>Underneath iWF, it’s still all about Temporal — it’s managed by iWF server for you.</p><h4>Standard Micro-service Architecture</h4><p>One of iWF’s most significant advantages is its embrace of standard microservice patterns.</p><p>Using Java as example below:</p><pre>// A regular Spring component - no special SDK constraints<br>@Component<br>public class PaymentProcessingState implements WorkflowState {<br>    // Standard dependency injection<br>    @Autowired<br>    private PaymentService paymentService;<br>    <br>    @Autowired<br>    private NotificationService notificationService;<br>    <br>    @Override<br>    public CommandRequest waitUntil(Context context, StateInput input, Persistence persistence) {<br>        // Your logic here<br>    }<br>    <br>    @Override<br>    public StateDecision execute(Context context, StateInput input, <br>                               CommandResults results, Persistence persistence) {<br>        // Use injected services directly<br>        Payment payment = paymentService.processPayment(input.getPaymentInfo());<br>        notificationService.sendConfirmation(payment);<br>        <br>        // Return state decision<br>        return StateDecision.singleNextState(StateMovement.create(NEXT_STATE));<br>    }<br>}</pre><p>This approach brings numerous benefits:</p><ol><li><strong>Standard Dependency Injection</strong>: Full support for Spring’s @Autowired annotations, constructor injection, etc.</li><li><strong>Configuration Management</strong>: Use @Value, @ConfigurationProperties, and other Spring configuration mechanisms</li><li><strong>Testing</strong>: Standard Spring testing tools (@SpringBootTest, @MockBean, etc.) work seamlessly</li></ol><h4>No Distinction Between Workflow and Activity Code</h4><p>Unlike Temporal, which enforces a strict separation between workflow code and activity code (with different programming models and constraints for each), iWF eliminates this boundary:</p><h4>iWF’s Operational Simplicity</h4><p>The push model brings significant operational advantages:</p><ul><li>Standard HTTP request/response metrics (latency, throughput, error rates)</li><li>Immediate task processing without polling delay</li><li>No need to configure or monitor polling rates</li><li>Simplified scaling — just add more application instances</li><li>No task queue partitioning required</li><li>Standard load balancer configurations apply</li></ul><p>This difference fundamentally transforms the operational experience, making iWF workflows much easier to monitor, troubleshoot, and scale.</p><h4>How iWF Leverages Temporal Underneath</h4><p>While iWF presents a different programming and deployment model, it’s built on Temporal’s reliable foundation:</p><pre>┌───────────────────────────────────────────┐<br>│                  iWF SDK                  │<br>└───────────────────────────────────────────┘<br>                      │<br>┌───────────────────────────────────────────┐<br>│               iWF Server                  │<br>└───────────────────────────────────────────┘<br>                      │<br>┌───────────────────────────────────────────┐<br>│      Temporal (optimized &amp; managed)       │<br>└───────────────────────────────────────────┘</pre><p>iWF intelligently leverages Temporal’s capabilities:</p><ul><li>Uses Temporal for workflow history and persistence</li><li>Automatically manages continue-as-new for long-running workflows</li><li>Optimizes costs through local activities when appropriate</li><li>Reduces timer usage through batching and optimization</li><li>Handles large snapshot that exceeds Temporal’s 4MB limit</li></ul><p>These optimizations happen transparently to developers, who only need to understand iWF’s simpler programming model.</p><h4>Continue-as-New: Automatic Management</h4><p>One of Temporal’s features for long-running workflows is “continue-as-new,” which iWF manages automatically:</p><ul><li>Default threshold of 100 steps (workflow states) before triggering continue-as-new</li><li>Can be overridden via WorkflowConfig when starting a workflow</li><li>Can be updated anytime during workflow execution</li><li>Not limited by Temporal’s 4MB gRPC limit — iWF uses an activity loop to page through previous run snapshots</li></ul><p>This automatic management means developers rarely need to think about continue-as-new, instead of requiring careful planning and implementation.</p><h4>Versioning: Eliminating Non-Deterministic Errors</h4><p>One of the most challenging aspects of Temporal is handling non-deterministic errors (NDEs) during versioning. An NDE occurs when code changes cause the execution sequence to differ from what’s stored in history, breaking the replay mechanism.</p><p>In iWF, there are no NDEs at all due to the architectural design:</p><ul><li>Whatever users define in code takes effect immediately upon deployment</li><li>No replay mechanism that can break with code changes</li><li>For data model changes, iWF provides a feature to manage in-flight workflows using system search attributes</li></ul><p>This fundamental difference dramatically simplifies the versioning and deployment process, eliminating one of the biggest pain points for Temporal users.</p><h4>Testing Approaches: Standard Practices Apply</h4><p>iWF simplifies testing by adopting the standard approaches:</p><ul><li>Unit testing uses standard libraries (e.g. JUnit, Mockito in Java)</li><li>Integration testing is simplified through explicit TimerCommand and WorkflowState interfaces</li><li>Timers can be skipped (fired immediately) for integration tests without waiting for actual durations</li><li>States can be tested individually, allowing for more focused and faster tests</li></ul><h4>Advanced Reset Capabilities</h4><p>Workflow reset operations are critical for handling production issues. While Temporal provides very powerful reset functionality, iWF extends this with more advanced control.</p><p>iWF allows resetting to a specific state execution, giving operators more granular control when addressing workflow issues in production.</p><h3>Strongly Typing Throughout: Maximizing Type Safety in Each Programming Language</h3><p>One of iWF’s core design principles is comprehensive type safety throughout the system. Unlike workflow engines that rely on string-based identifiers or untyped data, iWF embraces strong typing at every level.</p><p>Strong typing support varies in different SDKs due to the limitation of each language. But iWF maximize the usage of it. E.g. Python SDK support type hint to provide a similar experience like Java.</p><h4>Strongly Typed State Inputs and Outputs</h4><pre>// Define a strongly typed input data class<br>public class OrderProcessingInput {<br>    private String orderId;<br>    private CustomerDetails customer;<br>    private List&lt;OrderItem&gt; items;<br>    private PaymentMethod paymentMethod;<br>    <br>}<br><br>// State implementation with strongly typed input using generic type<br>@Component<br>public class PaymentProcessingState implements WorkflowState&lt;OrderProcessingInput&gt; {<br>    @Override<br>    public CommandRequest waitUntil(Context context, OrderProcessingInput input, Persistence persistence) {<br>        // Type-safe access to input properties<br>        String orderId = input.getOrderId();<br>        PaymentMethod method = input.getPaymentMethod();<br>        <br>        // Type-safe command creation<br>        return CommandRequest.forTimer(TimerCommand.createByDuration(30));<br>    }<br>    <br>    @Override<br>    public StateDecision execute(Context context, OrderProcessingInput input, CommandResults results, Persistence persistence) {<br>        // some logic here<br>        // Type-safe state transitions with input passing<br>        ShippingInput shippingInput = new ShippingInput(input.getOrderId(), input.getCustomer().getAddress());<br>        return StateDecision.singleNextState(StateMovement.create(ShippingState.class, shippingInput));<br>    }<br>}</pre><h4>Strongly Typed RPC Methods</h4><pre>// RPC method with strongly typed input and output<br>@Component<br>public class OrderManagementWorkflow implements ObjectWorkflow {<br>    <br>...<br>...<br><br>    @RPC<br>    public OrderUpdateResult updateShippingAddress(<br>            Context context, <br>            AddressUpdateRequest request, <br>            Persistence persistence, <br>            Communication communication) {<br>        <br>        // Type-safe data attribute access<br>        Order order = persistence.getDataAttribute(&quot;order&quot;, Order.class);<br>        <br>        // Update with strongly typed request object<br>        order.setShippingAddress(request.getNewAddress());<br>        persistence.setDataAttribute(&quot;order&quot;, order);<br>        <br>        // Type-safe channel publishing<br>        communication.publishInternalChannel(SHIPPING_UPDATE_CHANNEL, order.getId());<br>        <br>        // Return strongly typed response<br>        return new OrderUpdateResult(true, &quot;Address updated successfully&quot;);<br>    }<br>}<br><br>// Client-side invocation with type safety for RPC methods, input and output<br>final OrderManagementWorkflow rpcStub = iwfClient.newRpcStub(OrderManagementWorkflow.class, workflowId);<br>OrderUpdateResult result = client.invokeRPC(<br>    rpcStub::updateShippingAddress, request<br>);</pre><h4>Swarm for Team-Friendly Development</h4><p>While iWF loses some of the “fluentness” of writing workflows as monolithic functions, it gains significant advantages for team development:</p><ul><li>Teams can break up workflow logic into smaller WorkflowState components during design</li><li>States can be built and tested in parallel by different team members</li><li>This approach scales well for complex workflows developed across multiple teams</li><li>The explicit state model creates natural boundaries for work distribution</li></ul><h3>Conclusion: Architectural Choices Matter</h3><p>iWF’s architectural differences from Temporal aren’t just implementation details — they fundamentally change how developers build, deploy, and operate workflow applications:</p><ol><li><strong>Familiar Development Model</strong>: Standard microservices with REST interfaces</li><li><strong>Simplified Deployment</strong>: No special worker processes or task queue management</li><li><strong>Reduced Operational Complexity</strong>: Standard monitoring and scaling approaches</li><li><strong>Team-Friendly Development</strong>: Explicit state machines enable parallel development</li><li><strong>Enterprise Readiness</strong>: Advanced reset capabilities and precise control</li><li><strong>No Non-Deterministic Errors</strong>: Eliminating the biggest pain point in Temporal versioning</li><li><strong>Standard Testing Practices</strong>: Using familiar frameworks and approaches</li><li><strong>Cost Optimization</strong>: Smart use of Temporal resources under the hood</li></ol><p>By building on Temporal’s powerful foundation while introducing these architectural innovations, iWF offers a workflow system that fits more naturally into modern development practices while maintaining the reliability and durability expected from enterprise workflow systems.</p><p>For teams already using Temporal, understanding these architectural differences can help evaluate whether iWF’s approach might better suit their development model and operational requirements.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f91a0dbfd782" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Wait For State Completion Design Pattern in iWF]]></title>
            <link>https://medium.com/@qlong/wait-for-state-completion-design-pattern-in-iwf-81fb60497118?source=rss-56b397853e0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/81fb60497118</guid>
            <dc:creator><![CDATA[Long Quanzheng]]></dc:creator>
            <pubDate>Fri, 25 Apr 2025 18:14:43 GMT</pubDate>
            <atom:updated>2025-08-22T21:54:33.970Z</atom:updated>
            <content:encoded><![CDATA[<p>In modern distributed systems, coordinating between clients and long-running background workflow executions presents significant challenges. Applications often need immediate responses while workflows continue to process in the background. This <strong>Wait for State Completion Pattern</strong> addresses this challenge by allowing clients to wait for critical workflow states to complete before continuing, while letting the workflow proceed with other tasks asynchronously. This pattern enables more responsive applications while maintaining the reliability benefits of workflow-based processing.</p><h3>When to Use the Wait for State Completion Pattern</h3><p>This pattern is particularly valuable in these scenarios:</p><ul><li><strong>UI Rendering Dependencies</strong>: When frontend UI need to ensure data is persisted before proceeding, or some critical operation are successful</li><li><strong>Service Chain Optimization</strong>: When some services in a chain need immediate results while others can proceed asynchronously</li><li><strong>Replacing Polling Mechanisms</strong>: When you want to avoid implementing client polling loops to check operation status</li></ul><h3>Implementing the Wait for State Completion Pattern</h3><p>It’s extremely simple to implement this pattern using the Indeed Workflow Framework (iWF), it basically utlize the <a href="https://github.com/indeedeng/iwf/wiki/How-to-wait-for-a-workflow-state-to-complete">API of “wait for state execution completion”</a>.</p><p><a href="https://github.com/indeedeng/iwf-java-samples/tree/main/src/main/java/io/iworkflow/patterns/workflow/waitforstatecompletion">See the running sample code here.</a></p><h4>1. Define the Workflow Structure</h4><pre>public class DataProcessingWorkflow implements ObjectWorkflow {<br>    // Key for storing data in the workflow persistence layer<br>    public static final String PROCESSED_DATA_KEY = &quot;processed_data&quot;;<br>    <br>    // Service dependencies<br>    private final DataService dataService;<br>    private final NotificationService notificationService;<br>    <br>    @Override<br>    public List&lt;StateDef&gt; getWorkflowStates() {<br>        return Arrays.asList(<br>                StateDef.startingState(new PersistDataState(dataService)),<br>                StateDef.nonStartingState(new BackgroundProcessingState(notificationService))<br>        );<br>    }<br>    <br>    @Override<br>    public List&lt;PersistenceFieldDef&gt; getPersistenceSchema() {<br>        return Arrays.asList(<br>                DataAttributeDef.create(ProcessedData.class, PROCESSED_DATA_KEY)<br>        );<br>    }<br>    <br>    // RPC method to retrieve persisted data<br>    @RPC<br>    public ProcessedData getProcessedData(Context context, <br>                                        Persistence persistence, <br>                                        Communication communication) {<br>        ProcessedData data = persistence.getDataAttribute(PROCESSED_DATA_KEY, <br>                                                         ProcessedData.class);<br>        if (data == null) {<br>            throw new IllegalStateException(&quot;Data has not been processed yet&quot;);<br>        }<br>        return data;<br>    }<br>}</pre><h4>2. Implement the Critical State for Client Synchronization</h4><p>This is the state the client will wait for:</p><pre>class PersistDataState implements WorkflowState&lt;RawData&gt; {<br>    private final DataService dataService;<br>    <br>    @Override<br>    public Class&lt;RawData&gt; getInputType() {<br>        return RawData.class;<br>    }<br>    <br>    @Override<br>    public StateDecision execute(Context context, RawData input, <br>                               CommandResults commandResults, <br>                               Persistence persistence, Communication communication) {<br>        // Process and persist the data<br>        ProcessedData processedData = dataService.processData(input);<br>        <br>        // Store in the workflow&#39;s persistence layer for later retrieval via RPC<br>        persistence.setDataAttribute(DataProcessingWorkflow.PROCESSED_DATA_KEY, <br>                                   processedData);<br>        <br>        // Continue to the background processing state<br>        return StateDecision.singleNextState(BackgroundProcessingState.class, <br>                                          processedData);<br>    }<br>}</pre><h4>3. Implement the Background Processing State</h4><p>This state will continue execution after the client receives data:</p><pre>class BackgroundProcessingState implements WorkflowState&lt;ProcessedData&gt; {<br>    private final NotificationService notificationService;<br>    <br>    @Override<br>    public Class&lt;ProcessedData&gt; getInputType() {<br>        return ProcessedData.class;<br>    }<br>    <br>    @Override<br>    public StateDecision execute(Context context, ProcessedData input, <br>                               CommandResults commandResults, <br>                               Persistence persistence, Communication communication) {<br>        // Perform additional background processing<br>        // This can take longer and the client doesn&#39;t need to wait for it<br>        <br>        // For example, send notifications about the processed data<br>        notificationService.sendProcessingCompleteNotification(input);<br>        <br>        // Complete the workflow<br>        return StateDecision.gracefulCompleteWorkflow();<br>    }<br>}</pre><h4>4. Client-Side Implementation</h4><p>Here’s how a client would use this pattern to wait for the state in the backend API:</p><pre>@RestController<br>public class DataProcessingController {<br>    private final Client iwfClient;<br>    <br>    @PostMapping(&quot;/process-data&quot;)<br>    public ResponseEntity&lt;ProcessedData&gt; processData(@RequestBody RawData rawData) {<br>        // Generate a unique workflow ID<br>        String workflowId = &quot;data-processing-&quot; + UUID.randomUUID().toString();<br>        <br>        // Start the workflow with the wait-for-state-completion option<br>        iwfClient.startWorkflow(<br>                DataProcessingWorkflow.class,<br>                workflowId,<br>                3600, // timeout in seconds<br>                rawData,<br>                WorkflowOptions.extendedBuilder()<br>                        .waitForCompletionState(PersistDataState.class) // Specify which state to wait for<br>                        .getBuilder()<br>                        .build()<br>        );<br>        <br>        // Wait for the PersistDataState to complete<br>        iwfClient.waitForStateExecutionCompletion(workflowId, PersistDataState.class);<br>        <br>        // Retrieve the processed data via RPC<br>        DataProcessingWorkflow workflow = iwfClient.newRpcStub(<br>                DataProcessingWorkflow.class, workflowId);<br>        ProcessedData processedData = iwfClient.invokeRPC(workflow::getProcessedData);<br>        <br>        // Return the processed data to the client<br>        // Meanwhile, the workflow continues with background processing<br>        return ResponseEntity.ok(processedData);<br>    }<br>}</pre><h3>Key Benefits of the Wait for State Completion Pattern</h3><h4>1. Improved Responsiveness</h4><p>The pattern enhances system responsiveness:</p><ul><li>Clients receive data as soon as the critical processing is complete</li><li>No need to wait for the entire workflow to finish</li><li>Reduced perceived latency for user interfaces</li></ul><h4>2. Simplified Client Code</h4><p>From a development perspective, the pattern offers:</p><ul><li>Elimination of client-side polling loops</li><li>Synchronous-style programming for asynchronous operations</li><li>Clear demarcation between critical and background processing</li></ul><h4>3. Architecture Simplification</h4><p>The pattern provides architectural benefits:</p><ul><li>Replaces complex CDC (Change Data Capture) mechanisms</li><li>Reduces the need for message queues solely for status updates</li><li>Combines benefits of both synchronous and asynchronous processing models</li></ul><h3>Conclusion</h3><p>The Wait for State Completion Pattern provides a powerful approach to bridging the gap between synchronous client needs and asynchronous workflow processing. By allowing clients to wait for specific critical states to complete while workflows continue background processing, this pattern enables more responsive applications without sacrificing the reliability benefits of workflow-based architectures.</p><p>This pattern is particularly valuable for UI rendering, data persistence verification, and any scenario requiring immediate client feedback while additional processing continues in the background. By implementing the techniques and best practices described in this article, you can build more responsive, efficient applications that optimize both user experience and system resources.</p><h3>References</h3><ul><li><a href="https://github.com/indeedeng/iwf/wiki/How-to-wait-for-a-workflow-state-to-complete">How to wait for state execution completion in iWF</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=81fb60497118" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Graceful Timeout Design Pattern in iWF]]></title>
            <link>https://medium.com/@qlong/graceful-timeout-design-pattern-in-iwf-9a74f55e753a?source=rss-56b397853e0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a74f55e753a</guid>
            <dc:creator><![CDATA[Long Quanzheng]]></dc:creator>
            <pubDate>Fri, 25 Apr 2025 05:42:46 GMT</pubDate>
            <atom:updated>2025-08-22T21:51:51.320Z</atom:updated>
            <content:encoded><![CDATA[<p>In workflow automation, timeout management is a critical concern. Workflows can sometimes encounter unexpected delays or get stuck in processing loops, potentially running indefinitely and consuming valuable resources. The built-in timeout is a hardstop and doesn’t allow to perform any actions for cleanup.</p><p>This w<strong>orkflow Graceful Timeout Pattern</strong> addresses this challenge by providing a structured approach to enforce time constraints on workflow execution, ensuring that workflows either complete within a specified timeframe gracefully with appropriate handling.</p><h3>Implementing the Workflow Graceful Timeout Pattern</h3><p>Let’s explore how to implement this pattern using the Indeed Workflow Framework (iWF):</p><p><a href="https://github.com/indeedeng/iwf-java-samples/tree/main/src/main/java/io/iworkflow/patterns/workflow/timeout">See the running sample code here.</a></p><h4>1. Define the Basic Workflow Structure</h4><pre>public class GracefulTimeoutWorkflow implements ObjectWorkflow {<br>    @Override<br>    public List&lt;StateDef&gt; getWorkflowStates() {<br>        return List.of(<br>                StateDef.startingState(new InitState()),<br>                StateDef.nonStartingState(new TimeoutMonitorState()),<br>                StateDef.nonStartingState(new MainTaskState())<br>        );<br>    }<br>}</pre><h4>2. Implement the Initial State</h4><p>The initial state sets up both the main task and timeout monitor to run in parallel:</p><pre>class InitState implements WorkflowState&lt;WorkflowParameters&gt; {<br>    @Override<br>    public Class&lt;WorkflowParameters&gt; getInputType() {<br>        return WorkflowParameters.class;<br>    }<br>    <br>    @Override<br>    public StateDecision execute(Context context, WorkflowParameters params, <br>                               CommandResults commandResults, <br>                               Persistence persistence, Communication communication) {<br>        // Start both the main task and timeout monitor in parallel<br>        return StateDecision.multiNextStates(<br>                // The timeout monitor state<br>                StateMovement.create(TimeoutMonitorState.class, params.getTimeoutDuration()),<br>                <br>                // The main task state<br>                StateMovement.create(MainTaskState.class, params.getTaskInput())<br>        );<br>    }<br>}</pre><h4>3. Implement the Timeout Monitor State</h4><p>This state is responsible for tracking the elapsed time and terminating the workflow if the timeout is reached:</p><pre>class TimeoutMonitorState implements WorkflowState&lt;Duration&gt; {<br>    @Override<br>    public Class&lt;Duration&gt; getInputType() {<br>        return Duration.class;<br>    }<br>    <br>    @Override<br>    public CommandRequest waitUntil(Context context, Duration timeoutDuration, <br>                                  Persistence persistence, Communication communication) {<br>        // Wait for the specified timeout duration<br>        return CommandRequest.forAnyCommandCompleted(<br>                TimerCommand.createByDuration(timeoutDuration)<br>        );<br>    }<br>    <br>    @Override<br>    public StateDecision execute(Context context, Duration timeoutDuration, <br>                               CommandResults commandResults, <br>                               Persistence persistence, Communication communication) {<br>        // The timer has fired, meaning the timeout has been reached<br>        // Log the timeout event<br>        persistence.recordEvent(&quot;workflow-timeout&quot;, Map.of(<br>                &quot;workflowId&quot;, context.getWorkflowId(),<br>                &quot;timeoutDuration&quot;, timeoutDuration.toString(),<br>                &quot;timestamp&quot;, Instant.now().toString()<br>        ));<br>        <br>        // Force the workflow to fail, which will terminate the MainTaskState as well<br>        return StateDecision.forceFailWorkflow(<br>                &quot;Workflow exceeded maximum allowed execution time of &quot; + timeoutDuration<br>        );<br>    }<br>}</pre><h4>4. Implement the Main Task State</h4><p>This state performs the actual work of the workflow:</p><pre>class MainTaskState implements WorkflowState&lt;TaskInput&gt; {<br>    @Override<br>    public Class&lt;TaskInput&gt; getInputType() {<br>        return TaskInput.class;<br>    }<br>    <br>    @Override<br>    public CommandRequest waitUntil(Context context, TaskInput input, <br>                                  Persistence persistence, Communication communication) {<br>        // For time-consuming operations, we might need to wait for external events<br>        // This could be timers, signals, or other command completions<br>        <br>        // For demonstration, we&#39;ll use a timer to simulate work<br>        // In a real implementation, this might wait for actual task completion signals<br>        if (input.isSlowExecution()) {<br>            // Simulate a slow-running task<br>            return CommandRequest.forAnyCommandCompleted(<br>                    TimerCommand.createByDuration(Duration.ofMinutes(5))<br>            );<br>        } else {<br>            // Simulate a quick task<br>            return CommandRequest.forAnyCommandCompleted(<br>                    TimerCommand.createByDuration(Duration.ofSeconds(10))<br>            );<br>        }<br>    }<br>    <br>    @Override<br>    public StateDecision execute(Context context, TaskInput input, <br>                               CommandResults commandResults, <br>                               Persistence persistence, Communication communication) {<br>        // Task has completed successfully<br>        // Perform any final processing<br>        <br>        // Record successful completion<br>        persistence.recordEvent(&quot;task-completed&quot;, Map.of(<br>                &quot;workflowId&quot;, context.getWorkflowId(),<br>                &quot;timestamp&quot;, Instant.now().toString()<br>        ));<br>        <br>        // Force complete the workflow, which will terminate the TimeoutMonitorState as well<br>        return StateDecision.forceCompleteWorkflow(&quot;Task completed successfully&quot;);<br>    }<br>}</pre><h3>Conclusion</h3><p>The Workflow Graceful Timeout Pattern provides a robust solution for managing execution time in workflow systems. By implementing parallel execution paths for the main task and timeout monitoring, this pattern ensures workflows either complete successfully or terminate gracefully when time limits are exceeded.</p><p>This pattern is particularly valuable in scenarios involving external dependencies, long-running operations, or resource-constrained environments. By implementing the techniques and best practices described in this article, you can build workflow systems that maintain predictable execution times and gracefully handle timeout scenarios, enhancing overall system reliability and user experience.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a74f55e753a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Concurrency Control Design Pattern in iWF]]></title>
            <link>https://medium.com/@qlong/parent-child-workflow-design-pattern-in-iwf-046742cf435f?source=rss-56b397853e0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/046742cf435f</guid>
            <dc:creator><![CDATA[Long Quanzheng]]></dc:creator>
            <pubDate>Fri, 25 Apr 2025 05:32:29 GMT</pubDate>
            <atom:updated>2025-08-22T21:52:58.533Z</atom:updated>
            <content:encoded><![CDATA[<p>This Pattern establishes a hierarchical relationship between coordinating workflows (parents) and child workflows (usually for processing actual tasks).</p><h3>When to Use this Workflow Pattern</h3><p>This pattern is particularly valuable in these scenarios:</p><ul><li><strong>Large-Scale Data Processing</strong>: When processing large datasets that can be divided into smaller chunks</li><li><strong>High-Volume Task Execution</strong>: When handling a high volume of similar tasks that can be processed independently</li><li><strong>Controlled Parallelism</strong>: When system constraints require precise control over concurrency of child workflow executions</li><li><strong>Long-Running Operations</strong>: When individual tasks may take substantial time to complete</li><li><strong>Resource Management</strong>: When distribution of work across multiple processing units is needed</li></ul><h3><a href="https://github.com/indeedeng/iwf-java-samples/tree/main/src/main/java/io/iworkflow/patterns/workflow/scalableparallel">See it here.</a></h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=046742cf435f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Parallel Execution Design Pattern in iWF]]></title>
            <link>https://medium.com/@qlong/parallel-execution-design-pattern-in-iwf-17ed9f514194?source=rss-56b397853e0d------2</link>
            <guid isPermaLink="false">https://medium.com/p/17ed9f514194</guid>
            <dc:creator><![CDATA[Long Quanzheng]]></dc:creator>
            <pubDate>Fri, 25 Apr 2025 05:18:33 GMT</pubDate>
            <atom:updated>2025-08-22T21:48:11.184Z</atom:updated>
            <content:encoded><![CDATA[<p>In the world of workflow automation, efficiency is often measured by how quickly and effectively tasks can be completed. Sequential processing — where tasks run one after another — is sometimes necessary due to dependencies between steps. However, many workflow scenarios involve tasks that could be executed simultaneously, which is where the <strong>Parallel Execution Pattern</strong> shines. This pattern enables multiple workflow states to run concurrently, significantly improving throughput and reducing overall execution time.</p><h3>Understanding the Parallel Execution Pattern</h3><p>The Parallel Execution Pattern is a workflow design approach that allows multiple states or tasks to execute simultaneously when they don’t depend on each other’s results. This pattern comes in two primary variants:</p><ol><li><strong>Simple Parallel Execution</strong>: Multiple states run concurrently without waiting for each other’s completion</li><li><strong>Parallel Execution with Coordination</strong>: Multiple states run concurrently, with a coordinating state that waits for all parallel tasks to complete before proceeding</li></ol><p>Both variants offer significant performance benefits by maximizing resource utilization and minimizing end-to-end execution time.</p><h3>When to Use the Parallel Execution Pattern</h3><p>This pattern is particularly valuable in these scenarios:</p><ul><li><strong>Independent Operations</strong>: When tasks have no dependencies on each other’s outcomes</li><li><strong>Batch Processing</strong>: When processing multiple items of the same type</li><li><strong>Multi-Channel Communication</strong>: When sending notifications or messages through different channels</li><li><strong>Resource-Intensive Operations</strong>: When distributing workload across multiple execution threads</li><li><strong>API Fan-Out</strong>: When making multiple independent API calls to different services</li></ul><h3>Implementing the Parallel Execution Pattern</h3><p>Let’s explore how to implement this pattern using the Indeed Workflow Framework (iWF):</p><p>See the <a href="https://github.com/indeedeng/iwf-java-samples/tree/main?tab=readme-ov-file#design-patterns">sample code here</a>.</p><h4>1. Simple Parallel Execution</h4><p>The simplest form of this pattern executes multiple independent states in parallel:</p><pre>public class SimpleParallelWorkflow implements ObjectWorkflow {<br>    @Override<br>    public List&lt;StateDef&gt; getWorkflowStates() {<br>        return List.of(<br>                StateDef.startingState(new InitState()),<br>                StateDef.nonStartingState(new TaskAState()),<br>                StateDef.nonStartingState(new TaskBState())<br>        );<br>    }<br>}<br><br>class InitState implements WorkflowState&lt;UserData&gt; {<br>    @Override<br>    public StateDecision execute(Context context, UserData input, <br>                                CommandResults commandResults, <br>                                Persistence persistence, <br>                                Communication communication) {<br>        // Launch multiple states in parallel<br>        return StateDecision.multiNextStates(<br>                StateMovement.create(TaskAState.class, input.getEmailAddress()),<br>                StateMovement.create(TaskBState.class, input.getPhoneNumber())<br>        );<br>    }<br>}<br>class TaskAState implements WorkflowState&lt;String&gt; {<br>    @Override<br>    public StateDecision execute(Context context, String emailAddress, <br>                                CommandResults commandResults, <br>                                Persistence persistence, <br>                                Communication communication) {<br>        // Process email-related task<br>        sendEmail(emailAddress, &quot;Welcome to our platform!&quot;);<br>        <br>        // Complete the workflow from this branch<br>        return StateDecision.gracefulCompleteWorkflow();<br>    }<br>}<br>class TaskBState implements WorkflowState&lt;String&gt; {<br>    @Override<br>    public StateDecision execute(Context context, String phoneNumber, <br>                                CommandResults commandResults, <br>                                Persistence persistence, <br>                                Communication communication) {<br>        // Process SMS-related task<br>        sendSMS(phoneNumber, &quot;Thanks for signing up!&quot;);<br>        <br>        // Complete the workflow from this branch<br>        return StateDecision.gracefulCompleteWorkflow();<br>    }<br>}</pre><p>In this implementation, both TaskAState and TaskBState execute concurrently, and the workflow completes when both states complete.</p><h4>2. Parallel Execution with Await</h4><p>When you need to synchronize after parallel execution, use this variant:</p><pre>public class ParallelWithAwaitWorkflow implements ObjectWorkflow {<br>    public static final String COMPLETION_CHANNEL = &quot;task_completion_channel&quot;;<br>    <br>    @Override<br>    public List&lt;StateDef&gt; getWorkflowStates() {<br>        return List.of(<br>                StateDef.startingState(new InitState()),<br>                StateDef.nonStartingState(new ProcessItemState()),<br>                StateDef.nonStartingState(new CoordinatorState())<br>        );<br>    }<br>    <br>    @Override<br>    public List&lt;CommunicationMethodDef&gt; getCommunicationSchema() {<br>        return List.of(<br>                InternalChannelDef.create(String.class, COMPLETION_CHANNEL)<br>        );<br>    }<br>}<br><br>class InitState implements WorkflowState&lt;Integer&gt; {<br>    @Override<br>    public StateDecision execute(Context context, Integer itemCount, <br>                                CommandResults commandResults, <br>                                Persistence persistence, <br>                                Communication communication) {<br>        List&lt;StateMovement&gt; movements = new ArrayList&lt;&gt;();<br>        <br>        // Add coordinator state<br>        movements.add(StateMovement.create(CoordinatorState.class, itemCount));<br>        <br>        // Add parallel processing states<br>        for (int i = 1; i &lt;= itemCount; i++) {<br>            movements.add(StateMovement.create(<br>                    ProcessItemState.class, <br>                    &quot;Item-&quot; + i<br>            ));<br>        }<br>        <br>        // Launch all states in parallel<br>        return StateDecision.multiNextStates(movements);<br>    }<br>}<br>class ProcessItemState implements WorkflowState&lt;String&gt; {<br>    @Override<br>    public StateDecision execute(Context context, String itemId, <br>                                CommandResults commandResults, <br>                                Persistence persistence, <br>                                Communication communication) {<br>        // Process the individual item<br>        processItem(itemId);<br>        <br>        // Signal completion to the coordinator<br>        communication.publishInternalChannel(<br>                ParallelWithAwaitWorkflow.COMPLETION_CHANNEL, <br>                itemId<br>        );<br>        <br>        // End this state&#39;s execution path<br>        return StateDecision.deadEnd();<br>    }<br>}<br>class CoordinatorState implements WorkflowState&lt;Integer&gt; {<br>    @Override<br>    public CommandRequest waitUntil(Context context, Integer itemCount, <br>                                  Persistence persistence, <br>                                  Communication communication) {<br>        // Create commands to wait for all item completions<br>        List&lt;BaseCommand&gt; commands = new ArrayList&lt;&gt;();<br>        for (int i = 0; i &lt; itemCount; i++) {<br>            commands.add(InternalChannelCommand.create(<br>                    ParallelWithAwaitWorkflow.COMPLETION_CHANNEL<br>            ));<br>        }<br>        <br>        // Wait for ALL commands to complete<br>        return CommandRequest.forAllCommandCompleted(<br>                commands.toArray(new BaseCommand[0])<br>        );<br>    }<br>    <br>    @Override<br>    public StateDecision execute(Context context, Integer itemCount, <br>                                CommandResults commandResults, <br>                                Persistence persistence, <br>                                Communication communication) {<br>        // All parallel tasks have completed<br>        summarizeResults(commandResults);<br>        <br>        // Complete the workflow<br>        return StateDecision.gracefulCompleteWorkflow();<br>    }<br>}</pre><p>In this implementation, the CoordinatorState waits for all ProcessItemState instances to signal completion before proceeding to complete the workflow.</p><h3>Advanced Implementation Techniques</h3><h4>Dynamic Parallelism</h4><p>You can create parallel branches based on runtime data:</p><pre>@Override<br>public StateDecision execute(Context context, List&lt;String&gt; items, <br>                           CommandResults commandResults, <br>                           Persistence persistence, <br>                           Communication communication) {<br>    // Dynamically create state movements based on input list<br>    List&lt;StateMovement&gt; movements = new ArrayList&lt;&gt;();<br>    <br>    // Add coordinator<br>    movements.add(StateMovement.create(CoordinatorState.class, items.size()));<br>    <br>    // Add one state per item dynamically<br>    for (String item : items) {<br>        movements.add(StateMovement.create(ProcessItemState.class, item));<br>    }<br>    <br>    return StateDecision.multiNextStates(movements);<br>}</pre><h4>Parallel Processing with Batching</h4><p>Futher moore, you can process items in parallel, but in controlled batch sizes:</p><pre>@Override<br>public StateDecision execute(Context context, List&lt;String&gt; items, <br>                           CommandResults commandResults, <br>                           Persistence persistence, <br>                           Communication communication) {<br>    List&lt;StateMovement&gt; movements = new ArrayList&lt;&gt;();<br>    <br>    // Add coordinator<br>    movements.add(StateMovement.create(CoordinatorState.class, items.size()));<br>    <br>    // Determine optimal batch size based on load<br>    int batchSize = 10;<br>    <br>    // Create batched processing states<br>    for (int i = 0; i &lt; items.size(); i += batchSize) {<br>        // Calculate end index, ensuring we don&#39;t go out of bounds<br>        int endIndex = Math.min(i + batchSize, items.size());<br>        <br>        // Extract batch<br>        List&lt;String&gt; batch = items.subList(i, endIndex);<br>        <br>        // Create state for batch<br>        movements.add(StateMovement.create(BatchProcessState.class, batch));<br>    }<br>    <br>    return StateDecision.multiNextStates(movements);<br>}</pre><h3>Conclusion</h3><p>The Parallel Execution Pattern offers a powerful approach to improving workflow efficiency and throughput by executing independent tasks concurrently. Whether you’re implementing simple parallel branches or complex coordinated parallel processing with await mechanisms, this pattern can significantly reduce end-to-end processing time and improve resource utilization.</p><p>By understanding the key variants, implementation techniques, and best practices presented in this article, you can effectively apply the Parallel Execution Pattern to your workflow automation challenges, creating more efficient and responsive workflow applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17ed9f514194" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>