Get started with durable computing
In my article Fixing 30 years of software development pains? I spoke about how much easier software development used to be. The hardware was more reliable than todays commodity servers are, uptimes could be measured in years and not weeks because the computers had redundant CPUs onboard, error correcting memory, RAID disks that were directly on the bus and guaranteed data storage, and uninterruptible power supplies — when something failed you called in an engineer and they would replace parts while the computer continued to do its work, occasionally you might have to shut down temporarily to apply a patch. A contest held by Stratus to identify its longest-running server in 2010 found that a system had been running continuously for 24 years in Dearborn, Michigan. This kind of reliability made application programming easier because code did not have to handle failures, it was all business logic — a dream in todays development environment where teams are pushed to deliver business functionality rather than sweat over what needs to happen when things go wrong.
In contrast, early experiences with PCs could be very frustrating:
Customer: Hi. I was working on an Excel file yesterday and forgot to save it before my computer crashed. Can you help me recover it?
Microsoft Office Questions, justanswer.com
Work was lost when PCs crashed and the user had not diligently saved their work every so often, or if they accidentally pressed the wrong dialog button when quitting the application. A similar issue afflicts todays business computing.
When computing is distributed maintaining correct state becomes complicated, failures add complexity often to both the code and the systems architecture. The code may have to implement exception handling for recovery (often badly implemented), and use a consumer-producer design with the associated complications of introducing message-driven architecture. None of this incidental complexity arises from the business logic, it’s additional complication arising from the technical limitations. It would be great if all that added complication could be avoided, so that once again developers can focus on delivery of pure business logic.
Durable computing is a technique that makes it easier for teams to address resiliency and reliability in distributed systems. It involves handling state in such a way to ensure requests can be run to completion, reducing the risks of failure or downtime.
Durable computing, Richard Gall
Of course not every application requires this durability, however financial and other mission critical systems still need it, which is why banks are still running on systems with “six nines” uptime, instead of migrating to the cloud.
I recently showed how the ideas of inverted mocking and functional core imperative shell can be applied to implement a simple order processing application in TypeScript. Once business logic is functionally pure it then becomes fairly easy to implement durable computing — in this completely working demo project I used Temporal. What is Temporal?
Temporal
Temporal supports todays popular programming languages such as Python, Go, TypeScript, Ruby, C#, Java and PHP and it’s a mature technology used by businesses such as NVIDIA, Salesforce and Twilio.
Temporal is based on the concept of a workflow and a workflow consists of activities (input/output)— it’s these activities that Temporal can give assurances of completion thereby guaranteeing the workflow is a success (given there is no serious disaster). Not only this, but should the servers running your application with Temporal shut down, they will recover the workflows and restart them off from where they were terminated — “belt and braces”. This idiom lends itself well to implementing the Saga Design Pattern and makes it easy to implement the compensatory effects (assuming idempotency).
For todays applications that often make use of SAAS to (for example) process payments, complete a KYC verification, send messages and orchestrate numerous activities that can fail, this workflow idiom is designed to fit the need for distributed transactions without using heavyweight solutions like XA. This is also a useful for running long lived agentic workflows.
In my demo project the process order workflow module includes; retry configurations, proxies to wrap effects with Temporal assurances, and the implementation of an effects provider that delegates to the Temporal activities. The code snippet below shows how these 3 pieces work together to build a workflow including a call to an API.
const apiActivityOptions: ActivityOptions = {
startToCloseTimeout: '150s',
retry: {
initialInterval: 500,
backoffCoefficient: 2,
maximumAttempts: 10,
maximumInterval: 3000,
},
}
const {
getDiscountRules,
} = proxyActivities<Pick<Activities, 'getDiscountRules'>>(apiActivityOptions);
export async function processOrderWorkflow(orderId: string): Promise<Either<NonEmptyList<string>, ProcessedOrder>> {
const temporalEffects: AppEffects = {
...
pricing: {
getDiscountRules: getDiscountRules,
},
...When the business logic layer calls getDiscountRules, Temporal will take care of the retries as well as restore and continue the workflow should it be suspended. As a bonus it also provides observability of the workflow execution so you can follow everything that happens! You may still want a circuit breaker to prevent new workflows starting during failure, but all the workflows that are in flight can be assured of completion. The business logic stays pure and because of the functional purity, can be tested without the complexity of mocking the externalities.
It is quite easy to get set up and working with Temporal or even to just migrate critical parts of your code onto this platform — the focus of complication is shifted to hosting and deployment.
Golem Cloud
A relatively new platform that takes a more radical approach to durable computing is Golem Cloud. Golem currently supports agents written with TypeScript, Rust, MoonBit, Go and Scala and which get compiled to WASM. That limits external calls to using HTTP — although database connection is also supported, more recently interfaces to support agentic platforms were added. Why would you want to deploy your applications as WASM modules like that instead of as more typical docker containers?
Get John Coleman’s stories in your inbox
Join Medium for free to get updates from this writer.
Running WASM offers a number of powerful benefits to Golem:
- Golem can snapshot a program’s memory and execution state with extreme instruction point precision. This means that other than the mechanisms to utilise agents, there is minimal impact of the durability SDK on written code, so the learning curve is minimal.
- If a node goes down, Golem uses an “oplog” (operation log) to restore the worker on another machine at the millisecond it left off — note that this recovery process can introduce latency, as also the case with reloading passivated agents.
- Golem agents run in a restricted sandbox. Because this isolation is so strong and lightweight, Golem can safely run code from many different users on the same physical hardware without them interfering with one another. An agent that runs out of memory will not bring the entire runtime down.
- WASM modules can be compact (100s of kilobytes vs hundreds of megabytes) and therefore load in milliseconds vs full containers.
- Language interoperability guarantees*
The above benefits essentially provide the resilience of mainframes with the flexibility and scalability of distributed computing, and make development and deployment painless. This means applications can be written without the clutter of coping mechanisms such as retry handling, or other architectural complexity such as replication and message queuing, but you will still have to think about managing memory. The limitations of using WASM do introduce some bumps when dealing with remote systems that are not HTPP based — hopefully this will be addressed in future?
Golem is currently version 1.6 which includes many new features to suport agentic applications, but is yet to see widespread adoption. That aside, why use Golem instead of Temporal?
While temporal is opinionated on the workflow idiom, Golem makes no such assumptions and is more adaptable. Golem is agent based, and an agent could be workflow like or domain oriented — a number of patterns are possible. As Vigoo shows in the blog post Agent patterns in Golem, you can implement an agent as a state machine with workflow semantics to provide observability for multi-step workflows, rather like Temporal. Vigoo goes on to explain that agents can also represent domain entities, which is well suited to the Domain-Driven Design approach and fits the practice of building REST APIs that are entity orientated — as demonstrated, an agent could represent a Customer or an Order entity, for example (agents are also restrictive in terms of functions just working with primitive types).
Agents provide the capability to build code where the focus will be on implementing the domain in an intuitive code structure, as well as ensuring that the code execution is thoroughly robust. [I’m also thinking that domain modelling technology with code generation like Jetbrains MPS could be leveraged to generate agent code directly from the model — something I may explore when Golem is stable and more feature complete.]
If you are interested in trying Golem you can follow the official quickstart — I have used this quickstart to build a complete working TypeScript project for Node which you can find on my Github golem-agent-example repository. The main branch implements the example counter agent and includes all the instructions in one place to get set up and running easily with details for working on macOS, but you can easily adapt to other operating systems. Writing an agent is trivial as shown below, and the agents state is automatically persistent so no database required! Some ceremony is required to deploy the agent and expose the agent as a restful API (not shown), but from version 1.5, this is quite trivial.
@agent({
mount: "/counters/{name}"
})
class CounterAgent extends BaseAgent {
private readonly name: string;
private value: number = 0;
constructor(name: string) {
super();
this.name = name;
}
@prompt("Increase the count by one")
@description("Increases the count by one and returns the new value")
@endpoint({ post: "/increment" })
async increment(): Promise<number> {
this.value += 1;
return this.value;
}
}You can learn more about the above counter example in this YouTube video where John De Goes sets the project up and runs it as well as explaining more about the features of Golem and how they can be used to implement a database.
If you want to push ahead and run an integration test server on AWS for your team to collaborate for experimentation, the infra branch provides the deployment scripts you can run to set everything up using docker with EC2. Maybe you can use AI to convert the scripts to suit other cloud platforms? For production you can use Golem Cloud hosting, although the platform still seems to be developing.
If you want to self host for production using Golems open source containers, you will probably need to consider Kubernetes but as yet there is no documentation — this process will be time consuming but should be doable. For AWS this will likely require RDS and ElastiCache in addition to an EKS cluster. Unfortunately this is beyond the scope of this introduction.
The Tradeoffs
Of course there are always tradeoffs. Whether you use Temporal or another durable computing platform, you must learn some new technology and rethink how you build and deploy applications — but the approach ought to simplify development considerably. Consider also how you are going to host or utilise what is essentially a SAAS solution with ongoing costs that will probably introduce friction with established in-house CI/CD operations and change control mechanisms that also complicate delivery. However, for PoC, startups or applications where data residence and ownership is not an issue, some platforms have their own SAAS offerings available.
Final note: Achieving five nines or more availability with cloud service providers can still be a significant challenge and will likely involve a multi-region solution and then specialised hardware if you really need to reach those extra 9s — and even then you will never achieve the performance of working on a single high-availability machine.
If you want to dig into some of the pitfalls of durable execution Chris Gillums article Common Pitfalls with Durable Execution Frameworks, like Durable Functions or Temporal is a great start.
