Multi-Step Workflows and Durability
A deploy went out at 4:47 on a Tuesday and paid a customer back twice.
The run was ordinary. A customer asked for a refund, so a little task woke up to look up the order, issue the refund through our payments provider, email the customer to say it was on the way, and close the support ticket. Four steps. It had run maybe two thousand times without anyone noticing it existed. This time it issued the refund, sent the email, and then the machine it was running on got rolled in a routine deploy, right between the email and closing the ticket. Our job runner did exactly what a good job runner does with a task that never reported success. It ran it again. From the top. Looked up the order, issued the refund a second time, and sent a second email that opened with “Your refund is on its way.”
The refund was not the expensive part, though forty dollars is forty dollars. The expensive part was the trust. The customer now had two payments and two emails and a reasonable question about whether anyone was minding the store. The bug was not in any single step. Every step worked. The bug was that a run died in the middle and the retry replayed the parts that had already touched the world.
That is the whole subject of this article. A run is a sequence of steps, any of which can fail, and the naive fix (just retry it) repeats side effects. The good news is that this is not a new problem. Backend engineers have been making jobs survive crashes for decades. The only twist here is that one of the steps in the sequence is a language model that does not do the same thing twice.
A run is a chain of steps, and any link can snap
Go back to the agent loop. Strip away the framing and a run is a list of steps executed one after another: call the model, run a tool, call the model again, run another tool. Some steps are model calls. Some are tool calls that read data. A few are tool calls that change something, a charge, an email, a row written, a message posted. Those last ones are the dangerous steps, because they are the ones the outside world remembers.
Every one of those steps is a network call, and network calls fail in all the boring ways. A 429 when the provider throttles you. A timeout when an upstream is slow. A 500 from a flaky tool. And the failure that catches everyone at least once: the process itself vanishing mid-run. A crash, an out-of-memory kill, an autoscaler scaling you down, a deploy. The step did not fail. The whole world holding the step disappeared.
When that happens, you have exactly one instinct from ordinary code, and it is wrong here: retry. Retrying a pure function is free. Retrying a run that already charged a card is not. A blind retry starts the sequence over and re-runs every step, including the ones that already left a mark. That is the double refund. The failure was at step four, and the retry cheerfully redid steps one, two, and three to get back there.
Checkpoint after every step
The fix that does the most work for the least effort: after each step finishes, write down that it finished and what it returned. Then, if the run has to start again, skip the steps that already have an answer recorded and pick up at the first one that does not. You resume from the point of failure instead of the beginning.
Here is the buggy version first, so the fix is obvious. This is the refund run as I wrote it, and every line is a landmine on retry:
async function runTask(task) {
const order = await lookupOrder(task.orderId);
await issueRefund(order, task.amount); // money leaves the building
await emailCustomer(order.email); // "your refund is on its way"
await closeTicket(task.ticketId);
}
// the job runner retries runTask() on failure, from the top, every timeNow the checkpointed version. The trick is a tiny wrapper that records each step under a name. Run the step once, save its result, and on any future pass return the saved result instead of running it again:
async function runTask(runId, task) {
const state = (await loadRun(runId)) ?? { done: {}, out: {} };
const order = await step(state, "lookup", () => lookupOrder(task.orderId));
await step(state, "refund", () => issueRefund(order, task.amount));
await step(state, "email", () => emailCustomer(order.email));
await step(state, "close", () => closeTicket(task.ticketId));
}
async function step(state, name, fn) {
if (state.done[name]) return state.out[name]; // already ran once: skip it
const result = await fn();
state.out[name] = result;
state.done[name] = true;
await saveRun(state); // checkpoint, right here
return result;
}That is the entire idea. The step() wrapper is memoization by name, backed by durable storage instead of a Map. The first time through, every step runs and gets saved. If the process dies after the refund but before the email, the retry reloads state, sees refund is already done, skips it, and starts at email. The customer gets refunded once.
Keep the run in a store, not in a process
Look closely at loadRun and saveRun in that code, because they are carrying the actual weight. The checkpoint only helps if it outlives the thing that crashed. A Map in memory is useless here. The most common way a long run dies is the process going away, and the process going away takes your memory with it.
So the run has to live somewhere durable: a row in a database, a record in a key-value store, a document keyed by run id. After each step you write the updated state back. When a worker picks the run up (the first time, or after a crash) it reads that record and knows exactly how far the last attempt got. This is the same shape as any background job: the queue hands a worker a job, the worker makes progress and records it, and if that worker dies another worker can take the same job and carry on. The run is the job. The checkpoint is the job’s progress, saved outside any single worker.
There is a subtlety worth stating plainly, because it bites people who assume the store is magic. The checkpoint is only as good as when you write it. Write it after the step and a crash between the step and the write means the step ran but was never recorded, so the resume runs it again. Write it before the step and a crash means you recorded work that never happened. Neither ordering is perfect on its own, which is the entire reason for the next section.
Checkpoints do not cover side effects. Idempotency does.
Here is the gap. Your refund step calls the payments provider, the charge succeeds, and then the process dies, one line before saveRun. The money moved. The log does not know it moved. On resume, the step is not marked done, so it runs again, and the customer is refunded twice. Checkpointing narrowed the window from “always double-charges” to “double-charges only if it crashes in this exact half-second,” but the window is still open, and at scale a half-second window is a Tuesday.
You cannot close that window with more careful checkpoint ordering, because there will always be a moment between “the side effect happened” and “you recorded that it happened.” The fix lives at the side effect itself. Make the action idempotent: safe to call twice, because the second call recognizes the first and does nothing new.
The mechanism is an idempotency key. You compute a key that is the same on every attempt of that step, hand it to the provider, and the provider promises that two requests with the same key produce one effect. The key has to be derived from stable inputs, never from the clock or a random source, so a replay recreates the exact same key:
// same run, same step, same order => identical key on every replay
const key = `refund:${runId}:${order.id}`;
await step(state, "refund", () =>
issueRefund(order, task.amount, { idempotencyKey: key })
);Now the double-charge is impossible, checkpoint or no checkpoint. The first attempt reaches the provider, which records the key and moves the money. The replay carries the same key, the provider sees it has already handled that key, and it returns the original result without charging again. The step still returns a value, the run still proceeds, and the ledger only moved once.
Not every action has a provider that speaks idempotency keys for you. When it does not, you build the same guarantee yourself: before the effect, check whether you already did it (a unique row, a “processed” flag keyed by that stable id), and skip if so. The idempotency keys article covers the storage patterns. The lesson to carry into agents is that checkpointing and idempotency are two different jobs. Checkpointing keeps you from redoing work. Idempotency keeps a redo from duplicating an effect when the checkpoint could not save you. Long agent runs that touch money, mail, or anything a user sees need both.
Deterministic orchestration, non-deterministic steps
Step back and notice what the step() wrapper actually bought you. It split the run into two kinds of code. There is the plain control flow that decides what happens next (call lookup, then refund, then email), and there are the steps that do the work and touch the world. That split is the single most important idea in durable execution, and it is worth making explicit, because once you see it you will structure every long run this way.
The control flow has to be replayable. When a run resumes, that code runs again from the top, walking back through the sequence to figure out where it is. For that walk to land in the same place, the control flow must be deterministic: given the same recorded step results, it makes the same decisions. Anything that would come out different on a second run is poison here. A Date.now(), a Math.random(), a crypto.randomUUID(), a direct network call, a read of a mutable global. Run twice, get two answers, and the replay diverges from the original.
So the rule is blunt: nothing non-deterministic and nothing with a side effect goes in the orchestration. It all goes inside a step. A step runs once, its result is recorded, and every later replay reads that recorded value instead of running it again. That is how you pin down time, randomness, and IO. You do them once, inside a step, and freeze the answer.
// WRONG: computed in the orchestration, so a replay produces new values
// and every downstream decision drifts
const id = crypto.randomUUID();
const now = Date.now();
// RIGHT: wrapped as steps, so the values are recorded once and replayed
const id = await step(state, "id", () => crypto.randomUUID());
const now = await step(state, "now", () => Date.now());This applies to the model call too, and that surprises people. Wrap the model call as a step, and its output is recorded on the first run and replayed on resume rather than re-requested. That matters for two reasons. You do not pay for the same generation twice. And you do not risk the model taking a different path the second time and desyncing from the tool results already in your log. The model is the most non-deterministic thing in the whole system, which makes it the thing you most want to record and never re-run.
// the model call is a step: its output is saved, then replayed for free
const plan = await step(state, "plan", () => callModel(messages, { tools }));If this feels like a lot of ceremony, that is the honest cost, and it is why durable-execution engines exist. They give you the step() wrapper, the durable log, the deterministic-replay machinery, and the retry policies as a product, so you write something close to normal async code and the engine handles the checkpointing and replay underneath. Some engines are standalone. Some agent SDKs, as of 2026, let you tag a tool call so it becomes a durable step inside a workflow, which is the same split wearing a friendlier hat. Whatever you use, the rules above are what keeps it correct: deterministic orchestration, side effects only inside steps, stable step names.
A plan of retryable steps beats one long loop
The agent loop is one free-running while: the model decides the next move every turn, and the shape of the run is not known until it ends. That flexibility is the point of an agent, but it fights durability. A blob of “the model will figure it out” is hard to checkpoint into named steps, because the steps are invented on the fly.
For runs that are long or important, it often pays to add a layer: have the model produce a plan first, a short list of concrete steps, and then execute that list as durable steps you can retry one at a time. You get named units with clear boundaries, each independently retryable and resumable, instead of one opaque loop. The reliability math alone makes the case. A run of six steps where each step succeeds 95 percent of the time finishes on the first try only about 74 percent of the time (0.95 to the sixth). Make each step independently retryable and a single flaky step no longer sinks the whole run, and you push the finish rate up into the high nineties. Same steps, same flakiness, wildly different odds of getting to the end.
// 1. the model plans (as a recorded step, so the plan is fixed on replay)
const plan = await step(state, "plan", () => makePlan(goal)); // [{ tool, args }, ...]
// 2. execute the plan as durable steps, each retryable on its own
for (const [i, task] of plan.entries()) {
await step(state, `task:${i}`, () => runTool(task.tool, task.args));
}This is not free and it is not always right. A plan made up front can be wrong, and rigid execution of a bad plan is worse than a loop that adapts. Real systems mix the two: plan the parts you can, let the model re-plan when a step surprises it, and keep the free loop for the genuinely open-ended stretches. But the instinct to reach for, when a run matters, is fewer, named, retryable steps rather than one long improvisation. You can resume a list. You cannot resume a vibe.
Waiting for a human is just a very long step
Some steps should not be automatic. Publishing the post, sending the money, deleting the records, merging the change. For those you want a person to look and approve, which is the subject of human in the loop. The durability question it raises is a good one: how do you wait for a human without holding a process open for the hours or days they might take?
The answer falls straight out of everything above. Waiting for approval is just a step whose result arrives later, from outside. The run reaches the approval point, records that it is waiting, and stops. It holds no thread, no connection, no memory. It is a row in your store with status awaiting_approval. Later, whenever later is, the person clicks approve, that click hits a route or a webhook, you flip the record and re-enqueue the run, and it resumes from exactly that point and does the next step. Because the whole run is durable, the wait can span a deploy, a restart, a weekend. A run parked for four days costs you one row, not four days of compute.
await step(state, "draft", () => writeDraft(goal));
await step(state, "request", () => requestApproval(runId, state.out.draft));
await waitForSignal(state, "approved"); // parks the run; returns in seconds or days
await step(state, "publish", () => publish(state.out.draft));Give the wait a timeout, always. A human step with no deadline is a run that hangs forever when someone forgets. Park it for a day, and if no one has decided, escalate, cancel, or fall back to a safe default. The wait is durable. It should not be eternal.
Budgets have to survive the resume
The agent loop taught you to cap a run: a ceiling on steps, tokens, cost, and wall-clock, so a confused model cannot run up a bill. Durability adds a trap to that lesson, and it is an easy one to fall into. If the budget lives in a local variable, every resume starts it back at zero.
Picture a run that retries, resumes, retries, resumes. Each fresh attempt re-reads a budget of “200 steps” and starts counting from one. A run that should have been killed at 200 steps sails past 1,000 across attempts, because no single attempt ever saw more than a few hundred. The budget has to be part of the persisted state, counted across every resume, or it is not a budget at all.
state.budget ??= { steps: 0, costUsd: 0, deadline: Date.now() + 30 * 60_000 };
// counted in the durable state, so resumes accumulate instead of resetting
if (++state.budget.steps > 200) throw new Halt("step budget");
if (state.budget.costUsd > 5) throw new Halt("cost budget");
if (Date.now() > state.budget.deadline) throw new Halt("deadline");The per-step side of this is ordinary retries and backoff. Give each step its own timeout and a bounded retry with exponential backoff, so a slow or flaky step gets a few honest attempts and then gives up instead of hammering forever. Two layers, and they do different jobs. The per-step retry rescues a single transient failure. The per-run budget stops the whole thing when retries are not converging and the run is just thrashing. Ship both, and note the deadline goes in state too, so it is a wall-clock ceiling on the real run and not on one lucky attempt.
Watch a run die and come back
Enough theory. Below is the pipeline from this whole article, fetch then summarise then notify, with no model and no network anywhere on the page. Pick a mode, pick where the crash lands, and run it. In naive mode, a crash throws away all progress and the resume starts over from step one, which means it sends the notification a second time. In checkpointed mode, the resume reads what was already saved, skips the finished steps, and the notification goes out exactly once.
The sharp case is crashing right after the notify step. That is the double refund from the opening, in miniature: the side effect already happened, and only idempotency or a committed checkpoint stops the resume from repeating it. Try it both ways and watch the “emails sent” counter.
Run naive, crash after step 3, resume, and the counter reads two. Switch to checkpointed and do the exact same thing, and it reads one, because the resume saw notify was already saved and skipped it. Now try crashing after step 1: both modes end at one email, but watch the log, naive redoes the fetch and the summarise while checkpointed jumps straight past them. That gap is wasted work and wasted money on a real run, even when no side effect doubles up.
Seeing where the time and the money went
A run that spans retries, resumes, a human pause, and a deploy is nearly impossible to reason about from a flat log. You need to see it as one thing: which step retried five times, which model call cost the most, where the run sat parked for two days, which resume finally carried it home. That is a trace, and it is the same observability discipline from the server chapters pointed at an AI run.
The useful shape is a tree. One span for the whole run, child spans for each step, and inside a step, spans for the model call and the tool call with their token counts and latencies attached. As of 2026 there is a genuine standard forming for this, an open set of conventions for naming AI spans and their attributes, so a run traced in your code shows up sensibly in whatever backend you use. It is still stabilizing, especially around multi-step agent runs, so treat the exact attribute names as a moving target and the idea as solid. The idea is what matters: a durable run is long and lumpy, and the only way to know where it spent your time and your money is to record it as a trace while it happens.
Reach for this when it earns its keep
None of this is free, so do not cargo-cult it onto everything. A two-step run that finishes in three seconds and is cheap and harmless to redo does not need a durable log, a replay engine, and idempotency keys. Wrapping it in all that machinery buys you complexity and buys your users nothing. Retrying the whole thing on failure is completely fine when the whole thing is fast and its steps do not leave marks.
The machinery earns its cost when the run is long, so redoing it from scratch is painfully slow or expensive; when it has irreversible side effects, so a blind retry can double-charge or double-send; or when it waits, on a human or a slow external process, for longer than you can hold anything open. Those are the runs where “start over from the top” is the wrong answer, and where checkpoints, idempotency, and durable resumes stop being ceremony and start being the difference between a system you trust and a deploy that pays a customer twice.
Summary
- An agent run is a chain of model calls and tool calls. Any one can fail, and the failure that hurts most is the process vanishing mid-run: a crash, an out-of-memory kill, or a deploy.
- A blind retry replays the whole chain and repeats side effects. That is how a run double-charges a card or re-sends an email. Bring the same durability thinking you already use for background jobs to a worker that happens to be non-deterministic.
- Checkpoint after every step: record that it finished and what it returned, then on resume skip finished steps and continue from the first unfinished one. Name steps stably and deterministically or the log stops matching.
- Keep the run in a durable store, not in process memory, because the process is the thing most likely to die. A fresh worker reads the record and picks up where the last one stopped.
- Checkpoints stop you redoing work but cannot cover the gap between a side effect and its record. Make the effect idempotent with a stable key so a replay is a no-op. You need both.
- Split the run into deterministic orchestration (replayable, no clock or random or IO) and steps (all side effects and non-determinism, recorded once). Wrap the model call as a step so replay reuses its output instead of paying and re-rolling the dice.
- For long or important runs, prefer a plan of named, independently retryable steps over one free-running loop. You can resume a list; you cannot resume a loop mid-improvisation.
- Waiting on a human is just a durable step whose result arrives later. Park the run, hold zero compute, resume on the signal or a timeout. Always give the wait a deadline.
- Put budgets (steps, cost, deadline) in the persisted state so they accumulate across resumes instead of resetting to zero on every attempt. Add per-step timeouts and backoff underneath.
- Trace the whole run so you can see where a long, lumpy, resumed execution spent its time and money. See tracing an AI run.
- Skip all of it for short, cheap, harmless runs. Reach for it when a run is long, touches the world irreversibly, or waits.