Event-Driven Design
The order endpoint did five things in a row, and every one had to succeed before the customer saw a confirmation. Charge the card. Write the order. Send the receipt email. Decrement inventory. Fire an analytics event. On a normal Tuesday it returned in about ninety milliseconds and nobody thought about it.
Then the email provider had a bad afternoon.
Sends that used to take 200ms started taking thirty seconds, then timing out. Because sendReceipt was awaited right there in the handler, its latency became the endpoint’s latency. Requests piled up waiting on an email nobody was reading yet. The connection pool drained. And a store that was perfectly capable of taking money stopped taking orders, because the least important of the five steps was standing in the doorway.
That is the bug this whole topic exists to prevent. The receipt, the inventory sync, the analytics ping: none of them are the job. They are things that should happen because an order was placed. Welding them onto the critical path made a reaction into a dependency, and a dependency into a single point of failure.
The welded checkout
Here is the shape of it. Read it and the problem is not obvious, which is exactly why this code is everywhere.
// checkout.js: every reaction welded onto the critical path
async function placeOrder(cart, user) {
const charge = await payments.charge(cart.total, user); // the job
const order = await orders.insert(cart, user, charge); // the job
await email.sendReceipt(user, order); // a reaction, awaited
await inventory.decrement(order.items); // a reaction, awaited
await analytics.track("purchase", order); // a reaction, awaited
return order; // the customer waited for all five
}Two costs are baked in, and they compound. The first is latency: the customer’s wait is the sum of every line, so the slowest downstream service sets the floor for the whole endpoint. The second is coupling: checkout.js now imports the email client, the inventory client, and the analytics client, and it fails if any of them fails. Add loyalty points next quarter and someone edits placeOrder again. So does the fraud team. So does whoever wires up the warehouse. The one function about taking orders slowly accretes knowledge of every system that cares about orders.
Announce, don’t call
The fix is to flip who knows whom. Instead of checkout calling the things that should happen next, it announces that something happened and stops caring who reacts. It does its own job, the two writes that genuinely must be part of placing an order, then publishes an OrderPlaced fact and returns.
// checkout.js: do the job, announce the fact, return
async function placeOrder(cart, user) {
const charge = await payments.charge(cart.total, user);
const order = await orders.insert(cart, user, charge);
await bus.publish("OrderPlaced", {
id: crypto.randomUUID(), // the event's own id, we will need it later
orderId: order.id,
userId: user.id,
items: order.items,
at: new Date().toISOString(),
});
return order; // two writes and one publish, nothing downstream
}The reactions move out into their own files, each subscribing to the fact it cares about. checkout.js has never heard of any of them.
// email-service.js: its own module, maybe its own process and deploy
bus.subscribe("OrderPlaced", async (evt) => {
await email.sendReceipt(evt.userId, evt.orderId);
});
// inventory-service.js
bus.subscribe("OrderPlaced", async (evt) => {
await inventory.decrement(evt.items);
});Look at the import arrows. Before, checkout reached out to three modules. Now those modules reach in and ask to be told. The dependency reversed, and the payoff is direct: the customer waits only for checkout’s own work, a failure in analytics cannot touch the order, and adding a sixth reaction is a new file that subscribes, not an edit to the endpoint.
If that inversion feels familiar, it should. This is the observer and pub/sub pattern, the one behind addEventListener, promoted to the scale of a whole system. In one process, bus is the emitter you can build in fifteen lines. The difference here is what sits in the middle. Across processes, that bus becomes a message broker, a separate piece of infrastructure that receives events from producers and delivers them to consumers, and the network between them changes everything about how this can fail. The mechanism is the same. The consequences are not, and most of this article is about those consequences.
Facts, not orders
There is a distinction hiding in that refactor that decides whether your system stays decoupled or quietly re-couples. OrderPlaced is past tense on purpose. It is a fact: a thing that already happened, published to whoever cares, with the producer giving up all control over what happens next. That is different from a command like SendReceipt, which is an instruction aimed at one specific handler that you expect to carry it out.
The tense is not cosmetic. It tells you which way the coupling runs.
The smell to watch for is a “command dressed as an event.” If you publish SendWelcomeEmail to the bus, you have not decoupled anything. You have written a function call with worse ergonomics, because there is exactly one intended handler and you are furious if it does not run. Name it for what happened (UserRegistered) and let the email service decide that a welcome mail is its reaction. The producer should be able to delete every consumer and still be correct. If it cannot, you are sending commands.
Both have a place. Choreography runs on events; orchestration, which we will get to, runs largely on commands. The mistake is not picking one, it is blurring them so you cannot tell whether publishing something is a broadcast or a demand.
See the decoupling
Enough prose. Place an order below and watch checkout confirm immediately, in roughly the time its own two writes take, while the consumers react afterward on their own schedules. Toggle a consumer off and place another order: checkout is unchanged, that reaction simply does not happen. Turn on loyalty, a consumer that did not exist when checkout was written, and it starts reacting without a single line of checkout changing. That is the whole promise, made concrete.
Notice the shape of the log. The bold confirmation line always lands first and always at the same speed, no matter how many consumers are subscribed or how slow they are. The reactions trickle in afterward, out of order, each on its own clock. That gap between “confirmed” and “all reactions done” is not a glitch. It is the thing you are trading for, and it has a name.
Choreography or orchestration
Once services talk through events, a real order flow is rarely one event. Payment reacts to the order, shipping reacts to the payment, notifications react to the shipment. There are two ways to wire that multi-step process, and choosing between them is the main architectural decision in an event-driven system.
Choreography has no conductor. Each service subscribes to the events it cares about, does its part, and emits its own event, which the next service happens to be listening for. Nobody is in charge. The end-to-end flow is an emergent property of who listens to what. It is maximally decoupled, and you can add a step by adding a subscriber. The catch is that the whole sequence lives nowhere. To answer “what happens after an order is placed?” you read five services and hold the graph in your head.
Orchestration puts one component, often called a saga, in charge of the sequence. It sends ChargePayment, waits for the reply, sends ReserveStock, waits, sends Ship. The flow is explicit and sits in one file you can read top to bottom, which also makes compensation (refund the payment if shipping fails) something you can actually write down. The price is that the orchestrator is coupled to every service it drives, and it is one more thing to run and keep alive.
The honest rule: reach for choreography when the flow is short and the steps are genuinely independent, three or four services that each know how to react and nothing needs a bird’s-eye view. Reach for orchestration when the flow is long, branchy, or needs compensation and someone will inevitably ask “where is this order stuck?” A saga written as an explicit state machine answers that question. A choreography of eight services answers it with a week of log spelunking. Durable workflow engines exist precisely to make orchestration cheap to run, so you no longer have to choose between “explicit flow” and “one more fragile service.”
The costs you sign up for
None of this is free, and pretending otherwise is how teams end up worse off than the welded version. A synchronous call gives you a strong guarantee: when it returns, the thing happened, in order, exactly once, or you got an error. Events give up every word of that sentence. Go in knowing what you traded.
Eventual consistency
The moment checkout returns before the receipt is sent, your system has states that are true at different times. The order row exists. The receipt does not, yet. Inventory has not been decremented, yet. For a window measured in milliseconds or, on a bad day, minutes, the system is partly updated, and a read that lands in that window sees a half-applied world.
This is not a bug to fix, it is a property to design around. Sometimes the gap is fine: nobody minds that the analytics dashboard lags the sale by two seconds. Sometimes it needs product design: the confirmation page says “your receipt is on its way” rather than linking to a receipt that does not exist yet. And sometimes eventual consistency is flatly the wrong choice, because the user needs a definite answer now. “Did my payment go through?” is not a question to answer with “eventually.” When the reaction must be true before you can respond, keep it a synchronous call and stop reading here.
The flow hides in the gaps
In the welded version, a failed order was one stack trace in one service. In the event-driven version, a single order touches a dozen events across five services over several seconds, and when something goes wrong the story is scattered across all of them and across time. No single log has the whole picture.
The only thing that makes this tractable is threading a correlation id (say, the order id, or a dedicated trace id) through every event and every log line, so you can later gather everything tagged with it and reconstruct what happened. This is the whole job of observability in a distributed system, and it is the same trace-context discipline that tracing an AI pipeline relies on. Skip it and you are not debugging, you are guessing. Budget for it on day one, because retrofitting trace ids after the incident is its own miserable project.
At-least-once, so consumers must be idempotent
Here is the one that surprises people. A serious broker will not promise to deliver each event exactly once. It promises at-least-once: it keeps redelivering until the consumer acknowledges receipt, and if that acknowledgement is lost (the consumer processed the event and then crashed before its ack landed), the broker does the safe thing and delivers again. So your consumer will, occasionally, see the same event twice.
You will read that some brokers offer “exactly-once.” Treat that claim with suspicion. What is actually achievable across a network is effectively-once: at-least-once delivery plus a consumer that recognises duplicates and does the work a single time. The real exactly-once modes that exist are narrow, carry a throughput cost, and stop at the boundary of your own side effects anyway. The broker cannot un-send an email for you. So the rule is not “find a better broker,” it is make every consumer idempotent: carry an id on each event, record the ids you have processed, and make a repeat a no-op.
// email-service.js: safe to receive the same event twice
bus.subscribe("OrderPlaced", async (evt) => {
const fresh = await seen.add(evt.id); // INSERT ... ON CONFLICT DO NOTHING
if (!fresh) return; // already handled this exact event
await email.sendReceipt(evt.userId, evt.orderId);
});That seen.add is one indexed insert that returns whether the row was new. This is the same idea as an idempotency key on a payment endpoint, applied to a consumer, and it is not optional the way people treat it. It is the tax for at-least-once delivery, and you pay it in every consumer or you pay it in duplicate receipts and double-charged cards.
Nothing promises order
The second guarantee events give up is ordering. Events can arrive out of the sequence they were sent in, because retries jump the queue and different events may travel different paths. Do not assume OrderUpdated arrives after OrderCreated, or that OrderCancelled cannot land first.
Designs that survive this stop depending on arrival order. Attach a version or sequence number to each event and let a consumer ignore anything older than what it has already applied. Or route all events for one entity through the same partition or key, which buys you per-key ordering while giving up nothing important. What you must not do is write a consumer that only works if the events show up in the tidy order you imagined at your desk.
The dual write, and the outbox
This one is subtle and it bites hard. Look back at the emit-and-return checkout. It does two separate writes to two separate systems: it commits the order to the database, then it publishes to the broker. Two writes, no shared transaction. So there is a gap between them, and if the process dies in that gap, you get an order in the database that no event ever announced. No receipt. No shipment. A ghost order that looks fine in the admin panel and never happened as far as the rest of the system knows. Flip the order of the two writes and you get the opposite ghost: an event for an order that was never saved.
The standard fix is the transactional outbox. Instead of publishing to the broker directly, you write the event into an outbox table in the same database transaction as the order. Both land or neither does, because they are one atomic commit. A separate relay then reads unsent rows from that table and publishes them, retrying until the broker acknowledges, and marks them sent.
// one transaction: the order and the event commit together, or not at all
await db.transaction(async (tx) => {
const order = await tx.orders.insert(cart, user, charge);
await tx.outbox.insert({
id: crypto.randomUUID(),
topic: "OrderPlaced",
payload: { orderId: order.id, userId: user.id, items: order.items },
});
});
// a separate relay drains the outbox and publishes, retrying on failure
async function drainOutbox() {
for (const row of await db.outbox.takePending(100)) {
await bus.publish(row.topic, { id: row.id, ...row.payload });
await db.outbox.markSent(row.id);
}
}Because the relay can publish a row and then crash before marking it sent, it will sometimes publish the same row twice. Which is exactly why consumers had to be idempotent anyway. The outbox turns “maybe lost” into “at-least-once,” and the idempotent consumer turns “at-least-once” into “effectively-once.” The two patterns are a matched pair. This is one specific reason background jobs and queues sit next to your database, and the relay leans on ordinary database transactions and retries with backoff to do its job. The same reliability contract shows up when you deliver events to the outside world as webhooks: at-least-once, signed, and idempotent on the receiver.
When it is worth it, and when it is not
Event-driven design is a genuine improvement for a specific set of problems and a liability for the rest. Say yes when one fact has many independent reactions (an order that fans out to receipt, inventory, analytics, loyalty, fraud), when the reactions belong to different teams that should ship without coordinating, when the parts scale differently (checkout is spiky, analytics ingestion is steady), or when a queue in front of a slow consumer usefully absorbs a traffic burst. Those are real wins, and no amount of synchronous cleverness buys them.
Say no, or at least not yet, in the cases below. This is where I have watched teams talk themselves into pain.
The tell for a healthy event-driven boundary is that you can delete a consumer and the producer neither notices nor cares. If pulling a subscriber breaks the producer, the arrow is pointing the wrong way, and what you have is a command wearing an event’s clothes. Events are for facts you are willing to announce and then forget. Everything else is still a function call, and there is no shame in that.
Summary
- Event-driven design replaces “call the thing that should happen next” with “announce that something happened and let whoever cares react.” It decouples a producer from its consumers so they evolve, scale, and fail independently.
- It is the observer and pub/sub pattern at system scale. In one process the bus is an emitter; across processes it is a message broker, and the network between them is where the new problems live.
- Model events as facts (past tense,
OrderPlaced, broadcast, producer lets go) and keep them distinct from commands (imperative, addressed to one handler, caller stays in control). A command published as an event has decoupled nothing. - Choreography spreads a flow across services that each react and emit, with no conductor: loosest coupling, but the sequence lives nowhere. Orchestration puts a saga in charge: explicit and debuggable, but coupled to every service it drives. Short and independent leans choreography; long, branchy, or needs compensation leans orchestration.
- Eventual consistency is the price, not a bug. There is a real window where the system is partly updated. Design the product for it, or keep it synchronous when the user needs a definite answer now.
- Debugging moves into the gaps between services. Thread a correlation id through every event and log line or you are guessing. This is what observability is for.
- Brokers deliver at-least-once, so consumers see duplicates. “Exactly-once” is really “effectively-once”: at-least-once plus an idempotent consumer that dedupes on the event id. See idempotency keys.
- Ordering is not guaranteed. Use version numbers or per-key partitioning; never assume events arrive in the order you sent them.
- The dual-write problem can save the order but lose the event. The transactional outbox writes both in one commit and lets a relay publish with retries. It pairs with idempotent consumers by design.
- When to use: one fact with many independent reactions, teams or components that must decouple, uneven scaling, bursty load a queue can absorb. When to avoid: you need the answer synchronously, there is a single local consumer, or the “decoupling” is a distributed monolith in disguise.