Background Jobs and Queues
The signup endpoint did three things. Create the user, send a welcome email, return 201. It had run for a year without a single complaint, because for a year the email provider had been fast.
Then the provider had a bad afternoon.
Their API latency crept from 90ms to two seconds, then to six, and then calls started timing out entirely. Because the send happened inside the request handler, every signup inherited that delay. The p99 on POST /signup went vertical. The load balancer, seeing requests hang past its 10-second ceiling, started returning 504s to real people. And here is the part that turned a slow afternoon into a data-cleanup week: the user row was written to the database before the email call, so the people staring at an error page were, in fact, already signed up. They did the reasonable thing and hit the button again. Now there were duplicate accounts, and once the provider recovered, each of those accounts got two welcome emails.
None of that was a bug in the signup code. The code was correct. The mistake was structural. You did slow, third-party work while a paying customer and an open TCP connection sat there waiting for it to finish.
The fix is a single idea, and the rest of this article is the mechanics of doing it well. Do the fast, essential part now. Hand the slow part to something that runs later. Answer the request immediately.
What actually belongs in the background
Not everything should move off the request path. If the caller is sitting there waiting to see the result, a queue just adds latency and moving parts. So run the thing through four questions, and if any answer is yes, it is a candidate for the background:
- Is it slow? Rendering a PDF invoice, transcoding a video, generating a monthly report, resizing an upload into six thumbnails.
- Can it fail on its own schedule? Anything that depends on a system whose uptime you do not control.
- Does it talk to a third party? Email, SMS, push, payment capture, an outbound webhook to a customer’s server.
- Is it bursty? A campaign blast, a signup spike after a launch, a fan-out that turns one action into ten thousand.
The classic residents of the background are email and receipts, PDF and image processing, outbound webhooks, search index updates, nightly rollups, and cache warmups. The common thread: the user does not need the outcome in order to get on with their day.
The flip side matters just as much. Checking a password on login stays inline, because the answer is the response. Charging the card at checkout usually stays inline too, because the customer needs the yes or no right now. But the receipt email that follows the charge? That is background work. Learning to draw this line is most of the skill.
The shape of the thing
Once work moves off the request path, four pieces appear.
- The producer is your request handler. It enqueues a small message and returns.
- The broker holds the job durably, meaning it survives a process restart. This word is the whole game. An in-memory array is not a queue, it is a way to lose jobs on your next deploy.
- The worker is a separate process that pulls jobs and does the slow work at its own pace, with its own scaling and its own crashes.
Here is the anti-pattern and its fix, side by side. First, the version that ruined the afternoon:
// Inline: the response is hostage to the email provider
app.post('/signup', async (req, res) => {
const user = await db.users.create({ email: req.body.email });
await sendWelcomeEmail(user.email); // 4 seconds when the provider is sad
res.status(201).json(user);
});Now the enqueued version. Notice what goes into the job:
// Enqueued: the response leaves immediately
app.post('/signup', async (req, res) => {
const user = await db.users.create({ email: req.body.email });
const job = await emails.add('welcome', { userId: user.id });
res.status(202).location(`/jobs/${job.id}`).json({ status: 'queued', jobId: job.id });
});Put { userId } in the job, not the whole user object. The row already lives in your database. Enqueue the reference and let the worker load a fresh copy. A job that carries a snapshot of the user carries a copy that can already be stale by the time it runs, and it bloats the broker for no reason.
The request now says 202, not 200
When work moves to the background, your API can no longer honestly return 201 Created with the finished thing, because the thing is not finished. It returns 202 Accepted: I have your request, it is valid, I have not done it yet. That is not a lesser answer. It is a more truthful one.
A 202 should hand back a status resource, a URL the client can look at to find out how the work is going. The job moves through a small set of states, and that URL exposes them.
The two endpoints are small:
// POST /signup -> 202 with a pointer to the status resource
res.status(202)
.set('Location', `/jobs/${job.id}`)
.json({ status: 'queued', jobId: job.id });GET /jobs/9f3c-2a HTTP/1.1{ "id": "9f3c-2a", "state": "completed", "result": { "messageId": "e_1a2b" } }How does the client learn the work finished? Three options, in rough order of how much you should reach for them. Poll the status URL on a timer; it is boring and it works, so start here. Open a server-sent events stream and push the state change when it happens. Or, for machine-to-machine flows, fire a webhook at a URL the caller registered. Polling covers the vast majority of cases. Do not build the streaming version until someone asks for it.
At-least-once is the deal, and it is non-negotiable
Here is the uncomfortable truth that every queue eventually teaches you: you get at-least-once delivery. A job can, and eventually will, run more than once.
“Exactly-once” is mostly a phrase on a landing page. The reason is not laziness, it is physics. A worker does the work, then tells the broker “done.” Those are two separate steps. If the worker crashes in the gap between them, the broker never heard “done,” so it assumes the worker died and hands the job to somebody else. The side effect already happened. The acknowledgement did not.
Even the systems that advertise exactly-once put an asterisk on it. Amazon SQS FIFO queues deduplicate, but only within a five-minute window, and only for their own delivery. SQS cannot un-send an email your worker already handed to a mail provider. The broker’s bookkeeping can be exactly-once. Your side effects, out in systems the broker has never heard of, cannot be.
So you do not fight it. You make the work safe to repeat. Your job handler must be idempotent: running it twice must leave the world in the same state as running it once. This is important enough to have its own article on idempotency keys, but the shape inside a worker is simple:
new Worker('emails', async (job) => {
const user = await db.users.find(job.data.userId);
// the key makes a second delivery a no-op at the provider
await sendWelcomeEmail(user.email, { idempotencyKey: `welcome:${user.id}` });
}, { connection, concurrency: 8 });Derive a natural key from the work itself (welcome:${userId}), and either check-then-act, upsert, or pass that key to the downstream provider so it collapses the duplicate. Stripe, most email APIs, and every serious payment processor take an idempotency key for exactly this reason.
The job that runs twice because the worker was slow, not dead
When a worker claims a job, the broker hides it from every other worker for a while. That window is the visibility timeout. If the worker finishes and acks inside the window, the job is deleted. If the window closes with no ack, the broker assumes the worker is dead and makes the job visible again for someone else to grab.
The trap is right there in the design: the broker cannot tell “dead” from “slow.”
SQS defaults the visibility timeout to 30 seconds. If your job usually takes 5 seconds but occasionally takes 45, the broker will redeliver it while the first worker is still cheerfully working. Two workers, same job, nothing crashed.
BullMQ has the same story in different words. A worker holds a lock on the job (default lockDuration of 30 seconds) and renews it in the background roughly every 15 seconds. If the event loop is blocked, say you are doing a heavy synchronous CPU task, the renewal never fires. The lock expires, BullMQ marks the job stalled, and reruns it. After maxStalledCount stalls (default 1) it gives up and fails the job.
That connects two topics that look unrelated. CPU-bound work is dangerous in a Node worker not only because it slows throughput, but because a blocked event loop stops the heartbeat that tells the broker you are alive. This is one more reason that heavy CPU work must leave the event loop: if it does not, the queue thinks the worker died and duplicates the job.
You defend against it on four fronts. Set the visibility timeout comfortably above your p99 job time. For genuinely long jobs, extend the lock or send heartbeats while working. Keep the event loop free. And, above all, be idempotent, so that a double run is a shrug instead of an incident.
Concurrency and ordering
Two knobs control how jobs run: how many at once, and in what order.
Concurrency is the easy one. More workers, or a higher per-worker concurrency setting, means more jobs in flight. Tune it to the resource that actually bottlenecks (the database, the third-party rate limit, CPU) and not past it.
Ordering is where people overreach. It is tempting to say “process jobs in order,” and then to force everything through a single worker to guarantee it. Congratulations, you just deleted the entire benefit of a queue. Global ordering and parallelism are opposites.
The good news is you almost never want global ordering. What you actually want is per-key ordering: process the events for account 42 in the order they happened, while account 42 and account 99 run in parallel. The lever for this is a partition key. SQS FIFO calls it MessageGroupId: messages sharing a group id are strict FIFO, and different groups run concurrently.
Priorities, delays, retries, and the dead-letter queue
Real brokers give you a few more dials, and they all earn their keep.
Priority lets some jobs jump the line. A password-reset email should not sit behind a 50,000-message marketing blast. Most brokers take a priority number and serve higher priorities first.
Delayed and scheduled jobs run in the future. “Send this reminder in 10 minutes” is a delay. “Run the rollup every night at 2am” is a cron schedule. pg-boss and BullMQ both do these natively.
Retries with backoff handle the ordinary failure: a job throws, so you run it again, but not instantly and not forever. Instant retries hammer a dependency that is already struggling. The pattern is exponential backoff (wait 1s, then 2s, then 4s, then 8s), plus jitter, a random smear on each delay so that a thousand jobs failing at the same instant do not retry in perfect lockstep and knock the recovering service straight back over.
The dead-letter queue (DLQ) is where a job goes when it has failed its last retry. Do not drop it. A poison message that fails forever needs a human to look at it, so it lands in a side queue instead of vanishing or looping until the heat death of the universe. SQS moves a message to its DLQ after maxReceiveCount deliveries. BullMQ keeps exhausted jobs in a failed set you can inspect and replay. pg-boss has first-class dead-letter queues. Whatever the mechanism, graph your DLQ depth and alert on it. An empty DLQ is a system that is healthy. A filling one is a system quietly telling you something is broken.
Here is a realistic BullMQ producer, wired for all of this:
import { Queue } from 'bullmq';
const emails = new Queue('emails', { connection: { host: '127.0.0.1', port: 6379 } });
await emails.add('welcome', { userId }, {
attempts: 5, // up to five tries
backoff: { type: 'exponential', delay: 1000 }, // 1s, 2s, 4s, 8s...
priority: 1, // lower number = served sooner
});Watch a queue actually run
Numbers on a page are one thing. Turn the knobs and watch the counters move. Enqueue a burst, change the worker count, crank the failure rate, and flip the visibility timeout between short and long. Notice the trade: a short timeout recovers from a dead worker fast, but it also redelivers slow jobs, so the duplicate counter climbs. A long timeout kills duplicates but drags its feet when a worker really does die. There is no free setting. There is only the one that fits your job times.
An amber worker slot is a job that overran its lock. Its copy is already back in the queue, waiting to be picked up by someone else. That is a duplicate in the making, and it is exactly how a healthy-looking worker produces double sends.
The dual-write problem, and the outbox that fixes it
Look hard at the enqueued signup again. It performs two writes to two different systems: it inserts the user into Postgres, and it adds a job to Redis or SQS. What happens if the first succeeds and the second fails, because the Redis call timed out at the worst moment?
You now have a user with no welcome email and no job that will ever send one. Flip the order and it is just as bad: enqueue first, and you can dispatch a job for a user who never got committed. This is the dual-write problem, and “just be careful about the order” does not solve it, because every order has a crash window between the two writes.
The grown-up fix is the transactional outbox. Instead of writing to the broker at all, you write the job as a row into a table that lives in the same database as your business data, inside the same transaction:
BEGIN;
INSERT INTO orders (id, user_id, total_cents)
VALUES ($1, $2, $3);
INSERT INTO outbox (topic, payload)
VALUES ('order.created', $4);
COMMIT;Now it is one atomic commit. Either the order and the outbox row both land, or neither does. There is no window. A separate relay process then reads unpublished outbox rows, pushes them to the broker, and marks them sent. It claims a batch without fighting other relay instances using the same trick a Postgres-backed queue uses internally:
SELECT id, topic, payload
FROM outbox
WHERE published_at IS NULL
ORDER BY id
FOR UPDATE SKIP LOCKED
LIMIT 100;If the relay crashes after publishing but before marking a row sent, it will publish that row again on restart. That is fine. You already made the worker idempotent, so at-least-once from the relay is a solved problem.
Which broker
Pick by what you already operate, not by a throughput benchmark you will never approach.
My honest opinion, having watched teams do this the hard way: most reach for Redis or SQS well before they have a reason. If you already run Postgres, start your queue there. Add a dedicated broker when you have a measured problem (a latency target, a throughput wall, a feature the database queue lacks), not because a diagram somewhere had a Redis cylinder in it.
One last operational note, whichever you choose. Workers are long-running processes that will be restarted on every deploy, so they must shut down cleanly: stop claiming new jobs on SIGTERM, finish or release the ones in flight, and only then exit. A worker that gets killed mid-job just leaves that job for the visibility timeout to redeliver, which is survivable but sloppy. Doing it properly is the subject of graceful shutdown.
Summary
- Do not do slow, third-party, or bursty work inside the request. Enqueue it, answer immediately, and let a worker do it later. Coupling your latency to someone else’s uptime is the original sin here.
- The shape is producer, durable broker, worker. Enqueue references (an id), not fat snapshots that can go stale.
- Backgrounding changes your API contract: return 202 Accepted with a status resource the client can poll (or subscribe to). See status codes and errors.
- You get at-least-once delivery. Exactly-once delivery is a myth; exactly-once effect is real, and you build it by making every handler idempotent.
- The visibility timeout cannot tell a dead worker from a slow one, so a slow job gets redelivered and runs twice. Set the timeout above your p99, keep the event loop free (server concurrency), and stay idempotent.
- Want per-key ordering, almost never global ordering. Global order and parallelism are opposites.
- Use retries with exponential backoff and jitter, then a dead-letter queue for poison messages. Graph the DLQ depth and alert on it.
- The dual-write problem (row committed, job lost) is fixed by the transactional outbox: write the job as a row in the same transaction, and let a relay publish it. With a Postgres-backed queue, that outbox comes for free.
- Choose a broker by what you already run. Postgres is enough for most people; add Redis or SQS when you can measure why.