Scheduled Work: Cron in a Distributed World
The billing job worked fine for a year. It was a setInterval in the same Node process that served the API, it woke up at 03:00, it charged every customer with an outstanding balance, and nobody thought about it again.
Then Black Friday came and the team scaled the API from one instance to three. At 03:00 the next morning, three copies of that process woke up. Each one ran the loop. Each customer got charged three times. The refund and apology cost more than the whole feature had ever earned.
Nothing was “wrong” with the code. The problem is that a timer inside a web process is not a scheduler, and the reasons it isn’t are exactly what we are going to pull apart. Every fix below exists because someone already got burned by the thing it fixes.
setInterval is a timer, not a scheduler
A setInterval (or a bare in-process cron library) has four problems the moment you take it seriously.
It dies with the process. Deploy, crash, or scale-to-zero at 02:59 and the 03:00 run silently never happens. No error, no alert, just a gap.
It has no memory. Restart the process at 03:30 and it has no idea whether the 03:00 run already fired. It cannot catch up because it does not know what it missed.
It drifts. setInterval(fn, 3600_000) does not mean “every hour on the hour.” It means “roughly 3,600,000 milliseconds after the last time, whenever the event loop gets around to it.” Under load, with a busy event loop, those milliseconds stretch. A day later your “hourly” job fires at :07 past.
And it does not coordinate. Run more than one copy of the process and every copy fires. That is the three-charges bug.
The rest of this is how real systems answer each of those. Start with coordination, because it is the one that costs you money.
One job, many instances
You run three instances for availability and throughput. You want the nightly job to run once, not three times. So the instances have to agree on who runs it. That agreement is called leader election, and the usual tool for it is a distributed lock: a single shared record that only one instance can hold at a time.
Where does the lock live? Somewhere all instances can reach and that gives you atomic “grab it if free.” Common choices:
- A row in Postgres, using an advisory lock so you do not even need a table.
- A key in Redis set with
SET key value NX PX 60000(create only if absent, auto-expire in 60s). - Your orchestrator’s primitive: a Kubernetes lease, a cloud provider’s coordination service.
The Postgres advisory-lock version is the one I reach for first, because there is nothing to install and the lock is tied to your database session:
-- Non-blocking: returns true only for the first caller this second.
-- $1 is a stable 64-bit hash of the job name, e.g. hashtext('nightly-billing').
SELECT pg_try_advisory_lock($1);The instance that gets true runs the job. Everyone else gets false and goes back to sleep. When the session ends (or you call pg_advisory_unlock), the lock is released. Clean, and it survives nothing you have to babysit.
The lock that lies to you
Here is where people who have read one blog post get overconfident. A lock with an expiry (a TTL) is not a guarantee that only one process is running. It is a guarantee that only one process holds the key. Those are different, and the gap between them is a genuine 3am incident.
Walk the failure. Instance A acquires the lock with a 60-second TTL and starts charging customers. Forty seconds in, A hits a stop-the-world garbage-collection pause, or the VM it runs on gets live-migrated, or the kernel simply does not schedule the process for a while. From A’s point of view no time passes. From the wall clock’s point of view, 30 seconds pass, the 60-second TTL expires, and the lock service hands the now-free lock to instance B. B starts charging. Then A wakes up, still believing it holds the lock, and finishes charging. Two runs. Same corruption you started with, now harder to reproduce.
The honest fix is a fencing token. When the lock service grants the lock, it also hands out a number that only ever goes up. Grant one, token 33. Grant the next, token 34. The rule: every write to the protected resource carries the token, and the resource remembers the highest token it has accepted and rejects anything lower.
Now replay the stall. A got token 33. While A slept, B got token 34 and wrote successfully, so the database’s “highest seen” is now 34. When A finally tries to write with token 33, the database sees 33 is below 34 and rejects it. A is fenced off. Correctness holds even though, for a few seconds, two processes genuinely believed they owned the lock.
const { token } = await lock.acquire("nightly-billing", { ttlMs: 60_000 });
for (const account of accountsToBill) {
// The token rides along with every write. The DB (or a guarded
// stored procedure) rejects it if a higher token has been seen.
await billing.charge(account.id, account.amount, { fenceToken: token });
}If fencing tokens sound like a lot of ceremony for a nightly job, that is the correct reaction, and it is a big reason people push the actual work into a queue and make it idempotent instead. A double-fire that is harmless needs no fence.
Overlap: the run that laps itself
Different bug, same family. Your job is scheduled every hour, 0 * * * *. Usually it takes eight minutes. But the data grew, and one night it takes 70 minutes. At the top of the next hour the scheduler fires again, on schedule, while the previous run is still going. Now two copies of the same job are walking the same rows, and they interleave in ways your code never planned for.
Prevent overlap explicitly. Do not assume the last run finished. The same lock that elects a leader also serializes runs: the job takes the lock when it starts and holds it until it finishes, and a fresh tick that finds the lock held simply exits.
You have to decide what “found the lock held” should do, and it is a real choice:
- Skip the new run. Right for idempotent syncs where the next tick will catch up anyway.
- Queue it to run after the current one. Right when every scheduled instance must eventually run.
- Kill the old one and start fresh. Rarely what you want, occasionally exactly right (a “latest wins” recompute).
Good schedulers name this for you. Kubernetes CronJob calls it concurrencyPolicy with values Allow, Forbid, and Replace. pg_cron sidesteps it: it will not start a second copy of the same job while one is running, it queues it. If your scheduler has no such knob, the lock is your knob.
Missed runs: the tick that never happened
The service was down for a deploy from 02:58 to 03:12. The 03:00 job never fired. Now what? This is the question a naive timer cannot even ask, because it has no record that 03:00 was supposed to happen.
There are exactly two defensible answers, and the mistake is picking one by accident instead of on purpose.
Catch up when the run is about a fact that is still true. The billing job must collect money that is genuinely owed, so missing 03:00 is not permission to skip a day. Run it late.
Skip when the run is only useful at its time. A “good morning” digest that fires at 08:00 is worthless at 11:00, the moment has passed, send tomorrow’s instead. A cache warmer that missed its window has nothing to warm that the next tick will not cover.
Real schedulers give you a lever for this, usually shaped as “how late is too late.” Kubernetes uses startingDeadlineSeconds: miss the tick by more than that many seconds and the run is abandoned rather than fired late. Quartz calls the same idea a misfire policy. Whatever the name, set it deliberately, because the default is rarely the answer you would have chosen.
# Kubernetes CronJob: two of the problems above, solved by config
apiVersion: batch/v1
kind: CronJob
metadata:
name: nightly-billing
spec:
schedule: "0 3 * * *" # the cluster runs this in UTC
concurrencyPolicy: Forbid # overlap: never start a run atop a running one
startingDeadlineSeconds: 600 # missed runs: >10 min late, skip it
jobTemplate:
spec:
template:
spec:
restartPolicy: Never
containers:
- name: billing
image: myapp/billing:2026.7Timezones and the two mornings a year
Almost every scheduling bug that only reproduces twice a year is daylight saving time. The wall clock is not a monotonic number line. Twice a year it either skips an hour or repeats one, and any job scheduled inside that hour, in a zone that observes DST, does something surprising.
The rule that saves you: schedule in UTC. UTC has no daylight saving, so an interval-based or hour-based job never lands in a phantom or doubled hour. This is not a nice-to-have. It is why the cloud schedulers below all run in UTC by default: Cloudflare Cron Triggers execute in UTC, pg_cron interprets expressions as UTC unless you change a setting, and most managed schedulers do the same.
The complication is that humans want local times. “Send the invoice at 9am the customer’s time” is a real requirement. The move is to store the user’s IANA time zone (like Europe/Berlin, not a fixed offset like +01:00, because the offset itself changes with DST) and convert at the moment you compute the next run.
// Compute "next 09:00 in the user's zone" and store the resulting UTC instant.
// Intl carries the current IANA rules, so it handles the DST shift for you.
function next9amUtc(ianaZone) {
const fmt = new Intl.DateTimeFormat("en-US", {
timeZone: ianaZone,
hour12: false,
hour: "2-digit",
});
// ...find the next instant whose local hour in ianaZone is 09...
// The point: the schedule is anchored to the zone, the stored value is UTC.
}Cron expressions, read honestly
The syntax people paste from a search result is standard Unix cron: five fields, separated by spaces.
┌───────── minute (0-59)
│ ┌─────── hour (0-23)
│ │ ┌───── day of month (1-31)
│ │ │ ┌─── month (1-12 or JAN-DEC)
│ │ │ │ ┌─ day of week (0-6, Sunday = 0)
│ │ │ │ │
0 3 * * * → 03:00 every dayFour operators cover almost everything. * means every value. , is a list (1,15 = the 1st and 15th). - is a range (9-17 = 9 through 17). / is a step (*/15 in the minute field = 0, 15, 30, 45). So */15 9-17 * * 1-5 reads “every 15 minutes, 9am to 5pm, Monday to Friday.”
Two things regularly bite people, and neither is your fault.
The first is the day-of-week / day-of-month trap. When both of those fields are restricted (neither is *), cron runs the job when either matches, not both. So 0 0 13 * 5 is not “midnight on Friday the 13th.” It is “midnight on the 13th, and also every Friday,” which fires about nine times a month. This is written into the POSIX spec; Paul Vixie called it a bug and then declined to fix it because too many crontabs already depended on the behavior. If you need a true intersection you filter inside the job, or use a dialect with the # operator.
The second is that not all cron is the same cron. Quartz (the JVM world) and several managed platforms add a leading seconds field, so their expressions have six fields, and they number the days of the week differently: Unix cron uses Sunday = 0, Quartz and Cloudflare use Sunday = 1. Copy a five-field expression into a six-field engine and it means something else entirely. Always check which dialect you are feeding.
Type an expression below and see it decoded field by field, with the next few times it would fire. It is a small parser, but it handles the operators above and the day-of-week union rule, so you can catch a mistake before it ships:
Try 0 0 13 * 5 in there. The next-run list makes the union trap obvious the moment you see it firing on plain Tuesdays.
Where the schedule should actually live
You have the failure modes. Now the boring question that decides how much you will suffer: where do you run the thing? Four common homes, roughly in order of how much they hand you for free.
A cron inside your app process. The setInterval or node-cron we opened with. Fine for a single-instance side project. In anything scaled or important, it inherits every problem above and gives you no tools to fix them. Do not schedule money from here.
Your orchestrator. Kubernetes CronJob, a systemd timer, a cloud provider’s scheduler. This is a real step up: it runs in UTC, it survives your app restarting, and it hands you concurrencyPolicy and startingDeadlineSeconds so overlap and missed runs are config, not code. The gap is visibility. When the job fails, you are reading pod logs at 3am.
A platform cron trigger. Serverless platforms ship a scheduler: Cloudflare Cron Triggers, Vercel Cron, cloud provider event schedulers. You add a handler and a cron expression and forget about the machine. Read the fine print, because it moves: on Cloudflare the scheduled() handler runs in UTC and hands you the logical tick time; Vercel’s free tier limits you to one run per day and may fire it anywhere inside the scheduled hour, which is fine for a nightly job and useless for anything precise. Check the current limits before you design around them.
A job queue with delayed or repeating jobs. This is the one I reach for in real systems, and it is worth saying why plainly. A background queue already solves the hard parts you would otherwise rebuild: it persists the job so a restart cannot lose it, it retries with backoff when the work fails, it dedupes, and it gives you a dashboard where you can see that the run happened. Scheduling becomes “enqueue a job to run at time T,” which is a thing queues already do well.
In practice the two combine. Let the platform cron do one tiny thing, enqueue a job, and let the queue do everything that can fail:
// Cloudflare Worker: the cron's only job is to enqueue, not to do the work.
export default {
async scheduled(controller, env, ctx) {
// scheduledTime is the logical tick (ms since epoch), not "now".
// Using it as the id makes a double-fire enqueue the same job, not two.
const runId = new Date(controller.scheduledTime).toISOString();
await env.BILLING_QUEUE.send({ job: "nightly-billing", runId });
},
};In a plain Node deployment the same shape uses a library. BullMQ replaced its old repeatable-jobs API with job schedulers (upsertJobScheduler, from v5.16 on), and upsert is the whole point: call it a hundred times on boot across a hundred instances and you get one schedule, not a hundred.
import { Queue } from "bullmq";
const queue = new Queue("billing", { connection });
// Idempotent by id: every instance can run this on startup, safely.
await queue.upsertJobScheduler(
"nightly-billing",
{ pattern: "0 3 * * *", tz: "Etc/UTC" }, // schedule in UTC, always
{ name: "charge-all" },
);If Postgres is your whole world and you do not want another service, pg_cron schedules SQL directly, runs in UTC by default, and refuses to start a second copy of a job while one is running:
-- Runs in UTC unless cron.timezone is set. One instance of this job at a time.
SELECT cron.schedule('nightly-billing', '0 3 * * *', $$ CALL run_billing() $$);Make it idempotent, and make it loud
Two properties turn a scheduled job from a liability into something you can trust while you sleep.
Idempotent means running it twice does the same thing as running it once. Given everything above (leader elections that briefly overlap, retries, catch-up runs), some job is going to run twice eventually. Idempotency is what makes that a non-event instead of a double charge. The clean trick is to derive an idempotency key from the logical run time, not from Date.now(). Two fires of the 03:00 run share the key nightly-billing:2026-07-18T03:00Z, so the second one is recognized as a duplicate and skipped. If you had keyed on the actual wall-clock instant, the two fires would look like different jobs, and that is exactly the bug.
async function runOnce(jobName, logicalTime, work) {
const key = jobName + ":" + logicalTime.toISOString();
// INSERT ... ON CONFLICT DO NOTHING: the first run claims the key.
const claimed = await db.claimRun(key); // parameterized insert
if (!claimed) return; // someone already ran this exact tick
await work();
}Loud means the job tells you it ran, and screams when it does not. This is the one people skip, and it is the one that hurts most, because scheduled jobs fail silently. A web request that breaks throws an error someone sees. A nightly job that quietly stops firing produces nothing: no error, no page, just a slowly growing pile of uncollected invoices that someone notices three weeks later.
The fix is a dead-man’s switch. The job pings a monitor every time it finishes. The monitor knows the schedule, and if the ping does not arrive within a grace window, it alerts. You are no longer relying on a failure to announce itself, you are alerting on the absence of success, which is the only signal a silently-dead job actually emits.
Summary
- A
setIntervalor in-process cron is a timer, not a scheduler: it dies with the process, has no memory of what it missed, drifts under load, and fires once per instance. Scale to N instances and it runs N times. - Run a single copy with leader election via a distributed lock (a Postgres advisory lock, a Redis
NXkey, an orchestrator lease). The winner runs, everyone else skips. - A lock with a TTL can be held by two processes at once when the holder stalls past the TTL. For correctness-critical work, use fencing tokens: a monotonic number the resource checks so it rejects a stale writer. Redlock’s random value is not a fencing token.
- Prevent overlap explicitly. A long run should block the next tick (skip, queue, or replace), never assume the previous run finished.
concurrencyPolicyand its equivalents are how you say which. - Decide missed runs on purpose: catch up when the work is still owed (billing), skip when it is only useful at its time (a morning digest).
startingDeadlineSecondsand misfire policies are the levers. - Schedule in UTC so daylight saving never gives you a phantom or doubled hour. Store a user’s IANA zone and convert only for display or for genuinely local schedules.
- Know your cron dialect: five fields for Unix, six for Quartz, and the day-of-month / day-of-week fields combine with or, not and, when both are set.
- Prefer a queue with delayed jobs over bare cron: you get retries, dedup, and visibility for free. Let a thin cron trigger enqueue, and let the queue do the work.
- Make every scheduled job idempotent (key on the logical run time) and observable (heartbeat monitor that alerts on the run that did not happen), because they fail silently at 3am.