Caching Layers: Request, Data and CDN
A cache took our p99 from 780ms to 9ms on a Tuesday. The whole team was thrilled. Two Tuesdays later a customer opened their dashboard and saw a stranger’s unpaid invoice sitting at the top of the page.
Same feature. Both times.
Caching is the cheapest performance you will ever buy, and it is the most reliable source of bugs that make no sense at 3am. The speed is real. So is the staleness, and so is the one where two people quietly share a cached answer that was never meant to leave one of them. The hard part was never turning caching on. The hard part is knowing which layer you are caching in, what a hit actually saves you, and what it costs to throw the copy away once it goes wrong.
The layers, from the browser outward
There isn’t one cache. There’s a stack of them between a user’s screen and your database, and a request can be answered by any layer along the way. The closer to the user a request gets answered, the more work you skip. The closer to the user it lives, the harder it is to reach in and fix when it’s wrong.
one user
everyone near a POP
all app servers
one process
the DB itself
Read that top to bottom and the shape of every caching decision falls out of it.
The browser cache is the best hit you can get, because the request never even leaves the laptop. It’s also the one you cannot reach. Once a browser has stored something for an hour, you have no button that reaches across the internet and deletes it. You wait it out or you change the URL.
The CDN or edge is a shared cache sitting in a hundred cities, answering for your origin before the request travels the last, slow mile to your servers. This is the layer that carries most static traffic and increasingly whole rendered pages. You can purge it, usually within seconds, through an API. That’s covered more in edge and serverless runtimes.
A shared cache, almost always Redis or Valkey, sits next to your app servers and holds data you’d otherwise fetch from the database. Every app server sees the same copy, and you control it directly: set a key, delete a key. This is the layer most “add a cache” tickets are really about, and it’s the one that protects your database, the same instinct behind connection pooling.
In-process memory is a plain object or an LRU map inside a single Node process. Nanoseconds to read, no network at all. The catch is that every process has its own copy, so a value cached in eight processes is eight things you now have to invalidate, and you have no central place to do it.
At the bottom, the database’s own buffer pool caches hot pages and plans so most queries never touch the disk. You don’t manage this one. It’s why a “slow” query is often instant the second time you run it, and it’s related to how indexes keep the working set small enough to stay in memory.
HTTP caching is the free tier
Before you stand up any infrastructure, the browser and every CDN already speak a caching protocol built into HTTP. It costs nothing but a few response headers, and most teams either ignore it or get it subtly wrong. Get it right and a huge share of your traffic gets answered without a single byte reaching your code.
Cache-Control, and the words that lie
Cache-Control is the header that runs the show. A handful of directives do almost all the work, and two of them are named to mislead you.
Cache-Control: public, max-age=300, s-maxage=600max-age=300means fresh for 300 seconds. Note fresh from when: it’s measured from the moment the response was generated at the origin, not from when the browser received it. A response that spent 100 seconds sitting in a CDN arrives with anAge: 100header, and the browser treats it as already 100 seconds into its life.s-maxage=600overridesmax-age, but only for shared caches (CDNs, proxies). Private browser caches ignore it. This lets you keep something at the edge for ten minutes while browsers only hold it for five.publicmeans any cache may store it.privatemeans only the user’s own browser may, and shared caches must not. That single word is the difference between a CDN caching your public marketing page and a CDN caching a logged-in user’s account page for the next visitor to receive.
Now the two liars.
The mental model that keeps it straight: max-age says how long you can skip asking me, no-cache says always ask me but I’ll often say “still good”, and no-store says don’t even keep a copy to ask about.
ETag and the 304 handshake
“Revalidate before reuse” sounds expensive. It isn’t, because of a small, elegant handshake.
When the origin sends a response it can attach an ETag, an opaque string that identifies this exact version of the resource. Usually it’s a hash of the content or a version number. The client stores the body and the tag together.
GET /avatars/42.png
HTTP/1.1 200 OK
ETag: "9c1e-5f3a"
Cache-Control: max-age=60
Content-Length: 45213
...45 KB of PNG...Sixty seconds later the browser wants it again. Instead of asking blind, it sends the tag back in If-None-Match and says “only send the body if it changed”:
GET /avatars/42.png
If-None-Match: "9c1e-5f3a"
HTTP/1.1 304 Not Modified
ETag: "9c1e-5f3a"A 304 Not Modified carries no body. The server compared the tag, found a match, and answered in a couple hundred bytes instead of forty-five thousand. The browser reuses what it already has.
Two details worth carrying:
- A 304 still costs a round trip. You saved the body, not the latency of asking. For assets that truly never change, you want to skip the ask entirely (coming up under content hashing).
- ETags come in strong and weak flavors. A weak tag is written
W/"abc"and promises only that the content is semantically the same, not byte-identical.If-None-Matchcompares weakly, which is what you want for caching. Strong tags matter when byte-for-byte identity is required, like resuming a partial download.
Last-Modified, the cheaper validator
There’s an older validator that works the same way with dates instead of tags. The origin sends Last-Modified: Wed, 15 Jul 2026 10:00:00 GMT, the client echoes it back in If-Modified-Since, and an unchanged resource earns a 304.
It’s weaker than an ETag for one dull reason: HTTP dates have one-second resolution. If a file changes twice in the same second, the timestamp can’t tell the difference and a client can be handed stale content. Use Last-Modified when a real modification time is cheap and good enough. Reach for ETag when you need to detect any change, including two within a second. If you send both, the ETag wins.
stale-while-revalidate, the quiet hero
Here’s the directive that punches far above its reputation.
Cache-Control: max-age=600, stale-while-revalidate=30For the first 600 seconds the response is fresh and served instantly. Once it expires, plain caching would make the next visitor wait for a revalidation. stale-while-revalidate=30 changes that: for 30 seconds past expiry, the cache hands the stale copy back immediately and kicks off a refresh in the background. That visitor waits for nothing. The next one gets the updated value.
This is how you make a cache that is almost never a wall. Users see fast responses even at the moment of expiry, and your origin still gets refreshed. It pairs beautifully with a short max-age and a generous stale-while-revalidate: fresh often, but never a hard stall.
Content-hashed assets: cache forever, never invalidate
The cleanest cache invalidation is the one you never perform. Bundlers name built files after a hash of their contents:
/assets/app.3f9a2c1b.js
/assets/app.7d2e88f0.cssChange one character of source and the hash changes, so the filename changes, so it’s a different URL. That earns you the strongest caching directive there is:
Cache-Control: public, max-age=31536000, immutableA year, and immutable tells the browser not to even bother revalidating on a reload, because this URL’s bytes will never change. When you deploy, the HTML points at the new filenames; the old ones sit harmlessly in caches until they age out. You never purge anything. You just stop referencing it.
The one rule that makes this safe: the HTML (or whatever references the hashed files) must itself be short-lived, usually no-cache or a small max-age. Cache the fingerprinted leaves forever; keep the document that names them fresh. This is one of the biggest, easiest wins for Core Web Vitals, because repeat views load their JavaScript and CSS straight from disk.
Application caching
HTTP caching handles responses. Inside your server you also cache data: the result of a query, a computed aggregate, a rendered fragment. This is where Redis and Valkey and plain in-memory maps come in, and where a few named patterns save you from reinventing them badly.
Cache-aside is the default
The pattern you’ll use ninety percent of the time is cache-aside, also called lazy loading. Your code treats the cache as a fast thing it checks first and fills on demand.
async function getUser(id) {
const key = `user:${id}`;
const cached = await cache.get(key);
if (cached) return JSON.parse(cached); // hit: done
const user = await db.query( // miss: go to the source
"SELECT * FROM users WHERE id = $1",
[id]
);
await cache.set(key, JSON.stringify(user), { ttl: 60 });
return user;
}On a hit, the database is never touched. On a miss, you read the source of truth, stash it with a TTL, and return it. The next reader within 60 seconds gets it for free.
When the underlying data changes, the write path deletes the key rather than trying to update it in place:
async function updateEmail(id, email) {
await db.query(
"UPDATE users SET email = $1 WHERE id = $2",
[email, id]
);
await cache.del(`user:${id}`); // drop it; the next read repopulates
}Deleting is deliberate. If you tried to write the new value into the cache instead, two concurrent writers could interleave and leave the cache holding the older of the two. Delete-and-repopulate sidesteps that whole class of race: whoever reads next reloads fresh from the database.
Write-through and write-behind
Cache-aside fills lazily, on read. Two other shapes fill on write.
Write-through sends every write through the cache, which updates itself and the database together before returning. The cache is always warm and never stale relative to your writes. The price is slower writes and a cache full of things nobody has read back yet.
Write-behind (write-back) writes to the cache and returns immediately, flushing to the database asynchronously a moment later. Writes feel instant. The danger is real: if the process dies before the flush, those writes are gone. It’s a specialized tool for write-heavy systems that can tolerate a small loss window, and it’s rarely the right default for application data.
TTL: the humble answer to invalidation
Notice the { ttl: 60 } in the cache-aside code. That number is doing more than it looks.
A TTL (time to live) converts a hard correctness problem into a tunable staleness budget. You are no longer promising the cache is always right. You are promising it’s never wrong by more than 60 seconds, and that’s a promise you can actually keep without tracking down every code path that might change the data. A TTL also caps load: with a 60-second TTL, a single hot key hits your database at most once per minute no matter how many millions of reads land on it. It self-heals too, because a wrong entry, however it got wrong, evaporates on its own.
Pick the TTL from the business, not the code. How stale can this particular thing be before someone is annoyed or harmed? A public product listing, minutes. A user’s own settings right after they changed them, seconds or an explicit delete. Reference data that changes weekly, hours. When in doubt, a short TTL plus stale-while-revalidate behavior at the app layer gives you freshness and cheap misses at once.
The two genuinely hard problems
There’s an old line, usually credited to Phil Karlton, that there are only two hard things in computer science: cache invalidation and naming things. Caching is where you get to enjoy both at once, literally. Invalidation is getting rid of entries that went stale. Naming is choosing the cache key. Almost every serious cache bug is one of these two wearing a costume.
Naming: the cache key that ruins your week
A cache key must include everything that changes the answer. Miss one input and you serve one user’s answer to another. That’s the invoice from the opening, and it’s worth seeing exactly how it happens.
// Catastrophe: the key doesn't say whose dashboard this is
const html = await cache.get("dashboard");
if (html) return html;Two users, one key. Whoever misses first fills "dashboard" with their rendered page, and everyone after gets served that person’s private data until the TTL expires. The fix is to make identity part of the name:
const html = await cache.get(`dashboard:${userId}`);The same trap has a bigger version at the CDN. If a response depends on who’s asking (it sets a session cookie, it varies by auth), and you let a shared cache store it, the CDN can hand one logged-in user’s page to the next stranger. Two defenses, use both: mark per-user responses Cache-Control: private (or no-store) so shared caches refuse them, and use the Vary header to tell caches which request headers change the response.
Invalidation
You have four ways to get rid of a stale entry, in rough order of how much trouble they cause.
- Expiry (TTL). Let it die on a timer. The most forgiving option because it needs no coordination and self-heals. The cost is a bounded window of staleness you chose on purpose.
- Explicit delete on write. Delete the key the instant the source changes, as in
updateEmailabove. Precise and fresh, and the place bugs love to live: forget one code path that mutates the data and that path silently serves stale forever. Every writer must remember every key. - Versioned keys. Fold a version into the key, like
user:42:v7, and “invalidate” by bumping the version so old keys are simply never read again (they age out on their own). Great for invalidating a whole family of entries at once without hunting them down. - Event-driven purge. Emit an event on change and have a consumer purge the relevant keys or CDN paths. Powerful and the most machinery, so the most to get wrong.
In practice a healthy system leans on TTL as the safety net and adds explicit deletes only where staleness genuinely can’t be tolerated. TTL forgives your mistakes. Explicit invalidation punishes the one you forgot.
The stampede
Now the failure mode that takes down real systems, the one worth building an intuition for because a diagram alone won’t do it justice.
Picture one very hot key. Your homepage feed, cached with a 60-second TTL, serving 5,000 requests a second happily from cache. Then it expires. In the same instant, all 5,000 in-flight requests miss, and every one of them independently decides to rebuild the value from the database. Your database, which was handling roughly zero reads for this, is suddenly hit with 5,000 identical expensive queries at once. It slows. Slow queries hold connections longer, so the next second’s requests pile up behind them, miss too, and pile on more. This is a cache stampede, also called a thundering herd or a dogpile, and it can turn a one-key expiry into a full outage.
Watch it happen and then fix it. Set the TTL low, keep single-flight off, and the DB-load bar spikes red every time the key expires: many misses, many reads. Flip single-flight on and the spikes flatten to a single read per expiry, because only the first miss fetches and the rest ride along.
Fixing it
No single trick is the answer. You reach for a few, and they stack.
Single-flight (the big one). On a miss, let exactly one caller rebuild the value and make everyone else wait for that one result. In a single Node process it’s a map of in-flight promises:
const inFlight = new Map();
function singleFlight(key, load) {
if (inFlight.has(key)) return inFlight.get(key); // join the existing rebuild
const p = load().finally(() => inFlight.delete(key));
inFlight.set(key, p);
return p;
}Then the miss path becomes await singleFlight(key, () => db.query(...)), and a thousand concurrent misses collapse into one query while the other 999 await the same promise. Across many processes, one in-process map isn’t enough, so you take a short lock in the shared cache (SET key NX PX 5000 in Redis or Valkey) and let the loser either wait and re-read or serve stale. The same coordination shows up in rate limiting, where one shared counter arbitrates many workers.
Jittered TTL. Don’t let a batch of keys expire on the same tick. If you warm 10,000 cache entries at deploy with the exact same 300-second TTL, they all die together five minutes later and you get a synchronized stampede across every key at once. Add a little randomness:
const ttl = 300 + Math.floor(Math.random() * 60); // 300–360s, spread outThat small smear spreads expiries across a minute so misses never synchronize. It’s the same medicine that jitter gives retry storms.
Serve stale while you refresh. Keep the old value past its nominal TTL and refresh in the background (guarded by single-flight), so readers get an instant, slightly-stale answer and never see a raw miss. It’s stale-while-revalidate moved into your application layer, and it’s the most user-friendly of the fixes because the wait disappears entirely.
Probabilistic early expiration. The fancy option, from a well-known 2015 paper: instead of everyone treating the TTL as a hard cliff, each reader rolls dice that get likelier to come up “refresh now” as expiry approaches, so one lucky reader rebuilds the value a beat before the crowd needs it. Rarely necessary, good to know it exists.
What not to cache
Caching is a hammer, and some things are not nails.
- Data that must be correct to the second. Inventory count at the moment of checkout, a permission decision, a displayed account balance that’s treated as authoritative. Either don’t cache these or cache with a tiny TTL and be honest about the window. A stale “you may proceed” is a security bug, not a performance win.
- Highly personal, rarely reused data. If a value is unique per user and each user reads it roughly once, your hit ratio is near zero and you’re spending memory to cache things nobody asks for twice.
- Fast-changing values where the TTL would be too short to help. If correctness demands a two-second TTL on something read twice a second, the cache barely earns its keep and adds a moving part.
- Errors, for anything but a blink. Caching a 500 for five minutes turns a ten-second blip into a five-minute outage. Cache successful responses; cache failures for a second or two at most, if at all.
- Giant blobs that evict your hot keys. A cache is finite. Stuffing a 40 MB export into it can push out the thousand small keys that were carrying your real traffic. This is cache pollution, and it makes your hit ratio quietly collapse.
The number that tells you whether any of this is working is the hit ratio. Watch it. A hot path sitting well under 80% usually means the key is too specific, the TTL is too short, or you’re caching things that never get read again. Watch origin and database load alongside it, and treat a rise in “this looks stale” reports as the signal to check your invalidation, not to shorten every TTL in a panic.
Summary
- There isn’t one cache, there’s a stack: browser, CDN/edge, shared cache (Redis or Valkey), in-process memory, and the database’s own buffers. Higher up saves more per hit but is harder to purge; lower down the win shrinks and control returns.
- HTTP caching is free and mostly unused.
max-ageands-maxageset freshness,publicvsprivatedecides who may store, and the trap is thatno-cachemeans “revalidate every time,” whileno-storemeans “keep nothing.” - ETag plus
If-None-Matchgives you a bodiless 304 when nothing changed: you still pay a round trip but not the payload.Last-Modifiedis the weaker, date-based cousin. stale-while-revalidateserves a stale copy instantly and refreshes behind the scenes, so expiry stops being a wall. Confirm your CDN does it asynchronously.- Content-hashed filenames with
immutablelet you cache assets for a year and never invalidate, because a change is a new URL. Keep the referencing HTML short-lived. - Cache-aside is the default data pattern: check, fill on miss, delete-on-write. TTL turns invalidation into a chosen staleness budget and caps load on hot keys.
- The two hard problems are real: naming (a key missing the user id serves one person’s data to another) and invalidation (TTL forgives, explicit deletes punish the path you forgot).
- A cache stampede is one hot key expiring into thousands of simultaneous misses. Collapse it with single-flight, spread expiries with jittered TTL, and hide the wait with serve-stale-while-refresh. Never let the cache be a load-bearing dependency your database can’t survive without.