File Uploads and Object Storage
The invoices table was 4 gigabytes. About 30 megabytes of that was invoices.
The rest was PDFs. Someone had added a pdf bytea column two years earlier because it was one migration and zero new infrastructure, and it worked, right up until it didn’t. The nightly pg_dump crept from 90 seconds to 22 minutes. The read replica fell behind during business hours. And the afternoon a support rep bulk-downloaded 400 invoices, every unrelated query on that database got slow, because Postgres had evicted hot index pages out of its cache to make room for a few hundred megabytes of PDF that nobody needed cached.
None of that is a Postgres bug. It is the database doing exactly what you asked: keep this durable, replicate it, cache what gets read, put it in every backup. You just pointed all of that machinery at the wrong kind of data.
The bytes have to live somewhere, and it is not your database
A relational database is tuned around a working set: the slice of your data that is actually hot, which the engine keeps in RAM so queries touch memory instead of disk. Files wreck that. A 5 MB image is 5 MB of buffer cache it will happily blow through, evicting the index pages that make everything else fast. Every file you store also rides along in every backup, every replication stream, every restore. Your recovery time objective quietly becomes “however long it takes to move all the cat pictures”.
Object storage exists precisely so the database never holds a byte of file content. Amazon S3, Cloudflare R2, Google Cloud Storage: they are all the same shape. A flat namespace of keys mapping to blobs, addressed over HTTP, durable, cheap per gigabyte, and frontable by a CDN. They do one job and they are very good at it.
So the split is clean. The blob goes to object storage under a key you generate. The database stores a small row: the key, the owner, the size, the content type, a status. Metadata is tiny, queryable, and safe to cache. The bytes are somebody else’s problem.
The naive path, and exactly where it falls over
The browser side of an upload is a multipart/form-data request: the form fields and the raw file bytes packed into one body, split by a boundary string. If you have not seen how the browser builds that, FormData and the File object cover it. On the server, some body parser has to walk that format and hand you the file.
The tutorial version buffers the whole thing into memory. With Multer’s memory storage, that is one line and it looks harmless:
import multer from "multer";
// every uploaded byte lands in req.file.buffer, in RAM
const upload = multer({ storage: multer.memoryStorage() });
app.post("/avatar", upload.single("file"), async (req, res) => {
await saveSomewhere(req.file.buffer); // the whole file is already in memory
res.sendStatus(204);
});Here is the problem in one sentence: a 2 GB upload is a 2 GB allocation. Multer read the entire request body into a Buffer before your handler ever ran. Ten people uploading 2 GB at once is 20 GB of resident memory, and your process is dead long before that. On a container with a 512 MB limit it dies at the first big file, and it does not die politely. It gets OOMKilled, which takes down every other request that pod was serving, not just the one holding the giant buffer. That is the same failure mode as buffering a big response, covered in streaming a response.
Buffering into memory is fine for exactly one case: small files with a hard cap you actually enforce. A 256 KB avatar behind a limits: { fileSize: 512 * 1024 } is genuinely fine. Anything open-ended is a memory bomb with a fuse the length of your largest user’s patience.
Stream it instead of buffering it
The fix is the same as everywhere else in this part: do not hold the whole thing. A request body is a stream, so parse it as one and pipe each chunk straight to storage as it arrives. Busboy is an event-based multipart parser that hands you the file as a readable stream instead of a finished buffer, and the AWS SDK’s Upload helper consumes that stream, uploading in parts under the hood.
import Busboy from "busboy";
import { randomUUID } from "node:crypto";
import { S3Client } from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";
const s3 = new S3Client({ region: "auto" }); // "auto" for R2; a real region for S3
app.post("/upload", (req, res) => {
const bb = Busboy({ headers: req.headers, limits: { fileSize: 25 * 1024 * 1024 } });
bb.on("file", (field, stream, info) => {
const key = `uploads/${randomUUID()}`;
const upload = new Upload({
client: s3,
params: { Bucket: "my-bucket", Key: key, Body: stream, ContentType: info.mimeType },
});
upload.done()
.then(() => res.json({ key }))
.catch(() => { stream.resume(); res.status(500).end(); });
});
bb.on("error", () => res.status(400).end());
req.pipe(bb);
});Memory stays flat no matter how big the file is, because only a part or two is ever in flight. That is a real improvement over buffering. But look at the shape of it: the bytes still travel through your server on their way to the bucket. You are paying to receive them and paying again to send them, your process is busy shuttling data instead of answering requests, and on a serverless platform you may not even be allowed to do this.
Presigned URLs: get your server out of the data path
Here is the pattern you should reach for by default. Your server never touches the file. It authorizes the upload, hands the browser a short-lived, tightly-scoped URL, and the browser sends the bytes straight to the bucket. Storage validates the request against the constraints you signed into it and stores the object. Your API is consulted for permission and then steps aside.
This is not a micro-optimization. It is cheaper (you are not paying for ingress and egress on the file), it is faster (one hop instead of two, and the bucket’s edge is usually closer to the user than your origin), and it sidesteps a wall that stops the streaming approach cold on serverless. A Vercel function, as of 2026, rejects any request body over 4.5 MB with a 413 FUNCTION_PAYLOAD_TOO_LARGE before your code runs. AWS Lambda has its own 6 MB payload ceiling. You cannot stream a 40 MB video through a function that refuses to receive it. Presigning routes around the limit because the big body never goes to the function at all.
There are two flavors of presigned upload, and the difference between them is a real security decision, not a style choice.
A presigned PUT is the simple one. You sign a single URL for a PutObject, the browser does one PUT with the file as the body, done.
import { PutObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
app.post("/uploads/presign", requireAuth, async (req, res) => {
const key = `uploads/${req.user.id}/${randomUUID()}.jpg`;
const command = new PutObjectCommand({
Bucket: "my-bucket",
Key: key,
ContentType: "image/jpeg", // the client must send this exact header
});
const url = await getSignedUrl(s3, command, { expiresIn: 120 }); // 2 minutes, not 7 days
res.json({ url, key });
});The browser then does fetch(url, { method: "PUT", headers: { "Content-Type": "image/jpeg" }, body: file }). Simple, but notice what you cannot express: a maximum size. With a bare PUT the client controls Content-Length, and you have no way to say “reject anything over 5 MB” at the storage layer. You can pin the content type, you can pin the exact key, you cannot bound the size to a range.
A presigned POST can. It signs a policy, a list of conditions the upload must satisfy, and the bucket rejects the whole request if any of them fail:
import { createPresignedPost } from "@aws-sdk/s3-presigned-post";
const { url, fields } = await createPresignedPost(s3, {
Bucket: "my-bucket",
Key: `uploads/${req.user.id}/${randomUUID()}.jpg`,
Conditions: [
["content-length-range", 1, 5 * 1024 * 1024], // 1 byte to 5 MB, enforced by S3
["starts-with", "$Content-Type", "image/"], // must be some image/*
],
Fields: { "Content-Type": "image/jpeg" },
Expires: 120,
});
res.json({ url, fields }); // browser posts these fields + the file as multipart/form-dataThe browser builds a FormData, appends every entry from fields, appends the file last, and POSTs it to url. If the file is 40 MB, storage refuses it. That content-length-range is the difference that makes POST the right call whenever you accept files from people you do not trust, which is all of them.
Constrain the presign, then confirm it landed
Whatever you sign is the entire security boundary for that upload, so make it tight. Pin the key (or at least a starts-with prefix scoped to the user), pin or prefix the content type, bound the size, and set a short expiry. A loose presign like “any key, any type, any size, valid for a week” is functionally an open write handle to your bucket.
And then, crucially, do not trust the client to tell you the truth about what happened. The browser saying “upload complete” is a hint, not proof. Confirm it yourself with a HEAD:
import { HeadObjectCommand } from "@aws-sdk/client-s3";
app.post("/uploads/:id/complete", requireAuth, async (req, res) => {
const record = await db.upload.find(req.params.id); // status: "pending"
const head = await s3.send(
new HeadObjectCommand({ Bucket: "my-bucket", Key: record.storage_key }),
);
if (head.ContentLength > 5 * 1024 * 1024) {
await deleteObject(record.storage_key);
return res.status(413).json({ error: "too large" });
}
await db.upload.update(record.id, {
status: "ready",
size_bytes: head.ContentLength,
});
res.sendStatus(204);
});Only now does the row flip to ready. Before this call, the object might not exist, might be truncated, might be the wrong thing entirely. This confirm step is also the natural seam for the deeper validation we are about to get to.
Trust nothing the client said about the file
You now have an object in a bucket. Everything you “know” about it came from the client, and the client can lie about all of it. This is where uploads earn their reputation. Run every file through the same gauntlet.
Never trust the filename
req.file.originalname is an attacker-controlled string. If you join it onto a path and write it to disk, ../../../etc/cron.d/x is a real request and it does real damage. This class of bug is path traversal, and it also shows up as overwriting a sibling user’s file, or a null byte truncating the extension.
The fix is not to sanitize the name. It is to not use it at all. Generate your own key and, if you want a friendly extension, derive only the extension from the name after validating it against an allowlist:
import path from "node:path";
const ALLOWED_EXT = new Set([".jpg", ".jpeg", ".png", ".webp", ".gif", ".pdf"]);
function storageKey(userId, originalName) {
const ext = path.extname(originalName).toLowerCase(); // ".jpg", or ""
const safeExt = ALLOWED_EXT.has(ext) ? ext : "";
return `uploads/${userId}/${randomUUID()}${safeExt}`;
}Keep the original name if you want to show it in the UI, but store it as a plain database string, never as part of a filesystem path or a storage key. Display data and path data are different things and the whole exploit is confusing the two.
Never trust the content type
The Content-Type header and the file extension are both just labels the client attached. A file called avatar.png sent as image/png can be a Windows executable, an HTML page full of script, or a polyglot that is valid as two formats at once. The only way to know what a file is is to read its opening bytes, its magic number, and check the actual signature. A PNG starts with 89 50 4E 47. A JPEG starts with FF D8 FF. A PDF starts with 25 50 44 46, which is %PDF.
You do not hand-roll this. The file-type package (v21 as of 2026, ESM-only) reads the signature for you:
import { fileTypeFromBuffer } from "file-type";
const ALLOWED_MIME = new Set(["image/jpeg", "image/png", "image/webp", "application/pdf"]);
// the first few KB is enough; you do not need the whole file
const detected = await fileTypeFromBuffer(header); // header = first ~4100 bytes
if (!detected || !ALLOWED_MIME.has(detected.mime)) {
throw new Error("file content does not match an allowed type");
}Two honest caveats. file-type is best-effort by its own documentation, not a security oracle: it recognizes known binary signatures, it does not prove a file is well-formed or safe. And it works on bytes, so on the presigned path you have to fetch the first range of the object to check it, GetObject with a Range: bytes=0-4100 header, rather than reading a request body you never received. Detection tells you “this looks like a PNG”. It does not tell you “this PNG is safe to process”. For that you cap decoder work and push the heavy lifting into a sandboxed job.
Try it on your own files. The next demo reads a file entirely in your browser, shows the size and the type the browser reports, and dumps the first sixteen bytes so you can see the signature with your own eyes. Pick a .png, then rename a .zip to .png and pick it again. The extension changes. The bytes do not.
Treat SVG as executable, because it is
This one catches people who did everything else right. An SVG is not an image in the way a PNG is. It is an XML document, and the SVG spec allows <script> elements, on* event handler attributes like onload, <foreignObject> carrying arbitrary HTML, and javascript: links. When a browser renders an SVG served from your origin with Content-Type: image/svg+xml, it runs that script in your origin. That is textbook stored XSS: a user uploads logo.svg, another user views it, and the attacker’s script runs with your victim’s session. It sails straight past a naive extension check because .svg really is an image extension.
You have three defensible options, and a bad one:
- Serve it as an attachment. Send
Content-Disposition: attachmentand a locked-downContent-Security-Policy, so the browser downloads the file instead of rendering it live. Kills the script vector, but now it is a download, not an inline image. - Sanitize it server-side with a hardened XML/SVG sanitizer that strips
<script>, event handlers, and external references, and store only the cleaned output. Doable, but sanitizers and browser parsers disagree at the edges, and that gap is where bypasses live. - Serve user content from a separate origin, a different domain with no cookies and no session, so even if a script runs it runs in a sandbox that can reach nothing. This is why the big platforms serve uploads from
*.usercontent.comand the like. - The bad option: accept SVGs, serve them inline from your app origin, and hope. Do not.
If you only need raster output, the cleanest move is to convert uploads to a safe format in a background job and throw the original away.
Serving it back: who is allowed to see this?
Getting the file back to a user is its own decision with real tradeoffs, and the right answer depends on one question: how much control do you need over who downloads what, and when?
A public bucket is the whole file behind a plain URL, usually with a CDN in front. It is the fastest and cheapest way to serve, and it enforces nothing. Anyone with the URL has the file forever, so this is only for genuinely public assets: marketing images, avatars you have decided are public, static thumbnails.
A signed GET URL is the read-side mirror of the presigned upload. You mint a short-lived URL and the user fetches bytes straight from storage, so you still offload the transfer, but access is time-boxed:
import { GetObjectCommand } from "@aws-sdk/client-s3";
app.get("/files/:id/link", requireAuth, async (req, res) => {
const record = await db.upload.find(req.params.id);
if (!canRead(req.user, record)) return res.sendStatus(403);
const url = await getSignedUrl(
s3,
new GetObjectCommand({ Bucket: "my-bucket", Key: record.storage_key }),
{ expiresIn: 300 }, // 5 minutes
);
res.redirect(url);
});The catch is revocation. Once you hand out a signed URL, it is valid until it expires, and you cannot take it back short of rotating the account credentials, which nukes every URL at once. If a user’s access should end the instant you say so, a five-minute window is five minutes too long.
Streaming through your origin is the strict option. Every request hits your handler, you run whatever authorization you like, and only then do you pull the object and pipe it to the response:
import { GetObjectCommand } from "@aws-sdk/client-s3";
app.get("/files/:id", requireAuth, async (req, res) => {
const record = await db.upload.find(req.params.id);
if (!canRead(req.user, record)) return res.sendStatus(403); // checked on EVERY download
const object = await s3.send(
new GetObjectCommand({ Bucket: "my-bucket", Key: record.storage_key }),
);
res.setHeader("Content-Type", record.content_type);
res.setHeader("Content-Length", String(object.ContentLength));
object.Body.pipe(res); // Node SDK gives you a readable stream; stream, do not buffer
});You pay for it. The bytes flow through your process, so you carry the egress and the memory pressure and you lose the CDN, but in exchange you can authorize every single download, revoke access the moment a subscription lapses, and log exactly who fetched what.
Big files: multipart and resumable uploads
A single PUT has a ceiling. On S3 the largest object you can write in one request is 5 GB, and long before that a single request over a flaky connection is a bad bet, because one dropped TCP connection at 90% means starting over from zero.
The multipart upload API solves both. You split the file into parts (5 MiB minimum each, except the last, up to 10,000 parts, up to a 5 TiB object), upload the parts independently and in parallel, and finish with one call that stitches them together by their ETags. A failed part is retried on its own. Nothing else restarts.
You can presign each part, so a browser drives a big multipart upload directly to storage without your server ever seeing a byte, which folds every idea above into one flow. For truly hostile networks (mobile, spotty wifi) there is the tus protocol, an open standard for resumable uploads that survives a connection dropping and picking back up minutes later from the exact offset it left off.
One rule ties the big-file story together: anything expensive belongs in a job, not in the request. Generating thumbnails, transcoding video, running a virus scan, re-encoding an untrusted image to a safe format: none of that should block the upload response, and some of it (decoding attacker-controlled files) is dangerous enough that it wants an isolated worker anyway. The upload finishes fast, you enqueue the work, and a background job picks it up. Your handler stays quick and your process stays safe.
Summary
- Files do not belong in your database. Blobs bloat backups, blow out the buffer cache, and slow replication. Store bytes in object storage (S3, R2, GCS) and keep only a small pointer row in the database.
- Buffering an upload into memory is a memory bomb. A 2 GB upload is a 2 GB allocation; enough of them get your process OOMKilled. It is fine only for small files behind a hard cap.
- Streaming keeps memory flat but still routes every byte through your server, costs double the bandwidth, and hits serverless body limits (Vercel caps at 4.5 MB).
- Presigned direct-to-storage is the default. Your API authorizes and mints a short-lived, tightly-scoped URL; the browser uploads straight to the bucket. Use a presigned POST when you need
content-length-rangeto bound the size; a presigned PUT cannot. - Confirm the upload server-side. The client saying “done” is not proof.
HEADthe object, check its size and type, and only then mark the row ready. - Validate everything. Never build a path from the filename (traversal); store a generated key. Never trust the reported content type; sniff the magic bytes. Cap the size. Treat SVG as executable and never serve it inline from your app origin.
- Serving is a tradeoff: public bucket (fast, no control), signed URL (time-boxed, no early revoke), or stream-through your origin (per-request authorization and audit, but you carry the bytes). This course streams its PDF through the origin so it can authorize every download.
- Big files use multipart uploads (parts retried independently) or a resumable protocol like tus, and heavy processing belongs in a background job.