Running Models in the Browser
I built a notes app with a search box that actually understood what you meant. Type “trip costs” and it surfaced the note called “Lisbon budget,” even though the two share no words. The trick was embeddings: turn each note into a vector of numbers that captures its meaning, turn the query into a vector, rank by how close they sit. It worked well on the first try, which is usually when I stop paying attention.
Two problems showed up in the same week. The first was the bill. Every debounced keystroke fired a request to a hosted model to embed the query, so an idle afternoon of me typing in my own notes cost real money. The second was worse. To embed a private note, I was shipping that note to a third party. A journal app that couriers your journal to someone else’s server is not a journal app people trust.
Then I looked up the model that produces those embeddings. It is about 23 megabytes. It can run inside the tab. The query never has to leave the laptop, the note never has to leave the laptop, and after the one-time download it costs nothing to run a million times.
That is the whole pitch for on-device inference, and this article is about when it actually holds up.
The round trip you did not need
A normal model call is a network request. You already know the shape from fetch: your code sends data to a server, the server runs the model, the answer comes back. For a hard task that is exactly right, and most of this course is about doing it well. But look at what that round trip costs for a small task, the kind where the model is tiny and the work is trivial.
For a small model doing a small job, the network is most of the cost and most of the latency, and it is the entire privacy exposure. Take the network out and three problems disappear at once. You do not pay per call. You do not wait for a round trip. The data stays on the machine that produced it.
None of that helps for a genuinely hard task, because a small local model cannot do a genuinely hard task. That tension runs through the whole article. On-device is not a smaller version of the cloud. It is a different tool for a different slice of work.
What makes this possible now
Running a model means doing a great deal of matrix multiplication, fast. For years the browser had no good way to do that. JavaScript on the CPU was too slow, and the GPU was locked behind a graphics-only API. Two pieces changed that.
The first is WebGPU. It gives JavaScript direct access to the GPU for general computation, not just drawing. That distinction matters and it is the reason WebGPU exists as a separate thing from WebGL. WebGL was built to put triangles on screen, and people bent it into doing math by smuggling numbers through textures. WebGPU exposes compute shaders directly, which is exactly the primitive a model needs. The deeper story of how the GPU pipeline works is in Canvas, WebGL, and WebGPU; here the only thing you need is that WebGPU is the fast path for the linear algebra inside a model.
The second is WebAssembly as the fallback. When WebGPU is missing or blocked, the same library can run the model on the CPU through a WASM build that is far quicker than hand-written JavaScript. Slower than the GPU, but it works, and “works slowly” beats “does not run.”
On top of those two sits a library that hides the ugly parts: loading weights, building the compute graph, tokenizing input, running the forward pass. In JavaScript the common one exposes a single pipeline() call.
Where does this stand in mid-2026? WebGPU reached Baseline across the major browsers early this year, which is the honest word for “you can assume it on modern Chrome, Edge, Safari, and Firefox, but still feature-detect.” Chrome and Edge have shipped it the longest. Safari got it with the version 26 release across macOS, iOS, and iPadOS. Firefox rolled it out through the same window. Linux is the ragged edge, because GPU driver coverage there is uneven, so treat a WebGPU-less visitor as a normal case, not an exotic one.
Quantization is how the model gets small enough
A model is a big pile of numbers called weights. By default each weight is a 32-bit float. A model with 22 million weights at 32 bits each is around 90 megabytes, and that is a small model. Ship that as-is and every first-time visitor downloads 90 MB before your feature does anything.
Quantization is the trick that makes this practical. You store each weight in fewer bits: 8 instead of 32, sometimes 4. The number is less precise, but a model has so many weights that most tasks barely notice the rounding. The download shrinks by roughly the same ratio as the bit-width.
In a JavaScript inference library this is usually a single option when you load the model, a dtype set to something like "q8" for 8-bit or "q4" for 4-bit. The right choice is empirical. Try the smaller one, measure whether your task still passes, and drop to 8-bit or full precision only if it does not. For embeddings and classification, aggressive quantization is often invisible in the results. For a small chat model, over-quantizing shows up as noticeably worse writing, so you feel it sooner.
Local semantic search, start to finish
Here is the notes-search feature, done on-device. The shape is: load an embedding model once, turn every note into a vector ahead of time, and at query time embed the query and rank by similarity. The model name here is a placeholder for whatever small embedding model you pick, and it is meant to be swapped.
import { pipeline } from "@huggingface/transformers";
// Load once. First call downloads the weights; later calls read the cache.
// device:"webgpu" uses the GPU when available; dtype:"q8" is the 8-bit build.
const embed = await pipeline("feature-extraction", "your-small-embedding-model", {
device: "webgpu",
dtype: "q8",
});
// Turn text into one normalized vector (here, 384 numbers).
async function vector(text) {
const out = await embed(text, { pooling: "mean", normalize: true });
return out.data; // a Float32Array
}Embeddings are just vectors, so “similar in meaning” becomes “pointing in a similar direction.” Because we asked the model to normalize each vector to length 1, the cosine similarity between two of them is simply their dot product. No library needed.
// Both vectors are unit length, so the dot product IS the cosine similarity.
function similarity(a, b) {
let dot = 0;
for (let i = 0; i < a.length; i++) dot += a[i] * b[i];
return dot; // roughly 1 = same meaning, 0 = unrelated
}Embed the notes once and keep the vectors around. Then every search is pure arithmetic over numbers you already have, with no model call and no network at all.
// One-time: embed each note. Do this in the background, cache the result.
const notes = await loadNotes();
for (const note of notes) {
note.vec = await vector(note.body);
}
// Per keystroke: one embed of the query, then a cheap ranked scan.
async function search(query) {
const q = await vector(query);
return notes
.map((n) => ({ note: n, score: similarity(q, n.vec) }))
.sort((a, b) => b.score - a.score)
.slice(0, 5);
}That is the entire feature. On a machine with WebGPU, embedding a short query takes single-digit milliseconds after warm-up, the ranked scan over a few thousand notes is trivial, and not one byte of anyone’s notes left the device. Compare that to a debounced API call per keystroke, each one a round trip you pay for and a privacy exposure you have to justify.
Keep it off the main thread
There is a trap in the code above. That first model load, and every inference, is heavy synchronous-feeling work, and if you run it on the main thread the UI freezes. Buttons stop responding, scrolling jerks, the tab looks broken. This is the same single-threaded reality behind the event loop: one long task blocks everything else the page wants to do.
The fix is a Web Worker. Run the model in a worker, talk to it with messages, and the main thread stays free to handle clicks and paint frames while the GPU grinds.
The worker owns the model and answers requests:
// embed.worker.js
import { pipeline } from "@huggingface/transformers";
let embedPromise; // load lazily, once
function getEmbedder() {
embedPromise ??= pipeline("feature-extraction", "your-small-embedding-model", {
device: "webgpu",
dtype: "q8",
});
return embedPromise;
}
self.onmessage = async (e) => {
const embed = await getEmbedder();
const out = await embed(e.data.text, { pooling: "mean", normalize: true });
// Send the raw buffer; it transfers without a copy.
self.postMessage({ id: e.data.id, vec: out.data }, [out.data.buffer]);
};On the page side you post text and await the matching reply. Give each request an id so overlapping searches do not get their answers crossed.
const worker = new Worker(new URL("./embed.worker.js", import.meta.url), {
type: "module",
});
const pending = new Map();
worker.onmessage = (e) => {
pending.get(e.data.id)?.(e.data.vec);
pending.delete(e.data.id);
};
function vector(text) {
const id = crypto.randomUUID();
return new Promise((resolve) => {
pending.set(id, resolve);
worker.postMessage({ id, text });
});
}Same vector() signature as before, but now the UI never stalls, no matter how slow the device.
Feature-detect, then degrade
You cannot assume WebGPU, and you cannot assume the user’s phone has the memory for your model. So detect, and have a plan for every “no.” The check itself is one line.
function pickBackend() {
if ("gpu" in navigator) return "webgpu"; // fast path
if (typeof WebAssembly === "object") return "wasm"; // slower, still local
return "server"; // no local option, call the API instead
}Three tiers, worst case included. WebGPU when you can get it. WASM when you cannot, accepting that a big model may be too slow to be pleasant, so this tier suits the smallest models. And a server fallback for the rest, where you send the request off to a hosted model exactly like the cloud path this article opened by criticizing. That is not a contradiction. The point was never “never use the cloud.” The point is to stop using it for work the device can do for free.
Below is a live version of that detection running in your own browser right now, next to a small calculator for the cost question. No model is downloaded and nothing is sent anywhere. It only reads capability flags and does arithmetic, so you can see where on-device stops being a nice idea and starts being the obvious call.
The calculator is deliberately blunt about it. Per-call pricing accrues on every request for as long as the feature exists, while the on-device cost is a one-time download the user’s own machine pays for and then caches. For a high-volume, low-difficulty task, that is not a close call.
The hybrid pattern is usually the real answer
Pure on-device is the right architecture less often than pure cloud, and the honest sweet spot is neither. It is a split. Run a cheap first pass locally, and escalate to a cloud model only for the cases the local one cannot handle. Most requests never leave the device; the hard minority does.
Concretely, that looks like on-device embeddings feeding a local search, and a cloud model only when the user asks a real question about the results. Or a small local classifier that labels the obvious cases and passes the ambiguous ones up. Or a local model that drafts a reply instantly for the user to edit, with a “make this better” button that calls the big model on demand. The local pass is free and private; the escalation is where you spend money, and you spend it rarely. This is the same cost-and-reliability logic as server-side model routing and fallbacks, with the cheapest tier of all bolted on underneath: the user’s own device.
What about the model the browser ships itself?
There is a second way to run a model locally that does not involve you shipping any weights at all. Some browsers now bundle a small model and expose it through a built-in API, so your code asks the browser to run inference and the browser owns the download. As of mid-2026 this is real but early and uneven: one major browser has moved it toward a stable web API, others are behind or absent, and the model, its size, and its context limit differ per browser and keep changing.
Do not build a feature that only works in one browser’s preview API. Treat a built-in model as a fast lane you take when it happens to be there, with the library-on-WebGPU path as the version that actually ships everywhere.
Where on-device actually wins
Strip away the hype and the honest list is short and specific. On-device is the right call when the task is small enough for a small model, and at least one of these is true:
- Privacy is the point. The data is sensitive and never sending it anywhere is a feature, not a nice-to-have.
- It has to work offline. Cached weights mean the feature runs with no network at all.
- Volume is high and difficulty is low. Per-call pricing on a torrent of easy requests is exactly what a one-time download undercuts.
- Latency matters and the task is tiny. A local embedding or classification beats any round trip, because there is no round trip.
And it is the wrong call when the task needs frontier reasoning, long context, or an answer you could not get from a model small enough to fit in a tab. There is no shame in that. Send those to the cloud, and let the device handle the fat head of easy work it can do for free.
Summary
- The browser can run real models locally now, on WebGPU for GPU speed with a WASM CPU fallback, driven from JavaScript by a
pipeline-style library. - WebGPU is the compute-focused sibling of WebGL and is the reason the heavy matrix math is fast enough to be practical; as of 2026 it is broadly shipped but still needs feature detection.
- Quantization (storing weights in 8 or 4 bits instead of 32) is what shrinks a model to a browser-feasible download, for a small and usually acceptable accuracy cost.
- Weights download once and cache, so the size hit is a one-time cost, but that first load is real on mobile and storage quotas can evict big models.
- Run inference in a Web Worker so the heavy work never freezes the main thread, and feature-detect the backend so you degrade to WASM or a server instead of breaking.
- On-device wins for privacy-sensitive, offline, high-volume-low-difficulty, or low-latency small tasks, and loses for anything that needs a frontier model.
- The pattern that usually wins is hybrid: a free, private on-device first pass, escalating to the cloud only for the hard tail, the same routing logic as fallbacks and routing with the device as the cheapest tier.