PII, Redaction and Safety
The feature was a support-reply drafter. Paste a customer’s ticket, it writes a first draft in the company’s voice, an agent tweaks it and hits send. People loved it. It shipped in a week and cut reply times in half.
Then a lawyer asked one question in a planning meeting, and the room went quiet. “When a customer emails us to delete everything we hold about them, where does all of it actually live?”
The honest answer took a while to assemble. The customer’s data was in the database, obviously. It was also in the prompt logs in our observability tool, because we logged every request and response to debug the drafts. It was in an eval spreadsheet a contractor had downloaded to their laptop three months earlier. And some copy of it was sitting in a retention buffer at the model provider, for a length of time nobody in the room could state with confidence. Nobody had ever drawn the map of what left the building. The instant that feature called a model, it started shipping customer data to a third party and quietly keeping copies of it, and not one of us had noticed the line we walked across.
This article is about that line. Not the legal fine print, because I am an engineer and not a lawyer, and neither are you. It is about the handful of engineering decisions that decide whether you answer the lawyer’s question with a straight face or a long silence. Calling a model taught you how the request goes out. This is about what is riding in it.
What actually leaves your system
Start with the question everybody skips: what is in the payload? A user’s message is the obvious part. But the thing you send is rarely just their message. It is the message, plus whatever document or image they uploaded, plus the rows you retrieved to give the model context, plus tool results, plus a system prompt that might mention account details. Any of that can carry a name, an email, a phone number, a home address, a card number, a diagnosis, a salary. You assembled a rich packet of someone’s life and you are about to POST it to a company that is not yours.
The industry’s list of top LLM risks has this as its number two entry, right under prompt injection: sensitive information disclosure. It sits that high for a reason. Once the packet crosses your boundary, your code no longer governs what happens to it. The provider’s policy does.
Those three questions have real answers, and the answers move. Here is the durable shape as of 2026, stated carefully because the specifics rot fast.
The paid API tiers of the major providers generally do not train their models on the traffic you send. That is a genuine and important default, and it is different from the consumer chat products, where your conversations often can be used for training unless you opt out. API and chat are not the same contract. Know which one your code is on.
Retention is the sneakier one. Even when a provider does not train on your data, it usually keeps a copy for a while to monitor for abuse. That window has been anywhere from a few days to a month depending on the provider, and it moves. One major provider cut its API retention from thirty days to seven partway through 2025. So a request you thought was fire-and-forget is actually sitting on someone else’s disk for a stretch you probably could not name off the top of your head. For stricter needs there is usually a zero-retention arrangement, where prompts and responses are processed in memory and never written down, but it tends to be gated behind an enterprise agreement rather than a checkbox you can flip yourself.
The practical upshot is boring and important. Before you send real user data through a provider, read its data policy for your tier, and choose a provider partly on that basis, not only on model quality and price. Then tell your users, in plain language, that their input is processed by a third party. That last part is quickly becoming a legal requirement, not a courtesy, and we will get to why.
Send less than you think you need
The cheapest data to protect is the data you never sent. Before anything leaves, ask what the model actually needs to do its job. A summariser needs the text of the ticket. It almost never needs the customer’s full card number sitting in that ticket. So take it out.
This is data minimisation, and it has a mechanical version you can implement today: detect the sensitive spans, swap each one for a placeholder, send the placeholder, and swap the real value back in when the response returns. The model reasons over [CARD_1] and never sees the digits. You re-hydrate on your side, where the data was safe the whole time.
The detector has two halves, and they have different personalities. Structured identifiers such as emails, card numbers, and IBANs have rigid shapes, so a regex nails them with high precision. Names, addresses, and places have no fixed shape, so you need a named-entity model to catch them, and it will never catch all of them. That asymmetry matters when you decide how much to trust the output.
Here is the structured half, the part you can write in an afternoon. Each match becomes a stable placeholder, and the original is stashed in a vault that never leaves your process.
// High-precision structured PII. Names need an NER model, not this.
const PATTERNS = {
EMAIL: /\b[^\s@]+@[^\s@]+\.[^\s@]+\b/g,
CARD: /\b(?:\d[ -]?){13,19}\b/g,
PHONE: /\b\+?\d[\d ().-]{6,}\d\b/g,
};
function redact(text) {
const vault = new Map(); // placeholder -> original, stays on your side
let n = 0;
for (const [label, re] of Object.entries(PATTERNS)) {
text = text.replace(re, (match) => {
const key = `[${label}_${++n}]`;
vault.set(key, match);
return key;
});
}
return { masked: text, vault };
}Re-hydration is the trivial half. The model reply comes back mentioning [CARD_1], and you put the real value back before a human ever sees it.
function rehydrate(text, vault) {
for (const [key, original] of vault) {
text = text.split(key).join(original);
}
return text;
}One detail that separates a real detector from a toy: a card number should pass a Luhn checksum before you mask it, otherwise you will redact every nine-digit order ID and twelve-digit SKU in the text and hand the model a mess of placeholders it cannot reason about. Precision cuts both ways. Over-redaction breaks the feature; under-redaction leaks. The demo below runs a real Luhn check so you can watch it ignore an order number while masking a genuine card.
Try the round trip. Edit the text, then redact and send. The masked panel is exactly what would cross the boundary to the provider. Watch the name leak straight through, because regex cannot catch it, and watch the card get masked while the order number does not.
Tokenising like this is not a cure. It only helps for values the model does not need to understand. If the whole task is “explain this lab report,” you cannot mask the medical terms, because they are the job. There the answer is not redaction, it is choosing a provider and plan whose data terms you are comfortable putting health data through, and often signing the paperwork that makes that lawful. Redaction shrinks exposure. It does not erase the responsibility.
The quiet leak is your own logs
Here is the failure that got the lawyer’s question started, and it is the one almost everybody ships by accident. You want to debug bad drafts and build evals, so you log every prompt and every response. Reasonable. Except the prompt is the user’s data and the response often repeats it back, so your log store is now a growing pile of PII, sitting in a tool with looser access controls than your production database, retained forever because nobody set a policy, and copied into eval spreadsheets that wander onto laptops.
The maturing telemetry standards for AI got this right, and it is worth copying their default even if you roll your own logging. The convention is to record metadata always (which model, token counts, latency, the finish reason) and to make the actual prompt and completion content opt-in, stored as separate events you can filter or drop before they ever land in your backend. Content is treated as sensitive and large, so you capture it deliberately, redact it, truncate it, and keep it briefly, rather than firehosing raw prompts into a log you will keep for a year.
Three habits carry most of the weight:
- Run the same redactor from the last section over anything you log, so the log holds
[EMAIL_1]and not the address. If it was worth masking before it left, it is worth masking before it lands in a store your whole team can read. - Put a real retention policy on prompt and trace stores, measured in days for content, and enforce it with a job, not a good intention. This is the same discipline as any sensitive store, covered in observability and, for the AI-specific plumbing, tracing your model calls.
- Scope who can read it. Prompt logs deserve the access controls of the production database, because functionally they are a slower copy of it.
Guardrails: the output is not automatically safe
Everything so far guards the data on the way in. The other direction needs its own guard. A model can produce text that is toxic, biased, defamatory, wildly off-brand, or that leaks something from its context, and it can do this from a perfectly innocent prompt. In a product with real users, you cannot ship raw model output straight to a screen and hope. You wrap the call.
A guardrail is a layer that lives outside the model and enforces your policy on what goes in and what comes out. It is deliberately not the model, because asking the model to police itself puts the referee and the player in one body. Two passes: an input rail that inspects the user’s request before it reaches the model, and an output rail that inspects the generated response before it reaches the user. Each can allow, block, or rewrite. And the rule when a rail is unsure is to fail closed: a safe refusal beats a risky guess.
You do not have to build the classifiers yourself. As of 2026 there is a healthy spread of options, and they are provider-agnostic building blocks rather than a single product. Some providers ship a free moderation endpoint that scores text against a category list. There are dedicated open safety models you can run yourself that classify a message against a standard taxonomy of harms. There are cloud content-safety APIs, and programmable frameworks that let you write allow and deny rules as policy. They are lightweight enough (typically tens of milliseconds) that you can run them in parallel with, or in front of, the main call without wrecking latency.
In code the shape is the same whichever tool you pick. Screen, then generate, then screen again, and divert on a flag.
async function guardedReply(userText) {
// Input rail: cheap classifier + your own policy checks.
const inCheck = await moderate(userText);
if (inCheck.blocked) return refusal(inCheck.reason);
const draft = await model.generate(userText);
// Output rail: never trust the generation blindly.
const outCheck = await moderate(draft);
if (outCheck.blocked || leaksSecret(draft)) {
log.warn("output_rail_tripped", { reason: outCheck.reason });
return refusal("cannot_share");
}
return draft;
}Two honest caveats keep this from turning into theatre. First, a guardrail is your policy layer and it is not the same thing as the provider’s own content filter, the one that shows up as a content_filter stop reason in calling a model. The provider filter catches egregiously harmful requests to protect its own service. Your rail enforces your product’s rules, which are narrower and specific to you. Keep both. Second, these classifiers are themselves probabilistic and can be talked past, exactly like the model they guard. So a rail lowers risk, it does not zero it, and it belongs alongside the input-side defences from prompt injection, not instead of them.
Do not cross the tenant streams
One more leak deserves its own line, because it is embarrassing when it happens and easy to prevent. If your app serves many users or many customer organisations, the cardinal sin is putting one tenant’s data into another tenant’s prompt. It creeps in through shared context: a retrieval step that is not scoped to the current user pulls a neighbour’s document, a “recent examples” feature seeds the prompt with someone else’s chat, a cache keyed too loosely serves a stored answer across accounts.
The fix is the same boundary you already enforce everywhere else, applied to the prompt. Every retrieval is filtered by the caller’s identity before a single row reaches the context. This is ordinary authorization and multi-tenancy discipline, and the model does not get an exemption because it feels like a magic text box. The query that builds the context runs under the user’s scope, always.
-- The context you assemble for the prompt is scoped like any other read.
SELECT chunk_text
FROM documents
WHERE tenant_id = $1 -- never the whole table
AND owner_id = $2
ORDER BY embedding <=> $3
LIMIT 8;Treat the assembled prompt as a query result that crossed a trust boundary, because it did. If a row would not be allowed in the user’s API response, it does not belong in their prompt either.
Consent, control, and the delete button
Two obligations to your users flow directly from all of the above, and both are moving from nice-to-have toward required.
The first is telling people what is happening. If a human is interacting with an AI system, increasingly the law says you must say so, and if user input is processed by a third-party provider, honest disclosure of that belongs in your privacy notice, not buried where nobody reads it. Let people opt out where you reasonably can, and do not quietly feed their content into a training or fine-tuning pipeline without a clear, separate consent.
The second is deletion, which brings us back to the lawyer’s question. When someone asks you to delete their data, you have to actually reach every copy. Draw the map first, because you cannot delete what you have not enumerated.
The stores you own you can purge, if you built them to be purgeable: a real delete on the row, a job that sweeps the logs, a cache eviction. The provider’s retention buffer you cannot reach with your own code. You reach it through the contract, which is precisely why a short retention window and a data-processing agreement matter, and why zero-retention terms are worth paying for when you handle sensitive categories. Your deletion promise to a user can only be as strong as the weakest copy you are able to reach.
The rules, honestly
I will not pretend to give legal advice, and you should be suspicious of any engineer who does. But you cannot make sane design calls without a rough map of the terrain, and the terrain in 2026 is genuinely shifting, so here is the honest version.
The direction of travel is one way: more disclosure, more rights for the people whose data you handle, more accountability for what your system does with it. In Europe, the AI Act’s transparency duties come into force in 2026, which at minimum means telling users when they are dealing with an AI and marking AI-generated content. Notably, the timeline itself has moved, with the higher-risk obligations pushed out well past their original date, so even the calendar is not stable. General data-protection law already applies the moment a prompt contains personal data: you need a lawful basis, you owe people access and deletion, and your model provider is a processor you should have a signed agreement with.
The United States has no single federal AI law as of 2026 and instead a patchwork of state rules, some live, some rewritten before they took effect, with a federal move afoot to override the lot. It is unsettled enough that the specific citation matters less than the shape. Then there are the sector rules that predate all of this and bite hard: health data drags in HIPAA and a required agreement with any vendor that touches it, card data drags in PCI, and financial and education data have their own regimes.
None of this is the exciting part of building with models. It is the part that decides whether your feature is a quiet asset or the reason your company is in the news. The teams that stay out of trouble are not the ones with the fanciest model. They are the ones who can answer the lawyer’s question without the room going quiet.
Summary
- The moment your feature sends user text to a model provider, you are handing personal data to a third party and keeping copies of it. That is a data-handling responsibility, and it is a legal and trust problem before it is a technical one.
- Know what actually leaves: the message, uploads, retrieved rows, tool results, the system prompt. Read the provider’s data policy for your exact tier (do they train on it, how long do they retain it, where is it processed), pick accordingly, and disclose it. The specifics move, so check the current terms rather than trusting any number you read once.
- Minimise before you send. Detect structured PII with regex (Luhn-checked cards, emails, phones) and names with an NER model, swap sensitive spans for placeholders, send the placeholders, and re-hydrate on your side. Bias for recall, and treat it as a reducer, not a guarantee.
- Your own logs, traces, eval sets, and caches quietly become PII stores. Capture metadata by default and content only opt-in, redact and truncate what you keep, scope access like a production database, and put a short retention policy on it that a job enforces.
- Guardrails are a separate layer from the model. Run an input rail and an output rail around every call, fail closed on doubt, and keep them alongside, not instead of, your injection defences and the provider’s own content filter.
- Never let one tenant’s data into another tenant’s prompt. Scope every retrieval by the caller’s identity, and key caches per user, exactly as authorization and multi-tenancy already demand.
- Tell users when they are dealing with an AI, let them opt out and delete, and build every store you own to be purgeable so you can honour a deletion request. The regulations are tightening through 2026 and the timeline is shifting, but the five engineering habits that make you ready do not change.