Tracing and Observability for AI
A customer asked our support bot whether their Growth plan included priority support. It said yes, warmly, with a citation. It does not. Priority support is a Scale-plan feature, so the answer cost us a wrongly-set expectation, an escalation, and a ticket that read, in full: “the AI is hallucinating again.”
So I opened the logs. The request log had exactly one line for that interaction: POST /api/chat 200 OK 3.9s. That is everything my normal server logging knew about the most expensive request of my afternoon. The line was true. The HTTP call succeeded, the model returned, nothing threw. By every signal my web stack understands, that request was a clean success. It just produced a wrong answer, and my logs had no vocabulary for “wrong.”
You know how this ends because you have done it too. I bolted a console.log onto the final prompt, re-ran the same question, and there it was. The retrieval step had pulled a chunk from the Scale plan’s support section, the prompt handed the model the wrong tier’s text, and the model faithfully summarized the document it was given. The model was not hallucinating. It was doing precisely what I asked, with the wrong input, and I could not see the input.
That is the whole reason AI needs its own tracing. Most “the model is dumb” bugs are “the prompt we built was wrong” bugs, and from the outside they look identical. You cannot tell them apart, or fix either, without seeing what actually happened inside the chain. Your observability instincts from the server chapters all still apply. This is those instincts pointed at a system with a genuinely different shape.
One request is a tree, not a line
A normal web request is close to a straight line: it comes in, touches a couple of services, writes a row, returns. One AI request is a tree. A single question can fan out into a query rewrite, a vector retrieval, a rerank of the candidates, a couple of tool calls, and finally the generation, each one a separate model or service call with its own inputs and outputs. And every model step in that tree is non-deterministic, so you cannot reason about it by reading the code. You have to see the specific run.
This is the same span-waterfall you met in observability, and if you have adopted OpenTelemetry already, the mechanics are unchanged: a span is a named, timed operation with a parent, and assembling spans by parent gives you the tree. What changes for AI is which operations you make into spans and what you record on each one. A trace that only shows POST /api/chat 3.9s is worthless here. A trace that shows rewrite, retrieve, rerank, tool, generate as five child spans, each with its exact input and output, is the difference between guessing and knowing.
The agent loop makes this sharper. An agent can run the same tool three times, take seven steps one run and four the next, and quietly re-send a growing transcript on every turn. Without a trace tying the turns together, an agent that misbehaves in production is a black box you cannot replay. With one, you scroll the waterfall and see the exact turn it went sideways.
What to write down for every step
A span is only as useful as what you attach to it. Get the fields right and the trace answers your questions before you finish typing them. Get them wrong and you have a pretty timeline that still can’t tell you why the answer was bad.
The prompt and the response come first
If you record one thing, record the exact prompt you sent and the exact response you got back. Not the template. Not the user’s original message. The final, fully-assembled string of bytes that actually went over the wire, and the raw thing that came back. This single field resolves the majority of “the model is dumb” reports, because most of them are not about the model at all. They are about a retrieval that grabbed the wrong chunk, a variable that interpolated to undefined, a system prompt that got truncated, or a tool result that came back as [object Object]. All of those are invisible in a template and glaring in the assembled prompt.
Recording this in your own code is a few lines around each model call. If you have OTel, you open a span, set attributes on it, and let the exporter ship it wherever you look at traces:
import { trace, SpanStatusCode } from "@opentelemetry/api";
const tracer = trace.getTracer("ai");
async function generate(messages, opts) {
return tracer.startActiveSpan("chat llm-large", async (span) => {
// parameters: cheap, always safe to record
span.setAttribute("gen_ai.operation.name", "chat");
span.setAttribute("gen_ai.request.model", opts.model);
span.setAttribute("gen_ai.request.temperature", opts.temperature);
// the exact assembled prompt (see the privacy note before you keep this)
span.setAttribute("gen_ai.input.messages", JSON.stringify(messages));
try {
const res = await callModel(messages, opts);
span.setAttribute("gen_ai.output.messages", JSON.stringify(res.messages));
span.setAttribute("gen_ai.usage.input_tokens", res.usage.inputTokens);
span.setAttribute("gen_ai.usage.output_tokens", res.usage.outputTokens);
span.setAttribute("gen_ai.response.finish_reasons", res.finishReason);
return res;
} catch (err) {
span.recordException(err);
span.setStatus({ code: SpanStatusCode.ERROR });
throw err;
} finally {
span.end();
}
});
}The rest of the span
Beyond prompt and response, each span carries the facts that turn a shrug into a diagnosis:
- Model and parameters. Which model, at what temperature, with what
max_tokens, and which system prompt version. A regression that started “yesterday at 4pm” is very often a model or prompt change, and you can only see that if the version is on the span. - Token counts and cost. Input tokens and output tokens per call. Cost is derived from those, and it belongs on the trace so you can add it up per request and per feature. More on that below.
- Latency, split. Total duration, and for a streamed call the time to first token (TTFT) separately, because they fail for different reasons and users feel them differently.
- The retrieved chunks. For a retrieval span, the ids and similarity scores of what came back, and which ones survived the rerank. This is exactly where my priority-support bug lived, and it was one glance away in the trace.
- Tool calls, verbatim. For every tool call, the tool name, the arguments the model chose, and the result you handed back. When an agent does something baffling, the argument it invented is usually the answer.
- The finish reason.
stopmeans the model finished on its own.lengthmeans it hit the token ceiling and got cut off mid-thought, which explains a truncated answer instantly.tool_callsmeans it wants to act. A truncated response withfinish_reason: lengthis a five-second diagnosis and an afternoon of confusion without it.
A retrieval span looks much like the generation one, just with different attributes. The chunk ids and scores are the payload:
tracer.startActiveSpan("retrieval vector-store", (span) => {
span.setAttribute("gen_ai.operation.name", "retrieval");
span.setAttribute("retrieval.query", rewrittenQuery);
const hits = store.search(rewrittenQuery, { k: 8 });
// the ids and scores are what let you spot a wrong-chunk bug at a glance
span.setAttribute("retrieval.chunk_ids", hits.map((h) => h.id).join(","));
span.setAttribute("retrieval.scores", hits.map((h) => h.score.toFixed(3)).join(","));
span.end();
return hits;
});One id ties the whole journey
None of this helps if you cannot pull one user’s path out of the pile. As in the server chapters, you mint a correlation id at the edge, the first moment the request arrives, and thread it through every model call, every retrieval, every tool. If you adopt OpenTelemetry the id you propagate is the trace id, so your logs and your spans line up for free. When a specific user reports a specific bad answer, you filter to their id and the entire tree, rewrite through generation, comes back in order. The mechanics are covered in observability; the only AI-specific note is that the tree you reconstruct is deeper and every node in it can lie to you, which is exactly why you want the whole tree and not a single log line.
Cost, latency, quality: the three you watch together
Here is what makes AI observability its own thing rather than a reskin of your APM dashboard. You are watching three numbers at once, and traditional request tracing gives you a good view of only one of them.
Cost. Every model call has a price, and the price is input_tokens * inputRate + output_tokens * outputRate. Put the token counts on the span and cost is a derived sum you can roll up per request, per feature, per customer. This is where you discover that one rarely-used feature is 40% of your model bill because it stuffs a 30k-token document into every call. Do not hard-code the rates; they change often and vary by model, so keep them in config and look them up.
// rates in dollars per million tokens; keep them in config, they move
const RATES = {
"llm-large": { input: 3.0, output: 15.0 },
"llm-small": { input: 0.15, output: 0.6 },
};
function costUSD(model, usage) {
const r = RATES[model];
return (usage.inputTokens * r.input + usage.outputTokens * r.output) / 1_000_000;
}The details of counting and pricing tokens are their own topic, covered in cost and tokens. What matters for tracing is that cost is a first-class number on the trace, not a surprise on the monthly invoice.
Latency. You already track total duration. For AI, split out time to first token. On a streamed response the user starts reading as soon as the first token lands, so a 900ms TTFT with a slow tail can feel faster than a 600ms wait for a complete answer. TTFT and total latency also point at different causes. TTFT is dominated by prompt length, because the model has to read the whole prompt (the “prefill”) before it emits anything, so a context that grew from 4k to 12k tokens can double it. Total latency is dominated by how much the model generates. Watching only one number hides half the problem.
Quality. This is the signal generic request tracing has no word for, and it is the whole point of an AI system. A request can be fast, cheap, and 200 OK, and still be wrong, which is exactly the failure I opened with. So you attach a quality signal to the trace: an automated eval score (faithfulness to the retrieved context, or an LLM-as-judge rating), a thumbs up/down from the user, or a hard downstream signal like the user immediately rephrasing the question or escalating to a human. Now “show me the traces from last Tuesday with a faithfulness score under 0.5” is a query you can run, and it hands you your worst answers on a plate.
Turn real failures into tests
Once your worst answers are queryable, a genuinely useful loop opens up, and it is the payoff for doing all of the above. A bad answer in production is not just an incident to close. It is a test case you did not have yet.
The move is simple. When a trace shows a bad answer, you take its input, its retrieved context, and the correct output you wish it had produced, and you drop that into your eval dataset. Now that exact failure runs on every future change to the prompt, the model, or the retrieval, and if a change reintroduces the bug, the eval catches it before it ships. Production stops being a place bugs hide and becomes the richest source of tests you have.
This is why the trace has to carry the retrieved context, not just the final prompt. To turn a failure into a fair test, you need to reproduce the exact conditions: the same question and the same chunks. If your trace only kept the final answer, you cannot rebuild the case, and the failure is gone the moment the incident closes. Modern eval and tracing tools ship a literal “add to dataset” button on a trace for this reason; wired into your CI, the dataset becomes a regression suite that grows itself from real usage.
Prompts are full of PII
Everything above has a sharp edge. The single most useful field, the exact prompt, is also the field most likely to contain personal data, because users type their real problems into these systems: names, order numbers, medical questions, whatever they need help with. A trace store full of raw prompts is a database of sensitive user data wearing a debugging tool’s clothes, and it is subject to exactly the same rules as any other place you keep that data.
So the discipline from PII and safety applies unchanged. Scrub obvious identifiers before they land on a span. Put real access control on the trace UI, because “anyone on the eng team can read any user’s prompts” is a data-handling problem even when it feels like just a dev tool. Set retention so prompts age out instead of accumulating forever. And redact at the tracer, as a backstop, so a stray field does not leak just because someone forgot.
// scrub before the prompt ever reaches a span; redaction is a backstop, not the plan
function scrub(text) {
return text
.replace(/[\w.+-]+@[\w-]+\.[\w.-]+/g, "[email]")
.replace(/\b\d{12,19}\b/g, "[card]");
}
span.setAttribute("gen_ai.input.messages", scrub(JSON.stringify(messages)));You cannot keep every trace
At real volume, storing a full, content-rich trace for every request is a lot of data and a real bill. AI traces are heavier than normal ones, because a single trace can carry tens of thousands of tokens of prompt and response text. So you sample. The head-versus-tail split from observability carries over directly, and for AI the tail wins even more clearly.
Head sampling decides at the start, before you know anything, usually a fixed percentage. It is cheap, and it is a coin flip that throws away most of your errors and bad answers along with the boring successes. Tail sampling decides after the trace finishes, once you know how it went, so it can be smart: keep every trace that errored, every trace with a low quality score, every trace a user thumbed down, and a small percentage of the clean successes for baseline. You keep the traces you will actually open, and drop the ones you never will.
There is a standard now
You do not have to invent any of this from scratch, and you should not invent the field names. OpenTelemetry has GenAI semantic conventions: an agreed set of span names and attributes for exactly this. A model call is a span named by its operation and model (chat llm-large), a tool call is execute_tool <name>, a retrieval and an agent invocation have their own operation names. The attributes are the ones used throughout this article: gen_ai.request.model, gen_ai.usage.input_tokens and output_tokens, gen_ai.response.finish_reasons, with prompt and response content as opt-in message attributes.
Be honest about the maturity, though, because it is still moving. As of mid-2026 these conventions are widely implemented but formally still in “Development” status, so names shift between versions and you should pin the version you target rather than assume it is frozen. The practical upshot is good: instrument against the conventions and your traces are portable across backends, because the major observability vendors and the AI-native tools (things like the OpenLLMetry instrumentation, and platforms such as Langfuse, Phoenix, and others) all speak this vocabulary. As with all of Part 6, treat the specific tool names as examples that will churn. The concept, a standard span shape for AI calls, is the durable part.
See a trace you can open
Below is a real trace explorer, with a hardcoded trace and no network anywhere on the page. It is the same request from the diagrams: rewrite, retrieve, rerank, a tool call, then generate. Click any span in the waterfall to open it and read its prompt, response, tokens, cost, and latency. The header totals the cost and time and flags the slowest step, which is the first thing you look at when a request feels slow.
Click the retrieval span and look at the scores. The right chunk (growth#support, 0.74) barely edged out the wrong one (scale#support, 0.71). That 0.03 gap is the whole ballgame: on a slightly different phrasing the wrong chunk wins, the prompt gets the wrong tier, and you ship the exact bug from the top of this article. In a trace, that near-miss is right there in the numbers. In a request log, it does not exist.
Summary
- When an AI feature gives a bad answer, “it is broken” is not debuggable from a request log.
200 OKtells you the HTTP call worked, not whether the answer was right. You need to see inside the chain. - One AI request is a tree, not a line: rewrite, retrieve, rerank, tool calls, generate, each a non-deterministic step. Trace it as a waterfall of spans tied together by one correlation id, the same shape from observability pointed at a deeper, less trustworthy system.
- Per span, the highest-value field by far is the exact assembled prompt and the raw response, because most “the model is dumb” bugs are “the input we built was wrong” bugs. Also record model and parameters, token counts, cost, latency (split out time to first token), retrieved chunk ids and scores, tool calls with arguments and results, and the finish reason.
- Watch cost, latency, and quality together. Cost is
tokens * rate, rolled up per request and per feature. Latency splits into TTFT (driven by prompt length) and total. Quality is the signal generic APM lacks: an eval score or user feedback attached to the trace. - Feed production failures back into your eval dataset. A bad trace, captured with its input and retrieved context and the correct answer, becomes a regression test that runs in CI on every prompt or model change.
- Prompts contain PII. Scrub before spans, put access control and retention on the trace store, and keep content capture opt-in. The rules from PII and safety apply in full.
- Sample the tail. Keep 100% of errors and low-quality answers, plus a small slice of successes. A fixed head-sampling percentage throws away exactly the traces you would have opened.
- OpenTelemetry’s GenAI semantic conventions give a standard span shape (
chat,execute_tool,retrieval, and thegen_ai.*attributes). Widely adopted, still “Development” status in mid-2026, so pin the version. Treat specific tools as examples; the standard is the durable part.