Images, Audio and Documents as Input
The receipt scanner worked perfectly in the demo. Snap a photo, and the model reads back the merchant, the date, the line items and the total, all as clean JSON. We tested it on a dozen tidy sample images, felt good about it, and shipped. Then a real user pointed their phone at a crumpled receipt and uploaded the shot straight from the camera roll: a 12-megapixel image, 4,000 pixels on the long side. That one call cost more tokens than the entire conversation wrapped around it. A photo of a corner-store receipt, more expensive than three thousand words of chat.
Nothing was wrong with the code. We just did not understand that an image, to a model, is not a single thing you attach. It is a pile of tokens, and a big photo is a big pile.
Text stopped being the only thing you can send a while ago. As of 2026, essentially every frontier model reads images as a standard input, and many read audio and PDFs too. That opens a different class of feature than plain chat: describe this screenshot, pull the fields off this receipt, transcribe and summarise this sales call, answer questions about this chart. The mechanics are not hard. But three things surprise people every time, and this page is about all three: how the image actually goes in, what it costs, and where vision quietly gets things wrong.
A picture goes in the messages array
You already saw in system, user and assistant roles that a call is a list of messages, each with a role and some content. Up to now that content has been a plain string. The one change multimodal asks of you is small: a message’s content can be a list of parts instead of a single string, and each part is tagged with a type. A text part carries text. An image part carries an image. You put both in the same user turn, and the model reads them together.
In code, that user turn is just an object with an array for content:
const messages = [
{
role: "user",
content: [
{ type: "text", text: "What is wrong with this chart?" },
{ type: "image", image: "https://cdn.example.com/q3-revenue.png" },
],
},
];The exact field names shift between SDKs. Some spell the image part image_url with a nested { url }; some call it input_image; some want the media type spelled out. The shape underneath is the same everywhere: content becomes a list, and one of the entries is an image instead of text. Learn that shape once and every provider’s version reads the same.
An image part takes its data one of two ways, and the choice is practical, not deep. You either point at a URL the provider can fetch, or you inline the bytes as a base64 data URL right there in the request.
// Already hosted somewhere the provider can reach? Point at it.
{ type: "image", image: "https://cdn.example.com/q3-revenue.png" }
// Local, private, or ephemeral? Inline the bytes.
{ type: "image", image: "data:image/png;base64,iVBORw0KGgoAAAANSUhEUg..." }Use a URL when the file already lives somewhere public or behind a link the provider can load, which often means you uploaded it first and got back a URL. Inline base64 when the image is on the user’s device, is private, or only needs to exist for this one call and you would rather not host it. One honest tradeoff: base64 makes the JSON payload you send roughly a third larger than the raw file, because that is what base64 encoding does. It does not change what the model charges you, though. Whichever way the pixels arrive, the model turns them into the same tokens once they land, which is the part that actually costs you.
Why that photo cost so much
Here is the fact that blindsided our receipt scanner. An image is converted into tokens before the model reads it, exactly like text is in tokens, and why they cost you. And the count is not small. The image gets resized, sliced into a grid of tiles, and each tile is turned into a bundle of tokens that sit in the context window right alongside your words. More pixels means more tiles means more tokens. A postage-stamp thumbnail is cheap. A full-resolution phone photo is not.
The exact arithmetic differs by provider, and it changes often, so do not memorise anyone’s formula. But the two families are worth recognising because they behave differently. Some models cut the image into fixed-size tiles (say 512 pixels square), charge a flat rate per tile plus a small base, and cap how big the image can get first. Newer models lean on patches instead: chop the picture into tiny squares (16 or 32 pixels), turn each into a token-like vector, and let the count scale almost linearly with area up to a budget. Either way, resolution is the dial that sets the price.
Low detail and high detail
Because a full-resolution image is expensive, most vision APIs give you a detail setting, usually low and high. It is one of the highest-impact knobs in the whole part, and people leave it on the default without knowing it exists.
- Low detail downscales the image hard, to a small thumbnail, and charges a small fixed cost no matter how big the original was. The model sees the gist: rough layout, big shapes, obvious colours. It cannot read fine print.
- High detail keeps the resolution (up to the provider’s cap), tiles the whole thing, and charges per tile. Now it can read the small text on a receipt or the tick labels on a chart, and you pay for the privilege.
Play with the numbers. Set a width and height, flip between low and high detail, and watch the estimated tokens move. It uses a simple patch model with a rough budget cap, so it is an approximation, not any real provider’s tokenizer. The shape is the honest part: small images are cheap either way, and once an image gets big, high detail is where the tokens pile up.
Two habits fall out of this. Downscale before you send, because a model almost never needs a 12-megapixel photo to answer a question a 1,000-pixel one answers just as well. And reach for low detail by default, stepping up to high only when the task genuinely depends on small features. The receipt scanner got cheap the day we started resizing uploads to 1,500 pixels on the long side before they ever reached the model. Same accuracy, a fraction of the tokens. The full picture of where these tokens turn into a monthly bill is in cost and tokens.
What vision is good at, and what it fools you on
Vision models are genuinely useful, and they are also confidently wrong in specific, predictable ways. The trap is that the wrong answers sound exactly as fluent as the right ones. That same receipt scanner once read a total of $47.00 as $41.00 and stated it with total composure. So the rule is simple: know which questions land in the reliable column and which land in the verify-it column, and build checks around the second kind.
The left column is where these models shine. They read text embedded in images well, so a screenshot, a sign, a slide, a scanned page all become searchable. They grasp layout, so “which of these is the primary button” or “summarise this dashboard” works. They describe photos fluently, and they read the story a chart tells: up and to the right, a spike in March, a category that dominates.
The right column is where they slip, and the failures share a root cause. These models are not looking at pixels the way a measuring instrument does. They are pattern-matching a picture into words, and precise, quantitative, positional facts are exactly what that process smears.
- Counting. Ask how many people are in a crowd, or how many failed rows are highlighted in a table, and the answer is reliable up to a handful and then drifts. Past five or six, treat any count as an estimate. Research through 2026 keeps confirming this, models that ace hard reasoning still miscount a dozen objects.
- Coordinates and precise position. “Where exactly is the login button, in pixels” or “draw a box around the defect” tends to be loose. Tellingly, if you hand the model the coordinates as text, its spatial answers improve a lot, which says the weakness is reading position out of the pixels, not reasoning about it.
- Tiny or dense text. Small print, a wall of fine legal text, a spreadsheet zoomed out. This is partly the detail-level trap from earlier: at low detail the text is literally a blur, and even at high detail there is a size below which the model starts inventing plausible characters rather than reading real ones.
- Faithful spatial reasoning. Left of, right of, in front, behind, overlapping. Models under-use the spatial cues they do have, and compositional questions (“the mug that is behind the book, to the left of the lamp”) trip them up.
Documents and PDFs
PDFs are their own thing, because a PDF is two different animals wearing one file extension. A born-digital PDF (exported from a word processor or a web page) has real, selectable text inside it. A scanned PDF is just photographs of pages with no text layer at all. That difference decides how you should feed it to a model.
Native PDF input is the newest and easiest route where a model supports it: you attach the PDF as a document part, much like an image part, and the provider handles pulling out both the text and the visual layout. Convenient. It also tends to be the priciest, because behind the scenes it often renders pages to images and charges accordingly, so a long PDF can be a lot of tokens.
Extracting the text yourself is the workhorse for born-digital documents. A PDF library reads the embedded text layer straight out of the file, you send that text as an ordinary string, and the model does a plain text-in, text-out job that is fast, cheap and exact. No pixels, no vision tax. The catch is that you lose layout: a two-column page or a complex table can come out scrambled, and you get nothing at all from a scan.
Rendering pages to images is the fallback when the layout carries meaning or the text layer is missing. You rasterise each page to a PNG and send those as image parts, the same as any other image, detail levels and token costs included. This is what you reach for with scanned contracts, engineering drawings, or dense financial statements where the position of things is the information. It is the most expensive of the three, and for a long document it adds up fast, so use it where it earns its keep.
Most serious document pipelines end up mixing these: extract text where the text layer is clean, fall back to page images where it is not, and reserve native input for the cases that need both at once.
Audio
Audio splits the same way documents do, into two routes with different strengths.
The classic pipeline is transcribe first: run the audio through a speech-to-text model, get a transcript, then send that transcript to the language model as plain text. It is cheap, it is well understood, and for “summarise this meeting” or “what did the customer ask for” it is often all you need. The weakness is that a transcript is a flattening. Tone, hesitation, who spoke when, two people talking over each other, the sarcasm in “great, another meeting”, all of that is gone by the time it is words on a line. And your final answer is only ever as good as the transcript, so a model that mangles accents or background noise poisons everything downstream.
Native audio input feeds the sound straight into a multimodal model, no transcript in the middle. Because it hears the audio rather than a text summary of it, it can pick up tone, pacing, speaker changes and emotional register, which matters for things like call-quality review or a voice note where how something was said carries the meaning. The tradeoff is cost and availability: audio is metered by duration or by audio tokens, at rates meaningfully above text, and native support is less universal than image support. Reach for it when the sound itself is the signal, not just the words in it.
Same door out: text or structured data
Notice what the output has been this whole time. You send an image, an audio clip, a PDF, and what comes back is text. Multimodal is almost always about the input side. The model gains new senses; its mouth stays the same.
That is good news, because everything you already know about shaping output still applies. If you want the receipt back as fields you can act on rather than a paragraph you have to parse, you ask for structured output and get back typed JSON. The image is just where the data came from; the discipline of pinning down its shape is unchanged.
One thing worth stating plainly so you do not conflate them: a model that reads images is not the same as a model that makes them. Generating an image from a prompt, or synthesising speech, is a different class of model with its own API, its own costs, and its own failure modes. When someone says “the model is multimodal”, ask whether they mean it takes images in, puts images out, or both, because those are three different capabilities and a given model might have only one.
Sending someone’s photo is a data decision
The moment your feature accepts a user’s image, scan or voice note, you are shipping their data to a third party, and that is a decision with weight, not a detail. A photo can carry a face, a home address on an envelope, a card number on a receipt, a medical detail on a form. Audio carries a voiceprint. A scanned document can carry anything.
So treat every upload as the sensitive payload it might be. Ask what could be in it before you forward it. Know your provider’s retention and training policy, and whether the data leaves a region you are allowed to leave. Get consent where the law or plain decency calls for it. And here is the happy overlap with everything above: downscaling an image and stripping its metadata before you send both cuts your token bill and shrinks your exposure, and redacting an obvious card number or ID before it ever reaches the model is cheaper than explaining later why it was logged. The full treatment of handling user data through a model, including redaction and what providers do with what you send, is in PII and safety.
Multimodal input is one of the genuinely useful shifts of the last couple of years. It is also, underneath the demo magic, the same messages array you already use, plus a token bill that scales with pixels and a reader that is brilliant at gist and shaky at precision. Hold those three facts and you will build things that work and do not surprise you on the invoice.
Summary
- As of 2026, reading images is a standard model capability, and many models also take audio and PDFs. The output is still text (or structured data). Multimodal is mostly about what goes in.
- Images ride in the same messages array: a message’s
contentbecomes a list of parts, mixing text and an image, so one user turn can ask a question about a picture. The image is either a URL or an inline base64 data URL. - Images are converted to tokens, and the count scales with resolution. Providers tile or patch the image and charge per tile or patch, so a big photo can cost more tokens than a page of text.
- The low/high detail knob matters a lot. Low detail is a small fixed cost and sees only shapes; high detail keeps resolution and reads fine print, at a price. Match it to the question and downscale before sending.
- Vision is reliable at reading text, layout, scenes and chart trends, and shaky at counting many objects, exact coordinates, tiny text and precise spatial reasoning. Verify any number that drives something real.
- PDFs go in three ways: native document input where supported, extracting the embedded text for born-digital files (cheap and exact), or rendering pages to images for scans and layout-heavy docs (vision cost). Real pipelines mix them.
- Audio is either transcribed first (cheap, but loses tone and speakers) or fed in natively (hears tone and pacing, costs more). Pick based on whether the sound itself is the signal.
- Sending a user’s image, scan or voice to a provider is a data-handling decision. Know retention and residency, get consent, and redact and downscale before you send.