Multimodal — images, audio & documents

Key takeaways Some models are multimodal — they accept images, audio, and documents alongside text, not just words. That unlocks features like image understanding, OCR, and transcription from the same model API you already know. The output is usually still text, and it is still probabilistic — so treat extracted values as a draft to verify, not a fact. Not every model can do this; see the sibling types of models.

So far this path has treated a model as a thing you send text to and get text back from. But many models now accept more than text, and that opens a set of features you can’t build with words alone. This lesson covers what those inputs are, how you send them, and where the sharp edges are.

Beyond text

A multimodal model takes inputs beyond plain text. The most common and most mature is images — a photo, a screenshot, a chart, a scan. Increasingly, models also accept audio (a voice clip or recording) and whole documents such as a PDF, handed over as a file rather than pasted-in text.

What comes back is usually still text — a description, an answer, a transcript — or, when you ask for it, structured output your program can use directly. The extra modality is on the input side; the model reads or hears more, then responds in the familiar way.

What you can build

Concretely, multimodal input lets you:

  • Describe or analyze an image — hand the model a photo and ask what it shows, or what looks wrong with it.
  • Extract text or fields from a photo or scan (OCR) — pull the numbers off a receipt, the readings off a display, or the fields off a form.
  • Answer questions about a chart or document — “what’s the trend here?” over a graph, or “what does clause 4 say?” over a PDF.
  • Transcribe audio — turn a spoken clip into text you can store or search.
  • Reason over audio or a document — summarize a recording, or compare two scanned pages, rather than just converting them.

Each of these would be hard or impossible with a text-only model; the input modality is what makes the feature possible.

How you send them

You send non-text inputs as parts of a message. Instead of a message whose content is one string, you build a message whose content is a list: some text, plus an image, audio clip, or file. An image is typically provided either as data (the bytes, encoded) or as a URL the provider can fetch; audio and documents work the same way, as attached parts.

The important thing to internalize is that these parts consume tokens and cost money too. An image is converted into tokens the model reads, and a large or high-resolution one can use many of them; audio and documents are no different. So the cost, latency, and limits you learned to design around apply here as well — a picture is not a free addition to a prompt.

Limits & cautions

A few things to keep straight before you lean on this:

  • Not every model is multimodal. Support is a property of the specific model you call, not a given. Check the sibling types of models lesson for where multimodal and vision fit.
  • Quality varies with the input. Fine print, low resolution, poor lighting, handwriting, and unfamiliar accents all degrade results. Clean inputs read far better than messy ones.
  • The output is still probabilistic. A model reading an image is the same probabilistic component as one reading text — it can be confidently wrong. A misread digit looks exactly like a correct one.
  • Verify before trusting extracted values. Anything you pull out — a total, a serial number, a field — should go through the same parsing and validation as any other model output before your code acts on it.

A GopherTrunk example

Imagine feeding a model a screenshot of a waterfall or spectrum display, or a constellation plot, and asking it to describe the features it can see or to draft a plain-English explanation of what’s on screen. That’s a genuinely handy assistive aid — a way to get a second, verbal read on a picture — and it plays to a multimodal model’s strengths.

But frame it honestly: it is an assistant, not an authoritative decoder. The real decoding is done by GopherTrunk’s DSP, which measures the signal precisely. A model glancing at a screenshot might call a carrier “clean” that a proper measurement shows is degraded. Use the description to orient yourself, then trust the instrument for the numbers.

Quick check: what does "multimodal input" mean?

Recap

  • A multimodal model accepts inputs beyond text — images most maturely, and increasingly audio and documents.
  • The output is usually still text (or structured data); the extra modality is on the input side.
  • It unlocks real features: describing images, OCR, answering questions over charts and documents, and transcribing audio.
  • You send non-text inputs as parts of a message, and they cost tokens like everything else you send.
  • Not every model supports it, quality varies with the input, and the output is still probabilistic — verify extracted values before trusting them.

Next up: Module 4 grounds the model in your own data — why retrieval.