Skip to content

All lessons

Area 01 // LLMsLesson 27 / 14013 min read

Multimodal inputs

Multimodal models can take images, PDFs, audio and sometimes video as input alongside text, by turning each into tokens the model can attend to, which lets you build features like reading receipts, describing screenshots or answering questions about charts.

Think of it like this

Analogy

Explaining a broken washing machine over the phone is slow and error-prone: "there's a light... sort of orange... near the knob...". Sending a photo is instant: "Ah, that's the door-lock light."

A multimodal model is the friend you can send the photo to. It reads pictures, documents and sounds along with your words.

The problem it solves

A lot of real information isn't text: receipts, scanned forms, screenshots of errors, charts in reports, product photos, voice notes. Converting them to text first (manual typing, separate OCR, transcription) is slow, loses layout and context, and adds pipelines to maintain.

Multimodal models let you hand the image or document straight to the model with a question, which unlocks many features with very little code.

How it works, step by step

  1. Choose a multimodal model. As of November 2026, the flagship models from OpenAI, Anthropic and Google accept images and PDFs; many accept audio; some accept video. Open models such as Gemma, Qwen-VL and Llama 4 also handle images. Check each model's supported input types and limits.
  2. Turn the image into tokens. Inside a vision-language model, an image is resized, cut into small square patches, and each patch becomes a vector, giving image tokens that sit in the same sequence as text tokens. Attention then links words to image regions.
  3. Send it in a content block. In the API, the user message has a list of blocks: the image (as a URL or base64 data) and the text question. Put the image first, then the question.
  4. PDFs are handled as text plus page images, so charts and layouts are understood too (Anthropic: a document block; OpenAI: file inputs).
  5. Audio goes through speech models or audio-capable chat models (transcribe, then reason, or reason directly on audio).
  6. Pay for image tokens. Cost depends on image size and the provider's formula; large images are downscaled to a maximum. Resize or crop to the part that matters.
  7. Validate outputs. For extraction (receipts, forms), combine with structured outputs and checks.
Images become patch tokens that flow through the same transformer as the text.

See it in code

Needs
Nothing for this offline version (plain Python 3). Sending it needs an API key, as in lesson 10.

Builds a real (tiny) PNG in memory and wraps it the way both APIs expect. Swap in open("receipt.jpg", "rb").read() for a real image.

image_message.py
import base64, json, struct, zlib
 
def tiny_png(width=2, height=2, rgb=(255, 0, 0)):
    # Builds a real (tiny) red PNG in memory so this runs without any image file.
    raw = b"".join(b"\x00" + bytes(rgb) * width for _ in range(height))
    def chunk(kind, data):
        return struct.pack(">I", len(data)) + kind + data + struct.pack(">I", zlib.crc32(kind + data))
    header = struct.pack(">IIBBBBB", width, height, 8, 2, 0, 0, 0)
    return b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", header) + chunk(b"IDAT", zlib.compress(raw)) + chunk(b"IEND", b"")
 
image_b64 = base64.standard_b64encode(tiny_png()).decode()   # images travel as base64 text
 
# An Anthropic Messages API user turn: image block first, then the question.
anthropic_content = [
    {"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": image_b64}},
    {"type": "text", "text": "What colour is this image?"},
]
# The same idea in the OpenAI Responses API.
openai_input = [{"role": "user", "content": [
    {"type": "input_text", "text": "What colour is this image?"},
    {"type": "input_image", "image_url": f"data:image/png;base64,{image_b64}"},
]}]
 
print("PNG bytes:", len(tiny_png()), "| base64 chars:", len(image_b64))
print("anthropic block types:", [b["type"] for b in anthropic_content])
print("openai block types   :", [b["type"] for b in openai_input[0]["content"]])
print("request size (chars) :", len(json.dumps(anthropic_content)))
Output
PNG bytes: 73 | base64 chars: 100
anthropic block types: ['image', 'text']
openai block types   : ['input_text', 'input_image']
request size (chars) : 244

To send: anthropic.Anthropic().messages.create(model=..., max_tokens=500, messages=[{"role": "user", "content": anthropic_content}]), or OpenAI().responses.create(model=..., input=openai_input). Base64 makes files about a third bigger, so for large or repeated files use URLs or the providers' file upload APIs.

A real example

Example

Expense receipts in seconds

An expenses app lets employees snap a photo of a receipt. The app resizes it to about 1,500 pixels on the long side, sends it with a structured-output schema (merchant, date, total, currency, VAT, nullable fields), and pre-fills the claim form.

Accuracy on their 500-receipt eval set was 96% for totals. Failures were crumpled or faded thermal receipts, so the app shows the extracted values next to the photo for the user to confirm, and flags low-contrast images for a re-take. A month's work replaced a third-party OCR vendor.

The easy way

Start here

Take a screenshot of an error message, send it to a multimodal model with "What does this error mean and how do I fix it?". Then try a receipt with "Return merchant, date and total as JSON". That's 80% of what you'll build with images.

When you grow: the production setup

Production image pipelines resize and crop before sending, strip metadata (location in photo EXIF data), scan uploads for malicious content, use file APIs for repeated documents, combine vision with structured outputs and validation, keep humans in the loop for low-confidence extractions, and evaluate per document type. For large document collections, see multimodal RAG.

Trade-offs

What you gain

  • Understands layout, charts, photos and screenshots without separate OCR pipelines.
  • Very little code: an image block plus a question.
  • Opens features that were impractical before (visual QA, document understanding).

What it costs you

  • Images cost many tokens and add latency.
  • Weak on tiny text, exact counting and precise measurements.
  • Images can carry hidden instructions (text in the image) and private data.

When not to use it

  • You already have the content as clean text: send text, it's cheaper and more precise.
  • You need pixel-exact measurements or guaranteed OCR accuracy for compliance: use specialised tools, possibly alongside the model.

Common mistakes

  • Sending full-resolution photos when a resized crop would do.
  • Trusting extracted numbers without validation or user confirmation.
  • Forgetting that text inside images can contain prompt-injection instructions.
  • Leaking private metadata (GPS location, names) from user uploads.
  • Assuming every model supports every modality and file type.

Interview questions

Try answering out loud first, then open the model answer.

How do vision-language models process images?

Model answer

A vision encoder (often a vision transformer) splits the image into patches, embeds each patch, and projects the result into the language model's embedding space, producing image tokens that are interleaved with text tokens. The transformer attends across both, so text can refer to image regions. Images are resized within limits, which determines the token count and cost.

Design a receipt-extraction feature using a multimodal model.

Model answer

Client resizes and compresses the photo; server strips metadata, checks file type and size, and calls a multimodal model with a structured-output schema (merchant, date, total, currency, tax, nullable fields). Validate values (date format, totals sum, plausible ranges), show results with the image for user confirmation, route low-confidence cases to review, log corrections as eval data, and measure field-level accuracy per receipt type.

What new risks come with image inputs?

Model answer

Indirect prompt injection via text embedded in images, private data in images and metadata, higher cost and latency from large images, abuse via malicious or illegal content requiring moderation, and over-trust in extracted values. Mitigate with input scanning and moderation, metadata stripping, size limits, treating image text as untrusted data, validation and human confirmation.

Quiz

Pick an answer to see whether it’s right and why.

  1. 01What does a multimodal model accept?
  2. 02How does a vision-language model "see" an image?
  3. 03How are images usually sent to an API?
  4. 04Why resize a 6000×4000 photo before sending it?
  5. 05Which task is still risky for vision models?

0 of 5 answered

Key terms

Multimodal
Able to take (or produce) more than one kind of data, such as text plus images, PDFs or audio.
Vision-language model
A model that understands images together with text, for example answering questions about a screenshot.
Image tokens
The vectors an image is turned into (one per patch) so a transformer can process it with text. Billed like tokens.
Base64
A way to write binary data (like an image) as plain text characters so it can travel inside JSON. About a third larger than the original.

Before and after

Read first

Read next

Progress is saved in this browser only.