Area 02 // RAGLesson 44 / 14014 min read

Multimodal RAG: images, tables and PDFs

Multimodal RAG makes charts, tables, diagrams and scanned pages searchable, usually by turning each into good text (markdown tables, captions written by a vision model) or by embedding page images directly, while keeping a link to the original so the answering model and the user can see it.

On this page

Think of it like this
The problem it solves
How it works, step by step
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

Imagine a research assistant who can only read printed words. Hand them a report and they'll skip every chart, photo and diagram, then confidently tell you "the report doesn't mention Q3 revenue", even though there's a big bar chart of it on page 7.

Multimodal RAG gives the assistant eyes: someone describes every picture and table in words for the catalogue, and when one is needed, they look at the real picture.

The problem it solves

Business documents are full of information that isn't plain text: revenue charts, architecture diagrams, product photos, scanned forms, and tables that text extraction scrambles. Standard RAG pipelines extract text, so all of that is invisible. Questions whose answers live in a chart get "I don't know" or, worse, an answer from some other paragraph.

How it works, step by step

Detect non-text elements while parsing: images, charts, diagrams, tables, scanned pages (document parsing). Layout-aware parsers report them with their page and position.
Tables → structured text: convert to markdown (or HTML) with the table's title and surrounding context, keep each table whole as its own chunk.
Images and charts → descriptions: send each to a vision-language model with a prompt like "Describe this chart for search: title, axes, all values, trend." Embed the description.
Scanned pages → OCR (or a vision model) for their text.
Or embed images directly: multimodal embedding models put images and text in the same vector space, so a text question can retrieve a page screenshot. Some systems embed every page as an image ("visual document retrieval"), avoiding fragile parsing.
Keep links to originals: every text stand-in stores the image file, page number and document.
At answer time, send originals: when a caption or table chunk is retrieved, pass the actual image or page to a multimodal LLM along with the text chunks, so the answer is based on the real chart, not a summary.
Cite the page and show the image in the UI.

Retrieve using text stand-ins; answer using the originals.

See it in code

Needs: Nothing: plain Python 3. Captions are written by hand here; in a real pipeline a vision model writes them.

Turning a table and a chart into searchable text records that remember where the original lives.

multimodal_index.py

# Tables and images can't be embedded as raw pixels by a text embedding model.
# Common fix: turn each into searchable text, but keep a link to the original.
table = {
    "title": "Price list 2026",
    "columns": ["Plan", "Monthly price", "Users"],
    "rows": [["Basic", "9 GBP", "1"], ["Team", "29 GBP", "10"], ["Business", "99 GBP", "50"]],
}
 
def table_to_markdown(t):
    head = "| " + " | ".join(t["columns"]) + " |\n|" + "---|" * len(t["columns"])
    body = "\n".join("| " + " | ".join(r) + " |" for r in t["rows"])
    return f"{t['title']}\n{head}\n{body}"
 
image = {"file": "figures/q2-revenue-chart.png", "page": 7,
         "caption": "Bar chart: quarterly revenue 2026, Q1 12m, Q2 15m, Q3 14m GBP."}   # written by a vision model
 
index = [
    {"text": table_to_markdown(table), "kind": "table", "source": "pricing.pdf p.2"},
    {"text": image["caption"], "kind": "image", "source": f"report.pdf p.{image['page']}", "file": image["file"]},
]
for item in index:
    print(f"[{item['kind']}] from {item['source']}\n{item['text']}\n")

Output

[table] from pricing.pdf p.2
Price list 2026
| Plan | Monthly price | Users |
|---|---|---|
| Basic | 9 GBP | 1 |
| Team | 29 GBP | 10 |
| Business | 99 GBP | 50 |
 
[image] from report.pdf p.7
Bar chart: quarterly revenue 2026, Q1 12m, Q2 15m, Q3 14m GBP.

Each record's text is embedded and searched like any chunk; file and source let the answer step attach the original image and cite the page.

A real example

Example

Asking questions about board decks

A finance team's RAG assistant over board presentations kept failing: most numbers lived in charts on slides. They changed ingestion to render every slide as an image, have a vision model write a detailed description of each chart (title, every labelled value, the trend), store that alongside the slide's text, and keep the slide image.

At answer time, the top slides' images go to a multimodal model with the question. "What was EMEA gross margin in Q3?" now finds the right slide and reads the exact bar. They show the slide thumbnail next to every answer, which made reviewers comfortable trusting it.

The easy way

Start here

Start with the cheapest win: keep tables whole as markdown chunks. Then add vision-model captions for charts and diagrams, store the image path, and pass the image to a multimodal model when its caption is retrieved. Test with ten questions whose answers live only in charts.

When you grow: the production setup

Production pipelines use layout-aware parsers, choose per-element strategies (OCR, captioning, table extraction), evaluate visual document retrieval (page-image embeddings) against text stand-ins, store originals in object storage (S3), control vision costs with batching and resolution limits, and track retrieval quality per element type.

Trade-offs

What you gain

Makes charts, tables, diagrams and scans answerable.
Captions and markdown tables reuse your existing text RAG stack.
Sending originals at answer time gives precise, verifiable answers.

What it costs you

Vision captioning and OCR add ingestion cost and time.
Captions can miss or misread values.
Image inputs at answer time increase tokens and latency.

When not to use it

Your documents are almost entirely text: the extra pipeline isn't worth it.

Common mistakes

Assuming the PDF text extractor captured everything important.
Splitting tables across chunks.
Answering from captions alone and getting chart values slightly wrong.
Losing the link from a caption back to its image and page.
No eval questions that need visual content, so the gap stays invisible.

Interview questions

Try answering out loud first, then open the model answer.

How would you build RAG over PDFs full of charts and tables?

Model answer

Use a layout-aware parser to separate text, tables and figures. Convert tables to markdown chunks with titles and context; caption figures with a vision-language model including all values; OCR scanned pages. Embed all text stand-ins with metadata linking to the original element and page, optionally also embed page images with a multimodal embedding model. Retrieve with hybrid search and reranking, then pass both text chunks and original images to a multimodal LLM, citing pages and showing thumbnails. Evaluate with questions answerable only from visuals.

Captions vs multimodal embeddings for image retrieval: trade-offs?

Model answer

Captions reuse text infrastructure, are inspectable and searchable with BM25, but cost a vision call per image and may omit details. Multimodal embeddings skip captioning and can match visual content directly, but need a compatible embedding model and vector space, are harder to inspect, and don't support keyword search. Many systems use both and compare on evals.

Why pass original images to the LLM at answer time?

Model answer

Captions are lossy summaries written without knowing the future question, so they may omit the exact value asked about. Giving the multimodal LLM the original chart lets it read precisely what's needed, reducing errors, while the caption still served as the retrieval key.

Quiz

Pick an answer to see whether it’s right and why.

0 of 5 answered

Key terms

Multimodal RAG: RAG that also retrieves images, charts, tables and scanned pages, using text stand-ins or multimodal embeddings.
Multimodal: Able to take (or produce) more than one kind of data, such as text plus images, PDFs or audio.
OCR: Optical character recognition. Reading text from images of pages, such as scanned PDFs or photos.
Vision-language model: A model that understands images together with text, for example answering questions about a screenshot.
Embedding: A vector of numbers that represents the meaning of a piece of text (or an image), so that similar meanings get similar vectors.

Before and after

Read first