Area 02 // RAGLesson 34 / 14015 min read

Document parsing and ingestion pipelines

Before documents can be searched, they must be turned into clean, structured text (parsing), and a repeatable ingestion pipeline must keep the search index in sync as documents are added, changed and deleted.

On this page

Think of it like this
The problem it solves
How it works: document parsing
How it works: keeping the index fresh
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

A library can't shelve a book that arrives as a pile of torn pages with adverts glued in. Someone has to clean it up, put the pages in order, label the chapters, and note the author and date. When a new edition arrives, they replace the old one; when a book is withdrawn, they take it off the shelf.

Parsing is cleaning up each book. The ingestion pipeline is the library's process for keeping the shelves up to date.

The problem it solves

Real documents are messy: PDFs with two columns, headers and footers on every page, scanned images, tables that turn into a jumble of numbers, web pages full of menus and cookie banners. Feed that straight into chunking and embedding, and the index fills with noise. Retrieval returns half-sentences and footer text, and the model answers from garbage.

Then documents change. Policies get updated, pages are deleted, new files arrive daily. Without a pipeline, the index drifts out of date and the bot confidently quotes last year's rules.

How it works: document parsing

Identify the type: born-digital PDF, scanned PDF, Word, HTML, slides, spreadsheet, email.
Extract text with the right tool:
- Simple PDFs: PyMuPDF or pypdf.
- Complex layouts (columns, tables, figures): layout-aware parsers like Docling, Unstructured or LlamaParse.
- Scans and photos: OCR (Tesseract, cloud OCR) or a vision model.
- HTML: strip navigation, scripts, footers; keep headings and main content.
Keep structure: convert headings, lists and tables to markdown; keep page numbers. Structure helps chunking and gives the LLM context.
Clean: remove repeated headers and footers, fix hyphenated line breaks, normalise whitespace and odd characters.
Attach metadata: source ID, title, URL or path, section, page, author, last-updated date, language, and access permissions.
Spot-check parsed output by eye for each document type; parsing bugs are invisible otherwise.

How it works: keeping the index fresh

An ingestion pipeline runs on a schedule or on events (a file uploaded, a webhook from the wiki):

Detect changes: list documents with IDs and a content hash (or updated timestamp). Compare with what's indexed.
New documents: parse → chunk → embed → insert.
Changed documents: delete the old chunks for that document ID, then parse, chunk, embed and insert the new ones (an "upsert" by document).
Deleted documents: delete their chunks. Easy to forget, and a classic source of wrong answers.
Unchanged documents: skip (hash matches), saving embedding cost.
Re-embed everything only when you change the embedding model or chunking strategy; build the new index alongside the old one and switch when it passes evals.
Track it: log counts per run, failures per document, and index freshness (age of newest/oldest chunk).

An incremental ingestion run: only changed documents cost embedding money.

See it in code

Needs: Nothing: plain Python 3 (standard library only).

Turning a web page into clean, structured text (keeping headings, dropping menus and scripts), and the hash check that decides whether to re-embed.

parse_and_detect.py

import hashlib
from html.parser import HTMLParser
 
class TextExtractor(HTMLParser):
    """Keep headings and paragraphs, drop navigation, scripts and footers."""
    SKIP = {"nav", "script", "style", "footer"}
    def __init__(self):
        super().__init__(); self.parts, self.skip, self.tag = [], 0, None
    def handle_starttag(self, tag, attrs):
        self.skip += tag in self.SKIP; self.tag = tag
    def handle_endtag(self, tag):
        self.skip -= tag in self.SKIP
    def handle_data(self, data):
        if self.skip or not data.strip(): return
        prefix = "## " if self.tag in ("h1", "h2") else ""   # keep structure as markdown
        self.parts.append(prefix + data.strip())
 
page = """<html><nav>Home | Shop | Login</nav><h1>Returns policy</h1>
<p>You can return items within 30 days.</p><script>track()</script>
<h2>Exceptions</h2><p>Gift cards cannot be returned.</p><footer>(c) Acme</footer></html>"""
 
p = TextExtractor(); p.feed(page)
clean = "\n".join(p.parts)
print(clean)
 
# Keep the index fresh: only re-embed documents whose content actually changed.
stored_hash = "0" * 64                                    # hash saved at the last ingestion
new_hash = hashlib.sha256(clean.encode()).hexdigest()
print("changed -> re-chunk and re-embed" if new_hash != stored_hash else "unchanged -> skip")

Output

## Returns policy
You can return items within 30 days.
## Exceptions
Gift cards cannot be returned.
changed -> re-chunk and re-embed

For PDFs, the same idea with a real parser: pip install pymupdf, then import pymupdf; text = "".join(page.get_text() for page in pymupdf.open("policy.pdf")), or a layout-aware parser like Docling for tables and columns.

A real example

Example

The insurance bot that quoted a cancelled policy

An insurer's assistant kept telling customers about a "no-claims protection" add-on that had been discontinued months earlier. The product page had been deleted from the website, but the ingestion job only ever added and updated documents; it never removed chunks for deleted pages.

The fix had three parts: the job now lists every source document each night and deletes chunks whose document no longer exists; every chunk stores updated_at and the bot prefers recent sources; and a freshness dashboard shows the oldest content in the index. They also found their PDF parser had been turning premium tables into unreadable number strings, and switched to a layout-aware parser for those files.

The easy way

Start here

Start with one document type and look at the parsed text with your own eyes before you chunk anything. Store a document ID, URL and updated date on every chunk. Write the nightly sync so it handles add, change and delete from day one.

When you grow: the production setup

Production pipelines run as scheduled or event-driven jobs (background jobs) with retries and dead-letter queues, use connectors for each source system, apply PII redaction and permission mapping at ingestion (access control in RAG), version indexes, and alert on failures and staleness.

Trade-offs

What you gain

Clean, structured text makes every later step (chunking, retrieval, answers) better.
Incremental ingestion keeps the index fresh at low cost.
Rich metadata enables citations, filters, freshness rules and access control.

What it costs you

Parsing complex PDFs, tables and scans is hard and tool-dependent.
Pipelines need scheduling, monitoring and error handling like any data system.
Layout-aware parsers and OCR add cost and processing time.

When not to use it

A one-off prototype over a handful of clean markdown files doesn't need a full pipeline; a script is fine until the data starts changing.

Common mistakes

Never looking at the parsed text, so broken tables and footer noise go unnoticed.
Handling new and updated documents but not deleted ones.
Re-embedding everything every night instead of only what changed.
Dropping metadata (source, date, permissions) at parse time, making citations and filters impossible later.
Using one parser for every document type.

Interview questions

Try answering out loud first, then open the model answer.

How would you design an ingestion pipeline for a RAG system?

Model answer

Connectors list documents from each source with stable IDs and change markers. A scheduler or webhook triggers incremental runs: compute content hashes, parse new and changed documents with type-appropriate parsers, clean and preserve structure, chunk, embed in batches and upsert by document ID; delete chunks for removed documents. Attach metadata (source, section, page, updated_at, ACLs). Use queues with retries and dead-letter handling, log metrics, monitor freshness, and re-index fully only when changing the embedding model or chunking.

How do you handle tables and scanned documents?

Model answer

For tables, use layout-aware parsing and keep them as markdown or HTML, possibly as their own chunks with a generated description, so rows and columns stay aligned. For scans, run OCR or a vision-language model, check quality on samples, and keep page images for citation. Evaluate retrieval on questions that need table values.

How do you keep a RAG index consistent with its sources?

Model answer

Track every indexed chunk by document ID and version; run incremental syncs that compare source listings and hashes against the index, applying adds, updates (delete then insert by document) and deletes; make operations idempotent so retries are safe; record updated_at for freshness ranking; and alert when syncs fail or the index ages beyond a threshold.

Quiz

Pick an answer to see whether it’s right and why.

0 of 6 answered

Key terms

Document parsing: Turning PDFs, web pages, scans and office files into clean, structured text (headings, tables, page numbers) ready for chunking.
OCR: Optical character recognition. Reading text from images of pages, such as scanned PDFs or photos.
Ingestion pipeline: The repeatable process that loads, parses, chunks, embeds and stores documents, and keeps the index in sync as they change.
Metadata: Extra information stored with each chunk, such as source, page, date, owner and access permissions, used for citations and filters.
Content hash: A short fingerprint computed from a document's content. If it changes, the document changed and needs re-indexing.

Before and after

Read first

032 What RAG is: the whole pipeline