Skip to content

All lessons

Area 05 // AI System DesignLesson 91 / 14018 min read

Case study: a document Q&A system

Designing a document Q&A system for a large company means building an ingestion pipeline (parse, chunk, embed, index, keep in sync), a permission-aware retrieval and answering path with citations, and evals and monitoring, sized from the number of documents, queries and tokens.

Think of it like this

Analogy

A company library with a very good librarian. New books and memos arrive every day: someone catalogues them, notes which floors are allowed to read them, and shelves them (ingestion). When you ask a question, the librarian checks your badge, finds the relevant pages you're allowed to see, reads them and answers with page numbers (query path). Every month someone tests the librarian with known questions (evals).

The problem it solves

A company has 200,000 documents in wikis, drives and ticket systems. Employees want to ask questions and get correct, cited answers, but documents change daily, some are confidential to certain teams, PDFs have tables and scans, and nobody wants a system that confidently quotes last year's policy. This is the most common enterprise AI project, and it's mostly a data and retrieval problem.

How it works, step by step

  1. Requirements: answers grounded only in company documents, with citations; respect source permissions; new documents searchable within an hour; p95 answer under 5 s; about 50,000 questions a day.
  2. Estimate (see the code): corpus tokens, chunk count, vector storage, embedding cost, daily ingestion and query cost.
  3. Ingestion path (async, background jobs):
    • Connectors pull from each source, incrementally, using change events or modified dates.
    • Parse PDFs, slides and HTML, with OCR and table handling (parsing).
    • Chunk by structure with overlap (chunking); add contextual headers.
    • Embed with batching (batch inference) and store vectors with metadata: source, URL, title, date, and the ACL (who may read it).
    • Index in a vector database plus a keyword index; handle updates and deletions.
  4. Query path (online):
  5. Evals: a golden set of questions with known source passages; retrieval metrics (recall@k, MRR) and answer metrics (faithfulness, citation accuracy) in CI (RAG evals).
  6. Operations: ingestion lag and failure dashboards, freshness checks, feedback buttons, re-indexing when the embedding model changes.
Architecture
Loading diagram…
Two paths: background ingestion keeps the index fresh; the query path answers with permission-filtered retrieval.
  • Wikis, drives, tickets to Ingestion workers
  • Ingestion workers to Embedding model
  • Ingestion workers to Index
  • Employee to Q&A service
  • Q&A service to Index: filtered search
  • Q&A service to Answer model: top chunks

See it in code

Needs
Nothing: plain Python 3. Example prices as of November 2026 (embedding $0.02 per 1M tokens, mid-tier model $2/$10).

Sizing the ingestion and query sides.

docqa_sizing.py
# Sizing a document Q&A system: ingestion (one-off + daily) and query cost.
docs, pages_per_doc, tokens_per_page = 200_000, 15, 500
chunk_tokens, overlap = 500, 50
new_docs_per_day = 2_000
 
corpus_tokens = docs * pages_per_doc * tokens_per_page
chunks = corpus_tokens // (chunk_tokens - overlap)
embed_price = 0.02                                   # $ per 1M tokens (small embedding model, Nov 2026)
dims, bytes_per = 1_536, 4
print(f"corpus: {corpus_tokens / 1e9:.1f}B tokens -> {chunks / 1e6:.1f}M chunks")
print(f"one-off embedding cost: ${corpus_tokens / 1e6 * embed_price:,.0f}")
print(f"vector storage: {chunks * dims * bytes_per / 1e9:,.0f} GB float32 "
      f"({chunks * dims * 1 / 1e9:,.0f} GB at int8)")
daily = new_docs_per_day * pages_per_doc * tokens_per_page
print(f"daily ingestion: {daily / 1e6:.0f}M tokens -> ${daily / 1e6 * embed_price:.2f}/day to embed")
 
queries_per_day, ctx_tokens, out = 50_000, 6_000, 400
q_cost = queries_per_day * 30 * (ctx_tokens / 1e6 * 2.00 + out / 1e6 * 10.00)
print(f"queries: {queries_per_day:,}/day -> ~${q_cost:,.0f}/month on a mid-tier model")
Output
corpus: 1.5B tokens -> 3.3M chunks
one-off embedding cost: $30
vector storage: 20 GB float32 (5 GB at int8)
daily ingestion: 15M tokens -> $0.30/day to embed
queries: 50,000/day -> ~$24,000/month on a mid-tier model

Embedding is cheap; the real ingestion cost is engineering (connectors, parsing, permissions). 20 GB of vectors fits comfortably in pgvector or any vector database. The query bill is where caching, reranking to fewer chunks and routing pay off.

A real example

Example

The permission bug found by evals

A law firm built this system across 180,000 documents. Before launch, the team added eval cases that asked, as a junior associate, about a confidential merger only the deal team could see. The first run returned a cited answer: the ACL filter was applied to the vector search but not to the keyword search in the hybrid step.

After the fix, both retrievers shared one permission filter built from the user's groups, and a CI test asserted that the confidential question returned "I couldn't find this". The system launched with 94% recall@5 on 400 golden questions and an ingestion lag under 20 minutes.

The easy way

Start here

Start with the RAG capstone pattern (build a RAG app): one source, a parsing step, chunks with metadata in pgvector, hybrid search, citations and a small golden set. Add sources, permissions and sync one at a time.

When you grow: the production setup

Enterprise versions run connector frameworks with incremental sync and deletion handling, document-level ACLs mirrored from source systems, structure-aware parsing with OCR, contextual chunking, hybrid search with reranking, per-tenant or per-department indexes, query routing (Q&A vs search vs summarise), evals in CI and online, freshness SLOs, and re-embedding pipelines for model upgrades.

Trade-offs

What you gain

  • Answers stay current without retraining.
  • Citations let users verify answers.
  • Permission filtering keeps confidential data safe.

What it costs you

  • Connectors, parsing and permissions are a lot of work.
  • Quality depends on retrieval, which needs constant evaluation.
  • Long contexts make each query fairly expensive.

When not to use it

  • The corpus is small enough to fit in the prompt (a few hundred pages): use long context with prompt caching instead.

Common mistakes

  • Applying permission filters to only one of the retrievers.
  • Never deleting chunks of removed documents.
  • Losing tables and headings during parsing.
  • No "I don't know" path, so weak retrieval still produces confident answers.
  • Changing the embedding model without re-embedding the whole index.

Interview questions

Try answering out loud first, then open the model answer.

Design a document Q&A system for a large company.

Model answer

Split into ingestion and query paths. Ingestion: connectors with incremental sync, parsing with OCR and tables, structure-aware chunking with context, batched embedding, indexing into vector and keyword stores with metadata including ACLs, and handling of updates and deletions. Query: authenticate, rewrite, hybrid search filtered by permissions, rerank, answer with citations or abstain, guard and log. Size it (tokens, chunks, storage, costs), and measure with retrieval and answer evals plus online feedback.

How do you enforce document permissions in RAG?

Model answer

Store each chunk's ACL (allowed users or groups) as metadata copied from the source; at query time compute the user's groups from authentication and filter every retriever by them before ranking, so restricted text never reaches the model. Sync ACL changes quickly, test with eval cases for restricted content, and audit access.

How do you keep the index fresh?

Model answer

Use change events or periodic incremental crawls per source, re-process only changed documents, delete chunks of removed documents, track ingestion lag and failures with alerts, store document versions and dates, and show sources' last-updated dates in citations.

Quiz

Pick an answer to see whether it’s right and why.

  1. 01Which part of a document Q&A system runs asynchronously?
  2. 02Where must permission filtering happen?
  3. 03In the sizing example, what's the larger ongoing cost?
  4. 04A document is deleted in the source system. What must happen?
  5. 05How do you know retrieval is good enough?

0 of 5 answered

Key terms

Ingestion pipeline
The repeatable process that loads, parses, chunks, embeds and stores documents, and keeps the index in sync as they change.
Chunk
A small piece of a document (often a few hundred tokens) that is embedded, stored and retrieved as one unit.
Hybrid search
Running keyword search and vector search together and merging their results, to catch both exact terms and meaning.
Reranker
A model that re-scores a short list of retrieved candidates more carefully, so the most relevant ones end up first.
Citation
A reference in an answer that points to the exact source passage supporting a claim.
RAG access control
Filtering retrieval by the asking user's permissions (using access lists stored with each chunk), so restricted documents never reach the model or the answer.

Before and after

Read first

Read next

Progress is saved in this browser only.