Area 08 // BackendLesson 140 / 14014 min read

Testing APIs with pytest

Testing an AI backend with pytest means calling your endpoints in-process with a test client, replacing the real LLM with a fake through dependency injection so tests are fast, free and repeatable, and checking status codes, response shapes, prompts sent and edge cases, while evals cover answer quality separately.

On this page

Think of it like this
The problem it solves
How it works, step by step
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

Before a flight simulator session, nobody uses a real plane: the simulator reproduces the instruments and responses exactly, so pilots can practise engine failures safely and repeatably. Testing an AI API with a fake model is the same: you check how your system behaves in every situation (normal, empty input, model error) without paying for or depending on the real thing. Checking whether the plane itself flies well is a different test (evals).

The problem it solves

AI backends have lots of ordinary logic around the model: validation, auth, retrieval, prompt building, parsing outputs, tool execution, error handling. If tests call the real LLM they're slow, cost money, need keys in CI, fail randomly and can't easily simulate errors. If there are no tests, every change risks breaking an endpoint that worked yesterday.

How it works, step by step

Install and run: pip install pytest; put tests in tests/test_*.py; run pytest -q.
Write tests as plain functions with assert; pytest shows helpful diffs when they fail.
Use FastAPI's TestClient (or httpx.AsyncClient with an ASGI transport for async tests) to call endpoints in-process (FastAPI).
Make external services injectable: get the LLM client, database and other services through Depends(...), so tests can override them (app.dependency_overrides) with fakes that return fixed replies and record what they were sent.
Fixtures set up and tear down shared state (fake LLM, test database, temporary files); yield fixtures clean up afterwards.
Parametrize to cover many inputs with one test.
Test behaviour, including unhappy paths: validation errors (422), auth failures (401), empty inputs, provider errors and timeouts (fake raises), malformed model output, tool-call handling, and that the prompt contains the right context.
Other layers: integration tests against a real database (often in a container), a few smoke tests against staging, and evals for answer quality with real models (eval gates).
Run in CI on every push and pull request (GitHub Actions); keep the suite fast.

See it in code

Install: pip install fastapi httpx pytest
Needs: Python 3.10+. Save as test_summarise.py and run pytest -q, or run the file directly (it calls pytest on itself).

An endpoint whose LLM client is a dependency, tests that swap in a fake, and a parametrized edge case.

test_summarise.py

import re, subprocess, sys
import pytest
from fastapi import Depends, FastAPI
from fastapi.testclient import TestClient
 
# --- the app: the LLM client is a dependency, so tests can swap it for a fake ---
class LLMClient:
    def complete(self, prompt: str) -> str:
        raise RuntimeError("real API call: never in unit tests")
 
def get_llm() -> LLMClient:
    return LLMClient()
 
app = FastAPI()
@app.post("/summarise")
def summarise(body: dict, llm: LLMClient = Depends(get_llm)):
    text = body.get("text", "")
    if not text:
        return {"error": "empty text"}
    return {"summary": llm.complete(f"Summarise: {text}")[:100]}
 
# --- the tests ---
class FakeLLM:
    def __init__(self, reply="A short summary."):
        self.reply, self.prompts = reply, []
    def complete(self, prompt):
        self.prompts.append(prompt)
        return self.reply
 
@pytest.fixture
def client_and_llm():
    fake = FakeLLM()
    app.dependency_overrides[get_llm] = lambda: fake
    yield TestClient(app), fake
    app.dependency_overrides.clear()
 
def test_summarise_uses_llm(client_and_llm):
    client, fake = client_and_llm
    r = client.post("/summarise", json={"text": "Annual leave is 25 days."})
    assert r.status_code == 200 and r.json() == {"summary": "A short summary."}
    assert "Annual leave" in fake.prompts[0]                  # the prompt contains the input
 
def test_empty_text_never_calls_llm(client_and_llm):
    client, fake = client_and_llm
    assert client.post("/summarise", json={"text": ""}).json() == {"error": "empty text"}
    assert fake.prompts == []
 
@pytest.mark.parametrize("reply,length", [("x" * 500, 100), ("short", 5)])
def test_summary_is_truncated(client_and_llm, reply, length):
    client, fake = client_and_llm
    fake.reply = reply
    assert len(client.post("/summarise", json={"text": "t"}).json()["summary"]) == length
 
if __name__ == "__main__":
    out = subprocess.run([sys.executable, "-m", "pytest", "-q", "-p", "no:cacheprovider", __file__],
                         capture_output=True, text=True).stdout
    print(re.search(r"\d+ (passed|failed)", out).group(), "(and the real LLM was never called)")

Output

4 passed (and the real LLM was never called)

The parametrized test counts as two, hence four. The real LLMClient raises if called, so a missing override would fail loudly instead of quietly spending money. The capstone RAG app's tests follow this pattern with its mock provider (pytest -q in examples/rag-app).

A real example

Example

Tests that caught a prompt bug

A team refactored how their chat endpoint built prompts and accidentally dropped the retrieved documents from the context. Answers in manual testing still looked plausible (the model guessed). Their unit test with a fake LLM asserted that the prompt sent contained the retrieved chunk text and the citation instruction; it failed on the pull request, pointing straight at the line. Because the fake was injected, the whole suite of 140 tests ran in 4 seconds on every push without any API keys in CI.

The easy way

Start here

Put every external service behind a Depends function, write a FakeLLM that returns a fixed reply and records prompts, and write one test per endpoint for the happy path, a validation error and a provider error. Run pytest -q in CI.

When you grow: the production setup

Teams layer the test pyramid: fast unit tests with fakes, integration tests against real databases and queues in containers, contract tests on the OpenAPI schema, recorded-response tests for provider SDK integration, smoke tests on staging, load tests before launches, and evals plus red-team suites for model behaviour, all wired into CI with coverage tracking.

Trade-offs

What you gain

Fast, free, reliable tests on every change.
Error paths are easy to simulate with fakes.
Confidence to refactor prompts and pipelines.

What it costs you

Fakes can drift from real provider behaviour.
Unit tests don't measure answer quality.
Maintaining tests takes ongoing effort.

When not to use it

Don't use unit tests to judge model quality; that's what evals are for.

Common mistakes

Calling real LLM APIs in unit tests.
Asserting on exact model wording.
Only testing the happy path.
Global state shared between tests (forgetting to clear overrides).
A slow suite that developers skip.

Interview questions

Try answering out loud first, then open the model answer.

How do you test an API that calls an LLM?

Model answer

Inject the LLM client as a dependency and replace it with a fake in tests that returns controlled outputs (including malformed outputs and errors) and records prompts; use a test client to call endpoints; assert on status codes, response shapes, validation, the prompt and context sent, tool handling and error paths. Measure answer quality separately with evals against real models.

What are pytest fixtures and why use them?

Model answer

Functions that provide setup (and optional teardown via yield) to tests that request them by parameter name, such as a test client, fake services, temporary databases or files. They remove duplication, make dependencies explicit, and can be scoped per test, module or session.

Unit tests vs integration tests vs evals for an AI backend?

Model answer

Unit tests check your code's logic with fakes; fast and deterministic. Integration tests check real components working together (database, queue, sometimes recorded provider responses). Evals measure the AI system's output quality on representative cases with real models, with thresholds. All three run in CI at different frequencies.

Quiz

Pick an answer to see whether it’s right and why.

0 of 5 answered

Key terms

pytest: Python's most popular testing framework, with plain assert statements, fixtures and parametrised tests.
Unit test: A fast, automated test of one small piece of code in isolation, with external services replaced by fakes.
Test fixture: Reusable setup (and cleanup) for tests, such as a test client or a fake LLM, provided to tests that ask for it.
Dependency injection: Passing a component's dependencies (database, LLM client) in from outside instead of creating them inside, so they can be swapped, for example for fakes in tests.
Mock (fake): A stand-in object used in tests that imitates a real dependency, returning controlled results and recording how it was called.

Before and after

Read first