Area 05 // AI System DesignLesson 85 / 14015 min read

Batch inference and async job pipelines

Work that doesn't need an instant answer (summarising a million documents, nightly classification, long agent tasks) should run as background jobs, either through a provider's batch API at a large discount or through your own queue of workers, instead of inside a user's web request.

On this page

Think of it like this
The problem it solves
How it works: batch inference
How it works: async job pipelines
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

A dry cleaner doesn't make you wait at the counter while they clean your suit. You hand it in, get a ticket, and come back later (or they text you). Big hotel laundry goes out in bulk overnight at a cheaper rate. Behind the counter, a few machines work through the pile, and items that can't be cleaned go to a special shelf for the manager to look at.

That's async jobs, batch processing, worker concurrency and a dead-letter queue.

The problem it solves

Some AI work takes far longer than a web request should: summarising thousands of documents, re-embedding a corpus, classifying a month of tickets, running a 10-minute agent task. Doing it inside an HTTP request leads to timeouts, servers tied up waiting, lost work when a connection drops, and rate-limit storms when everything runs at once. And paying real-time prices for work nobody is watching wastes money.

How it works: batch inference

Collect the requests (e.g. one per document) in the provider's format: usually a JSONL file or a list, each with your own ID (custom_id).
Submit a batch job to the batch API (as of November 2026 OpenAI, Anthropic and Google all offer one at roughly half the normal price).
Wait: results arrive within a stated window (often up to 24 hours, usually sooner).
Poll the job status, then download results.
Match results by your ID, not by order (they can arrive in any order).
Handle per-request errors: some succeed, some fail or expire; resubmit failures.
Self-hosted batch inference: run large offline jobs through a serving engine at maximum batch size for throughput (vLLM).

How it works: async job pipelines

Accept the request and create an async job: store it with status queued and return a job ID immediately (HTTP 202).
Put it on a queue (message queues): Redis, SQS, RabbitMQ, a database table, or a workflow engine.
Workers pull jobs, call the model (or agent), and save results, updating status (running, done, failed).
Control concurrency (number of workers, semaphores) to stay within provider rate limits.
Retry transient failures with backoff; after N attempts move the job to a dead-letter queue for inspection.
Make jobs idempotent (idempotency): same job ID → same result, no duplicate side effects (idempotency keys).
Tell the user: polling endpoint, webhook, email, or a live update over SSE/WebSockets.
Observe: queue depth, job age, failure rate, cost per job.

Return a ticket now, do the work in the background.

See it in code

Needs: Nothing: plain Python 3 (asyncio). The model call is a stub; one document always fails to show error handling.

A queue with four workers but at most three concurrent model calls, failures recorded instead of crashing, and batch-API savings.

async_jobs.py

import asyncio
 
async def summarise(doc_id: int) -> str:
    await asyncio.sleep(0.01 * (doc_id % 3 + 1))     # stand-in for an LLM call
    if doc_id == 7:
        raise TimeoutError("model timed out")
    return f"summary of doc {doc_id}"
 
async def worker(name, queue, results, limit):
    while True:
        doc_id = await queue.get()
        async with limit:                              # cap concurrent model calls (rate limits!)
            try:
                results[doc_id] = ("done", await summarise(doc_id))
            except Exception as e:
                results[doc_id] = ("failed", str(e))   # in production: retry later or dead-letter queue
        queue.task_done()
 
async def main():
    queue, results, limit = asyncio.Queue(), {}, asyncio.Semaphore(3)
    for doc_id in range(10):                           # the API returns a job ID immediately...
        queue.put_nowait(doc_id)
    workers = [asyncio.create_task(worker(f"w{i}", queue, results, limit)) for i in range(4)]
    await queue.join()                                 # ...and workers process the jobs in the background
    for w in workers:
        w.cancel()
    done = sum(1 for s, _ in results.values() if s == "done")
    print(f"{done} done, {len(results) - done} failed:", [k for k, (s, _) in results.items() if s == "failed"])
 
asyncio.run(main())
 
# BATCH INFERENCE: the same 1M summaries through a batch API at half price (example prices, Nov 2026).
per_request = 2_000 / 1e6 * 2.00 + 200 / 1e6 * 10.00
print(f"1M summaries: real-time ${per_request * 1e6:,.0f} vs batch API ${per_request * 1e6 * 0.5:,.0f}")

Output

9 done, 1 failed: [7]
1M summaries: real-time $6,000 vs batch API $3,000

With Anthropic's Message Batches API, submission is client.messages.batches.create(requests=[{"custom_id": "doc-7", "params": {...}}, ...]); poll client.messages.batches.retrieve(batch_id) until it has ended, then iterate client.messages.batches.results(batch_id) and match results by custom_id.

A real example

Example

Classifying a year of support tickets for half the price

A company wanted every support ticket from the past year (1.2 million) tagged with category, sentiment and product. A real-time loop would have taken days, hit rate limits constantly and cost about £9,000. They wrote a script that built batch files of 10,000 requests each with ticket IDs as custom_id, submitted them to a batch API overnight, and resubmitted the 0.3% that failed. Total cost: about £4,500, finished in under a day, with zero rate-limit drama.

For new tickets arriving during the day, they use an async job pipeline: the ticketing system posts a job, a small worker pool tags it within a minute, and results are written back via webhook.

The easy way

Start here

If nobody is waiting for the result, use the provider's batch API: write a JSONL file with one request per line and an ID, submit, and collect the results the next morning. For user-triggered long tasks, return a job ID and process in a background worker.

When you grow: the production setup

Production pipelines use durable queues or workflow engines (Celery, RQ, SQS + Lambda, Temporal), worker autoscaling on queue depth, concurrency and token-rate limits per provider, retries with backoff and dead-letter queues, idempotent jobs, progress tracking, partial results, cost tracking per job, and alerts on job age and failure rates (background jobs).

Trade-offs

What you gain

Batch APIs cut cost roughly in half for non-urgent work.
Async pipelines keep web servers responsive and survive crashes.
Concurrency control avoids rate-limit storms.

What it costs you

Results aren't instant (minutes to hours).
More infrastructure: queues, workers, job state, notifications.
Partial failures need handling and resubmission.

When not to use it

A user is actively waiting for a short answer: use a normal (streaming) request.

Common mistakes

Running long AI work inside an HTTP request.
Matching batch results by position instead of by ID.
Unlimited concurrency, causing 429 storms.
Retrying forever instead of dead-lettering poison jobs.
Non-idempotent jobs that double-send emails or double-charge on retry.

Interview questions

Try answering out loud first, then open the model answer.

Design a system to summarise a million documents with an LLM.

Model answer

Use a batch API if latency allows: shard documents into batch files with stable IDs, submit jobs, poll, download results and match by ID, resubmit failures, store summaries with model and prompt versions. If self-hosting, run an offline job on a serving engine at maximum batch size. Track progress, cost and failures; make the pipeline idempotent and resumable; validate a sample of outputs with evals.

How do you build an async job pipeline for long AI tasks?

Model answer

The API validates the request, creates a job record and returns a job ID (202). The job goes onto a durable queue; a pool of workers with bounded concurrency processes jobs, calling the model with timeouts and retries, updating status and saving results idempotently. Failed jobs go to a dead-letter queue after N attempts. Clients poll a status endpoint or receive webhooks or live updates. Monitor queue depth, job age, failure rates and cost.

When would you use a batch API instead of real-time calls?

Model answer

For large volumes of independent requests where results can wait minutes to hours: backfills, nightly classification, evaluation runs, embedding or summarising corpora, synthetic data generation. The discount and separate rate limits make it cheaper and smoother; it's unsuitable for interactive features.

Quiz

Pick an answer to see whether it’s right and why.

0 of 5 answered

Key terms

Batch API: A way to submit many requests at once for processing within hours, at a large discount (often about 50%).
Batch inference: Running a large set of model requests offline, together, for maximum throughput and lower cost, when nobody is waiting for the results right away.
Async job: Work that the API accepts, returns a job ID for, and finishes later in the background; the client polls or gets notified when it's done.
Message queue: A buffer where producers put tasks and workers take them off one by one, so work can be spread out, retried and processed at a safe pace. Examples include SQS, RabbitMQ and Redis streams.
Dead-letter queue (DLQ): A separate queue for jobs that keep failing after their retries, so they can be inspected instead of blocking others or retrying forever.
Idempotency: The property that doing the same operation twice has the same effect as doing it once, which makes retries safe (no double charges or duplicate emails).

Before and after

Read first