Area 08 // BackendLesson 134 / 14014 min read

Message queues and background jobs

A message queue holds tasks that one part of your system produces and another part processes later, so slow or failure-prone work (document ingestion, long agent runs, emails) moves out of web requests into background workers that retry failures and scale independently.

On this page

Think of it like this
The problem it solves
How it works: message queues
How it works: background jobs
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

A print shop's order tray: the person at the counter takes your order, puts it in the tray and gives you a ticket. Printers in the back take orders from the tray one by one. If the shop gets busy, they turn on another printer; if a job keeps jamming, it goes on the "problem jobs" shelf so it doesn't block everyone else. If a printer breaks mid-job, the order goes back in the tray.

The problem it solves

Some work is too slow, unreliable or bursty for a web request: parsing and embedding a 300-page PDF, a 10-minute agent task, sending 5,000 emails, re-indexing after a model change. Doing it inline means timeouts, lost work when a server restarts, and web servers clogged during spikes. You need a buffer between "work requested" and "work done".

How it works: message queues

Producers (your API, a scheduler, a webhook) put messages (jobs) on a queue.
The queue stores them durably until a consumer takes them; messages can be ordered, delayed or prioritised.
Consumers (workers) take messages, process them and acknowledge them; unacknowledged messages become visible again after a timeout, so a crashed worker's job is retried.
Delivery is usually at-least-once, so duplicates are possible: make processing idempotent (idempotency).
Options: Redis lists or streams (simple, fast), RabbitMQ (flexible routing), cloud queues like SQS (managed), Kafka (high-volume event streams that many consumers can replay), workflow engines like Temporal (long, multi-step, durable workflows).
Pub/sub variants send one event to many subscribers (e.g. "document uploaded" → indexer, virus scanner, notifier).

How it works: background jobs

The API validates the request, creates a job record (status queued), enqueues it and returns 202 with a job ID (async jobs).
Workers run separately from web servers (their own processes or containers), often with a library: Celery, RQ, arq, Dramatiq, or cloud functions triggered by the queue.
Status and results go to a database or cache; clients poll GET /jobs/{id}, get a webhook, or receive live updates over SSE.
Retries: transient errors (429, timeouts) retry with exponential backoff; permanent errors fail immediately (retries).
Dead-letter queue: after N attempts the job moves aside for inspection (see the code).
Concurrency and scaling: cap concurrent LLM calls per worker to respect provider rate limits; autoscale workers on queue depth (autoscaling).
Observe: queue depth, oldest job age, failure rate, processing time per job type.
Scheduled jobs (nightly re-index, daily eval runs) use a scheduler that enqueues jobs on a timetable.

The API stays fast; workers process jobs, retry failures and park poison jobs.

See it in code

Install: pip install fakeredis
Needs: Python 3.10+. fakeredis stands in for Redis; production setups use a library like RQ, Celery or arq.

A Redis-list queue with a worker, retries, and a dead-letter queue for a file that can never be parsed.

queue_worker.py

import json
import fakeredis                      # stand-in for Redis; the same idea works with SQS, RabbitMQ, Celery or RQ
 
r = fakeredis.FakeRedis(decode_responses=True)
MAX_ATTEMPTS = 3
 
def enqueue(job: dict):
    r.lpush("jobs", json.dumps(job))               # producer: the API adds a job and returns at once
 
def summarise(doc: str) -> str:
    if doc == "corrupt.pdf":
        raise ValueError("cannot parse file")
    return f"summary of {doc}"
 
def worker():
    while (raw := r.rpop("jobs")) is not None:     # consumer: takes jobs one by one (in order)
        job = json.loads(raw)
        try:
            r.hset("results", job["id"], summarise(job["doc"]))
            r.hset("status", job["id"], "done")
        except Exception as e:
            job["attempts"] = job.get("attempts", 0) + 1
            if job["attempts"] < MAX_ATTEMPTS:
                r.lpush("jobs", json.dumps(job))   # retry later (real systems add a backoff delay)
            else:
                r.lpush("dead-letter", json.dumps({**job, "error": str(e)}))
                r.hset("status", job["id"], "failed")
 
for i, doc in enumerate(["policy.pdf", "corrupt.pdf", "handbook.pdf"], start=1):
    enqueue({"id": f"job-{i}", "doc": doc})
print("queued:", r.llen("jobs"), "jobs")
worker()
print("status:", r.hgetall("status"))
print("results:", r.hgetall("results"))
print("dead letters:", [json.loads(x) for x in r.lrange("dead-letter", 0, -1)])

Output

queued: 3 jobs
status: {'job-1': 'done', 'job-3': 'done', 'job-2': 'failed'}
results: {'job-1': 'summary of policy.pdf', 'job-3': 'summary of handbook.pdf'}
dead letters: [{'id': 'job-2', 'doc': 'corrupt.pdf', 'attempts': 3, 'error': 'cannot parse file'}]

The corrupt file was tried three times and parked, while the other two jobs finished. A real "cannot parse" error is permanent, so production code would dead-letter it on the first failure and only retry transient errors.

A real example

Example

Uploads that stopped timing out

A document Q&A product parsed, chunked and embedded uploads inside the upload request. Large PDFs took 90 seconds and the load balancer cut them off at 60, so users retried and created duplicates. The team switched to: upload to object storage (file uploads) → 202 with a job ID → an ingestion queue processed by autoscaled workers capped at 20 concurrent embedding calls → status updates streamed to the UI. Jobs became idempotent on the file's hash, so retries no longer duplicated chunks. Upload errors dropped to near zero, and a spike of 3,000 uploads simply grew the queue for 15 minutes instead of taking the API down.

The easy way

Start here

Use a simple Redis-backed job library (RQ or arq) or your cloud's managed queue: enqueue from the API, run one worker process, store status in your database, and add retries with a maximum attempt count.

When you grow: the production setup

Teams use durable managed queues or workflow engines, idempotent job handlers, separate queues per priority and job type, backoff and dead-letter queues with alerting, worker autoscaling on queue depth, concurrency limits per provider, job tracing across API and worker, scheduled jobs, and replay tools for dead letters.

Trade-offs

What you gain

Web requests stay fast; slow work survives crashes.
Bursts are absorbed by the queue.
Workers scale and fail independently.

What it costs you

More components to run and monitor.
Results are delayed; users need status updates.
Duplicates and ordering need careful handling.

When not to use it

Work that takes well under a second and must return a result immediately.

Common mistakes

Long work inside HTTP requests.
Non-idempotent jobs under at-least-once delivery.
Retrying permanent errors forever, with no dead-letter queue.
Unlimited worker concurrency that floods the LLM provider.
No visibility into queue depth or job age.

Interview questions

Try answering out loud first, then open the model answer.

When would you use a message queue in an AI system?

Model answer

For slow, bursty or failure-prone work that shouldn't block requests: document ingestion and embedding, long agent runs, batch summarisation, notifications, re-indexing and scheduled evals. Queues decouple producers from consumers, absorb spikes, enable retries and let workers scale independently.

How do you make background jobs reliable?

Model answer

Durable queues with acknowledgements; idempotent handlers (keyed by job ID or content hash); retries with exponential backoff for transient errors only; dead-letter queues with alerts; timeouts; job status stored in a database; concurrency limits; monitoring of depth, age and failure rate; and graceful shutdown so workers finish or return in-flight jobs.

Queue vs event stream (e.g. SQS vs Kafka)?

Model answer

A queue distributes each message to one consumer and deletes it after processing: ideal for work distribution. An event stream keeps an ordered, replayable log that many consumer groups read independently: ideal for event-driven architectures, analytics and rebuilding state. Workflow engines add durable multi-step orchestration on top.

Quiz

Pick an answer to see whether it’s right and why.

0 of 5 answered

Key terms

Message queue: A buffer where producers put tasks and workers take them off one by one, so work can be spread out, retried and processed at a safe pace. Examples include SQS, RabbitMQ and Redis streams.
Background job: Work that runs outside the web request, in a worker process, so the API can respond quickly.
Worker: A process that takes jobs from a queue and does them, separately from the web servers.
Dead-letter queue (DLQ): A separate queue for jobs that keep failing after their retries, so they can be inspected instead of blocking others or retrying forever.
At-least-once delivery: A queue guarantee that every message is delivered, but possibly more than once, so handlers must be idempotent.

Before and after

Read first