Area 08 // BackendLesson 128 / 14013 min read

Async and await

Async code lets one Python process handle many slow network operations at once, such as LLM calls, database queries and web requests, by switching to other work while each one waits, which makes it ideal for AI backends that spend most of their time waiting on APIs.

On this page

Think of it like this
The problem it solves
How it works, step by step
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

A waiter with ten tables doesn't stand at the kitchen door waiting for one dish. They take an order, hand it to the kitchen, and serve other tables while it cooks, coming back when it's ready. One waiter, many orders in progress. Async is that waiter; await is "I'll check back when the kitchen calls".

The problem it solves

An LLM call takes one to thirty seconds, and almost all of that is waiting for the network. If each request ties up a whole thread or process while it waits, a server handles only a few users at once, and agents that make several independent calls do them one after another. Async lets one process keep hundreds of calls in flight.

How it works, step by step

async def defines a coroutine: a function that can pause.
await pauses it until an awaited operation (network, timer, database) completes; meanwhile the event loop runs other ready coroutines.
Run things concurrently with asyncio.gather(...) or task groups: the total time is about the slowest call, not the sum (see the code).
Limit concurrency with asyncio.Semaphore(n) so you don't exceed provider rate limits or overload databases.
Use async clients: model SDKs offer async versions (e.g. AsyncAnthropic, AsyncOpenAI), plus httpx.AsyncClient and async database drivers.
Don't block the loop: time.sleep, requests, synchronous SDK calls or heavy CPU work inside async def freeze every other request. Wrap unavoidable blocking calls in asyncio.to_thread, and move CPU-heavy work to separate processes or background workers.
Timeouts and cancellation: asyncio.timeout(...) stops slow calls; when a user disconnects from a stream, cancel the upstream LLM call to save money.
In FastAPI: async def endpoints run on the event loop; plain def endpoints run in a thread pool, which is fine for blocking code.

See it in code

Needs: Nothing: plain Python 3 (asyncio). asyncio.sleep stands in for an LLM call.

Sequential vs concurrent vs concurrency-limited calls, and a blocking function moved to threads.

async_calls.py

import asyncio, time
 
async def call_llm(i: int) -> str:
    await asyncio.sleep(0.5)            # waiting on the network: the event loop runs other tasks meanwhile
    return f"answer {i}"
 
async def sequential():
    return [await call_llm(i) for i in range(5)]          # one after another
 
async def concurrent():
    return await asyncio.gather(*(call_llm(i) for i in range(5)))   # all at once
 
async def limited(max_at_once: int):
    sem = asyncio.Semaphore(max_at_once)                  # respect provider rate limits
    async def one(i):
        async with sem:
            return await call_llm(i)
    return await asyncio.gather(*(one(i) for i in range(5)))
 
for name, coro in [("sequential", sequential), ("gather", concurrent), ("gather, max 2", lambda: limited(2))]:
    start = time.perf_counter()
    results = asyncio.run(coro())
    print(f"{name:14} {len(results)} answers in {time.perf_counter() - start:.1f}s")
 
# A blocking call inside async code freezes everything: use asyncio.to_thread for blocking libraries.
async def blocking_mistake():
    start = time.perf_counter()
    await asyncio.gather(asyncio.to_thread(time.sleep, 0.5), asyncio.to_thread(time.sleep, 0.5))
    return time.perf_counter() - start
print(f"two blocking calls via to_thread: {asyncio.run(blocking_mistake()):.1f}s")

Output

sequential     5 answers in 2.5s
gather         5 answers in 0.5s
gather, max 2  5 answers in 1.5s
two blocking calls via to_thread: 0.5s

With a cap of 2, five calls run in three waves (2 + 2 + 1), so 1.5 s. With a real SDK the pattern is identical: await client.messages.create(...) on an AsyncAnthropic client.

A real example

Example

One sync call that slowed everyone

A team's FastAPI chatbot used async def endpoints but called the provider's synchronous SDK inside them. Under load, latency for every user grew with the number of active users: each call blocked the event loop for several seconds. Switching to the SDK's async client (and asyncio.gather for the three retrieval calls each request made) cut p95 latency from 14 s to 3 s on the same hardware, and a semaphore kept them inside the provider's concurrency limit.

The easy way

Start here

In FastAPI, use async def endpoints with async clients for LLMs, HTTP and databases. If a library is only synchronous, either make that endpoint a plain def (FastAPI runs it in a thread) or wrap the call in asyncio.to_thread.

When you grow: the production setup

Teams use async clients throughout, task groups and semaphores for bounded fan-out, timeouts on every external call, cancellation when clients disconnect, connection pooling, separate worker processes for CPU-heavy work, and monitoring of event-loop lag to catch accidental blocking.

Trade-offs

What you gain

Many concurrent I/O operations per process.
Parallel tool and retrieval calls cut latency.
Natural fit for streaming.

What it costs you

Easy to block the loop by accident.
Async and sync code don't mix freely; libraries must support it.
Harder debugging and stack traces.

When not to use it

CPU-bound work (embeddings on CPU, parsing big PDFs): use processes or background workers.

Common mistakes

Calling requests or sync SDKs inside async def.
time.sleep instead of await asyncio.sleep.
Unbounded gather over thousands of calls, triggering rate limits.
No timeouts, so one stuck call hangs a request forever.
Forgetting await, getting a coroutine object instead of a result.

Interview questions

Try answering out loud first, then open the model answer.

When does async improve performance, and when doesn't it?

Model answer

It improves throughput and latency for I/O-bound work such as network calls, database queries and file I/O, because waiting operations overlap on one event loop. It doesn't help CPU-bound work, which still runs one thing at a time per process; use multiple processes, workers or native libraries for that.

How would you call an LLM for 1,000 documents concurrently without breaking rate limits?

Model answer

Use an async client and asyncio tasks with a semaphore (or a worker pool reading from a queue) capping concurrency to fit the provider's limits, add timeouts and retries with backoff for 429 and 5xx, track tokens per minute, and collect results with error handling per document. For non-urgent work consider a batch API.

What happens if you block the event loop in an async web server?

Model answer

All coroutines on that loop stop making progress until the blocking call finishes, so every concurrent request stalls; latency spikes for everyone. Fix it by using async libraries, offloading blocking calls to threads with asyncio.to_thread, or using sync endpoints that run in a thread pool.

Quiz

Pick an answer to see whether it’s right and why.

0 of 5 answered

Key terms

Async (asynchronous code): Code that can pause while waiting for slow operations like network calls, letting the program work on other tasks meanwhile. In Python it uses async def and await.
Event loop: The scheduler at the heart of async Python that runs whichever task is ready while others wait for I/O.
Coroutine: A function defined with async def; calling it creates an object that runs when awaited and can pause at each await.
Concurrency: Making progress on several tasks in overlapping time periods, such as many LLM calls waiting on the network at once.
Semaphore: A counter that limits how many tasks can do something at the same time, for example at most 5 concurrent LLM calls.

Before and after

Read first

126 REST APIs with FastAPI