Area 05 // AI System DesignLesson 92 / 14018 min read

Case study: an AI coding assistant

An AI coding assistant has two very different workloads, instant autocomplete with a small fast model and short context, and chat or agent tasks with a strong model, repository context and tools that run in a sandbox, so the design separates them and controls latency, context, safety and cost for each.

On this page

Think of it like this
The problem it solves
How it works, step by step
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

A workshop with two helpers. One is a quick apprentice standing at your elbow who hands you the next tool before you ask: fast, simple, usually right, and fine to ignore (autocomplete). The other is a senior colleague you give a job to ("fix the broken drawer"), who reads the plans, tries a fix on a practice bench, checks it, and shows you the result for approval (the agent). You'd never make the senior wait at your elbow, or send the apprentice on a two-hour job.

The problem it solves

Developers expect suggestions as they type, answers about their own codebase, and increasingly agents that implement whole changes. The workloads pull in opposite directions: autocomplete needs very low latency at huge volume; agent tasks need deep context, strong models, tools and safety. A single design for both is either too slow, too expensive or unsafe.

How it works, step by step

Requirements: 5,000 developers; inline autocomplete with p95 under ~300 ms; chat about the codebase; agent tasks (fix a bug, add a test) producing reviewable diffs; code never leaves approved providers; no secrets in prompts.
Estimate (see the code): completions per day and peak QPS, cost per workload and per developer.
Autocomplete path:
- The IDE plugin waits for a short typing pause, cancels stale requests, and sends the code before and after the cursor plus snippets from open files.
- A small, fast model (hosted or self-served with vLLM) with short output and prefix caching; servers close to users (streaming and latency).
- Track acceptance rate per language and model version.
Context engine (shared): an index of the repository (symbols, file embeddings, keyword search), project rules files, recently edited files (context engineering); secret scanning before anything is sent.
Chat and agent path:
- A strong model with tool calling: search code, read files, edit files, run tests (coding agents).
- A harness with a loop, step and cost limits (stop conditions) and prompt caching for the large, stable repo context.
- Execution in a sandbox (container, limited network, no production credentials) (sandboxing); risky commands need approval (human in the loop).
- Output is a diff or pull request for human review, never a direct push to main.
Evals: a benchmark of real past tasks from your repos (tests must pass), acceptance-rate dashboards, and regression checks before model or prompt changes (agent evals).
Governance: per-team cost budgets, audit logs of agent actions (audit logging), data-retention settings with providers.

Architecture

Two paths sharing one context engine: fast autocomplete and a sandboxed agent.

See it in code

Needs: Nothing: plain Python 3. Example prices as of November 2026 (small $0.10/$0.50, mid-tier $2/$10 per 1M tokens).

Sizing both workloads for 5,000 developers.

coding_assistant_sizing.py

# Sizing an AI coding assistant: fast autocomplete + slower chat/agent requests.
devs = 5_000
completions_per_dev_hour, active_hours = 120, 6
agent_tasks_per_dev_day = 4
 
completions_per_day = devs * completions_per_dev_hour * active_hours
peak_qps = completions_per_day / (active_hours * 3600) * 2          # peak = 2x the working-hours average
print(f"autocomplete: {completions_per_day:,}/day, peak ~{peak_qps:.0f} QPS, needs p95 latency in the low hundreds of ms")
 
# Autocomplete: small model, short context, short output. Agent: big model, long context, many steps.
auto_cost = completions_per_day * (1_500 / 1e6 * 0.10 + 30 / 1e6 * 0.50)
agent_steps, agent_in, agent_out = 15, 25_000, 1_500                # per task (cached repo context helps)
agent_cost = devs * agent_tasks_per_dev_day * agent_steps * (agent_in / 1e6 * 2.00 * 0.4 + agent_out / 1e6 * 10.00)
print(f"autocomplete cost: ${auto_cost:,.0f}/day   agent cost: ${agent_cost:,.0f}/day")
print(f"per developer: ${(auto_cost + agent_cost) * 21 / devs:,.0f}/month (21 working days)")

Output

autocomplete: 3,600,000/day, peak ~333 QPS, needs p95 latency in the low hundreds of ms
autocomplete cost: $594/day   agent cost: $10,500/day
per developer: $47/month (21 working days)

Autocomplete is a latency problem; agents are a cost and safety problem. Agent steps dominate spend, so fewer steps (better tools and context) and prompt caching of the repo context matter more than anything on the autocomplete side.

A real example

Example

Rolling out an internal coding assistant

A software company with 1,200 engineers ran autocomplete on a self-hosted small code model behind vLLM in two regions (p95 210 ms, 31% acceptance), and used a hosted strong model for chat and agent tasks.

Agents ran in short-lived containers with a copy of the repo, no production credentials and network limited to the package mirror. Every task ended as a draft pull request with test results. A benchmark of 150 past bug fixes from their own repos was run before each model or prompt change; switching to a newer model raised the pass rate from 48% to 61%, and adding a "run the linter before finishing" step to the harness cut review comments by a fifth.

The easy way

Start here

Buy before you build: most teams start with an existing coding assistant product and configure it (approved models, project rules files, data-retention settings). Build your own only for special needs, starting with the agent harness and context engine, since those carry your team's knowledge.

When you grow: the production setup

Custom platforms add a repo-aware context engine with symbol graphs and embeddings, self-hosted completion models tuned on internal code, a gateway with per-team budgets, sandboxed agent runners with approval policies, PR-based output with CI checks, internal task benchmarks, acceptance and success dashboards, and secret scanning on every prompt.

Trade-offs

What you gain

Separate paths let each workload meet its latency and cost targets.
Sandboxes and PR review keep agent mistakes contained.
A shared context engine improves both completions and agents.

What it costs you

Two serving stacks and a context index to maintain.
Agent costs are high and grow with usage.
Code privacy needs careful provider agreements.

When not to use it

A small team without special needs: an off-the-shelf assistant is cheaper than building one.

Common mistakes

Using a large slow model for autocomplete.
Agents running commands outside a sandbox or with real credentials.
Sending files with secrets (like .env) in the context.
Agents pushing directly to main instead of opening reviewable changes.
Judging the assistant by demos instead of a benchmark of your own tasks.

Interview questions

Try answering out loud first, then open the model answer.

Design an AI coding assistant for 5,000 developers.

Model answer

Separate the workloads. Autocomplete: IDE plugin with debouncing and cancellation, small fast model with prefix caching near users, short output, acceptance-rate tracking; size around 3.6M completions a day and a few hundred QPS at peak. Chat and agents: a context engine over the repo (symbols, embeddings, rules), a strong model with tools in a harness with step and cost limits, sandboxed execution, approval for risky actions and diffs or PRs as output. Add evals on real tasks, secret scanning, audit logs, per-team budgets and cost estimates per developer.

How do you get relevant code context into the prompt?

Model answer

Combine the cursor neighbourhood and open files, symbols referenced near the cursor (definitions, call sites via a symbol index), embedding and keyword search over the repo for the task, recently edited files and project rules files, ranked and trimmed to a budget; for agents, let the model search and read files itself with tools rather than stuffing everything in up front.

How do you make coding agents safe?

Model answer

Run them in isolated sandboxes with least-privilege credentials and restricted network, treat repository text as untrusted input (prompt injection), require approval for risky commands, limit steps and spend, produce reviewable diffs or pull requests that go through CI, scan for secrets, and log every action for audit.

Quiz

Pick an answer to see whether it’s right and why.

0 of 5 answered

Key terms

Code autocomplete: Inline suggestions that complete code as a developer types, served by a small fast model within a few hundred milliseconds.
Code context: The parts of a codebase given to a coding model, such as the code around the cursor, open files, related symbols, search results and project rules.
Coding agent: An agent that reads, edits and runs code in a repository using file, search and shell tools, verifying its work with tests.
Sandbox: An isolated environment (container, VM, restricted process) where code or agents run with limited access, so mistakes and attacks stay contained.
Prompt caching: A provider feature that reuses the processed beginning of a prompt when it repeats exactly, making those tokens much cheaper and faster.
Agent evals: Tests that measure whether an agent completes realistic tasks correctly, safely and within budget, judged on final results and on the steps it took.

Before and after

Read first