Serving models with vLLM

How it works: inference serving with vLLM

1. Install and serve: on a Linux machine with a GPU, pip install vllm, then vllm serve <model-id> (e.g. a Hugging Face model). It starts an inference server on port 8000 by default.
2. OpenAI-compatible API: endpoints such as /v1/chat/completions (and an Anthropic Messages-compatible API and gRPC as of November 2026), so existing clients just change base_url.
3. Core techniques: continuous batching and PagedAttention (below), plus prefix caching, chunked prefill and speculative decoding.
4. Fit bigger models: quantized weights (FP8, INT8, INT4, GPTQ/AWQ, GGUF) and parallelism across GPUs (tensor, pipeline, expert, data, context parallelism).
5. Serve many adapters: multiple LoRA adapters on one base model (LoRA).
6. Tune: max model length, GPU memory fraction, max concurrent sequences, quantization, and parallel size, then benchmark throughput and latency on your workload.
7. Operate: run in containers (Docker) on GPU instances or Kubernetes, with metrics (Prometheus) for queue length, KV usage, tokens per second.

How it works: continuous batching

1. Static batching groups N requests, runs them together until all finish, then starts the next batch. Short requests wait for the longest one; slots sit idle.
2. Continuous batching (iteration-level scheduling) re-forms the batch at every decode step: finished sequences leave immediately and waiting requests join.
3. Result: the GPU stays full, waiting time drops, and throughput rises dramatically for workloads with mixed output lengths.
4. Chunked prefill splits long prompts into pieces so a big prefill doesn't stall everyone else's decoding.

How it works: PagedAttention

1. The problem: each request's KV cache grows token by token. Naive servers reserve a contiguous block big enough for the maximum context per request, wasting most of it.
2. PagedAttention splits the KV cache into small fixed-size blocks (pages, e.g. 16 tokens) and allocates them on demand, with a block table per request mapping logical positions to physical blocks, like an operating system's virtual memory.
3. Almost no waste: only the last block of each request is partly empty.
4. More concurrency: the saved memory holds more requests at once, enabling bigger batches.
5. Sharing: requests with the same prefix (system prompt, documents) can point at the same physical blocks, which is how prefix caching works.

# Requests that need different numbers of output tokens (decode steps).
lengths = [5, 40, 8, 6, 35, 7, 9, 4]
SLOTS = 4                                              # how many sequences fit on the GPU at once
 
def static_batching(lengths):
    """Old way: start a batch, wait until its LONGEST request finishes, then start the next."""
    steps = 0
    for i in range(0, len(lengths), SLOTS):
        steps += max(lengths[i:i + SLOTS])
    return steps
 
def continuous_batching(lengths):
    """vLLM way: every step, finished requests leave and waiting ones take their slot."""
    waiting, running, steps = list(lengths), [], 0
    while waiting or running:
        while waiting and len(running) < SLOTS:
            running.append(waiting.pop(0))
        running = [r - 1 for r in running if r > 1]      # one decode step for everyone
        steps += 1
    return steps
 
print("static batching    :", static_batching(lengths), "steps")
print("continuous batching:", continuous_batching(lengths), "steps")
 
# PagedAttention: KV cache in fixed-size blocks instead of one big reserved region per request.
BLOCK = 16
reserved_naive = len(lengths) * 2048                   # reserve max context for every request
used_paged = sum(-(-(100 + n) // BLOCK) * BLOCK for n in lengths)   # 100 prompt tokens + output
print(f"KV slots reserved: naive {reserved_naive:,} vs paged {used_paged:,}")

static batching    : 75 steps
continuous batching: 40 steps
KV slots reserved: naive 16,384 vs paged 960

pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct --max-model-len 16384   # OpenAI-compatible API on :8000

from openai import OpenAI
 
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")  # your vLLM server
r = client.chat.completions.create(model="Qwen/Qwen2.5-7B-Instruct",
                                   messages=[{"role": "user", "content": "Say hello in French."}])
print(r.choices[0].message.content)