Area 01 // LLMsLesson 15 / 14016 min read

The transformer architecture

A transformer is a stack of identical blocks, each letting tokens share information through attention and then think individually through a small neural network, and an LLM is a big decoder-only transformer that ends by scoring every possible next token.

On this page

Think of it like this
The problem it solves
How it works, step by step
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

Picture a team of editors improving a draft together, in rounds. In each round:

Meeting (attention): everyone reads everyone else's notes and borrows what's useful.
Desk work (feed-forward): everyone goes back to their own desk and refines their own note.

After 30 or 80 rounds, each note (each token) is rich with context. Finally, the last editor looks at the note at the end of the document and lists every possible next word with a score. That whole process is a transformer.

The problem it solves

You now know the pieces: tokens (lesson 12), embeddings and positions (lesson 13), attention (lesson 14), neural network layers (lesson 3). On their own they don't make a language model. The transformer is the recipe that arranges them so the result can be trained on trillions of tokens and still be stable, fast on GPUs, and able to scale from millions to trillions of parameters.

Knowing the recipe tells you where cost, speed and limits come from, which is what an AI engineer needs.

How it works, step by step

Tokenize the text into token IDs.
Embed each ID into a vector and add position information. You now have a grid: tokens × model dimension (e.g. 4,096 numbers per token).
Pass the grid through N identical transformer blocks (from about 30 to over 100 in big models). Each block:
1. Normalises each token's vector (layer norm or RMSNorm) to keep numbers stable.
2. Multi-head self-attention: tokens share information (with the causal mask).
3. Adds the result back to the input (a residual connection), so the block learns a correction instead of starting over.
4. Normalises again, then runs a feed-forward network on each token separately: expand to ~4× the size, apply an activation, shrink back. This is where most parameters live.
5. Adds that back too (another residual connection).
Final normalisation, then multiply by an output matrix to get logits: one score for every token in the vocabulary, at every position.
Use the last position's logits to choose the next token (decoding), append it, and repeat.

Architecture

A decoder-only transformer, the design behind GPT, Claude, Gemini and Llama. The middle box repeats N times.

Three families of transformer

Type	Attention	Typical use	Examples
Decoder-only	Causal (only earlier tokens)	Generating text: chat, code	GPT, Claude, Gemini, Llama, Mistral
Encoder-only	Bidirectional (all tokens)	Understanding: classification, embeddings, reranking	BERT and its descendants
Encoder-decoder	Encoder reads all; decoder generates	Translation, summarisation	T5, the original 2017 transformer

What the size numbers mean

When a model card says "32 layers, 4,096 dimension, 32 heads, 128k context, 8B parameters", those are the knobs: how many blocks, how wide each token's vector is, how many attention heads, how many positions it was trained for, and how many learned numbers in total.

See it in code

Install: pip install numpy
Needs: Python 3.10+

A complete (tiny, untrained) decoder-only transformer forward pass. Watch the shape stay the same through every block until the final projection to vocabulary scores.

tiny_transformer.py

import numpy as np
 
rng = np.random.default_rng(0)
n_tokens, d, vocab_size = 5, 16, 50
 
def layer_norm(x):
    return (x - x.mean(-1, keepdims=True)) / (x.std(-1, keepdims=True) + 1e-5)
 
def attention(x, Wq, Wk, Wv):
    q, k, v = x @ Wq, x @ Wk, x @ Wv
    s = q @ k.T / np.sqrt(x.shape[-1]) + np.triu(np.full((len(x),) * 2, -1e9), 1)
    w = np.exp(s - s.max(-1, keepdims=True)); w /= w.sum(-1, keepdims=True)
    return w @ v
 
def block(x, p):
    x = x + attention(layer_norm(x), *p["attn"])          # 1. tokens share information
    h = np.maximum(0, layer_norm(x) @ p["W1"])             # 2. feed-forward: think per token
    return x + h @ p["W2"]                                 #    (+ residual connections)
 
params = [{"attn": [rng.normal(0, .1, (d, d)) for _ in range(3)],
           "W1": rng.normal(0, .1, (d, 4 * d)), "W2": rng.normal(0, .1, (4 * d, d))}
          for _ in range(4)]                               # 4 stacked blocks
W_out = rng.normal(0, .1, (d, vocab_size))                 # vectors -> vocabulary scores
 
x = rng.normal(size=(n_tokens, d))                         # embeddings + positions
print("input            :", x.shape)
for i, p in enumerate(params):
    x = block(x, p)
    print(f"after block {i + 1}    :", x.shape)
logits = layer_norm(x) @ W_out
print("logits           :", logits.shape, "-> one score per vocab word, per position")
print("next-token guess : id", int(logits[-1].argmax()))

Output

input            : (5, 16)
after block 1    : (5, 16)
after block 2    : (5, 16)
after block 3    : (5, 16)
after block 4    : (5, 16)
logits           : (5, 50) -> one score per vocab word, per position
next-token guess : id 41

The weights are random, so the guess (id 41) is meaningless; training is what makes it good. The structure, though, is the real thing, just with 4 blocks of width 16 instead of 80 blocks of width 8,192.

A real example

Example

Why a bigger model is slower and pricier per token

A team compares two open models for an internal assistant: an 8B-parameter model (32 layers) and a 70B model (80 layers, much wider). Every generated token must flow through every layer, touching nearly all weights. So the 70B model needs roughly 9× the compute and memory per token: more GPUs, lower tokens per second, higher cost.

In their evals, the 8B model fine-tuned on their documents scored within a few points of the 70B one on 90% of questions. They used the small model by default and routed hard questions to the big one (model routing). Understanding the architecture made that trade-off obvious.

The easy way

Start here

Memorise the pipeline in one line: tokens → embeddings + positions → (attention + feed-forward) × N → logits → pick next token → repeat. When you read any model card, find the four numbers: layers, dimension, parameters, context length.

When you grow: the production setup

Production LLMs add refinements on the same skeleton: RMSNorm instead of LayerNorm, SwiGLU activations, rotary positions, grouped-query attention, and often mixture-of-experts feed-forward layers. Serving systems (vLLM) are built around this structure: caching keys and values per layer and batching many requests through the same blocks.

Trade-offs

What you gain

Highly parallel, so it trains efficiently on GPUs at enormous scale.
One simple repeated block scales smoothly: more layers and width, more capability.
The same design works for text, code, images, audio and video.

What it costs you

Attention cost grows with the square of context length.
Generation is sequential: one token at a time through every layer.
Huge memory for weights and the KV cache at serving time.

When not to use it

For small, structured prediction problems, a transformer is overkill; classic ML is cheaper and often better.

Common mistakes

Thinking the model writes a whole answer at once. It produces one token per full pass.
Confusing "encoder" models (great for embeddings and classification) with generative chat models.
Assuming more layers is always better without checking latency and cost.
Thinking the transformer "looks up" facts in a database; knowledge is spread through its weights.

Interview questions

Try answering out loud first, then open the model answer.

Walk through a forward pass of a decoder-only transformer.

Model answer

Tokenize the input, map IDs to embeddings and inject position (e.g. RoPE inside attention). For each of N blocks: pre-normalise, apply masked multi-head self-attention, add the residual; pre-normalise again, apply a position-wise feed-forward network (expand, non-linearity, project back), add the residual. After the last block, apply a final norm and project to vocabulary logits. The logits at the last position give the next-token distribution; generation repeats this autoregressively, with a KV cache to avoid recomputing earlier tokens.

What's the difference between encoder-only, decoder-only and encoder-decoder transformers?

Model answer

Encoder-only models use bidirectional attention over the full input to build representations, suited to classification, retrieval embeddings and reranking (BERT). Decoder-only models use causal attention and are trained to predict the next token, suited to generation (GPT, Claude, Llama). Encoder-decoder models encode the input bidirectionally and generate output with a decoder that cross-attends to the encoding, suited to sequence-to-sequence tasks like translation (T5).

Why are residual connections and normalisation important in transformers?

Model answer

Residual connections give gradients a direct path through deep stacks, preventing vanishing gradients and letting each block learn an incremental update. Normalisation (LayerNorm or RMSNorm, usually applied before each sub-layer) keeps activations on a stable scale, which makes training of very deep, very large models stable and less sensitive to learning-rate choices.

Quiz

Pick an answer to see whether it’s right and why.

0 of 6 answered

Key terms

Transformer: The neural network architecture behind modern LLMs, built from stacked blocks of attention and feed-forward layers.
Transformer block: One repeated unit of a transformer, containing attention and a feed-forward network, each wrapped with normalisation and a residual connection.
Feed-forward network (MLP): The part of each transformer block that processes every token's vector on its own; holds most of the model's parameters.
Residual connection: Adding a layer's input to its output, so information and gradients flow easily through very deep networks.
Layer normalisation: Rescaling each token's vector to a standard range inside the model, which keeps training stable. RMSNorm is a common variant.
Decoder-only model: A transformer that uses causal attention and generates text left to right. GPT, Claude, Gemini and Llama are decoder-only.
Logits: The raw scores a model outputs for every possible next token, before softmax turns them into probabilities.

Before and after

Read first

014 Self-attention