Cheat sheet · Area 01

LLMs

Tokenization

Before a language model can read text, a tokenizer chops it into tokens (whole words, pieces of words or symbols) and turns each one into a number, and those tokens are what models read, write, limit and charge for.

Token ≈ ¾ of an English word on average (about 4 characters). 1,000 tokens ≈ 750 words.
Tokenizer: text → token IDs (numbers). Each model family has its own tokenizer and vocabulary.
BPE: start from characters/bytes, repeatedly merge the most common pairs into new tokens.
You pay per token, limits are in tokens, speed is tokens per second.
Non-English text, code, numbers and JSON often use more tokens per character.
Count tokens with the provider's tool (tiktoken for OpenAI, count_tokens API for Claude).

Embeddings and positional encoding

Inside an LLM, every token ID is swapped for a learned vector (its embedding) that captures meaning, and position information is added so the model knows the order of the words.

Embedding table: one learned vector per token in the vocabulary; lookup by token ID.
Similar meanings end up with similar vectors because training rewards it.
Attention alone ignores order, so position information is added (sinusoidal, learned, or rotary/RoPE).
Input to the transformer = token embedding + position information, one vector per token.

Self-attention

Self-attention lets every token look at the other tokens in the text and pull in information from the ones that matter to it, which is how a model works out that "it" refers to "the animal" and not "the street".

Each token makes a query (what I'm looking for), a key (what I contain) and a value (what I'll share).
Score = query · key, scaled by √d; softmax turns scores into attention weights; output = weighted sum of values.
Causal mask: a token can only look at earlier tokens (no peeking at the future).
Multi-head: many attention heads in parallel, each learning different relationships.
Cost grows with the square of sequence length, which is why long contexts are expensive.

The transformer architecture

A transformer is a stack of identical blocks, each letting tokens share information through attention and then think individually through a small neural network, and an LLM is a big decoder-only transformer that ends by scoring every possible next token.

Pipeline: tokens → embeddings + positions → N transformer blocks → final norm → logits over the vocabulary.
Block = attention (tokens talk to each other) + feed-forward MLP (each token thinks alone), each with a residual connection and normalisation.
Residual connection: add the block's input back to its output, so information and gradients flow through deep stacks.
GPT-style LLMs are decoder-only (causal); BERT-style models are encoder-only; T5-style are encoder-decoder.
Size knobs: number of layers, model dimension, number of heads, vocabulary, context length.

Mixture of experts

A mixture-of-experts model replaces one big feed-forward network in each layer with many smaller "expert" networks plus a router that sends each token to only a few of them, so the model can store far more knowledge while doing much less work per token.

MoE layer = router + many experts; each token uses only the top-k experts (often 1–8 of dozens or hundreds).
Total parameters (stored) ≫ active parameters (used per token). Speed follows active; memory follows total.
Many current frontier and open models are MoE (e.g. Llama 4, Mistral Large 4, DeepSeek).
Trade-off: cheaper per token, but you still need memory for ALL experts.

Logits, decoding, temperature and top-p

At every step an LLM outputs a score (logit) for every possible next token, and a decoding strategy (greedy, beam, top-k, top-p, with a temperature) decides which token actually gets written, which controls how predictable or creative the output is.

Logits → (÷ temperature) → softmax → probabilities → decoding picks one token.
Greedy: always the top token. Fast, repetitive. Beam: keep the best few paths, pick the best whole sequence.
Top-k: sample from the k most likely tokens. Top-p (nucleus): sample from the smallest set whose probabilities add to p.
Temperature < 1: focused. > 1: varied. Change temperature OR top-p, rarely both.
As of Nov 2026, some newest models (e.g. Claude via API version 2026-10-21) no longer accept temperature/top_p/top_k.

Context windows and the KV cache

The context window is the maximum number of tokens a model can look at in one request (your input plus its output), and the KV cache is the memory of already-processed tokens that lets it generate each new token quickly instead of re-reading everything.

Context window = input tokens + output tokens for one request (e.g. 128K–1M+ as of Nov 2026).
The model remembers nothing between requests; your app resends the conversation each time.
Bigger context ≠ better answers: cost, latency and 'lost in the middle' all grow.
KV cache: store each token's keys and values once, so each new token only does new work.
Prefill (read the prompt, compute-heavy) vs decode (write tokens one by one, memory-heavy).
KV cache memory grows with context length × concurrent users; it often limits how many users a GPU serves.

Pretraining and post-training (SFT, RLHF, DPO)

An LLM is made in two stages, pretraining on huge amounts of raw text so it learns language and facts by predicting the next token, then post-training (supervised fine-tuning and preference training like RLHF or DPO) so it follows instructions, is helpful and refuses harmful requests.

Pretraining: predict the next token on trillions of tokens → a base model that continues text.
SFT: train on example conversations (prompt → ideal answer) → it follows instructions.
RLHF: humans rank answers → reward model → reinforcement learning towards preferred answers.
DPO: learn directly from chosen-vs-rejected pairs, no separate reward model; simpler and cheaper.
Reasoning models add RL on checkable tasks (maths, code) to learn to think step by step.
Base models complete text; chat/instruct models answer. Know which one you're using.

Prompt engineering basics and system prompts

Prompt engineering is writing instructions a model can't misunderstand, with a clear role, task, context, constraints and output format, and the system prompt is the standing set of instructions that frames every conversation.

A good prompt: role/context, the task, the inputs (clearly delimited), constraints, the output format, and what to do when unsure.
System prompt = standing instructions for the whole conversation; user messages = the specific request.
Be specific and positive ('reply in 3 bullet points') rather than vague ('be concise').
Wrap inputs in delimiters (XML tags, triple quotes) so data can't be confused with instructions.
Keep prompts in templates under version control and test changes with evals.

Few-shot prompting and chain of thought

Few-shot prompting means showing the model a few worked examples of the task so it copies the pattern, and chain of thought means asking it to reason step by step before answering, which helps on multi-step problems.

Zero-shot: instructions only. Few-shot: instructions + 2–8 worked examples.
Examples teach format and judgement better than descriptions; make them varied and representative.
Chain of thought: 'think step by step, then give the final answer in a marked line', and parse only that line.
Reasoning models think internally; for them, don't micro-manage the steps, just state the goal clearly.
CoT costs extra output tokens and latency; use it where accuracy on multi-step problems matters.

Reasoning models

Reasoning models are LLMs trained to think through a problem internally (planning, trying approaches, checking their work) before answering, which makes them much better at maths, code and multi-step tasks at the cost of more tokens and more time.

Reasoning model = thinks (hidden reasoning tokens) before answering; trained with RL on checkable tasks.
You pay for reasoning tokens as output tokens, even if you can't see them.
Control depth with an effort setting (e.g. OpenAI reasoning.effort, Claude output_config.effort).
Great for maths, code, planning, agents, complex analysis. Overkill for simple lookups and chat.
Prompt them with goals and constraints, not step-by-step scripts.

Structured outputs

Structured outputs make a model reply in an exact machine-readable shape (usually JSON matching a schema you define), so your code can use the answer directly instead of trying to parse free text.

Define the shape once (a Pydantic model or JSON Schema); the API constrains the model to it.
OpenAI: client.responses.parse(..., text_format=Model) → response.output_parsed.
Anthropic: client.messages.parse(..., output_format=Model) → response.parsed_output.
Constrained decoding guarantees the SHAPE, not the TRUTH: still validate values and business rules.
Keep schemas small and flat; use enums for fixed choices; allow null for 'not found'.

Function and tool calling basics

Tool calling lets a model ask your code to run a function (look up an order, search a database, send an email) by replying with the function's name and arguments; your app runs it and sends the result back so the model can finish its answer.

You describe tools (name, description, JSON Schema for arguments); the model decides if and when to call them.
The model never runs code itself: it returns a call request; YOUR app runs the function and returns the result.
OpenAI: output item type function_call → reply with function_call_output + call_id.
Anthropic: content block type tool_use (stop_reason tool_use) → reply with a tool_result + tool_use_id.
Validate arguments, check permissions, handle errors; treat tool calls like untrusted user input.

Prompt caching and token pricing

LLM APIs charge per million tokens, with output costing several times more than input, and prompt caching lets the provider reuse the work for a repeated beginning of your prompt, cutting the price and latency of those tokens dramatically.

Cost = input_tokens × input price + output_tokens × output price (prices per 1M tokens).
Output is usually 4–5× the input price. Reasoning tokens count as output.
Prompt caching: identical prefix (system prompt, tools, documents) is reused; cached reads cost ~10% of input.
Caching is a prefix match: put stable content first, anything that changes last.
OpenAI caches long prefixes automatically; Anthropic uses cache_control (explicit or automatic). Writes may cost a bit more.
Batch APIs give ~50% off for non-urgent work.

Hallucinations

A hallucination is when a model states something false or made up with full confidence, because it generates likely-sounding text rather than looking facts up; you reduce it by grounding answers in real sources, allowing "I don't know", and checking outputs.

Models predict plausible text, not verified facts, so fluent ≠ true.
Main causes: missing or outdated knowledge, vague prompts, pressure to answer, long noisy context.
Fixes: ground in retrieved sources (RAG), allow 'I don't know', ask for citations and verify them, use tools for facts and maths.
Measure it: faithfulness evals, citation checks, user reports.
You can reduce hallucinations a lot; you can't get them to zero. Design for that.

Multimodal inputs

Multimodal models can take images, PDFs, audio and sometimes video as input alongside text, by turning each into tokens the model can attend to, which lets you build features like reading receipts, describing screenshots or answering questions about charts.

Multimodal = more than text in (images, PDFs, audio, video) and sometimes out.
Images are cut into patches, turned into vectors ('image tokens') and processed with the text.
Send images as URLs or base64 in a content block, image first, then the question.
Images cost tokens too, roughly by resolution: resize to what the task needs.
Great for screenshots, documents, charts, photos; still weak at tiny text, exact counting and precise positions.

Fine-tuning with LoRA and QLoRA

Fine-tuning adapts an existing model to your task with your own examples; LoRA does it cheaply by freezing the original weights and training tiny add-on matrices, and QLoRA goes further by loading the frozen model in 4-bit so big models can be tuned on a single GPU.

Fine-tuning = more training on your examples. Full fine-tuning updates every weight (expensive).
LoRA: freeze W, learn a low-rank update B·A (rank r ≈ 8–64). Often <1% of parameters trained.
QLoRA: base model loaded in 4-bit + LoRA adapters in higher precision → fine-tune big models on one GPU.
Adapters are small files you can swap per customer or task on the same base model.
Fine-tune for behaviour, format, style and narrow skills; use RAG for facts that change.

Quantization and distillation

Quantization shrinks a model by storing its weights with fewer bits (16 → 8 → 4), and distillation trains a small "student" model to copy a big "teacher" model; both make models cheaper and faster to run with only a small loss in quality.

Memory for weights ≈ parameters × bits ÷ 8. 8B model: 16 GB at 16-bit, 8 GB at 8-bit, ~4–5 GB at 4-bit.
Quantization: fewer bits per weight → less memory, often faster; small quality loss (bigger at 3-bit and below).
Formats: GGUF (llama.cpp / Ollama), AWQ and GPTQ (GPU serving), FP8 (modern data-centre GPUs).
Distillation: a big teacher's outputs train a small student → most of the quality, a fraction of the cost.
Always re-run evals after quantizing or distilling.

Open vs closed models, small vs large

Closed models are used only through their maker's API and usually lead on quality, open-weights models can be downloaded and run anywhere you like, and small models are cheaper and faster while large ones are smarter; most products end up mixing them.

Closed: best quality, zero ops, pay per token, data goes to the provider, behaviour can change with versions.
Open weights: full control, privacy, fine-tune freely, fixed GPU cost; you run and secure it. Read the licence.
Small models: fast, cheap, can run on-device; weaker at hard reasoning and broad knowledge.
Large models: best at hard, open-ended tasks; slower and pricier.
Default plan: prototype with a strong closed model, then move each route to the cheapest model that passes evals.

Benchmarks and model leaderboards

Benchmarks are standard test sets that score models on skills like knowledge, maths or coding, and leaderboards rank models by those scores or by human votes; they're useful for a shortlist, but only your own evals tell you which model is best for your product.

Benchmark = fixed test set + scoring rule (e.g. % correct on maths problems, % of coding tasks passed).
Leaderboards rank by benchmark scores or by human preference votes (arena-style, Elo ratings).
Watch out for: contamination (test data in training), saturation (everyone near 100%), different settings, cherry-picking.
Small gaps are often noise: check confidence intervals and number of questions.
Use public benchmarks to shortlist; decide with your own eval set on your own tasks.