Area 01 // LLMsLesson 13 / 14012 min read
Embeddings and positional encoding
Inside an LLM, every token ID is swapped for a learned vector (its embedding) that captures meaning, and position information is added so the model knows the order of the words.
Think of it like this
Analogy
Imagine every word in the language has a passport with a list of scores: how much it's about animals, food, emotions, the past, plurals… That passport is the word's embedding. Words with similar passports mean similar things.
But a sentence is a queue of people, not a crowd. "Dog bites man" and "man bites dog" contain the same passports. So each person also wears a numbered seat ticket showing where they stand in the queue. That's positional encoding.
The problem it solves
After tokenization (lesson 12) you have token IDs like 4421. The number 4421 means nothing by itself; ID 4422 isn't "a bit more" than 4421. The model needs a representation where meaning lives in the numbers, so similar tokens look similar.
And the core operation of a transformer, attention (next lesson), compares tokens with each other but has no built-in sense of order. Without extra help, word order would be invisible.
How it works, step by step
- Each token ID picks a row from the embedding table, a big learned matrix with one row per vocabulary token. Each row is a vector with a few thousand numbers (the model's dimension).
- Training shapes these vectors. At first they're random. Because similar words help predict similar next words, gradient descent gradually moves related tokens close together.
- Add position information. The transformer gets, for every position (1st, 2nd, 3rd token…), a signal that says where the token is:
- Sinusoidal (the original 2017 transformer): fixed sine and cosine waves of different speeds, added to the embedding.
- Learned positions: one learned vector per position.
- Rotary (RoPE): used by many current models. Instead of adding, it rotates parts of each query and key vector by an angle that depends on the position, so attention naturally sees relative distance.
- The result is one vector per token that carries both "what this token is" and "where it is". That's the input to the first transformer block.
See it in code
- Install
pip install numpy- Needs
- Python 3.10+
import numpy as np
vocab = {"the": 0, "cat": 1, "sat": 2, "on": 3, "mat": 4}
d_model = 4 # size of each token's vector
rng = np.random.default_rng(0)
embedding_table = rng.normal(size=(len(vocab), d_model)).round(2) # learned in real models
def positional_encoding(n_positions, d):
# The original transformer's sine/cosine pattern: each position gets a unique "fingerprint".
pos = np.arange(n_positions)[:, None]
i = np.arange(d)[None, :]
angles = pos / (10000 ** ((2 * (i // 2)) / d))
return np.where(i % 2 == 0, np.sin(angles), np.cos(angles)).round(2)
tokens = ["the", "cat", "sat"]
ids = [vocab[t] for t in tokens]
token_vecs = embedding_table[ids] # 1. look up each token's meaning vector
pos_vecs = positional_encoding(len(tokens), d_model) # 2. a vector for each position
x = token_vecs + pos_vecs # 3. add them: meaning + where it is
print("embedding of 'cat':", embedding_table[vocab["cat"]])
print("position 1 vector :", pos_vecs[1])
print("input to the model (3 tokens x 4 numbers):")
print(x.round(2))embedding of 'cat': [-0.54 0.36 1.3 0.95]
position 1 vector : [0.84 0.54 0.01 1. ]
input to the model (3 tokens x 4 numbers):
[[ 0.13 0.87 0.64 1.1 ]
[ 0.3 0.9 1.31 1.95]
[ 0.21 -1.69 -0.6 1.04]]Real models use thousands of dimensions and a vocabulary of 100,000+ tokens, so the embedding table alone can hold hundreds of millions of parameters.
A real example
Example
Why a model understands 'not good' differently from 'good not'
A reviews-analysis tool sends "The food was not good" to an LLM. The embeddings of "not" and "good" carry their meanings, and the positional information tells the model that "not" comes right before "good". Later layers combine the two into "negative". If the same tokens arrived without positions, the model couldn't distinguish "not good" from "good, not bad". Order is meaning, and positions carry it.
The easy way
Start here
Remember: token ID → lookup → vector of meaning, plus a signal for position. You won't build this yourself; it lives inside every model. You'll use the same "meaning as vectors" idea directly when you build search with embedding models in RAG.
When you grow: the production setup
Position schemes matter in practice for context length: models trained with RoPE can be extended to longer contexts with scaling tricks, which is why context windows grew so fast. When you fine-tune, the embedding table is usually frozen or only lightly updated.
Trade-offs
What you gain
- Turns arbitrary IDs into rich, comparable meaning vectors.
- Learned automatically from data; no human labelling of meanings.
- Position schemes like RoPE let models handle long and variable-length inputs.
What it costs you
- The embedding table is large (vocabulary × dimension parameters).
- Embeddings inherit biases from training text.
- Models struggle beyond the context lengths their position scheme was trained for.
When not to use it
- Not a choice you make as an AI engineer for hosted models; it's built in. It matters if you train or extend models yourself.
Common mistakes
- Thinking a token ID's number carries meaning (ID 100 vs 101 are unrelated).
- Confusing the model's internal token embeddings with the sentence embeddings used for search.
- Forgetting that without positions, attention would ignore word order.
- Assuming embeddings are neutral; they reflect patterns, including biases, in their training data.
Interview questions
Try answering out loud first, then open the model answer.
What is an embedding layer in a transformer?
Model answer
It's a learned matrix of shape vocabulary size × model dimension. Each token ID indexes a row, giving a dense vector that represents the token. The vectors start random and are trained with the rest of the model, so tokens that behave similarly end up close together. Often the same matrix (transposed) is reused at the output to produce logits.
Why do transformers need positional encoding, and what kinds exist?
Model answer
Self-attention is permutation-invariant: it computes the same thing whatever the token order, so position must be injected. Options: fixed sinusoidal encodings added to embeddings (original transformer), learned absolute position embeddings (GPT-2, BERT), relative schemes, and rotary embeddings (RoPE) that rotate queries and keys by position-dependent angles, giving relative-distance awareness and enabling context extension. ALiBi adds distance-based biases to attention scores.
How are LLM token embeddings different from the embeddings used in semantic search?
Model answer
Token embeddings are per-token input vectors inside the model, without context. Search embeddings come from a dedicated embedding model that reads a whole passage (with attention) and outputs one vector for it, trained with contrastive objectives so semantically similar passages land close together. Both express meaning as vectors, but they're used and trained differently.
Quiz
Pick an answer to see whether it’s right and why.
0 of 5 answered
Key terms
- Embedding
- A vector of numbers that represents the meaning of a piece of text (or an image), so that similar meanings get similar vectors.
- Embedding table
- A learned matrix inside a model with one vector per vocabulary token. Looking up a token ID returns its vector.
- Positional encoding
- Information added to token vectors so a transformer knows word order. Variants include sinusoidal, learned and rotary (RoPE).
- Token ID
- The integer that stands for one token in a model's vocabulary. The number itself carries no meaning.
- Dimension
- The number of values in a vector. An embedding with dimension 1,536 is a list of 1,536 numbers.
Before and after
Progress is saved in this browser only.