Area 01 // LLMsLesson 12 / 14014 min read

Tokenization

Before a language model can read text, a tokenizer chops it into tokens (whole words, pieces of words or symbols) and turns each one into a number, and those tokens are what models read, write, limit and charge for.

On this page

Think of it like this
The problem it solves
How it works, step by step
See it in code
On the real platform
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

Think of a Scrabble set, but instead of single letters, your tiles include common chunks: "the", "ing", "un", "believ", "able". To spell "unbelievable" you'd grab three tiles (un believ able) instead of twelve letters. Rare words need more, smaller tiles.

A tokenizer is the rulebook for which tiles exist and how to spell any text with as few tiles as possible. A language model only ever sees the tiles.

The problem it solves

Neural networks only work with numbers (lesson 6). So text has to become numbers first. Using one number per letter would make sequences very long and slow. Using one number per whole word would need millions of numbers and still fail on new words, typos and code.

Tokens are the compromise. And because everything about using LLMs (price, speed, limits) is measured in tokens, an AI engineer has to understand them.

How it works, step by step

A tokenizer is trained once, before the model. It looks at a huge sample of text and builds a vocabulary: a fixed list of tokens, often 100,000–250,000 of them.
Most modern tokenizers use byte-pair encoding (BPE) or a close cousin. Start with every single byte as its own token. Find the pair of tokens that appears most often next to each other ("t" + "h") and merge it into a new token ("th"). Repeat tens of thousands of times. Common words end up as one token; rare words stay split.
Encoding text: the tokenizer splits your text into vocabulary pieces and replaces each with its token ID, an integer like 4421.
The model reads the IDs, turns each into a vector (next lesson) and predicts the next token ID.
Decoding: generated IDs are turned back into text pieces and joined.
Everything is counted in tokens: your bill (price per million tokens), the context window limit, max_tokens, rate limits and speed (tokens per second).

Text goes in as tokens and comes out as tokens. The model never sees raw letters.

Rules of thumb

Text	Rough token count
1 English word	~1.3 tokens
1,000 tokens	~750 English words, ~4,000 characters
A page of a novel	~400 tokens
Non-Latin scripts (Hindi, Thai…)	often 1.5–3× more tokens for the same meaning
Code, JSON, long numbers	more tokens than you'd guess (brackets, quotes, digits)

See it in code

Install: pip install tiktoken
Needs: Python 3.10+; runs offline after the first download of the tokenizer file.

tokens.py

import tiktoken
 
# o200k_base is the tokenizer used by recent OpenAI models.
# Other providers (Anthropic, Google, Meta) use their own, so counts differ a bit.
enc = tiktoken.get_encoding("o200k_base")
 
text = "Tokenization isn't magic: unbelievably simple!"
ids = enc.encode(text)                          # text -> list of token IDs
pieces = [enc.decode([i]) for i in ids]          # each ID back to its text piece
 
print("token IDs:", ids)
print("pieces   :", pieces)
print("count    :", len(ids), "tokens for", len(text), "characters")
 
# Same meaning, different languages and formats -> different token counts.
for sample in ["Good morning, how are you?",
               "सुप्रभात, आप कैसे हैं?",             # Hindi
               '{"user_id": 48213, "active": true}']:
    print(f"{len(enc.encode(sample)):>3} tokens: {sample}")
 
price_per_million = 2.00                          # example input price, $ per 1M tokens
print(f"cost of 1M such requests: ${len(ids) * price_per_million:,.2f}")

Output

token IDs: [4421, 2860, 12471, 19745, 25, 180692, 4705, 0]
pieces   : ['Token', 'ization', " isn't", ' magic', ':', ' unbelievably', ' simple', '!']
count    : 8 tokens for 46 characters
  7 tokens: Good morning, how are you?
 10 tokens: सुप्रभात, आप कैसे हैं?
 13 tokens: {"user_id": 48213, "active": true}
cost of 1M such requests: $16.00

Notice that spaces belong to the following token (' magic'), and that a short JSON object costs more than a whole sentence. For Claude, count exactly with the API: client.messages.count_tokens(model=..., messages=[...]).input_tokens.

On the real platform

What this looks like in OpenAI's tokenizer page and Anthropic's token counting · See the real screen in the official docs

A tokenizer web page showing text split into coloured tokens. Simplified view of an online tokenizer page, like OpenAI's tokenizer. — Simplified view of an online tokenizer page, like OpenAI's tokenizer.All About AI (simplified mock-up, not a real screenshot). Made for this course.

OpenAI Tokenizer (platform.openai.com/tokenizer): paste text and it colours each token, with a running count. Good for building intuition.
Anthropic has no visual page; use the count_tokens endpoint in code, or watch usage.input_tokens in responses.
Usage dashboards in both consoles show tokens per day and per model: the first place to look when a bill surprises you.

Try pasting the same paragraph in English and in your own language and compare the counts.

A real example

Example

A translation startup's surprise bill

A startup translates support articles from English into eight languages with an LLM. They estimated cost from English word counts. The first invoice was more than double the estimate: the Hindi, Thai and Arabic outputs used two to three times more tokens than the English source for the same meaning, and output tokens cost more than input tokens.

They fixed the budget by counting tokens with the real tokenizer on a sample of each language, set max_tokens per language, and chose a cheaper model for languages where quality held up in their evals.

The easy way

Start here

Use two numbers in your head: 1,000 tokens ≈ 750 English words, and output tokens cost more than input. When the number matters (budgets, limits), count with the real tokenizer instead of guessing. The cost calculator lesson turns this into budgets.

When you grow: the production setup

Production apps log input_tokens and output_tokens for every request, alert on unusual spikes, trim prompts that waste tokens (repeated instructions, verbose JSON), cap max_tokens per feature, and re-measure when switching models, because a new model can have a new tokenizer with different counts for the same text.

Trade-offs

What you gain

Handles any text, including new words, typos, emoji and code, with a fixed vocabulary.
Much shorter sequences than letter-by-letter, so models are faster and cheaper.
Common words compress to one token, which keeps English prompts compact.

What it costs you

Less common languages and scripts cost more and use more of the context window.
Models can't easily see individual letters, so spelling, counting characters and rhyming are harder.
Token counts differ between providers, so prices aren't directly comparable per token.

When not to use it

Don't do character-exact work (counting letters, checksums, exact string edits) with the model; do it in code.

Common mistakes

Estimating cost from word counts or characters instead of tokens.
Using OpenAI's tiktoken to count tokens for Claude or Gemini (different tokenizers).
Forgetting that output tokens usually cost several times more than input tokens.
Sending huge pretty-printed JSON when a compact format would use far fewer tokens.
Assuming the model sees letters, then being surprised when it miscounts them.

Interview questions

Try answering out loud first, then open the model answer.

What is tokenization and why does it matter to an AI engineer?

Model answer

Tokenization splits text into subword units from a fixed vocabulary and maps them to integer IDs, which is what the model actually processes. It matters because cost, latency, context-window limits, max output length and rate limits are all measured in tokens, and because token counts vary by language, format and provider. It also explains quirks like weak character-level reasoning.

How does byte-pair encoding work?

Model answer

Start with a base vocabulary of bytes or characters. Count adjacent pairs in a large training corpus, merge the most frequent pair into a new token, and repeat until the vocabulary reaches a target size. Encoding new text applies the learned merges. Frequent words become single tokens, rare ones split into subwords, and byte-level BPE can represent any input without unknown tokens.

Your multilingual chatbot is much more expensive for some languages. Why, and what can you do?

Model answer

The tokenizer was trained mostly on English-heavy data, so other scripts split into more tokens per word, increasing both input and output tokens. Measure tokens per language with the real tokenizer, set per-language budgets and max tokens, consider models whose tokenizers handle those languages more efficiently, keep system prompts short or cached, and evaluate whether a cheaper model is good enough per language.

Quiz

Pick an answer to see whether it’s right and why.

0 of 6 answered

Key terms

Token: A small chunk of text (a word, part of a word or a symbol) that a language model reads and writes one at a time. Usage and prices are counted in tokens.
Tokenizer: The component that splits text into tokens and maps them to integer IDs (and back). Each model family has its own.
Vocabulary: The fixed list of all tokens a tokenizer and model know, often 100,000 to 250,000 entries.
Byte-pair encoding (BPE): A way to build a tokenizer vocabulary by starting from single bytes or characters and repeatedly merging the most frequent adjacent pairs.
Token ID: The integer that stands for one token in a model's vocabulary. The number itself carries no meaning.
Context window: The maximum number of tokens a model can consider in one request, counting both the input and the generated output.

Before and after

Read first

010 Calling an LLM API for the first time