Area 00 // FoundationsLesson 05 / 14013 min read

Overfitting and train, validation and test splits

Overfitting is when a model memorises its training examples instead of learning the general pattern, so it looks perfect in training and fails on new data; you catch it by keeping some data aside that the model never learns from.

On this page

Think of it like this
The problem it solves
How it works, step by step
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

Two students prepare for a maths exam. The first memorises the answers to last year's paper word for word. The second practises the methods. On last year's paper both score 100%. On this year's paper, with new numbers, the first student fails and the second passes.

The first student overfitted: they learned the examples, not the skill. The only way a teacher can tell the two apart is to give a test with questions they haven't seen.

The problem it solves

A model can always get better on the data it trains on, all the way to perfect, by memorising it. That says nothing about how it will do on new data, which is the only thing that matters in real use.

If you judge a model by the same examples it learned from, you'll ship models that look brilliant in testing and fail in front of customers.

How it works, step by step

Split your data before training into three groups that never mix:
- Training set (~70–80%): the model learns from this.
- Validation set (~10–15%): you check choices on it, like "is 50 epochs better than 20?" or "is the bigger model worth it?".
- Test set (~10–15%): the final exam. Look at it once, at the end.
Train on the training set only.
Watch both training and validation loss. At first both go down. If training keeps improving but validation starts getting worse, the model has begun memorising: that's overfitting. Stop there ("early stopping").
If both are bad, the model is too simple or under-trained: underfitting. Use a bigger model or better features.
Report the test score as your honest estimate of real-world performance, called generalisation.

Split once, before anything else. The test set stays locked until the end.

Fixes for overfitting

Fix	Why it helps
More (and more varied) training data	Harder to memorise, easier to see the real pattern
A smaller or simpler model	Less capacity to memorise noise
Early stopping	Stop when validation loss starts rising
Regularisation (dropout, weight decay)	Penalises overly complex solutions

See it in code

Install: pip install numpy scikit-learn
Needs: Python 3.10+

The truth is a straight line plus noise. We fit a simple line (degree 1) and a very wiggly curve (degree 12) and compare their errors on training data and on held-out test data.

overfitting.py

import numpy as np
from sklearn.model_selection import train_test_split
 
rng = np.random.default_rng(0)
x = np.linspace(0, 1, 30)
y = 2 * x + rng.normal(0, 0.2, size=30)   # the truth is a straight line plus noise
 
# Hold back 30% of the data that the model never sees while learning.
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.3, random_state=0)
 
for degree in (1, 12):                     # a simple line vs a very wiggly curve
    coeffs = np.polyfit(x_train, y_train, degree)       # "train"
    err = lambda xs, ys: np.mean((np.polyval(coeffs, xs) - ys) ** 2)
    print(f"degree {degree:2}: train error {err(x_train, y_train):.3f}   test error {err(x_test, y_test):.3f}")

Output

degree  1: train error 0.031   test error 0.016
degree 12: train error 0.005   test error 26.564

The wiggly curve is better on training data (0.005) and catastrophically worse on new data (26.6). It bent itself through every noisy point instead of learning "it's a line". Without a test set you'd have picked the wrong model.

A real example

Example

The same trap in an LLM app

A team builds a support bot and tunes its prompt using 10 favourite customer questions. After a week of tweaks, all 10 answers are perfect. They launch, and real customers ask things phrased differently, and quality is poor.

They overfitted the prompt to 10 examples. The fix is the same as in classic ML: collect 200 real questions, tune on 150, and keep 50 as a test set they only check before each release. You'll do exactly this in the Evals area with golden datasets.

The easy way

Start here

Before you train or tune anything, run one line: train_test_split(X, y, test_size=0.2, random_state=0). Put the test part away. That habit alone prevents the most common beginner mistake in ML.

When you grow: the production setup

Teams use a fixed, versioned validation set for day-to-day decisions, a separate test set for release decisions, time-based splits for anything that changes over time, and cross-validation when data is small. For LLM apps the same discipline becomes an eval suite that runs in CI (eval gates).

Trade-offs

What you gain

Gives an honest estimate of real-world performance before launch.
Catches overfitting early, saving wasted training and bad launches.
Makes model comparisons fair.

What it costs you

Data held back for validation and testing isn't used for learning.
With very little data, the scores are noisy (cross-validation helps).
Needs discipline: one peek too many and the test set stops being honest.

When not to use it

Never skip it entirely. Even with tiny datasets, use cross-validation instead of no hold-out at all.

Common mistakes

Judging a model by its training score.
Splitting after some processing that used all the data (for example, normalising with the mean of the whole dataset), which leaks test information into training. This is data leakage.
Splitting time-series data randomly, so the model "sees the future".
Having the same customer or document in both training and test sets (near-duplicates count too).
Re-using the test set over and over until it's just another validation set.

Interview questions

Try answering out loud first, then open the model answer.

How do you detect and prevent overfitting?

Model answer

Detect it by comparing performance on training data with held-out validation data: a growing gap, or validation loss rising while training loss falls, means overfitting. Prevent it with more and more varied data, a simpler model, regularisation (dropout, weight decay), data augmentation and early stopping based on validation loss. Always confirm on an untouched test set.

Why do we need both a validation set and a test set?

Model answer

The validation set is used repeatedly to make decisions (hyperparameters, model choice, when to stop), so the model becomes indirectly tuned to it and its score becomes optimistic. The test set is used once at the end to get an unbiased estimate of real-world performance. If you only had one set, you'd have no honest final number.

What is data leakage? Give an example.

Model answer

Data leakage is when information that wouldn't be available at prediction time, or from the test set, gets into training, making scores misleadingly good. Examples: normalising features using statistics computed over the whole dataset before splitting; a feature that's recorded after the outcome (like "refund issued" when predicting fraud); or the same user's records in both train and test.

Quiz

Pick an answer to see whether it’s right and why.

0 of 6 answered

Key terms

Overfitting: When a model memorises its training examples, including their noise, so it does well on them but badly on new data.
Underfitting: When a model is too simple or under-trained to capture the pattern, so it does badly even on its training data.
Training set: The part of the data the model learns from.
Validation set: Held-out data used to compare models and settings during development, without touching the test set.
Test set: Held-out data used once, at the end, to estimate how the model will do on real new data.
Generalisation: How well a model performs on new data it never saw in training. The thing you actually care about.
Data leakage (ML): When information from the test set, or from the future, sneaks into training and makes scores look better than they really are.

Before and after

Read first

004 Weights, loss and gradient descent