Area 00 // FoundationsLesson 04 / 14015 min read

Weights, loss and gradient descent

Training means finding the weights that make the loss (how wrong the model is) as small as possible, and gradient descent does it by repeatedly taking a small step in whichever direction makes the loss go down.

On this page

Think of it like this
The problem it solves
How it works, step by step
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

You're standing on a foggy hillside and want to reach the lowest point of the valley, but you can only see the ground at your feet. What do you do? Feel which way the ground slopes down, take a small step that way, and repeat. Eventually you reach the bottom.

Your position is the model's weights.
Your height is the loss: how wrong the model is.
Feeling the slope is computing the gradient.
The size of your step is the learning rate.

That's gradient descent, the algorithm that trains almost every neural network, including GPT.

The problem it solves

A neural network might have billions of weights. Training means finding values for all of them that make the model right as often as possible. You can't try every combination: even a hundred weights would take longer than the age of the universe to search. You need a way to know, at every moment, which direction to adjust each weight to improve.

How it works, step by step

Start with random (or zero) weights. The model is useless at first.
Make predictions for a batch of training examples.
Compute the loss. For numbers, a common choice is mean squared error: the average of (guess − answer)². Squaring makes all errors positive and punishes big misses more.
Compute the gradient. For every weight: "if I increase this weight a tiny bit, does the loss go up or down, and how fast?" In a deep network, backpropagation does this efficiently, working backwards from the output layer.
Step downhill. weight = weight − learning_rate × gradient. The minus sign means "go the opposite way from uphill".
Repeat for every batch. One full pass over all the data is an epoch; training runs many.
Stop when the loss stops improving (and before the model starts memorising, see overfitting).

One training step. Real training repeats this millions of times.

Choosing the learning rate

Learning rate	What happens
Too small (0.000001)	Tiny steps. Correct, but takes forever.
About right (0.05 below)	Loss drops quickly and settles.
Too big (5.0)	Overshoots the valley, bounces, and the loss can blow up to infinity.

Modern training uses smarter versions of the same idea (Adam, AdamW) that adapt the step size per weight, but they're all gradient descent underneath.

See it in code

Install: pip install numpy
Needs: Python 3.10+

We'll recover the rule y = 3x + 2 from five examples, without telling the model the 3 or the 2. Watch the loss shrink and the weights walk towards the truth.

gradient_descent.py

import numpy as np
 
# Data that follows y = 3x + 2 (we pretend we don't know the 3 and the 2).
x = np.array([0, 1, 2, 3, 4], dtype=float)
y = np.array([2, 5, 8, 11, 14], dtype=float)
 
w, b = 0.0, 0.0        # weights start at zero: a model that knows nothing
lr = 0.05              # learning rate: the size of each step downhill
 
for step in range(301):
    pred = w * x + b                     # 1. guess
    loss = np.mean((pred - y) ** 2)      # 2. loss: mean squared error
    grad_w = np.mean(2 * (pred - y) * x) # 3. gradient: which way is uphill for w?
    grad_b = np.mean(2 * (pred - y))     #    ...and for b?
    w -= lr * grad_w                     # 4. step the other way (downhill)
    b -= lr * grad_b
    if step % 100 == 0:
        print(f"step {step:3}: loss={loss:8.4f}  w={w:.3f}  b={b:.3f}")

Output

step   0: loss= 82.0000  w=2.200  b=0.800
step 100: loss=  0.0006  w=3.014  b=1.960
step 200: loss=  0.0000  w=3.001  b=1.998
step 300: loss=  0.0000  w=3.000  b=2.000

Try lr = 0.2 and you'll see the loss explode instead (the steps are too big). Try lr = 0.001 and after 300 steps it's still far off (too small).

A real example

Example

Training a language model is the same loop, just enormous

When a lab trains an LLM, the "examples" are trillions of words of text. The model reads a chunk and predicts the next token; the loss (cross-entropy) measures how surprised it was by the real next token. Backpropagation computes gradients for every one of its billions of weights, and an optimizer like AdamW takes a small step. This runs on thousands of GPUs for weeks.

The loop is exactly the one in the snippet above: predict, measure loss, compute gradient, step. Only the size differs.

The easy way

Start here

Remember the hillside in fog. If someone asks "how does a model learn?", the answer is: it measures how wrong it is, finds which way is downhill for every weight, and takes a small step, over and over. Run the snippet and play with the learning rate; that one experiment teaches more than any formula.

When you grow: the production setup

In PyTorch the whole loop is a few lines: loss = loss_fn(model(x), y), loss.backward() (backpropagation), optimizer.step(). Automatic differentiation computes every gradient for you. When you fine-tune a model later (LoRA), you'll set the learning rate, batch size and number of epochs: the same knobs you just met.

Trade-offs

What you gain

Works for any model whose loss you can differentiate, from one weight to a trillion.
Each step only needs a batch of data, so it scales to huge datasets.
Simple to understand and implement.

What it costs you

Needs a well-chosen learning rate; bad choices waste days of GPU time.
Can get stuck in flat regions or poor valleys.
Expensive at scale: every step touches every weight.

When not to use it

A problem has an exact formula solution (simple linear regression can be solved directly).
You're using a model through an API: you can't (and don't need to) change its weights; you change the prompt or the data instead.

Common mistakes

Forgetting the minus sign: stepping with the gradient climbs the hill and makes the model worse.
Setting the learning rate once and never checking the loss curve.
Training until the training loss is zero. That usually means the model memorised the data (overfitting).
Confusing the loss (used for training) with the metric you care about (accuracy, user satisfaction). They're related but not the same.

Interview questions

Try answering out loud first, then open the model answer.

Explain gradient descent in plain language.

Model answer

The model's error on the training data is a function of its weights, like a landscape. The gradient tells you, for every weight, which direction increases the error. Gradient descent repeatedly moves each weight a small step in the opposite direction, scaled by a learning rate, so the error keeps going down until it settles near a minimum.

What happens if the learning rate is too high or too low?

Model answer

Too high and updates overshoot the minimum: the loss oscillates or diverges to infinity (you often see NaNs). Too low and training converges very slowly or stalls in a flat region, wasting compute. People use learning-rate schedules (warm-up then decay) and adaptive optimizers like Adam to get the best of both.

What is backpropagation and why is it needed?

Model answer

Backpropagation is the algorithm for computing the gradient of the loss with respect to every weight in a multi-layer network. It applies the chain rule from the output layer back to the input, reusing intermediate results, so the cost of computing all gradients is about the same as one forward pass. Without it, training networks with millions of weights would be impractical.

Quiz

Pick an answer to see whether it’s right and why.

0 of 6 answered

Key terms

Weight: A learned number that says how much one input matters to a neuron and in which direction. Training is mostly the search for good weights.
Loss: One number measuring how wrong a model's predictions are on a batch of examples. Training tries to make it small.
Gradient: For each weight, the direction and steepness in which the loss increases. Training steps the opposite way.
Gradient descent: The training method that repeatedly moves every weight a small step against its gradient so the loss goes down.
Learning rate: How big each gradient-descent step is. Too big and training blows up; too small and it crawls.
Epoch: One full pass of training over the whole training dataset.
Backpropagation: The algorithm that efficiently computes the gradient for every weight in a multi-layer network, working backwards from the output.

Before and after

Read first

003 Neural networks, starting from one neuron