Area 00 // FoundationsLesson 03 / 14015 min read

Neural networks, starting from one neuron

A neural network is many tiny calculators (neurons) stacked in layers; each one multiplies its inputs by weights, adds them up and passes the result on, and together they can learn very complicated patterns.

On this page

Think of it like this
The problem it solves
How it works, step by step
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

Think of a committee deciding whether to cancel the school picnic. Each member looks at one thing: the sky, the forecast, the temperature. Each gives a vote, but some members are trusted more than others, so their votes count extra. The head teacher adds up the weighted votes and, if the total is high enough, cancels.

That head teacher is one neuron. How much each member is trusted is a weight. How cautious the head teacher is (do they cancel at the first cloud?) is the bias. A neural network is many committees feeding their decisions into bigger committees.

The problem it solves

The rent model in lesson 2 had one number and could only draw a straight line. Real problems aren't straight lines. Whether a photo is a cat depends on thousands of pixels interacting in complicated ways. Whether a sentence is sarcastic depends on word order, tone and context.

We need a model that can represent almost any pattern, but is still built from simple pieces that a computer can tune automatically. Neural networks are exactly that.

How it works, step by step

A neuron takes some inputs. Numbers, like "how cloudy (0–1)" and "rain forecast (0–1)".
It multiplies each input by a weight. A big weight means "this input matters a lot". A negative weight means "this input argues against".
It adds them up, plus a bias. The bias is a starting push: a negative bias means "you need strong evidence before I say yes".
It passes the total through an activation function. Sigmoid squashes any number into 0–1 ("how sure am I?"). ReLU turns negatives into 0 and keeps positives. This bend is what lets networks learn curves instead of only straight lines.
Neurons are grouped in layers. Every neuron in one layer reads all the outputs of the layer before. Input layer → hidden layers → output layer.
Many layers = deep learning. Early layers learn simple things (edges in a photo, letters in text); later layers combine them into complex ideas (faces, meaning).
Training sets the weights. Nobody types them in. They start random and gradient descent tunes them. All the weights and biases together are the model's parameters.

Architecture

A small network: 2 inputs, a hidden layer of 3 neurons, 1 output. Every arrow carries a value multiplied by its own weight.

How big do they get?

Network	Parameters (roughly)
The umbrella neuron below	3
A handwritten-digit recogniser	~100,000
A small open LLM you can run on a laptop	1–8 billion
Frontier LLMs	hundreds of billions to trillions

The idea never changes, only the size. A large language model is this same weighted-sum-and-activation, repeated billions of times, arranged in a clever shape called a transformer.

See it in code

Install: pip install numpy
Needs: Python 3.10+

First one neuron, then a two-layer network that solves XOR ("exactly one input is on"), a problem a single neuron provably can't solve. The weights here are hand-picked so you can see the mechanics; in real life training finds them.

neurons.py

import numpy as np
 
# ---- One neuron: weigh the inputs, add a bias, squash with an activation ----
def neuron(inputs, weights, bias):
    total = np.dot(inputs, weights) + bias   # weighted sum: x1*w1 + x2*w2 + bias
    return 1 / (1 + np.exp(-total))          # sigmoid: squashes any number into 0..1
 
# "Should I take an umbrella?" inputs: [cloudiness 0-1, rain forecast 0-1]
weights = np.array([2.0, 4.0])   # rain forecast matters twice as much as clouds
bias = -3.0                      # needs fairly strong evidence before saying yes
print("sunny day :", round(neuron([0.1, 0.0], weights, bias), 2))
print("rainy day :", round(neuron([0.9, 0.9], weights, bias), 2))
 
# ---- A tiny network: 2 inputs -> 2 hidden neurons -> 1 output ----
# One neuron can't learn XOR ("exactly one is on"). Two layers can.
W1 = np.array([[20.0, -20.0], [20.0, -20.0]])  # hidden layer weights (already trained)
b1 = np.array([-10.0, 30.0])
W2 = np.array([20.0, 20.0])                    # output layer weights
b2 = -30.0
sigmoid = lambda z: 1 / (1 + np.exp(-z))
 
for x in ([0, 0], [0, 1], [1, 0], [1, 1]):
    hidden = sigmoid(np.array(x) @ W1 + b1)    # layer 1: two neurons at once
    out = sigmoid(hidden @ W2 + b2)            # layer 2: one neuron
    print(x, "->", round(float(out)))

Output

sunny day : 0.06
rainy day : 0.92
[0, 0] -> 0
[0, 1] -> 1
[1, 0] -> 1
[1, 1] -> 0

The hidden layer learned two helper questions ("is at least one on?" and "is not both on?"), and the output neuron combines them. That's the power of layers: each one builds on the simpler answers of the one before.

A real example

Example

Reading handwritten postcodes at the post office

Postal services sort millions of envelopes a day by reading handwritten postcodes. A neural network does this:

Input layer: the pixels of each scanned digit (for a 28×28 image, 784 numbers).
Hidden layers: the first learns strokes and edges, the next learns loops and corners, the next learns shapes like "a loop on top of a line".
Output layer: 10 neurons, one per digit 0–9. The one with the highest value is the answer.

Nobody programmed what a "7" looks like. The network saw tens of thousands of labelled digits and training set its weights.

The easy way

Start here

You don't need to build networks by hand as an AI engineer. Remember three facts: a neuron is a weighted sum plus a bend; layers stack neurons; training sets the weights. Then play with the umbrella neuron above: change the weights and bias and watch the answers flip.

When you grow: the production setup

In practice networks are built with frameworks such as PyTorch, where a layer is one line (nn.Linear(784, 128)) and training runs on GPUs. You'll mostly meet them as pre-trained models you download or call through an API.

Trade-offs

What you gain

Can learn almost any pattern given enough data and size.
The same building block works for images, audio, text and more.
Bigger networks keep getting better with more data and compute.

What it costs you

Need lots of data and compute to train.
Hard to see why a network made a decision: the "knowledge" is spread over millions of numbers.
Large networks need GPUs and lots of memory to run.

When not to use it

You have a small table of numbers: classic ML (decision trees, regression) is often as good, faster and easier to explain (lesson 8).
The decision must be fully explainable to a regulator line by line.

Common mistakes

Thinking artificial neurons work like brain cells. The name is inspired by the brain, but the maths is simple multiplication and addition.
Forgetting the activation function: without it, a hundred layers behave like one.
Believing more parameters always means a better model. Data quality and training matter as much.
Thinking you must understand every weight. Nobody can; you judge a network by testing its behaviour.

Interview questions

Try answering out loud first, then open the model answer.

Explain how a neural network works to a non-technical person.

Model answer

A neural network is a big stack of tiny calculators. Each one takes a few numbers, decides how much each matters, adds them up and passes on a single number. The first layer spots simple things, later layers combine them into complex ideas. During training it sees many examples with answers, and every calculator's settings are nudged until the network's answers match. After that it can handle new examples it's never seen.

Why are non-linear activation functions necessary?

Model answer

Each layer without an activation is a linear transformation, and a composition of linear transformations is still linear, so any number of layers collapses to a single layer that can only model linear relationships. A non-linearity such as ReLU, GELU or sigmoid between layers lets the network approximate complex, curved functions. That's what gives depth its power.

What does the number of parameters tell you about a model, practically?

Model answer

Roughly its capacity and its cost. Each parameter is stored in memory, so a 7B-parameter model needs about 14 GB in 16-bit precision just for weights, and every generated token touches most of them, which drives latency and GPU needs. More parameters often means better quality, but architecture (e.g. mixture of experts), training data and fine-tuning matter just as much, and quantization can shrink the memory cost.

Quiz

Pick an answer to see whether it’s right and why.

0 of 6 answered

Key terms

Neuron: The basic unit of a neural network. Multiplies its inputs by weights, adds them plus a bias, and passes the total through an activation function.
Weight: A learned number that says how much one input matters to a neuron and in which direction. Training is mostly the search for good weights.
Bias (in a neuron): A learned number added to a neuron's weighted sum, shifting how easily it activates. Not the same as unfair bias in data.
Activation function: A simple non-linear function (ReLU, sigmoid, GELU) applied to a neuron's output. Without it, stacked layers could only model straight lines.
Layer: A group of neurons that all read the same inputs. Networks stack layers so later ones build on what earlier ones found.
Neural network: A model made of layers of neurons connected by weights. Can learn very complex patterns from data.
Parameter: Any number a model learns during training (all its weights and biases). Model size is usually quoted in parameters, e.g. 8B = 8 billion.

Before and after

Read first

002 How a computer learns from examples