Area 00 // FoundationsLesson 08 / 14015 min read

Classic machine learning in one lesson

Before LLMs, most machine learning was (and much still is) three jobs, predicting a number (regression), predicting a category (classification) and finding groups (clustering), done with small, fast models like linear regression and decision trees.

On this page

Think of it like this
The problem it solves
How it works, step by step
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

A seasoned bank clerk does three kinds of judgement every day:

"How much will this house sell for?" A number. That's regression.
"Is this cheque genuine or forged?" A category. That's classification.
"These customers seem to fall into a few types…" Finding groups nobody labelled. That's clustering.

Classic machine learning automates these three judgements using past examples.

The problem it solves

LLMs get the headlines, but a huge share of real ML in companies is still prices, risk scores, churn predictions and fraud flags computed from tables of numbers. Using an LLM for those is like hiring a novelist to do your accounts: slow, expensive and not better.

An AI engineer needs to recognise these problems and know the small, fast tools that solve them, and to recognise when an LLM really is the better tool.

How it works, step by step

Decide the task type.
- Regression: predict a number (rent, delivery time).
- Classification: predict a category (spam / not spam, which product line).
- Clustering: find groups in unlabelled data (customer segments).
Pick a simple model first.
- Linear regression draws the best straight line (or flat surface) through the data.
- Logistic regression is the classification cousin: it outputs a probability for each class.
- A decision tree learns a flowchart: "size > 70 m²? → yes → near a station? → …". Many trees combined (random forests, gradient boosting like XGBoost or LightGBM) are often the best choice for tables.
- K-means clustering places k centres and assigns each point to the nearest one, moving the centres until they settle.
Split the data into training and test sets (lesson 5).
Train with .fit(), predict with .predict(), check with .score(). In scikit-learn, every model uses this same pattern.
Measure with the right metric. Regression: average error. Classification: accuracy, or precision and recall when one class is rare (lesson 68).

Task	Typical first model	Output example
Regression	Linear regression, gradient-boosted trees	1,915 (rent)
Classification	Logistic regression, random forest	"fraud" with 0.87 probability
Clustering	K-means	customer is in group 2

See it in code

Install: pip install scikit-learn
Needs: Python 3.10+

All three jobs in one file, using scikit-learn's built-in iris flower dataset (150 flowers, 4 measurements each, 3 species).

classic_ml.py

from sklearn.cluster import KMeans
from sklearn.datasets import load_iris
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
 
# 1) Regression: predict a NUMBER (rent from size).
sizes, rents = [[40], [55], [65], [90], [120]], [1050, 1400, 1650, 2300, 3050]
reg = LinearRegression().fit(sizes, rents)
print("regression  -> rent for 75 m²:", round(reg.predict([[75]])[0]))
 
# 2) Classification: predict a CATEGORY (iris flower species from 4 measurements).
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
tree = DecisionTreeClassifier(max_depth=3, random_state=0).fit(X_train, y_train)
print("classification -> test accuracy:", round(tree.score(X_test, y_test), 2))
 
# 3) Clustering: find GROUPS with no labels at all.
km = KMeans(n_clusters=3, n_init=10, random_state=0).fit(X)
print("clustering  -> flowers per group:", sorted(int((km.labels_ == k).sum()) for k in range(3)))

Output

regression  -> rent for 75 m²: 1915
classification -> test accuracy: 0.98
clustering  -> flowers per group: [38, 50, 62]

The tree gets 98% of unseen flowers right. K-means, with no labels at all, found three groups; the real species have 50 each, so it found one perfectly and mixed up the other two a little, because those two species really do look alike.

A real example

Example

A food-delivery app uses both kinds of AI

A delivery app predicts "your food arrives in 28 minutes". That's regression on features like distance, restaurant prep time, rider availability and weather, run millions of times a day with a gradient-boosted tree model that answers in under a millisecond.

The same app uses an LLM to summarise customer complaints and draft replies, where language understanding is the hard part. Each tool where it's strongest: a cheap classic model for numbers, an LLM for language.

The easy way

Start here

Learn the scikit-learn pattern (fit, predict, score) on the snippet above. For any table-of-numbers problem, try a gradient-boosted tree (HistGradientBoostingClassifier or ...Regressor in scikit-learn) before anything fancier.

When you grow: the production setup

Production tabular ML adds feature pipelines, cross-validation, hyperparameter search, model registries (lesson 121) and drift monitoring. Increasingly, teams combine both worlds: an LLM turns messy text into clean features, and a classic model makes the final cheap, explainable decision.

Trade-offs

What you gain

Very cheap and fast: train on a laptop, predict in microseconds.
Easy to explain (especially trees and linear models), which regulators and stakeholders like.
Often the most accurate option for tabular data.

What it costs you

Needs good features; it can't read raw text or images well on its own.
Needs labelled data for regression and classification.
Each model does one narrow job; an LLM can do many tasks from a prompt.

When not to use it

The input is free text, images or audio and the task needs understanding (summaries, open questions): use deep learning or an LLM.
You have no labelled examples and no time to collect them; a prompted LLM can be a quick zero-data start.

Common mistakes

Using an LLM for a problem that is really a table of numbers.
Judging a classifier by accuracy when one class is rare (99% "not fraud" accuracy can catch zero fraud).
Forgetting to scale or encode features the model needs (text categories must become numbers).
Skipping a simple baseline, so you can't tell whether the fancy model is actually better.

Interview questions

Try answering out loud first, then open the model answer.

When would you choose classic ML over an LLM?

Model answer

For structured, tabular data with a clear target (prices, churn, fraud, demand), where you have labelled history and need low cost, low latency, high throughput and explainability. Gradient-boosted trees typically beat LLMs there. I'd use an LLM when the input is unstructured language, the task is open-ended, or there's no labelled data, and sometimes combine them: LLM to extract features, classic model to decide.

Explain the difference between regression, classification and clustering with examples.

Model answer

Regression predicts a continuous number, like delivery time. Classification predicts a discrete label, like whether a transaction is fraudulent or which department a ticket belongs to. Both are supervised and need labelled examples. Clustering is unsupervised: it groups similar examples without labels, like discovering customer segments from purchase behaviour.

How does a decision tree make a prediction, and why are ensembles of trees popular?

Model answer

A tree splits the data with a sequence of feature thresholds chosen to make each branch as pure as possible; a new example follows the questions down to a leaf, which holds the prediction. Single trees overfit easily, so ensembles help: random forests average many trees trained on random subsets, and gradient boosting adds trees one at a time, each correcting the previous errors. They're accurate, handle mixed feature types and need little preprocessing.

Quiz

Pick an answer to see whether it’s right and why.

0 of 6 answered

Key terms

Regression: A machine learning task that predicts a number, such as a price or a delivery time.
Classification: A machine learning task that predicts a category from a fixed set, such as spam or not spam.
Clustering: Grouping similar examples together without any labels, for example finding customer segments.
Decision tree: A model that learns a flowchart of yes/no questions about the features, ending in a prediction. Easy to explain; ensembles of trees are strong on tables.
Accuracy: The share of predictions that are correct. Misleading when one class is rare; use precision and recall then.
scikit-learn: The standard Python library for classic machine learning, with the same fit / predict / score pattern for every model.

Before and after

Read first

005 Overfitting and train, validation and test splits