Area 01 // LLMsLesson 30 / 14014 min read
Open vs closed models, small vs large
Closed models are used only through their maker's API and usually lead on quality, open-weights models can be downloaded and run anywhere you like, and small models are cheaper and faster while large ones are smarter; most products end up mixing them.
Think of it like this
Analogy
Renting vs owning a car. A closed model is a premium taxi service: always the newest car, someone else maintains it, you pay per trip, and they decide the routes and prices. An open-weights model is buying a car: more up-front work, insurance and maintenance are yours, but you go wherever you want, it's private, and it's cheap per mile if you drive a lot.
Small vs large is a scooter vs a van. The scooter is perfect for quick trips across town. You only need the van for big loads. Most companies own both.
The problem it solves
"Which model should we use?" is one of the first questions on every AI project, and the answer affects cost, privacy, quality, latency and how much infrastructure you run. Teams often decide by hype or habit, then get stuck with a model that's too expensive, too slow, can't legally see their data, or changes behaviour under them.
You need a clear way to weigh open vs closed and small vs large, and to know that the answer can be "both".
How it works: open vs closed models
- Closed models (as of November 2026: OpenAI GPT-6, Anthropic Claude, Google Gemini) are available only through APIs and cloud platforms.
- Strengths: usually the highest quality, no infrastructure, new features (tools, caching, structured outputs) built in.
- Costs: per-token pricing, your data goes to the provider (under their data policies), rate limits, and models get updated or retired on their schedule.
- Open-weights models (Llama/Muse from Meta, Mistral, Qwen, Gemma, DeepSeek and more) can be downloaded and run anywhere: laptop, your cloud, on-premises.
- Strengths: full control and privacy, fine-tune freely, no per-token fees, behaviour never changes unless you change it, can run offline.
- Costs: you run GPUs, serving, scaling, security and updates; top open models may trail the best closed ones on some tasks.
- Read the model licence. Open weights doesn't automatically mean open source or unrestricted. Apache 2.0 and MIT are permissive; others add conditions (usage policies, user thresholds, attribution, restrictions on training other models).
- Middle routes exist: closed models deployed in your cloud account (Bedrock, Vertex AI, Azure/Foundry) with data-residency options, and providers' zero-data-retention agreements.
How it works: small vs large models
- Size drives capability and cost. Larger (or larger-active, for MoE) models know more and reason better, and are slower and pricier per token.
- Small models (roughly 1–15B parameters, or providers' "mini/lite/haiku/luna" tiers) are fast and cheap, can run on laptops and phones, and are often excellent at narrow tasks (classification, extraction, routing, autocomplete), especially when fine-tuned.
- Large models shine on open-ended reasoning, long complex documents, coding and agents.
- Mix them with routing: classify each request (or try the small model first) and escalate to a large model only when needed (model routing).
| Closed, large | Closed, small | Open, small (self-host) | Open, large (self-host) | |
|---|---|---|---|---|
| Quality on hard tasks | Highest | Medium | Lower–medium | High |
| Cost per token | High | Very low | Very low at scale | Medium (big GPUs) |
| Ops work | None | None | Low–medium | High |
| Privacy / control | Provider terms | Provider terms | Full | Full |
See it in code
- Needs
- Nothing: plain Python 3. It's a starting checklist, not a substitute for evals.
A first-pass decision helper that turns the main constraints into a recommendation with reasons.
# A first-pass checklist for "open or closed model?". Real decisions also need evals.
def recommend(data_must_stay_inside: bool, monthly_tokens: float, have_gpu_team: bool,
need_best_quality: bool, need_fine_tune_weights: bool) -> str:
reasons = []
if data_must_stay_inside:
reasons.append("data can't leave -> self-host open weights (or a private cloud deployment)")
if need_fine_tune_weights:
reasons.append("need full control of weights -> open weights")
if monthly_tokens > 5e9 and have_gpu_team:
reasons.append("very high volume + ops team -> self-hosting may be cheaper")
if need_best_quality and not data_must_stay_inside:
reasons.append("best quality, fast start -> closed API model")
open_votes = sum("open" in r or "self-host" in r for r in reasons)
choice = "open weights" if open_votes > len(reasons) - open_votes else "closed API"
return choice + "\n because: " + "; ".join(reasons or ["no strong constraint -> start with a closed API"])
print("startup MVP :", recommend(False, 2e7, False, True, False))
print("hospital :", recommend(True, 1e9, True, False, False))
print("big platform:", recommend(False, 2e10, True, False, True))startup MVP : closed API
because: best quality, fast start -> closed API model
hospital : open weights
because: data can't leave -> self-host open weights (or a private cloud deployment)
big platform: open weights
because: need full control of weights -> open weights; very high volume + ops team -> self-hosting may be cheaperA real example
Example
One product, four models
An e-commerce company's AI features, after a year of iteration:
- Search query understanding (millions per day): a small open model, fine-tuned, self-hosted. Fast, cheap, private.
- Product description drafts (thousands per day): a small closed model through an API. Good enough, no ops.
- Customer support (complex cases): a large closed model with RAG and tools, with a second provider as fallback.
- Data warehouse questions from analysts: a large open model in their own cloud, because the data can't leave.
Each choice came from evals and constraints, not loyalty to a provider. Their code calls models through one internal interface, so moving a route is a config change plus an eval run.
The easy way
Start here
Prototype with one strong closed model so you learn what "good" looks like. Build a small eval set. Then, for each feature, try a smaller or open model and keep it if it passes. Hide provider code behind one function so switching is easy.
When you grow: the production setup
Mature teams maintain a model catalogue with approved models per data classification, run evals for every candidate and every new release, route by task difficulty and tenant requirements, keep fallbacks across providers, track total cost of ownership for self-hosting (GPUs, engineers, on-call), and review licences with legal before shipping open models.
Trade-offs
What you gain
- Closed: top quality and zero infrastructure. Open: privacy, control, customisation and predictable behaviour.
- Small models: low cost and latency, on-device options. Large models: best results on hard tasks.
- Mixing them gets most of the quality at a fraction of the cost.
What it costs you
- Closed: per-token costs, data-sharing and dependency on the provider's roadmap.
- Open: operational burden and possibly lower top-end quality.
- Running several models adds complexity: more evals, routing logic and monitoring.
When not to use it
- Don't self-host open models just to save money at low volume; APIs are usually cheaper until traffic is large and steady.
- Don't default to the biggest model for every task without measuring whether it's needed.
Common mistakes
- Treating "open weights" as "no rules": licences have conditions.
- Underestimating the cost of running your own GPUs (idle capacity, engineers, on-call).
- Hard-coding one provider's SDK and features throughout the app.
- Picking a small model without evals, or a large one "just in case".
- Forgetting that closed models change; not pinning versions or re-testing.
Interview questions
Try answering out loud first, then open the model answer.
How do you decide between an open-weights and a closed model for a product?
Model answer
Start from constraints: data privacy and residency, quality bar from evals, latency, volume and cost, need for customisation, and the team's ability to operate GPUs. Closed models win for quality and speed to market; open models win when data can't leave, volume is high and steady, deep customisation is needed, or behaviour must be fixed. Licences and total cost of ownership matter. Often the answer is a mix behind a common interface.
When would you use a small model instead of a large one?
Model answer
When the task is narrow and well-defined (classification, extraction, routing, short rewrites), volume is high, latency matters, or it must run on-device, and the small model, possibly fine-tuned, passes the eval bar. Use large models for open-ended reasoning, long complex inputs, coding and agentic tasks, and consider routing to combine them.
What's the total cost of ownership of self-hosting an open model?
Model answer
GPU instances (including headroom and idle time, or reserved commitments), storage and networking, engineering time to build and maintain serving, scaling, monitoring, security patching and upgrades, on-call, evaluation of new versions, and compliance work. Compare against API cost at your real traffic profile, including peaks; self-hosting usually wins only at high, steady utilisation or when required for privacy.
Quiz
Pick an answer to see whether it’s right and why.
0 of 6 answered
Key terms
- Open weights
- A model whose trained parameters can be downloaded and run on your own hardware. The licence may still limit use.
- Closed model
- A model you can only use through its maker's API; the weights are not released.
- Model licence
- The legal terms for using a model's weights, from permissive (Apache 2.0, MIT) to custom licences with conditions.
- Small language model
- A model with relatively few parameters (roughly 1 to 15 billion) or a provider's cheapest tier. Fast and cheap, good at narrow tasks.
- Model routing
- Sending each request to the most suitable model, typically cheap models for easy requests and strong models for hard ones.
Before and after
Progress is saved in this browser only.