Area 07 // DeploymentLesson 116 / 14013 min read

Ollama for local models

Ollama downloads open models and runs them on your own laptop or server with one command, exposing a simple local HTTP API, which is ideal for offline development, privacy-sensitive experiments and cheap testing, as long as you evaluate whether a small local model is actually good enough for the task.

On this page

Think of it like this
The problem it solves
How it works, step by step
See it in code
A real example
The easy way
Trade-offs
Common mistakes
Interview questions
Quiz
Key terms
Before and after

Think of it like this

Analogy

Having a coffee machine at home instead of buying from a café. It's private, always available, works when the internet is down and costs nothing per cup. But a home machine isn't a professional barista: for some drinks the café is simply better, so you taste-test before serving guests.

The problem it solves

Calling hosted models needs an internet connection, an API key and money per token, and it sends your data to a provider. For development, demos on a plane, privacy-sensitive experiments or a quick test of an open model, you want a model running on your own machine with almost no setup. You also need to know how much weaker small local models are before you rely on one.

How it works, step by step

Install Ollama from ollama.com (Windows, macOS, Linux) or run the ollama/ollama Docker image.
Pull a model: ollama pull qwen3:0.6b (or llama3.2, gemma3, nomic-embed-text for embeddings; as of November 2026). Models come quantized, usually to about 4 bits per weight, so they fit in normal RAM (quantization).
Try it: ollama run qwen3:0.6b opens a chat in the terminal; ollama list shows what's downloaded.
Call the API: the server listens on http://localhost:11434 (/api/generate, /api/chat, /api/embed, plus an OpenAI-compatible /v1 endpoint), so most SDKs work by changing the base URL.
Use it from Python with the ollama package: the capstone app supports LLM_PROVIDER=ollama (build a RAG app).
In Docker Compose, run Ollama as a service with a volume for models (Compose).
Hardware: small models (under ~4B) run on CPU; larger ones want a GPU or Apple silicon (GPUs and memory).
Evaluate before trusting it: run your golden set against the local model; small models often struggle with long context, citations and strict instructions (golden datasets).
For production serving at scale, use a serving engine such as vLLM on GPU instances.

See it in code

Install: Ollama from ollama.com, then: ollama pull qwen3:0.6b
Needs: About 1 GB of disk and RAM for this model. The commands below ran on a GitHub Actions runner (CPU only).

Pulling a small model, calling Ollama's HTTP API, then plugging it into the capstone RAG app.

ollama_demo.sh

ollama pull qwen3:0.6b
ollama list
 
# Ollama's own HTTP API (no key, no internet needed after the pull):
curl -s http://localhost:11434/api/generate -d '{
  "model": "qwen3:0.6b", "prompt": "In one sentence, what is Docker?",
  "stream": false, "think": false }'
 
# The capstone RAG app with the local model instead of a hosted one:
cd examples/rag-app
pip install -r requirements.txt ollama
LLM_PROVIDER=ollama LLM_MODEL=qwen3:0.6b python -c "
from rag_app.app import RETRIEVER
from rag_app.generate import generate
q = 'How many days of annual leave do I get?'
print(generate(q, RETRIEVER.search(q, k=3)))"

Output

NAME          ID              SIZE      MODIFIED
qwen3:0.6b    7df6b6e09427    522 MB    Less than a second ago
 
Docker is a platform that allows developers to package and deploy software as a single,
portable executable image, enabling consistent and repeatable system deployment across
different environments.
tokens: 33 time_s: 1.9
 
I don't know based on the policies. [1]

Real output from the course's GitHub Actions run. The general question got a decent answer in under two seconds on a CPU. The RAG question did not: the retriever had put the correct passage ("25 days of annual leave") first, yet this 0.6-billion-parameter model replied "I don't know" and still cited [1]. The same pipeline answers correctly with larger models. This is the lesson: try a bigger local model (or a hosted one) and let the eval gate decide.

A real example

Example

Offline development, hosted production

A team building an internal policy assistant used Ollama with an 8B model on their laptops for day-to-day development: no keys to share, no cost, and no company documents leaving the machine. Their eval gate ran on every pull request against both the local model and the hosted production model. The local model scored about 15 points lower on faithfulness, which was fine for development but not for users, so production stayed on the hosted API. When a newer open model closed most of the gap, the same evals showed it within a day.

The easy way

Start here

Install Ollama, ollama pull llama3.2 (or a smaller model on a modest laptop), and point your app at http://localhost:11434. Use it for development and demos, and keep a hosted model for anything users rely on until your evals say otherwise.

When you grow: the production setup

Teams pin model tags and digests, run Ollama in containers with model volumes, use the OpenAI-compatible endpoint to switch providers with config only, evaluate local models on the same golden sets as production models, use local embedding models for privacy-sensitive indexing, and move to vLLM or another serving engine on GPUs for multi-user production traffic.

Trade-offs

What you gain

Free per token, private, works offline.
One command to try many open models.
Simple local API that many tools support.

What it costs you

Small models are noticeably weaker (as the RAG test showed).
Speed depends on your hardware; big models need a GPU.
Not built for many concurrent users.

When not to use it

High-traffic production serving or tasks where quality must match frontier models.

Common mistakes

Swapping in a tiny local model without running evals.
Pulling models larger than your RAM, then wondering why everything is slow.
Forgetting a volume in Docker, re-downloading models on every restart.
Exposing port 11434 to the internet without protection.
Assuming a model's licence allows your use without checking.

Interview questions

Try answering out loud first, then open the model answer.

When would you use Ollama in an AI project?

Model answer

For local development and testing without API keys or cost, offline demos, experiments with open models, and privacy-sensitive prototyping where data must stay on the machine. For production serving at scale, use a dedicated serving engine on GPUs, and use evals to decide whether an open model's quality is good enough.

How do you switch an app between a hosted model and a local one?

Model answer

Hide the model behind one interface (a provider setting or an OpenAI-compatible client with a configurable base URL), select it with configuration (environment variables), and run the same eval suite against both to compare quality, latency and cost before switching.

Why might a small local model fail at RAG even with the right context?

Model answer

Small models have weaker instruction following and reading comprehension, struggle with longer prompts and multiple sources, and may over-apply cautious instructions ("say you don't know"), giving wrong refusals or wrong citations. Larger models, better prompts with examples, fewer and shorter chunks, or fine-tuning for the task can help; evals measure the effect.

Quiz

Pick an answer to see whether it’s right and why.

0 of 5 answered

Key terms

Ollama: A free tool that downloads open-weights models and runs them on your own computer, with a simple API.
Open weights: A model whose trained parameters can be downloaded and run on your own hardware. The licence may still limit use.
Quantization: Storing a model's numbers with fewer bits (e.g. 4-bit instead of 16-bit) to save memory and speed it up, with a small quality loss.
Local model: A model running on your own computer or server (for example with Ollama) instead of a provider's API. Private and free per token, but often less capable.
Inference server: Software that loads a model onto hardware and answers requests over an API, handling batching, memory and streaming. Examples include vLLM, SGLang, TGI and Ollama.

Before and after

Read first