Quantization and distillation

How it works: quantization

1. Weights are normally stored as 16-bit numbers (BF16/FP16): 2 bytes each.
2. Quantization stores them with fewer bits: 8-bit (1 byte), 4-bit (half a byte), even lower. Each block of weights gets a scale factor; each weight becomes a small integer.
3. At run time, values are scaled back (approximately) for the maths, or the maths runs directly in low precision on hardware that supports it.
4. Memory drops roughly in proportion to bits (VRAM is often the limit), and generation often gets faster because less data moves through memory for each token.
5. Quality drops slightly: usually negligible at 8-bit, small at good 4-bit methods, more noticeable at 3-bit and below. Smart methods (GPTQ, AWQ, k-quants) protect the most important weights.
6. Formats you'll meet: GGUF (llama.cpp, Ollama, LM Studio; names like Q4_K_M, Q8_0), AWQ/GPTQ (GPU serving with vLLM), FP8 (data-centre GPUs).
7. The KV cache can be quantized too, freeing memory for longer contexts and more users.

How it works: distillation

1. Pick a strong teacher (a large model) and a small student architecture.
2. Generate training data with the teacher: its answers to many prompts (and sometimes its full probability distribution over next tokens, which carries more information than the single answer).
3. Train the student to reproduce the teacher's outputs (SFT on teacher answers, or matching its distributions).
4. Result: a model several times smaller that keeps much of the teacher's ability on the distilled tasks, at a fraction of the cost and latency.
5. Task-specific distillation is common in companies: use a top model to label or answer 50,000 of your real requests, then fine-tune a small model on them (LoRA).
6. Check the terms: some providers' terms restrict using their outputs to train competing models.

import numpy as np
 
# 1) Memory for the weights of an 8-billion-parameter model at different precisions.
params = 8e9
for name, bits in [("FP32", 32), ("FP16/BF16", 16), ("INT8", 8), ("4-bit", 4)]:
    print(f"{name:9}: {params * bits / 8 / 1e9:5.1f} GB")
 
# 2) Quantize some weights to 8-bit integers and back, and measure the error.
rng = np.random.default_rng(0)
w = rng.normal(0, 0.02, size=10_000).astype(np.float32)    # typical small weights
 
scale = np.abs(w).max() / 127                 # map the largest weight to 127
q = np.round(w / scale).astype(np.int8)       # store 1 byte per weight instead of 4
w_back = q.astype(np.float32) * scale         # what the model uses at run time
 
print("int8 values look like:", q[:6])
print(f"average error: {np.abs(w - w_back).mean():.6f} (weights are around {np.abs(w).mean():.4f})")

FP32     :  32.0 GB
FP16/BF16:  16.0 GB
INT8     :   8.0 GB
4-bit    :   4.0 GB
int8 values look like: [  4  -4  21   3 -17  12]
average error: 0.000154 (weights are around 0.0160)