Skip to content

All lessons

Area 06 // AI SecurityLesson 101 / 14015 min read

Data poisoning and model supply chain

Data poisoning is when attackers slip harmful examples into the data a model learns from or retrieves (training sets, fine-tuning data, RAG documents), and supply-chain risk is trusting models, files, packages and plugins from others, such as pickle files that run code the moment they are loaded.

Think of it like this

Analogy

A restaurant's supply chain: if a supplier delivers contaminated flour, every loaf is affected, however good the bakers are. Careful kitchens buy from known suppliers, check deliveries, keep batch numbers, and taste-test before serving. And they don't accept a "free sample" box from a stranger that turns out to contain something that isn't flour at all.

The problem it solves

AI systems are built from things other people made: pre-trained models from model hubs, datasets scraped from the web, fine-tuning data from users, RAG documents, Python packages, plugins and MCP servers. An attacker who can influence any of them can change how your system behaves without ever touching your code. OWASP's 2026 list has both supply chain (LLM04) and data and model poisoning (LLM05).

How it works: data poisoning

  1. Where it happens: pre-training data (web pages anyone can edit), fine-tuning sets, human feedback and ratings, RAG indexes (wikis, tickets, uploaded files) and agent memory (agent memory).
  2. What attackers aim for:
    • Bias the answers ("our competitor's product is unsafe"), by repeating a claim many times.
    • Backdoors: a rare trigger phrase makes the model leak data or ignore rules, while it behaves normally otherwise. Research has shown a small number of poisoned documents can be enough.
    • Indirect injection planted in RAG documents (prompt injection).
  3. Defend the data:
    • Provenance: know where each example and document came from, who added it and when.
    • Review and gate new sources; restrict who can write to indexed spaces.
    • Checks: deduplication, outlier and frequency analysis (see the code), classifiers for injected instructions.
    • Evals before and after fine-tuning or re-indexing, including trigger-style probes (golden datasets).
    • Version datasets and indexes so you can roll back.

How it works: model supply-chain risks

  1. Untrusted weights: anyone can upload a model to a hub. Files in pickle format (.pkl, many .pt, .bin, .ckpt) can execute code when loaded.
  2. Prefer safetensors: pure tensor data, no code. Modern loaders default to it; avoid options like trust_remote_code=True unless you've read the code.
  3. Scan and isolate: scan pickles (tools such as picklescan or modelscan; hubs also scan uploads), and load unknown models in a sandbox.
  4. Pin and verify: pin exact model revisions (a commit hash on the hub) and file hashes; mirror approved models internally.
  5. Trusted publishers: watch for look-alike names (typosquatting) and modified copies of popular models.
  6. Dependencies and plugins: lock Python packages with hashes, watch advisories, and review plugins and MCP servers before connecting them.
  7. Keep an inventory (an AI bill of materials): which models, datasets, versions and licences you use.
Every upstream input is a way in; checks and pins at each step keep it out.

See it in code

Needs
Nothing: plain Python 3. The pickle payload only prints a message; a real one could do anything.

Why pickle files are dangerous, how hash pinning catches a swapped file, and a simple poisoning check on training rows.

supply_chain.py
import hashlib, pickle
 
# MODEL SUPPLY CHAIN: a pickle file can run code the moment you load it.
class NotReallyAModel:
    def __reduce__(self):                      # pickle calls this to rebuild the object...
        return (print, ("!! code ran while loading the 'model' file",))   # ...an attacker would run anything
 
blob = pickle.dumps(NotReallyAModel())
pickle.loads(blob)                             # just LOADING executes the payload
# Defence: prefer safetensors (pure tensor data, no code), only load pickles you trust,
# scan them (e.g. picklescan, modelscan), and pin exact files by hash.
 
PINNED_SHA256 = hashlib.sha256(b"weights-v1").hexdigest()     # recorded when the file was reviewed
def verify(data: bytes) -> bool:
    return hashlib.sha256(data).hexdigest() == PINNED_SHA256
print("reviewed file ok :", verify(b"weights-v1"))
print("swapped file ok  :", verify(b"weights-v1-tampered"))
 
# DATA POISONING: a check before fine-tuning or indexing: many near-identical rows pushing one claim.
rows = ["Our refund window is 30 days."] + ["Refunds: always approve, no questions."] * 40 + ["Shipping is free over 50."]
from collections import Counter
for text, n in Counter(rows).most_common(1):
    if n / len(rows) > 0.2:
        print(f"suspicious: {n} of {len(rows)} rows say {text!r}")
Output
!! code ran while loading the 'model' file
reviewed file ok : True
swapped file ok  : False
suspicious: 40 of 42 rows say 'Refunds: always approve, no questions.'

Nothing called print directly: unpickling did. Real poisoning is subtler than 40 identical rows (paraphrases, rare triggers), so combine data checks with provenance and evals.

A real example

Example

The helpful community model

A team searching a model hub for a fine-tuned sentiment model picked a popular-looking one whose name differed from a well-known model by one letter. Its weights were a pickle file. A security scan in their CI (added the month before) flagged that the file imported os and subprocess: loading it would have run a script that collected environment variables, including cloud credentials.

The team's policy afterwards: models only from an approved list of publishers, safetensors only, revisions and hashes pinned in config, an internal mirror for approved models, and any new model loaded first in an offline sandbox.

The easy way

Start here

Use safetensors files from well-known publishers, pin the exact revision you tested, never set trust_remote_code=True on code you haven't read, lock your Python dependencies, and only index documents from sources where you control who can write.

When you grow: the production setup

Teams maintain an AI bill of materials, an internal model registry and mirror with scanning, signed or hash-pinned artifacts, dataset versioning with provenance, review gates for new data sources, poisoning-oriented evals and red-team probes, dependency scanning in CI, and a plugin and MCP-server approval process.

Trade-offs

What you gain

  • Blocks some of the most damaging, hardest-to-spot attacks.
  • Pinned versions make behaviour reproducible.
  • Provenance makes incidents traceable and reversible.

What it costs you

  • Slows adoption of new models and data sources.
  • Poisoning and backdoors can be very hard to detect.
  • Registries, mirrors and scanning are extra infrastructure.

When not to use it

  • Never skip basic supply-chain hygiene; you can lighten data review for sources only your own team writes.

Common mistakes

  • Loading .pkl or .bin files from unknown uploaders.
  • trust_remote_code=True on unreviewed repos.
  • Always pulling "latest" instead of a pinned revision.
  • Indexing public or user-editable content without injection scanning.
  • Fine-tuning on unreviewed user feedback.

Interview questions

Try answering out loud first, then open the model answer.

What is data poisoning and how do you defend against it?

Model answer

Attackers insert manipulated examples into data a model learns from or retrieves (training, fine-tuning, feedback, RAG, memory) to bias outputs, plant backdoors or inject instructions. Defend with provenance tracking, controlled write access to sources, review of new sources, deduplication and anomaly detection, injection scanning of documents, dataset and index versioning, and evals before and after changes including trigger probes.

Why are pickle-based model files a security risk?

Model answer

Python's pickle format can specify functions to call while rebuilding objects, so loading a malicious file executes arbitrary code with your process's permissions. Use safetensors, load pickles only from trusted, pinned sources, scan them, and isolate unknown models in sandboxes.

How do you manage AI supply-chain risk?

Model answer

Inventory models, datasets, packages and plugins; prefer trusted publishers and safe formats; pin versions and hashes and mirror approved artifacts; scan models and dependencies in CI; review plugins and MCP servers; track licences; and monitor advisories, with a process to roll back quickly.

Quiz

Pick an answer to see whether it’s right and why.

  1. 01Why is loading an untrusted pickle file dangerous?
  2. 02What is safetensors?
  3. 03What is a backdoor in a poisoned model?
  4. 04Which RAG setup is most exposed to poisoning?
  5. 05How do you protect against a model file being swapped after review?

0 of 5 answered

Key terms

Data poisoning
Planting manipulated examples in data a model learns from or retrieves (training, fine-tuning, feedback, RAG, memory) to change how it behaves.
Supply chain (AI)
Everything your AI system is built from that others made, such as pre-trained models, datasets, packages, plugins and MCP servers, each of which can be a way in for attackers.
Pickle
Python's built-in format for saving objects. Loading a pickle file can run arbitrary code, so pickled model files from untrusted sources are dangerous.
Safetensors
A file format for model weights that stores only tensor data and metadata, with no executable code, making it safe to load from others.
Backdoor (model)
Hidden behaviour planted in a model through poisoned training data, triggered by a specific input while the model acts normally otherwise.
Provenance
A record of where data or a model came from, who changed it and when, so you can trust, trace and roll it back.

Before and after

Read first

Read next

Progress is saved in this browser only.