Skip to content
← All cheat sheets

Cheat sheet · Area 06

AI Security

Prompt injection, indirect injection and jailbreaks

Prompt injection is when text in a model's input overrides the instructions the developer gave it, typed directly by a user or hidden in content the model reads (indirect injection), while a jailbreak tricks the model into ignoring its safety rules; none can be fully prevented inside the model, so you limit what a fooled model can do.

  • Prompt injection: input text overrides the developer's instructions. No model is immune (OWASP LLM01:2026).
  • Indirect injection: the attack hides in content the model reads (web pages, emails, PDFs, tool results).
  • Jailbreak: tricking a model past its safety training (role-play, hypotheticals, encodings).
  • Main defence: limit what a fooled model can do (least privilege, policy checks in code, human approval).
  • Plus: mark untrusted content, detection classifiers, output checks, red teaming. Layers, not one fix.

The OWASP Top 10 for LLM apps

The OWASP Top 10 for LLM Applications is a community list of the ten most important security risks in apps built on language models, from prompt injection to unbounded consumption, and it works best as a checklist for threat-modelling your own app.

  • OWASP = Open Worldwide Application Security Project; the LLM list is from its GenAI Security Project.
  • 2026 edition (Aug 2026): 01 Prompt Injection, 02 Sensitive Info Disclosure, 03 Excessive Agency, 04 Supply Chain, 05 Data and Model Poisoning.
  • 06 Unbounded Consumption, 07 Misinformation, 08 Hidden Context Exposure, 09 Vector and Embedding Weaknesses, 10 Improper Output Handling.
  • Use it to threat-model: which risks apply to my features, and which control covers each?
  • For agents, also read the OWASP Top 10 for Agentic Applications (Dec 2025).

Input guardrails, output validation and moderation

Guardrails are the concrete checks around a model, input guardrails that screen requests, output validation that makes sure answers have the right shape and stay within business rules, and content moderation that catches harmful text, built from rules, schemas and small classifier models.

  • Input guardrails: size limits, injection/jailbreak classifiers, topic limits, PII handling.
  • Output validation: parse JSON against a schema (Pydantic), enforce business rules, escape before HTML/SQL.
  • Moderation: a model scores text by category (harassment, self-harm, violence...); you choose thresholds per category.
  • Tools (as of Nov 2026): provider moderation endpoints, Llama Guard 4, Llama Prompt Guard 2, Guardrails AI, NeMo Guardrails.
  • On failure: retry with the error, fall back to a safe reply, or escalate; log every block.

PII redaction and data leakage

PII redaction removes or replaces personal details (names, emails, phone and card numbers) before text reaches a model, logs or a third party, and data-leakage prevention stops private information escaping through answers, prompts, logs, caches or training data.

  • Redact: replace PII with placeholders like <EMAIL_1> before the model sees it; restore in the reply if needed.
  • Detect with rules (regex, checksums) + NER models (e.g. Presidio); test on your real data formats.
  • Leakage paths: answers, hidden prompts, logs and traces, caches, RAG indexes, provider retention, fine-tuning data.
  • Canary tokens in hidden prompts reveal leaks automatically.
  • Data minimisation: don't send what the model doesn't need; check provider retention and region settings.

Least-privilege tools and sandboxing

Least privilege means giving an AI agent only the tools, permissions and data it needs for its job (read-only where possible, narrow scopes, short-lived credentials), and sandboxing means running anything risky, such as model-written code or shell commands, in an isolated environment where mistakes and attacks can't reach real systems.

  • Ask of every tool: what's the worst thing a fooled model could do with it? Then shrink that.
  • Narrow tools beat general ones: get_order(id) instead of run_sql(query).
  • Read-only credentials, narrow scopes, short-lived tokens, per-user identity; enforce in the system, not the prompt.
  • Sandbox code and commands: container or microVM, no secrets, limited network, CPU/memory/time limits, throwaway filesystem.
  • Combine with approvals for anything that changes the world (next lesson).

Authorization and human approval for agents

Authorization for agents decides, in code and for every single tool call, whether this agent acting for this user may do this action on this resource, and human approval adds a person's explicit yes before high-impact actions such as payments, deletions or messages to outsiders.

  • Authentication = who is this? Authorization = may they do this, to this resource, right now?
  • Agents act ON BEHALF OF a user: effective permissions = agent's allowed scopes ∩ user's permissions.
  • Check every tool call in code: scope, resource ownership, limits. Never ask the model whether it's allowed.
  • Human approval for high-impact, irreversible or unusual actions; show exactly what will happen.
  • Delegated access via OAuth scopes and short-lived tokens; MCP uses OAuth 2.1 for remote servers.

Secrets management and API key rotation

Secrets such as model-provider API keys must live in a secrets manager or the platform's secret store (never in code, prompts, logs or images), be scoped and given to each service at runtime, and be rotated on a schedule and immediately after any leak, with an overlap period so nothing breaks.

  • Never put keys in code, git, prompts, client apps, logs or screenshots.
  • Store in a secrets manager (cloud KMS/Secrets Manager, Vault, GitHub/hosting secrets); inject at runtime as env vars or files.
  • One key per service and environment, least privilege, spending limits on provider keys.
  • Rotate: create new → deploy → overlap → revoke old. After a leak: revoke first, then investigate.
  • Scan for secrets in commits (pre-commit, push protection) and in prompts/logs.

Access control in RAG

Access control in RAG makes sure a user can only retrieve, and so only get answers from, documents they are allowed to see, by storing each chunk's permissions with it, filtering retrieval by the user's identity before ranking, and keeping those permissions in sync with the source systems.

  • Copy each document's ACL (allowed users/groups) onto every chunk as metadata at ingestion.
  • At query time: user identity → groups → filter retrieval BEFORE ranking, in every retriever (vector AND keyword).
  • Post-filtering the top-k loses good results and risks leaks; pre-filter or use filtered search.
  • Sync permission changes and deletions quickly; test with 'can user X see document Y?' evals.
  • Never rely on the model to withhold text it was given.

Data poisoning and model supply chain

Data poisoning is when attackers slip harmful examples into the data a model learns from or retrieves (training sets, fine-tuning data, RAG documents), and supply-chain risk is trusting models, files, packages and plugins from others, such as pickle files that run code the moment they are loaded.

  • Poisoning targets: pre-training data, fine-tuning sets, feedback/RLHF data, RAG indexes, agent memory.
  • Backdoor: a trigger phrase makes a poisoned model misbehave while it looks normal otherwise.
  • Defend data: provenance, review of new sources, dedup and anomaly checks, evals before and after training.
  • Pickle (.pkl, .pt, .bin) can execute code on load. Prefer safetensors; scan and load only trusted files.
  • Pin model versions and hashes, use trusted publishers, lock dependencies, review plugins and MCP servers.

MCP server trust

Connecting an MCP server gives its tool descriptions a voice inside your model's context and its tools a path to your data, so you should only connect servers you trust, review and pin their tool definitions, run them with least privilege, and treat everything they return as untrusted input.

  • An MCP server's tool names and descriptions go straight into the model's context: they can carry instructions.
  • Tool poisoning: hidden instructions in a description. Rug pull: definitions change after you approved them.
  • Only install from trusted publishers; read the code or use vetted registries; pin versions.
  • Pin tool definitions (hash) and re-approve on change; run local servers sandboxed with minimal access.
  • Remote servers: OAuth 2.1 per the MCP spec, narrow scopes; confirm tool calls that change things.

Audit logging

An audit log is a tamper-resistant record of who did what, when, on whose behalf and with what result, including every tool call, approval and data access an AI agent makes, kept so you can investigate incidents, prove compliance and spot abuse.

  • Record: who (user, agent, on behalf of), what (action, tool, arguments, resource), when, result, approver, trace ID.
  • Audit logs ≠ debug logs: structured, complete for security events, append-only, access-controlled, kept longer.
  • For agents: log every tool call, denial, approval, model/prompt version and data source accessed.
  • Make tampering detectable: write-once storage, hash chains, separate permissions.
  • Redact PII/secrets; set retention; alert on suspicious patterns.

Red teaming

Red teaming means deliberately attacking your own AI system the way real adversaries would, with prompt injection, jailbreaks, data extraction and tool abuse, to find weaknesses before others do, then turning every finding into a fix and an automated test.

  • Red team = authorised attackers testing your system; blue team = defenders.
  • Scope first: assets, threat model, rules of engagement, what counts as a failure.
  • Attack categories: injection (direct and indirect), jailbreaks, data and prompt extraction, tool abuse, cost abuse, harmful content.
  • Mix manual creativity with automated suites (tools such as PyRIT, garak, promptfoo as of Nov 2026).
  • Measure attack success rate per category; every finding becomes a fix + a regression test in CI.

Denial of wallet and cost abuse

Denial of wallet is an attack (or accident) that makes your AI system spend huge amounts of money, through floods of requests, giant prompts, very long outputs, runaway agent loops or stolen keys, and you stop it with input limits, output caps, per-user budgets, rate limits and spending alerts.

  • Every request costs real money, so an attacker (or a bug) can bankrupt the feature instead of crashing it.
  • Vectors: request floods, huge inputs, long outputs/reasoning, agent loops, free-tier abuse, stolen keys, resale of your endpoint.
  • Controls: auth, rate limits, max input size, max_tokens, step limits, per-user and per-tenant daily budgets.
  • Provider-side: spending caps and alerts per key; separate keys per feature.
  • Monitor cost per user; alert on spikes; have a kill switch.

Responsible AI basics

Responsible AI is the practice of building AI systems that are fair, private, transparent, safe and legally compliant, which in practice means checking for bias, protecting personal data, respecting copyright and licences, telling people when they're dealing with AI, keeping humans accountable, and following regulations such as the EU AI Act.

  • Bias: test outcomes and error rates per group; fix data, features and thresholds; keep humans on high-stakes decisions.
  • Privacy: minimise data, get a lawful basis, honour deletion, check provider retention (GDPR and similar laws).
  • Copyright: respect licences of data, models and outputs; don't present generated work as someone else's.
  • Transparency: say when users talk to AI, label generated content, document the system (model cards).
  • Regulation: EU AI Act by risk level; high-risk deadlines moved to Dec 2027 / Aug 2028 (as of Nov 2026, check current text).