Area 06 of 08

AI Security

Defending AI apps: prompt injection, guardrails, private data, safe tools and agents, secrets, supply chain, red teaming and responsible AI.

Printable cheat sheet for this area

1. 093

   Prompt injection, indirect injection and jailbreaks

   Prompt injection is when text in a model's input overrides the instructions the developer gave it, typed directly by a user or hidden in content the model reads (indirect injection), while a jailbreak tricks the model into ignoring its safety rules; none can be fully prevented inside the model, so you limit what a fooled model can do.

   CoversPrompt Injection · Indirect Prompt Injection · Jailbreaks

2. 094

   The OWASP Top 10 for LLM apps

   The OWASP Top 10 for LLM Applications is a community list of the ten most important security risks in apps built on language models, from prompt injection to unbounded consumption, and it works best as a checklist for threat-modelling your own app.

3. 095

   Input guardrails, output validation and moderation

   Guardrails are the concrete checks around a model, input guardrails that screen requests, output validation that makes sure answers have the right shape and stay within business rules, and content moderation that catches harmful text, built from rules, schemas and small classifier models.

   CoversInput Guardrails · Output Validation · Content Moderation

4. 096

   PII redaction and data leakage

   PII redaction removes or replaces personal details (names, emails, phone and card numbers) before text reaches a model, logs or a third party, and data-leakage prevention stops private information escaping through answers, prompts, logs, caches or training data.

   CoversPII Redaction · Data Leakage

5. 097

   Least-privilege tools and sandboxing

   Least privilege means giving an AI agent only the tools, permissions and data it needs for its job (read-only where possible, narrow scopes, short-lived credentials), and sandboxing means running anything risky, such as model-written code or shell commands, in an isolated environment where mistakes and attacks can't reach real systems.

   CoversLeast-privilege Tools · Tool Sandboxing

6. 098

   Authorization and human approval for agents

   Authorization for agents decides, in code and for every single tool call, whether this agent acting for this user may do this action on this resource, and human approval adds a person's explicit yes before high-impact actions such as payments, deletions or messages to outsiders.

   CoversAuthorization for Agents · Human Approval for Actions

7. 099

   Secrets management and API key rotation

   Secrets such as model-provider API keys must live in a secrets manager or the platform's secret store (never in code, prompts, logs or images), be scoped and given to each service at runtime, and be rotated on a schedule and immediately after any leak, with an overlap period so nothing breaks.

   CoversSecrets Management · API Key Rotation

8. 100

   Access control in RAG

   Access control in RAG makes sure a user can only retrieve, and so only get answers from, documents they are allowed to see, by storing each chunk's permissions with it, filtering retrieval by the user's identity before ranking, and keeping those permissions in sync with the source systems.

9. 101

   Data poisoning and model supply chain

   Data poisoning is when attackers slip harmful examples into the data a model learns from or retrieves (training sets, fine-tuning data, RAG documents), and supply-chain risk is trusting models, files, packages and plugins from others, such as pickle files that run code the moment they are loaded.

   CoversData Poisoning · Model supply-chain risks (untrusted weights, pickle files)

10. 102

    MCP server trust

    Connecting an MCP server gives its tool descriptions a voice inside your model's context and its tools a path to your data, so you should only connect servers you trust, review and pin their tool definitions, run them with least privilege, and treat everything they return as untrusted input.

11. 103

    Audit logging

    An audit log is a tamper-resistant record of who did what, when, on whose behalf and with what result, including every tool call, approval and data access an AI agent makes, kept so you can investigate incidents, prove compliance and spot abuse.

12. 104

    Red teaming

    Red teaming means deliberately attacking your own AI system the way real adversaries would, with prompt injection, jailbreaks, data extraction and tool abuse, to find weaknesses before others do, then turning every finding into a fix and an automated test.

13. 105

    Denial of wallet and cost abuse

    Denial of wallet is an attack (or accident) that makes your AI system spend huge amounts of money, through floods of requests, giant prompts, very long outputs, runaway agent loops or stolen keys, and you stop it with input limits, output caps, per-user budgets, rate limits and spending alerts.

14. 106

    Responsible AI basics

    Responsible AI is the practice of building AI systems that are fair, private, transparent, safe and legally compliant, which in practice means checking for bias, protecting personal data, respecting copyright and licences, telling people when they're dealing with AI, keeping humans accountable, and following regulations such as the EU AI Act.