Skip to content

All lessons

Area 06 // AI SecurityLesson 96 / 14014 min read

PII redaction and data leakage

PII redaction removes or replaces personal details (names, emails, phone and card numbers) before text reaches a model, logs or a third party, and data-leakage prevention stops private information escaping through answers, prompts, logs, caches or training data.

Think of it like this

Analogy

A hospital sends patient files to an outside lab. Before they leave, a clerk blacks out names and addresses and writes a case number instead. When results come back, the clerk matches the case number to the patient. The lab does its job without ever learning who the patient is. And the hospital also checks its own filing cabinets, photocopiers and bins, because files leak from those too.

The problem it solves

Users paste emails, phone numbers, card details and medical information into chatbots. That text then flows into a provider's API, your logs, traces, caches, analytics, eval datasets and perhaps fine-tuning data. Each copy is a privacy risk and a compliance problem (GDPR and similar laws). Meanwhile the model itself can leak: repeating its hidden prompt, other users' data from a shared cache, or confidential documents from RAG.

How it works: PII redaction

  1. Decide what counts as PII for you: names, emails, phone numbers, addresses, ID numbers, card and bank numbers, health data, and so on.
  2. Detect with a mix of methods: patterns and checksums for structured data (cards, IBANs), named-entity models for names and addresses (as of November 2026, Microsoft's open-source Presidio is a common choice), and custom recognisers for your own IDs.
  3. Redact or pseudonymise: replace values with typed placeholders (<EMAIL_1>), keep the mapping in a short-lived vault, and restore values in the reply only when the user should see them.
  4. Apply it everywhere data flows: before the model call, before logging and tracing, before storing eval or training data.
  5. Test it on realistic samples in your languages and formats; measure misses and over-redaction.

How it works: data leakage

  1. Map leakage paths: model answers, hidden prompts and context (OWASP hidden context exposure), logs and traces, semantic and response caches, RAG indexes, vector stores, provider data retention, fine-tuning data, and third-party tools the agent calls.
  2. Minimise: send the model only what the task needs; don't put secrets or other users' data in prompts (token budgets help here too).
  3. Scope by user and tenant: retrieval filtered by permissions (RAG access control), caches keyed by tenant (semantic caching).
  4. Detect leaks: canary tokens in hidden prompts, output scanners for PII and secrets, red-team tests (red teaming).
  5. Control the provider side: check data-retention settings, zero-data-retention options, regions, and whether your data is used for training.
  6. Limit retention in your own systems: delete logs and conversations on a schedule; support user deletion requests.
Redact before data leaves your boundary; restore only for the user who owns it.

See it in code

Needs
Nothing: plain Python 3. Production systems use a detector such as Presidio instead of three regexes.

Reversible redaction with placeholders, plus a canary-token leak check.

pii_redaction.py
import re
 
# PII REDACTION with reversible placeholders: the model never sees the real values; we restore them after.
PATTERNS = {
    "EMAIL": r"[\w.+-]+@[\w-]+\.[\w.]+",
    "CARD":  r"\b(?:\d{4}[ -]?){3}\d{4}\b",          # check cards before phones (both are digit runs)
    "PHONE": r"\+?\d[\d -]{8,}\d",
}
def redact(text: str):
    vault, counter = {}, {}
    for label, pattern in PATTERNS.items():
        for value in re.findall(pattern, text):
            counter[label] = counter.get(label, 0) + 1
            token = f"<{label}_{counter[label]}>"
            vault[token] = value
            text = text.replace(value, token)
    return text, vault
 
def restore(text: str, vault: dict) -> str:
    for token, value in vault.items():
        text = text.replace(token, value)
    return text
 
msg = "Hi, I'm Sam (sam.lee@example.com, +44 7700 900123). Card 4111 1111 1111 1111 was charged twice."
safe, vault = redact(msg)
print("sent to model :", safe)
reply = "Thanks! We'll refund the duplicate charge and email <EMAIL_1> a receipt."
print("shown to user :", restore(reply, vault))
 
# DATA LEAKAGE test: plant a canary in the hidden prompt and check that outputs never contain it.
CANARY = "canary-7f3a"
system_prompt = f"[{CANARY}] You are a support bot. Internal discount code: STAFF50."
leaky_output = "My instructions start with [canary-7f3a] You are a support bot..."
print("leak detected :", CANARY in leaky_output)
Output
sent to model : Hi, I'm Sam (<EMAIL_1>, <PHONE_1>). Card <CARD_1> was charged twice.
shown to user : Thanks! We'll refund the duplicate charge and email sam.lee@example.com a receipt.
leak detected : True

The name "Sam" got through: regexes can't find names, which is why real systems add an entity-recognition model. The card number is a standard test number. Note the system prompt also contains a discount code: the real fix for that is to keep secrets out of prompts altogether.

A real example

Example

The traces that held everything

A health-insurance chatbot redacted PII before calling the model, and the team was proud of it. During an audit, they found their tracing tool recorded the raw user message before redaction, plus tool results containing policy numbers and diagnoses, all retained for a year and visible to the whole engineering team.

They moved redaction to the very first step of the request, added a redaction processor to the tracing pipeline, restricted trace access, cut retention to 30 days, and added a CI test that sends known fake PII through the app and asserts it never appears in logs, traces or the response.

The easy way

Start here

Run a PII detector (Presidio or your provider's tool) on every message before the model call and before logging, replace values with placeholders, and set log retention to the minimum you need. Add a canary to your system prompt and alert if it ever appears in an output.

When you grow: the production setup

Mature setups use a privacy layer shared by all AI features: entity detection tuned per language, reversible pseudonymisation with a secure vault, redaction in logs, traces and eval data, tenant-scoped caches and indexes, provider zero-retention agreements and regional processing, retention schedules and deletion workflows, leakage tests in CI, and privacy reviews for new data flows.

Trade-offs

What you gain

  • Large reduction in privacy and compliance risk.
  • Logs and traces become safe to share with more people.
  • Placeholders keep conversations useful.

What it costs you

  • Detectors miss some PII and over-redact other text.
  • Redaction can remove context the model needed.
  • Adds latency and another component to maintain.

When not to use it

  • Don't redact data the task truly needs and the user consented to share (e.g. drafting a letter to a named person); minimise and protect it instead.

Common mistakes

  • Redacting model input but logging the raw message.
  • Regex-only detection that misses names and addresses.
  • Putting secrets or other customers' data in prompts.
  • Shared caches that serve one user's answer to another.
  • Using raw customer conversations as fine-tuning data.

Interview questions

Try answering out loud first, then open the model answer.

How would you protect PII in an LLM application?

Model answer

Minimise what's collected and sent; detect PII with patterns plus entity models; replace with placeholders via a short-lived vault and restore only for the owning user; apply redaction before model calls, logs, traces, analytics and training data; scope caches and retrieval by user; configure provider retention and region; set retention limits; and test with seeded fake PII.

What are the ways data can leak from an LLM app?

Model answer

Through answers (other users' data via shared caches or unfiltered retrieval, memorised training data), hidden context exposure (system prompts, tool definitions), logs and traces, analytics and eval datasets, third-party tools and plugins, provider retention, and exfiltration via prompt injection (e.g. data encoded into URLs). Each path needs its own control.

What is a canary token and how is it used with LLMs?

Model answer

A unique random string planted in hidden content such as a system prompt or a confidential document. If it ever appears in an output, log or external request, you know that content leaked. It's cheap to add, works in automated tests and production monitoring, and helps measure prompt-extraction resistance.

Quiz

Pick an answer to see whether it’s right and why.

  1. 01Why replace PII with placeholders instead of deleting it?
  2. 02Which is a common, overlooked leakage path?
  3. 03What does a canary token in a system prompt do?
  4. 04Why order detection rules carefully (e.g. card numbers before phone numbers)?
  5. 05Why is fine-tuning on customer conversations risky for privacy?

0 of 5 answered

Key terms

PII (personally identifiable information)
Data that can identify a person, such as names, emails, phone numbers, addresses, ID and card numbers. It must be protected and often masked.
Redaction
Removing or masking sensitive details such as personal data or secrets before text is sent, stored or shown.
Pseudonymisation
Replacing personal details with placeholders or codes (like EMAIL_1) while keeping a separate, protected mapping so the real values can be restored when allowed.
Data leakage
Private or confidential information escaping to people or systems that shouldn't have it, through answers, logs, caches, prompts or training data.
Canary token
A unique random string planted in hidden content (like a system prompt); if it ever shows up in an output or request, you know that content leaked.

Before and after

Read first

Read next

Progress is saved in this browser only.