Threat modeling for agent systems

Why this matters

Agents introduce attack surfaces that classical web threat models miss: untrusted natural-language input drives privileged actions; tools span tenants; long-lived credentials sit in memory; sub-agents proliferate; LLM outputs can leak secrets verbatim. A structured threat model written before you build saves you incident reviews later.

Learning objectives

1. Apply STRIDE + LINDDUN to an agent system.
2. Enumerate the OWASP Top 10 for LLMs and the OWASP Top 10 for Agentic AI.
3. Map threats to controls covered later in the curriculum.
4. Score risk with a simple matrix per agent + per tool.
5. Produce an architecture review document you'd defend in front of a security lead.

1. STRIDE — quick recap

Microsoft's STRIDE categorises threats:

For privacy specifically, use LINDDUN (Linking, Identifying, Non-repudiation, Detecting, Data disclosure, content Unawareness, policy Non-compliance) on the agent's data flows.

2. OWASP Top 10 for LLM Applications (2025+)

Track these explicitly when reviewing an agent product:

1. Prompt Injection (direct + indirect).
2. Sensitive Information Disclosure (leaking PII / training data / system prompts).
3. Supply Chain (model, plugin, MCP server, embedding model provenance).
4. Data and Model Poisoning (RAG corpus poisoning; fine-tune poisoning).
5. Improper Output Handling (LLM output → SQL / shell / HTML without escaping).
6. Excessive Agency (agent acts beyond intended authority).
7. System Prompt Leakage (not just embarrassing — often contains business logic + creds).
8. Vector and Embedding Weaknesses (retrieval poisoning; embedding inversion).
9. Misinformation / Overreliance (humans trusting hallucinated outputs).
10. Unbounded Consumption (cost, GPU, recursion).

For agents specifically, OWASP's Agentic AI Top 10 (in active maintenance) adds:

• Goal manipulation (subverting the agent's plan).
• Memory poisoning (long-term memory feeding back into decisions).
• Tool misuse (calling a tool in unintended ways).
• Cascading sub-agent failures.
• Identity-spoofing across agents.
• Lack of human oversight on high-stakes actions.

Every threat above maps to a control covered later in this folder.

3. The agent-specific attack tree

A reusable structure for any agent product:

ROOT: Unauthorized action performed by Agent X
├── 1. Steal a credential
│   ├── 1a. Compromise the agent host (cloud workload identity → tokens)
│   ├── 1b. Exfiltrate via LLM output (system prompt or tool output contains secret)
│   ├── 1c. Steal refresh token from agent state store
│   └── 1d. Compromise a tool provider that issued long-lived API key
│
├── 2. Trick the agent into using its own authority badly
│   ├── 2a. Direct prompt injection in user message
│   ├── 2b. Indirect prompt injection in retrieved content (RAG poisoning)
│   ├── 2c. Tool-output injection (e.g., HTML tags returned by an API)
│   ├── 2d. Confused-deputy across tenants
│   └── 2e. Sub-agent persuades parent to escalate
│
├── 3. Bypass policy
│   ├── 3a. Skip approval step via crafted argument names
│   ├── 3b. Override scope via token-exchange path
│   ├── 3c. Replay tokens captured from logs
│   └── 3d. Forge JWTs (alg=none / key confusion)
│
└── 4. Hide the attack
    ├── 4a. Disable telemetry by getting agent to call its own admin tool
    ├── 4b. Use legitimate flows that don't get logged
    └── 4c. Race conditions in audit pipeline

Use this as a checklist on a whiteboard with the product team. Each leaf becomes a specific test case.

4. Data classification + blast radius

For every tool the agent can call, write down:

• Data classification of inputs and outputs (public / internal / confidential / restricted / PII / PHI / PCI).
• Side effects (read-only, idempotent write, irreversible write, financial impact, external comms).
• Blast radius if misused (per-user, per-tenant, system-wide).
• Reversibility (instant undo, manual undo, irreversible).

A useful table:

Rule of thumb: anything in the bottom two rows requires step-up authentication and an audit log entry per call. Lessons 5, 6, and 7 cover the controls.

5. Threat-modeling workshop — 60-minute template

Run this once per agent:

1. Diagram (15 min) — principals (lesson 1.2), trust boundaries, data flows, persistence. One whiteboard.
2. Asset list (5 min) — what's worth attacking? (User data, money, brand, infra.)
3. STRIDE per element (15 min) — walk each data-flow arrow; ask each STRIDE letter.
4. LLM-specific overlay (10 min) — walk the OWASP LLM Top 10 + agentic Top 10.
5. Score risk (5 min) — Likelihood × Impact, 1-5 each.
6. Mitigations + owners (10 min) — pick top 5 risks; assign owners; set due date.

Capture in a Markdown doc in the same repo as the agent. Re-run after major architecture changes.

6. Risk register entry — example

Agent:   helpdesk-v1
Threat:  Indirect prompt injection in customer-emailed PDFs
ID:      THR-2026-014
Category: T (STRIDE) + LLM-01 (Prompt Injection) + Agentic-04 (Tool Misuse)
Asset at risk: Customer PII in CRM
Likelihood: 4 / 5 (PDFs are routinely uploaded by customers)
Impact:     4 / 5 (CRM has cross-customer notes)
Risk score: 16 (High)
Controls:
  - C1: Strip suspicious markup from extracted text before sending to LLM
  - C2: Tool-call allow-list excludes `crm.search_global` when context contains user-uploaded files
  - C3: HITL approval required before any `crm.update`
  - C4: Per-tenant scoping enforced on CRM API (defence in depth)
Owner: @sec-team
Due:   2026-07-30
Status: in-progress
Tests: tests/sec/test_pdf_injection.py

Maintaining this register is the single most useful security artefact for an agent program.

7. Linking threats to controls in this curriculum

8. Hands-on lab (2 h)

1. Pick a real or imagined agent product (e.g. "AI Recruiter" that reads CVs, schedules calls, posts on LinkedIn, sends emails).
2. Diagram principals + trust boundaries + data flows.
3. List all tools and fill in the blast-radius table.
4. Run the 60-min workshop solo (or with a friend).
5. Produce a risk register with at least 8 entries; rank top 3.
6. Identify which lessons of this folder address each top-3 risk.

9. Common pitfalls

1. Threat modeling only the happy path — attackers don't use the happy path.
2. Treating prompt injection as a generic "input validation" line item.
3. No data classification — over-scoping every tool by default.
4. Skipping the agent-vs-LLM distinction; the same LLM can serve multiple agents with very different blast radius.
5. Reviewing once at launch and never again.
6. Threat model lives in someone's laptop, not the repo.

10. Self-check

1. STRIDE letters + an agent example for each.
2. Three threats unique to indirect prompt injection.
3. What "denial of wallet" is.
4. Why blast-radius classification matters.
5. The 60-minute workshop structure.

11. References

• OWASP "Top 10 for LLM Applications" (2025 version).
• OWASP "Agentic AI Top 10" working draft.
• Microsoft "Threat Modeling AI/ML Systems" docs.
• NIST AI RMF 1.0 + Generative AI Profile.
• "Confused Deputy" — Hardy 1988.
• "Indirect Prompt Injection" — Greshake et al., 2023.
• MITRE ATLAS (adversarial threats to AI systems).
• Google "Secure AI Framework" (SAIF).