Threat modelling an LLM application: the complete system view

Draw data and authority together

A normal data-flow diagram shows where information moves. For an LLM system, add authority: which component may retrieve a document, call an API, send a message or change a record. A low-trust document should not inherit the authority of the agent that reads it.

Mark every boundary where content changes role—from external text to prompt context, from model output to code input, and from a recommendation to an approved action.

• User and tenant boundary
• Application-to-model provider boundary
• Retrieval index and document source boundary
• Tool execution and downstream system boundary

Model concrete abuse paths

Write scenarios in plain language. A supplier document contains hidden instructions; a customer extracts another tenant’s context; a model output injects markup into an admin page; an agent chooses a high-impact tool without approval.

Connect each scenario to prevention, detection and recovery. Prevention alone is fragile because language systems and external content are variable.

• Direct and indirect prompt injection
• Sensitive information disclosure
• Data or model poisoning
• Insecure output handling
• Excessive agency and unbounded consumption

Design controls at enforcement points

Prompts can guide behaviour but should not be the only control. Enforce access in retrieval services, tool gateways and downstream APIs. Validate structured output before use and keep user-visible confirmation for consequential actions.

Reduce the blast radius with scoped identities, short-lived credentials, allowlisted operations and transaction limits. The agent should have less authority than the human operator, not a pooled super-user account.

• Authorize every retrieval against the requester
• Separate content from control instructions
• Use typed schemas and strict output validation
• Require step-up approval for irreversible actions

Test and observe the threat model

Turn threat scenarios into repeatable tests and detection logic. Include documents with hostile instructions, ambiguous requests, cross-tenant identifiers and unavailable dependencies.

Production logs should connect the user, prompt category, retrieved sources, model version, tool decision, authorization result and final outcome without unnecessarily storing sensitive content.

• Red-team realistic workflows
• Review tool denials and override patterns
• Retest new connectors and data sources
• Update scenarios from incidents and near misses

A practical 30-day field plan

Week one — Inputs. Users, files, web content and system prompts. Put one accountable owner in the room, agree which business service or decision is in scope, and record the assumptions the team is making. Resist the urge to begin with a technology purchase; the first deliverable is a shared view of the problem and the authority to change it.

Week two — Context. Retrieval, memory, tools and provider boundary. Walk through the current process with the people who operate it. Compare the written design with real access paths, data flows, exceptions and on-call practice. Mark every point where an owner is missing or where the team cannot produce evidence that a control works.

Week three — Decision. Model output, validation and approval. Choose a narrow pilot that can be observed safely. Define the expected result, the rollback path and the person who may accept a trade-off. Capture operational friction as product feedback; controls that are difficult to use will eventually be bypassed.

Week four — Effect. Display, database write, message or transaction. Review the pilot with engineering, operations, security and the service owner. Close urgent gaps, assign longer work to a funded backlog and set the next evidence review. The month should end with a repeatable operating rhythm, not a one-time presentation.

Evidence worth keeping

Good evidence is understandable outside the team that created it. Keep a concise record that connects the decision, owner, technical implementation and observed result. Screenshots can support evidence, but configuration, logs, test output and approved records are stronger because another person can reproduce or challenge them.

• User and tenant boundary — owner, current state, last validation and any open exception
• Direct and indirect prompt injection — owner, current state, last validation and any open exception
• Authorize every retrieval against the requester — owner, current state, last validation and any open exception
• Red-team realistic workflows — owner, current state, last validation and any open exception
• Decision log showing who approved residual risk and when it will be reviewed
• Test or exercise result with the actual outcome, not only a pass label

Questions for the leadership review

Use these questions to keep the discussion connected to operating risk rather than tool activity. A useful answer names a person, a service and evidence.

• Who is accountable for the ai threat modelling outcome when teams disagree about delivery and risk?
• Which critical service or customer promise would be affected by a failure in this area?
• What evidence would tell us the design is working in production today?
• Which exception creates the largest concentration of access, dependency or recovery risk?
• What would the team contain first, and how would it restore a trustworthy service?
• Which improvement can be completed in the next 30 days without waiting for a large programme?