Intake and design
At intake, ask what outcome the feature creates and whether AI is necessary. Document sensitive data, affected users, autonomy, external providers and failure impact. The security architect then maps data and trust boundaries with the product team.
Threat modelling should cover both ordinary software risks and AI-specific paths: untrusted instructions, retrieval poisoning, sensitive context, model supply chain, unsafe output use and excessive tool permissions.
- Confirm business purpose and accountable owner
- Classify input, training and retrieved data
- Model abuse as well as accidental failure
- Design a non-AI fallback for critical operations
Build and test
Use separate environments and identities for development and production. Keep secrets outside prompts and repositories. Restrict model endpoints, vector stores and tools by workload identity rather than shared keys.
Evaluation must be versioned with the system. Test intended quality, misuse resistance, privacy, authorization and operational failure. Human reviewers should record why a result is acceptable, not merely click a pass box.
- Pin and inventory models and dependencies
- Validate retrieved content against user authorization
- Treat model output as untrusted input
- Test indirect prompt injection through documents and websites
Release and change control
A release package should state the use-case tier, known limits, test result, monitoring plan, rollback route and named approver. High-impact changes require staged exposure and stronger independent review.
Model swaps, new tools, expanded data sources and prompt changes can all alter risk. Define what counts as a material change before the team is under delivery pressure.
- Use canary release and bounded permissions
- Set transaction and rate limits
- Require approval for consequential actions
- Re-run evaluations for material changes
Operations and retirement
Monitor access denials, unusual prompt patterns, retrieval anomalies, tool-call failures, sensitive output and human overrides. Join these signals with identity, application and cloud telemetry so responders can reconstruct an event.
Retirement includes revoking keys, deleting temporary indexes, enforcing data-retention rules, preserving required evidence and removing unused network access. An abandoned prototype can remain a live data path.
- Assign an AI incident playbook
- Keep provider change notifications
- Review continued business value and risk
- Verify data and access removal at retirement
A practical 30-day field plan
Week one — Discover. Purpose, owner, users, data and impact. Put one accountable owner in the room, agree which business service or decision is in scope, and record the assumptions the team is making. Resist the urge to begin with a technology purchase; the first deliverable is a shared view of the problem and the authority to change it.
Week two — Design. Threat model, boundaries and control plan. Walk through the current process with the people who operate it. Compare the written design with real access paths, data flows, exceptions and on-call practice. Mark every point where an owner is missing or where the team cannot produce evidence that a control works.
Week three — Assure. Evaluate, red-team and approve release. Choose a narrow pilot that can be observed safely. Define the expected result, the rollback path and the person who may accept a trade-off. Capture operational friction as product feedback; controls that are difficult to use will eventually be bypassed.
Week four — Operate. Monitor, respond, change and retire. Review the pilot with engineering, operations, security and the service owner. Close urgent gaps, assign longer work to a funded backlog and set the next evidence review. The month should end with a repeatable operating rhythm, not a one-time presentation.
Evidence worth keeping
Good evidence is understandable outside the team that created it. Keep a concise record that connects the decision, owner, technical implementation and observed result. Screenshots can support evidence, but configuration, logs, test output and approved records are stronger because another person can reproduce or challenge them.
- Confirm business purpose and accountable owner — owner, current state, last validation and any open exception
- Pin and inventory models and dependencies — owner, current state, last validation and any open exception
- Use canary release and bounded permissions — owner, current state, last validation and any open exception
- Assign an AI incident playbook — owner, current state, last validation and any open exception
- Decision log showing who approved residual risk and when it will be reviewed
- Test or exercise result with the actual outcome, not only a pass label
Questions for the leadership review
Use these questions to keep the discussion connected to operating risk rather than tool activity. A useful answer names a person, a service and evidence.
- Who is accountable for the ai engineering outcome when teams disagree about delivery and risk?
- Which critical service or customer promise would be affected by a failure in this area?
- What evidence would tell us the design is working in production today?
- Which exception creates the largest concentration of access, dependency or recovery risk?
- What would the team contain first, and how would it restore a trustworthy service?
- Which improvement can be completed in the next 30 days without waiting for a large programme?
Common questions
Is AI red teaming enough for release?
No. It is one assurance method. Secure architecture, authorization, data controls, ordinary application testing and runtime operations are equally important.
Should prompts be treated as code?
Important prompts should be versioned, reviewed and tested like configuration because they can materially change system behaviour.