SECAI Core · phase 14 of 15
AI threat modeling
The process of identifying potential threats and analyzing the risks in AI systems
The Explain card
- Plain English
- AI threat modeling is the structured process of identifying who might attack an AI system, how they could do it and what the damage would be, then prioritising the risks before building defences.
- Example
- Modeling a customer-facing chatbot with tool access, a team maps threats such as prompt injection via uploaded documents, training data poisoning and exfiltration through tool calls, and rates each by likelihood and impact.
- Why it matters
- AI systems have attack surfaces ordinary applications lack: the model, its data and its prompts. Threat modeling makes those surfaces visible so defences go where the risk is.
- Hook
- Draw the map before the enemy does.
Where it sits in the deck
Phase 14: Incident Response, Evaluation, and Knowledge Bases
When detection fires, teams need structured response plans, scoring frameworks, and curated knowledge bases to triage, measure, and learn from AI security events.
