SECAI Core · phase 6 of 15
Overreliance
When human practitioners lose critical thinking capabilities and become overly dependent on AI recommendations
The Explain card
- Plain English
- Overreliance is when practitioners lean so heavily on AI recommendations that they stop thinking critically, accepting outputs without checking them.
- Example
- A SOC analyst lets an AI triage assistant close alerts for months because it is usually right. When an attacker crafts activity that the assistant labels benign, the analyst no longer looks, and the intrusion runs unnoticed.
- Why it matters
- Overreliance turns a helpful tool into a single point of failure, and attackers only need to fool the model, not the person. Defenders counter it with spot checks, explanations of why a recommendation was made, and keeping humans practised.
- Hook
- Trust the model the way you trust autocorrect: helpful, often right, never unsupervised.
Where it sits in the deck
Phase 6: Model Quality, Ethics, and Responsible AI Principles
Before deploying a model, understand the quality and ethical dimensions — bias, fairness, explainability, and the human oversight needed to govern outputs responsibly.
