SECAI Core · phase 14 of 15
Recall-oriented understudy for gisting evaluations
A set of metrics for evaluating how good a system-generated summary or text is by comparing it to one or more human (reference) summaries, with a focus on how much of the reference content is included
The Explain card
- Plain English
- ROUGE is a family of metrics for scoring machine-generated summaries by how much of the content in human-written reference summaries they capture, with the emphasis on recall.
- Example
- A summarisation model is evaluated on incident reports. Its ROUGE scores stay high, but reviewers find it fabricating details, showing that overlap with references does not guarantee accuracy.
- Why it matters
- Defenders increasingly rely on AI summaries of logs and alerts. ROUGE helps measure whether a model keeps the important facts, but it does not catch hallucination, so pair it with human review.
- Hook
- Did the summary remember the things that mattered?
Where it sits in the deck
Phase 14: Incident Response, Evaluation, and Knowledge Bases
When detection fires, teams need structured response plans, scoring frameworks, and curated knowledge bases to triage, measure, and learn from AI security events.
