SECAI Core · phase 3 of 15
Deduplication
A technique for removing duplicate copies of repeated data. In SIEM, the removal of redundant information provided by several monitored systems.
The Explain card
- Plain English
- Deduplication removes repeated copies of the same data. In a SIEM it means collapsing the same event reported by several monitored systems into one.
- Example
- A firewall, an endpoint agent and a proxy all log the same download. Deduplication turns three alerts into one, and in a training corpus it stops a document appearing hundreds of times.
- Why it matters
- Duplicates inflate a model's confidence in whatever they contain. Repeated examples are memorised more readily, which increases data leakage and makes injected content far more influential.
- Hook
- Say it a thousand times and the model believes it.
Where it sits in the deck
Phase 3: Data Fundamentals: Types, Pipelines, and Preparation
Models are only as good as their data — understand the raw material and the engineering that shapes it before covering how it flows or breaks.
