SECAI Core · phase 3 of 15 · noun
Deduplication
A technique for removing duplicate copies of repeated data. In SIEM, the removal of redundant information provided by several monitored systems.
The Explain card
- Plain English
- Deduplication removes repeated copies of the same data. In a SIEM it means collapsing the same event reported by several monitored systems into one.
- Example
- A firewall, an endpoint agent and a proxy all log the same download. Deduplication turns three alerts into one, and in a training corpus it stops a document appearing hundreds of times.
- Why it matters
- Duplicates inflate a model's confidence in whatever they contain. Repeated examples are memorised more readily, which increases data leakage and makes injected content far more influential.
- Hook
- Say it a thousand times and the model believes it.
Word knowledge
How the word is built, where it came from, and what it sits beside in memory.
In a sentence
The FineWeb build ran MinHash deduplication on Common Crawl WARC records before it wrote the training shard.
How it is built
- de- prefix reverse, off
- duplic root twofold, double
- -ation suffix act or process
Where it came from
- Origin
- English de- plus duplication, from Latin duplicatio, from duplicare, to double, from duplex, twofold, from duo, two, plus plicare, to fold
- Entered the language
- late 1960s
- What changed
- Duplication named a doubling from the early 1400s. English added de- in late-twentieth-century library and computing work, first for catalog records, then disk arrays, then training corpora.
How it is spelled
- Pattern
- de- keeps its e before a consonant, so de plus duplicate is deduplicate, never dduplicate
- Pattern
- Verbs in -ate take nouns in -ation: duplicate becomes duplication
- Pattern
- tion after a spells /ʃən/
- Breaks the pattern
- Older computing and library writing still uses de-duplication with a hyphen
- Breaks the pattern
- Shop talk clips it to dedupe, which dictionaries treat as informal
Spelled like
- duplication
- reduplication
- decryption
- replication
Broken into chunks
-
de
- decode
- decrypt
- denoise
-
duplic
- duplicate
- duplex
- duplicity
-
ation
- replication
- application
- authentication
What it sits beside
Same subject
Same shape
- reduplication
- decompression
- replication
Data preparation
Reversing de- nouns
- decompression
- decryption
- demodulation
The duplic family
- duplicate
- duplication
- reduplication
Where it sits in the deck
Phase 3: Data Fundamentals: Types, Pipelines, and Preparation
Models are only as good as their data — understand the raw material and the engineering that shapes it before covering how it flows or breaks.
