dataduplication

Dataduplication, or data deduplication, is a data reduction technique that eliminates duplicate copies of repeating data to reduce storage requirements and optimize network bandwidth. It identifies identical data blocks or files and stores only a single copy, replacing duplicates with references.

Deduplication can operate at different granularities. File-level deduplication, also known as single-instance storage, removes duplicate files. Block-level deduplication divides data into chunks and detects duplicates within and across files, achieving higher efficiency for changing data. Chunking can be fixed-size or content-defined; the latter, based on rolling hashes, adapts boundaries to data patterns and often yields better savings for variable data.

Deduplication can be implemented in inline mode, performing the reduction as data is written, or in post-process mode, running after data has been stored. It can be local, within a single storage system, or global (cross-system) across multiple storage targets to maximize savings. Source-side deduplication occurs before data is transmitted, reducing network traffic, while target-side deduplication happens on the storage side.

A typical implementation uses a metadata index of unique data blocks identified by cryptographic hashes. When a block with an existing hash is encountered, a reference is stored instead of a new copy. This approach raises considerations about hash collisions, metadata reliability, and the need for efficient garbage collection and integrity checks.

Benefits include reduced storage capacity, lower bandwidth usage, and faster backups for suitable data sets. Limitations involve added CPU and memory overhead, potential latency, management complexity, and decreased effectiveness with highly encrypted or highly random data. Dataduplication is widely used in backup systems, archival storage, virtual machine images, and cloud storage environments.