deduplication

Deduplication, or data deduplication, is a data compression technique that eliminates redundant copies of data to reduce storage needs and improve efficiency. It breaks data into chunks, computes a signature for each chunk, and stores only one copy of identical chunks while replacing duplicates with references. It can be inline or post-process, and at different granularities: file-level, block-level (fixed or variable-sized), and byte-level or content-defined chunking.

Applications include backup, archival, primary storage, cloud storage, and virtualized environments where many similar images or VMs share blocks. Deduplication reduces storage and typically lowers bandwidth, speeding backups and restores.

Benefits: reduced storage footprint, lower network and I/O load, faster transfers, and potential cost savings. Considerations: deduplication works best on repetitive data and is less effective on highly random data; it adds processing and metadata overhead and can affect write latency for inline implementations. Recovery relies on deduplication metadata.

Encryption can reduce deduplication effectiveness if data is encrypted before deduplication. Some systems perform dedup before encryption or use architectures that preserve dedup while maintaining security.

Common terms include inline versus post-process deduplication, source-side versus target-side deduplication, and local versus global deduplication.