zstd

Zstandard, commonly referred to as zstd, is a data compression algorithm and file format developed by Facebook (Meta) and released in 2016. It aims to provide fast compression and decompression with high throughput and strong compression ratios, across a wide range of data types. It is distributed as an open-source library and has become a widely used general-purpose compressor.

The algorithm uses a modern combination of techniques: an LZ77-style sliding-window dictionary for detecting repeated sequences, a token-based encoding of literals and matches, and an entropy coder based on Finite State Entropy (FSE). This design supports both speed and compression ratio. Zstd also supports dictionary compression, allowing pre-trained data patterns to improve results on small inputs or similar datasets.

The zstd file format is frame-based and supports streaming. Data are organized into frames containing blocks, each with its own compression parameters, optional dictionary identifier, and a frame checksum. The format supports multi-block frames, recovery from errors, and streaming decompression. A command-line tool and a reference library provide APIs for compression and decompression, with multiple compression levels.

Performance characteristics: low compression levels favor speed, high levels favor compression ratio. Compared with traditional compressors, zstd often achieves faster speeds than gzip with similar or better ratios, and is competitive with or superior to other modern algorithms like LZ4 at mid-range settings. It supports multi-threading and is available for numerous programming languages via libzstd bindings.

Usage and ecosystem: Zstandard is widely adopted in software packaging, data storage, and network protocols. It has a cross-language library, bindings for C, C++, Python, Java, Rust, Go, and more, and is used in operating systems, packaging tools, and backup solutions.