filegz

Filegz is a cross-platform data compression tool and file format designed as a flexible, gzip-compatible solution with extensions for streaming and random access to compressed data. The implementation focuses on performance and compatibility, enabling both traditional gzip workflows and newer data-processing patterns. It provides a high-level toolchain and a library interface that can be embedded in applications.

Origin and design: The project emerged in 2019 from a coalition of open-source contributors seeking to extend gzip’s capabilities. It preserves byte-for-byte decompression compatibility with standard gzip tooling while adding features such as chunked streaming, random access in compressed streams, and rich metadata.

Features and architecture: The core library is written in C with bindings for Python, Rust, and Go. It supports streaming compression and decompression, multi-threading, and optional dictionaries to improve compression ratios on repetitive data. Filegz files include a header with metadata (original size, CRC, and parameters) and may use a .filegz container format for extended functionality, without breaking compatibility with existing gzip readers.

Usage and ecosystem: Filegz is used to compress and decompress data, validate integrity, and inspect metadata. It integrates with build systems and data pipelines, and you can employ existing gzip tooling for basic work, while taking advantage of the extended features through its API and CLI.

Status and licensing: As of the latest release, filegz is released under the MIT License and maintained in a public repository. It runs on Linux, Windows, and macOS, with ongoing development and community contributions.