zarr

Zarr is an open format and software ecosystem for the storage of chunked, compressed, N-dimensional arrays. It is designed to enable scalable analytics on large datasets, particularly in cloud environments and high-performance computing, by allowing parallel and selective input/output without requiring a single monolithic file.

In Zarr, an array consists of chunks laid out in a multidimensional grid. Each chunk is stored as a separate file or as a block within a single store, and metadata files describe the array’s shape, data type, chunk shape, and compression settings. The group object organizes arrays and metadata hierarchically; the metadata includes files named .zgroup and .zattrs, and each array has a .zarray with configuration information.

Backends for Zarr include local filesystems and object stores such as Amazon S3, Google Cloud Storage, or Microsoft Azure, with support typically provided through the fsspec abstraction. Compression options include Blosc, zlib, zstd, and others; compression is optional and chunk size is tunable to balance I/O performance and storage. Access patterns support random access to chunks and partial I/O for efficient workflows.

Ecosystem and bindings: the Python reference implementation is the zarr library (often referred to as PyZarr), which provides chunking, compression, and metadata handling. Other language bindings exist, including JavaScript (zarr.js), enabling cross-language data sharing. Zarr integrates with data analysis ecosystems such as Dask and Xarray to enable scalable, parallel processing.

History and usage: Zarr was developed to address limitations of traditional formats like HDF5 for cloud storage and parallel I/O. It has been adopted across disciplines including earth science, astronomy, and neuroimaging, and it aims for interoperability and scalable data sharing. Related formats include NetCDF, HDF5, and the N5 format, with Zarr often compared to cloud-optimized storage approaches.