HDF5

Hierarchical Data Format version 5 (HDF5) is a data model, library, and file format for storing and organizing large and complex data. It is designed to be portable, scalable, and self-describing, enabling efficient management of scientific, engineering, and analytical data across platforms and over time.

The core data model uses a hierarchical structure of groups and datasets. Groups function like folders and can contain other groups and datasets; datasets store multi-dimensional arrays and are accompanied by attributes that describe their metadata. The file itself is self-describing: internal objects, links, and metadata describe the data layout, datatypes, and filters applied to data. The format supports a rich set of datatypes, extensible metadata, and references between objects.

Key features include support for large file sizes, a flexible type system, chunked storage, and data compression through filters (for example gzip and other algorithms). Data can be stored with multiple dataspace configurations, and datasets can be extended. The format supports soft and hard links, external references, and user-defined filters. Parallel I/O is available when built with MPI support, enabling scalable reads and writes on high-performance computing systems.

APIs exist for multiple languages, notably C and Fortran, with bindings or wrappers for Python (such as h5py), Java, MATLAB, and R, contributing to broad adoption in scientific computing. The format is cross-platform and machine-independent, with metadata and data modeled independently from the host architecture.

HDF5 was developed and is maintained by The HDF Group. It is widely used in disciplines such as climate science, geophysics, biology, and engineering, often for data archives, simulation outputs, and large-scale experiments.