PDFile

PDFile is a cross-platform binary data format and accompanying software library designed for efficient storage and interchange of structured data. It uses a schema-based encoding to represent nested records, arrays, and primitive types, and supports optional compression to reduce storage and I/O costs. The format is optimized for streaming and random access through a chunked payload layout and a central index that enables efficient seeks within large files. PDFile emphasizes portability, data integrity, and ease of interoperability across programming languages.

PDFile originated in the PDFile Consortium in the early 2010s as an open standard for scalable data interchange. Version 1.0 was released publicly in 2013, followed by subsequent releases that added schema evolution, encryption, and improved performance. The project maintains an open-source repository with multiple language bindings and command-line tools for validation, inspection, conversion, and data extraction.

The file structure consists of a header containing magic bytes and version information, a metadata block with global properties, a schema section describing the data types, and a sequence of data blocks. Each data block can be compressed with algorithms such as Zstandard or LZ4, and includes per-block checksums alongside an overall file hash to ensure data integrity. Random access is supported by a per-block index, enabling efficient seeks without full file reads.

PDFile is used in data analytics pipelines, scientific datasets, and archival storage where well-defined schemas and performance matter. Supported languages include C, C++, Java, and Python, with tooling to validate, query, and convert PDFile data. Ecosystem integrations exist for common data processing frameworks.

The project is released under the MIT license and receives ongoing development.