datach

Datach is a data architecture concept that treats data as a sequence of linked chunks, combining chunked storage with cryptographic chaining to support integrity, provenance, and scalable processing. It is discussed in data infrastructure design as a generic framework rather than a single standard technology.

In a datach model, data is partitioned into chunks. Each chunk carries payload, metadata, and a hash that depends on its own content and the previous hash, producing a chain. A registry or ledger can store chunk ids and relationships to enable provenance verification.

A typical datach stack includes a chunking module, a hash-chain manager, a metadata catalog, and a processing engine. Chunking methods vary (fixed-size vs content-defined), affecting deduplication and performance. The registry supports versioning and snapshotting for reproducible results.

In ingestion and streaming scenarios, datach supports incremental updates and chunk-level querying. The hash chain enables end-to-end integrity checks, aiding governance and regulatory compliance. It is used in data lakes, data pipelines, and research projects requiring reproducible analytics.

Advantages are data integrity, transparent provenance, and scalable storage with potential deduplication. Challenges include system complexity, hashing overhead, coordination in distributed setups, and interoperability with existing data formats and tooling.

Related concepts include data provenance, content-addressable storage, Merkle trees, and data lineage.