mergei

Mergei is an open-source framework for merging data from multiple sources using declarative merge rules. It aims to produce unified records while preserving provenance and enabling reproducible data transformations.

The core feature is a domain-specific language (DSL) for defining merge rules. Rules specify merge keys, attribute reconciliation, and conflict-resolution strategies such as last-write-wins, consensus-based decisions, or user-in-the-loop arbitration. The engine applies these rules to input datasets and emits a unified output with versioned snapshots.

The architecture consists of a core merge engine, adapters that connect to databases, files, or streaming sources, and a rule compiler that translates DSL definitions into executable plans. It supports schema mapping, deduplication, data lineage, rollback, and incremental merging.

Typical applications include customer data platforms, catalog unification in e-commerce, and merging health or scientific records across disparate systems. It is commonly used within ETL pipelines, data lakes, and data governance workflows to reduce duplicates and harmonize attributes.

Limitations include the need for careful rule design to avoid inconsistencies, potential performance overhead for large datasets, and the requirement for governance to audit rules and outcomes. Advocates emphasize the value of transparent provenance and reproducibility.

See also: Data integration, Master data management, Data fusion, ETL, Schema mapping.