duplikatmatching

Duplikatmatching is a data management technique focused on identifying and reconciling duplicate records that refer to the same real-world entity across one or more datasets. The goal is to create a single, canonical representation of that entity to improve data quality and interoperability.

Methods combine deterministic rules and probabilistic or machine learning approaches. Blocking or indexing reduces the number of comparisons. Similarity measures such as Levenshtein distance, Jaccard similarity, cosine similarity, and numeric tolerances assess attribute likeness. Approaches include rule-based matching, supervised or unsupervised learning, and graph-based clustering. Probabilistic linkage methods, exemplified by the Fellegi-Sunter framework, estimate match probabilities to decide whether two records refer to the same entity.

Applications include customer data integration, healthcare records, bibliographic databases, product catalogs, and fraud detection, where accurate deduplication improves analytics and operational efficiency.

Challenges include data quality issues, missing or conflicting attributes, scalability for large datasets, multilingual or culturally variant names, privacy concerns, and evaluating performance without ground truth.

Duplikatmatching is related to deduplication, entity resolution, and record linkage. It is widely used in data cleaning, data integration, and master data management, and is an active area of research and practice.