nearduplicate

Near-duplicate refers to items that are highly similar but not identical. In information retrieval and data management, near-duplicate detection aims to identify pairs or groups of items whose content overlaps substantially, such as web pages, documents, or images that have been revised, translated, or copied with minor edits. Detecting near-duplicates helps reduce storage, indexing workload, and user-visible redundancy in search results.

Common methods rely on compact representations and similarity estimation. Shingling or n-grams convert a document into a set of overlapping tokens; similarity is often measured with Jaccard similarity. To scale to large collections, approximate techniques such as minhash and locality-sensitive hashing are used to quickly identify candidate pairs. For images, perceptual hashing extracts features invariant to minor edits and compares hash values. Combined, these techniques enable efficient deduplication and clustering of related content.

Applications include search engine result deduplication, content management systems, data cleaning, and copyright enforcement. Challenges include selecting appropriate similarity thresholds, handling paraphrase, multilingual content, and obfuscated or untranslated variants. Scalability, noise sensitivity, and the trade-off between precision and recall are ongoing concerns in industrial deployments.

Evaluation often uses pairwise similarity metrics, precision, recall, and F1 on labeled benchmarks, along with resource usage such as memory and processing time. Near-duplicate detection remains an active area in information retrieval and data mining.