datasettiincore

Datasettiincore is a community-driven specification and ecosystem designed to standardize the core datasets used in machine learning, data science, and related research. It provides a common data model, metadata schema, and governance practices intended to improve interoperability, provenance, and reproducibility across projects that rely on publicly available or licensed data.

The core of datasettiincore is its modular data model. It defines a minimal set of metadata fields for each dataset, including identifiers, versioning information, domain, size, formats, licensing, provenance, collection methods, sampling strategies, and quality indicators. The specification also describes recommended schemas for data itself, as well as for accompanying documentation such as data dictionaries and readme files.

Licensing and governance are central to datasettiincore. The project promotes open licenses compatible with research and commercial use and advocates clear attribution, auditability, and consent management where applicable. A community steering council coordinates updates, ensures backward compatibility, and maintains conformance tests for repositories and tooling that claim datasettiincore compatibility.

Adoption of datasettiincore aims to enhance discovery, comparison, and reproducibility of experiments. By enabling consistent metadata and versioning, it supports reproducible benchmarks, fair evaluation, and easier data provenance tracing across domains such as vision, language, and tabular data. Tooling and repositories aligned with the standard offer APIs for discovery and programmatic access while preserving data governance.

As an open specification, datasettiincore invites participation from researchers, data stewards, and platform providers.