MLStest

MLStest is a software toolkit designed to enable rigorous evaluation of machine learning models through standardized experimentation and statistical testing. It provides a framework to plan, execute, and compare model performance across datasets and experimental configurations, with an emphasis on reproducibility and statistical validity.

The toolkit offers evaluation metrics for classification and regression, supports common statistical tests (paired and unpaired), including t-tests, non-parametric tests, and bootstrap confidence intervals. It supports cross-validation, multiple runs, and resampling; allows defining experiments via a Python API or a command-line interface; integrates with popular machine learning frameworks and data sources; and outputs summaries, plots, and machine-readable reports (JSON/CSV) suitable for dashboards and journals.

Its architecture is modular, comprising an experiment manager, metric calculators, a statistical testing engine, data adapters, and a reporting component. This separation helps extend MLStest with new metrics, tests, or backends. Typical use cases include model selection, ablation studies, and benchmarking across datasets. Limitations include reliance on appropriate data splits and test assumptions, potential computational overhead for large-scale comparisons, and the need for careful interpretation of p-values and effect sizes. See also: model evaluation, benchmarking, statistical testing in machine learning.