metatests

Metatests are evaluations of testing procedures themselves rather than the data analyses they produce. In statistics and psychometrics, a metatest examines properties such as the Type I error rate, power, calibration, robustness to model misspecification, and measurement validity or reliability across a range of conditions. The term is sometimes used more broadly to describe assessments of software test suites or other quality-assurance tests that judge the adequacy of tests rather than the subject data.

Methodologically, metatesting typically relies on simulation studies, where data are generated under controlled scenarios, or on cross-dataset benchmarking. A set of candidate tests is applied to each scenario, and performance is summarized with aggregate metrics such as average power, false discovery rate, familywise error rate, and calibration plots. Resampling, bootstrap, or permutation-based approaches may be used to obtain distribution-free estimates. Results are reported across scenario grids to reveal conditions under which each test performs well or poorly.

Purposes include selecting appropriate testing procedures for given data-generating processes, understanding sensitivity to assumptions, and guiding reporting standards. Metatests can also help compare competing tests, highlight robustness to outliers or nonnormality, and inform recommendations in methodological guidelines.

Limitations include reliance on the realism and breadth of simulated or benchmark scenarios, potential bias in scenario design, and substantial computational cost. Because conclusions depend on the chosen conditions, metatesting complements but does not replace theoretical analyses or empirical validations. See also: meta-analysis, simulation study, test validity, calibration.