Testsum

Testsum is a scalar metric used to summarize the outcomes and significance of a set of software tests. It aggregates per-test results into a single number that can be tracked over time, reported on dashboards, or used in release decisions. The concept is a design choice for teams seeking a compact indicator that accounts for both test outcomes and the relative importance of tests.

Computation is defined for a suite with tests i = 1 to n. Each test has an outcome o_i in {0,1} where 0 = fail and 1 = pass, and a nonnegative weight w_i representing importance, risk, or execution cost. A per-test score s_i can be defined as o_i, or adjusted to include partial credit for flaky or inconclusive results. The unnormalized Testsum is T = sum_{i=1}^n w_i s_i. The normalized Testsum, T_norm = T / sum_{i=1}^n w_i, lies in the interval [0,1].

Variants and extensions include time-based decay, which gives more weight to recent tests, and alternative scoring schemes that reflect coverage or severity of failures. Tests can also be grouped by subsystem to produce sub-sums for focused views.

Interpretation of results: A higher Testsum indicates greater passing coverage of important tests; a lower value points to more failures or less critical test coverage. It is typically used alongside other metrics such as test pass rate and coverage data to inform quality assessments and release readiness.

Limitations: The metric depends heavily on the chosen weights and scoring rules, and may mask flaky tests or distributional details of failures. Etymology: a portmanteau of “test” and “sum,” describing the aggregation of test results into a single score.