Simpsonparadox

Simpson's paradox, also known as the Yule-Simpson effect, is a statistical phenomenon in which a trend that holds within multiple groups reverses or disappears when the groups are combined. It shows that the direction or strength of an association between two variables can depend on how the data are partitioned and aggregated.

The paradox arises from confounding or non-homogeneous groups. When groups differ in how the data are distributed—for example, in sizes or in baseline risk—the weighting that occurs in pooling can change the overall relationship. Even if one variable appears to be positively associated with another within each subgroup, aggregation across subgroups can produce a negative association.

A common illustrative scenario involves two treatments or categories evaluated in two separate groups. In each group, treatment A may have a higher success rate than treatment B. Yet, when the results from both groups are combined, treatment B can appear to have the higher overall success rate. This reversal occurs because the groups contribute unequally to the total, often reflecting a lurking variable such as severity, age, or prior risk that affects outcomes differently across groups.

Historical notes: the paradox is named after Edward H. Simpson, who described it in 1951 in contingency-table analysis; it is sometimes called the Yule-Simpson effect, acknowledging earlier related observations by partially overlapping researchers.

Implications and approaches: Simpson's paradox cautions against drawing causal conclusions from aggregated data without considering potential confounders. Methods to address it include stratification, regression controlling for confounders, and causal modeling techniques such as the Mantel-Haenszel adjustment or more modern causal graphs.