Misgeneralization

Misgeneralization is the error of extending conclusions or patterns beyond the evidence in a way that leads to incorrect predictions or beliefs. It occurs when a general rule or pattern learned from data or experience does not hold in new contexts, or when the rule is formed from biased or limited information. While generalization aims to apply learned knowledge to novel situations, misgeneralization results when the extension is inappropriate or unsupported.

In machine learning and artificial intelligence, misgeneralization happens when a model relies on correlations that do not persist outside the training data or when labels are noisy or inconsistent. Causes include non-representative training data, distribution shift between training and deployment environments, overfitting, and overly simplistic inductive biases. Consequences are degraded performance on out-of-distribution inputs and brittle behavior in real-world settings.

In human cognition, misgeneralization arises when people infer broad rules from a few observations. This can lead to stereotypes, overconfident judgments, or faulty causal inferences. Heuristics, selective memory, and cognitive biases contribute to such generalizations, especially under uncertainty or time pressure.

Common examples include a classifier that performs well on one domain but fails on another due to different feature distributions, or a person inferring that “all X are Y” from limited experiences. Distinguishing misgeneralization from legitimate generalization requires careful evaluation across diverse, representative scenarios and awareness of potential biases.

Mitigation strategies encompass using diverse and representative data, validating on out-of-distribution samples, regularization, domain adaptation, and robust evaluation in ML; and in human reasoning, training with counterexamples, metacognitive checks, and debiasing techniques.