informationcriterion

Information criterion is a class of statistical tools used for model selection. It provides a quantitative basis to compare competing models by balancing how well a model fits the observed data against the complexity of the model. The core idea is to approximate the information lost when a model is used to represent the data, often formalized through a likelihood function and a penalty for the number of estimated parameters. In practice, an information criterion assigns a numerical score to each candidate model, and the model with the lowest score is typically chosen.

Several well-known information criteria are widely used in statistics and econometrics. The Akaike Information Criterion (AIC) combines the maximum log-likelihood with a penalty term proportional to the number of estimated parameters: AIC = -2 log L + 2k, where L is the likelihood and k is the number of parameters. The Bayesian Information Criterion (BIC), also called the Schwarz criterion, adds a penalty that grows with both the number of parameters and the sample size: BIC = -2 log L + k log n. A corrected version for small sample sizes is AICc. The Minimum Description Length (MDL) principle and related criteria originate from information theory and seek models that minimize the total description length of the model and the data. Other variants include the Hannan–Quinn and CAIC criteria.

Information criteria are used to compare both nested and non-nested models and do not provide formal hypothesis tests. Their performance depends on the context, such as the true data-generating process and sample size; AIC tends to favor more complex models, while BIC tends to favor simpler models as the sample size grows. In practice, researchers may report multiple criteria and consider theoretical justification and practical interpretability alongside the scores.