ExpectationMaximization

Expectation-Maximization is an iterative method for finding maximum likelihood estimates of parameters in statistical models that depend on unobserved latent variables or incomplete data. The algorithm alternates between an expectation step (E-step), where the expected value of the log-likelihood is computed with respect to the current parameter estimates and the distribution of the latent variables, and a maximization step (M-step), where these expectations are maximized with respect to the model parameters to obtain updated estimates.

In the E-step, one computes the expected sufficient statistics or the conditional distribution of the latent variables given the observed data and current parameters. In the M-step, these expectations are used to produce parameter updates that maximize the expected log-likelihood. Each iteration increases or preserves the observed-data likelihood, and the process converges to a stationary point, typically a local maximum rather than a guaranteed global optimum.

EM is widely used for models with latent variables or incomplete data, including Gaussian mixture models, hidden Markov models, and factor analysis, as well as for imputing missing values. It provides a general framework for maximum likelihood estimation when direct optimization is difficult due to hidden structure.

Extensions and variants of EM include ECM (expectation conditional maximization), ECME, Monte Carlo EM (which uses sampling to approximate expectations), and Variational EM (which incorporates Bayesian ideas). While powerful, EM can be sensitive to initialization, may converge slowly near a maximum, and can become trapped in local optima.