MLEs

Maximum likelihood estimation, or MLE, is a method of estimating the parameters of a statistical model. Given data x1, …, xn assumed to be drawn independently from a distribution with density or mass function f(x|θ), the likelihood is L(θ) = ∏ f(xi|θ). The maximum likelihood estimator is any θ̂ that maximizes L(θ) (equivalently the log-likelihood ℓ(θ) = ∑ log f(xi|θ)). For many models, θ̂ is found by solving score equations ∂ℓ/∂θ = 0 or by numerical optimization.

Under regularity conditions, θ̂ is consistent and asymptotically normal with √n(θ̂ − θ) →d N(0, I(θ)−1), where I(θ) is the Fisher information. Inference often uses the observed or expected information to estimate standard errors. The MLE is invariant under reparameterization: the MLE of φ = g(θ) is φ̂ = g(θ̂).

Computationally, closed-form solutions exist for some models (e.g., the mean of a normal distribution with known variance). In many cases, numerical methods are required (Newton-Raphson, gradient ascent, or the EM algorithm for incomplete data).

Advantages of MLE include consistency and efficiency under correct model specification. Limitations include sensitivity to model misspecification, potential bias in small samples, multiple local maxima, and non-identifiability. Penalized likelihood or Bayesian methods can address some issues, leading to penalized or MAP estimates.

Examples: estimating the mean of a normal distribution with known variance; estimating the rate parameter λ of an exponential distribution; fitting logistic regression by maximizing the Bernoulli likelihood.