MLE

Maximum likelihood estimation (MLE) is a method of estimating the parameters of a statistical model by choosing the values that maximize the likelihood function, the probability of observing the given sample under the model. Given independent observations x1, ..., xn drawn from a distribution with density or mass function f(x; theta), the likelihood is L(theta) = product of f(xi; theta). The log-likelihood is l(theta) = sum of log f(xi; theta). The MLE theta_hat is any value that maximizes L (or l). In many cases, derivatives can be set to zero; if a closed-form solution exists, it can be solved analytically; otherwise numerical optimization methods are used. MLEs are invariant under reparameterization: if theta_hat maximizes L, then g(theta_hat) maximizes L as a function of g(theta).

Common examples illustrate the approach. For data Xi ~ N(mu, sigma^2) with known sigma^2, the MLE for mu is the sample mean mu_hat = x-bar. For an exponential model with rate lambda, the MLE is lambda_hat = n / sum Xi.

Properties of MLEs are studied under regularity conditions. Consistency means theta_hat converges to the true parameter theta as the sample size n grows. Under regularity, MLEs are asymptotically normal: sqrt(n)(theta_hat - theta) converges in distribution to a normal with mean 0 and variance I(theta)^{-1}, where I is the Fisher information. Asymptotically, the MLE is efficient in the sense that it achieves the Cramér-Rao lower bound among regular estimators.

Limitations include dependence on correct model specification and regularity conditions, sensitivity to outliers, potential for multiple local maxima, and numerical challenges. In small samples, MLEs can be biased, and identifiability issues may arise; in such cases, regularization or Bayesian approaches may be considered.