noisecontrastive

Noise-contrastive estimation (NCE) is a statistical method for estimating parameters of unnormalized probabilistic models by turning density estimation into a supervised classification task. The idea is to compare samples from the target data distribution with samples drawn from a known noise distribution. The model defines an unnormalized density, often expressed as an exponential-family score function of the input, and learning proceeds by training a binary classifier to distinguish real data from noise.

During training, for each data example, multiple noise samples are drawn. The classifier's goal is to assign high probability to data and low probability to noise. The objective is the log-likelihood of the correct labels with respect to the parameters; this yields gradients that update the scoring function. Because the model is unnormalized, the partition function does not need to be computed at training time, enabling scalable estimation for large models. Under suitable conditions, the estimator is consistent for the parameters (and the normalization constant can be recovered with extra steps).

Noise distribution selection and the number of noise samples per data point (k) influence efficiency and accuracy. Choosing a noise distribution close to the data distribution improves learning. NCE has been applied in natural language processing, computer vision, and energy-based modeling, particularly when exact normalization is intractable. It also influenced contrastive learning approaches that use similar discrimination-based objectives to learn representations.

Historical note: Noise-contrastive estimation was introduced by Martin Gutmann and Aapo Hyvärinen in 2010, with subsequent analyses on consistency and practical considerations. The method remains a practical tool for training unnormalized models and for density estimation where computing the partition function is prohibitive.