RNADE

RNADE, or Real-valued Neural Autoregressive Distribution Estimator, is a probabilistic generative model for real-valued data. It factors the joint density of a D-dimensional vector x into an autoregressive product: p(x) = ∏_{i=1}^D p(x_i | x_1, ..., x_{i-1}). Each conditional distribution is modeled with a neural network that outputs the parameters of a real-valued distribution for x_i given the preceding components.

In RNADE, the conditionals are typically modeled as mixtures of Gaussians. The neural network provides the mixture weights, means, and variances for p(x_i | x_{<i}). Conditioning on the previous components is realized through a shared hidden representation or sequential processing, enabling parameter sharing across conditionals and scalable training. The use of Gaussian mixtures allows the model to capture multimodal and complex conditional structures.

Training is performed by maximum likelihood using backpropagation, with exact computation of the log-likelihood afforded by the autoregressive factorization. Sampling proceeds naturally in a forward pass: sample x_1 from p(x_1), then x_2 from p(x_2 | x_1), and so on, producing samples from the joint distribution.

RNADE is related to other autoregressive density estimators such as NADE and MADE and belongs to a class of models that aim for tractable, likelihood-based density estimation of real-valued data. It has been applied to various high-dimensional real-valued datasets, including image patches and audio features. Limitations include sensitivity to the order of variables, computational cost for high-dimensional data, and the need to manage the number of mixture components to control model size.