advi

ADVI, short for Automatic Differentiation Variational Inference, is an algorithm for approximate Bayesian inference that automates the construction of a variational posterior and uses automatic differentiation to optimize the evidence lower bound (ELBO). Introduced to make variational inference accessible without manual gradient calculations, ADVI is designed to work with complex probabilistic models by transforming parameters from constrained spaces to an unconstrained Euclidean space and then fitting a Gaussian variational family there.

The method typically uses either a mean-field Gaussian (diagonal covariance) or a full-rank Gaussian (full covariance) as the variational family. It estimates the ELBO and its gradients with respect to the variational parameters via Monte Carlo samples and the reparameterization trick, with gradients computed through automatic differentiation. This enables scalable, gradient-based optimization that can handle large datasets and high-dimensional models. The resulting approximation q(θ; φ) serves as a surrogate posterior, facilitating posterior predictive checks and uncertainty quantification.

ADVI is implemented in several probabilistic programming environments, notably Stan, and has inspired a broader ecosystem of automated variational inference tools. Its advantages include avoiding manual derivation of gradients, applying to a wide range of differentiable models, and delivering fast approximate inference. Limitations include reliance on Gaussian variational families that may underestimate uncertainty and struggle with multimodality or highly skewed posteriors, as well as potential difficulties with discrete or non-differentiable components. As a result, ADVI is commonly used as a fast baseline or starting point before more expressive variational families or exact methods are applied.