Reparameterising

Reparameterising, or parameterising, is the practice of rewriting a model or probability distribution in terms of alternative parameters or latent variables. The aim is to modify the representation to improve properties such as identifiability, numerical stability, or computational efficiency in estimation and inference.

In Bayesian inference, reparameterisation often reduces correlations between parameters or between parameters and data, which can improve the convergence and mixing of samplers. A common example is switching from a centered to a non-centered parameterisation in hierarchical models, where latent effects are expressed as standardised variables plus scale and location parameters. Reparameterisation can also make prior specification or posterior exploration more robust when parameters are constrained or highly scale-dependent.

In optimization and machine learning, reparameterisation helps enforce constraints and improve conditioning. For positive quantities, parameters may be redefined as exponentials of unconstrained variables; for components that must sum to one, a simplex parameterisation may employ softmax or related transforms. A notable technique is the reparameterisation trick used in variational autoencoders, where sampling from a complex distribution is reformulated as a deterministic transformation of a simple noise variable, enabling straightforward gradient-based optimisation.

From a mathematical perspective, changing parameterisations requires transforming probability densities via the Jacobian determinant of the transformation, to preserve proper probabilistic reasoning. This introduces considerations about identifiability and interpretability, and may require back-transformation for interpretation of results.

Limitations include potential loss of interpretability, the need to account for the Jacobian, and the possibility that a reparameterisation offers little practical benefit for a given problem.