Hscores

Hscores, commonly called Hyvärinen scores, are a class of proper scoring rules used to evaluate and fit probabilistic density models, especially unnormalized ones. They are local in nature, depending on derivatives of the log-density, and do not require knowledge of the normalization constant.

The concept originates from Aapo Hyvärinen’s score matching approach for estimating unnormalized models without computing the partition function. For a parametric density p_theta(x) with log-density l_theta(x) = log p_theta(x), the multivariate Hyvärinen score at a point x is S_theta(x) = sum_i ∂^2 l_theta(x)/∂x_i^2 + 1/2 sum_i [∂ l_theta(x)/∂x_i]^2. The empirical risk is the average of S_theta over data, and parameter estimates minimize this risk.

Hyvärinen scoring can be extended to higher dimensions and used with gradient-based optimization, since it relies on derivatives of the log-density rather than the normalization constant. It is invariant to additive constants in l_theta and is well suited for models where the partition function is intractable.

Applications include training energy-based models and other unnormalized density models, density estimation for complex or high-dimensional data, and anomaly detection. Advantages include avoiding the partition function and leveraging local information from the log-density; limitations involve the need for differentiable densities, reliable gradient calculations, and less widespread adoption compared with maximum likelihood in some domains.

Related topics include score matching and other proper scoring rules.