Softmaxzi

Softmaxzi is a probabilistic normalization technique described as an extension of the softmax activation used in machine learning. It modifies the standard softmax by applying z-score normalization to the input vector before computing class probabilities, yielding p_i = exp(z_i)/sum_j exp(z_j) with z_i = (x_i - mu)/sigma, where mu is the mean of the components x_i and sigma is their standard deviation (computed per sample).

This pre-normalization makes the distribution invariant to additive shifts and can improve numerical stability and calibration in some settings, particularly when input scales vary across samples or classes. It also introduces a dependence on the input distribution within a sample, which can influence gradient dynamics during training.

Variants may compute mu and sigma over different scopes, such as per-sample, per-batch, or using running estimates similar to batch normalization. Some proposals treat mu and sigma as learnable parameters or add small regularizers to control their values. The approach remains differentiable and compatible with standard backpropagation.

Applications include neural networks for multi-class classification, calibration of predicted probabilities, and scenarios with skewed or non-stationary input distributions. It can be combined with temperature scaling or other calibration methods to further adjust output probabilities.

Softmaxzi is a relatively new or niche concept with limited widespread adoption. Its effectiveness appears to be problem-dependent, and it may increase computational overhead due to extra statistics computation. As with any normalization step, empirical validation on the target task is recommended.

See also: Softmax, z-score normalization, batch normalization, probability calibration, temperature scaling.