centerembeddings

Centerembeddings is a term used in representation learning to describe embedding vectors that have been adjusted to have zero mean across a collection of items. The typical operation, called centering, involves subtracting the global mean vector mu from each embedding v_i: mu = (1/n) sum_i v_i, v_i' = v_i - mu. After centering, the set of vectors has approximately zero average, which can change their geometric properties in ways that are beneficial for downstream tasks. In some pipelines, centering is followed by normalization to unit length.

Variants include per-dimension centering, where the mean is subtracted separately for each coordinate, and online or streaming centering, where the mean is updated incrementally as new embeddings arrive.

Purposes and effects: centering removes the overall offset of the embedding space, which can improve the interpretability of similarities and distances, help with alignment across models or modalities, and stabilize optimization in downstream tasks such as clustering or retrieval. In kernel and linear methods, centering is closely related to using the centering matrix H = I - (1/n)11^T to center data in feature space.

Limitations: centering can discard information contained in the global mean, which may be meaningful in some tasks. It also depends on the choice of dataset over which mu is computed; re-centering may be required when embedding sets change significantly or when using embeddings from different sources.

See also: mean-centering, zero-mean, normalization, standardization, whitening, embedding alignment.