Embeddings

Embeddings are mappings from discrete items to dense, real-valued vectors in a continuous space. They are designed so that geometric relationships among vectors reflect semantic or syntactic relationships among the items. This dense representation contrasts with one-hot encodings, which are high-dimensional and sparse. The arrangement of vectors often enables simple operations, such as measuring similarity with cosine similarity or performing vector arithmetic that mirrors linguistic relations.

Word embeddings are a common focus. Static word embeddings assign a single vector to each token, learned from large text corpora using methods such as Word2Vec (skip-gram and CBOW), GloVe, or fastText. Static embeddings have limited handling of polysemy and out-of-vocabulary words. Contextual embeddings, produced by models like BERT or GPT, generate token representations that vary with surrounding text, capturing context-dependent meaning. FastText improves handling of rare or misspelled words by incorporating subword information. Sentence or document embeddings can be created by aggregating word vectors or by specialized models such as Sentence-BERT.

Graph embeddings map nodes in a network to vectors that preserve neighborhood structure, enabling tasks such as link prediction, node classification, and recommendation. Techniques include DeepWalk, node2vec, LINE, and graph neural networks.

Image embeddings are feature vectors extracted from neural networks trained on large image datasets, used for retrieval, clustering, and downstream recognition tasks. Training of embeddings is typically unsupervised or self-supervised, using predictive objectives, matrix factorization, or contrastive learning. Evaluation uses intrinsic tasks (similarity, analogies) and extrinsic tasks (downstream performance). Common word embedding dimensions range from about 50 to 300. Limitations include biases inherited from training data and challenges with out-of-vocabulary terms or highly dynamic contexts.