CBOW

CBOW, short for Continuous Bag of Words, is a neural network model used in the Word2Vec family to learn dense vector representations of words from large text corpora. It is trained to predict a target word from its surrounding context words within a fixed window.

In CBOW, the input consists of several context words. Each context word is mapped to a continuous vector, and these vectors are averaged (or summed) to form a single context representation. This representation is then fed into a shallow neural network with one hidden layer to produce a probability distribution over the vocabulary. The model is trained to maximize the probability of the actual target word given the context, typically using cross-entropy loss. After training, the learned input-to-hidden weights serve as the word embeddings.

Training CBOW efficiently over large vocabularies often involves approximations such as negative sampling or hierarchical softmax, since computing a full softmax over all words can be expensive. A typical setup uses a window size that determines how many words on each side are considered as context, such as two to five tokens.

CBOW is contrasted with the Skip-gram model, which learns to predict context words from a target word. In practice, CBOW tends to perform well for frequent words and benefits from larger corpora, producing embeddings that reflect semantic and syntactic relationships. These embeddings are widely used in downstream NLP tasks, with similarity between words measured by cosine similarity in the embedding space.

Limitations include reliance on fixed context windows, ignoring word order and long-range dependencies, and reduced effectiveness for rare words. Variants and extensions, including negative sampling and subword information, address some practical concerns. CBOW remains a foundational approach in neural word representation learning.