Word2VecCBOW

Word2VecCBOW, short for Continuous Bag-of-Words, is a neural-network-based method for learning dense vector representations of words from large text corpora. It is one of the two architectures in the Word2Vec family, the other being Skip-gram.

CBOW trains to predict a target word from its surrounding context words within a fixed window. Given a context window of size m, the model uses the average (or sum) of the input vectors for the context words as the hidden representation. A softmax layer computes the probability distribution over the vocabulary for the target word, and the model is trained to maximize the probability of the actual center word. The weights connecting input and hidden layers are the word embeddings.

Training uses either hierarchical softmax or negative sampling for efficiency. In practice, due to the large vocabulary, negative sampling is common, which approximates the softmax by distinguishing the true target from noise samples.

Compared with the Skip-gram model, CBOW tends to perform better with larger datasets and frequent words, while Skip-gram can perform better on rare words. Embeddings learned by CBOW are dense, low-dimensional, and capture semantic and syntactic relationships, enabling vector arithmetic such as "king" - "man" + "woman" ≈ "queen".

Applications include providing feature representations for downstream NLP tasks, initialization for neural networks, similarity searches, clustering, and transfer learning. Limitations include a single, context-independent embedding per word, inability to resolve polysemy, reliance on non-contextualized context, and sensitivity to corpus quality and window size.

Word2VecCBOW was introduced by Mikolov et al. in 2013 as part of the Word2Vec framework; many pretrained embeddings and tools exist in various libraries.