Word2Vec

word2vec is a family of neural network models that produce dense vector representations of words from large text corpora. Developed by Tomas Mikolov and colleagues at Google and first described in 2013, the models learn embeddings that place semantically similar words closer in a continuous vector space. These embeddings can be used to measure similarity, perform arithmetic on meanings, and serve as features in downstream NLP tasks.

word2vec encompasses two main architectures: Continuous Bag of Words (CBOW) and Skip-gram. CBOW predicts a target word from the surrounding context words, while Skip-gram does the opposite, predicting context words from a given target word. Both are implemented as shallow, single-hidden-layer neural networks that take one-hot word representations as inputs and produce probability distributions over the vocabulary.

To train efficiently on large vocabularies, word2vec uses optimization techniques such as hierarchical softmax or negative sampling. These methods reduce computational cost and enable training on very large corpora, such as Google News. Typical embedding dimensionalities range from about 100 to a few hundred, and training often relies on subsampling of frequent words to improve performance.

Word vectors capture many linguistic regularities, enabling simple vector operations such as king minus man plus woman approximating queen. They are useful for measuring word similarity, clustering, and serving as input features for downstream tasks. However, word2vec produces static embeddings that assign a single vector to a word regardless of sense, and embeddings can reflect biases present in the training data. Contextualized models released later, such as ELMo and BERT, address these limitations.

Word2vec had a major impact on natural language processing by providing a scalable, effective way to learn word representations from raw text. While newer models have supplanted it for many applications, word2vec remains a common baseline and a foundational concept in embedding methods and semantic analysis.