skipgram

Skip-gram is a neural network model used to learn dense vector representations of words, often referred to as word embeddings. It was introduced as part of the Word2Vec framework by Tomas Mikolov and colleagues at Google in 2013. The core idea is to use a target word to predict its surrounding context words within a fixed window.

In practice, the model takes a center word and aims to maximize the probability of nearby words appearing in the context. For each position in a text, the center word w_t is used to predict the sequence of context words w_{t+j} for j in a chosen window around the center. The traditional formulation uses a shallow neural network with one hidden layer: the input is a one-hot vector for the center word, the hidden layer produces a continuous embedding, and the output layer attempts to assign high probabilities to actual context words.

Training is typically expensive with a full softmax over a large vocabulary, so efficient approximations are commonly used, such as negative sampling or hierarchical softmax. Negative sampling updates only a small number of word vectors per training example, while hierarchical softmax uses a binary tree to reduce computation.

Skip-gram tends to perform well on infrequent words and captures semantic and syntactic relationships, which can be assessed through analogy and similarity tasks. It contrasts with the Continuous Bag of Words (CBOW) model, which predicts the center word from its context rather than the reverse. Subword information, as in fastText, can be incorporated to improve handling of rare or unseen words.