OutofVocabularyProbleme

Out of vocabulary (OOV) refers to tokens that are not present in the vocabulary used by a natural language processing system. In many NLP models, a fixed vocabulary maps words to embeddings or probability distributions. When a word or token not included in this set is encountered, it is treated as OOV, triggering fallback mechanisms designed to maintain processing.

OOVs arise for several reasons. New names and neologisms, technical jargon, multilingual input, misspellings, and morphological variations can all fall outside a model’s vocabulary. Languages with rich morphology tend to produce many surface forms of a word, increasing OOV risk. In dynamic text streams, such as social media, the appearance of creative spellings and memes further contributes to OOV occurrences.

The presence of OOVs can impact performance in language modeling, translation, search, and information retrieval. Common remedies replace unseen tokens with a generic unknown token, but this can obscure important information. More sophisticated approaches aim to preserve information through alternative representations.

Mitigation strategies include subword tokenization methods such as byte-pair encoding (BPE), WordPiece, and SentencePiece, which break words into smaller units that are more likely to appear in training data. Character-level models and hybrid approaches model text at multiple granularities. Dynamic vocabularies, transliteration for proper nouns, and morphological analysis to decompose words into known morphemes are additional techniques. Each method balances coverage, efficiency, and contextual fidelity, and the choice often depends on the application and data domain.