subtoken

A subtoken is a fragment of a word or a symbol that is used in natural language processing. Instead of treating an entire word as a single unit, subtokens break down words into smaller, more manageable pieces. This approach is particularly useful for handling rare words, misspellings, or words with complex morphology (word structures). For example, the word "unhappiness" might be broken down into subtokens like "un", "happi", and "ness".

The use of subtokens is a core component of many modern natural language processing models, especially those based on transformer architectures like BERT and GPT. These models learn representations for individual subtokens and then combine them to understand the meaning of whole words and sentences. This allows the models to generalize better to unseen words and to represent a much larger vocabulary efficiently.

Subtokenization typically involves using algorithms that learn common subword units from a large text corpus. Popular subtokenization algorithms include Byte Pair Encoding (BPE), WordPiece, and SentencePiece. Each algorithm has its own method for determining the optimal set of subtokens. BPE, for example, iteratively merges the most frequent pairs of characters or subtoken sequences. WordPiece, used by BERT, prioritizes merges that maximize the likelihood of the training data. SentencePiece treats the input as a sequence of Unicode characters and learns a subword vocabulary. By leveraging subtokens, NLP models can achieve better performance on a wide range of tasks, from translation and text generation to sentiment analysis and question answering.