BPE

Byte Pair Encoding (BPE) is a tokenization method used in natural language processing to convert text into a sequence of subword units. It originated as a simple data compression technique but gained prominence in machine translation and language modeling after being adapted to create a fixed-size vocabulary of frequently co-occurring character sequences. BPE helps bridge the gap between character-level representations and word-level models.

How it works: start with an initial alphabet consisting of individual characters or bytes. Scan a large corpus to count the frequency of adjacent symbol pairs. Replace the most frequent pair with a new symbol, and repeat this merging process a fixed number of times. After all merges, the final vocabulary contains both original characters and merged subword units, which are used to tokenize text.

Applications and benefits: by representing words as sequences of subword units, BPE reduces out-of-vocabulary words and decreases the size of the final vocabulary while preserving enough granularity to reconstruct most inputs. It is language-agnostic, works well for morphologically rich languages, and integrates smoothly with neural models for training and inference. Common settings produce tens of thousands of subword tokens.

Variants and related methods: BPE-based approaches include WordPiece and SentencePiece, which introduce probabilistic or language-model based criteria during segmentation. BPE is often used with neural machine translation and large language models, sometimes in combination with dropout or other regularization techniques to improve generalization.

Limitations: BPE's merges depend on the training corpus and can yield token boundaries that are linguistically opaque. It may struggle with extremely rare words or multilingual text where scripts differ. Despite these limitations, BPE remains a widely adopted, efficient, and effective subword tokenization method in modern NLP.