BPEs

BPEs, short for Byte Pair Encodings, are a family of subword tokenization algorithms used in natural language processing. They derive from a data compression technique and are employed to construct a fixed-size vocabulary of subword units by repeatedly merging the most frequent adjacent symbol pairs in a corpus. The process typically starts with a vocabulary of individual characters (or bytes in byte-level variants). The most frequent pair of neighboring symbols is merged to form a new symbol, and this continues until a target vocabulary size is reached. The resulting set of merges defines the subword units used to tokenize text.

For encoding, text is segmented by applying the learned merges to produce the longest possible subword units that appear in the vocabulary. Byte-level BPE variants operate directly on raw bytes, allowing them to handle any text without language-specific preprocessing and to deal more robustly with unusual or unseen characters.

BPEs have become widely used in NLP because they mitigate out-of-vocabulary issues while keeping vocabulary size manageable. They are particularly effective for morphologically rich languages and multilingual settings, where full word vocabularies would be impractically large. Typical BPE vocabularies range from about 8,000 to 60,000 tokens. They have influenced a range of subword tokenizers, including related approaches like WordPiece and SentencePiece, and are implemented in many NLP frameworks and pretrained models.

Limitations include dependence on the training corpus and domain, potential creation of overly small or opaque subword units, and sensitivity to hyperparameters such as vocabulary size and merge rules. Despite these considerations, BPEs remain a foundational tool for scalable and flexible text representation in modern NLP.