tokenizers

Tokenizers are software components used in natural language processing to convert raw text into a sequence of tokens that can be processed by models. They also map tokens to numerical identifiers in a model's vocabulary and may apply normalization, case handling, and language-specific rules. A tokenizer aims to produce stable, reproducible inputs while controlling for vocabulary size and handling unknown words.

Tokenizers are typically categorized by granularity: character-level tokenizers split text into individual characters; word-level tokenizers split on whitespace and punctuation; and subword tokenizers strike a balance by decomposing rare or unknown words into smaller units. Subword approaches are widely used in modern models because they reduce vocabulary size and improve robustness to out-of-vocabulary words.

Leading subword algorithms include byte-pair encoding (BPE), WordPiece, and unigram language model tokenizers. BPE starts with individual characters and iteratively merges the most frequent adjacent pairs to form subword units. WordPiece, used by BERT, uses a probabilistic criterion to decide token boundaries. Unigram, employed by SentencePiece, searches for a subword set that maximizes likelihood under a unigram model.

Tokenizers also manage pre-processing steps such as Unicode normalization, punctuation handling, and lowercasing, as well as the insertion of special tokens for model tasks. They must be deterministic, support multilingual text, and be able to operate efficiently on long inputs. The choice of tokenizer affects model performance, speed, and the ability to generalize to unseen text.

Common tooling includes libraries that implement fast tokenizers and pre-tokenization pipelines, such as HuggingFace's tokenizers, which provide BPE and WordPiece variants, and SentencePiece. In practice, tokenizers form the first step in training, evaluating, and running language models.