pretokenization

Pretokenization is a preprocessing step in natural language processing that converts raw text into an initial, coarse-grained representation before the main tokenization stage. The goal is to standardize input, reduce variability, and make downstream encoding more predictable for models that operate on tokens or subwords.

Pretokenization usually involves breaking text into manageable units while controlling how punctuation, whitespace, and Unicode elements are handled. Common techniques include whitespace tokenization, punctuation splitting, and Unicode normalization; grapheme cluster segmentation for user-perceived characters; and case folding. Some pipelines employ byte-level representations to avoid issues with rare characters and scripts.

Pretokens feed into subword tokenizers such as BPE or SentencePiece. After pretokens, the chosen subword model segments the units into subword tokens, which may be further mapped to vocabulary indices. Pretokens are often configurable and may include language-specific rules, URL or number handling, and contractions.

Trade-offs: pretokens influence vocabulary size, token granularity, and processing speed. An overly aggressive pretokens scheme can fragment useful words; too coarse may fail to capture distinctions. The choice depends on language, domain, and model architecture. Pretokenization is widely used in modern NLP pipelines to enable consistent handling of diverse text, support multilingual data, and improve the performance of subsequent tokenization and modeling steps.