Tokenization

Tokenization is the process of breaking text into units called tokens, which can be words, subwords, or characters. It is a foundational step in natural language processing and other text-based computing tasks. The resulting stream of tokens is used as input to algorithms, models, or search systems.

There are rule-based tokenizers that apply language-specific syntax and punctuation rules, and statistical or neural approaches that learn token boundaries from data. Subword tokenization methods, such as Byte-Pair Encoding, WordPiece, and SentencePiece, split rare or unknown words into smaller known units, enabling stable vocabularies. Each token is typically mapped to an identifier for embedding or indexing.

Tokenization is essential for language models, text classification, information retrieval, and many NLP pipelines. Challenges include language diversity, different scripts, and inconsistent punctuation. Some languages have no clear word boundaries, like Chinese or Japanese, requiring character-level or specialized segmentation. Handling contractions, hyphenation, and proper names also affects downstream performance.

In security and data management, tokenization can also refer to replacing sensitive data with non-sensitive tokens to protect privacy and simplify compliance. This form is unrelated to linguistic tokenization but shares the same goal of abstracting original data. In finance and blockchain, tokens can represent assets or rights on a distributed ledger.