tokenizer

A tokenizer is a software component that converts a raw text stream into a sequence of tokens, the basic units used for further processing. Tokenization is a common first step in natural language processing, as well as in compilers and interpreters for programming languages.

Tokenization methods vary. Rule-based tokenizers use patterns or regular expressions to split text, while statistical or neural approaches learn how to segment text from labeled data. The tokens produced can be characters, whole words, or subword units such as Byte-Pair Encoding (BPE) or WordPiece.

In NLP pipelines, tokenizers typically map each token to an integer id in a fixed vocabulary and may generate special tokens to indicate sentence boundaries, padding, or classification tasks. They also handle punctuation, case normalization, and language-specific issues, and may operate with language models that rely on subword representations to manage unknown words.

In programming languages, a tokenizer, often called a lexer, analyzes source code and emits tokens with types such as identifier, keyword, literal, and operator. This stage feeds the parser and helps enforce the language’s syntax.

Challenges in tokenization include handling multilingual text, emojis, hyphenation, contractions, and scripts without clear whitespace boundaries. Tokenizers balance granularity, speed, and memory use, and the choice of method can influence downstream performance and generalization.

Common families of subword tokenizers include BPE-based, WordPiece-based, and unigram models. Tokenization is a foundational step in many AI systems and language tools, shaping vocabulary size, OOV handling, and model efficiency.