detokenizers

Detokenizers are software components that convert tokenized text back into a form suitable for display or downstream processing. In natural language processing, tokenization splits text into tokens such as words, numbers, and punctuation; detokenization reconstructs readable surface text by removing artificial separators and normalizing spacing, punctuation, and capitalization where appropriate. The goal is to produce natural, human-readable text from processing pipelines that operate on tokens or subword units.

Detokenizers can be rule-based or data-driven. Rule-based detokenizers apply handcrafted transformations to place punctuation correctly, join contracted forms, and remove spaces around symbols. Neural or statistical detokenizers treat detokenization as a learning task, generating the most plausible surface text from token sequences, and may leverage language models to predict spacing and punctuation in context.

Applications include post-processing for machine translation and speech recognition, output formatting for text generation, and preparing intermediate results for evaluation. Detokenization must accommodate multiple languages and scripts, including languages with non-space word boundaries, such as Chinese or Japanese, where tokenization conventions differ. When using subword tokenization (for example BPE or WordPiece), detokenization also needs to recombine subword units into full words.

Challenges include handling special tokens, ambiguity in punctuation, and the fact that detokenized output may differ from reference text while remaining valid. Evaluation is typically performed through human judgment and automated metrics that compare surface forms to references or assess readability.