detokenisering

Detokenisering, also written detokenisering in some Scandinavian languages, is the process of converting a sequence of tokens into a fluent natural language text. It is the inverse operation of tokenization and aims to restore appropriate spacing, punctuation, and formatting that may have been altered during tokenization.

In natural language processing, detokenization is performed after models generate token sequences. It is an essential step in applications such as machine translation, language modeling, speech recognition, and text-to-speech, where human-readable output is required.

Detokenization faces several challenges. Languages differ in punctuation rules, spacing around punctuation, and the treatment of quotes, dashes, abbreviations, and numbers. Many modern tokenizers split text into subword units (such as Byte-Pair Encoding or WordPiece), requiring careful merging back into complete words. Hyphenation, clitics, capitalization, and language-specific conventions (for example non-breaking spaces in some languages) add further complexity.

Approaches to detokenization include rule-based methods, which encode language-specific spacing and punctuation rules, and statistical or neural methods, which learn detokenization patterns from data. Many NLP toolchains combine both approaches. Well-known examples include rule-based detokenizers used in the Moses project and various neural post-processing components in contemporary systems.

Evaluation of detokenization accuracy can be manual or automatic. Common measures include detokenization error rates and the impact of detokenization on downstream metrics such as BLEU or ROUGE. In practice, detokenization is typically tuned to the target language and the specific tokenization scheme used in the preceding steps.