wordpiece

WordPiece is a subword tokenization method used in natural language processing to convert text into a fixed-size vocabulary of tokens. It enables robust handling of rare and unseen words by decomposing them into smaller, known units, reducing out-of-vocabulary issues while keeping the vocabulary size manageable. WordPiece tokens can include a prefix marker to indicate that a piece is a continuation of a word.

How it works: WordPiece builds its vocabulary iteratively starting from individual characters and merging pairs of symbols to form larger subword units. Each merge is chosen to maximize the likelihood of the training corpus under a probabilistic language model. The result is a fixed vocabulary of subword tokens, typically including a special marker to denote continuation pieces. For tokenization, a word is segmented into vocabulary tokens that maximize the model’s overall likelihood; if a word is already in the vocabulary, it is emitted as a single token; otherwise it is split into smaller pieces from the vocabulary.

History and usage: WordPiece was introduced by Google researchers in 2016 in the paper WordPiece: A simple and efficient subword tokenizer for neural machine translation. It gained prominence as the tokenization scheme for BERT, aiding handling of multilingual and morphologically varied text.

Relation to other methods: WordPiece is related to byte-pair encoding (BPE) but employs a likelihood-based training objective rather than pure frequency-based merges. It shares the subword approach with other tokenizers, but its specific optimization and vocabulary construction distinguish it from alternative methods.

Implementation and impact: WordPiece has been implemented in major NLP frameworks and widely incorporated in transformer-based models. Typical vocabularies consist of tens of thousands of tokens, balancing expressiveness with computational efficiency. Its use improves handling of morphological variation and reduces fragmentation of words into meaningless units.