WordPieces

WordPieces are a subword tokenization method used in natural language processing. They represent a compromise between word-level and character-level tokenization. Instead of treating each word as a single unit, WordPieces break down rare or unknown words into smaller, meaningful subword units. This approach allows models to handle a larger vocabulary and better understand out-of-vocabulary words by composing them from known subwords.

The process of creating WordPieces typically involves a greedy algorithm. It starts with a base vocabulary of characters and iteratively merges the most frequent pairs of existing tokens to form new, longer tokens. This continues until a desired vocabulary size is reached. The resulting vocabulary contains common words, frequently occurring subword units, and individual characters for less common parts.

When a new piece of text is tokenized using a pre-trained WordPiece model, it is first split into words. Then, each word is greedily matched against the WordPiece vocabulary. If a word is present in the vocabulary, it's tokenized as is. If not, the algorithm tries to find the longest subword prefix of that word that exists in the vocabulary. This process is repeated for the remainder of the word until the entire word is represented by a sequence of WordPieces. Unknown words that cannot be broken down into known subwords are often represented by a special unknown token.

WordPiece tokenization is widely used in transformer-based language models such as BERT and its successors. Its ability to balance vocabulary size, handle rare words, and represent words compositionally contributes to the strong performance of these models on various NLP tasks.