sentencepiece

SentencePiece is an open-source text tokenizer and detokenizer designed for neural network-based natural language processing. It provides language-independent subword units by learning a vocabulary directly from raw text, without relying on language-specific tokenization rules. It is widely used in machine translation and other sequence labeling tasks.

It supports two model types: BPE (Byte Pair Encoding) and a Unigram language model. It operates on Unicode input and can be used with or without an explicit pre-tokenization step. It produces a fixed-size vocabulary of subword units and can handle large character sets, enabling consistent tokenization across languages and scripts. The tool is language-agnostic and suitable for multilingual corpora.

Training and encoding: Users train a model with spm_train on a corpus, specifying parameters such as model_type, vocab_size, character_coverage, and input_sentencepiece. After training, encoding and decoding can be performed with spm_encode and spm_decode. The resulting piece sequences replace words with subword tokens, reducing out-of-vocabulary issues.

Implementation and availability: SentencePiece is implemented in C++ with Python bindings and released as open-source software under the Apache 2.0 license. The project provides command-line utilities and APIs for integration into NLP pipelines and popular frameworks.

See also: subword tokenization, Byte-Pair Encoding, Unigram language model, neural machine translation.