Tokenise

Tokenise is an English verb meaning to divide a text or data sequence into tokens. In natural language processing, tokenisation is the process of converting a text string into a sequence of tokens, typically words, subwords, or punctuation. Tokenisation is a prerequisite for many linguistic analyses and downstream tasks such as parsing and machine learning.

In data security, tokenisation refers to replacing sensitive data with non-sensitive placeholders, or tokens, that map to the original data through a secure vault. Tokens preserve the original data’s format and length to minimize system changes. Tokenisation is designed to reduce exposure of sensitive data and is distinct from encryption, as tokens have no intrinsic meaning and can be reversible only via the vault.

In finance and blockchain, tokenisation refers to representing rights to an asset as a digital token on a distributed ledger. Fungible tokens and non-fungible tokens are common forms. Tokenisation can enable fractional ownership, programmable rights, and faster transfer, but raises legal, regulatory, and custody considerations.

In NLP applications, tokenisation approaches range from simple whitespace or punctuation-based tokenisation to advanced subword tokenisation methods such as Byte Pair Encoding, WordPiece, and SentencePiece, which help manage large vocabularies and rare words. Tokenisers must handle multilingual text, language-specific boundaries, and edge cases such as contractions.