tokenisation

Tokenisation is the process of converting input data into tokens that stand in for the original content. In computing, the term has several uses, including linguistic tokenisation, data security tokenisation, and the tokenisation of assets in finance and blockchain contexts. The common idea is to replace sensitive or complex input with simpler, more manageable representations that can be mapped back to the original data under controlled conditions.

In natural language processing and information retrieval, tokenisation refers to dividing text into tokens such as words, subwords, or punctuation marks. The chosen unit depends on the task and language, ranging from simple whitespace splits to more sophisticated approaches that handle clitics, punctuation, and multiword expressions. Tokenisation enables downstream processing such as parsing, indexing, and model training, but it faces challenges with multilingual scripts, ambiguous boundaries, and nonstandard text.

In data security, tokenisation substitutes sensitive data (for example, payment card numbers) with non-sensitive tokens. The mapping between tokens and the original data is stored in a secure token vault, accessible only to authorized systems. Tokens can be used in processing without exposing underlying data, reducing risk in environments like payment processing. Reversal requires secure access controls and governance over the vault.

In finance and asset management, tokenisation represents ownership or claims as digital tokens on a blockchain or distributed ledger. This can enable fractional ownership, programmable rights, and easier transfer of assets such as real estate or art. Regulatory, custody, and rights frameworks govern issuance and transfers.

Notes and limitations: tokenisation is not encryption; it relies on secure token vaults and governance. Interoperability, performance, and risk management are ongoing considerations.