tokenisaation

Tokenisaation, or tokenization, is a term used in multiple domains to describe the process of transforming a sequence into tokens or discrete units. In general, it refers to breaking data into smaller pieces that are easier to process, store, or analyze. In data security, tokenisation replaces sensitive information with non-sensitive placeholders, or tokens, that retain some utilitarian value without exposing the original data.

In natural language processing (NLP), tokenisation is the preprocessing step that splits text into tokens such as words, numbers, or punctuation marks. Tokens form the basic units for subsequent analysis, tagging, or modeling. Languages differ in tokenisation challenges: some have clear word boundaries, while others (like Chinese or Thai) require specialized segmentation. Tokenisation methods range from simple whitespace and punctuation-based rules to more sophisticated statistical or neural approaches.

A common development in NLP is subword tokenisation, which breaks text into smaller units than words, such as syllables or frequent character sequences. Techniques like Byte-Pair Encoding (BPE), WordPiece, and SentencePiece enable models to handle rare or unseen words more effectively by building a controllable vocabulary of tokens.

In data security, tokenisation substitutes sensitive data with tokens that can be mapped back to the original values only by authorized systems. Tokens may be reversible or non-reversible and are used to reduce exposure of payment card numbers, personal data, or other identifiers. Tokenisation is distinct from encryption and is often implemented in token vaults or third-party services to support compliance with privacy and security regulations.

Overall, tokenisaation encompasses both linguistic text processing and data protection practices, with methods and implications that vary by domain.