Tokens

Tokens are digital units used to represent access, identity, value, or data. In computing, a token is a stand-in for something else, enabling secure, stateless operations such as authentication, authorization, and data handling without exposing underlying credentials. Tokens can be issued, validated, and revoked by trusted services to control access and track usage.

Authentication and authorization tokens include session tokens, API tokens, and bearer tokens such as JSON Web Tokens (JWT). OAuth 2.0 access tokens provide delegated access with scoped permissions and expiration. Tokens are typically issued by an identity provider and presented to a resource server to access protected resources. Security considerations include protecting token confidentiality, binding tokens to recipients, implementing revocation, rotating credentials, and setting appropriate lifetimes.

In blockchain and cryptocurrency, tokens represent digital assets or rights within a network. Fungible tokens, such as those following the ERC-20 standard, are interchangeable units. Non-fungible tokens, like ERC-721, represent unique assets. Other token categories include security tokens, which may represent shares or stakes, and governance tokens, which confer voting rights. Tokens can thus symbolize currency, tokenized assets, access rights, or other value within decentralized applications.

In natural language processing, a token is the basic unit produced by a tokenizer. Tokens can be words, subwords, or characters, depending on the tokenization scheme. Tokenization is a preprocessing step for text analysis, indexing, and model training, and it influences vocabulary size, handling of punctuation, and language-specific features.

Security hardware tokens and one-time password devices provide additional methods of authentication by generating short-lived codes or cryptographic proofs. In data handling, tokenization often refers to replacing sensitive data with non-sensitive placeholders to reduce exposure during processing.