Tokentypes

Tokentypes are the categories used by a lexical analyzer to classify sequences of characters into tokens during the first phase of compiling or interpreting a program. Each token has a type and a value; the type identifies the syntactic category of the token, while the value stores the literal text matched. Token types are language-specific and defined in a language’s lexical specification. Common token types include identifiers, keywords, literals (such as integers, floating-point numbers, strings, and booleans), operators, punctuation or separators (such as commas, semicolons, parentheses, and braces), comments, and end-of-file markers. In many languages, whitespace is treated as ignorable or as a separate token class only in certain contexts.

Token types are typically defined by rules expressed as regular expressions or simple pattern actions. The lexer scans the input and applies the rules to produce a stream of tokens, usually following concepts such as the longest-match rule and a specified rule priority. The resulting token stream is then consumed by the parser, which uses the token types to apply the language’s grammar and construct an abstract syntax tree.

A notable design choice is how keywords are handled: some languages assign distinct token types for reserved words (IF, ELSE, WHILE), while others recognize them as identifiers and differentiate them via a symbol table. Whitespace and comments are often discarded or kept as separate tokens for error reporting or tooling support. Token types can be implemented directly or generated by lexer generators such as Lex, Flex, or ANTLR, and they form the bridge between raw source text and the structured representation used by parsers and compilers.