lexers

Lexers, or lexical analyzers, are a core component of compilers and interpreters. They read source text and convert it into a stream of tokens. Each token has a type, such as identifier, keyword, literal, operator, or punctuation, and often carries the token’s text and its position in the source file. The lexer also removes whitespace and comments and applies basic normalization when appropriate.

Lexical analysis is typically implemented with regular expressions and finite automata. A lexer scans input using a maximal munch rule, selecting the longest valid token at each step. It supports features such as multi-character operators, string and numeric literals, escape sequences, and error handling for illegal characters.

Lexers can be written by hand or generated from specifications using tools such as Lex or Flex, ANTLR, or re2c. The produced token stream is consumed by a parser, which applies a formal grammar to build a parse or abstract syntax tree. The lexer may also provide line and column information to aid error reporting.

Beyond compilers, lexers are used in syntax highlighting, code analysis, and data extraction. They are typically deterministic and must be fast, as they run before parsing or interpretation.

In practice, lexers and parsers are designed to work together, though some languages require context-sensitive lexing or separate phases for certain constructs. The distinction is that a lexer identifies tokens, while a parser expresses the language grammar over those tokens.